Autonomous Incident Triage Agent for On-Call SREs

What This Builds

When something breaks at 2 AM, the information needed to resolve it is scattered across logs, deployment pipelines, configuration histories, and third-party monitors. An SRE responding to a page has to manually correlate telemetry from multiple sources, trace dependencies, and form hypotheses — a process that routinely takes hours.

This recipe builds an SRE agent that does the first pass for you. A PagerDuty incident triggers a LangGraph agent. The agent pulls the relevant logs, recent deploys, and config changes for the affected service, forms a ranked root-cause hypothesis, and posts a structured triage summary into the incident’s Slack channel before the human has finished reading the page.

It does not auto-remediate. It collapses the “gather and correlate” phase so the human starts the decision phase with evidence already in hand. AWS reports their managed DevOps Agent running this loop can take MTTR from hours to minutes on scoped incidents.

The Stack

• LangGraph orchestrates the agent as an explicit state graph: detect, gather, hypothesize, report. A graph (not a free-form ReAct loop) keeps the investigation auditable and bounds tool calls.
• PagerDuty is the trigger. A webhook on incident creation fires the graph; PagerDuty’s own AI agents work the same way using LangGraph.
• Amazon CloudWatch (or your log/metrics backend) is the read-only evidence source the agent queries.
• Amazon Bedrock (or any chat model) runs the hypothesis step.
• Slack is the output surface — the agent writes its findings into the incident channel.

Step-by-Step Outline

1. Register a PagerDuty webhook on incident.triggered. Point it at an HTTP endpoint that starts the LangGraph run with the incident id, service, and severity.
2. Gather node: fetch the last N minutes of error logs for the affected service from CloudWatch, plus the deploy timeline and recent config/IaC changes. Keep every tool read-only.
3. Correlate node: align the error spike against deploy and config events to find what changed just before the incident.
4. Hypothesize node: ask the model for a ranked list of likely causes, each tied to specific evidence (a log line, a deploy SHA, a config diff).
5. Report node: post a Slack message to the incident channel: one-line summary, top hypothesis with evidence links, and suggested next checks.
6. Log every run (inputs, tool calls, output) so you can evaluate accuracy and tune which signals matter most.

Start narrow: one service, one log source, hypothesis-only output. Add deploy and config correlation once the first version posts useful summaries.

Source

• AWS: Leverage Agentic AI for Autonomous Incident Response with AWS DevOps Agent
• PagerDuty: How PagerDuty Built AI Agents with LangGraph