Set up an alert-diagnosis workflow for Kubernetes incidents
An n8n flow takes Alertmanager alerts, checks several MCP-connected systems, and writes a read-only diagnostic reply in Mattermost.
- What they did
- A webhook from Alertmanager kicks off an n8n workflow. It groups duplicate alerts across a 48-hour window, pulls the PromQL query that fired from the Prometheus Rules API, then hands the case to an AI agent. That agent looks through Kubernetes, Grafana, DigitalOcean, GitHub, and a Qdrant index of internal docs, all in read-only mode, and the workflow finds the original Mattermost alert post to answer in the thread.
- How it went
- The output is a diagnostic report that includes a short summary, a timeline of events, up to two likely root causes, and suggested next steps. The source does not report any measured improvement. The author is still considering adding more signals, including database metrics and pipeline status.
- Tips
- Keep retries to a single attempt for each failed tool call so the agent does not loop. Remove noisy labels such as job and instance before sending the alert into the prompt, and set up your own MCP endpoints and credentials.
Try it yourself with n8n
Create an n8n workflow that starts from an Alertmanager webhook. First, deduplicate matching alerts over a 48-hour window and fetch the firing PromQL from the Prometheus Rules API, then pass the case to an AI agent with read-only access to my MCP tools for [Kubernetes], [Grafana], [DigitalOcean], [GitHub], and a Qdrant store of internal docs. The agent should investigate the alert, gather evidence, and draft a report with a brief summary, an event timeline, and up to two ranked root-cause hypotheses plus troubleshooting steps. Before any message is posted, stop and show me the draft for approval; only after I approve should the workflow reply in the existing [Mattermost thread]. Do not modify the cluster, and limit each failed tool call to one retry.
Fill in the [brackets]. Agents can make mistakes, so review anything before it is sent, paid or deleted.
2 more reports of this
Get 3 agent uses like this every morning
With the day's top AI news. Free, and one click to unsubscribe.
Related agent uses
Auto-review GitHub pull requests with n8n
They used an n8n workflow to inspect new GitHub pull requests, generate a review with an LLM, and post feedback back to the PR.
Review GitHub PRs with a structured agent checklist
Use Hermes to inspect pull request diffs, rank issues by severity, and turn the workflow into a scheduled repo watcher.
Build and ship your first iOS app with Replit
They used Replit and ChatGPT to make a first iPhone app, then spent weeks getting it through App Store review.
Find the bad config and draft the rollback
During a simulated outage, an SRE used Gemini CLI to trace logs, identify a bad config push, and prepare the revert and follow-up work.
Test an iOS app and collect bugs automatically
Use Claude Code to drive an iOS app through its accessibility tree, visit screens, and return a structured bug report.
Use a long-running agent to chase a stubborn bug
A developer let Codex work overnight against a repeatable test, using debug prints and hypotheses to track down an unexplained bug.