Spot flaky tests and trace likely causes

Use an agent to score test instability from JUnit history, then draft a root-cause report for the flaky cases.

Done with Gemini· Intermediate· Nov 2026

What they did
A Python script first parsed JUnit XML results and scored each test by how often it both passed and failed across commits. It then saved JSON reports for a Strands Agents setup running on Gemini 2.5 Flash. That agent had one tool to pull reports for a selected time window and another to export a Markdown summary, then it reviewed error text, guessed likely causes, and wrote suggested fixes.
How it went
On sample data, the process flagged 12 tests out of 109 after parsing 19,644 results, and the agent produced a root-cause writeup. The author noted that this is only an initial pass, and infrastructure problems can be mistaken for flaky tests.
Tips
This approach only catches tests that have both pass and fail results on the same commit, so it will miss other instability patterns. Make sure your results include commit_sha and run_id, and that your CI setup matches GitHub Actions.

Try it yourself with Gemini

Review the test results for [repo or test suite] from [time range or CI logs]. Identify tests that show inconsistent pass/fail behavior across commits, then inspect the related failure messages and summarize the most likely cause for each one. Before you recommend or apply any fix, pause and ask me to approve each change. Finish with a ranked Markdown report listing the suspicious tests, your confidence, and a suggested next step for each.

Fill in the [brackets]. Agents can make mistakes, so review anything before it is sent, paid or deleted.

Read the original on dev.to ↗

Get 3 agent uses like this every morning

With the day's top AI news. Free, and one click to unsubscribe.