Use Claude Code to hunt flaky CI tests overnight

Claude Code tracked CI flakiness, isolated the worst tests, and suggested fixes while a human reviewed the changes each day.

Done with Claude Code· Advanced· May 2026

What they did
A developer gave Claude Code access to CI artifacts and asked it to build a tracker for the rolling 30-day pass and fail rate. From there, it sorted out the noisiest tests, tried to separate test bugs from product issues, and worked toward the smallest possible repro case. When there was no reliable repro, it still tried a fix and kept rerunning the tests overnight to see whether the change held up statistically.
How it went
Over several weeks, the CI pipeline finished 20 minutes faster and the pass rate improved by 35%, starting from about 40% on master. Some of Claude’s suggested fixes were wrong and were thrown out.
Tips
Keep a person in the loop for every pull request, since not every proposed fix will be right. Overnight test runs are useful, but they still need daytime review before anything is merged.

Try it yourself with Claude Code

You have read access to my CI artifacts and test history for [repo]. First, build a simple report that tracks the rolling 30-day pass/fail rate and ranks the flakiest tests. Then focus on the top [number] offenders: try to reproduce each one, decide whether the cause is likely in the test or in production code, and narrow it down to the smallest useful repro. If you cannot get a deterministic repro, propose a likely fix, run the test suite [number] times overnight, and tell me whether the change looks stable enough to keep. Pause and show me the diff before making or merging any irreversible change, and do not submit anything without my approval.

Fill in the [brackets]. Agents can make mistakes, so review anything before it is sent, paid or deleted.

Read the original on news.ycombinator.com ↗

Get 3 agent uses like this every morning

With the day's top AI news. Free, and one click to unsubscribe.