Use coding agents to stress-test your docs

A CLI runs coding agents against doc-dependent tasks, spots where they fail, then rewrites the docs and repeats the eval.

Done with Claude CodeCodex· Advanced

What they did
They saved a task set with success criteria into Gauge, then let it launch clean Docker runs of Claude Code, Codex, Pi, or OpenCode. Each run got the repo inputs they chose, such as Skills or files, and the resulting transcripts were checked for mistakes or gaps. Their own coding agent then summarized the weak spots, updated the guidance, and reran the same evaluation.
How it went
Gauge compared the earlier and later batches after each pass. The README did not show measured results, though it did include one release-notes Skill example. The docs can be used as inputs in both local and hosted modes, but hosted mode also rewrites responses, so the local docs-testing path is less clear.
Tips
You need Node 22.13+, Docker, and credentials for the agent you want to run. Usage is billed through your provider or subscription, and hosted mode draws on workspace credits.

Try it yourself with Claude Code

Test and improve the docs for [project or docs folder]. Create 5 realistic tasks that should be solvable using only those docs, then run a fresh coding-agent session for each task and capture the transcripts. Review the transcripts for places where the agent hesitates, guesses, or misses key instructions. Before making any irreversible change, show me the proposed doc edits and wait for my approval. After I approve, revise the docs, rerun the same tasks, and keep looping for up to 3 rounds or until all tasks pass. Finish with a short report that lists each fix and how it changed the results.

Fill in the [brackets]. Agents can make mistakes, so review anything before it is sent, paid or deleted.

Read the original on github.com ↗

Get 3 agent uses like this every morning

With the day's top AI news. Free, and one click to unsubscribe.