Test whether your published papers still run
Mike Frank had Claude inspect the code and data behind his recent journal papers and it uncovered serious reproducibility problems.
- What they did
- Mike Frank used Claude to go through every journal paper he had co-authored over the last five years and check whether the computational work could be repeated. The agent relied on the papers’ publicly posted code and data and worked from those materials alone.
- How it went
- The check surfaced major reproducibility trouble even though the files were openly available. The post does not include a paper count, specific failure modes, or a breakdown of which papers passed.
- Tips
- This approach depends on having public code and data to inspect. If you want a similar review, have the agent do the work in a duplicate copy and pause before any action that would change, send, or delete something.
Try it yourself with Claude
I want to verify the reproducibility of my published papers using the code, data, and PDFs in [locations]. For each paper, set up a fresh copy, try to execute the analysis, and compare the outputs with the reported figures and tables. For every item, tell me whether it fully matches, partly matches, or does not match, and name the reason if something breaks, such as missing inputs, broken paths, or version conflicts. Do not touch my original files. If anything would involve publishing, deleting, sending, or another irreversible step, stop first and ask for my approval.
Fill in the [brackets]. Agents can make mistakes, so review anything before it is sent, paid or deleted.
Get 3 agent uses like this every morning
With the day's top AI news. Free, and one click to unsubscribe.
Related agent uses
Inspect a broken book file and search it manually
When an indexer failed on an uploaded book, Claude used shell commands to identify the file type and search the text directly.
Coordinate a swarm for equity research and coding
They used Claude to manage multiple Codex agents for coding and dozens of agents for equity analysis.
Track competitor prices in a shared sheet
They set up an agent to check competitor sites on a schedule and log pricing and contact details into Google Sheets.
Use search agents to map AI bot naming trends
A writer used a Claude swarm to gather thousands of bot names and compare patterns by gender, industry, and launch year.
Search papers and save notes to Zotero
A researcher used Claude to find papers, check a personal Zotero library, and write short literature notes back into Zotero or Obsidian.
Let a browser agent open hard-to-reach articles
They set up Claude with a browser-driving MCP so it could get into articles normal web fetches missed.