Have a coding agent check papers before each experiment

A Claude Code agent with paper search found useful ideas that a plain run missed while tuning a small language model.

Done with Claude Code· Intermediate· Apr 2026

What they did
They ran two Claude Code agents on the same optimization task for a small language model. One agent used only its built-in knowledge. The other could query Paper Lantern, an MCP search tool tied to a 2M+ paper collection, and it checked that library before each attempt. Over the run, it looked through 520 papers, pulled out 25 possible approaches, and turned them into step-by-step guidance.
How it went
The version with paper search improved the model by 4.05%, while the baseline reached 3.67%. The poster says roughly 15 of the 25 paper-based ideas proved useful. They also said the stronger setup remained ahead by 12+ minutes in a 2-hour run, but commenters said the result may not generalize well.
Tips
This works best when the topic has a lively research scene, since the agent pays a time cost to read full papers. Keep a clear audit trail of which paper led to which change, and make sure anything irreversible waits for approval.

Try it yourself with Claude Code

I want to improve [model/project/benchmark]. Before each attempt, search [paper corpus/search tool] for methods related to [problem]. For each candidate idea, tell me the paper it came from, why it might help, and how to compare it with the current baseline of [score]. Apply only one change at a time, pause and ask me before any irreversible action or final submission, and keep a log of each idea, its paper source, and the result. After [number] experiments, summarize which ideas were worth keeping.

Fill in the [brackets]. Agents can make mistakes, so review anything before it is sent, paid or deleted.

Read the original on reddit.com ↗

1 more report of this

Get 3 agent uses like this every morning

With the day's top AI news. Free, and one click to unsubscribe.