Give a coding agent a paper corpus to find new techniques
Quick experiment I ran. Task: Quick experiment I ran. Took two identical AI coding agents (Claude Code), gave them the same task — optimize a small language model. One agent worked from its built-in knowledge. The other had access to a search engine over 2M+ computer science research papers. Agent without papers: did what you'd expect. Tried well-known optimization techniques. Improved the model by 3.67%. Agent with papers: searched the research literature before each attempt. Found 520 relevant papers, tried 25 techniques from them — including one from a paper published in February 2025, months after the AI's training cutoff. It literally couldn't have known about this technique without paper access. Improved the model by 4.05% — 3.2% better. The interesting moment: both agents tried the same idea (halving the batch size). The one without papers got it wrong — missed a crucial adjustment and the whole thing failed. The one with papers found a rule from a 2022 paper explaining exactly how to do it, got it right on the first try. Not every idea from papers worked. But the ones that did were impossible to reach without access to the research. AI models have a knowledge cutoff — they can't see anything published after their training. And even for older work, they don't always recall the right technique at the right time. Giving them access to searchable literature seems to meaningfully close that gap. I built the paper search tool (Paper Lantern) as a free MCP server for AI coding agents: https://code.paperlantern.ai Full experiment writeup: https://www.paperlantern.ai/blog/auto-research-case-study; tools: Claude Code. Human role: The original post does not say how a person stayed involved.; limitations: Prerequisites: Access to Claude Code, a small language model, and a paper search MCP server. Constraints: AI knowledge cutoffs may limit Agent A's recall; Agent B's success depends on paper relevance. Source reliability: Unverified secondary source; HTTP 403 error on original URL. Human approval required before applying any code changes..
Tools
Published on Agent Work Atlas:
Overview
Quick experiment I ran.
Workflow
Quick experiment I ran. Took two identical AI coding agents (Claude Code), gave them the same task — optimize a small language model. One agent worked from its built-in knowledge. The other had access to a search engine over 2M+ computer science research papers. Agent without papers: did what you'd expect. Tried well-known optimization techniques. Improved the model by 3.67%. Agent with papers: searched the research literature before each attempt. Found 520 relevant papers, tried 25 techniques from them — including one from a paper published in February 2025, months after the AI's training cutoff. It literally couldn't have known about this technique without paper access. Improved the model by 4.05% — 3.2% better. The interesting moment: both agents tried the same idea (halving the batch size). The one without papers got it wrong — missed a crucial adjustment and the whole thing failed. The one with papers found a rule from a 2022 paper explaining exactly how to do it, got it right on the first try. Not every idea from papers worked. But the ones that did were impossible to reach without access to the research. AI models have a knowledge cutoff — they can't see anything published after their training. And even for older work, they don't always recall the right technique at the right time. Giving them access to searchable literature seems to meaningfully close that gap. I built the paper search tool (Paper Lantern) as a free MCP server for AI coding agents: https://code.paperlantern.ai Full experiment writeup: https://www.paperlantern.ai/blog/auto-research-case-study
Reported result
Unknown actual outcomes. The source text claims Agent A improved the model by 3.67% and Agent B by 4.05%, but no independent verification exists. Proposed acceptance check: Measure improvement percentage and technique success rate for both agents in a controlled environment.
Human role
The original post does not say how a person stayed involved.
Limitations
Prerequisites: Access to Claude Code, a small language model, and a paper search MCP server. Constraints: AI knowledge cutoffs may limit Agent A's recall; Agent B's success depends on paper relevance. Source reliability: Unverified secondary source; HTTP 403 error on original URL. Human approval required before applying any code changes.
Try it
I want to test if giving a coding agent access to research papers improves its optimization skills. Use Claude Code. Set up two agents: one with only built-in knowledge, and one connected to a paper search tool like Paper Lantern. Task: Optimize a small language model [replaceable input: specify model name and dataset]. Steps: 1. Run Agent A for 5 optimization attempts. 2. Run Agent B for 5 attempts, requiring it to search papers before each try. 3. Record improvement metrics and technique success rates. 4. Compare results. Output: A report showing percentage improvement for each agent and a list of techniques tried. Measure success by higher improvement rate and fewer failed attempts. Handle missing inputs by asking for model details. Do not apply any changes to the model without my explicit confirmation after reviewing the first sample result.
Source reports are not independently verified. Publication date is the catalog publication date, not the original article date.