Featuring Jiacheng Miao, Postdoctoral Fellow, Stanford Institute for Human-Centered AI
At our latest Snorkel AI Reading Group, Jiacheng Miao (Stanford Institute for Human-Centered AI) presented Paper2Agent, a framework that turns a research paper and its codebase into an interactive AI agent that can run the paper’s methods. Each agent is packaged as an MCP server, tested against the paper’s own results, and hosted so anyone can query it in natural language. Of 100 computational biology papers, 74 became working agents, and on 300 benchmark questions the agents scored 91.2% accuracy, compared with 80.3% for Claude Code given the paper’s GitHub repo (both using Sonnet 4). The talk asks what scientific communication looks like when every paper ships with an agent.
Transcript
Lightly edited for readability.
Why a 350-year-old format still limits how we use papers
Miao started with the research paper itself, the traditional unit of scientific communication. One of the first modern-style scientific papers was published 350 years ago, and people still use papers to share scientific knowledge. They used to be printed. Now they are PDFs.
But when we read a paper, we rarely want only to learn what’s new. We want to apply it to our own problem. Can I run this training method on the data I just generated? Can I test it on my own samples? Editors and reviewers ask a different question: can I reproduce the paper’s results? And when a paper lines up with our interests, we might email the author to propose working together, because we have data that could pair with their method.
In practice, Miao said, interacting with papers is “really passive and fragmented.” His example was the AlphaGenome paper from DeepMind, which trains foundation models on sequence-to-function prediction. To understand what it offers, you read a PDF of roughly 200 pages, which he said no human can realistically get through. You then hunt through the supplementary notes, because many of the key details are not in the main text. If you want to train your own models, you have to work out which data the authors used. If you want to apply AlphaGenome to your own problem, you have to figure out how to use their code and APIs from the GitHub repo they released. That, he argued, is how we currently communicate scientific knowledge.
The proposal: release a paper agent, not just a PDF
The question driving the project, Miao said, is whether we can reimagine how people interact with papers. Over the past year, large language models have become powerful enough to do real research. If we have agents, an AI co-scientist, how should we rethink the way scientific knowledge is communicated?
His proposal is that instead of publishing only a passive paper in a single PDF, researchers should also release a paper agent, and his team can help them build it.
He walked through the AlphaGenome case. Suppose you want to know whether people who carry certain DNA mutations are likely to develop a disease, and through what mechanism. Traditionally you would dig into AlphaGenome, work out how to use its code, and set up the Python environment. With a paper agent, you ask a natural-language question instead. In his example, the query asked the agent to analyze gene expression profiles to explain why a set of mutations leads to disease. The agent used AlphaGenome to make predictions, applied biological reasoning to AlphaGenome’s output, and returned candidate causal genes. A pharma or biotech company developing a therapy could then consider those genes as targets.
The broader point is that papers should be written not only for humans to read but also for agents to use, so that an agent can easily apply a paper’s methods to complex tasks.
How Paper2Agent turns a paper into an MCP server
The overall workflow starts with a paper as input and ends with an MCP server. Miao asked how many people in the room had heard of MCP servers, and many hands went up. Coding assistants such as Claude Code connect to MCP servers natively, so adapting papers to that format makes them usable by tools people already have.
Each paper’s MCP server has three components:
- MCP tools. These wrap the paper’s computational methods, so a user can apply them to their own data.
- MCP resources. These expose the manuscript, links to the code repo, and the training data described in the paper, so a coding agent can look up what data a paper used without guessing.
- MCP prompts. Reading a paper is rarely a single analysis. It is a chain, where some steps build on the output of earlier ones. To let an agent reproduce a paper’s results or apply its method to new data, the server includes step-by-step instructions that follow what the paper did. Miao compared this to skills today, but said skills did not exist when the team started building Paper2Agent.
Because few researchers have the compute to run these servers themselves, Miao advocates hosting them in the cloud, on platforms such as Hugging Face or AWS. He likened it to having an API for a paper: you call it, and it returns the paper’s knowledge. You can then chat with the paper agent, and when you want to apply the method to your own data, the agent writes the code to do the analysis.
Miao described the result as a virtual corresponding author. Anyone who has emailed a real corresponding author knows they may or may not reply with feedback, data and code. The paper agent is always there to respond.
Making paper agents reliable
Early on, the team found that even the best coding agents, run zero-shot, sometimes misunderstand what a paper does and write the wrong code. Miao compared this to humans trying to replicate a paper: we fail at first and learn from the failures. So the team built a multi-agent system that creates a virtual environment where an agent can work with the data and everything else related to a paper, build up experience, and learn to use the paper reliably.
His example was teaching an agent to use AlphaGenome. One set of agents identifies the codebase, installs the necessary dependencies, and works out which tasks AlphaGenome is suited for. The team then extracts benchmark examples, essentially question-and-answer pairs, from the paper and its GitHub repo. A loop of testing agents runs those Q&As against the MCP server that the building agents produced, and the server is hardened over several iterations. When the agent with that server can reproduce everything the paper reported, including every notebook in the GitHub repo, that is a signal it uses the paper well.
On a single AlphaGenome example, where the task was to score DNA mutations across several modalities, a one-shot Claude Code with access to the repo often missed on accuracy. With the team’s loop, accuracy came up quickly, and Miao said the trained paper agent also handles the task faster and, he thinks, much more cheaply than the alternatives. The team also wrote examples by hand to check that the agent generalizes to novel inputs and is not just remembering what is in the paper.
For a broader test, they turned 100 papers into 100 agents and curated 300 questions based on what people would want to do with such papers. The paper agents scored over 90% accuracy, compared with about 80% for Claude Code with access to the GitHub repo. Miao’s takeaway: if you build a virtual environment and let the agent self-play in it against benchmarks, performance improves.
Use case: finding causal genes, and checking a paper’s claims
Miao then turned to applications, starting with one of his favorites. The task comes straight from the AlphaGenome paper: given a set of DNA mutations, predict the causal genes. Based on AlphaGenome’s predictions, the paper names two genes as the likely culprits.
The team gave a paper agent the same task. The agent started with a plan for which AlphaGenome tools to use, ran them to produce tables and figures, interpreted those outputs with biological reasoning, and proposed a different gene as causal. Miao noted this is a well-known hard problem in human genetics: when several genes sit close together, it is difficult to tell which one is responsible. He said experiments have, to his knowledge, validated the agent’s pick, while AlphaGenome’s predictions alone pointed to two other genes.
The larger point is about reliability. A paper makes claims, and you may not know whether to trust them. Miao suggested the AlphaGenome authors come more from an AI background than from human genetics, so they may not be aware of parts of that literature. Language models, by contrast, have read it. With a single prompt, he said, you can evaluate a published conclusion using independent, model-based evidence, and a paper agent can act as a verifier that compares the human result against the agent’s result on a complex scientific task.
From there he moved to what he sees as the future of scientific collaboration. Imagine one paper with a new method, and another with new data that cost millions of dollars to generate. Someone could apply the method to the data and find something new. The team is testing whether paper agents can do that kind of collaboration on their own.
Agents collaborating across papers: a causal gene for psoriasis
His favorite example involved three paper agents working together to find a causal gene for psoriasis, an inflammatory disease. One was the AlphaGenome agent. The other two came from papers whose main contribution was CRISPR screening data of the kind used to train virtual cell models.
The AlphaGenome agent went first. Based on AlphaGenome’s predictions, it proposed that GPR137 may be the causal gene in specific cell types. The two CRISPR-screening agents then tried to validate that.
Miao explained the logic. In genetics, the question is how DNA mutations change genes and lead to disease. Some CRISPR screens perturb the variants and measure changes in downstream genes. Others perturb the genes themselves and measure the same downstream changes. If the link from variant to gene holds, with only one edge between them, then the variant’s effect on downstream genes should be proportional to the gene’s effect. In other words, the two vectors of downstream effects should correlate. The agent’s proposal was to pull the screening results for the variant and for the gene and compute that correlation. Miao showed a comment from his advisor, a leading human geneticist, who said he had never seen this analysis before and considered it a novel idea.
The result was specific. The correlation was high only in stimulated T cells, meaning activated immune cells, and not in the resting state. It held for GPR137 but not for other genes in the same region as the mutations. Miao said this shows paper agents can support autonomous, AI-driven discovery and validation when several of them collaborate.
Demo and lessons learned
Miao then showed the hosted agents at paper2agent.ai, which are free to use. In the first demo, he asked an agent to reproduce the conclusions reported in a paper and generate a report with figures. The agent read the paper, located the ADHD case study, and worked out what analysis it contained. It then went into the supplementary tables, which the paper attached as an Excel sheet, to see what had been reported, read the paper’s figures, and decided what to reproduce, noting nuances along the way. As Miao put it, the agent behaves like the author of the paper: it understands the paper and writes code to reproduce it. He left that run going.
The second demo was a run already in progress: the AlphaGenome query from earlier in the talk. The agent used the tools built from the AlphaGenome paper to score variants, identified the highest-scoring one, wrote code, and named the likely causal gene. It produced a report with figures and a reproducible Jupyter notebook so that anyone can rerun the analysis.
He closed the section with lessons learned. Not every paper turns easily into a robust agent. In many cases the original codebase is incomplete, poorly documented, or contains unresolved errors, so the team cannot reliably expose it as working tools. They have also tried building agents when there is no codebase at all, generating the code from the paper’s text alone. Miao said current models behave much better at this, but at the time of the study the team was working with much earlier models, so there was a big gap.
That difficulty is itself informative. Whether a paper is reproducible has been hard to measure: some papers take other researchers three months to replicate. Miao proposed using agents to reproduce papers and treating the result as a new reproducibility metric. If a paper is very hard to turn into a robust agent, it may be missing key information, and the paper is not yet something people can use, only something they can read.
Beyond one paper: agent swarms and training for statistical reasoning
Miao described two follow-up projects. The first starts from the observation that a paper reports a single analysis, but a lot can happen behind it. He cited a study that asked whether immigration affects support for social welfare. Researchers surveyed in that work tended to report results in line with their own views: pro-immigration researchers reported positive results and anti-immigration researchers reported negative ones, even though they analyzed the same data and asked the same question. The team saw the same behavior in agents. When told it was pro-immigration, an agent tended to report that immigration raises support for social welfare, because it chose which method to run and which sample to analyze.
Their proposal is to stop relying on the one result a paper reports. Instead, roll out thousands of agents to run the analysis thousands of times, and count how often the agents report results more extreme than the paper’s. If the paper’s result is more extreme than most of what the agents find, it may have been cherry-picked, perhaps without anyone intending it. Miao pointed to incentives to publish and to the well-known cluster of reported p-values around 0.05 in the scientific literature. Agent swarms are a way to check a paper’s own analysis against that.
The second project is about training models to do research. Hypothesis testing sits at the center of modern science, from clinical trials asking whether a drug works to policy questions about who a policy benefits. Agents can already take a dataset and produce conclusions and p-values. The team built a benchmark of 300 tasks, which Miao compared to Terminal-Bench but for statistical reasoning: the agent gets a study design and data and has to write code to answer the question. The GPT model they tested in April did not perform very well, but the team found recipes for training agents at scale to do much better. The work is a NeurIPS oral this year.
Miao summarized the vision. With agents, researchers get more than static knowledge: they get active collaborators, virtual corresponding authors that help run analyses and adapt and communicate what a paper knows. Papers today carry data availability statements so others can reproduce them. He expects a future with agent availability statements, where every paper is paired with an agent that people can talk to and that can perform complex scientific tasks. Those agents could collaborate and generate new knowledge, which in turn produces new paper agents. He called that recursive self-improvement, or RSI, for knowledge. The Paper2Agent paper was published in Nature about two months ago, and the GitHub repo is available.
Q&A
Q: How will the roughly 10% accuracy gap change with newer frontier models?
An attendee asked whether the gap between paper agents and Claude Code with the repo is shrinking as newer frontier models arrive. Miao said the team did test this, after a reviewer asked for it. There is a recent trend, but it has not closed the gap: if you use the frontier models to train the agents, their scores rise too. The gap remains, though he observed that it closes a little.
He added that the more important question is how we publish. Academia can spend a year getting from preprint to paper, while coding models evolve far faster, and he sees that as real pressure to find better ways to publish.
Q: What context could a paper agent use beyond a four-to-eight-page paper?
An attendee observed that a paper is a compressed version of months of research, limited to a few pages because human context windows are small, and asked what richer context behind the paper could unlock. Miao said that providing the raw code and data may be the way to go, though few researchers want to share them because they are messy, and agents are good at writing code. For machine learning papers, such as those at NeurIPS, the valuable part is the algorithm written in code, so access to the code and the training data may be all an agent needs, rather than polished text meant to satisfy reviewers.
He also noted that reading a paper can be like reading a novel. Good writers tell good stories, and that part is not for agents. It is for people to enjoy.
Q: How should machine-readable and human-readable insight fit together?
Another attendee framed papers and code as two different ways of compressing knowledge, and asked whether the work had changed how Miao thinks about capturing the why and the what. Miao said it depends on the purpose. If the goal is only to generate new knowledge, agents can do that, and it is already happening: AI can produce proofs that no human understands. The open question is whether we want that, or want humans in the loop guiding the process.
His team is exploring proactive agents that work with humans. Agents are very good at analysis today but not at proposing research ideas. In his experience, when an agent comes up with 10 ideas and a human researcher selects one, that one is typically really good. He sees that as a promising interface, with machine-readable code on one side and high-level human guidance on direction on the other.
Q: How did you build the evaluation questions?
An attendee asked how the team assembled the questions for the 100-paper benchmark, given that what a reader wants depends on their purpose. Miao said the aim was to test whether paper agents can reproduce a paper’s results and generalize to new datasets. The team extracted question-and-answer pairs from what each paper reports, such as running the paper’s method on a given dataset and checking the result. They also created new datasets by hand to see what results the agents would produce there.
He said some things are much harder to benchmark, such as whether an idea proposed by agents collaborating across papers is actually novel.
Q: Did you see reward hacking when training for statistical reasoning?
An attendee asked whether such a long, complex task led to reward hacking in the evals. Miao said yes, a lot. That is why the final reward is outcome-based. The team tried rubric-based rewards during training, and the agent was always able to hack them. For example, they wanted the agent to report a confidence interval along with the p-value, and the agent could report negative infinity to positive infinity and still score highly. Writing good rubrics that scale across a diverse set of questions, rather than one specific question, proved very hard.
The attendee asked whether the graders were programmatic, and Miao confirmed they were. On data, he said the final evaluation uses real data collected from top journals, while training uses synthetic data. Because the team generates the data, it knows the distribution and can hide part of it, so the agent has to work out what was generated and propose a suitable method. With programmatic grading the team saw less reward hacking, though not none, because a scientific question can be answered in more than one valid way, and that creates room for hacking.
Q: How should we think about attribution and the role of the human researcher?
The host asked about attribution when novel ideas come from several paper agents working together, and about the role of the human researcher. Miao said every researcher runs into this. Agents do very well when told exactly what to do, but with vague prompts or vague ideas they tend to go nowhere. His team is doing two things. First, it keeps a human in the loop: when the agent is uncertain, the human gives feedback and chooses which ideas to pursue and which papers should collaborate. Second, it is training models to choose which two or three papers to combine. That is a research taste question: out of 100 papers a person has read, which three would they decide to combine? He believes that if a language model can be trained to do this well, a lot of research taste questions become solvable.
FAQ
Q: What is Paper2Agent?
Paper2Agent is an open-source framework that turns a research paper and its codebase into an interactive AI agent that can run the paper’s methods. A multi-agent pipeline locates the code, builds the environment, runs the paper’s tutorials, and packages the methods as an MCP server with tools, resources, and prompts. You then query the agent in natural language through a coding agent such as Claude Code.
Q: Is Paper2Agent open source?
Yes. The code is on GitHub at jmiao24/Paper2Agent, and the work is described in a Nature paper and an arXiv preprint.
Q: Where can I try a paper agent?
Miao said the hosted agents at paper2agent.ai are free to use, and you can request an agent for whichever paper you want. The agents write code to run the analysis for you and can return a report with figures and a reproducible Jupyter notebook.
Q: How is a paper agent different from giving Claude Code a paper’s GitHub repo?
A paper agent is built through a build-and-test loop in a virtual environment. Agents install the dependencies, extract question-and-answer examples from the paper and its repo, and test the resulting server until it can reproduce the paper’s reported results. Miao said that zero-shot coding agents sometimes misunderstand a paper and write the wrong code. In the Nature paper’s benchmark of 300 tutorial-derived questions, Paper2Agent with Sonnet 4 scored 91.2% accuracy, compared with 80.3% for Claude Code with direct repository access using the same model, and 86.3% with Sonnet 4.6. Of the 100 computational biology papers tested, 74 were successfully turned into agents.
Q: Does Paper2Agent work on papers with incomplete or poorly documented code?
Not always. Miao said some codebases are incomplete, poorly documented, or contain unresolved errors, which means the team cannot reliably turn them into working tools. The pipeline validates tools against the original paper’s outputs, repairs the ones that fail, and drops the ones that cannot be fixed. The team is also exploring whether an agent can generate the code itself from a paper’s text when no codebase exists.
Speaker bio
Jiacheng Miao is a postdoctoral fellow at the Stanford Institute for Human-Centered AI. He is an AI researcher focused on agentic AI for biomedical and therapeutic discovery. He received his PhD at UW-Madison, and his work has been featured by Anthropic, The Scientist, Stanford Medicine, and Nature, among other places. He is also a recipient of the NOMIS & Science Young Explorer Award. Learn more at jiachengmiao.com.
Links
- Watch the talk: YouTube
- Paper: Nature and arXiv preprint
- Hosted agents: paper2agent.ai
- Code: GitHub
- Jiacheng Miao: website, X, LinkedIn
- Snorkel AI Reading Group: all sessions and upcoming events
- More from the series: Continual Learning Bench, Agents’ Last Exam, Collaborative Gym
Recommended articles




