State of AI with Nathan Benaich - AI needs science_s search history

0:00Nathan BenaichYou're listening to Science Needs a Search History for AI to Learn Taste on Air Street Press. Papers are the happy path. If we want AI with research taste, it has to learn from the branches that were considered, rejected, and never written down. Actually, that's where the goal is, said Alistair Russell, my graduate school friend who leads a preclinical genome editing group at Cambridge. He was talking about the winding road of science that is omitted from published papers. When you're discussing how to do the experiment, and why this way is better than another way. And what does the data really mean? I know what it shows, but what does it mean? At Ray's 2026, he described how his group has begun logging the twisting path of discovery as it happens, recording the verbal and written exchanges that normally disappear. Ideas become nodes in a living graph. They branch when a meeting produces two plausible experiments, merge when separate lines of evidence converge and go dark.

0:59Nathan Benaichwhen someone quietly stops pursuing them. In his implementation, one agent scores novelty, while another tries to learn how scientists think and how they navigate through a complex world of data so that high potential nodes can trigger deeper investigation. As frontier AI labs deploy agents towards scientific discovery, giving AI authentic scientific taste remains a trillion-dollar dilemma. The bottleneck is that the record we have kept for centuries of science might be insufficient to get us there.

1:29Nathan BenaichThe experiments that never make the paper. Peter Mediwar, 1960. Noble Laureate and father of transplantation asked in 1963 whether the scientific paper was a fraud and answered yes. The form of the paper misrepresents the thinking that produced it. 60 years on, the diagnosis is unchanged. A paper presents a clean progression from hypothesis to result to conclusion. Lost along the way are the unconventional theories, the abandoned or unaffordable methods, and the underwhelming and inconclusive data. It reads like a browser history with every dead end deleted, the 10 open tabs, the four rephrased queries, the wrong turns down a sub-forum, all scrubbed, leaving one clean path from question to answer as the canonical path. What has changed since then is that the discarded material is now worth something.

2:23Nathan BenaichNo high-impact journal wants these artifacts, but raw trial and error is the most likely source of the training data required to develop scientific intuition or what researchers call taste. Earlier attempts to capture the discards were motivated by scientific integrity. The Journal of Negative Results in Biomedicine launched in 2002 to publish rigorous studies that disputed established models or exposed ineffective treatments. Its archive preserved negative conclusions after they had become papers.

2:52Nathan Benaichrather than the live alternatives and arguments that produce them. Biomed Central closed it in 2017, saying the mission had been served now that other journals publish null results. A less generous reading is that in 15 years, it published around 200 papers because almost nobody wants to read a negative result, let alone write one up when it will not count toward academic tenure. AI models are different. Even an uninteresting negative result can be useful, provided it is labeled.

3:21Nathan BenaichEarlier this year, Anthropic put a version of this to the test. The company pulled 129 decision points from real Claude code sessions between January and March 2026 showed models the work up to a human detour and asked what should happen next. Claude mythos preview beat the human choice 64% of the time while Opus 4.5 managed 51%. The comparison was tilted toward the models because Anthropic deliberately chose moments where the human decision had room for improvement. On 127 further scenarios where the human action was already strong, the models improved on it only about 20% of the time. The study was possible because anthropics researchers work inside a tool that logs by default. The reasoning, detours, and outcome are produced in the same working environment. Biology has no equivalent. Its reasoning happens in hallways, at benches, in slack threads, and on whiteboards.

4:19Nathan Benaichwhile the outcome arrives weeks later somewhere else. The record has to emerge from the work itself. Ask scientists to reconstruct it afterwards and we'll create another polished account. A log is not a label. Before any of this becomes training data, a negative result has to say what failed. The first paper published in the Journal of Negative Results in Biomedicine examined 234 negative studies across five leading medical journals. Only 30% discussed statistical power and half clearly defined a primary outcome. Telemodel and experiment failed without telling it whether the assay was underpowered, a reagent had degraded, or the hypothesis was simply wrong, and it will learn noise with confidence. Decision histories have a second missing label too. When a lab considers five experiments and runs one, only the chosen branch returns an outcome. The other four are experimental counterfactuals. Researchers call this the selective labels problem.

5:19Nathan Benaichthe data reveal results only for the action someone chose to take. Such a record can teach a model to imitate a lab's taste, but it cannot establish that the taste was good. It also smuggles local constraints into general lessons. A lab that never ran CryoEM because it did not own a microscope teaches a model that CryoEM is rarely the right call. Alastair's nodes preserve the candidate set, which is the necessary first step. To be useful, each decision record needs six fields.

5:48Nathan Benaichthe evidence available at that moment, and nothing that arrived later. The candidates considered, including the ones dismissed in a sentence. For each candidate, the expected result, the confidence attached to it, and the cost and time and money, the root chosen and the reason. What happened next? How the evidence changed the scientist's view. These must be time stamped before the outcome because hindsight turns uncertainty into inevitability. Disagreement also has to survive.

6:16Nathan BenaichAveraging three scientists into one clean rationale reproduces exactly the information loss we are trying to fix. But there is an obvious failure mode here. Once decisions are logged and scored, people log for the record post facto. Anyone who has watched an electronic lab notebook fill up with retrospective tidying knows how quickly a research tool becomes a compliance exercise. A record that costs a scientist 10 minutes of honest reflection per decision is worth far more than one that costs an hour of performance.

6:46Nathan BenaichA publication initiative called Registered Reports offers a useful starting point. Here researchers submit their rationale, methods and analysis, plan for peer review before the data exists, and the journal commits in principle to publish if they follow the approved plan. Nature has now expanded the format across every field it covers. This approach to paper writing timestamps intent before the outcome, but it freezes one plan. A useful search history must also preserve how the plan changed.

7:15Nathan Benaichwhich alternatives were rejected, when, and on what evidence. Run the runner up. Autonomous labs show what happens when decisions and outcomes are connected in a loop. Liverpool's mobile robotic chemist ran 688 experiments over eight days in a 10-variable formulation space. A batched Bayesian search used each result to choose the next experiments and found photocatalyst mixtures six times more active than the starting formulations.

7:43Nathan BenaichEvery completed experiment changed what the system did next. The robot optimized within a goal and search space that humans had already chosen. It did not decide which scientific question mattered. So what this work demonstrates is narrower, but still useful. A search history pays off when the options return standardized outcomes quickly. Open-ended biology is harder because branches can take weeks and the discarded alternatives may never be run. Some exploration therefore has to be bought.

8:13Nathan BenaichWhere two branches are plausible and the stakes justify the cost, a lab should sometimes run the runner up. Otherwise, the record captures what today's scientists usually chose and stays silent on what they systematically overlooked. A funder could ring fence a small fraction of a grant for the branch not taken, provided that the decision record and the outcome are both deposited. That costs money, but so does having every lab rediscover the same abandoned path. Test whether taste transfers.

8:44Nathan BenaichWhen it comes time to evaluate if our new AI system exhibits taste, the comparisons should run across laboratories and fields. At selected decision points in a live research campaign, we should freeze the six fields, the evidence, the candidates, expected results, confidence, costs, and the choice made. Experts then rank the options before the outcomes are known, and those outcomes are checked independently later. We could give models one of two diets, papers alone or papers plus decision histories, and ask them to rank the next experiment on unfamiliar projects. Then we score information, gain per dollar in week, calibration, and how quickly week branches are abandoned. If history has improved choices only inside the lab that produced them, they amount to useful organizational memory. If they improve choices on unfamiliar problems elsewhere, Alastair's group will have captured something science has never managed to write down, a transferable record of taste.

9:40Nathan BenaichUntil that happens, a scientific search history is a promising record, not yet a training set. Thanks for listening to Science Needs a Search History for AI to Learn Taste on Air Street Press.