AI Models Are Acing Biology Exams While Failing Real Lab Work

The gap between "knowing biology" and "doing biology" just got a benchmark — and current frontier models drop up to 46% in accuracy when faced with tasks that reflect how science actually works.

~5 min read · Edison Scientific / FutureHouse · 2026-04-14 · Benchmark

TL;DR LABBench2 is a ~1,900-task successor to the LAB-Bench biology benchmark, redesigned to reflect realistic research workflows rather than simplified multiple-choice proxies. Frontier AI models score dramatically lower on LABBench2 than on its predecessor — accuracy drops of 26–46% — revealing that progress on benchmarks has been masking real gaps in practical scientific capability.

There's a quietly uncomfortable pattern in AI for science: models keep hitting near-saturation on biology benchmarks, yet no one is automating their lab. LABBench2 is a direct response to that tension — a new benchmark designed not to measure what AI knows about biology, but whether it can actually do biology.

The Problem with the Original Benchmark

LAB-Bench, introduced in 2024, was the first serious attempt to benchmark AI systems on practical biological research tasks: retrieving literature, interpreting figures and tables, troubleshooting lab protocols, querying databases, and handling molecular biology problems. It was genuinely ahead of its time.

But the benchmark also made concessions that mattered. To make grading tractable, it relied heavily on multiple-choice answers. To accommodate the limitations of models at the time, it simplified inputs — DNA sequences were provided inline rather than fetched from databases. Task framing was sometimes artificial: asking a model to extract information from a provided figure rather than locating the right figure within a full paper.

Frontier models have improved substantially since then, and those concessions have turned into blind spots. Several LAB-Bench subcategories have reached near-saturation or even superhuman performance — which would be exciting if it meant AI systems were ready to work alongside biologists. It mostly means the tasks got too easy.

💡 The Core Idea LABBench2 changes the inputs, not the subjects. The biology tasks are largely the same; what's different is that models now have to find the right document, navigate to the right figure, handle raw files, and produce open-ended answers — just like a human researcher would. The difficulty jump this creates is the whole point.

What Changed

LABBench2 preserves the same broad task families as LAB-Bench — literature retrieval, figure/table understanding, protocol reasoning, database access, molecular biology — but re-frames them to reflect how these tasks actually appear in research practice:

Open-response answers replace multiple choice, so models can't guess their way through
Retrieval-dependent framing means models must locate the relevant paper or supplemental file themselves, not have it handed to them
File-dependent tasks require working with actual sequence files and databases, not inline text snippets
The benchmark spans nearly 1,900 tasks in total

The result is a consistent, substantial drop in performance across every task family and every model tested.

The Numbers That Matter

flowchart LR
    A[LAB-Bench\nMultiple choice\nSimplified inputs\nProvided context] -->|Realistic redesign| B[LABBench2\nOpen response\nRetrieval-dependent\nFile/database access]
    B --> C[26–46% accuracy drop\nacross all frontier models]

Across all task families and models evaluated, moving from LAB-Bench to LABBench2 produces accuracy drops ranging from −26% to −46%. This isn't a minor recalibration — it's a signal that prior benchmark performance was a poor proxy for real-world utility.

Tool augmentation (primarily web search and code execution) helps meaningfully in some areas. Tasks involving literature retrieval — including the LitQA3 and Patent/TrialQA subtasks — benefit substantially from giving models access to search tools. Accessing supplemental materials also improves with tool use, likely because web search can locate the right documents.

But tools don't close all the gaps.

⚠️ Watch out for

Retrieval and localization failures: The largest drops occur when models must first identify the correct source, then find a specific figure, table, or supplemental file within a long document. Better reasoning alone doesn't fix this — it requires better document navigation.

Fragile exact-input handling: Tasks like SeqQA2 and CloningQA expose that even conceptually straightforward operations break down when they require exact string-level fidelity — say, faithfully manipulating a DNA sequence through a multi-step protocol. This is a known failure mode that human researchers have built dedicated tools to work around.

Non-text database access: Querying specialized scientific databases that don't return plain text remains a substantial capability gap — and one that blocks many high-value research applications.

Why It Matters

The benchmark saturation problem in AI for science is more than academic. When a model appears to "master" biology tasks on a benchmark but the tasks are fundamentally easier than real research work, it creates false confidence in AI tools before they're ready — and risks premature deployment in contexts where errors have real consequences.

LABBench2 is a correction mechanism. By raising the floor to match what biology research actually demands, it makes visible the capabilities that still need work: navigating heterogeneous scientific documents, faithfully handling precise biological data, and integrating multiple retrieval and reasoning steps without compounding errors.

The benchmark is released with a public dataset on HuggingFace (futurehouse/labbench2) and an open eval harness on GitHub (EdisonScientific/labbench2), which should make community uptake straightforward.

For practitioners building AI-assisted research tools, the takeaway is specific: if your system relies on models that perform well on LAB-Bench, LABBench2 will likely surface failure modes that matter for deployment. The gap between benchmark-ready and research-ready is measurable now — and large.

Source: LABBench2: An Improved Benchmark for AI Systems Performing Biology Research Authors: Jon M Laurent, Albert Bou, Michael Pieler, Conor Igoe, Alex Andonian, Siddharth Narayanan, James Braza, Alexandros Sanchez Vassopoulos, Jacob L Steenwyk, Blake Lash, Andrew D White, Samuel G Rodriques Published: 2026-04-14 PDF: https://arxiv.org/pdf/2604.09554