SciConBench

Princeton University Center for Information Technology Policy
Authors
  • Hayoung Jung 1 Pedro Viana Diniz 2 José Reinaldo Corrêa Roveda 2
  • Abner Fernandes da Silva 2 Haeun Jung 3 Enoch Tsai 4
  • Aleksandra Korolova *1 Manoel Horta Ribeiro *1

1 Princeton University 2 Universidade Federal de Minas Gerais
3 Stony Brook University 4 Hackensack Meridian School of Medicine
* Jointly advised this work

Acknowledgement
  • AI agents are increasingly used for the long-horizon task of synthesizing scientific conclusions for consequential decisions, particularly in high-stakes domains such as health and medicine [1, 2].
  • SciConBench tests whether AI agents can synthesize factually correct, complete scientific conclusions using SciConHarness, a clean-room harness that prevents access to benchmark answers from open-web search and browsing.
  • Our preprint’s best agent reached just 0.337 factual F1. As SciConBench adds new samples monthly, this live dashboard tracks frontier-model progress on fresh samples and a fixed core set.

AI Agent Progress on SciConBench

Model Families
All models

Latest News

    Frequently Asked Questions

    Introduction

    Overview of the SciConBench benchmark, agentic synthesis task, clean-room harness, evaluation pipeline, and results
    Figure 1 from the preprint. (1) SciConBench launched with 9.11K questions paired with expert-written conclusions and continues to grow as a live benchmark (9.19K samples as of August 2026!). (2) Agents synthesize scientific conclusions using web tools. (3) SciConHarness enforces clean-room evaluation by blocking ground-truth artifacts. (4) An expert-validated pipeline decomposes generated and reference conclusions into atomic facts and computes factual precision, recall, and F1. (5) Low clean-room factual F1 scores show that reliable scientific conclusion synthesis remains difficult.

    Data + Method

    SciConBench is a live, agentic benchmark for scientific conclusion synthesis, pairing 9,000+ questions from the Cochrane Database of Systematic Reviews with expert-written conclusions. At the start of every month, our pipeline finds newly published reviews, replaces superseded reviews with their latest editions, converts reviews’ Objectives sections into scientific questions, breaks their Authors’ Conclusions sections into atomic, checkable facts, and publishes the growing benchmark on Hugging Face.

    This longitudinal dashboard tracks frontier AI agents’ scientific conclusion synthesis over time. Each month, we evaluate the latest model from seven leading labs alongside three fixed controls on a stable core set and rolling panels of newly published reviews. Agents use SciConHarness to search the web, find papers, and browse pages while blocking the target review and answer-revealing sources, ensuring the task requires synthesis rather than lookup. Each rolling panel is grouped by its publication month and finalized only after that month ends. Newly released models are evaluated on the core set and up to four of the most recent rolling panels, then on each new panel as it is introduced on the dashboard.

    For details, see why the benchmark has to be live, how SciConHarness prevents answer lookup, which models are evaluated and how they are configured, and why we employ both the core set and rolling panels. See Quantitative Results below for how conclusions are scored.

    Evaluation set size over time.

    Fixed core set plus all rolling panels included each month.

    Core Set Existing Rolling Panels New This Month

    Composition

    Sample SciConBench questions

    Eight examples across diverse areas of health and medicine.

    Quantitative Results

    Leaderboard

    Scores range from 0 to 1, with higher scores indicating better precision, recall, and F1. Metrics are macro-averaged over the selected evaluation sample (i.e. core set, rolling panels, all reviews). All model evaluation uses the clean-room protocol.

    # Model Precision ↑ Recall ↑ F1 ↑

    How Often Do AI-Synthesized Conclusions Contradict Expert-Written Systematic Reviews?

    Across models, a high proportion of AI-synthesized conclusions contain at least one fact that contradicts the expert-written Cochrane systematic review. Rows are sorted from highest to lowest contradiction rate; lower is better.

    Model At least one contradicted fact ↓

    Tool Usage

    SciConHarness provides three tools: google_search via Serper, web_browse via Jina, and paper_search via Semantic Scholar. All tool calls pass through the clean-room protocol, which blocks access to ground-truth answers on the open web. For each model, the table reports average tool calls and agent turns per review across all reviews it has evaluated. Tool distribution shows each tool’s share of that model’s total tool calls, with the model’s most-used tool bolded.

    # Model Tool distribution

    Citation