Skip to content
Claimband
How it works

LLMs propose. Code verifies.

A language model reads the paper and the repository and suggests what to run. Everything that decides a verdict is ordinary code over recorded runs.

  1. 01
    LLMs propose, code verifies

    The model proposes claim extractions, mappings and fixes. Deterministic code parses metrics, computes bands and assigns verdicts. The model never outputs a final number or verdict.

  2. 02
    An environment failure is not a scientific failure

    Blocked is reported separately and never counted against the paper.

  3. 03
    Noise is not failure

    A gap inside the statistically justified band is reproduction.

  4. 04
    Per claim, never one score

    A paper-level summary always shows coverage, blocked rate and evidence strength next to the reproduction rate.

  5. 05
    Evidence or it didn't happen

    Every sentence that asserts something links to a log line, file:line or table cell.

  6. 06
    The GPU never touches untrusted code

    Reasoning runs on the GPU host. Experiments run only in the CPU sandbox, which has no network at run time.

Architecture

From upload to verdict

Five swimlanes. Hover or focus an agent to see what it takes in, what it hands on, its tools and the failures it handles.

Web UILane 1Upload, live replay, audit workspace, reportOrchestratorLane 2LangGraph state machine: stages, retries, human gatesReasoning layerLane 3Self-hosted open-weight coding LLM on one H100; JSON-schema outputs onlySix agentsLane 4CPU sandboxLane 5Docker + gVisor · no network at run time · cgroup limits · read-only root · non-root userrepaircounterfactual rerunsA1PaperA2RepoA3EnvironmentA4ExecutionA5ComparisonA6Discrepancy & Report

Solid arrows carry artifacts forward. Dashed loops are repair (environment fixes after a failed run) and counterfactual reruns (isolating the cause of a gap).

The trust boundary

The GPU never touches untrusted code

Uploaded code is untrusted. It runs only on the far side of a narrow, fixed-schema API, on a machine with no GPU and no network.

GPU reasoning host

  • Runs the open-weight model only
  • Never executes uploaded code
  • Outputs are JSON-schema constrained
  • Verdicts are not produced here

Narrow exec API

  • Fixed request and response schema
  • Accepts: env spec, entry point, seed, limits
  • Returns: run records only
  • No shell, no file transfer, no free text executed

CPU sandbox

  • Docker + gVisor
  • Network off at run time
  • cgroup caps on CPU, RAM, processes
  • Read-only root, non-root user
Tolerance-band test

Noise is not failure

A claim reproduces when our mean lands inside a band around the reported value. The band is the widest of four things: δ = max(precision, author std, seed spread, platform floor × policy). Move the sliders and watch which term wins.

Reported value86.70
Our mean (fixed)86.55
Reported precision (decimals)1
Author std0.00
Our runs (k)5
Our seed std0.20
Platform floor0.10
Tolerance policy
86.1586.5586.95ourspaper
Reproduced|ours − reported| = 0.150 ≤ δ = 0.248

The chip shows the band test alone. A full audit may also change a verdict through counterfactual reruns and the human gates.

The rule

δ = max(precision = 0.050,author std = 0.000,seed spread = 0.248,platform floor × policy = 0.100) = 0.248

The widest term sets the band. Here that is our seed spread. The seed spread term is the t-based confidence half-width of our k runs (5 runs, t = 2.776, s = 0.200).

Rubric

Six verdicts, two flags

Every claim gets exactly one status, and may carry flags.

LabelMeaning
ReproducedThe faithful run matches within the tolerance band, with no source-modifying fixes.
Partially reproducedDirection or ranking holds but magnitude is off; or it matches only under the paper's stated settings (repo defaults differ); or a source fix was needed; or evidence is too thin (k = 1).
Not reproducedA faithful run in a verified environment falls outside the band, and counterfactuals did not close the gap.
BlockedCould not execute: missing artifact, dependency failure or over budget. Not counted against the paper.
UnverifiableNo code path can be mapped, or required details are missing (for example, metric averaging is unspecified).
Not computationalTheorem, human study, private data or a scale far beyond the CPU budget. Classified when the ledger is built.
Code ≠ PaperFlagThe implementation differs from the described method; the audit shows a diff.
Reproducible but suspectFlagThe number reproduces, but a methodological red flag was found (leakage, tuning on the test set).

Paper-level summary, never one number

Coverage
Executed claims ÷ computational claims. Executed means a verdict of Reproduced, Partially reproduced or Not reproduced.
Reproduction rate
Weighted share of executed claims: Reproduced = 1.0, Partially reproduced = 0.5, Not reproduced = 0. Main-table claims weigh 3; ablation, figure and in-text claims weigh 1.
Blocked, Unverifiable, Not computational
Shown separately, never folded into the rate.

Claims confirmed and evidence strength

Claims confirmed
Reproduced ÷ executed, drawn against Raff's 75% paper-level threshold line (source 2).
Evidence strength
High: k ≥ 5 and the mapping is confirmed by the README and a dry run. Medium: in between. Low: k = 1, or the mapping comes from the model only.

Three-way cause separation

A claim that is not green can have three different causes: the environment broke, seed noise moved the number, or the paper and the code genuinely disagree. A plain pass/fail check cannot tell them apart. Here they are kept apart, using audit RL-0412.

Loading…
Human gates

You decide at three points

Three optional checkpoints keep a person in charge of anything the model proposed.

Gate 1

Ledger

Opens after the Claim Ledger is built.

You confirm the extracted claims, values and which are computational. Edits are logged, and nothing downstream runs on a claim you rejected.

Gate 2

Mapping

Opens when a mapping has low confidence.

When the claim-to-code link rests on the model alone, or two scripts could produce the same table, you pick the right one before anything is run.

Gate 3

Source edits

Opens when a fix would change source code.

Environment-only fixes (pins, path remaps) need no approval. A source-modifying edit waits for you. Approving a source edit caps affected verdicts at Partially reproduced.

Security model

Threats and controls

ThreatControl
Malicious install scripts (setup.py, post-install hooks)Dependencies build inside the sandbox image only. No network at run time, non-root user, read-only root filesystem, nothing from the host mounted.
Sandbox escapegVisor user-space kernel on top of Docker, seccomp profile, dropped capabilities, no privileged containers. The GPU host is a different machine.
Exfiltration of data or secretsNo outbound network at run time and no secrets in the sandbox. The only way out is the narrow exec API, which returns fixed-schema run records.
Fork bombs and runaway jobscgroup limits on processes, CPU and memory, plus wall-clock caps per run. A killed run is Blocked, never counted against the paper.
Prompt injection from the paper or repoModel output is JSON-schema constrained and verdicts come from code, so injected text cannot set a verdict. The Integrity Shield also scans the paper and repo for hidden instructions (white-on-white text, invisible characters, comment payloads) and reports them as findings.
Sources

Citations

  1. 1Pimentel, Murta, Braganholo, Freire. "A Large-scale Study about Quality and Reproducibility of Jupyter Notebooks." MSR 2019. doi:10.1109/MSR.2019.00077 doi.org/10.1109/MSR.2019.00077
  2. 2Raff, E. "A Step Toward Quantifying Independently Reproducible Machine Learning Research." NeurIPS 2019. arXiv:1909.06674 arxiv.org/abs/1909.06674
  3. 3NeurIPS 2025 Program Chairs. "Reflections on the 2025 Review Process." blog.neurips.cc, 30 Sep 2025. blog.neurips.cc/2025/09/30/reflections-on-the-2025-review-process-from-the-program-committee-chairs/
  4. 4Codesota. "Papers with Code shutdown record." codesota.com/papers-with-code/shutdown www.codesota.com/papers-with-code/shutdown
  5. 5Starace et al. "PaperBench: Evaluating AI's Ability to Replicate AI Research." ICML 2025.
  6. 6Siegel, Kapoor, Nadgir, Stroebl, Narayanan. "CORE-Bench." TMLR 2024. arXiv:2409.11363 arxiv.org/abs/2409.11363
  7. 7Kapoor, Narayanan. "Leakage and the Reproducibility Crisis in Machine-Learning-Based Science." Patterns, 2023. doi:10.1016/j.patter.2023.100804 doi.org/10.1016/j.patter.2023.100804
  8. 8Pineau et al. "Improving Reproducibility in Machine Learning Research." JMLR 22(164), 2021.
  9. 9Bouthillier et al. "Accounting for Variance in Machine Learning Benchmarks." MLSys 2021. arXiv:2103.03098 arxiv.org/abs/2103.03098
  10. 10Picard, D. "torch.manual_seed(3407) is all you need." arXiv:2109.08203, 2021. arxiv.org/abs/2109.08203
  11. 11Li et al. "ReproRepo: Scaling Reproducibility Audits with GitHub Repository Issues." arXiv:2606.18237, 2026. arxiv.org/pdf/2606.18237
  12. 12ACM. "Artifact Review and Badging" (current version). acm.org/publications/policies/artifact-review-and-badging-current www.acm.org/publications/policies/artifact-review-and-badging-current
  13. 13Menke et al. "The Rigor and Transparency Index Quality Metric…" (SciScore). iScience, 2020.
  14. 14Sumner et al. "ripetaScore." Frontiers in Research Metrics and Analytics, 2021; and Chemistry World coverage of Ripeta in Editorial Manager.
  15. 15"Reproducing ICML 2026" community challenge on Hugging Face (15 Jul – 2 Aug 2026), as reported by howaiworks.ai.
  16. 16"Paper2Agent." Nature, September 2026. doi:10.1038/s41586-026-11044-y doi.org/10.1038/s41586-026-11044-y
  17. 17Seo et al. "Paper2Code." ICLR 2026. arXiv:2504.17192 · Zhao et al. "AutoReproduce." ACL 2026. arXiv:2505.20662 arxiv.org/abs/2504.17192
  18. 18PyTorch documentation, "Reproducibility / Randomness" notes. pytorch.org/docs/stable/notes/randomness.html
  19. 19Nikkei Asia, July 2025: hidden prompts in arXiv preprints.
  20. 20ML Reproducibility Challenge 2026, official NeurIPS track.