We don't check whether the code runs. We check whether the claim holds.
Give Claimband a paper and its repository. It finds every number the paper claims, runs the code that produces it in a locked-down sandbox, and tells you claim by claim what reproduced, what didn’t, and why.
- Three-way cause separation
- Per-claim verdicts
- Statistical tolerance bands
- Counterfactual explanations
- Evidence on every line
| Dataset | Model | Acc (%) | ECE (%) | AUC | Brier |
|---|---|---|---|---|---|
| Adult | CalibBoost | 87.4 ± 0.2 | 1.9 ± 0.1 | – | – |
| German Credit | XGBoost | – | – | 75.8 | – |
| German Credit | CalibBoost | – | – | 77.9 | 0.162 |
| Dataset | Model | Acc (%) | ECE (%) | AUC | Brier |
|---|---|---|---|---|---|
| Adult | CalibBoost | 87.4 ± 0.2 | 1.9 ± 0.1 | – | – |
| German Credit | XGBoost | – | – | 75.8 | – |
| German Credit | CalibBoost | – | – | 77.9 | 0.162 |
| Dataset | Model | Acc (%) | ECE (%) | AUC | Brier |
|---|---|---|---|---|---|
| Adult | CalibBoost | 87.4 ± 0.2 | 1.9 ± 0.1 | – | – |
| German Credit | XGBoost | – | – | 75.8 | – |
| German Credit | CalibBoost | – | – | 77.9 | 0.162 |
| Dataset | Model | Acc (%) | ECE (%) | AUC | Brier |
|---|---|---|---|---|---|
| Adult | CalibBoost | 87.4 ± 0.2 | 1.9 ± 0.1 | – | – |
| German Credit | XGBoost | – | – | 75.8 | – |
| German Credit | CalibBoost | – | – | 77.9 | 0.162 |
“It runs” is not an answer.
One green tick for the whole repository. It says nothing about any number in the paper.
Same repository. A very different story.
Three causes, kept apart
A pass/fail check lumps every failure together. Claimband separates why a claim is not green: the environment broke (Blocked, and never held against the paper), it is seed noise (inside the tolerance band), or it is a genuine discrepancy (outside the band in a verified environment). In RL-0412 that is 1 blocked, 4 inside the band and 3 outside it; select any claim to open it.
The problem in numbers
- 24% / 4%
of GitHub Jupyter notebooks ran without errors / reproduced the same results.
Pimentel et al., MSR 2019 [1] - 63.5%
of 255 ML papers were independently reproduced.
Raff, NeurIPS 2019 [2] - 21,575
main-track submissions to NeurIPS 2025, reviewed by about 20,500 people.
NeurIPS 2025 program chairs [3] - July 2025
Papers with Code was retired; nothing public records whether linked code reproduces a paper.
Codesota [4]
Six steps from PDF to verdict
- 1
Read
Pull every quantitative claim, with its table cell and quote.
Claim Ledger - 2
Map
Link each claim to the script, config and metric that produce it.
Mapping - 3
Build
Pin the environment to the paper’s date; log every fix.
Env Lock - 4
Run
Rerun each claim k times on CPU, network off.
Run Records - 5
Judge
Compare against a statistical tolerance band; assign one of six verdicts.
Verdicts - 6
Explain
Rerun one-factor variants to explain each gap; write the report.
Report
Six things no other tool does
Every visual below is drawn from audit RL-0412. Open any card to see it in the workspace.
Seed Lottery
Where does the paper’s number sit among 50 seeds of the same code?
The published margin is a 1-in-25 seed.
Open in auditRot Radar
How long does a paper stay reproducible, and which release broke it?
Hatched = blocked after the first breaking releaseUnpinned shelf life: ~2 months.
Open in auditIssue Radar
What real users reported, linked to claims and confirmed by execution.
- #7AttributeError: 'DataFrame' object has no attribute 'append'Confirmed
- #12Can't reproduce Table 3 ablation (I get ~77.5, not 74.8)Confirmed + explained
- #15Where is data/covertype_splits.pkl?Confirmed
- #21Brier score on German Credit is ~0.18, not 0.162Confirmed
- #24Which F1 is reported in Section 5.1?Confirmed
5 of 5 reproduction issues confirmed.
Open in auditFix-Pack
A patch the authors can apply: pins, seeds, config fixes, a claim manifest.
README.md−pip install -r requirements.txt+pip install -r requirements.txt # versions pinned as of 2023-02-14−Tested with Python 3.10.+Tested with Python 3.10.9. A `Dockerfile` with the same pins is included.From diagnosis to pull request: +109 −33 across 10 files.
Open in auditChecklist Truth Table
What the authors said on the checklist vs what the code did.
- ClaimsPartially verified
- Experimental result reproducibilityPartially verified
- Open access to data and codePartially verified
- Experimental setting / detailsContradicted
‘5 seeds’, one hard-coded seed.
Open in auditReproCert
A per-claim certificate and badge anyone can verify in the browser.
Claimband4/7 claims · StandardRLC-0412-7F3A · SHA-256Edit one field and it breaks.
Open in audit
- Integrity ShieldHidden-instruction scan on every paper and repo.
- Human gatesYou approve the claim ledger and mappings before runs.
- Time-travel pinningEnvironments pinned to the paper’s date.
- Environment-only vs source-fix logEvery fix logged, classed by what it touches.
- Forest plotsEvery claim’s band and paper value on one axis.
- Verdict JSONThe whole audit as machine-readable JSON.
Six verdicts. Two flags. No single score.
- Reproduced
Faithful run matches within the tolerance band.
- Partially reproduced
Direction holds, or it matches only under the paper's stated settings.
- Not reproduced
Outside the band in a verified environment; counterfactuals did not close the gap.
- Blocked
Couldn't run. Not counted against the paper.
- Unverifiable
Can't be mapped to code, or key details are missing.
- Not computational
Nothing to rerun (theorem, human study, private data or GPU-scale).
They attach to any verdict.
- Code ≠ Paper
The implementation differs from the method described in the paper.
- Reproducible but suspect
The number reproduces, but a methodological red flag was found (for example leakage).
Where Claimband fits
| Uses the authors’ code | Per-claim verdict | Statistical band | Explains gaps | Fix for authors | Verifiable certificate | |
|---|---|---|---|---|---|---|
| PaperBench | No. | No. | No. | No. | No. | No. |
| Paper2Code · AutoReproduce | No. | No. | No. | No. | No. | No. |
| CORE-Bench agents | Yes. | Partial | No. | No. | No. | No. |
| Community reproduction logbooks | Yes. | No. | No. | Partial | No. | Partial |
| SciScore · Ripeta | No. | No. | No. | No. | Partial | No. |
| ACM artifact badges | Yes. | Partial | No. | No. | No. | Partial |
| Claimband | Yes. | Yes. | Yes. | Yes. | Yes. | Yes. |
Based on public descriptions as of October 2026. ✓ yes · – no · Partial = some of it.
Built for the people who have to trust the number
Authors
Run it before you submit. Fix what reviewers would find.
Open Fix-PackReviewers and area chairs
See which headline numbers hold in two minutes.
Open Venue consoleReproducers
Go from clone to a first faithful run, with a hypothesis for every gap.
Open DiscrepanciesPractitioners
Check the number is real before you build on it.
Open Seed Lottery
The GPU never touches untrusted code.
Reasoning runs on a GPU host. Experiments run only in a CPU sandbox with no network at run time. Verdicts are computed by deterministic code that text in a paper can’t change.
- gVisor isolation, non-root, read-only root
- Network off at run time; packages only from an allow-listed mirror at build
- Hidden-instruction scan on every paper and repo
Bring a paper. Leave with evidence.
Sources (20)
- 1Pimentel, Murta, Braganholo, Freire. "A Large-scale Study about Quality and Reproducibility of Jupyter Notebooks." MSR 2019. doi:10.1109/MSR.2019.00077The problem in numbers
- 2Raff, E. "A Step Toward Quantifying Independently Reproducible Machine Learning Research." NeurIPS 2019. arXiv:1909.06674The problem in numbers
- 3NeurIPS 2025 Program Chairs. "Reflections on the 2025 Review Process." blog.neurips.cc, 30 Sep 2025.The problem in numbers
- 4Codesota. "Papers with Code shutdown record." codesota.com/papers-with-code/shutdownThe problem in numbers
- 5Starace et al. "PaperBench: Evaluating AI's Ability to Replicate AI Research." ICML 2025.Where we fit
- 6Siegel, Kapoor, Nadgir, Stroebl, Narayanan. "CORE-Bench." TMLR 2024. arXiv:2409.11363Where we fit
- 7Kapoor, Narayanan. "Leakage and the Reproducibility Crisis in Machine-Learning-Based Science." Patterns, 2023. doi:10.1016/j.patter.2023.100804
- 8Pineau et al. "Improving Reproducibility in Machine Learning Research." JMLR 22(164), 2021.
- 9Bouthillier et al. "Accounting for Variance in Machine Learning Benchmarks." MLSys 2021. arXiv:2103.03098
- 10Picard, D. "torch.manual_seed(3407) is all you need." arXiv:2109.08203, 2021.
- 11Li et al. "ReproRepo: Scaling Reproducibility Audits with GitHub Repository Issues." arXiv:2606.18237, 2026.
- 12ACM. "Artifact Review and Badging" (current version). acm.org/publications/policies/artifact-review-and-badging-currentWhere we fit
- 13Menke et al. "The Rigor and Transparency Index Quality Metric…" (SciScore). iScience, 2020.Where we fit
- 14Sumner et al. "ripetaScore." Frontiers in Research Metrics and Analytics, 2021; and Chemistry World coverage of Ripeta in Editorial Manager.Where we fit
- 15"Reproducing ICML 2026" community challenge on Hugging Face (15 Jul – 2 Aug 2026), as reported by howaiworks.ai.Where we fit
- 16"Paper2Agent." Nature, September 2026. doi:10.1038/s41586-026-11044-y
- 17Seo et al. "Paper2Code." ICLR 2026. arXiv:2504.17192 · Zhao et al. "AutoReproduce." ACL 2026. arXiv:2505.20662Where we fit
- 18PyTorch documentation, "Reproducibility / Randomness" notes.
- 19Nikkei Asia, July 2025: hidden prompts in arXiv preprints.
- 20ML Reproducibility Challenge 2026, official NeurIPS track.