A reproducibility tool has to be reproducible itself. These are the checks we run on the pipeline, and two of them you can run yourself, right here.
We describe how each check works. We do not print aggregate scores for Claimband here; the audits in the library show what happened on specific papers.
We fork a repository with known ground truth and plant one defect at a time: a changed config value, a removed stratification, a leak before the split, a deleted seed call, a breaking dependency pin, a missing artifact, an absolute path, a hidden instruction. A detection counts only if the right detector fires and the verdict or flag matches the plant. The playground below replays these.
We rerun the same code under different seeds on pairs where nothing was changed, then check how often the band rule calls pure noise a failure. The explorer below runs this experiment on the decision rule itself, so you can see the false-alarm share move with k, policy and seed spread.
Papers with hand-annotated claim ledgers are the reference. We compare extracted claims field by field (metric, value, dataset, split, location). Every extracted field must also carry a source span that exists in the paper text; fields without one are rejected before the ledger is shown.
For repositories where the authors name the script behind each table, we check whether the Claim→Code Mapping lands on the same file and metric site, and whether the dry run confirms it. Low-confidence mappings are routed to a human gate rather than guessed.
We run anonymised tasks from the public CORE-Bench benchmark (source 6) through the pipeline and compare against its answer keys. The goal is to catch systematic disagreement with an independent reference, not to publish a leaderboard score.
The same paper and repository are audited repeatedly. Because verdicts come from deterministic code over run records, differences can only come from the proposal steps (extraction, mapping, fixes). We inspect every case where two audits of one input disagree.
Each toggle is a defect planted in the CalibBoost fork (audit RL-0415). Verdicts shown are computed by the engine from that audit under the Standard policy.
Both the author's number and our k runs are drawn from the same distribution, so there is no real difference to find. Any Not reproduced call is a false alarm of the tolerance-band test.
Fixed for this experiment: true mean 85, reported to 2 decimals, platform floor 0.10 (scaled by the policy).
Property of the decision rule, computed in your browser.
Language-model agents are judged by pass^k: the chance that the agent succeeds on all k independent attempts at the same task. If one attempt succeeds with probability p, pass^k is p to the power k, so reliability falls quickly as the number of repeats grows.
That is the reason the model never writes a verdict or a number. It proposes claims, mappings and fixes; deterministic code parses metrics, computes the band and decides. The same run records always give the same verdict, so the only thing that can vary between audits is a proposal, and every proposal is checked against the text or the code before it counts.
Where a proposal is uncertain, the audit says so: the evidence strength drops and a human gate opens.
| Repeats (k) | Chance all k succeed |
|---|---|
| 1 | 90% |
| 3 | 73% |
| 5 | 59% |
| 10 | 35% |
For an illustrative step that succeeds 90% of the time per attempt. A deterministic verdict step has no such decay: same inputs, same output.