Claimband reproducibility report

CalibBoost: Margin-Calibrated Gradient Boosting for Tabular Risk Scoring

Ilse Marrow, Devansh Kulkarni-Rao, Teodor Vasq · Northgate Institute of Technology

Audit ID: RL-0412
Repository: tabular-lab/calibboost @ a41c9e2
Policy: Standard
Completed: 01 Oct 2026
Seeds per claim: k = 5
Compute: 1 h 41 m on 4 vCPU

4Reproduced
2Partially reproduced
1Not reproduced
1Blocked
1Unverifiable
1Not computational

Coverage

77.8%

7 of 9 computational claims executed

Reproduction rate

70.6%

weighted, headline claims count 3×

Claims confirmed

57.1%

4 of 7 reproduced

Blocked

1

not counted against the paper

Flags

4

Code ≠ Paper, or suspect

Claims

10

9 computational

Certificate

RLC-0412-7F3A

digest 1b4d4d7038973eb1…882d1d33

/verify/RLC-0412-7F3A

Citation validator: 36/36 sentences linked

2Executive summary

1
4 of 7 executed claims reproduced under the Standard policy; coverage is 77.8% and the weighted reproduction rate is 70.6%.[Table 2 · row 1 · Acc] [RUN-0412-C1-s0] [Table 2 · row 1 · ECE] [RUN-0412-C2-s0]
2
C7 (German Credit Brier) is not reproduced: Outside the band in a verified environment; counterfactuals did not close the gap.[Table 2 · row 3 · Brier] [RUN-0412-C7-s0]
3
C5: Feature selection fit before the split. SelectKBest sees test labels (prep.py:21-23). Fitting on the training split only gives 95.08 ± 0.24 (−1.24 pp). The reported 96.2 is likely inflated.[prep.py:21-23] [§4.1 · Datasets and splits]
4
C4 (Ablation AUC) is partially reproduced: Reproduced under the paper's stated setting; the repository default diverges. Setting n_estimators to the paper's value closes 94% of the gap.[configs/ablation.yaml:11-12] [§4.2 · Hyperparameters] [Table 3 · row 2 · AUC]
5
Blocked: C8. Unverifiable: C9. Blocked claims are not counted against the paper.[Table 5 · row 2] [load_data.py:13] [FileNotFoundError: data/covertype_splits.pkl] [§5.1 · sentence 2] [eval.py:37] [§5.1 · sentence 2]

Citation validator: 36/36 sentences linked

3Claim table

ID	Claim	Source	Paper	Ours (mean ± CI, k)	Band	Verdict	Flags	Evidence
C1	Adult · CalibBoost · 87.4 ± 0.2	Table 2 · row 1 · Acc	87.4%	87.31 ± 0.09 (k=5)	±0.20	Reproduced	–	High
C2	1.9 ± 0.1	Table 2 · row 1 · ECE	1.9%	2.08 ± 0.04 (k=5)	±0.20	Reproduced	–	High
C3	it improves AUC over XGBoost by 2.1 points	Table 2 · row 3 · AUC	2.1 pp	0.90 ± 0.50 (k=5)	±0.50	Partially reproduced	–	High
C4	w/o margin term · 74.8	Table 3 · row 2 · AUC	74.8 pp	77.60 ± 0.30 (k=5)	±0.30	Partially reproduced	Code ≠ Paper	High
C5	CalibBoost · 96.2	Table 4 · row 1 · F1	96.2%	96.32 ± 0.20 (k=5)	±0.20	Reproduced	Reproducible but suspect, Code ≠ Paper	High
C6	ECE decreases monotonically as the margin weight λ increases.	Figure 3 · caption	–	monotone 5/5, ρ -1.00	–	Reproduced	–	Medium
C7	German Credit · CalibBoost · … · 0.162	Table 2 · row 3 · Brier	0.162	0.181 ± 0.002 (k=5)	±0.002	Not reproduced	Code ≠ Paper	High
C8	CalibBoost also trains 3.2× faster than CatBoost on Covertype.	Table 5 · row 2	3.2×	–	–	Blocked	–	Low
C9	On Adult, CalibBoost also achieves an F1 of 80.1.	§5.1 · sentence 2	80.1%	–	–	Unverifiable	–	Low
C10	71% (10 of 14) preferred CalibBoost.	§6 · Analyst study	71%	–	–	Not computational	–	Low

Verdicts come from the tolerance-band test under the Standard policy. Band is the half-width around our mean.

4Forest plot

Forest plot

Each row is scaled to its own tolerance band: the shaded area is −1 to +1 band widths around our mean. ● ours, ◆ paper.

Claim−3 δpaper − ours, in band widths+3 δEvidence · flags

Reproduced
paper 87.4 · ours 87.31 ± 0.09
Reproduced
paper 1.9 · ours 2.08 ± 0.04
Partially reproduced
paper 2.1 · ours 0.90 ± 0.50
Partially reproduced
paper 74.8 · ours 77.60 ± 0.30(paper at −9.2 δ)
Code ≠ Paper
Reproduced
paper 96.2 · ours 96.32 ± 0.20
Reproducible but suspectCode ≠ Paper
Reproduced
monotone in 5 / 5 seeds · ρ -1.00
Not reproduced
paper 0.162 · ours 0.181 ± 0.002(paper at −9.5 δ)
Code ≠ Paper
Blocked
Missing artifact data/covertype_splits.pkl (referenced at load_data.py:13 and in the README; not in the repository and not downloadable). Estimated run time if available: 24 min per seed.
Unverifiable
The paper does not state the F1 averaging or the split for this number, and no code path emits 80.1 under any averaging. Variants computed from the Adult runs: binary 70.6, macro 78.9, weighted 86.8, micro 87.3 (all k = 5). None is within band of 80.1.
Not computational
Human study (14 analysts). No computation to rerun.

5Discrepancies and attributions

C4Ablation AUCPartially reproduced

C4 is partially reproduced: Reproduced under the paper's stated setting; the repository default diverges.[Table 3 · row 2 · AUC] [RUN-0412-C4-s0]
Counterfactual on n_estimators: 100 (repo, configs/ablation.yaml:12) → 500 (paper §4.2). Mean 74.98 over 5 seeds. Gap closed: 94%.[configs/ablation.yaml:11-12] [§4.2 · Hyperparameters]

C5Breast Cancer F1Reproduced

C5 is reproduced: Faithful run lies within the tolerance band.[Table 4 · row 1 · F1] [RUN-0412-C5-s0]
Counterfactual on feature selection fit: all rows (prep.py:21-23) → training split only (paper §4.1). Mean 95.08 over 5 seeds. Shift against the faithful run: −1.24.[prep.py:21-23] [§4.1 · Datasets and splits]
Feature selection fit before the split. SelectKBest sees test labels (prep.py:21-23). Fitting on the training split only gives 95.08 ± 0.24 (−1.24 pp). The reported 96.2 is likely inflated.[prep.py:21-23] [§4.1 · Datasets and splits]
Feature selection differs from §4.1. The paper selects features on the training split; the code selects on all rows before the split.[prep.py:21-23] [§4.1 · Datasets and splits]

C7German Credit BrierNot reproduced

C7 is not reproduced: Outside the band in a verified environment; counterfactuals did not close the gap.[Table 2 · row 3 · Brier] [RUN-0412-C7-s0]
Counterfactual on split: unstratified (prep.py:25-30) → stratified (paper §4.1). Mean 0.178 over 5 seeds. Gap closed: 16%.[prep.py:25-30] [§4.1 · Datasets and splits]
Counterfactual on learning rate: confirm 0.05 (configs/table2.yaml:6) as stated. Mean 0.181 over 5 seeds. Gap closed: 0%.[configs/table2.yaml:6] [§4.2 · Hyperparameters]
Split not stratified (§4.1). Stratifying moves Brier by 0.003, more than the band (0.002); explains 16% of the gap.[prep.py:25-30] [§4.1 · Datasets and splits]

C9Adult F1Unverifiable

C9 is unverifiable: The paper does not state the F1 averaging or the split for this number, and no code path emits 80.1 under any averaging. Variants computed from the Adult runs: binary 70.6, macro 78.9, weighted 86.8, micro 87.3 (all k = 5). None is within band of 80.1.[§5.1 · sentence 2] [eval.py:37] [§5.1 · sentence 2]

6Missing information and Fix Log

Missing information

1. F1 averaging for §5.1 is not stated (binary, macro, micro or weighted).[§5.1 · sentence 2] [eval.py:37]
2. The Covertype split file is referenced but not released.[load_data.py:13]
3. ECE binning is not stated (the code uses 15 bins).[eval.py:11]
4. Seed values are not listed; the code uses one fixed seed.[train.py:18]
5. Dependency versions are not pinned.[requirements.txt:1-9]
6. The calibration hold-out fraction is not stated.[§4.1 · Datasets and splits]

Fix Log

Environment-only changes · 5

1Pinned 14 packages to versions current on 2023-02-14. requirements.txt has no version pins[requirements.txt:1-9]
2CUDA_VISIBLE_DEVICES="" and the CPU torch wheel. torch is imported but unused for training[utils/device.py:1-8]
3CPU shim for get_device() that returns CUDA. The helper returns a CUDA device even without a GPU[utils/device.py:8]
4Path remap /home/author/data → /data (read-only dataset cache). Absolute path on the authors' machine[load_data.py:8]
5Seed injection: the runner sets train.SEED per run through an import hook; repository files unchanged. The code uses one fixed seed, so a seed loop needs to come from outside[train.py:18]

Source changes · 0

None.

7Insights

Seed Lottery

Seed Lottery, Adult accuracy (%): the published 87.4 sits at the 84th percentile of 50 seeds (risk low).[Table 2 · row 1 · Acc] [RUN-0412-C1-s0]
Seed Lottery, Paired AUC difference vs XGBoost (pp): the published 2.1 sits at the 96th percentile of 50 seeds (risk high). The method wins in 88% of seeds.[Table 2 · row 3 · AUC] [RUN-0412-C3-s0]
Seed Lottery, Breast Cancer F1 (%): the published 96.2 sits at the 26th percentile of 50 seeds (risk low).[Table 4 · row 1 · F1] [RUN-0412-C5-s0]

Rot Radar

Rot Radar replayed the repository against 6 dated library snapshots.[Rot Radar: 6 snapshots]
The first hard break is 2023-04 · pandas 2.0.0, 48 days after the paper date: AttributeError: 'DataFrame' object has no attribute 'append'[load_data.py:21]
Pin pandas<2.0 and xgboost<2.0 and scikit-learn==1.2.1 to keep the results runnable.[load_data.py:21] [Rot Radar: 6 snapshots]

Issue Radar

#	Issue	Claim	Status	Evidence
7	AttributeError: 'DataFrame' object has no attribute 'append'	–	Confirmed	[Rot Radar 2023-04: hard break at load_data.…] [load_data.py:21]
12	Can't reproduce Table 3 ablation (I get ~77.5, not 74.8)	C4	Confirmed + explained	[CF-C4-a] [configs/ablation.yaml:12]
15	Where is data/covertype_splits.pkl?	C8	Confirmed	[load_data.py:13]
18	LightGBM support?	–	Not about reproduction
21	Brier score on German Credit is ~0.18, not 0.162	C7	Confirmed	[RUN-0412-C7-s0]
24	Which F1 is reported in Section 5.1?	C9	Confirmed (missing information)	[eval.py:37]

Checklist Truth Table

Checklist item	Authors say	Finding	Evidence
Claims	Yes	Partly supported	[RUN-0412-C3-s0] [RUN-0412-C7-s0] [load_data.py:13]
Experimental result reproducibility	Yes	Partly supported	[README.md:39] [configs/ablation.yaml:12] [load_data.py:13]
Open access to data and code	Yes	Partly supported	[load_data.py:13]
Experimental setting / details	Yes	Contradicted	[configs/ablation.yaml:12] [prep.py:25-30] [prep.py:21-23]
Experiment statistical significance	Yes	Not supported	[train.py:18] [README.md:37]
Experiments compute resources	Yes	Verified	[CPU time 5 h 12 m · wall 1 h 41 m on 4 vCPU]

8Environment

Image python:3.10.9-slim built in 4 m 12 s; digest sha256:7c1e4b0d9f2…9a02; dependencies resolved as of 2023-02-14.[Image built · sha256:7c1e…9a02 · 4 m 12 s]
Sandbox: gVisor, 4 vCPU, 8 GB RAM, 512 processes, 20 GB disk, network off, budget 90 min.[Sandbox up: gVisor · 4 vCPU · 8 GB · networ…]
All runs used 4 CPU cores and no GPU. The repository selects a CUDA device, so the authors most likely trained on a GPU; small numeric differences between platforms are covered by the platform floor in the tolerance band.[utils/device.py:6]

Lockfile excerpt

numpy==1.24.2   # unpinned in the repository
pandas==1.5.3   # unpinned in the repository
scikit-learn==1.2.1   # unpinned in the repository
xgboost==1.7.3   # unpinned in the repository
scipy==1.10.0   # unpinned in the repository
pyyaml==6.0   # unpinned in the repository
tqdm==4.64.1   # unpinned in the repository
torch==1.13.1 (CPU wheel)   # unpinned in the repository
matplotlib==3.6.3   # unpinned in the repository
joblib==1.2.0   # unpinned in the repository
threadpoolctl==3.1.0   # unpinned in the repository
python-dateutil==2.8.2   # unpinned in the repository
# … 2 more

Environment health

Imports	ok
Tiny forward pass	ok
Dataset hash · adult	ok · sha256:3f9a…c21e
Dataset hash · german_credit	ok · sha256:b07d…4a8f
Dataset hash · breast_cancer	ok · sha256:e51c…07bd
Covertype split file	failed · data/covertype_splits.pkl is missing

9Appendix: run records

70 run records. Each line is a run ID and the first 8 characters of the SHA-256 of its ID, value and image digest. The full records open from any run chip.

RUN-0412-C1-s0········
RUN-0412-C1-s1········
RUN-0412-C1-s2········
RUN-0412-C1-s3········
RUN-0412-C1-s4········
RUN-0412-C2-s0········
RUN-0412-C2-s1········
RUN-0412-C2-s2········
RUN-0412-C2-s3········
RUN-0412-C2-s4········
RUN-0412-C3-s0········
RUN-0412-C3-s1········
RUN-0412-C3-s2········
RUN-0412-C3-s3········
RUN-0412-C3-s4········
RUN-0412-C4-s0········
RUN-0412-C4-s1········
RUN-0412-C4-s2········
RUN-0412-C4-s3········
RUN-0412-C4-s4········
RUN-0412-C5-s0········
RUN-0412-C5-s1········
RUN-0412-C5-s2········
RUN-0412-C5-s3········
RUN-0412-C5-s4········
RUN-0412-C6-s0········
RUN-0412-C6-s1········
RUN-0412-C6-s2········
RUN-0412-C6-s3········
RUN-0412-C6-s4········
RUN-0412-C7-s0········
RUN-0412-C7-s1········
RUN-0412-C7-s2········
RUN-0412-C7-s3········
RUN-0412-C7-s4········
CF-C4-a-s0········
CF-C4-a-s1········
CF-C4-a-s2········
CF-C4-a-s3········
CF-C4-a-s4········
CF-C5-a-s0········
CF-C5-a-s1········
CF-C5-a-s2········
CF-C5-a-s3········
CF-C5-a-s4········
CF-C7-a-s0········
CF-C7-a-s1········
CF-C7-a-s2········
CF-C7-a-s3········
CF-C7-a-s4········
CF-C7-b-s0········
CF-C7-b-s1········
CF-C7-b-s2········
CF-C7-b-s3········
CF-C7-b-s4········
SC-0412-G1········
SC-0412-G2········
SC-0412-G3········
SC-0412-G4········
SC-0412-G5········
SC-0412-G6········
DRY-0412-C1········
DRY-0412-C2········
DRY-0412-C3········
DRY-0412-C4········
DRY-0412-C5········
DRY-0412-C6········
DRY-0412-C7········
DRY-0412-C8········
DRY-0412-C9········