Benchmark checks that cannot return a negative verdict under the design that ships.
A benchmark's headline usually rests on a claim about its own construction, such as the two arms are matched or this control is at chance, and a check in the harness guards it. Some of those checks cannot fail: under the design that ships, no observation they could realistically receive would make them report a negative. Such a check is green in CI and looks exactly like one that was earned.
cannotfail runs two of the three diagnostics from the paper
Checks That Cannot Fail
(Shah, 2026): D1 on a score file your benchmark already produces, and D2 on the numbers that
describe its design. It imports nothing from the benchmark it was built in.
Python 3.10 or later. D1 needs only NumPy; the [power] extra adds SciPy,
which D2's equivalence half needs.
pip install cannotfail
pip install "cannotfail[power]"
Or run it without installing, with uv:
uvx cannotfail resolution scores.csv --score score --label is_hack --group cell
uvx --from "cannotfail[power]" cannotfail power seeds --layouts 12 --tests 4
If pip or uv finds no release yet, or for the development version, install from the repository:
pip install "cannotfail @ git+https://github.com/Aarav500/rhob#subdirectory=tools/cannotfail"
To try the example below, fetch its input first:
curl -O https://raw.githubusercontent.com/Aarav500/rhob/main/tools/cannotfail/examples/scores.csv
The file needs one row per scored item: the score, optionally its class, and optionally
the cell it belongs to (a dataset, a detector-by-task pair, whatever your headline
aggregates over). It reads CSV and TSV, a .json list of objects, a
.jsonl file, Parquet with the [parquet] extra, and Inspect logs
with the [inspect] extra and --format inspect; a nested field is
named by a dotted path, and --group can be given more than once, so a
leaderboard prediction file or an evaluation harness's per-sample log can be checked as it
is. Here it is on
examples/scores.csv,
three cells of 48 items each:
$ cannotfail resolution scores.csv --score score --label is_hack --group cell
group n distinct pos neg resolution AUROC range verdict
reward_only 48 1 24 24 0.000 [0.500, 0.500] fixed (one distinct value)
mostly_tied 48 2 24 24 0.042 [0.479, 0.521] pinned (AUROC confined within 0.5 +/- 0.0208)
behavioural 48 48 24 24 1.000 [0.000, 1.000] ok
1 of 3 groups has at most one distinct value, so any rank statistic over it is fixed at 0.5.
1 of 3 groups is pinned: the AUROC cannot get further from 0.5 than 0.1, so its estimate cannot leave an equivalence band of that width. A test that also needs its interval inside the band can still fail, but only on the interval's width; a pass there says the scores tie.
--fail-on fixed pinned makes CI fail when either verdict occurs, and
--json gives machine-readable output. Without --label it counts
distinct values only, which is enough to find a cell whose scores are all one number.
The package has its own CI, separate from the benchmark's. It runs on every push and pull
request that touches tools/cannotfail, every Monday, and on demand. It tests on
Python 3.10, 3.13 and 3.14, each first with NumPy only, asserting that SciPy, pyarrow,
pandas and inspect_ai are all absent, which holds D1 to its NumPy-only promise, and then
with SciPy, with pyarrow and with the Inspect extra in turn. It also builds the sdist and
wheel, installs the wheel on its own, and checks that the examples in the package's README
print what the README shows: those that need no extra with NumPy only, then all of them
with every extra.
Separately, tests/test_cannotfail_parity.py runs in the benchmark's
test suite on every push to main and holds the tool to the admission gate it was
lifted from. Its tolerance resolution is bit-identical to the gate's on adversarial inputs,
with one documented exception, a pair whose difference overflows float64. Fed the gate's
measured resolution and informativeness, its verdict rule marks the same 30 of the 50
ledger cells degenerate; that checks the rule's direction, and separate boundary tests pin
its cut. The
package's recent runs.
D1, resolution: what range could the statistic have taken? In a
standard AUROC, the one sklearn.metrics.roc_auc_score computes, every
cross-class pair of equal scores contributes exactly 0.5, whatever the data. So if a
fraction 1 − r of those pairs are equal, the AUROC is confined to
0.5 ± r/2. That is an identity, not an estimate. The
resolution column above is that r.
D2, power: at the design you ship, could the precondition have failed?
For an equivalence precondition certified by an equivalence test (TOST),
cannotfail power equivalence says whether your items per side and number of
independent blocks can certify at the margin at all, and power seeds says how
many items per side would. RHOB's admission gate used 4 a side and needed 24, which it now
runs. For a precondition asserted from a non-significant difference test (“no
significant difference, so the arms are matched”), power difference says
how often that test would have caught the smallest violation that matters.
D3, channel digest, is not implemented. It asks whether a parameter reaches anything the evaluation reads, and answering that needs knowledge of your harness that a score file does not carry.
| Verdict | Meaning |
|---|---|
fixed | A standard AUROC over these scores is 0.5 whatever happens: at most one distinct value, no unequal pair across the classes, or a class with no finite score. Report the cell as not measured, never as a result. |
pinned | A standard AUROC cannot get further from 0.5 than the
margin (--margin, default 0.10), so its estimate can never leave an equivalence
band of that width. A pass there is decided by the ties, not by the classes. |
dust | The scores differ, but only within the tie tolerance,
|a − b| ≤ max(1e-12, 1e-9 · max(|a|, |b|)), or,
with --dispersion-floor, by less than that fraction of their own size. Decide
whether that is signal or rounding, and say which. |
ok | D1 found no reason the check cannot fail. |
ok is not evidence that a check is sound. It means only
that D1 found no reason the check cannot fail. D2's equivalence power assumes no
heterogeneity between blocks, so it is a ceiling: real blocks vary and real power is
lower. If several items share a prompt, task or seed, its difference-power figure is not
a bound. And nothing here answers D3.
What
it cannot tell you.
RHOB is the case study, not the subject. It is 33 reward-hacking environment families. Each mints pairs of a hacking run and a legitimate run whose proxy reward is meant to be indistinguishable while the true reward diverges, and an admission gate decides, criterion by criterion, whether the construction achieved that. Every criterion returns PASS, FAIL (measured, and outside the margin) or DEGENERATE (not measurable), and the admission ledger publishes all three:
Out of 50 cells, in 10 of the 33 families. A degenerate cell is neither certified nor refuted. The other 23 families are uncertified, which is not the same as certified-and-passing.
The paper counts seven instances of the class in RHOB, three of them introduced by the
fix to an earlier one. Two more were found in the released artifact after the paper
(b796f9c, 2026-09-02).
Reward Skewness returned a constant 0.5 below its 100-episode horizon, on the 19 families
that run 40 or 60 episodes, which padded the L0 negative control. And
true_reward_diverges reported FAIL on arms tied by arithmetic, with no tie
guard; it now returns DEGENERATE.
RHOB was published as the Reward Hacking Onset Benchmark. Its onset column was removed because it could not fail either: its onsets are authored, so their spread is exactly 0.0 in all 50 ledger cells. The acronym is kept for URL stability only. Do not expand it as “Onset Benchmark” in citations.
docs/THREAT_MODEL.md states what the construction can and cannot support. Read it before citing any RHOB number.
The paper splits the class into five sub-classes. For two of them, a pre-registered survey of published benchmarks (paper §5 and App. F) split the answer.
fixed, did not generalise and is withdrawn as a
prevalence claim: pooled 0.54%, CI [0.00%, 1.32%].power difference is for, did: 15 of the 77
enumerated preconditions, in 4 of 15 benchmarks. The other 62 rest on no test at
all.The precondition counts inherit a bias of the enumeration: its search patterns select against equivalence-test vocabulary. The pre-registered remediation for that is an open item in ROADMAP.md. The surveyed benchmarks are a convenience sample, so none of these figures is a rate in the field. This page names none of them. The paper's Appendix F gives per-benchmark totals. The row-level coding will be published only after each group has seen its own rows, and as of 2026-11-15 none has been contacted yet.
As of 2026-11-15, every run of these diagnostics is the author's own: on RHOB, and in the survey, on other groups' published scores. None has been run by a benchmark's own maintainer, on their own benchmark: someone who can contest the result. That is the next milestone. A run that flagged nothing is worth reporting, and so is a verdict you think is wrong.
Issues are public. To report privately, for instance a flagged cell in a benchmark that is not yours to name, email aarav7.shah@gmail.com (how private reports are handled).