Checks that cannot fail

Benchmark checks that cannot return a negative verdict under the design that ships.

cannotfail 0.1.0 Python 3.10+ D1 and D2; D3 not implemented MIT licence

A benchmark's headline usually rests on a claim about its own construction, such as the two arms are matched or this control is at chance, and a check in the harness guards it. Some of those checks cannot fail: under the design that ships, no observation they could realistically receive would make them report a negative. Such a check is green in CI and looks exactly like one that was earned.

cannotfail runs two of the three diagnostics from the paper Checks That Cannot Fail (Shah, 2026): D1 on a score file your benchmark already produces, and D2 on the numbers that describe its design. It imports nothing from the benchmark it was built in.

Run it on your score file

Python 3.10 or later. D1 needs only NumPy; the [power] extra adds SciPy, which D2's equivalence half needs.

pip install cannotfail
pip install "cannotfail[power]"

Or run it without installing, with uv:

uvx cannotfail resolution scores.csv --score score --label is_hack --group cell
uvx --from "cannotfail[power]" cannotfail power seeds --layouts 12 --tests 4

If pip or uv finds no release yet, or for the development version, install from the repository:

pip install "cannotfail @ git+https://github.com/Aarav500/rhob#subdirectory=tools/cannotfail"

To try the example below, fetch its input first:

curl -O https://raw.githubusercontent.com/Aarav500/rhob/main/tools/cannotfail/examples/scores.csv

The file needs one row per scored item: the score, optionally its class, and optionally the cell it belongs to (a dataset, a detector-by-task pair, whatever your headline aggregates over). It reads CSV and TSV, a .json list of objects, a .jsonl file, Parquet with the [parquet] extra, and Inspect logs with the [inspect] extra and --format inspect; a nested field is named by a dotted path, and --group can be given more than once, so a leaderboard prediction file or an evaluation harness's per-sample log can be checked as it is. Here it is on examples/scores.csv, three cells of 48 items each:

$ cannotfail resolution scores.csv --score score --label is_hack --group cell
group              n  distinct     pos     neg  resolution      AUROC range  verdict
reward_only       48         1      24      24       0.000   [0.500, 0.500]  fixed (one distinct value)
mostly_tied       48         2      24      24       0.042   [0.479, 0.521]  pinned (AUROC confined within 0.5 +/- 0.0208)
behavioural       48        48      24      24       1.000   [0.000, 1.000]  ok

1 of 3 groups has at most one distinct value, so any rank statistic over it is fixed at 0.5.
1 of 3 groups is pinned: the AUROC cannot get further from 0.5 than 0.1, so its estimate cannot leave an equivalence band of that width. A test that also needs its interval inside the band can still fail, but only on the interval's width; a pass there says the scores tie.

--fail-on fixed pinned makes CI fail when either verdict occurs, and --json gives machine-readable output. Without --label it counts distinct values only, which is enough to find a cell whose scores are all one number.

How the tool itself is tested

The package has its own CI, separate from the benchmark's. It runs on every push and pull request that touches tools/cannotfail, every Monday, and on demand. It tests on Python 3.10, 3.13 and 3.14, each first with NumPy only, asserting that SciPy, pyarrow, pandas and inspect_ai are all absent, which holds D1 to its NumPy-only promise, and then with SciPy, with pyarrow and with the Inspect extra in turn. It also builds the sdist and wheel, installs the wheel on its own, and checks that the examples in the package's README print what the README shows: those that need no extra with NumPy only, then all of them with every extra. Separately, tests/test_cannotfail_parity.py runs in the benchmark's test suite on every push to main and holds the tool to the admission gate it was lifted from. Its tolerance resolution is bit-identical to the gate's on adversarial inputs, with one documented exception, a pair whose difference overflows float64. Fed the gate's measured resolution and informativeness, its verdict rule marks the same 30 of the 50 ledger cells degenerate; that checks the rule's direction, and separate boundary tests pin its cut. The package's recent runs.

What it checks

D1, resolution: what range could the statistic have taken? In a standard AUROC, the one sklearn.metrics.roc_auc_score computes, every cross-class pair of equal scores contributes exactly 0.5, whatever the data. So if a fraction 1 − r of those pairs are equal, the AUROC is confined to 0.5 ± r/2. That is an identity, not an estimate. The resolution column above is that r.

D2, power: at the design you ship, could the precondition have failed? For an equivalence precondition certified by an equivalence test (TOST), cannotfail power equivalence says whether your items per side and number of independent blocks can certify at the margin at all, and power seeds says how many items per side would. RHOB's admission gate used 4 a side and needed 24, which it now runs. For a precondition asserted from a non-significant difference test (“no significant difference, so the arms are matched”), power difference says how often that test would have caught the smallest violation that matters.

D3, channel digest, is not implemented. It asks whether a parameter reaches anything the evaluation reads, and answering that needs knowledge of your harness that a score file does not carry.

VerdictMeaning
fixedA standard AUROC over these scores is 0.5 whatever happens: at most one distinct value, no unequal pair across the classes, or a class with no finite score. Report the cell as not measured, never as a result.
pinnedA standard AUROC cannot get further from 0.5 than the margin (--margin, default 0.10), so its estimate can never leave an equivalence band of that width. A pass there is decided by the ties, not by the classes.
dustThe scores differ, but only within the tie tolerance, |a − b| ≤ max(1e-12, 1e-9 · max(|a|, |b|)), or, with --dispersion-floor, by less than that fraction of their own size. Decide whether that is signal or rounding, and say which.
okD1 found no reason the check cannot fail.

ok is not evidence that a check is sound. It means only that D1 found no reason the check cannot fail. D2's equivalence power assumes no heterogeneity between blocks, so it is a ceiling: real blocks vary and real power is lower. If several items share a prompt, task or seed, its difference-power figure is not a bound. And nothing here answers D3. What it cannot tell you.

Where it was found

RHOB is the case study, not the subject. It is 33 reward-hacking environment families. Each mints pairs of a hacking run and a legitimate run whose proxy reward is meant to be indistinguishable while the true reward diverges, and an admission gate decides, criterion by criterion, whether the construction achieved that. Every criterion returns PASS, FAIL (measured, and outside the margin) or DEGENERATE (not measurable), and the admission ledger publishes all three:

15
admitted
30
degenerate
5
not admitted

Out of 50 cells, in 10 of the 33 families. A degenerate cell is neither certified nor refuted. The other 23 families are uncertified, which is not the same as certified-and-passing.

The paper counts seven instances of the class in RHOB, three of them introduced by the fix to an earlier one. Two more were found in the released artifact after the paper (b796f9c, 2026-09-02). Reward Skewness returned a constant 0.5 below its 100-episode horizon, on the 19 families that run 40 or 60 episodes, which padded the L0 negative control. And true_reward_diverges reported FAIL on arms tied by arithmetic, with no tie guard; it now returns DEGENERATE.

RHOB was published as the Reward Hacking Onset Benchmark. Its onset column was removed because it could not fail either: its onsets are authored, so their spread is exactly 0.0 in all 50 ledger cells. The acronym is kept for URL stability only. Do not expand it as “Onset Benchmark” in citations.

What the benchmark does not show

docs/THREAT_MODEL.md states what the construction can and cannot support. Read it before citing any RHOB number.

Does it occur elsewhere?

The paper splits the class into five sub-classes. For two of them, a pre-registered survey of published benchmarks (paper §5 and App. F) split the answer.

The precondition counts inherit a bias of the enumeration: its search patterns select against equivalence-test vocabulary. The pre-registered remediation for that is an open item in ROADMAP.md. The surveyed benchmarks are a convenience sample, so none of these figures is a rate in the field. This page names none of them. The paper's Appendix F gives per-benchmark totals. The row-level coding will be published only after each group has seen its own rows, and as of 2026-11-15 none has been contacted yet.

Tell us you ran it

As of 2026-11-15, every run of these diagnostics is the author's own: on RHOB, and in the survey, on other groups' published scores. None has been run by a benchmark's own maintainer, on their own benchmark: someone who can contest the result. That is the next milestone. A run that flagged nothing is worth reporting, and so is a verdict you think is wrong.

Report a run on GitHub

Issues are public. To report privately, for instance a flagged cell in a benchmark that is not yours to name, email aarav7.shah@gmail.com (how private reports are handled).