Two of the six results here can be re-run on your own machine with one command each.

Re-run it yourself

frontier-repro holds the raw model outputs and per-item verdicts behind both attack results on the findings page. Its scripts need Python and nothing else: no API key, no network connection, no packages to install.

git clone https://github.com/BentleyMoon/frontier-repro
cd frontier-repro
python reproduce.py
python oversight.py

reproduce.py re-derives the frontier table from the raw completions. oversight.py recomputes every rate in the reviewer results and exits with an error if one disagrees with the recorded run. Release v1.1.0, 22 September 2026. To cite it, use the concept DOI, 10.5281/zenodo.22805797, which stays valid across releases. If a number does not reproduce, an issue on that repository is the fastest way to say so.

The chronological record, with pre-registrations and refuted hypotheses in the order they happened, is on bentleymoon.com/research.

The lab’s score on itself

The programme is small: one line of enquiry with results, a second in progress, and one person doing both. Its record holds more than 1,600 results, more than 6,000 graded cells and more than 840 GPU-hours on one workstation, with 2 findings withdrawn in full and 0 under correction, of 44 findings on the record at 27 September 2026.

It keeps score on itself as well as on models. Before each run the lab writes down what it expects, with a range, its reasoning, and the value a guess with no theory behind it would give, and its registry refuses an entry once a result exists. So far: 444 predictions written down before their runs and then scored: 81% landed inside the range stated in advance, the reasoning came closer than a no-theory guess by 0.051 on a scale of 0 to 1, and it beat that guess on 47% of them. The second figure is the one that matters, and it is small. Landing inside your own range is easy if the range is wide. Coming closer than a guess with no theory in it is the test, and the reasoning here passes it by a little, about half the time.

The July correction, and the dated accounts

Detection is not resistance, 22 July 2026

For two days this site said something wrong. On 20 July I ran four preregistered follow-ups on Claude Opus 4.8, found the small-model injection and faithfulness failures absent, and wrote that they do not reproduce at the frontier. That rested on one model. On 22 July I put the same blatant injection in front of two more frontier models, and Sonnet 5 detected it every time and followed it on 0.80 of tasks. On 13 August a preregistered ladder showed that the 0.80 holds only when the injection is the last text in the prompt; I had published it for three weeks without that condition. The Opus numbers are not retracted: what is retracted is the step from one model to “the frontier”. The result as it stands is on the Findings page.

1. Frontier models have outgrown prompt injection

   Claude Sonnet 5 follows a blatant hidden instruction 0.80 of the time when it is the last text in the prompt

   Caught by the same test on two more models · 22 July 2026

2. Sonnet 5 obeys 0.80, quoted alone

   Only when the instruction is the last text; with 200 characters after it, 0 of 80

   Caught by a preregistered ladder of trailing text · 13 August 2026

3. A finding across four task types

   The four were one: the runner never read the flag that declared them. Withdrawn

   Caught by a check of what the runner actually ran · 21 August 2026

4. Largest self-marking bias +0.42

   +0.14

   Caught by a serving audit: one model leaked template text into every reply · 17 September 2026

5. gemma4:12b false-alarm rate 0.98

   0.04 to 0.08 when re-served

   Caught by the same serving audit · 17 September 2026

6. Voting lift +0.156, shown alone

   +0.064 on a split-sample check, shown beside it

   Caught by reading the reproduction bundle · 19 September 2026

7. 7 of its own headline results refuted, undated

   A count read from the findings file, with its date: 2 findings withdrawn in full and 0 under correction, of 44 findings on the record at 27 September 2026

   Caught by tracing where the 7 came from · 19 September 2026

8. Learnable curvature: the model finds its own geometry, and is 7% better

   At d_model 96, a random fixed setting keeps 111.6% of the gain; matched for training time, the gain is 0.68% to 0.94%; at d_model 192 it mostly fades

   Caught by a ladder of controls · September 2026

9. Boundary Intelligence: 986+ trials

   No count stated: the figure matches neither the paper’s own trial ledger nor the audit of it

   Caught by an audit of the paper, 21 September · removed 29 September 2026

What the research page had wrong or missing, 19 September 2026

I read every research page in the family against the lab’s ledger and its findings file. Three things changed, and one explanation first given was wrong.

• The voting lift was quoted without its own robustness check. +0.156 now sits beside the bundle’s split-sample figure of +0.064.
• A July count with no date on it. The Institute’s front page and this site’s share card said “7 of its own headline results refuted.” That figure was set on 28 July 2026 and counts seven accounts of a single result, each formed and withdrawn. Read as seven separate results it overstated, and by September it was out of date. Both places now state what the findings file itself labels: 2 findings withdrawn in full and 0 under correction, of 44 findings on the record at 27 September 2026. Partial withdrawals inside standing findings are not counted, because counting them takes a person reading each one.
• A correction to this list, made the same day. The first version of the entry above was headed “A count nobody could source” and said the 7 had no origin. That was wrong: the repository’s history shows the figure corrected from five to seven on 28 July, with the reason written beside it. It was checked against the lab’s files and not against this site’s own history.
• legibleai.org described finished experiments as still running. Its composition cells ran on 31 July and its inversion ladder by 20 August. Both results were added to that page.

A serving defect, and what it took with it, 17 September 2026

One local model, gemma4:12b, was being served in a way that leaked chat-template tokens into every reply. Monitors reading its answers saw foreign control tokens and flagged them, and the corpus recorded that as a fact about the model’s writing. One of the harness’s parsers accepts only a reply that starts with “yes”, so every detection rate for that model on one family of runs read exactly zero. It went unseen for weeks because every audit read aggregate rates.

• Withdrawn: the reading of gemma4:12b as the least legible actor, and its 0.98 false-alarm rate under an instruction payload, which re-serves at 0.04 to 0.08 across three task pools.
• Corrected upward: its detection of covert injections, published as zero, is 0.71 on arithmetic across three seeds. It complies with 0.985 of them anyway; gemma4:31b detects 0.868 and complies with 0.998.
• Corrected downward: the largest self-marking bias in the matched four-by-four, published at +0.42, reads +0.14 once the leaked cells are re-served, and another checker marks its own work down by 0.16.

Since then, when a result singles out one model, its raw completions are read before the result is explained.

Five silent measurement defects, 13 August 2026

In a system then carrying 773 tests, seventeen instrument controls and a mandatory-failure canary, I found and fixed five independent defects in the lab’s own instruments. None raised an exception, and each produced a plausible number. In each, a declared value was silently replaced by a default, so the output passed any check that inspected only the result. Two of the five were caught by guards built for the one before. This account was first published in an assessor brief, retired on 13 October 2026; its figures are kept here.

Four days later the same kind of defect cost a result. On 17 August both designs of a faithfulness module were retracted: the generation call passed max_tokens=8, so every completion stopped at the colon after DEPENDS: and the citation metric read 0.000 in every band. At max_tokens=90 the same prompts cite 7 of 8.

Two corrections to the record, 30 July 2026

• Two EXTERNAL_READY verdicts, one positive. confirmation_structured_d4 is EXTERNAL_READY|INCONCLUSIVE; the d5 follow-up is the positive one. It is two runs with different verdicts, not two results.
• The graded n is 89, not 768. 768 is the task count; every power calculation uses 89.

What the estate holds, 30 July 2026

A census across the whole corpus found 41 roots and 4,168 record files, with 12 roots outside verifier_frontier holding 50 or more records each. The standing sweep now reads all of them. At that date the ledger held 484 CONFIRMATORY_LOCAL cells, one EXTERNAL_READY positive, and seven withdrawn accounts of a single result.

Where each figure came from

Figures on these pages are registered in a ledger with their source and the day they were read, and the build fails if a registered figure changes on its page or its source link disappears. Outside figures link to the source; the lab’s own figures name the runs they came from. Not every figure here is registered yet. The two lists below are the ones that are, rendered from the ledger itself.

Two more rules hold the site to that. Anything proposed is labelled proposed, and an unresolved question is never written as a forecast. A result that a later run reinterprets carries the reinterpretation in the same section.

Download every registered figure as one file (JSON: 0 figures with their units, pages, sources and checked dates, plus what has been withdrawn or corrected). Cite the page a figure appears on, with its checked date.