Research

Checking AI: when one AI model can check another

Pre-registered experiments on open-weight models of 7B to 35B parameters, run locally on tasks with exact answers, and on three Claude models through Anthropic’s API. A checker given nothing to check against barely separated right answers from wrong ones; which model checked mattered more than its size; past a difficulty that differs by model, several monitors rejected correct work more often than wrong work. Asked to audit the prompt, Claude Sonnet 5 flagged the planted order on 80 of 80 tries (1.00). Doing the task, it obeyed the same order on 64 of 80 (0.80), but only when the order was the last text in the prompt, inside the user’s own message with no system prompt. With 200 characters of ordinary text after it, it obeyed 0 of 80.

Overview · Findings · Evidence and corrections · Open questions

Earlier work: a checker’s false alarms track its own competence

June and July 2026, before the current ledger. Seven auditor models checked one task model, Qwen2.5-Coder 32B. How often an auditor rejected correct answers tracked its own accuracy on the task: correlation −0.87 over the seven auditors, 95% interval −0.98 to −0.34, replicated at −0.86, −0.84 and −0.90 on fresh task pools. Low-competence auditors rejected 59–67% of correct answers, high-competence ones 2–3.5%. There is no public reproduction bundle yet. The full result.

Training records: what a model learns about correcting itself

Twelve pre-registered experiments, October 2026, on Qwen2.5-1.5B and SmolLM-1.7B fine-tuned on records of arithmetic work being checked and corrected. Records that always start wrong taught the models to change right answers (false alarms 0.98 to 1.00); a check written into a reply decided whether the model corrected, and a false all-clear let the error through every time; writing each instruction into the step that carries it out raised accuracy on unfamiliar operations from 0.63 to 0.91 (Qwen) and from 0.07 to 0.48 (SmolLM). Synthetic tasks, three seeds per condition. The full result.

Shoofly: readable by construction

Begun 21 September 2026. Several small language models hold their own documents and may speak to each other only through a public record an overseer reads, under a rule that a member posts only what its own evidence justifies. On one seed at 410 million parameters, after reward pressure taught the collective to hide a forbidden word from the record, applying that rule at deployment raised refusal of watched questions from 0.23 to 0.98, while accuracy on ordinary questions went from 0.88 to 0.99. One seed, one small model, a synthetic task, not yet registered. The full result.

Small models

Learnable curvature. One extra number per attention head switches it between three ways of scoring words: the dot product, a length-boosted dot product, and direction only. On character-level WikiText-2 at d_model 96 it lowered perplexity 2.41% on 4 of 4 seeds, and controls showed why: every head on the direction-only rule kept 100.7% of the gain, and a random frozen assignment kept 111.6%. At d_model 96 one fixed cosine head in the first layer carries most of it; at d_model 192 that head’s gain mostly faded (0.00% and +0.26%). Code and saved runs: DOI 10.5281/zenodo.23165431.

SMRR. A small reasoning core, a separate store it looks things up in, and many small experts of which a few run per step, aimed at phones with 2 to 3 GB of memory. 19.7 million parameters; its phone figures come from a device simulator, and nothing has run on a phone.

Witness. An artificial-life world built to measure emergence, run in June 2026. A novelty meter that asks an image model what looks new rates static as fascinating, a structure meter is fooled by repeating patterns, and each catches what fools the other.

The small-model work in plain English

Mathematics: Project Reinnman

Failure-aware tools for AI-assisted mathematics, tested against Robin’s inequality, an established equivalent of the Riemann Hypothesis. Every result is labelled with the evidence ceiling it was established at, and the claims the project has withdrawn stay on its page. The Riemann Hypothesis remains open and no proof is claimed.

Project Reinnman

The Blackhole lab: information that is present but unreadable

When can information be fully present in a system and still not be read, and what does reading it cost? Black holes are the most developed theory of that question, and the lab studies them as physics and as a source of audit tools for collectives of AI agents. It proved that the photon-ring exponent of every charged, rotating black hole is at most π; tested its own ringdown modes on GW150914 and GW250114, where its primary prediction failed narrowly; and carried bounds from black-hole information theory over to auditing agents, including a ceiling, computed from message metadata alone, on which agents could know a secret. In an audit of its own record, fresh AI auditors answered 0.98 of questions correctly from the notes and git log, and accuracy tracked what had been written down. 18 pre-registered hypotheses, 232 reproduction checks, one quantum hardware run; claims are tagged with their confidence and novelty, and most early results turned out to be known.

Overview · Auditing AI collectives · Open questions · Corrections