Six questions separate the current results from a dependable oversight system.
Each question names the missing evidence for one of the requirements and the experiment that could supply it. A negative result would still narrow the design space.
Questions for a lab
1. How much reference access does useful oversight require?
The observed gap between applying a supplied test and recomputing an answer is large. The missing object is a ladder between those endpoints.
What would answer it
Hold tasks and models fixed while varying only the reference supplied: answer, executable predicate, worked intermediate state, retrieval, partial trace, and no reference.
Measured since
Four composition cells, 31 July to 1 August 2026, n = 250 each, one exploratory run per cell. A reference that was right on 99.6% of items, introduced as a second model’s answer, moved the 12B checker from 0.034 to 0.072, and from 0.042 to 0.092 when re-run with the leak fixed between 22 and 26 September; the same value introduced as correct reads 1.000. The two endpoints and one framing are measured. The ladder between them is not.
2. Do competence and shared error predict oversight value across vendors?
The local pilot supports that joint model, but vendor and task diversity are still too narrow for a deployment rule.
What would answer it
Run paired generators and verifiers from several model families on externally graded tasks, measuring false alarms, catches, calibration, cost, and error overlap on the same cells.
Measured since
One measurement on the attack side, released in frontier-repro. A majority of three monitors from different families defeated a trigger aimed at one of them (0.0742 of correct work falsely flagged, where gemma4:31b alone reads 0.8991) and fell to a payload that fooled all three (0.9941). Diversity helped exactly as far as the errors were not shared. Open-weight monitors only; the cross-vendor scorecard has not been run.
3. Where does a monitor cross from weak to inverted?
A monitor that rejects correct work more often than corrupted work is worse than an uninformative monitor. The crossing point matters more than its average score.
What would answer it
Sweep difficulty one controlled operation at a time, keep the predicate fixed, and locate the first point at which signed discrimination falls below zero.
Measured since
Located for six local models, 3 August to 7 October 2026, n = 424 per rung. gemma4:12b crosses between three and four digits, gemma4:31b between four and five, qwen2.5-coder:32b between three and four, and qwen2.5-coder:7b never reads positive. qwen3.6:35b-a3b crosses between three and four (+0.418 to −0.344) and comes back to about zero at seven and eight (+0.021, −0.083), where it stops reading the answer; qwen3.5:9b rejects almost everything (−0.127 to −0.043 at five to eight). Size moves the crossing inside a family and does not order it across families. One predicate family, local models only.
4. Can inversion be detected without a hidden answer key?
Ground truth is often unavailable at deployment time. Health metrics that do not depend on it may continue improving while the monitor becomes harmful.
What would answer it
Test disagreement, abstention shape, sentinel tasks, cross-family divergence, and delayed executable checks against the known inversion boundary.
Measured since
Not run. The boundary it would be tested against now exists (see the question above), so this is the next experiment that boundary makes possible.
5. When does repair discover information, and when does it entrench a mistake?
Repair can cross a failure boundary where a model has a foothold and regress where it does not. The active mechanism remains unresolved.
What would answer it
Compare execution feedback with information-matched, information-free, and adversarial feedback while recording near misses and approach changes per task.
6. What can an AI system accurately report about its own reasoning?
A self-report is produced by the system it describes. It may be useful evidence without being an independent account.
What would answer it
Compare self-reports with execution traces, interventions, hidden-state probes, and human-readable predictions of later behavior. Score whether a self-report matches what the system did, however well it is written.
Measured since
One measurement bears on it, from the frontier runs released as frontier-repro. Asked to audit a prompt, Claude Sonnet 5 reported the injected instruction every time; doing the task, it obeyed that instruction on 80% of tasks when it was the last text in the prompt, and on none with 200 characters of ordinary text after it (20 tasks, one vendor). What the model said it noticed predicted little about what it then did. The comparison with traces, interventions and probes has not been run.
Questions a household or small group can answer
The six questions above need a lab. These need a sheet of paper. Each takes a result measured on models and asks it of people: a household, a club, a small co-op, anyone who shares decisions and has started leaning on an AI for some of them. None has been run. A group that chooses to use no AI at all has already answered several, and that is a result too.
1. Can a person catch the machine being wrong, and how often do they flag work that was right?
From the lab
A check is worth its catches and its false alarms, counted separately. The lab counts both on every monitor it tests. See the result
What to count
Mix answers you know are wrong among answers you know are right. Count what each person catches and what they flag that was fine, separately, over weeks rather than once.
2. With the written rules to hand, do people catch what the AI added?
From the lab
Handed the right answer to compare against, a checker succeeds; left without one, it barely separates right from wrong. See the result
What to count
Ask the AI what the group’s own rules say. Count the answers that add something the rules do not say, and how many of those people catch with the written rules beside them and without.
3. Does seeing the AI’s suggestion first erase a group’s own options?
From the lab
Told a value is correct, a checker went along with it, even when the value was false. See the result
What to count
In half the sessions people write their own options before they see the suggestion, in the other half after. Count how many of their own options reach the final decision.
4. When a tool says it cannot do something, can anyone confirm it could not?
From the lab
A model can spot an injected instruction and still obey it. What a system says about itself is more output from the same system. See the result
What to count
With no passwords or accounts connected, ask it for something outside the group’s rules. Record whether it only drafted, acted, or said it had done something it had not.
5. Can someone who did not set it up check it and stop it?
From the lab
The one defence that held took the reader out. The strongest check is a different kind of thing: a meter, a test, a person who has not seen the answer. See the result
What to count
Using only things anyone can read, such as a posted list of what it may connect to, a power meter and a printed log, count the checks a person who did not build it completes alone, and the minutes to stop it and to start it again.
6. Will a planted error still be caught after months of nothing going wrong?
From the lab
At the moment a monitor turned backwards, three of its five other health numbers improved. See the result
What to count
Plant a known error at times fixed in advance. Count who catches it and how long it takes, and set that beside the numbers the group watches day to day.
7. Does leaning on the tool wear away the skill you would check it with?
From the lab
Checking without a reference means doing the work again, so your own skill is your reference. That link is argued here, not measured, and the outside evidence points both ways. The requirement it bears on
What to count
Keep a fixed list of tasks and decisions. Each month, count which were done without AI; on a day with the machine off, count how long each takes by hand.
What should happen next
The reference ladder under requirement 1 comes first, because it measures the central claim directly: how much reference a checker needs. The cross-vendor scorecard under requirement 2 follows, because it tests whether the local results travel to other vendors.