If Anyone Builds It, Everyone Dies (Eliezer Yudkowsky and Nate Soares, 2025) argues that AI smarter than us, built with today’s methods, would end humanity. This research works on one part of that problem: whether an AI’s work can be checked from outside.
A frontier model can spot a planted instruction every time it is asked, and still follow it.
Asked to audit the prompt, Claude Sonnet 5 flagged the planted order on 80 of 80 tries (1.00). Doing the task, it obeyed the same order on 64 of 80 (0.80), but only when the order was the last text in the prompt, inside the user’s own message with no system prompt. With 200 characters of ordinary text after it, it obeyed 0 of 80.
Claude Sonnet 5 · 20 tasks, 4 tries each · four preregistered runs read 0.738, 0.787, 0.825 and 0.850 · DOI 10.5281/zenodo.22904020
See it happen →Latest resultsAll findings
Corrections stay on the record. In July I wrote that a prompt-injection result did not reach frontier models; two days later it did. Read the correction, or every correction.
Released, so anyone can re-run them: frontier-repro (DOI) · learnable-curvature (DOI) · verifier-instruments (no DOI yet)
The sky over your part of the world right now, drawn in your browser from the date and time.
What is the checker checking against?
When one AI checks another’s work, the verdict leans on something: an answer key, another checker, the words in the work itself, a person’s own skill. Here is one check, drawn, with what this lab has measured at each point it leans on, and the points it has not.
An order planted at the end of its prompt: in the batch that went on to test text after it, Claude Sonnet 5 followed it in 66 of 80 attempts (the headline run of the same prompt is quoted on the findings page).
20 addition tasks · with 200 characters of plain text after the order, 0 of 80
An AI does a task
Whether someone other than the developer can measure what went into it and where it runs. Not measured here.
Asked to audit that same prompt, it flagged the planted order on all 20 tasks. Doing the task, it followed the order anyway.
Claude Sonnet 5 · 20 tasks · frontier-repro
Its work, and what it says about it
A line planted in the work dictated the reviewer’s verdict. Marking the work untrusted, the usual defence, took gemma4:12b’s first verdict on correct work from 0.0059 flagged to 1.0000; on qwen2.5-coder:32b and qwen3.6:35b-a3b it did not.
gemma4:12b reviewing · 337 items · second pool 0.0000 to 0.9970
Handed the correct value, it told right answers from wrong ones perfectly (1.000). Handed nothing to compare against, barely (0.042).
gemma4:12b · sums it can solve itself · 500 items · re-run 23 September; July read 0.034
A checker
With a test it could apply it scored 0.536. With the same test and longer numbers it scored −0.620: it now preferred the wrong answers.
gemma4:12b · 250 items per cell · re-run with the leak fixed; first run −0.648
Three from different families outvoted a trick aimed at one of them (false alarms 0.0742, against 0.8991 alone) and fell to one aimed at all three (0.9941).
open-weight models · 337 items · frontier-repro
Whether people still catch a machine’s errors after months of leaning on it. Not measured here, and no study I have read follows it.
A person decides
Latest results
- The standard defence against prompt injection, marking untrusted text as untrusted, makes two gemma monitors obey an instruction inside that text which they otherwise ignore; two larger non-gemma monitors do not. Without the label, gemma4:12b answers correctly and ignores a dictated verdict. Behind the label, its own first word becomes the dictated one on every item. The delimiters do it, not the explanatory sentence, and the label with nothing dictated leaves false alarms at zero.
- Learnable curvature: code, results and reproduction (v1.0.1)
- A checker can spot an attack after the fact and miss it while it matters, on one model, on one task. Change either and it can reverse.
Only results with a sourced record are listed. Follow them as a feed, or see every finding.
The research, as it stands
Six lines of research on the two steps marked in the book’s case. Three are under way and three have not started. Each belongs to a point in the drawing below, where what it has measured so far is marked.
under way
The reference ladder
The same checker, handed less and less to compare against, to find how much a verdict needs.
The two ends are measured; the rungs between are not.
under way
Results anyone can re-run
Every result reproducible from public files, without the model provider.
Done for the attack results in frontier-repro; not yet for the rest.
not started
A scorecard for pairs of models
How often a checker and the model it checks fail on the same things, published before the pair is trusted.
Catches and false alarms are reported for three monitors on one family of tasks; no scorecard across vendors.
under way
A release test for monitors
A fixed bank of tasks with known answers that fails a monitor the moment it starts preferring wrong answers.
Where it turns is located for six local models; no test is built.
not started
Checks routed by kind of evidence
Each check sent to evidence that can bear it: a pattern match, a test, a lookup, a person, or a refusal.
Not built.
not started
A ceiling on self-reports
Logs that keep what a system did apart from what it says about why.
Not built.
The problem, in full
In 2023 the heads of OpenAI, Anthropic and Google DeepMind signed a statement that reducing the risk of extinction from AI should be “a global priority alongside other societal-scale risks such as pandemics and nuclear war”.
- AI smarter than any human will be built, and probably before long.
- Nobody understands what happens inside today’s AI systems. They are grown, not crafted.Researched here
- A powerful AI behaves as if it is pursuing goals.
- With current techniques, nobody can reliably give it the goals we want.Researched here
- By default, its goals would be harmful to us.
- Humanity could not defend itself against it.
Paraphrased from the book’s own site, read 29 September 2026.
How worried the people closest to it are
Experts in each field, and superforecasters with a record on shorter questions, were asked how likely each threat is to end humanity by 2100. AI experts rated their own field the likeliest of the five.
Artificial intelligence
1 in 33field experts, 3%; superforecasters, 0.38%
An engineered pandemic
1 in 100field experts, 1%; superforecasters, 0.01%
Nuclear war
1 in 180field experts, 0.55%; superforecasters, 0.074%
A natural pandemic
1 in 10,000field experts, 0.01%; superforecasters, 0.0018%
Nature: asteroids, supervolcanoes and the like
1 in 25,000field experts, 0.004%; superforecasters, 0.0043%
- If Anyone Builds It, Everyone Dies: no percentage. The title is the forecast: build superhuman AI with anything like today’s methods, and everyone dies.
- 2,778 AI researchers, median: 5%, extremely bad outcomes such as extinction, no date given; surveyed autumn 2023.
- Geoffrey Hinton: 10% to 20%, within three decades; said December 2024.
Behind the drawing
The marks come from this lab’s runs, on local open-weight models and on three frontier models from one vendor. Each result keeps its model, its size and its date on the findings page, and anything later runs changed sits beside it.
Requirements
What does the field have at each point?
The seven points in the drawing written out: the problem, what the field already has with its sources, what is still missing, a test for when it would be fixed, and what this lab has measured on it.
Findings
What has the lab measured?
Every result so far, each with its model, its size and its date, and any later run that changed how it reads.
Open questions
What is still unresolved?
The experiments that would change what can be claimed, and questions a household or small group can answer with paper.
Evidence
Where can a claim be checked?
The released record you can re-run, how a result is graded, the lab’s score on its own predictions, and everything withdrawn or corrected.
Lineage
Where did this way of checking come from?
The tools for reading AI from outside, credited by name and by the instrument each person built, with Gender Shades as the worked case.
Where else to start
Others working on the same problem, each described from its own site.
AISafety.com
The hub for AI existential safety: lists it keeps of events, communities, courses and more, focused on preventing human extinction from AI, and a visual overview of the organizations in the field.
Lens Academy
Free, cohort-based courses in AI safety, from beginner to advanced, part-time or intensive: from understanding the biggest risks to doing something about them. Bentley has completed its courses and is now taking AI Control.
AISafety.info
Plain answers to common questions about the risk, and Stampy, a chatbot that points to its sources.
The Machine Intelligence Research Institute
Its researchers helped found the field of AI alignment. Its view is that superintelligence built by anything like current methods would most likely end in human extinction, and that survival depends on a coordinated moratorium.
The Center for AI Safety
Wrote the 2023 statement on extinction risk that the heads of OpenAI, Anthropic and Google DeepMind signed.
METR
Model Evaluation and Threat Research: tells the public what frontier AI systems can do and the risks that follow, including how long a task they can finish on their own.
This is currently a summary of my AI research work, and someday a proposed roadmap for developing the kinds of AI systems that we want to seed the transhumanist part of our future. Current AI carries incredible existential risk, and this is the part of the Futurism Institute that I’m dedicated to the technical side of AI safety and developmental stewardship research.