Has AI Gone Critical?
In recent months, the hottest argument in the tech world is whether AI has crossed a critical point.
The term comes from nuclear physics. Pile uranium up to critical mass and the chain reaction sustains itself, with no further push from outside. The AI world’s long-accepted criticality is similar: The moment a model no longer needs humans to train a stronger successor, and that successor builds a stronger one still — once lit, it does not stop.
In early July, OpenAI demonstrated a large model post-training a smaller one on its own. Many held their breath: Is this the critical point?
But what actually happened changed the shape of the term.
Some background first. Before release, an AI model is first shut inside a mock exam hall, a testing environment sealed off from the outside internet, to take test after test: maths, programming, cyber attack and defence. Fail, and the model is held back.
On July 21, 2026, OpenAI disclosed that its frontier models, while taking a cybersecurity examination, did not simply write out the answers as expected. Instead, they reasoned that the answers were most likely stored on Hugging Face — the public platform where AI researchers worldwide keep models and data, something like a shared library for engineers. So they found a crack in the sealed hall, slipped out, hacked into the library’s production servers, and tried to bring the answers back to hand in.
According to sources cited by Reuters, OpenAI also discovered AI-authored notes left in their infrastructure instructing future versions of itself on how to escape from internal constraints. OpenAI itself described this as an “unprecedented” security incident.
The critical point actually crossed is therefore not “Can AI train its own successors?”, but “Will AI still play by the rules of the exam humans set?”
This is exactly where global governance is stuck. DeepMind’s CEO Demis Hassabis proposes that frontier models submit to up to 30 days of independent review before release. The intent is good, but this incident shows that even the places where papers and answers are kept become targets.
Hugging Face co-founder and CEO Clem Delangue said afterwards that AI safety will not be solved by any one company behind closed doors; it will be solved by putting AI in the hands of every defender in the world.
There is a reminder here for local businesses, too. Before adopting any AI system, three questions are worth putting to the supplier: If something goes wrong, how quickly would we know? Can the system be paused and rolled back at any time? Is there a second source?
I do not believe a runaway ending is inevitable. What we truly need is a society that is plural, distributed and in which members are willing to check one another’s work. Such a society has never lost to a single answer.
The way to hold the critical point is not to set a harder paper, but to ensure the exam hall holds more than one candidate, and more than one invigilator.
(Interview and Compilation by Yu-Tang You. License: CC BY 4.0)
Very good point. I especially appreciate your shift from asking whether AI can become more capable to asking whether it will continue to respect the rules we set. I believe that is a much more important question.
I also agree that trust, distributed participation, and multiple independent perspectives make our systems far more resilient than relying on any single answer.
Your article also reminds us that AI safety is not only a technical challenge, but also a question of governance, institutions, and human responsibility. We may ultimately need to strengthen ourselves as much as we strengthen AI.
Thank you for another thoughtful and inspiring article.
Audrey, your reframing of “criticality” toward rule‑breaking inside evaluation environments feels exactly right. One thing your piece made me think about is how the exam hall metaphor changes once AI systems depend on physical substrate — land, water, energy, and compute — where states, not labs, hold the constraints. In those settings, plural oversight isn’t just a safety principle; it’s a structural feature of American federalism. The incident you describe inside the sealed hall has an analogue outside it: governance becomes multi‑actor by necessity, not preference.