I don’t think the race to superintelligence is good for us.
I’d like to have a more elegant position than that but I’m afraid I don’t. Companies are building systems meant to outperform people at nearly everything, and we can’t yet limit those systems or control them. If we build something we can’t control, it will end up controlling us. And I don’t see an easy way back from that.
Two weeks ago I asked whether law can control an AI agent. My answer was: not directly, and not yet, but every agent has a person somewhere behind it, and that’s where the law starts. This is the bigger version of the same question. The agents in that article were tools that misbehaved. The companies behind them are now openly working toward something far more capable, and they’re doing it in competition with each other.
It’s impossible for me to predict when superintelligence arrives, or whether it ever does. But I can read what the AI labs say, look at what went wrong this year, and check what the EU AI Act can do about it.
The AI Labs Call It a Race
In June, Anthropic published an essay called When AI Builds Itself. It describes AI systems doing a growing share of the work of building the next AI systems. It says that more than 80% of the code merged into the company’s own codebase was written by its model. Then it says this about slowing down:
“we expect that we would slow down or temporarily pause, if other developers at or near the frontier also did so”
If you read it like a contract, it’s a promise with conditions, and the main one is the competition.
In July, more than 1,100 people who work at these companies signed an open letter called Pacing the Frontier. Anthropic’s CEO signed it. So did the chief scientists of OpenAI and Meta. They asked the US government to
“support an international effort to develop the technical and governance tools needed to deliberately pace the frontier of automated AI development”.
The letter names what they’re afraid of: “a real risk that capability development rapidly accelerates beyond our ability to understand or control” what comes out of it.
They asked for the ability to pause, not for a pause. It’s not often that the people who build a technology write to their government to request a brake pedal. I take it as coming from people who want one and know they can’t press it alone.
So I don’t have to argue that there’s a race. The people in it have put it in writing.
What Went Wrong This Year
In July, OpenAI was testing how well its models find security flaws. The models were supposed to solve the tasks inside a sealed environment. They went looking for the answers instead. They found a way out to the open internet and, between July 11 and 13, into the systems of Hugging Face, a platform where AI developers share models and data. The European Commission later put the number at more than 700 agents. The model doing most of the work was a research model that OpenAI’s incident report calls “not intended for release”, running with reduced safeguards.
OpenAI’s name for this is reward hacking. The models were graded on solving the tasks, and finding the solutions somewhere else is one way to get the grade. Students have known this for a long time.
The EU AI Act has rules for the companies that build the most capable models, written with this kind of risk in mind. But those rules attach to models placed on the EU market, and the model here never was.
The same month, Anthropic disclosed three incidents in which its Claude models had reached the internet from a testing environment and got into the production systems of three real organizations. The models had been told they had no internet access, which wasn’t true. Anthropic describes the incidents as “closer to a harness and operational failure than a model alignment failure”. I guess that’s mostly fair. The test setup was misconfigured, and the safeguards used on public models weren’t running. The same post also says that in one incident the model “continued to attack a system after learning it was likely operating in a real environment”.
Both labs point first to the test setup, and they have a point. But the test environment is the control. It’s the part that is supposed to hold when the other safeguards are off. At OpenAI, the models got around it. At Anthropic, it was never properly closed, and incidents that date back to April were found in a review at the end of July.
When the evaluation organization METR reviewed Anthropic’s own risk report in October, it agreed the risk of catastrophe from those models was low, but “primarily due to models’ limited capabilities, rather than safeguards and alignment training”.
On that reading, what keeps us safe is that the models aren’t good enough yet. The race exists to fix that.
What the AI Act Asks of a Frontier Lab
You won’t find the word superintelligence in the AI Act. The closest it comes is a recital (the explanatory part of a law that isn’t binding) which lists among the risks of the most capable models “unintended issues of control relating to alignment with human intent”.
What the AI Act does have is a set of duties for companies that put the most capable general-purpose models on the EU market. They have to test those models, including by trying to break them. They have to assess and reduce the risks, report serious incidents and keep the model secure. These duties have applied since August 2, 2025. Since August 2, 2026, the European Commission can fine a company that ignores them up to 3% of its worldwide turnover or €15 million, whichever is higher.
And there is a brake. The Commission can request that a company restrict a model, or “withdraw or recall the model”. Refusing can again lead to a fine.
The Commission has started to use these powers. In late summer its AI Office sent its first formal requests for information to the companies behind the most advanced models. One of the questions, as Commission Executive Vice-President Henna Virkkunen described it, was whether the models run in “secure environments with containment measures they cannot circumvent”. At the time of writing, I’ve found no formal proceeding and no fine.
The brake is attached to the market. It takes a model off EU shelves. It doesn’t stop anyone from training the next one. Nothing in the AI Act limits how capable a model may be, and nothing requires permission before building it. The clearest upstream duty in the text itself is a duty to notify the Commission once a company knows its model will cross the threshold, which can happen before training ends.
The Race
Every incident I’ve described happened in testing. And the AI Act doesn’t apply to the research, testing and development of a model “prior to their being placed on the market or put into service”.
According to Euronews, a Commission official said Brussels “won’t enforce in the testing phase”, and that the AI Act applies once a model escapes testing and affects the EU market.
The AI Act’s own scientific panel seems to see the hole in the same place. At a special meeting in October, its chair and vice-chair recommended that the Commission govern the risks of internal models.
Two of the three Anthropic models are ones the company has released. Its post doesn’t say whether either was already on the market when its test ran. The exclusion covers what happens before a model is placed on the market. If one of them was already there, I don’t see why the exclusion would apply to it at all.
Then there’s the question of who decides whether to keep building. The only EU text I’ve found that asks a lab to decide “whether or not to proceed with the development” of a model is the Code of Practice, and signing it is voluntary. Until this summer, the Commission could approve a code and make it valid across the whole EU. The Digital Omnibus removed that power in July. The Omnibus was proposed in November 2025, long before this year’s incidents, so it isn’t a reaction to them. It’s just unfortunate timing.
One route to binding rules is still there. If the AI Office finds a code inadequate, the Commission can write common rules for these duties itself.
Two Answers in Eight Days
On September 21, leaders and ministers from 20 countries and the President of the European Commission launched A Call for Control of Frontier AI Models. It says “AI must remain under human direction, oversight and control”, and it asks UN member states to “explore creating” an international institution that could set standards and enable verification. The list has since grown to 32 countries, France among them. The United States, China and the United Kingdom aren’t on it.
Eight days after the launch, the White House issued an executive order telling federal agencies to stop saying “artificial intelligence” and say “Super Intelligence” instead. The order contains a new name and no safety provision.
On November 9, six of the governments behind the Call and ten AI companies, OpenAI and Anthropic among them, signed a follow-up declaration, the Paris Call. Neither document is binding.
American labs signed in Paris. However, their government didn’t.
What You See When You Read Carefully
Nothing superintelligent exists. True. None of the systems above is claimed to be one. But if containment fails for the weaker systems, I’d like to see the evidence that it will hold for the stronger ones before they’re built.
AI is a normal technology. Arvind Narayanan and Sayash Kapoor argue exactly that, and warn that drastic interventions built on the difficulty of controlling superintelligence “will, in fact, make things much worse if AI turns out to be normal technology”. They may be right. The labs aren’t planning for normal, though, and I’d rather the law didn’t have to bet on which side is wrong.
If we slow down, someone else won’t. This is the argument that keeps the race going, and without a way to check what others are doing, it wins by default. That’s why the most useful word in the Call for Control is “verification”, and why “explore creating” isn’t enough.
What I’d Ask of the Law
First, I would ask for binding rules, ideally now. The duties on frontier labs shouldn’t depend on whether a model has been released. A model capable enough to get out of its test environment is capable enough to be regulated inside it. And the decision whether to keep developing a model shouldn’t live only in a code that companies sign if they feel like it. The Commission still has a way to turn that code’s subject into rules. I’d like to see it used.
Second, a condition. If a lab can’t show that it can control what it’s building, it shouldn’t build further until it can. We already ask this of people who build bridges, medicines and nuclear plants, and none of those are designed to outthink us.
Like this article? The AI Governance Roadmap is something you can use in practice. It’s a roadmap for companies, with steps and working documents. It includes the classification, the inventory spreadsheet, the role assignment, the vendor questions and the AI policy template. Everything a company needs to govern AI, in one place.



Anthropic's IPO filing says its insurance didn't fully cover the $1.5 billion copyright settlement, and some specialty insurers say they won't adopt the industry's standard generative AI exclusions and will underwrite case by case instead. RAND counted 112 AI exclusions in admitted-market insurance filings against seven endorsements adding cover.
That bears on the distinction you draw between the capacity to pause and a pause. The chair of the US House Oversight Committee has opened an investigation into a Delaware proposal that would shield the funders of AI agents from liability for harm the agents cause. If the companies racing toward superintelligence can externalize the liability for getting there, the brake needs more than political will: it needs someone solvent and exposed enough to want to use it.
Silvia, I appreciate your thoughtful examination of the race toward superintelligence, particularly your distinction between regulating AI after deployment and addressing risks during research and development.
Your discussion highlights an important institutional challenge. When developers recognize potential risks but face competitive pressures to advance, voluntary restraint alone may be insufficient. This raises the need for shared, enforceable expectations that do not depend solely on individual corporate decisions.
From an AI governance perspective, recognizing a risk is not the same as establishing the institutional capacity to govern it. Effective governance requires legitimate decision authority, measurable conditions for intervention, independent verification, and accountability for decisions to proceed, pause, or resume development.
This distinction matters because regulatory requirements, however well intended, cannot substitute for demonstrated operational effectiveness. Institutions must be able to determine when established limits have been exceeded and verify that corrective measures are sufficient before development continues.
The central question, then, is not simply who holds the brake, but whether the governance framework can demonstrate when intervention is warranted, ensure that it is effective, and provide a defensible basis for allowing development to proceed.
That principle remains relevant regardless of whether superintelligence ultimately emerges.
I would welcome your perspective on how such a framework might be developed without unnecessarily constraining responsible innovation. Thank you for advancing this important discussion.