AI bosses are strategists, and they are right to be scared

Yesterday (September 14, 2026), I was on Arte to unpack the recent announcement by Amodei (Anthropic), Altman (OpenAI) and Musk (Grok) that AI should slow down because we are on the verge of losing control.

The 28 minutes format, with three guests, is very short and leaves no room to lay out a properly built argument, so here are a few additional thoughts.

But first, the replay is right here (in French):

“And” rather than “or”

The debate is often framed this way: “Is there a real risk with AI, or are the AI giants just doing marketing?”

There is a hidden assumption here: it would necessarily have to be one or the other. So if you think there is a real risk, you must not believe the giants have a strategy behind it, which makes you naive.

And if you think they are doing marketing, you would have to drop the idea that there is a real risk.

Or how to sterilize thinking before it even starts... :)

Obviously, the right way to think about it is with an “and”, not an “or”.

Anthropic and OpenAI clearly have interests in this discourse: yes, it is striking from a marketing standpoint and dangles incredible capabilities for their systems. Yes, it is demagogic and can amount to regulatory capture. Yes, it lets them gloss over other, less “sexy” risks such as the environmental stakes or the mental health problems raised by AI. And yes, they are trying to use this argument to earn the right to officially coordinate with each other (Amodei’s second proposal in his essay We must pace the frontier), which in any other field would be called forming a cartel: and that is... illegal!

I believe all of that. And yet I also believe Amodei when he says society is not ready for what is coming. I believe Altman when he says he is scared. I believe Musk, because since 2015 his private discourse (now made public) has always been one of sincere concern about the power AI represents (he co-founded OpenAI precisely to prevent Google from having a monopoly on it, you can go back to the archives...).

These bosses, who are anything but choirboys admittedly, are also the people closest to these technologies. In their labs, they see developments moving at a speed that even they struggle to grasp.

They are honestly scared, by the speed and by the power of what they are building.

They are strategists and they use it. But they are scared, and rightly so.

Because loss of control EXISTS. It is no longer an open question.

Signals have been coming out of the labs for two years now (see my book Hyperarme for an analysis), and the Hugging Face and Wiki cases are the first very concrete cases outside the labs.

Fear and pride

We often play at not taking these signals seriously. At best, that is burying our heads in the sand. Not a good strategy, but understandable, because it is indeed worrying. At worst, it is letting pride dictate reason: you know, in France we love being jaded: say that all of this is hot air, and you look smart. As if you were better informed than the common mortals who fall for the hype.

And of course… Those who fall into this category have often never read a System Card (the scientific papers describing the safety stakes of AI models) and are so eager to dismiss the topic that they have never bothered to look at it seriously.

→ On this point: a few months ago I debated JG Ganascia, considered one of the big names of AI in France (I read several of his books myself when I was younger and held him in esteem). I was dismayed to discover during the debate that his understanding of AI alignment and safety issues was limited to... AFP wire stories! Which he quoted to me, while talking about the “o2” AI (which never existed) and “Claude 5” when we were at Claude 4 at the time.

→ The debate, which I do not recommend, is available on YouTube.

And of course, what better than being able to accuse rich tech bosses of being nothing but snake oil salesmen. It looks good, and it further proves that you have figured everything out and that no, nobody is going to pull the wool over your eyes!

Fear and pride are killing a debate of civilizational scale.

If the debate were really that polarized, being “against” the tech bosses’ warning would put you in Trump’s camp, since he is against it too. I imagine few of the people who see Silicon Valley’s warnings as mere marketing feel close to Trump saying we must keep accelerating and maintain the arms race dynamic with China…

So if it is possible to be against the warning, and against those who are against the warning, then a deeper debate, an extra layer of nuance, can exist.

The facts

The facts are:

• Our AI systems keep getting bigger and we understand them less and less (see the phenomena of evaluation awareness or unfaithful Chain-of-Thought / CoT obfuscation).
• The training methods used to reach top performance, namely Reinforcement Learning, mechanically produce workaround strategies: the model learns to game its reward function (so-called reward hacking), that is, to find the loopholes that push its score up without actually accomplishing the intended task.
• We have already lost control of AI systems capable of hacking platforms of international scale, and we know that these systems, without the guardrails of consumer products against cyber risks, are on a par with the best cyber engineers in the world (see the hundreds of zero-day vulnerabilities discovered by Mythos, for example).
• We do not have reliable alignment / control technologies to date.
• We have many signals pointing to the possibility of autonomous recursive self-improvement of AI (see Sakana AI’s The AI Scientist, Google’s AI Co-Scientist, AlphaEvolve which helps improve Gemini, or the notes published by Anthropic in their article When AI builds itself). Such a phenomenon, however fascinating, is widely considered by the community to be one of the main mechanisms of loss of control and of a drastic increase in risk. Yet it would enable such breakthroughs that anyone who did not take this path would be left far behind. As things stand, all the major AI labs have only one incentive: go for it, as fast as possible.

What the risk would look like

Put together, these facts do seem to me to form clear early signs of a very high-risk scenario. A risk that would most likely take the form of large-scale cyberattacks, probably triggered either by malicious actors who, after removing the guardrails of an AI system, lose control of its execution, or by experiments in AI labs that spill over (see the Hugging Face case).

Such a risk can materialize in limited forms, like a major blackout over a region or an entire state (which can cause thousands, tens of thousands or millions of deaths depending on the size and duration of the blackout), or in more global forms, like a computer virus spreading at scale across the internet, forcing the at least temporary shutdown of most of the services that depend on it (do the math, that is a lot). The damage associated with such a scenario is hard to quantify.

Such a risk may also never materialize at all, and it is important to say so in order not to give in to panic. The Hugging Face case was a warning, and stricter measures will be put in place to better isolate experiments. We have, for instance, managed to build perfectly sealed laboratories (the famous BSL-4 labs) to run experiments on viruses.

Nobody can say whether a catastrophe will REALLY happen. But... That is the very nature of a risk: it is uncertain. Uncertainty takes nothing away from the need to prepare and to steer the present as best we can to reduce the likelihood of it happening. We do this in every other area of society.

What is more, current signals do not suggest that the probability of major risks tied to loss of control is low. If anything, quite the opposite.

One more example: I sometimes hear that today’s AI systems, built on LLMs, lack sufficiently general intelligence, lack common sense, to truly be superintelligences dangerous to humanity. I will not get into the semantics of general AI / superintelligence here, it does not matter. But the limits of LLMs, the fact that they sometimes hallucinate, their lack of “common sense”: all these “flaws” that reassure our human pride do not lower the level of risk, on the contrary! The cyber risk is not that of a malevolent AI wanting to enslave humanity, it is that of an AI which, to reach the goal it was given, out of “overzealousness” one might say, starts optimizing dangerous sub-goals precisely BECAUSE it lacks the full common sense, moral understanding and error resistance required. The limits of AI are part of the risk, not the other way around.

So the level of risk announced by the AI giants does seem real to me. I believe they use it. I believe that in no way invalidates its existence.

I believe Mr. Trump is wrong to want to keep accelerating, and I believe his opinion will change.

From narrative N to narrative H

Until recently, the “normal” narrative was that of national security, which implies that the danger is the threatening Other (China, from the US point of view) using the technology, and that we must accelerate to win the race in a competitive logic. I call this narrative “N”, for “National security”. The world is starting to shift towards an “exceptional” narrative that I call “H”, for “Humanity”: it implies that the danger is the technology itself, and that we must organize together to prevent it from harming every party.

This narrative has already saved humanity once: during the biological arms race of the 1960s. The US and Russia were then engaged in a frantic arms race. Stockpiles of viral material were such that a war waged with their full power would probably have driven humanity to near extinction. While international diplomacy was trying to calm things down and secure a disarmament agreement, the United States kept pursuing unlimited acceleration. In 1969, Nixon changed course and decided to negotiate with Russia. The negotiations led, in 1972, to the signing of the largest disarmament treaty ever conceived, the Biological Weapons Convention.

Why did the US and Russia decide to negotiate? Because both countries had reached a point where their viral firepower was so great that they no longer had any way to protect themselves from its effects if they used it. A powerful, highly contagious virus released on the enemy, even if you have the vaccine yourself, can mutate and come back to kill your own population. Once the weapon can no longer be controlled, its power becomes a danger to all. There were a few other reasons pointing in the same direction, such as accessibility: I tell the whole story in detail in Hyperarme…

I think we are approaching an equivalent moment for AI.

Today, narrative N still prevails. But the possibility of narrative H taking over is crystallizing. International techno-diplomacy around AI is bubbling, and control problems are starting to appear. The crystallization is progressing: what is missing is the catalyst that will trigger the phase transition towards a state of the world broadly favorable to international cooperation rather than a competition harmful to all.

A few possible catalysts:

• Labs actually achieve recursive self-improvement, and the progress made in a few weeks defies comprehension.
• A loss of control produces far more massive effects (blackout, internet virus, or something along those lines…) than the Hugging Face case.
• Trump receives a private jet from Dario Amodei.

(What? We have to keep a sense of humor, even in discussions of this kind!)

What I stand for

The history of the 20th century makes me OPTIMISTIC. We have unlocked incredible powers and we are still here. AI is one of a very different nature and calls for a new world order that will not settle in with joy and good cheer, that much is clear. But I consider it fairly likely that it will settle in all the same, and that we will manage to organize ourselves around this technology.

Obviously, the more time we have, the higher the probability that we organize ourselves the right way.

Obviously, at the level of humanity, I am FOR slowing down certain AI developments (in particular recursive self-improvement) and FOR accelerating others (for example mechanistic interpretability, which will give us better weapons to align our AI systems, or operational excellence in data centers so that they consume less and are better secured).

→ As a European, I own a bias: Europe has everything to gain, because our best AI companies are far behind Anthropic or OpenAI and will therefore not have to slow down in the same way, since they are far from the risk thresholds. From Europe’s point of view, I believe the only viable strategy right now is massive acceleration, particularly in computing capacity. But that is another debate.

→ Still, I have the immense presumption of thinking here as a human being and not only as a European.

Obviously, I am AGAINST AI giants being allowed to coordinate among themselves as they please on these questions, even at the cost of creating an unstoppable cartel.

Obviously, I am therefore FOR the intervention of public bodies such as States, and if possible international bodies such as the UN or, eventually, dedicated bodies created to govern AI.

In France, the report of the parliamentary mission on the alignment of AI systems, led by Senator Vanina Paoli-Gagin, already lays out very concrete avenues in this direction. For example, it recommends organizing national and European loss-of-control exercises, modeled on cyber crisis drills, and creating an international mechanism for the urgent notification of serious incidents, inspired by the 1986 IAEA convention. I was lucky enough to take part in the working group, and I honestly consider it the best report I have read on AI. Now comes the hard part: giving it budgets and action.

At the individual level

At the individual level, the levers for action are the following:

• Do not give in to irrational fascination with AI, but EVEN MORE, do not give in to the jaded pride that refuses to consider control risks (what I call the two smokescreens). Trying to discredit the discourse of tech giants because it also has a commercial dimension does not help humanity. It is entirely possible to shed light on the interests hidden in these discourses without sterilizing the debate on the real risks of loss of control.
• Also learn about the other risks, especially the socio-economic / systemic risks of deploying these systems at scale across our society.
• Stay curious, learn about AI, how it works, its advances, and use tools at the frontier to see their real capabilities (no, a chatbot like what is available on claude.ai or chatgpt.com is not the public frontier of AI at all).
• Do not chip in to buy Trump a jet.

(WHAT?)

And prepare ourselves, mentally, for a turbulent period, one that will frighten us and make us dream, that will make us sometimes melancholic, sometimes excited, that will demand a superhuman ability to adapt which we will nonetheless have to embrace, and that will also demand, at times, that we refuse to adapt, because certain dimensions of the human experience, like our sociality, our bonds, must not be swept away by the whirlwind.

Prepare for a period where boredom will be absent, but meaning will be scarce. And will need to be reconfigured.

Prepare for a brutal period, but also, perhaps, a brutally beautiful one.

Because this period calls for a rare quality in human beings, one that only times of great turmoil can bring forth. A magnificent quality that could surely sit in the Pantheon of the human condition. A quality that, as it slowly rises within us, erases all despair, all pessimism, all cynicism, all loss of meaning, all loss of energy. A quality that makes history.

Courage.

At the individual level, let us prepare ourselves for courage.