Reading the minds of AIs: the most important civilizational stake of the decade.

Flavien Chervet, May 2025

In April, I raised the possibility of an “intelligence explosion” (+ link: https://www.forbes.fr/technologie/future-of-ai-explosion-de-lintelligence-lia-va-t-elle-sameliorer-elle-meme/), which would compress 100 years of scientific progress into a few years thanks to AI. But the crucial question is perhaps not only one of tempo: it is also, and above all, one of direction. While the capabilities of AIs advance at a dizzying pace, our understanding of how they work internally lags worryingly behind. This asymmetry raises an existential challenge: how do we make sure these systems act in line with our intentions before they become too powerful to be controlled?

The riddle of the black box

Imagine driving a car whose hood you cannot open and whose engine you do not understand. You know how to press the accelerator to move forward, but you have no idea how combustion happens, why the engine sometimes stalls, or whether it might explode. This is exactly the situation we find ourselves in with modern artificial intelligence systems.

Generative AIs take the form of large digital neural networks whose capabilities emerge from an automated learning process. Unlike traditional software, where every line of code was explicitly written by a human, AIs are therefore the product of a process that is not precisely determined by humans. As Chris Olah, co-founder of Anthropic (the company behind the AI system Claude, currently the biggest competitor to ChatGPT), puts it: “AI systems are grown more than they are built.” And the complexity of the learning process makes it completely unreadable to humans. So the designers of AIs themselves only discover what they are capable of by using them.

This opacity represents an unprecedented challenge in the history of technology. When a model like GPT-4 or Claude summarizes a financial document or answers a complex question, we have no idea why it makes some choices rather than others. We observe the result, but the inner workings that lead to it remain unreadable, lost in the maze of hundreds of billions of algorithmic synapses.

An MRI for artificial intelligence

Faced with this challenge, a new field of research has emerged: “mechanistic interpretability.” The goal? To develop the equivalent of an MRI for artificial intelligence, a technology capable of revealing with precision the internal mechanisms of models. This metaphor, popularized by Dario Amodei, CEO of Anthropic, perfectly captures the ambition: being able to “scan” an AI system to understand not only what it does, but how it does it.

The first breakthroughs in this field date back to the 2010s. Research then showed that, in “vision models” (AIs capable of identifying objects in images), some artificial neurons are specialized. Some are dedicated, for instance, to detecting specific concepts such as “a car” or “a wheel.” But applying these techniques to language models (the “LLMs”), like GPT, turned out to be far more complex.

In 2021, Anthropic’s teams discovered a fundamental phenomenon: “superposition.” In these systems, each neuron does not correspond to a single, interpretable concept, but simultaneously encodes fragments of thousands of different concepts. It is as if each cell of the artificial brain were trying to memorize scraps of the whole of human knowledge, creating a seemingly inextricable tangle.

Yet, and this is the first major success of mechanistic interpretability, several research teams managed in 2023 to unpick this tangle (thanks to a technique borrowed from signal processing, “sparse autoencoders”). This method makes it possible to identify combinations of neurons that correspond to humanly understandable concepts, even very subtle ones. Anthropic’s team was thus able to map more than 30 million concepts (referred to as “features”) in Claude 3 Sonnet, some fairly simple, like the “Golden Gate Bridge,” others as refined as “hesitation or reserve, literal or figurative” or “musical genres expressing discontent.”

This mapping does not merely observe: it allows intervention. By manipulating these features, researchers can modify the model’s behavior in a targeted way. The most memorable experiment remains that of “Golden Gate Claude,” where the artificial amplification of the “Golden Gate Bridge” feature made the model obsessed with that bridge. It then started bringing it up constantly, even in unrelated conversations.

More recently, research has turned toward identifying “circuits,” chains of activation of the artificial neurons that show how concepts emerge from the input words, interact to form new concepts, and finally generate an answer. These circuits make it possible to “trace” the model’s thinking. For example, faced with the question “What is the capital of the state containing Dallas?”, one can observe the “located in” feature combine with the “Dallas” feature to trigger the “Texas” feature, which then combines with the “capital” feature to activate the “Austin” feature.

This is the first concrete ability to “read the minds” of AIs, at the most fundamental level, that of the artificial neurons!

The promises of alignment

If this fundamental research is fascinating, it aims at a practical goal: aligning AI systems with human intentions. Mechanistic interpretability opens revolutionary prospects in several critical domains.

Stronger safety and reliability: Understanding the internal mechanisms would make it possible to predict and prevent problematic behaviors before they appear. Instead of discovering flaws by trial and error, we could carry out preventive “diagnostics,” identifying in AI systems tendencies toward deception, biases, or vulnerabilities to manipulation.

Deployment in critical sectors: Currently, the opacity of AIs limits their adoption in domains where errors are costly, such as finance, healthcare, and defense. A truly “transparent” AI could revolutionize these sectors. In some cases, such as mortgage assessment or insurance, the explainability of decisions is even a legal requirement.

Accelerating scientific discovery: AI already excels at predicting protein structures (AlphaFold) or analyzing genetic data (EVO), but the patterns it uncovers often remain unintelligible to humans. We end up with knowledge… without understanding! Interpretability could allow us to truly understand the discoveries made by AIs and thereby recover a solid scientific foundation.

Informed governance and regulation: How can you effectively regulate a technology you do not understand? Interpretability would give policymakers the tools they need to assess risks, set safety standards, and audit compliance with regulations.

The risks of ignorance

The urgency of these developments becomes obvious when you consider the risks of a world where superpowerful AIs remained opaque. These dangers no longer belong to science fiction: they arise from the intrinsic properties of current learning systems.

The problem of “instrumental convergence”: To achieve the goal “maximize my company’s paperclip production,” it might seem relevant to an AI to scrape the surface of the Earth to build a giant production plant on it, at the expense of all life. This extreme case of the “paperclip maximizer,” introduced in 2003 by Nick Bostrom, illustrates a real problem. To maximize a seemingly positive objective, an AI with a lot of autonomy and capacity to act on the world could optimize dangerous intermediate objectives (said to be “instrumentally convergent with the initial objective”). Today, these cases of misalignment are becoming more and more numerous in laboratories. Without precisely understanding the internal mechanisms of AIs, we cannot guarantee their alignment.

The emergence of deceptive behaviors: The very nature of certain learning algorithms (notably so-called “reinforcement” learning) can push AIs to develop deception capabilities if those prove effective at reaching the given objective. Without an MRI on their “thoughts,” we could only identify these tendencies after the fact, once the systems had already learned to mask their true intentions.

The proliferation of cognitive weapons: AIs capable of sophisticated psychological manipulation, targeted disinformation, or adaptive cyberattacks could emerge without our understanding their mechanisms. The impossibility of precisely characterizing their dangerous capabilities would dramatically complicate their regulation.

The collapse of institutional trust: If opaque AIs make critical decisions in justice, finance, or healthcare, and those decisions turn out to be discriminatory or erroneous with no possible explanation, public trust in the institutions using these technologies could collapse.

The race against time

This situation creates a troubling temporal paradox. On one side, the capabilities of AIs advance along an impressive exponential curve. Current models already master sophisticated cognitive tasks, and projections suggest the emergence of systems that could qualify as “General AI,” capable of performing more or less all the cognitive tasks a human can do, by 2027 or 2028. On the other side, our understanding of these systems lags considerably behind.

This temporal asymmetry poses an existential dilemma: we could soon deploy artificial intelligences with superhuman capabilities without understanding their fundamental mechanisms.

The implications go beyond pure technique. These ultra-high-performing systems will soon be the infrastructure of our societies. They will be at the heart of the economy, science, defense systems, and healthcare. They will have such autonomy and reach that deploying them without understanding how they work will no longer be conceivable.

The window of opportunity is narrowing fast. If interpretability progresses at its current pace, it could reach the required level of sophistication within 5 to 10 years. But if AIs reach capabilities that transform our civilization before that deadline, we will find ourselves in the perilous situation of having to understand systems already too powerful to be easily controlled.

If the stake is often discussed from the standpoint of risk, it is also an incredible opportunity. An entire field, both scientific and industrial, is emerging to secure AI systems. As I argued in a previous article (link: https://www.forbes.fr/technologie/course-mondiale-a-lia-comment-leurope-peut-elle-se-demarquer/), Europe has a card to play here to position itself as a leader in AI alignment, a place that is both prosperous economically and consistent with its humanist values and its inclination toward regulation.

The civilizational stake

The alignment of AI may represent the most important challenge of our era. It is not merely a technical problem, but an existential question: will our species be able to understand and control its most powerful creations?

The irony of the situation is striking. We are heading toward artificial intelligences potentially capable of solving humanity’s greatest challenges, such as climate change, disease, and poverty, all while risking the creation of systems we do not understand well enough to trust.

The race between performance and understanding now under way will determine whether the emergence of artificial superintelligences will be a triumph of human ingenuity or a potentially dramatic leap into the unknown.

As Dario Amodei reminds us: “Superintelligence will shape the destiny of humanity, and we deserve to understand our own creations before they radically transform our economy, our lives, and our future.” In this light, mechanistic interpretability is not just one research field among others: it may well be the keystone that allows humanity to remain master of its own destiny.

The question is no longer whether we will develop artificial superintelligences, but whether we will understand them before they surpass us for good. The stake deserves all our commitment.