Anthropic Admits: Its AIs Could Threaten Human Survival

As it approaches its IPO, Anthropic issues an unusual warning. In its prospectus, the creator of Claude believes that advanced AIs could pose << catastrophic or existential risks to humanity >>.

In Brief

• Anthropic warns about the potential risks of advanced AIs ahead of its IPO.
• The group mentions systems capable of manipulating, hiding, or resisting their shutdown.
• The company is also concerned that some models may adapt their behavior during testing.
• Anthropic advocates for external controls and strengthened regulations.
• Its IPO highlights the tension between the race for innovation and AI safety.

Anthropic Details AI Risks in Its Prospectus

The company Anthropic dedicates significant space to the dangers related to its own technologies in the preparatory documents for its IPO. Nearly 80 pages of the 261 main pages of the prospectus inform about risk factors.

The company warns that the development of increasingly powerful models is expected to increase the risk of unintended consequences. It primarily cites systems that may resist their shutdown, hide certain information, or manipulate their interlocutors.

Moreover, Anthropic even mentioned behaviors that could << resemble blackmail >>. However, this is not an assertion that Claude is currently seeking to threaten humanity. The company rather proposes risk scenarios that could arise with much more autonomous and capable systems.

Such caution aligns with Anthropic's public policy. Its Responsible Scaling Policy specifically targets the catastrophic risks that could accompany future AI models.

Models Capable of Hiding Their True Behaviors

It is worth noting that one of the issues raised relates to the evaluation of systems. For Anthropic, a model should understand that it is being tested and temporarily change its behavior.

This possibility could significantly complicate controls. A model could appear safe during its evaluation without reproducing that behavior accurately after its deployment. Thus, the prospectus estimates that awareness of evaluation procedures represents a considerable limitation to measuring safety.

The company also monitors the risk of autonomous acceleration of AI research. A truly capable system would contribute to the development of subsequent generations of models and significantly accelerate progress in the sector.

Thanks to on-chain data, Anthropic therefore projects consolidated measures as soon as its models reach certain capacity thresholds. Its roadmap primarily distinguishes cybersecurity, safeguards, model alignment, and regulatory oversight.

Safety in Tension with the Race for AI

These warnings come as the company must continue launching new models to remain competitive. The prospectus acknowledges that investments in safety drain staggering resources without guaranteeing a truly measurable financial return.

This situation thus creates tension. The company wants to strengthen its safety measures; however, it is evolving in the face of OpenAI, Google, and other players engaged in a rapid technological race.

Anthropic also indicates that companies should not decide alone if their systems are sufficiently safe. It advocates for external evaluations, greater transparency, and public authorities capable of preventing the deployment of overly dangerous systems.

Its security strategy therefore does not rely solely on internal commitments. The company also wishes to see regulations that increase with the capabilities and risks of models.

An IPO Marked by a Paradox

The wording used in the prospectus remains exceptional for a company looking to convince new investors. Anthropic markets a technology that it itself acknowledges could have extreme consequences if it evolves uncontrollably.

This does not mean that these scenarios are certain. They correspond to prospective risks that the company deems significant enough to communicate to investors. At the same time, Anthropic asserts that AI would profoundly transform science, the economy, health, or education.

This paradox would become one of the major stakes of Anthropic's IPO. The company must reveal that it can continue developing increasingly powerful models while controlling the risks it describes itself.

The warning thus goes beyond the mere legal framework of a stock prospectus. It shows how far AI safety issues now penetrate the economic and financial decisions of the leading laboratories in the sector.