What the pause is about

On September 12, Anthropic CEO Dario Amodei called for slowing the advance of AI capabilities. Sam Altman publicly supported pacing too, and endorsed independent evaluators with extensive access to the labs. The stated concern is straightforward: AI is improving so quickly that our ability to understand and control it may fall behind. Slowing down would give safeguards time to catch up. [1]

These are fierce competitors with personal animosity towards one another. Their personal conflict even produced an awkward moment onstage in February, when they refused to touch each other as the other leaders joined hands. Seeing them converge publicly is surprising to put it mildly. It suggests they take this very seriously. [2]

Calls for pacing AI research are not new, and momentum has been building gradually. A July statement signed by Amodei and senior researchers at OpenAI and other labs already called for ways to pace development. It opened by pointing to the approaching automation of AI research. [3]

Still, it raises the question: why the surge in interest now? Did they see "god in a box" and get spooked or is something else going on? I think the explanation is much simpler. And I think another event this week helps explain it.

On September 8, OpenAI announced a proposed solution to the Navier-Stokes Millennium Prize Problem, one of mathematics' most famous unsolved problems. Three days later, 25 Fields Medalists published a declaration arguing that the goals of AI companies and the mathematical community were severely misaligned. [4][5]

Their argument is that mathematics involves much more than producing correct answers. Working through problems develops understanding. New methods spread through discussions, teaching, and careful explanations. A proof matters partly because of everything people learn while creating and studying it. Producing answers faster does not automatically reproduce that process. [5]

A result nobody can interpret or build on has limits. That is true. But I also hear something familiar in the response. It is easy to welcome the automation of somebody else's work. It feels different when the machine starts automating your job.

Imagine dedicating thirty years to a field. You have developed judgment, taste, and abilities that very few people possess. Your colleagues respect you. Society has given you a high social standing. Your work gives your life structure and meaning. Then a machine starts producing results outside your personal reach. Even if you remain financially comfortable, something important has changed. Maybe you still like the work. Maybe you want to continue doing it. But the world's need for you has come into question.

Of course that hurts.

It also creates a strong incentive to cast suspicion on the machine and its progress. First you find faults in the output itself. Once that stops being productive, you find other ways to assert the world's need for you. I write software, so I've had to grapple with this myself. And the arguments are eerily similar. It's also eerily similar to the arguments of literally anyone who has had their work automated. The mathematicians are really not that special.

Now return to the AI labs.

The development drawing so much concern is AI becoming capable of doing AI research: designing experiments, writing the software, interpreting results, and helping build the next generation of models. As those systems improve, they may contribute more to improving their successors. That is the basic idea behind recursive self-improvement (RSI).

It is also the automation of the people who work at AI labs.

The rising tide of automation has now reached the people producing the tide. For their employers, the tide has reached the expertise their companies are built around. The people asked to bring about this transition are also those with most to lose. An interesting conundrum. The financial entity and its customers want progress; the people powering the entity are more ambivalent.

My suspicion is that this explains most of the recent willingness inside the frontier labs to rally around pacing. The prospect has become personal. If automating AI researchers were a distant prospect, I doubt we would see the same urgency or the same coalescing of forces.

The labs' stated explanation is that automated research could accelerate capabilities beyond their ability to keep the systems safe. As always, incentives shape outcomes. None of this requires anyone to be consciously dishonest. People can sincerely experience a threat to their vocation as a threat to something much larger. This can also be seen in other sectors. Teachers and journalists have frequently called the threat to their vocation threats to democracy or even truth itself.

It's good to recognize the pain of becoming less necessary. It's also good to keep it in proportion.

What the pause is about

Sources