Anthropic Tells Newsom Courts Should Be Able to Halt Risky AI Models

Anthropic has told Gov. Gavin Newsom that California should let its attorney general ask a court to suspend an AI model's deployment, internal use or further development if independent evaluators find a near-term risk of catastrophic harm, according to an Oct. 15 letter first shared with Politico.

The letter is Anthropic's response to Newsom's Sept. 18 executive order (N-9-26), which asks state officials to study ideas such as embedded independent auditors and a "kill switch" for frontier models and report back by Nov. 16. Nothing in it is law.

Anthropic also said large developers should publish risk reports at least every six months, and that companies should pay into a state-run fund rather than hire their own outside auditors. On kill switches, it said every frontier company should be able to rapidly shut down its models but warned that a shutdown-only law "could provide a false sense of reassurance."

OpenAI sent Newsom a separate letter last week; Politico reported it did not go as far as Anthropic on letting the government block a model. Newsom's office told InsideAIPolicy it was reviewing OpenAI's letter; no response to Anthropic's was found.

Sources: Anthropic letter to Gov. Newsom; Politico; California Governor's Office.