California ordered a kill switch for frontier AI. Defence systems build that in from day one.

Governor Newsom's new executive order gives California's Government Operations Agency until November 16 to hand back recommendations on three things: a "kill switch" mandate for frontier AI models, independent auditors embedded onsite inside the big labs, and an expanded list of incidents companies must report, including loss-of-control events. It reads like the state finally caught up to what safety-critical engineering has known for decades: you cannot regulate a system you cannot see inside, and you cannot trust a shutdown mechanism you have never tested under load.

I've spent years building systems where "what happens when this goes wrong" isn't a policy question, it's a spec line. Real-time drone threat detection for a defence deployment doesn't ship with a kill switch as an afterthought. The fail-safe is architected in from the first design review, tested adversarially before anything is fielded, and it triggers locally, on the device, in milliseconds, because the alternative is a threat detector that keeps running while someone three time zones away waits for a server to respond.

That's my problem with treating "kill switch" as the headline here. A shutdown mechanism that depends on a request reaching a vendor's cloud, routed through whatever infrastructure that vendor runs that week, isn't a kill switch. It's a support ticket with better branding. If a frontier model can act autonomously enough to need an emergency stop, the stop has to be as fast and as local as the failure mode it's meant to catch. Bolt a switch onto a system after the fact and you've built a governance ritual, not a control.

The part of the order that actually matters is the onsite auditors. Embedding an independent verification organization inside a lab, with standing access to test and evaluate before something ships, is closer to how defence programs already work: independent test and evaluation isn't optional before you field a system that can hurt someone, it's the gate. Applying that discipline to frontier labs is a real structural change, not a press release. It's also the harder thing to build, because it requires access, technical depth, and enough independence that the auditor isn't just reading the lab's own eval numbers back to them.

Worth remembering that Newsom vetoed SB 1047 in 2024, which would have required developers to prove they could shut a system down promptly. This order is the same governor, two years later, walking that requirement back in through the front door with more structure around it. That's not a contradiction, it's what happens when the incident reports start piling up faster than the political cover for ignoring them.

None of this works if the kill switch and the audit are things a company can plausibly claim rather than things a regulator can independently verify. I've published on formal verification because "the model behaved correctly in testing" and "the model is provably constrained" are different sentences, and only one of them survives contact with an adversarial input the developers didn't think of. A frontier lab self-attesting that its shutdown path works is exactly the kind of unverified claim that formal methods exist to catch. If California wants this to be real, the November recommendations need to specify how the kill switch itself gets tested, not just that one must exist.

Regulators are right to worry about loss-of-control events. They're wrong if they think a switch on paper is the same as a switch that's been fired in a drill. Build the audit requirement first. Make the shutdown path something the verification organization has actually triggered, under adversarial conditions, before anyone calls it a kill switch. Otherwise November 16 produces a document, not a control.