The agent kill switch: stop conditions, safe state and a drill that proves it

Speed is the appeal of autonomy; the ability to stop is what makes the speed survivable. Every agent that can act needs two distinct stop mechanisms: a kill switch a human can hit, and a circuit breaker that trips itself. The requirement is no longer just engineering taste. For high-risk systems, Article 14(4)(e) of the EU AI Act spells it out. Overseers must be able “to intervene in the operation of the high-risk AI system or interrupt the system through a ‘stop’ button or a similar procedure that allows the system to come to a halt in a safe state”. Most agents will not be high-risk systems; the design standard is worth stealing anyway, because the regulation’s phrase safe state names exactly the hard part.

Two mechanisms, two failure speeds

The circuit breaker predates agents by decades: a component that watches for failure conditions and opens the circuit before damage compounds. Applied to an agent, it is the automatic layer — it reacts in milliseconds to conditions you defined in advance, and it does not need anyone awake. The kill switch is the manual layer for the failure you did not predict: a human sees something wrong and stops everything, without needing to diagnose it first.

The division of labour matters because each covers the other’s blind spot. A breaker cannot trip on a condition nobody wrote down; a human cannot react in the 400 milliseconds it takes an agent to issue its next tool call. Design them separately, wire them to the same halt path, and never let one’s existence excuse skipping the other.

Trip conditions you can copy

Breaker thresholds are workload-specific; the shape of a good trip set is not. Rates catch runaway behaviour while there is still budget left. Totals only confirm the damage afterwards:

# Circuit breaker trip conditions — tune multiples to your baseline.
- trip: cost_rate
  when: cost_per_hour > 3 * baseline_hourly     # runaway loop or context bloat
- trip: loop_depth
  when: steps_in_run > 25                        # thrashing, not a hard task
- trip: tool_error_burst
  when: tool_errors_in_5m >= 5                   # the world broke; stop acting on it
- trip: novel_write
  when: write_action AND domain NOT IN allowlist # first-ever side effect => pause + page
- trip: oversight_offline
  when: no_heartbeat_from_monitor > 10m          # nobody is watching => degrade to read-only
on_trip:
  - halt_new_actions
  - cancel_queued_jobs
  - revoke_short_lived_credentials
  - page_owner_with_last_trace

The last trip condition is the one teams resist and later thank themselves for: if the monitoring that would catch a failure is itself down, the agent should not keep acting on the world. An unwatched agent and a broken agent look identical from the outside. The signals that feed these conditions come straight from the observability layer — an untraced agent cannot even tell you it needs stopping.

Safe state is the actual design problem

Halting is easy; halting safely is the work. Killing the agent’s process leaves in-flight tool calls completing, queued jobs running, sub-agents working and tomorrow’s scheduler ready to resurrect the whole loop. The Act’s phrase — “come to a halt in a safe state” — translates into four checkable properties. No new actions start. In-flight actions complete atomically or roll back, never half-finish. Credentials the agent holds stop working. And the state left behind is one a human can inspect and resume from. Write those four down for your agent specifically, because “safe” is different for an agent that sends emails, one that moves money, and one that edits code.

What no breaker looks like: a case from this portfolio

The most instructive stop-layer failure I have operated was an agent with no stop layer at all. One site in this portfolio ran two scheduled agents that refreshed content and published straight to production every morning. No cost breaker was needed — the runs were cheap. No error breaker tripped — the pipeline was green every day. The condition nobody wrote down was external acceptance. Google’s crawler quietly stopped indexing the output, and by the time the drop showed in Search Console, 209 of 213 pages were out of the index. The automation had run flawlessly for weeks producing something the outside world had already rejected. When you do not define your trip conditions, the environment defines them for you — later, and at a worse price. The novel_write and outcome-rate conditions above exist so that the system’s definition of fine and the world’s definition of fine cannot drift apart unobserved for a month.

Make the stop path boring

Three properties turn a stop design from a diagram into a control. Independence: the halt path must not run through the agent. A stop that asks the agent to please stop is a suggestion — revoke at the gateway and credential layer the agent cannot touch. Reachability: the person on call can trigger it in seconds without a deploy, and knows they are allowed to — a stop that needs a change request is not a stop. Rehearsal: drill it quarterly with a synthetic runaway; measure time-to-halt; verify the four safe-state properties held. Gate each increase in autonomy on those drills passing, the same staged way deployment autonomy is earned. The human side of the design — who watches, who decides — is the subject of human oversight and agent autonomy. For the systems the AI Act does classify as high-risk, the compliance timeline says when Article 14 stops being advice.