A blast-radius metric for autonomous agents — and how to cap it

Here is a AI risk management problem that deserves more rigour than it usually gets.

The mechanism

Risk management quantifies what can go wrong, its likelihood and impact, and the residual exposure once controls are applied. For agentic AI — systems that take actions (payments, writes, messages, code changes), not just emit text — the exposure concentrates in the action path. Model it explicitly: enumerate failure modes (prompt injection, tool misuse, data exfiltration, unsafe autonomy), score inherent risk on a likelihood × impact scale, then apply layered controls and track residual risk against a declared appetite.

Why it compounds

As deployments move from assistive (the model suggests) to agentic (the model acts), errors stop being isolated outputs and become state changes — money moved, records altered, messages sent — that propagate downstream. A small per-action failure rate, multiplied across thousands of autonomous steps, becomes material operational, legal and financial exposure. The differentiator is not adoption speed but the ability to bound, observe and reverse what the system does.

Controls that apply

• A risk register separating inherent vs. residual risk, with named owners and a review cadence.
• Action-path controls: human approval on irreversible/sensitive actions, least-privilege scopes, full payload logging, redaction, and a kill switch.
• Continuous monitoring of key risk indicators — drift, fairness/disparate-impact, guardrail-breach and prompt-injection rates — not a one-time review.
• A rehearsed incident-response runbook with defined severities and escalation paths (cf. SR 11-7 model-risk lifecycle).

What to test

• Instrument one high-impact use case end-to-end with per-action logging.
• Define quantitative thresholds (error rate, autonomy level, blast radius) and an automatic stop.
• Run a bounded pilot; compare incident rate, review latency and residual risk against the baseline.
• Capture near-misses as structured signals — they are the cheapest telemetry you have.