How do I roll out an AI agent safely (shadow, 10%, kill switch)?
Start in shadow mode, where the agent sees real traffic and drafts its actions but writes nothing and talks to nobody, then send it 10% of traffic, then 50% including peak hours, then 100%, with a tested human fallback and a kill switch at every stage. Move up only when the checks you agreed with the customer pass and a named person on their side approves.
Before shadow: test on their history
Replay the customer’s own past calls or tickets first, not synthetic personas. Rokan’s catalog uses the last 90 days, at least 20 cases per case type and every case of the rarer types, with 1 in 5 held out that nobody tunes against. Run each case several times: single runs overstate reliability, and in τ-bench GPT-4o passed the same retail task 8 times in a row only about 25% of the time (τ-bench paper). Review every failure with the customer before anything goes live.
The stages
Hold times are Rokan’s planning assumptions, not measurements. A stage ends when its exit check passes and the customer’s named approver says go, not when the days run out.
- Shadow mode (0%). The agent sees real traffic and drafts its actions. Nothing is written and nobody hears from it. Staff compare its drafts with what they did. Move on when: The agreed checks pass on shadow traffic and staff sign off on the failures. Hold: about 3 days.
- 10% of traffic. One in ten calls or conversations goes to the agent. The rest follow today’s route. Move on when: No unsafe writes, escalations within the agreed range, and every failure reviewed. Hold: about 3 days.
- 50% of traffic. Half of the traffic goes to the agent, including peak hours. Move on when: The same checks hold at peak volume and the fallback team keeps up. Hold: about 4 days.
- 100% of traffic. All traffic goes to the agent. The fallback and the kill switch stay in place. Move on when: After launch, replay checks rerun whenever a connected system, prompt or model changes. Hold: ongoing.
- Kill switch, at every stage. One action sends all new traffic back to today’s route, usually the customer’s human team. Drill it in shadow mode and again at 10%. Move on when: Pull it on a wrong-record write, error or escalation rates above the agreed threshold, a connected system failing or changing its fields, or staff reporting a problem they can’t work around. Hold: always on.
The fallback and kill-switch drills
- Name who can pull it, on your side and the customer’s, and time how long it takes from the decision to 0% agent traffic.
- Call the fallback at the hours the agent runs, including after hours. A fallback number nobody answers at 9 p.m. isn’t a fallback.
- Transfers carry a summary, so the caller doesn’t have to start over with a person.
- Revoke-all drill. Every token and credential the agent holds can be cut in one step. Security reviewers ask for this after the Salesloft Drift breach (Google Threat Intelligence).
- Check capacity before 50%. Retell’s default is 20 concurrent calls; past that, inbound callers wait about 40 seconds, then go to the fallback number or drop (Retell docs). Compare with peak concurrency in the history.
Mistakes that turn rollouts into rollbacks
- Cutting staff before proof. Commonwealth Bank of Australia reversed a plan to replace call-centre roles with AI (Finextra). Track resolution and repeat contacts, not containment.
- Changes that skip the gate. A model provider’s own bugs can degrade an agent with no customer change (Anthropic postmortem). Rerun replay on every prompt, model or connector change.
- Confident wrong answers. Air Canada was held liable for its chatbot’s refund answer (BC Civil Resolution Tribunal). Ground policy and price answers in the customer’s current documents.
- Model deprecations. OpenAI shut down o4-mini-2025-04-16 on October 23, 2026 (OpenAI). Keep a deprecation calendar and requalify on the replay suite before switching.
- No written exit criteria. Without them, a pilot never ends. Agree the numbers at signature (Gartner).
Short answers
What is shadow mode for an AI agent?
The agent sees real traffic and drafts what it would do, but writes nothing and nobody hears from it. Staff compare its drafts with what they actually did before any live traffic.
What share of traffic should an AI agent start with?
A common sequence is 10%, then 50% including peak hours, then 100%, moving up only when the agreed checks pass and the customer approves each step.
What should trigger an AI agent’s kill switch?
A write to the wrong record or account, error or escalation rates above the agreed threshold, a connected system failing or changing its fields, or staff reporting a problem they can’t work around.