Feature Flags and Kill Switches for AI Model Rollouts

Shipping a new model version is not like shipping a CSS tweak. A bad prompt can raise cost 10× in an hour. A tool schema change can break every agent turn. A provider swap can silently degrade quality while your dashboards still look "green." If your only rollback is a hot hotfix PR, you are already late. Feature flags and kill switches are how production AI teams change models safely — and stop damage in seconds, not deploy cycles.

Flag the units that actually change behavior

Do not hide an entire "AI feature" behind one boolean. Split the blast radius. You want independent control over the model id, the prompt/contract version, retrieval config, tool allowlists, and temperature or decoding settings. When something goes wrong, you need to know which lever to pull — and you need to pull only that lever.

• Model flag: provider + model id + max tokens (and optional fallback model)
• Prompt flag: contract version or template hash — never "latest" in production
• Retrieval flag: index, top-k, filters, and embedding model separately
• Tools flag: which tools the agent may call for this tenant or surface
• UX flag: streaming on/off, citations required, or degrade-to-search mode

Roll out by cohort, not by hope

Percentage rollouts work for AI the same way they work for web — if you define sticky assignment. Hash user or tenant id so the same customer does not bounce between model A and B mid-session. Start with internal dogsfood, then a small paying cohort with known eval baselines, then widen. Hold a fixed control group on the previous config so you can compare cost, latency, and task success — not just "it felt better in Slack."

A sensible default ladder

• 0%: flag exists, shadow traffic or offline eval only
• 1–5%: internal + friendly tenants; watch p95 latency and $/successful task
• 10–25%: expand if eval gates and error budgets hold
• 50%+: only after you have a documented kill path and a stable fallback
• 100%: promote the config to default — keep the old flag for 1–2 release cycles

“If you cannot name who is on the new model and who is on the old one, you do not have a rollout — you have a rumor with an API key.”

Build a kill switch that kills

A kill switch is not "open a PR to revert." It is a control that stops or degrades AI paths without waiting on CI. Wire it to three outcomes: hard off (feature returns a clear unavailable state), soft degrade (faster/cheaper model or retrieval-only answers), and spend circuit (cap tokens or requests per tenant when cost spikes). Test the switch in staging the way you test failover — click it, confirm traffic moves, confirm invoices stop climbing.

• Global kill: disable a surface for all tenants in under a minute
• Tenant kill: isolate one customer without taking down everyone
• Model kill: pin traffic back to the last known-good model id
• Spend kill: trip when $/hour or tokens/minute exceed a budget

Observe the flag, not just the model

Every AI request should record which flags resolved: model, prompt version, tools, rollout bucket. Your traces and cost dashboards must slice by those dimensions. Otherwise a 5% bad cohort looks like noise. Alert on regression relative to the control bucket: eval score drop, refusal spike, tool-error rate, TTFT, and cost per successful task. Flags without observability are theater.

Treat rollouts as release engineering

Put flag names in the PR template. Require an owner, a success metric, a rollback owner, and a kill-switch rehearsal before any model goes past 5%. Expire stale flags so you do not drown in half-dead experiments. At Brutanix Studios we ship AI the way we ship product infrastructure: small cohorts, sticky assignment, measurable promotion criteria, and a kill path that works when the pager goes off — not after the blog post about the outage.