Model Routing and Cascades That Cut Cost Without Killing Quality

Sending every request to your strongest model is simple — and expensive. Sending every request to your cheapest model is also simple — and users notice. Production AI products need a third option: route by task, confidence, and risk so easy work stays cheap and hard work still reaches a frontier model. At Brutanix Studios we treat model routing like traffic engineering, not a prompt trick: explicit policies, measurable promotion rules, and fallbacks that fail closed on quality.

Routing is a product policy, not a model opinion

Do not ask the expensive model whether it should have been called. Decide in code. A router is a deterministic (or lightly learned) policy that maps request features to a model tier before generation starts. Features that matter: intent/task type, input length, required tools, tenant plan, latency budget, and whether the action is reversible. If those dimensions are not logged on every request, you cannot tune the router — you can only argue about invoices.

• Tier 0: rules, templates, or retrieval-only — no generative model
• Tier 1: small/fast model for classify, extract, rewrite, and short Q&A
• Tier 2: mid-size generalist for grounded answers with light tool use
• Tier 3: frontier model for hard reasoning, ambiguous intent, or high-stakes drafts

Start with intent routing before clever cascades

Most cost savings come from not calling a large model for work a small one can finish. Classify the job first: FAQ, data extraction, code edit, multi-step agent turn, creative draft, or support reply with refund risk. Map each intent to a default tier and a max spend. A cheap classifier (or even regex + embeddings for closed intents) pays for itself quickly when 60–80% of traffic is short and repetitive.

Intent → tier examples

• Normalize a shipping address or extract invoice fields → Tier 1 with schema validation
• Answer from a known FAQ / policy chunk with citations → Tier 1 or retrieval-only
• Copilot that may edit code or call write tools → Tier 2 with tool allowlists
• Executive summary across messy multi-doc context → Tier 3 with length and spend caps

“If your router cannot explain why this request skipped the frontier model, you do not have routing — you have a random discount on quality.”

Cascades: try cheap, escalate on signal

A cascade runs a cheaper model first and escalates only when a gate fails. Gates are the load-bearing part: confidence score, schema validity, citation coverage, self-consistency checks, or a second-pass verifier. Escalation must be bounded — one hop is usually enough. Infinite retry loops to "better" models turn latency spikes into cost spikes.

• Generate → validate: Tier 1 draft; escalate if JSON/schema or required fields fail
• Answer → verify: Tier 1 answer; Tier 2/3 only if citation overlap or faithfulness score is below threshold
• Draft → critique → revise: small model drafts; stronger model reviews only high-risk intents
• Shadow escalate: log what Tier 3 would have said for a sample without serving it — promote cascade rules from disagreement data

Fallbacks that protect UX when a provider blinks

Routing is also resilience. Providers rate-limit, region-fail, and silently degrade. Define a fallback chain per surface: primary model → secondary provider with the same contract → degrade mode (cached answer, retrieval-only, or honest unavailable). Sticky-hash tenants to a primary so A/B and support debugging stay sane, but allow failover when error rates or p95 latency trip a circuit.

• Timeouts and budgets per tier — never let Tier 1 hang until the user feels Tier 3 latency
• Circuit breakers on provider error rate and token cost per minute
• Degrade messaging in the UI when you skip generation entirely
• Idempotent tool execution so a retry after failover does not double-send side effects

Measure routing like you measure product quality

Dashboards that only show average cost hide a bad router. Slice by intent, tier, tenant plan, and cascade depth. Track: % of requests per tier, escalate rate, cost per successful task, task success / eval score by tier, and p95 latency including cascade hops. A cascade that escalates 70% of the time is a frontier model with extra latency — fix the gate or the Tier 1 prompt, do not celebrate the architecture diagram.

Promotion rules worth writing down

• Move an intent down a tier only when eval score and support tickets hold for a fixed cohort window
• Move an intent up a tier when escalate rate or human override rate exceeds a budget
• Never change default tiers without a flag, a control group, and a kill path back to the previous map

Ship a router you can explain on a one-pager

Before you add ML-based routers or multi-hop cascades, write the policy: intents, default tiers, escalate gates, fallbacks, and the metrics that justify a change. Keep the map in versioned config next to your prompt contracts. At Brutanix Studios we build AI products so most traffic stays on the cheapest tier that still passes the job contract — and the expensive model is reserved for the work that actually needs it. That is how you cut cost without teaching users that "AI mode" means roulette.