Latency Budgets for AI Features: Design for Time, Not Just Tokens

Token quality gets the roadmap debates. Latency gets the churn. Users abandon copilots that stall, hang up on voice agents that pause too long, and stop trusting dashboards that spin while a model "thinks." If you only optimize prompts and evals, you ship a smart product that feels broken. Latency is a product requirement — write it down, measure it, and design the experience around the budget you can actually hit.

Write the budget before you pick the model

Start from the interaction, not the provider price sheet. A side-panel rewrite can tolerate a few seconds if it streams. A voice turn cannot. An autocomplete suggestion that arrives after the user has moved on is wasted spend. Capture three numbers per surface: time to first useful token (or first audible syllable), time to complete answer, and a hard abort ceiling after which you cancel, degrade, or escalate.

• Chat / rewrite: first token under ~1s when possible; full answer paced by streaming
• Inline autocomplete: useless after ~300–500ms — cache and smaller models win here
• Tool-using agents: show named progress within 1–2s even if tools take longer
• Voice: aim for sub-second turn-taking feel; silence longer than ~800ms needs filler or barge-in design

Spend latency where users feel it least

Not every millisecond is equal. Users forgive work they can see and hate work they cannot interrupt. Stream tokens early so the UI proves progress. Prefetch retrieval for likely intents. Parallelize independent tools instead of serializing a "planner then worker" chain that looks idle. Move expensive ranking offline when the corpus is stable. Keep the critical path short: auth, retrieve, generate, validate — and cut anything that does not change the next visible frame.

Streaming is not optional decoration

A 4-second full-block response feels slower than an 8-second streamed one with the same content. Reserve layout space, stream into it, and expose cancel. For structured outputs, stream a partial object only when your parser can show safe intermediate UI — otherwise stream a short status narrative while the structured payload finishes server-side.

Account for tools, retrieval, and the network

Model generation is often not the bottleneck. RAG round-trips, cold vector indexes, third-party APIs, and JSON schema retries inflate p95. Instrument each stage with spans: queue wait, retrieval, prompt build, model TTFT, tool calls, validation. When p95 slips, you should know which span owns the regression — not guess from a single end-to-end timer.

• Cache embeddings and frequent retrieval hits with explicit TTL and tenant keys
• Cap tool fan-out and set per-tool timeouts lower than the user-facing abort
• Fail fast on schema retries — two retries max, then a recoverable error UI
• Pin warm paths for peak hours: keep a small pool ready instead of cold-starting every request

“If your only latency metric is "model response time," you are blind to the half of the wait that lives in your own stack.”

Design fallbacks that protect the budget

A latency budget without a fallback is a hope. Define what happens when you blow it: switch to a faster model for draft quality, return a cached prior answer with a freshness label, skip optional tools, or hand off to a human with context. For voice, prepare short bridging audio and a graceful "let me connect you" path. For chat, prefer a partial useful answer over a perfect late one. Tell the user what changed — silent downgrades erode trust.

Gate releases on latency the way you gate on quality

Add latency checks next to eval scores in CI and canaries: synthetic cases for TTFT and end-to-end, plus production SLOs with alerts on p95 and error-budget burn. A prompt change that improves accuracy by 2% and doubles wait time is not a win. At Brutanix Studios we treat AI latency as part of the product contract — budgets on the page, spans in the traces, and UX that shows progress, offers cancel, and degrades on purpose when time runs out.