How to Make LLM Features Reliable

Model providers go down. On June 9, 2026, a routine operating system update on OpenAI’s cloud-hosted GPU servers knocked a large number of nodes off the network. API error rates peaked around 25%, availability dropped to 75%, and full recovery across all models took until the afternoon of the following day [1]. A month later, another incident produced elevated error rates across the API, ChatGPT, and Codex simultaneously [2].

This is not a criticism of any one provider. It is the baseline condition of the dependency you just added to your product. An LLM call is a remote call to a capacity-constrained service that can be slow, throttled, or unavailable, and unlike most of your other dependencies it takes seconds rather than milliseconds even when everything is working.

The good news: the engineering is well understood. Timeouts, retries with backoff and jitter, circuit breakers, fallbacks, and graceful degradation are decades-old distributed systems practice. This guide applies them to LLM features specifically, and covers the failure modes that are unique to model APIs.

It is part of our series on adding AI features to an existing product.

Know your failure modes

LLM calls fail in ways that need different responses. Treating them all as “the API errored” produces either retry storms or unnecessary failures.

Two of these deserve emphasis. 429 and 529 are different problems. A 429 means you exceeded your own limits, so the fix is on your side: respect retry-after, reduce concurrency, and smooth your traffic [3]. A 529 means the provider is saturated across all customers, so retrying harder does not help and failing over does [3]. 4xx errors other than 408 and 429 are not worth retrying at all [4]; they will fail identically every time and just burn your retry budget.

The last row is the sneakiest. A provider can serve responses that are slower or subtly worse without erroring. Only evals and quality telemetry catch that, which is why they belong in production monitoring and not just CI.

Set a latency budget per feature

Decide, before you build, how long each AI interaction is allowed to take, then design to it. The following are practitioner guidelines rather than measured benchmarks; calibrate against your own users.

The levers that matter most: stream tokens so perceived latency is time to first token rather than total time; run independent calls (retrieval, classification, guardrails) in parallel rather than in sequence; use a smaller model on latency-sensitive paths; cache long static prompt prefixes, which cuts cost and time to first token together; and move anything that does not need an immediate answer off the request path entirely.

Measure at the tail, not the mean. p95 and p99 are what users remember, and a median of 1.2 seconds with a p99 of 30 seconds is a feature people stop trusting.

Retry correctly, or make things worse

Retries are the most common source of self-inflicted outages. When a provider is already struggling, a fleet of clients retrying immediately multiplies the load exactly when capacity is lowest. Google’s SRE material is blunt about this: retries can amplify a low error rate into a much higher traffic level and drive cascading failure [5].

The rules that keep retries safe:

• Set explicit timeouts on every model call, both connection and request. Too short, and you trigger a retry storm of your own making [4].
• Use exponential backoff with jitter. The randomization matters as much as the backoff: without it, all your clients line up and retry in sync. Full or decorrelated jitter substantially reduces total client work under contention compared with simple additive randomness [6].
• Honor retry-after when the provider sends it, rather than substituting your own schedule [3].
• Cap retries at two or three attempts for interactive paths. Beyond that the user has already left.
• Bound retries globally, not just per call. A retry token bucket lets normal traffic through while limiting total retry volume during an incident: successes deposit fractional tokens, retries consume whole ones, and when the bucket empties, retries stop [7]. AWS has shipped this behavior in its SDKs since 2016 [4].
• Only retry what is safe. Read-only calls are naturally idempotent. Calls with side effects, such as a copilot that creates records or sends messages, need idempotency keys before they are safe to retry [4].
• Use the provider SDK’s built-in retry logic where it exists, and configure it rather than layering your own on top. Double-layered retries multiply attempts silently.

For agent loops, retry budgets need to be per task, not per step. A fifteen-step agent with three retries per step can generate 45 extra calls from one bad afternoon. See AI agents in the enterprise.

Add a circuit breaker

When a provider is failing consistently, every additional request wastes time and pushes the user’s wait longer. A circuit breaker trips after a threshold of failures, fails fast (or routes to the fallback) for a cooldown period, then lets a trial request through to test recovery.

One refinement worth knowing: a client-side breaker that stops all traffic can make a partial outage worse, because it cuts off capacity that was still working. Breaking only the retries, while letting first-attempt traffic through, avoids that failure mode [7].

Design the fallback chain

Every AI feature should have a defined answer to “what happens when the model call fails.” Ordered from best to worst:

1. Retry on the same provider for transient errors, within the limits above.
2. Fail over to a second provider or platform that has passed the same eval suite. This is the step most teams skip and most need during a real incident. Because the same model families are now available across multiple platforms, the fallback can often be the same model with a different operator: Claude through Bedrock or Google Cloud when the direct API is saturated [3], or an equivalent tier from a different vendor. See Azure OpenAI vs OpenAI API vs AWS Bedrock and choosing an LLM for your product.
3. Degrade to a cheaper or smaller model that clears a lower bar, and label the result accordingly.
4. Serve a non-AI result. Return the underlying search results without the generated summary, show the raw records, or fall back to a template.
5. Fail clearly. Tell the user the AI feature is temporarily unavailable, keep the rest of the product working, and let them do the task manually.

Steps 3 and 4 are graceful degradation, which SRE practice describes as serving a lower-quality but cheaper response rather than failing entirely [8]. For an AI feature, the most important form of this is simple: the product must work without the AI. If a model outage takes down the ticket view, the checkout, or the dashboard, the AI feature was built as a hard dependency and should not have been.

A fallback you have never tested is a hypothesis. Run a game day: disable the primary provider in staging, confirm the chain executes, and check that the user-facing messages make sense.

Control your own load

Much of what looks like provider unreliability is self-inflicted traffic shape.

• Smooth your traffic. Sudden spikes trigger rate limiting; steady, predictable usage does not. Ramp up gradually when launching to a large tenant [3].
• Queue instead of bursting. Put a work queue with bounded concurrency in front of non-interactive workloads.
• Move bulk work to batch APIs, which are built for volume and keep it off your interactive rate limits.
• Cache long static prefixes, which reduces both tokens billed and load generated [3].
• Set per-tenant limits so one heavy customer cannot exhaust shared capacity for everyone else. This doubles as cost control; see LLM API costs in production.
• Shed load deliberately when you are over capacity, rejecting low-priority AI requests early and cheaply rather than letting queues back up [8].

Monitor what actually predicts failure

Standard uptime monitoring misses most LLM problems. Track, per feature and per provider:

• Error rate split by cause: 429, 529, 5xx, timeout, schema validation failure.
• Latency at p50, p95, and p99, measured to first token and to completion.
• Retry rate and fallback rate, which usually move before your error rate does.
• Circuit breaker state changes.
• Cache hit rate, since a silent drop raises both cost and latency.
• Output quality signals from production: schema failures, guardrail rejections, and user behavior such as regenerate and discard. See AI copilot UX patterns.
• Queue depth and per-tenant consumption against limits.

Alert on the leading indicators. A rising fallback rate means the primary provider is degrading; waiting for a user-visible error rate to spike means you learn about it from customers.

Also subscribe to your providers’ status pages and, where possible, pipe those alerts into the same channel as your own monitoring, so an on-call engineer can immediately tell an internal problem from an external one.

What to tell users

Reliability engineering ends at the interface. During an incident, users should see an honest message rather than a spinner that never resolves: that the AI feature is temporarily unavailable, that their work is safe, and what they can do in the meantime. If you degraded rather than failed, say so, so nobody mistakes a fallback result for your best output. And keep the rest of the product fully usable, which is the whole point of not building AI in as a hard dependency.

Reliability checklist

• Every model call has explicit connection and request timeouts.
• Retries use exponential backoff with jitter, honor retry-after, and are capped.
• Non-retryable errors (4xx other than 408 and 429) fail immediately.
• Total retry volume is bounded globally, not just per call.
• Calls with side effects use idempotency keys before they are retried.
• A circuit breaker exists, and it breaks retries rather than all traffic.
• A second provider has passed the same eval suite and is wired as a fallback.
• The product remains usable when every AI feature is unavailable.
• Per-tenant rate limits and bounded queues protect shared capacity.
• Error rate by cause, tail latency, retry rate, and fallback rate are monitored with alerts.
• The fallback chain has been tested in a game day, not just written down.

Keep reading

• How to add AI features to an existing software product (the pillar for this series)
• LLM API costs in production: how to estimate and cut them
• Azure OpenAI vs OpenAI API vs AWS Bedrock for enterprise apps
• AI copilots inside B2B SaaS: UX patterns users actually adopt
• Our enterprise RAG guide

CMW AI Labs builds AI features with the reliability engineering included: latency budgets, retry and fallback chains across providers, and telemetry that catches degradation before customers do. If your AI feature is fragile under load, talk to our team.

Sources

1. OpenAI, “Elevated error rates,” incident write-up, June 2026. https://status.openai.com/incidents/01JXCAW3K3JAE0EP56AEZ7CBG3/write-up
2. Unite.AI, “Global outage hits OpenAI’s ChatGPT, API and Codex,” July 25, 2026. https://www.unite.ai/global-outage-hits-openais-chatgpt-api-and-codex/
3. Anthropic, “Errors,” Claude API documentation. https://docs.anthropic.com/en/api/errors
4. Marc Brooker, “Timeouts, retries, and backoff with jitter,” Amazon Builders’ Library. https://aws.amazon.com/builders-library/timeouts-retries-and-backoff-with-jitter/
5. Google SRE Book, “Production Services Best Practices.” https://sre.google/sre-book/service-best-practices/
6. Marc Brooker, “Exponential Backoff and Jitter,” AWS Architecture Blog. https://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/
7. Marc Brooker, “Fixing retries with token buckets and circuit breakers,” February 2022. https://brooker.co.za/blog/2022/02/28/retries.html
8. Google SRE Book, “Handling Overload.” https://sre.google/sre-book/handling-overload/