I’m building an app that can send requests to OpenAI, Claude, Grok, and DeepSeek. What’s a sensible way to handle rate limits and temporary API failures across providers without either spamming retries or making users wait through a long chain of failed calls?
The tricky part is that “retry the request” sounds like one simple rule, but a rate limit, a server error, and a timeout are different problems. Provider APIs also vary in their error details and headers, so it helps to keep their raw responses at the integration boundary and translate them into a small set of internal failure types.
Start by classifying the failure
A useful first distinction is between errors that may clear on their own and errors that need a change to the request:
- Rate limited: A 429 response usually means the provider is temporarily refusing requests. If the response includes a retry delay, respect it. Otherwise, use a bounded backoff.
- Temporary server failure: A 5xx response may be transient, but repeated retries can add load and increase latency. Retry only a small number of times.
- Timeout or connection loss: The request may have reached the provider even if your application never received the response. Treat this differently from a confirmed rejection.
- Invalid request or authentication error: Retrying the same payload generally won’t help. Return a useful error or fix the configuration instead.
Normalize these outcomes into provider-neutral categories such as rate_limited, temporary_failure, timeout, and permanent_failure. Keep the provider name, status code, and request ID in logs, too; those details are often necessary when investigating an incident or contacting support.
Use bounded backoff, not a tight retry loop
For retryable failures, use exponential backoff with jitter: wait longer after each failed attempt, then add a small random amount so that many clients don’t all retry at the same moment. Set a maximum delay and a maximum number of attempts. A few carefully spaced attempts are usually more helpful than a long queue of retries that the user can’t see or control.
When a provider supplies a Retry-After value or equivalent retry guidance, treat it as a meaningful signal. Don’t retry earlier just because your local backoff timer is shorter. If that delay exceeds the time your product is willing to wait, report that the request is temporarily unavailable rather than keeping the user’s request open indefinitely.
Be careful with timeouts and duplicate work
A timeout does not prove that the provider never processed the request. The model may have finished generating while the response was lost on the way back. For a simple text-generation request, retrying can therefore produce a second billable generation or a different answer. Where an API supports idempotency keys, use them for requests that might be repeated. Otherwise, make the retry policy aware that an uncertain timeout is not the same as a definite 429.
Streaming needs its own policy. If a stream fails before any content arrives, a retry may be reasonable. If the user has already received part of the answer, silently restarting can show duplicated or contradictory text. Decide whether to stop and show a partial-response message, or explicitly restart the answer with clear UI feedback. Avoid combining the first partial output and a second generation as if they were one continuous response.
Keep provider limits separate
Don’t assume one global request-per-minute limit applies to every model or provider. Limits may be scoped by model, account, organization, or token usage, and a provider can impose separate constraints on concurrent requests. Track provider-specific rate limits and concurrency in your adapter or request scheduler. That makes it easier to pause traffic to one constrained service without unnecessarily blocking requests to another.
It’s also worth distinguishing a retry from a fallback. A retry asks the same provider to handle the same request again; a fallback sends it somewhere else. Fallback can be useful for availability, but it may change model behavior, latency, data handling, and cost. Make that choice explicit in your product and logs rather than treating every 429 as permission to send the prompt to another company.
Make retries visible and measurable
Set an overall deadline for the user’s request, not just a timeout for each individual attempt. Record how many attempts were made, which failure triggered each one, how long the waits lasted, and whether the request ultimately succeeded. These metrics reveal whether a retry policy is recovering from occasional blips or hiding a provider problem behind rising latency and cost.
A practical default is to retry only clearly temporary failures, respect provider retry guidance, cap attempts within an overall deadline, and avoid automatic replays after uncertain outcomes unless you can make them safe. The exact limits depend on whether the request is interactive, part of a background job, or costly enough that a duplicate generation matters. I’d be interested to hear what retry limits have worked for other teams running multiple LLM APIs.
