What’s the Right Way to Handle Rate Limits in a Multi-Provider LLM Integration?

I’m building a service that can send requests to OpenAI, Claude, and other LLM APIs. I understand that each provider has rate limits, but I’m unsure how to handle them without either hammering the API with retries or making users wait unnecessarily. Should I retry, queue requests, or switch providers when one starts returning 429 errors?

A useful way to think about this is that a rate limit is a signal to slow down, not an invitation to immediately resend the same request. The right response depends on which limit you hit and whether the request can safely be retried.

Track limits separately for each provider and model

Don’t treat “the provider” as a single bucket. Limits may apply to a particular model, project, organization, or API key, and may separately count requests and tokens. A workload can be under its requests-per-minute allowance but still exceed its token-per-minute quota because a few prompts have large contexts.

Keep a separate limiter for each relevant combination, such as provider, account, and model. If the provider returns rate-limit headers, use them to update your view of the remaining capacity, but don’t assume every API exposes the same headers or defines limits in the same way. Leave some headroom; estimates of token usage and concurrent work are rarely perfect.

Retry briefly, with a limit

For a 429 response, check for a Retry-After header and wait at least as long as it requests. If no such header is present, use bounded exponential backoff with a little random jitter. Jitter matters when many workers hit a limit together: without it, they can all wake up and retry at once.

Here’s a small synchronous example for a request function that returns a response object with a status_code and headers. A production worker may instead schedule the next attempt in a queue so it isn’t occupied while waiting.

import random
import time


def send_with_retry(send, maattempts=4):
    # Retry only responses that are commonly temporary; don't retry ordinary 4xx errors.
    retryable = lambda status: status == 429 or 500 <= status <= 599

    for attempt in range(maattempts):
        response = send()

        if not retryable(response.status_code):
            return response

        if attempt == maattempts - 1:
            # Let the caller inspect or handle the final failed response.
            return response

        retry_after = response.headers.get("Retry-After")
        if retry_after is not None:
            # Honor a numeric Retry-After value when the API provides one.
            delay = float(retry_after)
        else:
            # Increase the wait each time and add jitter to stagger concurrent retries.
            delay = min(30, 2 ** attempt) + random.uniform(0, 0.5)

        time.sleep(delay)

    raise RuntimeError("Unreachable")

This is only a starting point: a real implementation should handle malformed headers, timeouts, cancellation, and the provider’s documented behavior. Also, avoid retrying indefinitely. A request that keeps failing should eventually return a clear error or be placed in a dead-letter path for inspection.

Use a queue when traffic is bursty

A queue gives you control over how many requests are in flight and lets you pace work at the rate the account can actually sustain. Apply limits to both request count and estimated token volume. The queue should have a maximum wait time or capacity, too; otherwise a temporary provider slowdown can quietly turn into a long backlog.

Keep the provider-specific queue or scheduler state isolated. A busy model shouldn’t block unrelated requests to another provider that still has capacity. At the same time, a global concurrency limit can protect your own application from launching more work than it can serve.

Make provider fallback a deliberate decision

Sending a request to a fallback provider can help keep a service available, but it isn’t a universal fix for rate limits. The alternate model may have its own quota, different latency, different output behavior, or a separate data-handling policy. Make sure the task is allowed to move providers and that the alternate model is suitable for it.

Finally, distinguish a 429 from an ambiguous timeout. A rate-limited request is generally rejected, but after a network timeout you may not know whether the provider already processed the generation. Retrying could create duplicate generation requests and extra work. Use provider-supported idempotency features where available, or track your own request IDs and decide how duplicates should be handled.

In short: pace requests before they fail, retry transient responses a limited number of times, and treat fallback as an application decision rather than an automatic response to every 429. That combination is usually more dependable than a loop that simply tries again.