Count active requests before starting another
A concurrency limit controls how many requests can be active at the same time. Think of a checkout with a fixed number of open lanes: customers can wait, but only those at a register are being served. For an AI application, each active model request occupies a lane until it finishes, fails, or is canceled.
This is different from a rate limit, which restricts how many requests or tokens you can use within a time window. You can stay within a rate limit and still exceed a concurrency limit by launching too many requests together. A queue helps manage the overlap, but does not replace rate limiting.
Put pending work in a bounded queue
Instead of sending every request immediately, place new work in a queue. A small set of workers takes jobs from that queue. Each worker handles one request at a time, then picks up another when its current request ends.
For a first implementation, choose a modest local cap, such as three active requests. That is an application setting, not a claim about any provider’s allowance. Check the limits that apply to your account and model, and keep your cap within them.
- Pending: The job is waiting and has not started a model request.
- Active: A worker has started the request and holds a concurrency slot.
- Finished: The job has succeeded, failed permanently, or been canceled.
Give the queue a maximum size too. An unlimited queue can consume memory and leave users waiting for work that is no longer useful. When the queue fills, reject additional work clearly or ask the caller to try again later.
Keep each slot until the request really ends
The basic worker loop is straightforward: take a pending job, acquire a slot, send the request, record the outcome, and release the slot. Release it in cleanup logic that runs on both success and failure. Otherwise, one exception can leave your application believing a slot is occupied forever.
Streaming needs extra care. Receiving response headers does not mean the request is finished. Keep the slot until the stream ends or is explicitly closed. If the user cancels, propagate that cancellation to the underlying request rather than merely hiding the output.
A cap inside one application process only controls that process. If you run several server instances, their active requests add together. Use shared coordination when you need one limit across the whole application.
Retry rejected requests without creating a second pileup
A queue cannot prevent every rejection. Other clients may share your account, and upstream capacity can change. Inspect the response rather than treating every error as a concurrency problem. A 429 response can indicate a rate or capacity restriction; invalid credentials and malformed requests need a fix, not repeated attempts.
- Honor a Retry-After instruction when the service provides one.
- Otherwise, use exponential backoff with jitter: increase the wait between attempts and add randomness.
- Set a maximum attempt count and an overall job deadline.
- Make delayed retries acquire a slot again before sending another request.
Keep delayed retries outside the active worker slots where practical, so sleeping jobs do not block ready work. Track attempts on the original job instead of creating unlimited copies. A timeout does not prove that upstream processing stopped, so retrying may duplicate work and token charges.
Connect SwiftRelay behind the same queue
SwiftRelay is a practical option for routing model requests through an OpenAI-compatible SDK with pay-as-you-go, per-token billing. Set the SDK base URL to https://api.fastapirelay.com/v1, and keep your queue in front of the SDK calls. The relay does not remove the need for application-level concurrency control.
Sign up for SwiftRelay to try this setup with the small gift credit available to new accounts. Keep API credentials on your server, not in browser code.
Check queue health before raising the cap
Track active requests, queue depth, waiting time, rejections, retries, and canceled jobs. Start conservatively and adjust using observed behavior and documented limits. The goal is controlled admission: accept work you can handle, keep waiting work visible, and stop retries from becoming another overload.