A batch of model requests should not become a burst of uncontrolled connections. Put a queue between incoming work and your API client, then let a fixed number of workers drain it. This sprint builds a practical starting point for applications using SwiftRelay, with explicit rules for admission, streaming, retries, and shutdown.
1. Set a local concurrency budget before sending requests
Concurrency is the number of requests currently in flight. It is different from requests per minute and tokens per minute. A concurrency cap controls overlapping work; it does not guarantee that you stay within every rate limit.
Start with a conservative local cap, such as four active requests, and make it configurable. Treat that number as your application setting, not a published SwiftRelay allowance. Check the limits applicable to your account and selected model before increasing it.
2. Put a bounded queue ahead of the API client
Every incoming job should enter the same admission path. Store a job identifier, model, input, creation time, deadline, and retry count. Avoid placing credentials inside job records.
Give the queue a maximum size. An unlimited queue only moves overload into memory and makes old work accumulate. For a first implementation, use a first-in, first-out queue with a clear rule when it fills:
- Interactive work: reject excess jobs with a clear busy response.
- Background work: pause producers or persist jobs in a durable queue.
- Expired work: remove jobs whose deadlines have passed before dispatch.
3. Give each worker exactly one active request
Create as many workers as your concurrency budget allows. Each worker takes one queued job, sends its request, records the result, and only then takes another job. Alternatively, use a semaphore that every outbound call must acquire.
SwiftRelay fits naturally behind this worker layer because it works with OpenAI-compatible SDKs. Configure the SDK base URL as https://api.fastapirelay.com/v1 and load your API key from a secret store or environment variable. Sign up for SwiftRelay to test this queue with its small new-account gift credit.
Keep queue management separate from model selection. For example, a job targeting Claude Sonnet 5.5 should pass through the same admission and cleanup logic as other jobs.
4. Hold the slot until streaming actually finishes
A streaming request is still active after the response headers arrive. Release its slot only when the stream finishes, fails, or is cancelled and the underlying connection is closed. Releasing it when the stream starts can silently exceed your intended cap.
Always release capacity through guaranteed cleanup. Parsing errors, disconnects, and cancellation must all reach that cleanup path. When a user abandons a request, propagate cancellation to the API client rather than merely hiding the result.
5. Schedule retries without creating another burst
Classify failures before retrying. Authentication and invalid-input errors need correction, not repetition. Rate-limit responses and some temporary server failures may be retryable; honor retry guidance returned by the service when available.
Use capped exponential backoff with random jitter and a maximum attempt count. Put delayed retries into a scheduled queue rather than holding a worker slot while sleeping. Each retry must reacquire capacity through the same worker pool.
A timeout does not prove that the remote request never ran. Retrying can duplicate generation and token charges. Retry interrupted streams cautiously, and do not automatically replay tool actions or other side effects.
6. Share the budget across application instances
Four workers in each of three processes permit twelve active calls, not four. If your budget must apply across the deployment, use a shared queue with a controlled worker count or a distributed semaphore with leases and crash recovery.
When several users share the queue, add per-user admission limits so one large batch cannot occupy every waiting position.
7. Check queue health before raising the cap
Track queue depth, queue wait, active calls, rejected jobs, retry counts, cancellations, and error categories. During shutdown, stop accepting new jobs, preserve queued work where required, and allow active requests to finish within a configured deadline.
Raise concurrency gradually only after checking applicable limits and observed errors. A successful first lap means bounded waiting work, predictable admission, and no leaked request slots—not simply more simultaneous calls.