A successful response can hide several paid attempts. Your SDK retries a connection failure, your worker repeats the job, and your application finally returns one answer. The user sees one result; your token ledger may show several generations. This pit stop gives you a practical sequence for finding repeated work and putting a spending boundary around it.
1. Map every layer that can repeat a request
Start at the user action and follow the request through your application, job queue, SDK, relay, and model provider. Check which layers can retry and what triggers them. Do not assume that setting one retry option controls the entire chain.
Record whether each setting means total attempts or additional retries. Three application attempts combined with three SDK attempts can produce nine upstream attempts, even though neither setting looks especially aggressive. Inspect defaults as well as explicit configuration.
2. Give each user action a traceable attempt trail
Create one logical operation identifier for the user action and a separate attempt identifier for each outbound call. Preserve the operation identifier when a queue redelivers work. Capture upstream request identifiers when available, without assuming they provide deduplication.
- Operation record: model, prompt version, input size, and spending allowance.
- Attempt record: retry owner, trigger, response status, and whether output began.
- Usage record: reported input tokens, output tokens, and any available billing details.
Keep credentials and sensitive prompt content out of these logs. The goal is to connect repeated attempts, not create another store of private conversations.
3. Separate rejected requests from uncertain outcomes
A timeout means your client stopped waiting. It does not prove that generation stopped upstream. Likewise, a broken stream may leave you with partial output while the provider has already performed billable work. Failure status alone is not a reliable billing signal.
Classify errors using the actual API contract. Authentication failures and invalid inputs generally need correction, not repetition. Rate-limit responses and temporary service failures may justify bounded retries. For connection losses after submission, mark the outcome as uncertain and check available request records before replaying expensive work.
4. Put one component in charge of retries
Choose a single retry owner wherever practical. If your application owns retry policy, inspect whether the SDK also retries and disable that behavior where supported. Account separately for queue redelivery and any documented intermediary retry behavior.
Use bounded exponential backoff with jitter for retryable failures, and honor server retry guidance when supplied. Backoff reduces repeated pressure, but it does not prevent repeated token consumption. You still need a total attempt limit and an operation-level spending boundary.
5. Reserve a token allowance before each replay
Estimate the next attempt using the complete input you will resend, including conversation history and tool results, plus a bounded output allowance. Check the model's current pricing rather than treating every token category as equivalent.
For a workflow using Claude Sonnet 5.5, repeating the full conversation can replay substantial input even when the requested answer is short. Reserve estimated cost before dispatch, then reconcile against reported usage. Treat missing usage as unresolved exposure rather than zero cost. If the remaining allowance cannot cover another attempt, stop and surface a recoverable failure.
6. Protect partial answers and tool side effects
Do not automatically restart an entire generation after delivering partial output. Decide whether to preserve it, offer an explicit retry, or issue a carefully scoped continuation. A continuation is another request, not a guaranteed resumption of the original generation.
Apply separate safeguards to tool execution. Repeating a model call must not repeat a payment or create another ticket. Use application-level deduplication and tool-specific idempotency where supported. Never assume a model endpoint honors an idempotency key unless its documentation explicitly says so.
7. Reconcile retry spend before the next release
SwiftRelay fits this workflow when you want pay-as-you-go, per-token access through a relay. For an OpenAI-compatible SDK, use https://api.fastapirelay.com/v1 and inspect that SDK's retry configuration. Sign up for SwiftRelay to test your retry policy with the small gift credit provided to new accounts.
Before release, compare token usage per logical operation with usage per attempt. Investigate successful operations with multiple generations, missing usage records, or repeated full-context submissions. Your finish line is a bounded, observable recovery path—not retries disabled everywhere.