Your AI feature needs more than a log line saying the model call failed. It needs enough evidence to distinguish a broken credential, an exhausted budget, a provider rejection, and an answer your application cannot use. This first-lap checklist builds that visibility without turning a small service into an observability project.
1. Define what a failed AI request means
Start at the user-facing boundary. A request fails when your service cannot deliver its promised result, even if the upstream API returns a successful HTTP status. For a document classifier, that includes invalid JSON, missing required fields, and categories outside your allowed list.
Separate transport success from application success. Track connection failures, HTTP errors, response parsing failures, schema validation failures, and explicit model refusals as distinct outcomes. A refusal may be expected behavior rather than an incident; classify it according to your product contract.
2. Put one monitored wrapper around model calls
Route every model request through a shared function that records the outcome and then returns a result or raises a typed error. Keep monitoring outside individual feature handlers so new endpoints inherit the same coverage.
SwiftRelay fits here as a pay-as-you-go relay accessible through an OpenAI-compatible SDK. Set the SDK base URL to https://api.fastapirelay.com/v1 and keep the API key in your secret store. Sign up for SwiftRelay to use the small gift credit for your initial instrumentation checks.
Record the configured model identifier rather than guessing it from response text. If you evaluate Claude Haiku 4.5, for example, verify the supported API identifier before deploying your configuration.
3. Write structured events without customer content
Emit a structured event for each completed call, including failures. Use a consistent field set so you can search across routes and group incidents without reading free-form exception messages.
- Correlation: application request ID, trace ID, and upstream request ID when available.
- Context: environment, deployment version, route name, configured model, and attempt number.
- Outcome: success flag, error category, HTTP status when present, and sanitized provider error code.
- Usage: reported input and output token counts when available; leave them unknown when absent.
Do not log API keys, authorization headers, raw prompts, uploaded documents, or complete model responses. Apply an allowlist to logged fields and sanitize exception messages, which can contain request content. Keep request IDs out of metric labels to avoid unbounded cardinality.
4. Classify failures before deciding to retry
Map authentication and permission failures to configuration checks. Treat invalid request parameters as application defects. Route quota and billing-related errors to the person responsible for account limits. Distinguish these from temporary connection failures and upstream server errors.
Retry only failures that your integration considers transient. Use a capped attempt count, exponential backoff with jitter, and Retry-After guidance when supplied. Check the SDK's built-in retry behavior first so your wrapper does not multiply attempts unexpectedly.
A timeout does not prove the upstream request was never processed. Retries can consume additional tokens, and repeated tool execution can duplicate side effects. Design tool actions with idempotency safeguards rather than assuming model requests are safe to repeat.
5. Build a dashboard around user-visible failures
Start with request volume, terminal failure rate, failure categories, retry attempts, validation failures, and reported token usage. Break them down by environment, route, model, and deployment version using bounded labels.
Count attempts separately from logical requests. A request that succeeds after a retry should appear as one successful user request plus an additional attempt. Otherwise, retries can make your service look busier and less reliable than users experience.
6. Alert on actionable patterns
Page someone for sustained user-visible failures, not every isolated exception. Pair failure-rate alerts with a minimum request count so a quiet service does not generate misleading alarms. Send configuration failures and unusual token consumption to an owner who can act.
Include the affected route, error category, deployment version, and a saved log query in each alert. Add a short runbook: inspect recent changes, verify credentials and limits, check upstream status, then choose rollback, temporary feature disablement, or a validated fallback.
7. Test the monitoring before real traffic
Use mocked upstream responses to exercise rejected credentials, rate limits, server errors, malformed output, and connection failures. Confirm that sensitive content stays out of logs, retries stop, and terminal failures reach the dashboard. Trigger a test alert and follow its runbook. Your first lap is complete when a teammate can identify the failure and choose a safe next action.