A checkout test fails in continuous integration, passes on rerun, and disappears into the backlog. Giving a coding agent permission to “fix the flake” can make things worse: it might add sleeps, weaken assertions, or hide the failure behind retries. Give it a bounded investigation instead. This routine turns an intermittent failure into a reproducible experiment, a small patch, and an evidence-backed review.
1. Give the Agent a Narrow Failure Contract
Start with one failing test and one expected behavior. Identify the repository, test command, failing CI job, and relevant commit. State what the agent may change and what it must preserve.
Suggested instruction: “Investigate this intermittent checkout failure. Preserve the payment-success assertion. Do not skip the test, add fixed sleeps, increase global timeouts, or change retry settings. First report your reproduction plan.” Require approval before changes to application behavior or shared test infrastructure.
2. Capture the Failure Before Editing Anything
Ask the agent to assemble an evidence bundle: assertion output, browser trace, screenshot, console errors, relevant network responses, and test-run metadata. Record the browser, operating system, environment variables, worker count, retry configuration, and commit identifier. Redact credentials and customer data before sending artifacts to a model.
Have the agent distinguish observations from guesses. “The confirmation request returned an error” is evidence. “The server was overloaded” is a hypothesis until logs support it. If artifacts are missing, improve failure capture before attempting a repair.
3. Build a Reproduction Matrix Without Retries
Disable retries for the diagnostic runs so each failure remains visible. Repeat the target test with a fixed run count, then compare isolated execution with its normal suite context. Keep other settings unchanged wherever possible.
- Run the test alone with a single worker.
- Run it with the usual parallel worker configuration.
- Run the surrounding tests in their normal order.
- Compare a fresh account and clean data with reused fixtures.
Record outcomes for each condition. If the failure does not reproduce, preserve that uncertainty rather than letting the agent claim a confirmed cause.
4. Rank Hypotheses Against the Collected Evidence
Require a short hypothesis table in the agent’s investigation notes: suspected cause, supporting evidence, and an experiment that could disprove it. Useful candidates include shared accounts, duplicate records, ambiguous locators, unfinished background work, and assumptions about time zones or generated dates.
Change one variable per experiment. If parallel execution fails but isolated execution does not, inspect shared resources before touching browser waits. If the wrong element receives a click, inspect locator matches before blaming the network. Let evidence choose the next action.
5. Replace the Missing Contract, Not the Assertion
Ask for the smallest patch that addresses the demonstrated cause. For synchronization problems, wait for the state the user actually needs: an enabled control, a completed operation, or a visible confirmation. Avoid replacing one arbitrary delay with another.
For isolation problems, give each test ownership of its records and cleanup. For ambiguous selectors, target an accessible role and name or an explicit test identifier. Preserve the original business assertion. If the application has a real defect, report it rather than teaching the test to accept incorrect behavior.
6. Verify the Patch Against the Same Conditions
Repeat the original reproduction matrix with the patch applied. Keep the environment and run counts comparable, then run the affected neighboring tests. Check that cleanup still executes after failures and that the patch has not introduced dependencies on execution order.
Ask a separate review pass to inspect the diff for weakened assertions, hidden retries, broad configuration changes, and accidental production edits. Report observed passes and failures; do not describe a finite clean run as proof that flakiness is impossible.
7. Package the Evidence and Keep Agent Access Portable
Finish the pull request with the failure signature, reproduction command, supported root cause, patch rationale, and before-and-after results. Include remaining uncertainty and sanitized artifact references so another developer can continue the investigation.
If you use Claude Code, Codex, or Gemini CLI for this routine, SwiftRelay is a pay-as-you-go relay that fits those workflows. Keep the same investigation contract when changing models, and review tool permissions separately from model access. Sign up for SwiftRelay to try the routine with a small gift credit, and keep your first task scoped to one flaky test.