Start the lap with one unreliable browser test
A flaky end-to-end test passes sometimes and fails sometimes without an intentional change to the application. Because it exercises a real user journey through a browser, the cause can sit anywhere along that journey: page rendering, network requests, shared accounts, or leftover data.
A coding agent can inspect files, run tests, and propose changes. Give it a narrow investigation rather than asking it to “fix the test suite.” Start with one failing test, its failure message, and the command used by your continuous integration system, usually called CI.
Give the agent a small investigation contract
Before allowing edits, define what the agent may change and what evidence it must collect. This prevents an apparently successful patch from quietly weakening the test.
Try this instruction: Investigate the checkout confirmation test. Read the repository instructions first. Reproduce the failure before editing. Inspect the failure artifacts and relevant application code. Propose the smallest evidence-backed fix. Do not remove assertions, add unconditional delays, increase retries, or change production behavior without approval.
Run the agent in an isolated development environment. Use test accounts and nonproduction credentials, and require approval before destructive commands or changes outside the agreed scope.
Capture the failure before changing the test
Ask the agent to record the exact command, browser, relevant configuration, and observed failure. For Playwright, a targeted command might be npx playwright test tests/checkout.spec.ts. Match the CI browser project and configuration when reproducing a CI-only failure.
Collect a trace, screenshot, browser console output, and relevant request failures when available. A trace is a recording of test actions and page state that helps explain what happened before an assertion failed. Treat these artifacts as potentially sensitive; redact tokens and customer information before sharing them.
If the failure does not reproduce, the agent should report that limitation. It should not invent a cause from a single error message.
Turn the failing assertion into a testable hypothesis
Have the agent explain what the assertion expected, what actually appeared, and which observation would distinguish likely causes.
- Readiness: Did the test click before the intended control became usable, or check the page before the expected result appeared?
- Isolation: Did another test modify the same account, order, or database record?
- Selection: Did the locator match multiple buttons or depend on changing page structure?
- Dependencies: Did an external service fail, or did the application return an unexpected response?
For example, “the confirmation message was missing” is an observation. “The assertion runs before checkout completes” is a hypothesis. The trace and network activity should support that explanation before the agent edits anything.
Patch the cause without relaxing the promise
If readiness is the problem, wait for the user-visible result with an assertion that automatically checks again until its timeout, rather than inserting a fixed sleep. In Playwright, checking that a confirmation message becomes visible usually expresses the requirement better than pausing for an arbitrary interval.
If data is shared, create unique test records and clean up only records the test owns. If a locator is ambiguous, use an accessible role and name, or an agreed test identifier. Keep the original user outcome intact.
A genuine application race may require an application fix. Ask the agent to flag that separately for review instead of disguising it with more permissive test expectations.
Verify the patch and leave an investigation note
Run the targeted test repeatedly, then run its neighboring tests together to check for interference. Where relevant, repeat with CI-like parallel execution. Passing runs increase confidence; they do not prove that flakiness is impossible.
Request a handoff containing the evidence, root-cause hypothesis, changed files, commands run, results, and remaining uncertainty. Keep retries as diagnostic information, not permission to ignore failures.
If you use Claude Code, Codex, or Gemini CLI for this routine, SwiftRelay provides pay-as-you-go model access with per-token billing. Sign up for SwiftRelay to try the workflow with a small gift credit, while keeping the same evidence-first investigation checklist.