A concurrency bug appears when overlapping tasks interact in an unsafe way. The same code may work repeatedly, then fail when operations happen in a different order. Reasoning models can help investigate these failures, but a convincing explanation is not proof. Your first comparison should test whether each model can show exactly what goes wrong and propose a repair that survives scrutiny.
Build a small concurrency bug test set
Start with three short examples whose behavior you understand. Keep unrelated application code out of the first run. You want to evaluate reasoning about shared state, not the ability to navigate a large repository.
- A lost update: two workers read the same counter value, calculate an increment, and overwrite each other’s result.
- A deadlock: two tasks acquire the same pair of locks in opposite orders and can end up waiting forever.
- A safe control: a shared-state operation protected correctly by a lock, included to catch models that invent bugs.
Choose a language you can verify locally. State its relevant concurrency rules, especially whether an operation is atomic or whether a lock protects every access. For the deadlock example, specify that acquiring a held lock waits. Ambiguous assumptions make model answers difficult to compare fairly.
Give each model the same investigation prompt
Use identical code, context, and instructions for every candidate. For example, you could compare Claude Sonnet 5.5 and Gemini 3.1 Pro without assuming either will perform better. Start a fresh conversation for each case so earlier answers do not reveal the expected diagnosis.
A useful prompt is: Inspect this code for concurrency bugs. State your assumptions. If a bug exists, show a concrete sequence of operations that triggers it, identify the rule being violated, propose a minimal fix, and describe a regression test. If the code is safe under the stated assumptions, explain why.
Keep tool access consistent. If one model can run tests while another only reads code, you are comparing different workflows. Record model settings and any requested reasoning effort so you can reproduce the setup.
Check the thread schedule, not the confidence
A thread schedule is simply an ordered list of what each worker does. It turns a vague warning into something you can inspect.
For a counter starting at zero, a valid lost-update schedule is: worker A reads zero; worker B reads zero; A writes one; B writes one. Two increments have occurred, but the final value is one. An answer that only says “there is a race condition” has not demonstrated this failure.
For deadlocks, require the same precision: A holds the first lock, B holds the second, and each requests the lock held by the other. Reject explanations that rely on impossible execution steps or ignore locks already present.
Score repairs and tests separately
A correct diagnosis does not guarantee a correct patch. A model might protect the write but leave the read exposed, or fix one lock-order problem while introducing another.
- Diagnosis: does it identify a real bug without falsely flagging the safe control?
- Evidence: does its operation sequence actually produce the claimed failure?
- Repair: does the patch protect the complete operation and preserve intended behavior?
- Verification: does the proposed test target the failure rather than merely exercise the code?
Prefer tests that deliberately coordinate workers at the dangerous point. Repeating a test many times can reveal a problem, but a passing stress test does not prove the absence of one. Review the synchronization argument as well.
Run your first model comparison through SwiftRelay
SwiftRelay is a practical option for trying both candidates through an OpenAI-compatible SDK with per-token billing. Sign up for SwiftRelay to use the small gift credit for an initial comparison, then track token usage alongside your review notes.
Choose based on demonstrated correctness, clear assumptions, and repair quality across your cases. Treat the result as evidence about your workload, not a universal ranking. Keep human review and executable tests between any suggested patch and production.