Start with one Go function and two model candidates
A useful model comparison starts with a small task you understand, not a request to test an entire repository. Pick a Go function with clear inputs, outputs, and failure conditions. A configuration parser, validation helper, or price calculator makes a manageable first run.
You could compare Claude Sonnet 5.5 and GPT-6.1 Sol as candidates. Treat their names as labels, not evidence that either will produce better tests. Your goal is to discover which output needs less correction in your project.
A good generated test checks the intended behavior, fails when that behavior breaks, and remains readable to your teammates. Confident explanations and long test files do not establish any of those qualities.
Give both models the same Go testing brief
Provide the function, relevant types, module package name, and behavioral requirements. Include existing testing conventions if your repository has them. Without this context, a model may invent constructors, import unavailable libraries, or assume behavior that your application never promised.
For example, imagine a function called ParsePort that accepts a string and returns a port number or an error. Tell both models whether whitespace is allowed, whether zero is valid, and which values are out of range. Otherwise, different tests may reflect different assumptions rather than different model quality.
Use an identical prompt for each candidate:
Write table-driven unit tests for ParsePort using Go's standard testing package. Follow the supplied behavior specification. Cover valid input, boundary values, malformed input, and error cases. Do not change production code or add dependencies. Identify any ambiguity before making assumptions. Return the test file and briefly explain what each group of cases protects.
Check whether the generated tests actually run
Save each candidate's output separately and run go test ./... in the same repository state. If the project has unrelated failing tests, isolate the relevant package and record that limitation. Never silently repair one candidate's output while judging the other exactly as written.
- Compilation: Do imports, function signatures, and package declarations match your code?
- Dependencies: Does the output honor the request to use only the standard library?
- Isolation: Can each test run without network access, shared files, or execution-order assumptions?
- Diagnostics: Would a failure message tell you which input failed and what result was expected?
Track required edits in plain language. “Needed the package name corrected” and “assumed an unavailable helper” are more useful observations than “felt smarter.”
Separate meaningful assertions from empty coverage
A passing test is not automatically a useful test. Inspect what each assertion proves. For valid ports, tests should compare the returned value and check that no error occurred. For invalid input, they should verify the documented error behavior rather than merely print the result.
Look closely at boundaries. If the specification accepts ports from 1 through 65535, useful cases include both endpoints and values just outside them. Empty input, letters, and whitespace deserve attention when the contract defines their behavior. Do not reward extra cases that simply invent requirements.
Then deliberately introduce a small defect in a disposable branch, such as incorrectly rejecting the largest valid port. Run the tests again. A relevant test should fail. Restore the code afterward. This checks whether the assertions can catch a real mistake, not just execute the function.
Choose a model using repair effort and repeatable evidence
Record each candidate's strengths under the same headings: runnable output, behavior coverage, clear assertions, and repair effort. Repeat the exercise on another representative function before choosing a default. A parser alone cannot tell you how well a model handles interfaces or cancellation.
SwiftRelay is a practical way to try model candidates through one relay using an OpenAI-compatible SDK or supported coding tools, with per-token billing. Sign up for SwiftRelay to run your own Go testing comparison. Keep the prompt, repository state, and review checklist consistent so your decision rests on test quality rather than presentation.