How to Give an AI Coding Agent a Reproducible Test Case

Summary

• A reproducible test case is a small, self-contained package that lets an AI coding agent reliably observe the same failure you see.
• Start by defining the exact expected vs actual behavior, then reduce the scenario to the smallest input and environment that still fails.
• Include deterministic setup steps (versions, seed, fixtures) and a single command the agent can run to reproduce.
• Provide a clear output structure: files, commands, observed logs, and acceptance checks the agent must satisfy.
• Review your test case for hidden dependencies, nondeterminism, and missing artifacts before handing it to the agent.

If an AI coding agent cannot reproduce your bug, it will guess. A reproducible test case removes guesswork by giving the agent a minimal environment, exact inputs, and a single run command that reliably produces the failure. This article shows how to turn your “it breaks” report into a concrete deliverable an agent can execute, debug, and fix.

What “reproducible test case” means for an AI coding agent

A reproducible test case is not a long description of what happened. It is a runnable artifact (or set of artifacts) that produces the same failing behavior on demand.

For an AI coding agent, a good test case has three properties:

• Runnable: There is a specific command (or short sequence) to run.
• Deterministic: The failure happens consistently, not “sometimes.”
• Minimal: It includes only what is needed to trigger the bug.

The deliverable you should hand to the agent (usable output structure)

Use this structure as the “package” you paste into the agent or commit to a branch. It is designed so the agent can act without asking basic follow-ups.

Repro Case Template (copy/paste)

• Title: One-line description of the failure
• Goal: What should happen (expected behavior)
• Actual: What happens instead (include exact error text)
• Repro command: One command (or a numbered list of 2–4 commands)
• Environment:
  • OS:
  • Language/runtime version:
  • Package manager + version (if relevant):
  • Key dependencies + versions (only those involved):
• Inputs: Exact input data (inline or as files)
• Observed output: Logs/stack trace (verbatim)
• Expected output: What the command should print/return/do
• Acceptance checks: Bullet list of checks that prove the fix works
• Constraints: Anything the fix must not change (API, performance budget, behavior)

This template is intentionally explicit. It forces you to provide the pieces an agent needs to reproduce, diagnose, and verify a fix.

Step-by-step: how to create a reproducible test case

1) Write the “expected vs actual” in one screen

Before you touch code, write two short statements:

• Expected: “When I do X with input Y, the system should do Z.”
• Actual: “Instead, it does W (error: …).”

Keep it concrete. Avoid interpretations like “it fails to parse correctly” without showing the incorrect output.

2) Identify the smallest trigger

Reduce the scenario until removing any remaining piece makes the bug disappear. Common reduction moves:

• Replace large inputs with the smallest input that still fails.
• Remove unrelated modules, routes, UI, or background jobs.
• Replace external services with a stubbed response or a local fixture.
• Move from “full app run” to a single function call or unit test when possible.

The goal is not elegance; it is to isolate the failure mechanism.

3) Freeze the environment details that matter

AI agents can reason about code, but they cannot infer your runtime. Provide the versions that influence behavior:

• Runtime (Node/Python/JVM/.NET) version
• Dependency versions involved in the failing path
• OS details if the bug is platform-specific (filesystem, line endings, case sensitivity)

If you cannot provide exact versions, provide the closest you can and note uncertainty. The agent can still proceed, but it may need to reproduce in a controlled container or lockfile-based setup.

4) Make the repro runnable with one command

Give the agent a single command that reproduces the failure. If you need multiple commands, keep them short and ordered.

• Good: “npm test -- test/repro.spec.js”
• Good: “python -m pytest -k repro_case”
• Good: “go test ./... -run TestRepro”

If the repro requires setup (migrations, seed data), include it as explicit steps and keep it minimal.

5) Capture the failure output verbatim

Paste the exact stack trace, error message, and any relevant logs. Do not paraphrase. Agents use exact strings to locate code paths and confirm they are seeing the same failure.

6) Add acceptance checks (how the agent knows it is done)

Acceptance checks are the “definition of done” for the fix. They should be testable and specific.

• The repro command exits with code 0.
• The output contains a specific expected value.
• A unit test is added that fails before the fix and passes after.
• No changes to public API signatures (if required).

Invented example: turning a bug report into a reproducible test case

Invented example (for illustration): A JSON parsing function incorrectly rejects valid Unicode escape sequences.

Raw bug report (too vague)

“Our parser fails on some JSON from a customer. It works locally sometimes but fails in CI.”

Repro Case (agent-ready)

• Title: JSON parser rejects valid Unicode escape in string
• Goal: Parsing the input JSON should succeed and return the expected string value.
• Actual: Parser throws InvalidEscapeError: unsupported escape sequence when encountering \u2028.
• Repro command: python -m pytest -k test_repro_unicode_escape
• Environment:
  • OS: Ubuntu (CI), macOS (local)
  • Python: 3.11.x
  • Dependency: internal parser module in src/parser.py
• Inputs: Save as fixtures/repro.json:
  • {"message":"line-sep:\u2028end"}
• Observed output: Stack trace excerpt:
  • InvalidEscapeError: unsupported escape sequence at position 20
• Expected output: Parsed object where obj["message"] equals "line-sep:\u2028end" (actual line separator character in the string).
• Acceptance checks:
  • The test test_repro_unicode_escape passes.
  • No regression in existing parser tests.
• Constraints: Do not change the public parse function signature.

Invented example test file

Invented example (for illustration):

# test/test_repro_unicode_escape.py
from src.parser import parse_json

def test_repro_unicode_escape():
    data = '{"message":"line-sep:\\u2028end"}'
    obj = parse_json(data)
    assert obj["message"].endswith("end")
    assert "line-sep:" in obj["message"]

Why this works: it gives the agent a single failing test, the exact input, and a clear expected behavior. The agent can now focus on the parser logic instead of reconstructing the scenario.

What to include (and what to avoid) when the agent needs a “minimal repo”

Sometimes a single test file is not enough. If the bug depends on build tooling, bundling, or runtime configuration, provide a minimal repository layout.

Minimal repo checklist

• README_REPRO.md with the repro command and steps
• Lockfile (if your ecosystem uses one) so dependency resolution is consistent
• One failing test or a small script that exits non-zero on failure
• Fixtures (input files) stored locally in the repo
• Config only if it affects the failing path (e.g., a single config file)

Avoid these pitfalls

• Hidden dependencies: “Works on my machine” because you have a local service running, a file in your home directory, or an environment variable set.
• Time-based behavior: Tests that depend on the current time without freezing it.
• Network calls: External APIs that can change or rate-limit. Prefer fixtures or stubs.
• Randomness: If randomness is involved, set a seed and record it.

Review checks before you send the test case to an AI coding agent

Run these checks yourself. They reduce back-and-forth and help the agent converge faster.

• Fresh clone check: Can you reproduce after deleting build artifacts and reinstalling dependencies?
• Single-command check: Is there one command that triggers the failure?
• Minimality check: Can you remove any file, dependency, or step without losing the failure? If yes, remove it.
• Determinism check: Does it fail 3 times in a row? If not, identify the nondeterministic input (time, concurrency, network, randomness).
• Clarity check: Would a new teammate understand expected vs actual without asking you questions?

How to phrase the handoff message to the agent (so it produces the right deliverable)

Once you have the repro case, your message to the agent should request a specific output: a fix plus proof.

Agent handoff prompt (copy/paste)

• Task: Fix the bug shown by the repro case below.
• Deliverables:
  • A code change that makes the repro command pass
  • A brief explanation of root cause
  • Any additional tests you add (keep them minimal)
• Do not: Refactor unrelated code or change public APIs unless required.
• Repro case: (paste the template-filled content)

This keeps the agent focused on the failing behavior and the verification path.

Frequently Asked Questions

Back to FAQ Table of Contents

Back to FAQ Table of Contents

Back to FAQ Table of Contents

Back to FAQ Table of Contents

Back to FAQ Table of Contents

Back to FAQ Table of Contents

Back to FAQ Table of Contents

Back to FAQ Table of Contents

Related Guides

• What Is an AI Context Pack?
• How to Prepare Better Context for ChatGPT
• How to Organize Copied Text for AI Work
• Source-Labeled Context for AI: Why It Matters
• ChatGPT Context Pack for Consultants