AI-Generated Tests Are Producing Garbage: How to Guide LLMs to Write Tests You Can Trust

If you've used a chatbot or a coding agent to generate tests in 2025–2026, you've seen the failure mode: tests that look correct, pass on green-path scenarios, and silently fail to catch real bugs. The World Quality Report 2025–2026 cited this as the #1 concern with AI in QA — "unguided AI testing generates noise: false positives, brittle coverage, edge cases that slip through because no one thought to encode the right context."

The fix isn't to stop using AI. It's to stop using AI unguided. After two years of pairing with LLMs on QA work across five client projects, here's the prompt-and-review workflow that produces tests I'd ship.

Table of Contents

• The 5 failure modes of unguided AI test generation
• Prompt principle 1: Tests need a contract before they need code
• Prompt principle 2: Provide existing patterns, not blank slates
• Prompt principle 3: Negative cases by construction
• The review checklist before merging AI-generated tests
• Metrics that catch garbage AI tests after they ship
• Real prompts that work on Claude / Cursor / Copilot
• FAQs

The 5 Failure Modes of Unguided AI Test Generation

1. Hallucinated routes. Agent writes a test against /api/users/me when the actual endpoint is /v1/account. Test fails on first run. Costs you nothing if you run it; costs you a flaky reputation if you merge without running.
2. Tautological assertions. Agent fills the email field with "test@test.com" then asserts the email field contains "test@test.com". Passes always. Tests nothing.
3. Happy-path-only coverage. Agent writes 8 tests, all of them success scenarios. Negative cases (declined card, expired session, invalid input) are absent because the agent didn't think to test them.
4. Locator decay built-in. Agent uses CSS classes (.btn-primary.checkout-btn) instead of role-based selectors. Tests pass today, break in two weeks. See my CSS locators migration post.
5. Missing async waits. Agent reads the trace viewer but emits page.waitForTimeout(2000) anyway. Tests are slow and still flaky. See my web-first assertions post.

All five failure modes have one root cause: the agent doesn't know your codebase's conventions and the prompt didn't tell it. Garbage in, garbage out.

Prompt Principle 1: Tests Need a Contract Before They Need Code

The wrong prompt: "Write Playwright tests for the checkout page."

The agent has to decide what "the checkout page" means, which features to test, what success looks like, and what the edge cases are. Each of those decisions is a place hallucination creeps in.

The right prompt: write the test plan first. Either you write it, or you have the planner agent generate it (see my Playwright 1.59 agents post) and you edit it. Then hand the plan to the generator.

Example plan I'd hand to an agent:

## Test plan: /checkout

### Critical paths
1. Empty cart -> redirect to /products
2. Single item, valid card -> success page (assert order ID in URL)
3. Multiple items, valid card -> success, total includes tax line

### Negative paths
4. Declined card (4000000000000002) -> inline error, cart preserved
5. Expired card -> client-side validation prevents submit
6. Invalid email -> field-level error, no API call

### Skip
- 3DS (requires real card, manual verification)
- Apple Pay (mobile only, separate suite)

### Conventions
- Use getByRole and getByTestId, never CSS class selectors
- Wait via page.waitForResponse, never waitForTimeout
- Test data: import { createTestUser, addToCart } from '@/fixtures'

The conventions section is critical. It's the agent's only chance to learn your team's patterns.

Prompt Principle 2: Provide Existing Patterns, Not Blank Slates

If you have an existing checkout test file in your repo, give it to the agent as a reference:

npx playwright test --agent=generator \
  --plan=plan.md \
  --context=tests/checkout/successful-payment.spec.ts,tests/fixtures/auth.ts \
  --output=tests/checkout/declined-card.spec.ts

The agent now knows your file structure, your imports, your fixture patterns, your assertion style. It will mimic them.

For chat-based tools (Claude, ChatGPT), paste the existing similar test in the prompt:

"Here's an existing test for our checkout flow. Match this style — same fixtures, same assertion patterns, same selector approach. Now write a new test for the declined-card scenario."

[paste 60–80 lines of existing test]

The output quality jumps dramatically. You're not asking the agent to invent your conventions; you're asking it to apply yours.

Prompt Principle 3: Negative Cases by Construction

Agents skew toward happy paths because most training data does. Counteract this in the prompt.

I append a checklist to every test-generation prompt:

"For each user action you test, also write a test for: (a) the input being missing, (b) the input being invalid, (c) the user not being authorized, (d) the network call failing with a 500, (e) the network call timing out. Skip a category only if it's not reachable for that action."

The agent now produces 5x the test count, most of which are negative cases. You'll prune some — not every action has all five — but the bias toward edge cases is established.

The Review Checklist Before Merging AI-Generated Tests

I never merge an AI-generated test without reading it line by line. This is the checklist:

1. Run it first. If it fails on the first run, the agent hallucinated. Don't waste time reading.
2. Read every assertion out loud. If the assertion is "the thing I just typed is the thing I just typed," delete it.
3. Check selectors. Any CSS class? Replace with role-based or test-id.
4. Check waits. Any waitForTimeout? Replace with web-first assertion.
5. Check test data. Hardcoded emails, names, IDs? Move to fixtures.
6. Check the negative cases. Did the agent write any? If only happy paths, send back.
7. Run the full suite. Did this test cause others to flake (shared state)? Fix isolation.

Time per spec file: roughly 10–15 minutes. Faster than writing it from scratch (45–90 minutes), slower than blindly merging (3 minutes, but you ship trash).

Metrics That Catch Garbage AI Tests After They Ship

Even with review, some bad tests slip through. These metrics catch them post-merge:

• Pass rate over time. Tests that pass 100% of the time for 30 days might be tautological. Investigate.
• Mutation testing score. Run Stryker (or Mutation in your language) periodically. Tests that don't catch mutations aren't testing anything real.
• Bug-escape correlation. When a customer-reported bug ships, check: should an existing test have caught it? If yes, the test was wrong.
• Flakiness rate. Persistently flaky tests are usually badly-written tests, not victims of "flaky systems." See my race conditions post.

Real Prompts That Work on Claude / Cursor / Copilot

Three templates I actually use, ready to copy:

Template 1: New test from a plan

You're writing a Playwright test in TypeScript. The codebase uses:

• Role-based locators (getByRole, getByLabel) and getByTestId as fallback. NEVER CSS classes.
• Web-first assertions (await expect(locator).toHaveText(...)). NEVER waitForTimeout.
• Test data via fixtures from @/fixtures.
• Network waits via page.waitForResponse with the URL pattern, in Promise.all with the click that triggers it.

Here's a representative existing test for context: [paste 60 lines]

Now write a test that does this: [paste plan section]

Output the spec file only. No explanation.

Template 2: Fixing an AI-generated test that's failing

This test is failing in CI but passes locally. Here's the test: [paste]. Here's the trace viewer summary: [paste trace text].

Diagnose the root cause. Don't add retries or longer timeouts. Common causes: race conditions on API responses, hydration timing, animation overlay, locator brittleness. Fix the root cause.

Template 3: Reviewing AI output for quality

Review this AI-generated Playwright test for quality issues. Flag any of: (1) CSS class selectors, (2) waitForTimeout, (3) tautological assertions, (4) hardcoded test data, (5) missing negative cases, (6) shared state with other tests. Be specific about line numbers. Suggest fixes.

[paste test]

FAQs

Should I disable AI for test writing on my team?

No — properly guided, AI is a 30–50% productivity boost. Disable unguided use. Require the test plan + existing-pattern context for any AI-generated test that gets merged.

How do I prove AI tests are low quality to my team?

Run mutation testing on a sample of AI-generated tests vs human-written tests. The mutation score (% of code mutations the tests catch) is usually visibly lower for unguided AI tests.

What model produces the best test code?

Claude Sonnet 4.6 produces the cleanest Playwright code in my experience. GPT-4 is a close second. Codex is okay for autocomplete but not for whole-test generation. The model matters less than the prompt structure.

Are the Playwright 1.59 agents better than chat-based generation?

Slightly — they have access to the running app via browser.bind and can verify their tests as they write. But the same prompt principles apply.

What about AI for test maintenance, not generation?

Maintenance is where AI shines. The healer agent fixes locator drift in seconds. New test generation is where it struggles.

Should I commit a CONTRIBUTING.md with conventions for AI agents?

Yes. Same file used by humans. Your CI agent reads it as part of context. Single source of truth.

How long should an AI-generated test review take?

10–15 minutes per spec file. If it's faster, you're skipping. If it's slower, the prompt was too vague — improve the prompt for next time.

What about agentic test generation that runs continuously?

Run on a schedule (e.g., weekly). Output to a separate branch. Human reviews and merges selectively. Never auto-merge agent-generated test commits.

How do I track AI test quality over time?

Tag every AI-generated test with a comment: // gen: claude-sonnet-4.6, prompt: v3, date: 2026-04-15. Then you can correlate model + prompt version + bug-escape rate.

Will the AI eventually replace the QA engineer?

The role shifts toward judgment and prompt design. Writing the test code is becoming AI-assisted. Deciding what's worth testing is not.

Wrap-Up

The World Quality Report flagged AI-generated test garbage as a real industry problem. The fix isn't to ban AI — it's to guide it with structured prompts, existing-pattern context, explicit negative-case requirements, and a strict review checklist. Done right, it's a 30–50% productivity boost. Done wrong, it's a future regression-debt machine.

If your team is adopting AI test generation and wants help building the prompt+review workflow, I cover this in framework engagements. Or book a free call.

Related reading:

• Playwright 1.59 Agents Explained
• Migrate to Role-Based Locators
• Hardcoded Waits Are Killing Your Tests
• AI in QA Automation 2026