13 KiB
13 KiB
Strict TDD Module — Verify Phase
This module is loaded ONLY when Strict TDD Mode is enabled AND a test runner is available. If you are reading this, the orchestrator already verified both conditions. Follow every instruction.
TDD Verification Philosophy
When Strict TDD Mode is active, verification goes beyond "does the code work?" to "was the code built correctly?" — meaning: was TDD actually followed? The apply phase reports TDD evidence; your job is to validate that evidence against reality.
Step 5a: TDD Compliance Check (includes Assertion Quality Audit)
Read the apply-progress artifact and verify that TDD was actually followed:
Read apply-progress artifact:
├── Find the "TDD Cycle Evidence" table
├── FOR EACH task row:
│ ├── RED column:
│ │ ├── Must say "✅ Written"
│ │ ├── Verify: test file EXISTS in the codebase
│ │ └── Flag: CRITICAL if test file does not exist
│ │
│ ├── GREEN column:
│ │ ├── Must say "✅ Passed"
│ │ ├── Cross-reference with Step 5b test execution results:
│ │ │ └── The test file listed must PASS when you run it
│ │ └── Flag: CRITICAL if test fails now (was it really green?)
│ │
│ ├── TRIANGULATE column:
│ │ ├── If "✅ N cases" → verify N test cases exist in the test file
│ │ ├── If "➖ Single" → verify spec truly has only one scenario for this task
│ │ └── Flag: WARNING if spec has multiple scenarios but only 1 test case
│ │
│ ├── SAFETY NET column:
│ │ ├── If "✅ N/N" → existing tests were run before modification (good)
│ │ ├── If "N/A (new)" → verify the file was actually NEW (not modified)
│ │ └── Flag: WARNING if file was modified but safety net shows "N/A"
│ │
│ └── REFACTOR column:
│ ├── Not strictly verifiable (subjective quality)
│ └── Skip verification, trust the report
│
├── If NO "TDD Cycle Evidence" table found:
│ └── Flag: CRITICAL — apply phase did not report TDD evidence
│ (Strict TDD was enabled but apply did not follow the protocol)
│
└── Summary: "{N}/{total} tasks have complete TDD evidence"
Step 5 Expanded: Test Layer Validation
Classify ALL test files related to this change by their testing layer:
Scan test files created/modified by this change:
├── Classify each test file:
│ ├── Unit test: tests a single function/class in isolation
│ │ └── Indicators: no render(), no page., no HTTP calls, mocked dependencies
│ ├── Integration test: tests component interaction or user behavior
│ │ └── Indicators: render(), screen., userEvent., testing-library imports
│ ├── E2E test: tests full system through real browser/HTTP
│ │ └── Indicators: page.goto(), playwright/cypress imports, browser context
│ └── Unknown: cannot classify → report as-is
│
├── Report distribution:
│ ├── Unit: {N} tests across {N} files
│ ├── Integration: {N} tests across {N} files
│ ├── E2E: {N} tests across {N} files
│ └── Total: {N} tests
│
├── Cross-reference with capabilities:
│ ├── If integration tests exist but tools not in capabilities → how?
│ ├── If E2E tests exist but tools not in capabilities → how?
│ └── Flag: WARNING if tests use tools not detected in capabilities
│
└── For each spec scenario: note which layer covers it
└── Flag: SUGGESTION if critical business logic only has unit tests
(only if integration/E2E tools are available)
Step 5d Expanded: Changed File Coverage
When coverage tool is available, report coverage for CHANGED files specifically:
IF coverage tool available (from cached capabilities):
├── Run: {test_command} --coverage (or equivalent)
├── Parse the coverage report
├── Filter to ONLY files created or modified in this change
│ (get file list from apply-progress "Files Changed" table)
├── Report per-file:
│ ├── File path
│ ├── Line coverage %
│ ├── Branch coverage % (if available)
│ ├── Uncovered line ranges (specific lines, not just %)
│ └── Flag per file:
│ ├── ≥ 95% → ✅ Excellent
│ ├── ≥ 80% → ⚠️ Acceptable
│ └── < 80% → ⚠️ Low (list uncovered lines)
├── Report aggregate:
│ ├── Average coverage of changed files
│ ├── Total uncovered lines in changed files
│ └── Compare to threshold if configured
└── Flag: WARNING if any changed file < 80% coverage
IF coverage tool NOT available:
└── Report: "Coverage analysis skipped — no coverage tool detected"
(NOT a failure — just not available)
Step 5e: Quality Metrics (if tools available)
Run quality checks ONLY on changed files, ONLY if tools are available:
Read quality tools from cached capabilities:
IF linter available:
├── Run linter on changed files only
├── Report: errors and warnings
└── Flag: WARNING for errors, SUGGESTION for warnings
IF type checker available:
├── Run type checker (usually whole-project, not per-file)
├── Filter output to changed files
├── Report: type errors in changed files
└── Flag: WARNING for type errors
IF neither available:
└── Report: "Quality metrics skipped — no tools detected"
Report Template Extension
When Strict TDD Mode is active, your verification report MUST include these additional sections:
### TDD Compliance
| Check | Result | Details |
|-------|--------|---------|
| TDD Evidence reported | ✅ / ❌ | {Found in apply-progress / Missing} |
| All tasks have tests | ✅ / ❌ | {N}/{total} tasks have test files |
| RED confirmed (tests exist) | ✅ / ⚠️ | {N}/{total} test files verified |
| GREEN confirmed (tests pass) | ✅ / ❌ | {N}/{total} tests pass on execution |
| Triangulation adequate | ✅ / ⚠️ / ➖ | {N} tasks triangulated / {N} single-case |
| Safety Net for modified files | ✅ / ⚠️ | {N}/{total} modified files had safety net |
**TDD Compliance**: {N}/{total} checks passed
---
### Test Layer Distribution
| Layer | Tests | Files | Tools |
|-------|-------|-------|-------|
| Unit | {N} | {N} | {tool} |
| Integration | {N} | {N} | {tool or "not installed"} |
| E2E | {N} | {N} | {tool or "not installed"} |
| **Total** | **{N}** | **{N}** | |
---
### Changed File Coverage
| File | Line % | Branch % | Uncovered Lines | Rating |
|------|--------|----------|-----------------|--------|
| `path/to/file.ext` | 95% | 90% | — | ✅ Excellent |
| `path/to/other.ext` | 82% | 75% | L45-48, L62 | ⚠️ Acceptable |
| `path/to/new.ext` | 100% | 100% | — | ✅ Excellent |
**Average changed file coverage**: {N}%
{or "Coverage analysis skipped — no coverage tool detected"}
---
### Assertion Quality
| File | Line | Assertion | Issue | Severity |
|------|------|-----------|-------|----------|
| ... | ... | ... | ... | ... |
**Assertion quality**: {N} CRITICAL, {N} WARNING
{or "✅ All assertions verify real behavior"}
---
### Quality Metrics
**Linter**: ✅ No errors / ⚠️ {N} warnings / ❌ {N} errors / ➖ Not available
**Type Checker**: ✅ No errors / ❌ {N} errors / ➖ Not available
Step 5f: Assertion Quality Audit (MANDATORY)
Scan ALL test files created or modified by this change and check for trivial/meaningless assertions:
FOR EACH test file related to the change:
├── Read the file content
├── Scan for BANNED assertion patterns:
│ ├── Tautologies: expect(true).toBe(true), assert True, expect(1).toBe(1)
│ ├── Orphan empty checks: expect(result).toEqual([]) or assert len(result) == 0
│ │ └── UNLESS there is a companion test with same setup that asserts NON-EMPTY
│ ├── Type-only assertions used alone: toBeDefined(), not.toBeNull(), typeof checks
│ │ └── These are OK if COMBINED with value assertions in the same test
│ ├── Assertions that never call production code (no function call, no render, no request)
│ ├── Ghost loops: assertions inside for/forEach over queryAll/filter results
│ │ └── Check if the collection could be empty — if so, the assertions NEVER RUN
│ │ Flag: CRITICAL — a loop over an empty array is a test that ALWAYS passes
│ ├── Incomplete TDD cycle: test passes because preconditions prevent code from running
│ │ └── e.g., testing behavior of a component that is never rendered due to state
│ │ Flag: CRITICAL — test must set up conditions where the code path IS exercised
│ ├── Smoke-test-only: render() + toBeInTheDocument() without behavioral assertions
│ │ └── "Renders without crash" is NOT a valid test — it must assert WHAT was rendered
│ │ Flag: WARNING — smoke tests do not count toward TDD coverage
│ ├── Implementation detail coupling: assertions on CSS classes, internal state, mock call counts
│ │ └── expect(el.className).toContain("text-xs") or expect(mock.calls.length).toBe(3)
│ │ Flag: WARNING — tests must assert behavior, not implementation
│ └── Mock/assertion ratio: count vi.mock() calls vs expect() calls per test file
│ └── If mocks > 2× assertions → Flag: WARNING — "Mock-heavy test ({N} mocks, {N} assertions)"
│ Recommend: extract logic to pure function or move to higher test layer
│
├── For each violation found:
│ ├── Record: file, line number, the assertion, why it's trivial
│ └── Classify:
│ ├── CRITICAL: tautology (expect(true).toBe(true)) — test proves NOTHING
│ ├── CRITICAL: assertion without production code call — test exercises nothing
│ ├── CRITICAL: ghost loop — assertions inside loop over possibly-empty collection
│ ├── WARNING: empty collection without companion non-empty test
│ ├── WARNING: type-only assertion without value assertion
│ ├── WARNING: smoke-test-only — render + toBeInTheDocument without behavioral check
│ ├── WARNING: CSS class / implementation detail assertion
│ └── WARNING: mock-heavy test (mocks > 2× assertions) — wrong test layer
│
├── Check triangulation quality:
│ ├── Count distinct test cases per behavior
│ ├── If only 1 test case exists for a behavior with multiple spec scenarios:
│ │ └── Flag: WARNING — "Insufficient triangulation for {behavior}"
│ ├── If all test cases assert the SAME type of value (e.g., all check empty arrays):
│ │ └── Flag: WARNING — "No variance in test expectations — all assert empty/trivial"
│ └── A well-triangulated behavior has tests asserting DIFFERENT expected values
│
└── Summary: "{N} trivial assertions found across {N} files"
Assertion Quality Report Table
Include this table in the verification report when any issues are found:
### Assertion Quality
| File | Line | Assertion | Issue | Severity |
|------|------|-----------|-------|----------|
| `path/test.ts` | 15 | `expect(true).toBe(true)` | Tautology — proves nothing | CRITICAL |
| `path/test.ts` | 23 | `expect(result).toEqual([])` | Empty without companion non-empty test | WARNING |
| `path/test.ts` | 31 | `expect(result).toBeDefined()` | Type-only — no value asserted | WARNING |
**Assertion quality**: {N} CRITICAL, {N} WARNING
If zero issues found, report: "Assertion quality: ✅ All assertions verify real behavior"
Rules (Strict TDD Verify specific)
- ALWAYS check the TDD Cycle Evidence table from apply-progress — it's the primary artifact
- ALWAYS cross-reference reported test files against actual execution — don't trust the report blindly
- ALWAYS run the Assertion Quality Audit (Step 5f) — trivial tests are WORSE than missing tests
- If apply-progress has no TDD evidence table, flag as CRITICAL — the protocol was not followed
- If tautology assertions are found (expect(true).toBe(true)), flag as CRITICAL — these MUST be rewritten
- Coverage and quality metrics are informational, NOT blocking — only flag as WARNING, never CRITICAL
- Test layer distribution is informational — SUGGESTION level only
- DO NOT fix issues — only report. The orchestrator decides.
- If coverage/quality tools are not available, say so cleanly and move on — never flag missing tools as failures