You are measuring code coverage wrong
If your coverage check compares one percentage with a threshold, it will pass a pull request that removes the test for your most important code. Now that AI agents write more of that code, the problem gets worse.
Here is how it happens, in an example storefront.
The wrong way: one percentage
A developer, or an agent, adds gift card payments to the storefront. The new code is well tested, overall coverage goes up, and the gate passes.
What the coverage gate saw
83.54%
Up 0.78 points. The target is 80%.
New code: 149 of 150 lines tested.
Check passedWhat the gate did not show
2 lines
In refunds.py, 2 lines lost their only test.
The refund limit is now untested.
Not reportedTo speed up the test suite, the same pull request deletes a slow test:
src/storefront/checkout/giftcards.py (new file: 150 lines, 149 tested)
+class GiftCardPayment:
+ ...
tests/checkout/test_refunds.py
-@pytest.mark.slow
-def test_refund_cannot_exceed_capture(payment):
- ...
That test was the only one that ran two lines in refunds.py. Those lines stop the system from
refunding more money than it captured. The pull request does not change them, so they are not in
the diff. But after this pull request, no test runs them.
41def refund(payment, amount):42 if amount > payment.captured_amount - payment.refunded_amount:43 audit.log("refund_rejected", payment_id=payment.id, amount=amount)44 raise RefundExceedsCapture(payment.id, amount)45 ...
41def refund(payment, amount):42 if amount > payment.captured_amount - payment.refunded_amount:43 audit.log("refund_rejected", payment_id=payment.id, amount=amount)44 raise RefundExceedsCapture(payment.id, amount)45 ...
The percentage cannot see this. It is an average over every line in the codebase. Add 150 well-tested lines, leave 2 important lines without a test, and the average goes up.
Patch coverage (also called diff coverage) does not see it either. It checks only the lines that a pull request changes, and the refund lines did not change. A deleted test removes coverage from lines that the pull request never touches.
Every line in the codebase
Lost their testOutside the diff, where patch coverage never looks.
Changed linesThe only lines patch coverage checks.
TestedNot testedLost its test in this pull request
The right way: compare line by line
Compare coverage line by line against the branch that the pull request merges into. If a line had a test before and has none now, fail the check and name the line.
1. Coverage on main
41424344
refunds.py: every line has a test.
2. Coverage in the pull request
41424344
Lines 43 and 44 have no test.
3. The check
Failed
refunds.py, lines 43 and 44 lost their test.
Of the three checks, only this one catches the deleted test:
| Check | Catches the lost test? |
|---|---|
| Coverage percentage How much of the code runs under tests? |
No |
| Patch coverage Do the changed lines have tests? |
No |
| Line by line Did a tested line lose its test? |
Yes, names 2 lines |
It also changes what you block. Green is the right call, and red is the wrong one:
| Pull request | Percentage gate | Line by line |
|---|---|---|
| Adds an untested log line (coverage down 0.1%) | Blocks it | Lets it through |
| Deletes the refund test (coverage up 0.78%) | Lets it through | Blocks it |
Building this yourself? Four details to get right
| Detail | What goes wrong | What to do |
|---|---|---|
| Line numbers move | Line 43 on main is line 51 in the branch, so every edit looks like lost coverage. | Map lines through the diff before you compare them. |
| The base commit | Unrelated changes on main show up as losses in someone else's pull request. | Take base coverage from the commit that the pull request branched from. |
| Lines that run only sometimes | Code that depends on timing or test order causes false alarms, and people ignore the check. | Find these lines and handle them. |
| Generated and vendored code | The report fills with lines that nobody intends to test. | Leave it out. |
Why agents make this urgent
For most of the history of code review, a person who read the diff was the last check. That person could see that a pull request about gift cards deleted a test about refunds, and ask why. That person reads fewer diffs every month.
We ditched code review at @DuckbillHQ (mostly)
The most important word in that post is "mostly". Teams that stop reading every diff replace that reading with checks on what breaks production, such as test coverage. Then the checks are the review, and a percentage is not good enough to be the review.
Agents make this worse in three ways:
- They aim for a passing build. When a test is slow or fails, the shortest path to a passing build is often to delete or skip it. A percentage gate rarely notices.
- They write a lot of tested code. A large, well-tested pull request raises the percentage, and that hides the lines it left without a test.
- Nobody reads the diff. When an agent opens the pull request and another agent reviews it, the deleted test can merge before a person sees it.
Agents are also good at the fix, if you give them the right signal. "Coverage dropped 0.1%" gives an
agent nothing to act on. "Lines 43 and 44 in refunds.py lost their test" tells it which test to
write.
How Drape does it
Drape compares every pull request with its base, line by line. When tested lines lose their tests, Drape fails the check and names the lines, whatever the percentage does. Here is the example pull request again, from the failed check to the fix:
CI fails the pull request and names the lines
$ drape upload coverage coverage.xml Head: 83.54% (+0.78%) Regressed: 2 lines Regressed files: src/storefront/checkout/refunds.py: 2 lines (43-44) Result: FAILED - Coverage regression: 2 previously-covered lines lost coverageYour agent asks Drape why through Drape's MCP server
> Why is coverage failing on PR 451? ● drape · get_pr_report(repo="storefront", pr=451) refunds.py 43-44: covered on main, not in this pull request ● The over-refund guard lost its only test. Restoring it as a fast unit test. + def test_refund_cannot_exceed_capture(payment): + with pytest.raises(RefundExceedsCapture): + refund(payment, payment.captured_amount + 1)The fix passes with no lost lines
$ drape upload coverage coverage.xml Head: 83.61% (+0.85%) Regressed: 0 lines Result: PASSED
Whatever tool you use, stop asking "what is the percentage?" Ask "which code stopped being tested?"