← All posts

Flaky tests teach your team and your agents to ignore a red build

A flaky test fails on code that works. Rerun it, and it passes. Each rerun costs CI time and slows the merge. But the worst cost is the habit: once people expect a red build to be flaky, they rerun the real failures too. Coding agents make this worse, because an agent cannot tell the two apart.

Every team has them

At Google, almost 16% of tests fail now and then on code that works. So a red build is often noise:

Pass to fail, caused by a flaky test

84%

At Google, most times a test went from pass to fail, a flaky test was involved.

Google

Commits with a flaky red build

1 in 11

At GitHub, before they built tooling to manage flaky tests.

GitHub

They waste CI hours and slow every merge

A rerun runs the whole job again, and someone waits for it. At Atlassian, about 15% of failures in the Jira backend repository come from flaky tests. The reruns waste over 150,000 hours of developer time each year. A pull request that should merge in ten minutes waits an hour. A deploy that should go out today waits behind a build that has to pass twice.

The real cost is trust

…engineers quickly learn to ignore it and eventually remove it.

Meta engineering, on flaky tests · Source

Here is how that goes in an example storefront. test_partial_refund fails now and then on main. Everyone knows it, so when it goes red they rerun the build. Then a pull request breaks refunds for real. The test fails, someone reruns it, and the second run happens to pass. The break ships.

Without the history

Rerun

"That test is flaky." A rerun passes, and the break ships.

With the history

Real

It failed all 3 attempts on this commit, so it is a new failure.

Agents make it worse

A coding agent sees one red build and no history. It does not know that test_partial_refund failed on main last week. So it treats every red build as a bug to fix, and it has three bad options:

1. Change working code

Wrong fix

It edits the refund logic until the test passes, and that passing run was luck.

2. Weaken the test

Lost check

It adds a retry, a sleep or a skip, so the test stops checking anything.

3. Rerun until it passes

Lost hours

It loops on CI and burns build minutes and tokens.

Tell the agent the test is flaky, and it fails the other way: it ignores the real failure too. An agent needs the same thing a person needs, which is to know how this test behaved before.

The big teams built tools for this

Every company that fixed flaky tests did it the same way. They recorded every test run, across commits, and used the history to decide which failures are real:

Team What they built
Google Flaky-result tracking across every test run
Meta A flakiness score for every test
GitHub Flaky-test detection, with 18x fewer flaky builds
Slack Automatic detection and suppression
Spotify A dashboard of test results that cut flakiness from 6% to 4%

Most teams do not have a team to build this. Drape gives them the same history.

How Drape does it

Drape records every test run in your CI and lays them out in a history grid. A test that passes and fails on the same commit is flaky, and the grid shows it at a glance:

Test history: the last six commits on main, then the pull request

Test3e19c0b4271de5a2fbPR 512
test_totals✓✓✓✓✓✓✓
test_partial_refund✓✗✓✓✓✗✓✓✗✗✗
test_store_credit✓✓✓✓✓✓✓
test_cancel_order✓✓✓✓✓✓✓

Passed Failed, then passed on a retry (flaky) Failed every attempt

On main, test_partial_refund flaked on two of six commits. On the pull request it failed all three attempts, so this failure is real. Drape reads the grid the same way, for each red build:

  1. CI fails the pull request on one test

    FAILED tests/test_refunds.py::test_partial_refund
      3 attempts, 3 failures
  2. Your agent asks Drape before it reruns through Drape's MCP server

    > Why did CI fail on PR 512?
    ● drape · triage_ci(repo="storefront", pr=512)
      test_partial_refund: new failure
      flaky on main, but failed every attempt on this commit
    ● This is not the flake. The change broke partial refunds. Fixing the code.
  3. The fix passes on the first run

    PASSED tests/test_refunds.py::test_partial_refund
  • Known flaky failures stop blocking merges. You can suppress a flaky test in Drape, or let Drape suggest it. A suppressed failure no longer fails the build, and Drape keeps recording its runs.
  • New tests prove they are stable first. Drape can run a new test many times in your CI, before it can flake in everyone's build.
  • You see what flakiness costs. Drape shows the CI time lost to reruns, split into flaky tests, infrastructure and real failures.

A red build should mean that something broke. Drape makes it mean that again, for people and for agents.

Stop re-running flaky builds.

14-day free trial. No credit card required.