Flaky or real? Diagnose a red CI build

Goal

Your CI build went red. Before you change any code, find out whether the failure is a real regression, a flaky test, or broken infrastructure — with commands you can copy, expected output, and a decision at the end.

Why this matters

A red build is not the same as a real bug. Google's testing team reported in 2016 that almost 16% of their tests showed some level of flakiness, that about 1.5% of all test runs reported a flaky result, and — the number that should change how you triage — about 84% of pass-to-fail transitions involved a flaky test, not a real regression. Microsoft has attributed roughly 25% of CI failures to flaky tests, with other studies putting the figure as high as 50–56% before remediation (Amikoo, October 2026). Research published in the IEEE Transactions on Software Engineering in 2026 found undetected flaky failures made up 9.8%–16.3% of failed CI pipeline runs in the projects studied.

The real danger is not the noise — it is what the noise teaches your team. Google's engineers wrote that it is "quite common to ignore legitimate failures in flaky tests due to the high number of false-positives," and that it is human nature to ignore alarms with a history of false signals. Once developers learn to re-run a red build without investigating, real regressions start looking exactly like noise.

Flaky, real, or infra?

Use this decision table on the failing build's log before touching code:

• Real regression — the test fails consistently on a commit where it used to pass, and nothing else in the run looks off. The failing test's name, error, and line are the same on every re-run.
• Flaky — the test passes and fails on the same commit with no code change in between, or flips frequently across recent runs. Google's definition is exactly this: a test that exhibits both a passing and a failing result with the same code.
• Infrastructure — the failure is not in your code at all: npm install/pip install failed, the runner ran out of memory (OOM kill), a network fetch timed out, the compiler itself crashed. The log dies before your tests even start.

Two traps: do not trust a manual re-run that goes green — a passing retry proves nothing about the root cause and buries the flake for next time. And do not delete a test you suspect is flaky — quarantine it (see below) so the signal is not lost.

Detect

1. Capture the full build log to a file — you cannot triage what you cannot re-read:

   npm test 2>&1 | tee ci.log

   Copy

   Expected: the full test output in your terminal and in ci.log. On GitHub Actions, add one step after your test command:

   - name: Run tests
     run: npm test 2>&1 | tee ci.log

   Copy

2. Find the first real error — everything after it is often collateral. Scan for the framework's failure markers:

   grep -n -m1 '●' ci.log    # jest first failure
   grep -n -m1 'FAILED' ci.log # pytest first failure

   Copy

   Expected: one line number pointing at the first failing test, e.g. 47: ● Cart pricing › applies bulk discount. If the log dies during npm install, a Docker build, or a compiler step, this is infrastructure — skip the rest and fix the runner.

3. Check the failing test's history on the same commit. Did it pass here before? A failure on an unchanged commit is the strongest flake evidence there is:

   cifd record --history .cifd-history.jsonl --commit $GITHUB_SHA < ci.log
   cifd history

   Copy

   Expected: one line per recorded run with its verdict. A test that failed on abc1234 but passed on the same commit earlier is a classic flake — no code change in between, so the code cannot be the cause.

4. Run the automated triage — first error, verdict, and repeats in one report:

   npx ci-failure-detective analyze ci.log --format md

   Copy

   Expected output (from the tool's own demo on real fixture logs):

   ## 🟡 CI Failure Detective — flaky
   
   **Category:** Test failure
   **Framework:** jest
   **Signature:** `a3f9c1e2b4d6` — seen 12× before (first 2026-10-18)
   
   **First error** (line 47):
   ```
   ● Cart pricing › applies bulk discount over 10 items
   ```
   
   **Failing tests:**
   
   - `Cart pricing › applies bulk discount over 10 items` (src/cart/pricing.test.js) — **🟡 flaky**
     - Failed on abc1234 but passed on the same commit before — classic flake, no code change in between.
   
   **Suggested next steps** (advisory only — nothing is applied automatically):
   
   - **1 flaky test — quarantine suggested (not applied)**
     - Consider quarantining via --testPathIgnorePatterns or jest.retryTimes

5. Run tests verbose from now on so future triage has evidence. Same-commit flake detection needs per-test pass records, and plain logs only record failures:

   npx jest --verbose        # instead of npx jest
   pytest -v               # instead of pytest

   Copy

   With verbose logs, a test that passes on a commit is visible; without them, same-commit evidence does not exist and verdicts fall back to flip-rate and regression heuristics.

Quarantine policy

What do you do with a test once you know it is flaky? Three practices from industry teams:

• Quarantine, don't ignore, don't delete. Google marks flaky tests so they report a failure only if they fail 3 times in a row, and teams re-run only the failing tests instead of the whole suite. The quarantine list is a tracked debt item that must shrink — a flaky test stays out of the blocking pipeline until its root cause is fixed.
• Fix the root cause, not the alarm. The most common sources are timing/synchronization problems (async waits), race conditions in concurrent code, shared state between tests, test-order coupling, environment drift, and flaky third-party services. Replace fixed sleeps with condition-based waits, isolate shared state, freeze clocks, seed randomness.
• Automatic re-runs are the last resort, not the first. Plugins like pytest-rerunfailures or jest.retryTimes retry a failing test a fixed number of times and report success if any attempt passes. They mask the symptom without fixing the cause, and they teach the team that green-after-retry means "fine" — which is how real failures start hiding.

Martin Fowler's warning from 2011 still holds: left uncontrolled, non-deterministic tests can completely destroy the value of an automated regression suite.

Limits

No detection method is perfect: a test that flakes once a month can pass 100 clean runs and still fail on release day — detection is a history question, not a single-run question. A flaky test can also be signal, not noise: a test that intermittently times out may be catching a genuinely slow service. Investigate before quarantining.

Our tool

ci-failure-detective is our local-first triage step: it reads a failed CI log and reports the first real error, a verdict per test — flaky, real, or unknown with a confidence score — and clusters repeats by a stable 12-character failure signature. A test is called flaky when it fails and passes on the same commit (0.95 confidence) or flips frequently across runs (0.70); a test that fails consistently where it used to pass is a real regression (0.80). With no history it answers unknown — it never guesses. It parses Jest, pytest, go test, and Mocha, and its failure classification works on any raw log: npm/pip/go install failures, tsc errors, Docker builds, OOM kills. It runs inside your runner — no logs leave your repo, no signup. Suggested actions (quarantine hints, retry guidance) are advisory only: quarantine stays suggest-only by design, because automatically retrying or silencing failures hides real bugs.

Honest comparison (checked 2026-10-24): BuildPulse, Trunk, Datadog, and Mergify ($18–99/month) give you dashboards, trend analytics, auto-quarantine, and alerts — at the price of uploading all your test data and logs to their cloud. Gradle Develocity adds AI-powered failure grouping and predictive test selection, enterprise sales-led. The closest open-source rivals are gh-failmap (failure clustering CLI, but GitHub-only and no flaky verdicts) and flakestat (local-first flaky hunting, but flakes only — no clustering, no log reading). LLM-based triage tools send your logs to a third-party API and bill per run. CI Failure Detective has no dashboard and no alerting — it is a local triage step, not a platform.

Sources

• Google Testing Blog, "Flaky Tests at Google and How We Mitigate Them" (May 2016) — testing.googleblog.com
• Amikoo, "Real Bug or Flaky Test? How to Diagnose Failures Before You Fix Anything" (October 2026) — amikoo.ai
• Qable, "Regression Testing Strategy: What to Re-Run in 2026" (October 2026), citing Memon et al., "Taming Google-Scale Continuous Testing" (ICSE-SEIP 2017) — qable.io
• ci-failure-detective README, techdevfun/ci-failure-detective, honest-comparison section (checked 2026-10-24)