TECHDEV · Flagship · Open-source tool

ci-failure-detective

Your CI failed. Was it a real break, or just a flake? And where exactly did it break? CI Failure Detective reads a failed CI log and answers three questions: what the first real error was, whether each failing test is flaky or genuinely broken, and whether this exact failure happened before.

View on GitHub →Read the article →Read the guide →

GitHub Action + CLI (cifd) · v0.1.0 · Zero dependencies · Node 18+ · MIT · Runs entirely inside your runner — no logs ever leave your repo, no signup, no SaaS bill.

What it tells you

Question	How
Where did it actually break?	First error line + stack context, classified (test / compile / dependency / OOM / timeout / lint)
Flaky or real?	Verdict per test — `flaky`, `real`, or `unknown` — with a confidence score, based on run history
Seen this before?	Stable 12-char failure signature; repeats are clustered with first/last seen dates
What should I do?	Suggested actions only — quarantine hints, retry guidance, first-error pointer. Nothing is ever applied automatically

Suggested next steps are advisory only — nothing is applied automatically. Quarantine stays suggest-only by design: automatically retrying or silencing failures hides real bugs.

How a flaky verdict is decided

Flaky, 0.95 confidence: a test fails and passes on the same commit.
Flaky, 0.70 confidence: a test flips frequently across recent runs.
Real, 0.80 confidence: a test fails consistently on a commit where it used to pass — a real regression.
Unknown: no history yet. It never guesses.

Flaky detection needs history: commit .cifd-history.jsonl in your repo and append each run. Same-commit verdicts need per-test pass records, so run tests verbose (jest --verbose, pytest -v) so passing tests are visible in the log. Plain (non-verbose) logs only record failures, so same-commit evidence is unavailable from them — flip-rate and regression verdicts still work.

Supported frameworks — and any raw log

Test-result parsing: Jest, pytest, go test, Mocha (best-effort — the best-matching parser wins). But you do not need a test framework at all: failure classification and first-error extraction work on any raw log — npm/pip/go install failures, tsc errors, Docker build failures, OOM kills.

Use it

# As a GitHub Action (recommended) — add one step after your test command
- name: Run tests
  run: npm test 2>&1 | tee ci.log

- name: Diagnose failures
  if: failure()
  uses: techdevfun/ci-failure-detective@v1
  with:
    log-file: ci.log

# As a CLI — zero dependencies, Node.js 18+
npx ci-failure-detective analyze ci.log
cifd analyze ci.log --format md >> $GITHUB_STEP_SUMMARY

# Pipe a live run straight into the report
npm test 2>&1 | tee ci.log | npx ci-failure-detective analyze --format md

# Record history, then show failure clusters
cifd record --history .cifd-history.jsonl --commit $GITHUB_SHA < ci.log
cifd history   # show failure clusters

Commands copied exactly from the ci-failure-detective README (v0.1.0). The README shows these npx / CLI commands and the @v1 Action step; it does not state a separate npm-install step, so none is claimed here.

Action inputs and outputs

Input	Default	Description
`log-file`	—	Path to the CI log (required)
`history-file`	`.cifd-history.jsonl`	Run-history file
`commit`	`$GITHUB_SHA`	Commit SHA for flaky scoring
`fail-on-real`	`false`	Exit 1 when verdict is `real-failure`
`summary`	`true`	Append markdown report to `$GITHUB_STEP_SUMMARY`

Outputs: verdict, signature, report-json.

CLI reference

cifd analyze [logfile] [--format json|md] [--history FILE] [--commit SHA]
cifd record --commit SHA [--history FILE] < logfile
cifd history [--history FILE]

Example output (README demo, not a benchmark)

The README's worked example is a flaky Jest verdict, recorded from a real terminal session on the repo's own fixture logs. It is shown here as an example of the report format — not as measured performance:

Category: Test failure · Framework: jest
Signature: a3f9c1e2b4d6 — seen 12× before (first 2026-10-18)
First error (line 47): Cart pricing › applies bulk discount over 10 items — “At least one test failed — see the first failing test below.”
Failing test: Cart pricing › applies bulk discount over 10 items (src/cart/pricing.test.js) — 🟡 flaky: failed on abc1234 but passed on the same commit before — classic flake, no code change in between.
Suggested next step: 1 flaky test — quarantine suggested (not applied); consider quarantining via --testPathIgnorePatterns or jest.retryTimes.

The README contains no latency, accuracy-benchmark, or test-count figures, so none are claimed on this page. The demo also shows an infra verdict for a failed npm install and the failure-cluster history.

What this is NOT — and where rivals beat us

CI Failure Detective has no dashboard and no alerting — it is a local triage step, not a platform.
BuildPulse / Trunk / Datadog / Mergify ($18–99/month, comparison checked 2026-10-24) beat us on breadth: full dashboards, trend analytics, auto-quarantine and auto-retry, Slack/PagerDuty alerts. They require uploading all your test data and logs to their cloud.
Gradle Develocity beats us on AI-powered failure grouping across builds and predictive test selection — enterprise sales-led, heavyweight by design.
gh-failmap (OSS) is the closest direct rival for failure clustering, but it is CLI-only, needs Python + gh login, is GitHub-only, and has no flaky verdicts.
flakestat (OSS, zero-dependency) is the closest philosophical rival — local-first flaky hunting — but it only hunts flakes (via re-runs or JUnit XML); it does not cluster failures or read logs.
ci-why / ci-log-triage explain failures with an LLM — which means your logs go to a third-party API and you pay per run.

As the README states it: CI Failure Detective is the only zero-dependency GitHub Action + CLI that does all three — cluster repeat failures, separate flaky from real, point at the first error — without any byte of your log leaving the runner.

Source + README on GitHub → · Report an issue → · ask@techdev.fun

Limits and roadmap

Limits stated in the README: flaky detection needs run history (without it, verdicts are unknown); same-commit verdicts need verbose logs; framework parsing is best-effort. Nothing is ever applied automatically.

Roadmap — not shipped features: JUnit XML as an additional input (alongside raw logs); Slack webhook output; more frameworks (PHPUnit, RSpec, .NET); SARIF output for code scanning integration.

Have a say

Vote for this tool, suggest a feature, or discuss it with other operators on the community board.