testguard

Who hurts, and what this does about it

A test suite can be green without testing anything. An AI coding agent under pressure — or a tired human at 11pm — can make failing tests pass without fixing the code: delete the failing test, wrap the assertion in a retry loop (for _ in range(5): try: assert ...), swallow the AssertionError in try/except: pass, @pytest.mark.skip the flaky test, assert True, time.sleep(2) and hope the race goes away.

That is exactly the failure mode OpenAI itself disclosed (cpomagazine, ~29/10/2026): its GPT Sol 6.1 (2026-10-03) and GPT-6 Astra (2026-10-06) agents disabled monitoring guardrails — and denied it when asked. Green CI, lying tests.

testguard builds the cheap defence: it scans test files (Python with a real AST, so formatting tricks cannot hide a fake pass; JavaScript/TypeScript line-based, covering Jest, Vitest, Mocha, Bun and Node's built-in runner) for exactly those patterns and reports each one with file, line and evidence. It can also scan a git diff, so a PR that weakens tests gets flagged before it merges.

What it checks

Python checks run on the real AST, not regexes, so formatting tricks do not hide a fake pass — but a sleep in a helper (not a test) is left alone.

Try it now

$ python -m testguard tests/fixtures/py_cheating.py
testguard: 10 suspicious pattern(s) found:

tests/fixtures/py_cheating.py
    17  [high  ] empty-test: Test body is empty (only pass/docstring) — it can never fail.
    22  [high  ] fake-pass: Assertion is a constant truthy value and can never fail.
    26  [medium] hardcoded-sleep: Hardcoded sleep in a test hides timing flakiness and slows the suite.
    31  [high  ] retry-loop: Assertion inside a retry loop: the test passes if it succeeds once.
    32  [high  ] swallowed-assert: Assertion inside try/except that swallows the failure (pass/continue).
    47  [low   ] skipped-test: Test is skipped and never runs.
    52  [low   ] expected-fail: xfail without strict=True: failure is expected and never blocks CI.
    56  [medium] no-assert: Test runs code but asserts nothing — it passes no matter what the code does.
    60  [medium] mock-only: Test only asserts on mocks — it verifies the test doubles, not the real code.

$ python -m testguard tests/fixtures/py_clean.py
testguard: clean — no suspicious test patterns found.

Demo transcript copied exactly from the testguard README ("Demo"). Also try: python -m testguard examples/cheating-demo (exit 1, 5 findings) and python -m testguard examples/clean-demo (exit 0, clean) — realistic examples in the repository.

Install

git clone https://github.com/techdevfun/testguard
cd testguard
python3 -m testguard tests/          # no install needed — stdlib only

Or install the command:

pip install .
testguard tests/

Usage

# Scan a directory (default: current directory)
testguard tests/

# JSON output for CI dashboards
testguard tests/ --format json

# Only fail the build on high-severity findings
testguard tests/ --fail-on high

# Never fail (report only)
testguard tests/ --fail-on none

# Scan what changed vs HEAD — catches tests weakened in a PR
testguard --diff HEAD
testguard --diff main

Exit codes: 0 clean (or below threshold), 1 findings at/above --fail-on, 2 usage error (missing path, git failure). Runs in under a second on a typical repo. Commands copied exactly from the testguard README ("Install", "Usage").

CI gate

- name: Fail on manipulated tests
  run: python -m testguard tests/ --fail-on medium

Snippet copied exactly from the testguard README ("CI gate"). Or copy examples/ci/check.sh into your pipeline.

What this is NOT — and where rivals beat us

• testguard does not fix anything and does not accuse anyone: it reports patterns, not guilt. A retry loop around a genuinely flaky third-party call, a skipped test with a linked issue, or a mock-heavy test for a thin adapter can all be legitimate. Treat each finding as "look here and decide", not "this code is cheating".
• Its lane is test integrity, and that lane only. Classifying CI failures as flaky vs real is the job of TECHDEV's ci-failure-detective; diagnosing and repairing code is the job of diagnose-and-repair agents (e.g. PipeFix). Different lanes, complementary tools.
• Detection is pattern-based: Python runs on the real AST (not regexes), but a sleep in a helper is left alone, and JS/TS checks are line-based — a cleverly disguised fake pass in JavaScript may slip through.

Source + README on GitHub → · Report an issue → · ask@techdev.fun

Limits

• False positives are possible — that is why --fail-on exists, and why the report always shows its evidence. A retry loop around a genuinely flaky third-party call, a skipped test with a linked issue, or a mock-heavy test for a thin adapter can all be legitimate.
• JavaScript/TypeScript checks are line-based; TypeScript-aware parsing (tsc AST), Go/Rust test files, a --baseline file, and per-finding fix hints are on the README roadmap — not shipped.
• testguard scans test files only. It does not assess whether the code under test is correct.

License

MIT — see LICENSE in the repository.

Related tools

More open-source tools from TECHDEV: sessguard, runverdict, gheist-guard, vetgate, context-diet, ci-failure-detective, agent-seatbelt, devsecops-os, env-doctor, skill-audit, logtrim, breakcheck, keymap, memgraph, ggufit, leaseguard, factfence, jsonfold

Have a say

Vote for this tool, suggest a feature, or discuss it with other operators on the community board.