TECHDEV · Open-source tool

factfence

AI assistants now write with total confidence about statistics that were never true: a plausible percentage, a named report, a link that 404s — or worse, a link to a real page that never contained that number. Human reviewers skim past them because each claim looks sourced. factfence is a small, zero-dependency CLI that reads a markdown or text file, pulls out every quantitative claim, fetches each citation, and tells you whether the cited page actually contains the numbers. It takes about ten seconds to run and tells you exactly which three sentences deserve your skepticism.

View on GitHub →Read the article →

Python 3.10+ · Zero dependencies · MIT

What it does

Given a file like this (a typical AI-drafted paragraph):

According to the 2026 State of DevTools Survey, 73% of developers say
AI-generated code needs human review before merging
([survey](http://127.0.0.1:18347/source-a.html)).

Some teams claim 99.9% of AI suggestions are production-ready, citing an
internal study (http://127.0.0.1:18347/source-b.html).

It is often repeated that 60% of all code will be AI-written by 2027,
though nobody cites a source.

Input copied exactly from the factfence README ("What it does"). factfence reports (real output from examples/run-demo.sh):

[✓ SUPPORTED]
  According to the 2026 State of DevTools Survey, 73% of developers say AI-
  generated code needs human review before merging
  ([survey](http://127.0.0.1:18347/source-a.html)).
    ↳ http://127.0.0.1:18347/source-a.html
      HTTP 200, matched 73%
    cited page contains 73%

[? UNVERIFIED]
  Some teams claim 99.9% of AI suggestions are production-ready, citing an
  internal study (http://127.0.0.1:18347/source-b.html).
    ↳ http://127.0.0.1:18347/source-b.html
      HTTP 200
    cited page(s) fetched but contain none of the claim's numbers

[! UNCITED]
  It is often repeated that 60% of all code will be AI-written by 2027,
  though nobody cites a source.
    quantitative claim with no citation — verify by hand

6 claim(s) checked: 2 supported, 1 cited, 1 unverified, 1 broken, 1 uncited. 3 need review.

Output copied verbatim from the factfence README (real output from examples/run-demo.sh).

The five verdicts

Verdict	Meaning
`SUPPORTED`	A cited page loaded and contains the claim's numbers.
`CITED`	A source-less assertion ("studies show…") with a working link — nothing numeric to cross-check.
`UNVERIFIED`	The cited page loads but doesn't contain the numbers. The classic AI hallucination shape.
`BROKEN`	Every cited link failed (404, timeout, DNS).
`UNCITED`	A quantitative claim with no citation at all.

Verdict table copied exactly from the factfence README ("The five verdicts").

Quick start

# zero dependencies — Python 3.10+ stdlib only
git clone https://github.com/techdevfun/factfence.git
cd factfence

# 1. point it at any AI-generated text
python3 -m factfence draft.md

# 2. read the verdicts (exit code 1 = something needs review)
# 3. pipe it into CI
python3 -m factfence --json draft.md | python3 -m json.tool

Commands copied exactly from the factfence README ("Quick start"). Or install it as a command: pip install ., then factfence draft.md.

How it works

Extract — code blocks are stripped, prose is split into sentences, and sentences with strong numbers (percentages, money, large counts, multipliers, ratios) become statistic claims; "studies show…" / "according to…" sentences become assertion claims. Citations are detected as inline links, reference links, footnotes, and bare URLs.
Fetch — each unique URL is fetched once (8s timeout, 1.5MB cap, polite user-agent), reduced to plain text, and normalized the same way the claim's numbers are normalized.
Verify — a claim is SUPPORTED if any fetched page contains any of its signature numbers. Best verdict wins when several URLs are cited.

Exit codes: 0 clean · 1 something needs review · 2 usage/IO error — so factfence --json draft.md drops straight into CI.

Behavior from the factfence README ("How it works"), not a copied snippet.

What this is NOT — and where rivals beat us

factfence deliberately does not judge whether a source is right — only whether the numbers appear in it. A SUPPORTED claim can still be wrong; the tool catches the far more common failure of confident numbers with dead, irrelevant, or missing citations.
Existing citation checkers (Citely, CiteProof, academic DOI verifiers) mostly verify that a reference exists — aimed at bibliographies and legal briefs, usually as paid web services. factfence asks the narrower question for everyday prose: does the cited page contain the actual numbers? — as a free, local, offline-friendly CLI you can run before you hit publish.
Always read the source yourself before publishing a number. factfence verifies citation support, not truth.

Source + README on GitHub → · Report an issue → · ask@techdev.fun

Limits

Heuristic sentence splitting; English-focused. Tables, PDFs, and non-English prose are out of scope.
Number matching is textual, not semantic: "a 73% increase" won't match a page that says "rose by nearly three quarters".
Pages behind login walls, JS-rendered apps, and PDFs can't be checked (reported as fetch failures, not as verdicts on truth).
It verifies citation support, not truth. Always read the source yourself before publishing a number.

Limits from the factfence README's own "Honest limitations" section.

Roadmap — not shipped yet

--offline mode (report citation presence without fetching)
DOI / arXiv identifier resolution
Markdown table cell support
Optional LLM-free "number paraphrase" matching (73% ≈ "nearly three quarters")

Roadmap copied from the factfence README; all four items are still open checkboxes upstream.

License

MIT — see LICENSE in the repository.

Have a say

Vote for this tool, suggest a feature, or discuss it with other operators on the community board.