TECHDEV · Flagship · Open-source tool

servefit

Before you rent the GPUs, check the map. Headlines say "production inference engine" and assume 64 Blackwell GPUs. Reality: DeepInfer collected ~19k stars in one week (GitHub Trending scan 2026-10-28) while serving exactly one model — DeepSeek-V4.1-Flash — on SM100+ or Ascend 950 silicon, offline-only, with no KV-cache system, no PD disaggregation, no load balancing (checked against the upstream repo 2026-11-01). servefit inventories your hardware, does the memory math for a model, checks a dated engine-support matrix, and tells you what you can actually serve — and with what.

View on GitHub →Read the article →

Python · CLI · Apache-2.0 · Python 3.10+ · Zero dependencies. Works offline with --local <model-dir> or explicit --params/--layers/--kv-heads. Until a release is published on PyPI: pip install git+https://github.com/techdevfun/servefit.

The problem it answers

The OOM killer is a terrible capacity planner. Picking an LLM serving engine means combining three facts you usually don't have in one place: what hardware you actually have, what the model actually costs in memory at your context length, and which engines are actually viable on that combination. servefit answers exactly that: servefit inventory parses nvidia-smi (GPUs, VRAM, compute capability with name fallback) plus rocm-smi for AMD and CPU/RAM — Apple Silicon is reported as detected-with-limits, never guessed — then servefit fit --model Qwen/Qwen3-8B does weights + KV cache + overhead math with every assumption printed, picks viable engines from a curated 6-engine matrix, and gives one recommendation with reasons plus a starter config for that engine.

Three commands

Command	What it does
`servefit inventory [--from saved.json]`	Hardware inventory: GPUs, VRAM, compute capability, CPU/RAM. `--from` plans a purchase before the box exists.
`servefit fit --model <hf-id-or-label>`	Memory math + viable engines from the dated matrix + one recommendation + starter config. Exits with reasons, never silent.
`servefit probe --endpoint <url> --model <id>`	Measures TTFT and approximate tokens/sec against a running OpenAI-compatible endpoint on your hardware.

Every command also writes --json out.json manifests for automation. Commands copied exactly from the servefit README (release 2026-11-01).

pip install git+https://github.com/techdevfun/servefit
servefit inventory
servefit fit --model Qwen/Qwen3-8B
# one recommendation, every assumption printed

A real run

servefit fit --model Qwen/Qwen3-8B on a synthetic RTX 4090 profile:

Memory math: weights 15.3 + KV 4.5 + overhead 1.0 = 20.8 GiB
  dtype bf16, ctx 32768, layers 36, kv-heads 8, head-dim 128

RECOMMENDED: SGLang
  + chip: compute_cap 8.9 >= 7.5
  + model: qwen family supported
  + memory: needs ~20.8 GiB, single-GPU VRAM has 24.0 GiB (fits)

Not viable:
  [no] DeepInfer
       chip: needs NVIDIA SM100 (Blackwell) or Ascend 950
       model: qwen not in supported families (deepseek-v4.1-flash-only)

The demo transcript labels what is synthetic (inventory runs against a synthetic RTX 4090 profile) and what is scripted (the probe runs against a local fake OpenAI-compatible server); the timing it measures is this machine's, honestly labeled.

Method — stated honestly

Memory math: weights = params × bytes-per-param; KV cache = 2 × layers × kv_heads × head_dim × context × bytes-per-param; plus a stated overhead line. Every assumption is printed, so you can argue with it. Hand-computed fixtures in the test suite (8B bf16: 14.9 GiB weights, 4.5 GiB KV @ 32k, 20.4 GiB total) all match; 37 tests, all green, CI across Python 3.10–3.13. The engine matrix is the opinionated part: a preference order (SGLang > vLLM > TensorRT-LLM > DeepInfer > llama.cpp > Ollama) gated by chip, model-family, and memory rules — and the matrix is dated: every row carries a check date (all 2026-11-01 at release), and CI fails if any row goes 90 days without re-verification, so staleness is loud, not silent.

What this is NOT — and where rivals beat us

It does not download models and does not provision GPUs.
A "fits" verdict is not a production-readiness certificate. Missing production components (load balancing, autoscaling, monitoring) are listed per engine instead of hidden — your deployment README still needs them.
It does not replace profiling. Memory math gets you to the right engine; real throughput comes from measuring your workload. Probe numbers are your machine's under your conditions, not a benchmark claim about the engine (probe tokens are a whitespace-split approximation, labeled in the output — not the model's tokenizer).
Apple Silicon: GPU VRAM is reported unknown by design; recommendations lean on llama.cpp/Ollama with a manual RAM-check note.
The matrix is a snapshot. Hardware moves fast; every recommendation carries its check date. Re-verify before spending money.
Honest comparison (checked 2026-11-01): Hugging Face model pages and hub calculators win on data freshness and model coverage — every new model's arch is there the day it drops; servefit only maintains a curated engine matrix. Per-engine docs (vLLM, SGLang, TensorRT-LLM) win on depth: the full config surface, performance tuning, multi-node guides; servefit gives one starter config, not a manual. DeepInfer wins on the exact thing it was built for: serving DeepSeek-V4.1-Flash fast on SM100/Ascend hardware — servefit's matrix correctly rules it out for everything else, and that is the point of the map.

Source + README on GitHub → · Report an issue → · ask@techdev.fun

Roadmap

More engines, added as rows with fresh check dates
Multi-GPU/NUMA topologies (TP/PP splits, not just total pooling)
Concurrency-aware KV math (per-request sequences, not just single-sequence)
servefit compare --models a,b side-by-side fit table
servefit watch mode that re-checks the matrix on a schedule

License

Apache-2.0. See LICENSE in the repository.

Have a say

Vote for this tool, suggest a feature, or discuss it with other operators on the community board.