← All projects
Tooling · Benchmark · 2026
ollama-inference-benchmarks
A reproducible test bench for choosing a local model on measured criteria instead of gut feeling.
The problem
Everyone claims their model is "fast". On a GPU-less VPS, latency depends on the model, the quantization and the output length. Without a reproducible protocol the choice is made on a feeling — and the first run always lies, because the model is not loaded yet.
What I built
A Typer CLI with Pydantic models, a nine-prompt suite across three categories (short reasoning, structured extraction, long text), cold/warm latency measurement via `--runs`, two swappable backends (llama.cpp and vLLM through an OpenAI-compatible interface) and CSV export. Shipped with a Docker image, GitHub Actions CI, 43 tests, strict mypy and clean ruff.
Technical points
- Backends are swappable behind one interface: changing engine costs a subclass.
- The cold/warm distinction is measured, not assumed — it is what invalidates most published benchmarks.
- Concrete outcome: Qwen3 1.7B gives the best quality/latency trade-off on this hardware.