Tibério Konate Technical portfolio
← All projects

Tooling · Benchmark · 2026

ollama-inference-benchmarks

A reproducible test bench for choosing a local model on measured criteria instead of gut feeling.

View repository ↗

The problem

Everyone claims their model is "fast". On a GPU-less VPS, latency depends on the model, the quantization and the output length. Without a reproducible protocol the choice is made on a feeling — and the first run always lies, because the model is not loaded yet.

What I built

A Typer CLI with Pydantic models, a nine-prompt suite across three categories (short reasoning, structured extraction, long text), cold/warm latency measurement via `--runs`, two swappable backends (llama.cpp and vLLM through an OpenAI-compatible interface) and CSV export. Shipped with a Docker image, GitHub Actions CI, 43 tests, strict mypy and clean ruff.

Technical points

  • Backends are swappable behind one interface: changing engine costs a subclass.
  • The cold/warm distinction is measured, not assumed — it is what invalidates most published benchmarks.
  • Concrete outcome: Qwen3 1.7B gives the best quality/latency trade-off on this hardware.