Skip to content

fix: use engine-reported model metadata; make roofline honest (v1.1.4) - #11

Merged
Prajwal7-2002 merged 1 commit into
mainfrom
fix/real-model-metadata
Nov 5, 2026
Merged

Prajwal7-2002 merged 1 commit into
mainfrom
fix/real-model-metadata

Conversation

@Prajwal7-2002

Copy link
Copy Markdown
Owner

Summary

servebench's core measurements are accurate (Ollama qwen2.5:0.5b: 201 tok/s and 43 ms TTFT measured vs Ollama's own 187-221 tok/s). Its derived analyses, though, relied on guesses. This is groundwork for using servebench results in a public benchmark database, where every number must be measured or clearly labelled.

Fixed

  • Metadata missing from exports: detection ran once before the run; a server still starting up left engine and model_info as null (true for every export from a day of sweeps). servebench now re-detects after the run.
  • Real model size: Ollama (/api/show, /api/tags) and llama.cpp (/v1/models meta) report the exact parameter count and weights size → model_info.parameter_count, model_info.weights_gb. LM Studio reports quantization and architecture.
  • servebench roofline:
    • uses the engine-reported size instead of guessing from the name, which counted 4-bit Ollama models as 16-bit;
    • no longer invents prompt throughput (was 20-25% of output);
    • reports MBU/MFU above 100% instead of clamping them, with a note;
    • flags unknown GPUs and name-based guesses;
    • requires a run file instead of printing made-up "representative" numbers.
  • GPU table: RTX 3050 split into 6 GB laptop (168 GB/s) / 4 GB laptop (192 GB/s) / desktop (224 GB/s, 36.4 TFLOPS); RTX 4050/4060/4070/4070 Ti/4080/4080 Super moved from sparsity to dense TFLOPS.
  • mini-infer's version is no longer a hard-coded "0.1.0".
  • README: the roofline example is now real output; it previously showed specs matching no table entry. Version 1.1.4.

Test verification

  • New tests:
    • Ollama metadata, with per-endpoint fakes (the old test stubbed every URL with the same response and didn't stub /api/show);
    • llama.cpp and LM Studio metadata;
    • mini-infer version;
    • detection retry after the run;
    • engine size vs name guess;
    • 100% reporting, prompt throughput not invented, unknown-GPU flag, CLI requires a run file.

  • The 4 new detection tests fail on the old code, and pass here.
  • Full suite: 161 passed (was 152); ruff clean.

Live check (Ollama 0.35.1, RTX 3050 6GB laptop):

  • The export now records parameter_count: 494032768, quantization: Q4_K_M, weights_gb: 0.398 and on_gpu: 100%.
  • roofline on a 1/2/4-user sweep (189 / 310 / 403 tok/s) gives MBU 44.8% / 36.7% / 23.8%, matching the hand calculation (tok/s × 0.398 GB ÷ users ÷ 168 GB/s). The old name guess would have exceeded 100% and been clamped.

Not tested by hand yet: llama.cpp server and LM Studio metadata, which are covered only by fake-server tests.

- re-detect the engine after a run when the server was not ready at
  start (exports had engine/model_info null)
- read exact parameter count and weights size from Ollama (/api/show,
  /api/tags) and llama.cpp (/v1/models meta); quantization/arch from
  LM Studio
- roofline: use engine-reported size instead of the name guess (4-bit
  Ollama models were counted as 16-bit), stop inventing prompt
  throughput, report MBU/MFU > 100% instead of clamping, flag unknown
  GPUs and guesses, require a run file instead of made-up examples
- GPU table: split RTX 3050 6GB/4GB laptop/desktop; RTX 40-series dense
  TFLOPS instead of sparsity figures
- stop reporting a hard-coded mini-infer version
- README: real roofline output from an Ollama sweep; bump to 1.1.4
@Prajwal7-2002
Prajwal7-2002 merged commit fbbc9d1 into main Nov 5, 2026
12 of 13 checks passed
@Prajwal7-2002
Prajwal7-2002 deleted the fix/real-model-metadata branch November 5, 2026 18:10
Prajwal7-2002 added a commit that referenced this pull request Nov 5, 2026
TTFT steps of 40 ms against a 150 ms target left 30 ms of headroom at 3
users; a slow macOS runner turned 3 users into 2 (failed once on PR #11).
Use 200 ms steps and a 700 ms target: 100 ms margin on each side.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant