Skip to content

bench: add generic runner and M1 Max Qwen3.6 results#28

Open
kira-ariaki wants to merge 1 commit into
basecompute:mainfrom
kira-ariaki:contrib/m1max-bench
Open

bench: add generic runner and M1 Max Qwen3.6 results#28
kira-ariaki wants to merge 1 commit into
basecompute:mainfrom
kira-ariaki:contrib/m1max-bench

Conversation

@kira-ariaki

Copy link
Copy Markdown

Summary

  • add the generic benchmarks/scripts/run_bench.sh entrypoint already referenced by the benchmark docs and manual workflow
  • parse basert-bench output fail-closed into normalized CSV with engine-free tests
  • run benchmark-tool checks in CI
  • publish BaseRT 0.1.7 Qwen3.6-35B-A3B default-q4 results from an M1 Max 64 GB
  • preserve model/source SHA-256 provenance, converter details, and complete raw second-run output
  • clarify independent prefill/decode microbenchmark semantics

Results contributed

Five measured repetitions after one warm-up iteration:

  • pp128: 552.07 tok/s
  • pp256: 692.73 tok/s
  • pp512: 851.39 tok/s
  • pp2048: 1025.34 tok/s
  • pp6000: 878.60 tok/s
  • pp43000: 574.93 tok/s
  • tg128: 95.99–97.15 tok/s across the six independent invocations

These are single-stream engine microbenchmarks, not server throughput or decode-after-prefill measurements.

Provenance

  • source revision: Qwen/Qwen3.6-35B-A3B@995ad96eacd98c81ed38be0c5b274b04031597b0
  • conversion: basert pull Qwen/Qwen3.6-35B-A3B, base-convert v0.1.6, profile default-q4
  • .base SHA-256: 74c4a9256e64971e919c41b78b03ff8e0ec74b4330058113f5785097706ed1d0
  • BaseRT 0.1.7 basert-bench SHA-256: f81d0ddd3c3eae1ff43f734aa0c76786b8b2e9b9fe3e1c5ec28f8d581a972721

Verification

  • python3 -m unittest discover -s benchmarks/tests -v — 5/5 passed
  • ruff check benchmarks/scripts/parse_bench.py benchmarks/tests
  • python3 -m py_compile benchmarks/scripts/parse_bench.py benchmarks/tests/*.py
  • bash -n benchmarks/scripts/run_bench.sh
  • reran the complete six-point sweep end-to-end through the new runner; all throughput values reproduced within 1.2%, and the committed CSV/raw output record that second run

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant