Skip to content

1-Build-Paths.md

Latest commit

 

History

History
250 lines (190 loc) · 5.78 KB

1-Build-Paths.md

File metadata and controls

250 lines (190 loc) · 5.78 KB
Raw

BUILD PATHS

There are 3 Paths you can take in pipeline

Fastest start: Path A on any machine with Python 3.10+ — Only needs pip install -e ".[all]" and clang++.

  • No GPU, no CUDA, no TensorRT required until Path C.

Pipeline Flow

flowchart TD
    SETUP["🔧 Setup\nclone · venv\npip install -e '.[all]'"]
    S0["📥 Stage 0 · Import\nHuggingFace · PyTorch · GGUF\nSafeTensors · Ollama · NGC"]
    S1["📦 Stage 1 · Export → ONNX\ntorch.onnx.export  opset 17\ndynamic axes · shape inference"]
    S2["🔐 Stage 2 · Optimize\nFP16 · INT8 · FP8 E4M3 · W4A16 GPTQ"]

    SETUP --> S0 --> S1 --> S2

    S2 -->|no GPU needed| A3
    S2 -->|CUDA GPU| B3
    S1 -->|GPU + TensorRT| C1b

    subgraph PA ["Path A — 💻 CPU only"]
        A3["🏋️ Stage 3\nclang++ · CPU backend\nno CMake · no deps"]
        A4["📊 Stage 4\nBenchmark\n--backends cpu onnx"]
        A3 --> A4
    end

    subgraph PB ["Path B — 🟦 ONNX Runtime + CUDA EP"]
        B3["🏋️ Stage 3\nCMake\n-DWITH_ONNXRUNTIME=ON"]
        B4["📊 Stage 4\nBenchmark\n--backends cpu onnx"]
        B3 --> B4
    end

    subgraph PC ["Path C — 🟥 TensorRT  (full GPU)"]
        C1b["📦 Stage 1b\nBuild TRT engine\ntensorrt_builder  --precision fp16"]
        C3["🏋️ Stage 3\nCMake\n-DWITH_ONNXRUNTIME=ON\n-DWITH_TENSORRT=ON"]
        C4["📊 Stage 4\nBenchmark\n--backends cpu onnx trt"]
        C1b --> C3 --> C4
    end

    A4 --> S5
    B4 --> S5
    C4 --> S5

    S5["🏆 Stage 5 · Dashboard\nreport.html · SVG charts · Markdown table"]

    style SETUP fill:#1a1f1a,stroke:#4a7a30,color:#c8d4c8
    style S0    fill:#1a1f1a,stroke:#4a7a30,color:#c8d4c8
    style S1    fill:#1a1f1a,stroke:#4a7a30,color:#c8d4c8
    style S2    fill:#1a1f1a,stroke:#4a7a30,color:#c8d4c8
    style S5    fill:#2e2800,stroke:#c8a010,color:#e8d070
    style PA    fill:#1e2a1e,stroke:#4a7a30,color:#c8d4c8
    style PB    fill:#1a1e2e,stroke:#3a5a9a,color:#c8d4c8
    style PC    fill:#2e1a1a,stroke:#9a3a3a,color:#c8d4c8
Loading

Path A — 💻 CPU only (no GPU, runs anywhere)

Clone & set up Python env

git clone https://github.com/HiteshSahu/Model-Gym && cd Model-Gym

python3 -m venv .venv && source .venv/bin/activate
pip install -e ".[all]"

Stage 0 — Import model from HuggingFace

python -m src.importer.huggingface \
--model bert-base-uncased \
--task  text-classification

Stage 1 — Export to ONNX

python -m src.export.onnx_exporter \
--model  bert-base-uncased \
--output models/bert.onnx \
--opset  17

Stage 2 — Optimize (two variants, no GPU needed)

python -m src.optimize.fp16 \
--input  models/bert.onnx \
--output models/bert_fp16.onnx
python -m src.optimize.int8 \
--input  models/bert.onnx \
--output models/bert_int8.onnx \
--mode   dynamic

Stage 3 — Build C++ inference engine (CPU backend, no CMake)

clang++ -std=c++17 -O2 -I inference/include \
inference/src/core/tensor.cpp \
inference/src/backends/cpu_backend.cpp \
inference/src/main.cpp -o model_gym

Run it

./model_gym \
    --backend cpu \
    --hidden 768 \
    --output 1000 \
    --batch 4 \
    --seq-len 128

Stage 4 — Python benchmark harness (cpu + onnx backends)

mkdir -p results
python -m src.benchmark.harness \
--backends   cpu onnx \
--model      models/bert.onnx \
--batch-sizes 1 8 32 \
--warmup     5 \
--iters      50 \
--out        results/bench.json

Stage 5 — Dashboard

mkdir -p dashboard/output
python -m src.dashboard.report \
--results results/bench.json \
--out     dashboard/output \
--format  html markdown console

Output in dashboard/output/: report.html, report.md, and 4 SVG charts.

Path B — 🧮 With GPU + ONNX Runtime CUDA EP

Same as Path A, but swap step Stage 3 — & Step 4

Stage 3 — CMake build with ONNX Runtime

cmake -B build -DCMAKE_BUILD_TYPE=Release -DWITH_ONNXRUNTIME=ON \
-DONNXRUNTIME_ROOT=/path/to/onnxruntime
cmake --build build -j$(nproc)

Run with CUDA EP

./build/inference/model_gym \
--backend onnx \
--model   models/bert.onnx \
--cuda \
--batch 4 --seq-len 128

Stage 4 — benchmark all three backends

python -m src.benchmark.harness \
--backends cpu onnx \
--model    models/bert.onnx \
--batch-sizes 1 8 32 \
--out      results/bench_gpu.json

Path C: 🧮 Full GPU pipeline with TensorRT

Stage 1 — build TRT engine (requires GPU + TRT installed)

python -m src.export.tensorrt_builder \
--input       models/bert.onnx \
--output      models/bert_fp16.engine \
--precision   fp16 \
--workspace-gb 4

Stage 3 — CMake build with TensorRT

cmake -B build -DCMAKE_BUILD_TYPE=Release \
-DWITH_ONNXRUNTIME=ON -DWITH_TENSORRT=ON \
-DTENSORRT_ROOT=/path/to/TensorRT
cmake --build build -j$(nproc)

Run TRT backend

./build/inference/model_gym \
--backend trt \
--engine  models/bert_fp16.engine \
--batch 4 --seq-len 128

Stage 4 — benchmark all backends

python -m src.benchmark.harness \
--backends cpu onnx trt \
--model    models/bert.onnx \
--engine   models/bert_fp16.engine \
--batch-sizes 1 8 32 \
--out      results/bench_trt.json

Stage 5 — dashboard (same command regardless of backends)

python -m src.dashboard.report \
--results results/bench_trt.json \
--out     dashboard/output \
--format  html markdown console

C++ micro-benchmarks 📟

optional, separate from Stage 4

cmake -B build -DCMAKE_BUILD_TYPE=Release && cmake --build build --target bench_inference
./build/benchmark/cpp/bench_inference --benchmark_format=json > results/cpp_bench.json

© 2026 Hitesh Kumar Sahu · Licensed under Apache 2.0