tiny-llm (a course on deploying large language models (LLMs)) is designed for macOS users; it is not very user-friendly for Windows-only users. This repository has been adapted to work with a specific version of PyTorch, making it easier for Windows-only users to use.
tiny-llm (A course on LLM serving) 适合 macos 玩家食用，作为纯 win 党不便于学习。 本仓适配了一个 torch 版本，便于纯 win 党玩耍。
- run inference
- without kvcache
pdm run main --solution myllm --loader week1 --model qwen2-0.5b --prompt "Give me a short introduction to large language model"- with kvcache
pdm run main --solution myllm --loader week2 --model qwen2-0.5b --prompt "Give me a short introduction to large language model"- with flash-attn
pdm run main --solution myllm --loader week2 --model qwen2-0.5b --enable-flash-attn --prompt "Give me a short introduction to large language model"- run one test case
pdm run pytest test/test_week_1_day_1.py::test_task_1_simple_attention- run all test cases
pdm run pytest test/test_week_1_day_1.py -x