Add vLLM optimization guide and skill#32
Conversation
|
|
||
| - [Cloud Intel Xeon AI Performance Advisor](https://xeonprocessoradvisor.intel.com/csp-ai-performance-advisor) | ||
| - [On-prem Intel Xeon AI Performance Advisor](https://xeonprocessoradvisor.intel.com/on-prem-ai-performance-advisor) | ||
| - [Intel AI Software Catalog - Model Guidance](https://swcatalog.intel.com/models) |
There was a problem hiding this comment.
First two requires end user to provide email address. Lets move the third bullet to the top, since that link has direct access to content.
| @@ -0,0 +1,312 @@ | |||
| # vLLM on Intel Xeon Processors | |||
There was a problem hiding this comment.
Hi, Thanks for the contribution!
Consider adding a table of contents and breaking down the document into sections/subsections.
There are three broad topics being covered -
vLLM on Xeon guidance,
AI agent skill installation and benchmarking workflow.
Probably we can move the "Use with AI Coding Agents" towards the end, so we don't confuse and redirect someone who lands on the page for immediate tuning guidance.
| | `--max-num-batched-tokens` | Online: `2048`; offline: `4096` | Maximum number of batched tokens per iteration. Tune for prefill throughput and time to first token. | | ||
| | `--max-num-seqs` | Online: `128`; offline: `256` | Maximum number of sequences per iteration. Tune for decode throughput and inter-token latency. | | ||
| | `VLLM_CPU_SGL_KERNEL` | `0`, or try `1` for low-latency SLM serving | Experimental x86 small-batch kernels; requires AMX, BF16 weights, and compatible shapes. | | ||
|
|
There was a problem hiding this comment.
Can you elaborate if the above performance guidance settings are environment settings? or some flags/arguments for a benchmark?
| | OS | Linux | | ||
| | Python | 3.10 through 3.13 | | ||
| | vLLM | `v0.17.0` or newer | | ||
| | Intel AMX Xeon CPU Flags | 4th Gen or newer with `amx_tile`, `amx_bf16`, and `amx_int8` for best BF16/INT8 performance | |
There was a problem hiding this comment.
| | Intel AMX Xeon CPU Flags | 4th Gen or newer with `amx_tile`, `amx_bf16`, and `amx_int8` for best BF16/INT8 performance | | |
| | Intel AMX related Xeon CPU Flags | 4th Gen Intel Xeon or newer with `amx_tile`, `amx_bf16`, and `amx_int8` for best BF16/INT8 performance | |
|
|
||
| | Setting | Guidance | Why it matters | | ||
| | --- | --- | --- | | ||
| | `--dtype=bfloat16` | Use `bfloat16` on Intel Xeon with Intel AMX | Enables the preferred CPU dtype and AMX | |
There was a problem hiding this comment.
Does this enable the preferred CPU dtype and enables Intel AMX acceleration?
| | `--dtype=bfloat16` | Use `bfloat16` on Intel Xeon with Intel AMX | Enables the preferred CPU dtype and AMX | | ||
| | `VLLM_CPU_KVCACHE_SPACE` | `20` to `40` GiB or larger | Larger values allow more concurrency and context, but must fit per NUMA node. | | ||
| | `VLLM_CPU_OMP_THREADS_BIND` | `auto` | Binds OpenMP worker threads to NUMA-local cores. Use ranges such as `0-31\|32-63` for manual control, `auto` preferred. | | ||
| | `VLLM_CPU_NUM_OF_RESERVED_CPU` | `1` | Sets a core for API serving, tokenization, networking, logging, and OS work. | |
There was a problem hiding this comment.
| | `VLLM_CPU_NUM_OF_RESERVED_CPU` | `1` | Sets a core for API serving, tokenization, networking, logging, and OS work. | | |
| | `VLLM_CPU_NUM_OF_RESERVED_CPU` | `1` | Reserves one core for API serving, tokenization, networking, logging, and OS work. | |
| | `VLLM_CPU_KVCACHE_SPACE` | `20` to `40` GiB or larger | Larger values allow more concurrency and context, but must fit per NUMA node. | | ||
| | `VLLM_CPU_OMP_THREADS_BIND` | `auto` | Binds OpenMP worker threads to NUMA-local cores. Use ranges such as `0-31\|32-63` for manual control, `auto` preferred. | | ||
| | `VLLM_CPU_NUM_OF_RESERVED_CPU` | `1` | Sets a core for API serving, tokenization, networking, logging, and OS work. | | ||
| | `--tensor-parallel-size` | Use default for single NUMA or set to NUMA node count | Keeps model shards close to local memory; current vLLM CPU releases do not support `6`. | |
There was a problem hiding this comment.
| | `--tensor-parallel-size` | Use default for single NUMA or set to NUMA node count | Keeps model shards close to local memory; current vLLM CPU releases do not support `6`. | | |
| | `--tensor-parallel-size` | Use the default for a single NUMA node, or set it to the NUMA node count | Keeps model shards close to local memory; current vLLM CPU releases do not support --tensor-parallel-size=6. | |
|
|
||
| This summarizes the official benchmarking and tuning guidance from the vLLM documentation, with a CPU focus. Always consult the [official benchmarking docs](https://docs.vllm.ai/en/latest/benchmarking/cli/) for the latest recommendations and tools. | ||
|
|
||
| Start the Docker. If it is running in the foreground, open another terminal for these checks: |
There was a problem hiding this comment.
| Start the Docker. If it is running in the foreground, open another terminal for these checks: | |
| Start the Docker container. If it is running in the foreground, open another terminal for these checks: |
| Start the Docker. If it is running in the foreground, open another terminal for these checks: | ||
|
|
||
| ```bash | ||
| # Docker path, because the server above is named vllm-cpu. |
There was a problem hiding this comment.
| # Docker path, because the server above is named vllm-cpu. | |
| # Docker path, because the container above is named vllm-cpu. |
| ``` | ||
|
|
||
| ### Benchmark | ||
|
|
There was a problem hiding this comment.
Can you add a few lines on what is vllm bench serve before the docker commands. Is it a subset of the Benchmark Suite you have there later?
Also, the execution context changes frequently between host/native install, docker container, virtualenv. Someone may land up running host command in container or vice versa.
For example, Concurrency sweep is provided for native environment.
Also, are the utility tools, h/w validation etc. applicable before running the vLLM benchmark suite as well? or just for vllm bench serve?
Once you have sections/subsections hierarchy it will be more clear.
2 participants
This pull request adds comprehensive support and documentation for running vLLM (an efficient LLM inference engine) on Intel Xeon CPUs. It introduces a new vLLM section to the main documentation, provides a detailed user guide for vLLM CPU deployment and tuning, and supplies an AI agent skill for automating vLLM setup and benchmarking on Xeon systems.
The most important changes are:
Documentation and User Guidance:
README.md, making vLLM CPU guidance discoverable alongside other AI workloads.software/vllm/README.md, a thorough guide covering installation, hardware validation, Docker-based deployment, performance tuning, benchmarking, and troubleshooting for vLLM on Intel Xeon CPUs. This guide includes practical tips, recommended settings, and links to upstream documentation.AI Agent Skill for Automation:
software/vllm/skill/SKILL.md, an agent skill enabling AI coding agents (like GitHub Copilot and Claude Code) to automate vLLM deployment, tuning, hardware validation, and benchmarking on Xeon CPUs. The skill provides step-by-step procedures and best practices, and is designed for easy integration with agent workflows.