Roy WangandClaude Opus 5.5
b515648641
Signed-off-by: esmeetu <jasonailu87@gmail.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
250 lines
7.3 KiB
YAML
250 lines
7.3 KiB
YAML
meta:
|
||
title: "Qwen3-32B"
|
||
slug: "qwen3-32b"
|
||
provider: "Qwen"
|
||
description: "Qwen3 32B dense model with hybrid thinking/non-thinking modes — verified on TPU v6e (Trillium)."
|
||
date_added: 2025-10-15
|
||
date_updated: 2026-09-10
|
||
tasks:
|
||
- text
|
||
performance_headline: "Verified on TPU v6e (Trillium) and v7 (Ironwood) with BF16"
|
||
related_recipes:
|
||
- "Qwen/Qwen3-4B"
|
||
hardware:
|
||
trillium: verified
|
||
ironwood: verified
|
||
arc_pro_b60: verified
|
||
arc_pro_b70: verified
|
||
|
||
model:
|
||
model_id: "Qwen/Qwen3-32B"
|
||
min_vllm_version: "0.8.5"
|
||
architecture: dense
|
||
parameter_count: "32B"
|
||
active_parameters: "32B"
|
||
context_length: 40960
|
||
base_args: []
|
||
base_env: {}
|
||
|
||
features:
|
||
tool_calling:
|
||
description: "Enable automatic tool choice with the Hermes parser"
|
||
args:
|
||
- "--enable-auto-tool-choice"
|
||
- "--tool-call-parser"
|
||
- "hermes"
|
||
reasoning:
|
||
description: "Enable Qwen3 thinking mode (chain-of-thought) with the Qwen3 reasoning parser"
|
||
args:
|
||
- "--reasoning-parser"
|
||
- "qwen3"
|
||
long_context:
|
||
description: "YaRN RoPE scaling to the model card's validated 131,072-token context. Static YaRN holds the scaling factor constant regardless of input length, so leave this off unless you actually serve long prompts — it can cost quality below 32K."
|
||
args:
|
||
- "--hf-overrides"
|
||
- '{"max_position_embeddings":131072,"rope_parameters":{"rope_type":"yarn","factor":4.0,"original_max_position_embeddings":32768}}'
|
||
- "--max-model-len"
|
||
- "131072"
|
||
|
||
spec_decoding:
|
||
description: "Eagle3 speculative decoding with externally trained draft model; the served checkpoint is unchanged."
|
||
args:
|
||
- "--speculative-config"
|
||
- '{"model": "RedHatAI/Qwen3-32B-speculator.eagle3", "num_speculative_tokens": 3, "method": "eagle3"}'
|
||
|
||
opt_in_features:
|
||
- spec_decoding
|
||
- long_context
|
||
|
||
variants:
|
||
default:
|
||
precision: bf16
|
||
vram_minimum_gb: 77
|
||
description: "Full precision BF16 — TPU v6e 2x2 (4 chips, TP=4) or a single 80GB+ GPU"
|
||
fp8:
|
||
model_id: "Qwen/Qwen3-32B-FP8"
|
||
precision: fp8
|
||
vram_minimum_gb: 39
|
||
description: "Qwen official FP8 checkpoint — single 40 GB GPU"
|
||
awq:
|
||
model_id: "Qwen/Qwen3-32B-AWQ"
|
||
precision: int4
|
||
vram_minimum_gb: 20
|
||
description: "AWQ 4-bit quantized weights — fits on a single 24 GB GPU"
|
||
extra_args:
|
||
- "--quantization"
|
||
- "awq"
|
||
|
||
redhat_fp8:
|
||
model_id: "RedHatAI/Qwen3-32B-FP8-dynamic"
|
||
precision: fp8
|
||
label: "FP8 Dynamic (RedHat)"
|
||
vram_minimum_gb: 39
|
||
description: "RedHatAI FP8 (E4M3) with dynamic per-token activation quantization"
|
||
extra_args:
|
||
- "--kv-cache-dtype"
|
||
- "fp8"
|
||
|
||
redhat_fp8_block:
|
||
model_id: "RedHatAI/Qwen3-32B-FP8-block"
|
||
precision: fp8
|
||
label: "FP8 Block (RedHat)"
|
||
vram_minimum_gb: 39
|
||
description: "RedHatAI FP8 block quantization"
|
||
extra_args:
|
||
- "--kv-cache-dtype"
|
||
- "fp8"
|
||
|
||
nvfp4:
|
||
model_id: "RedHatAI/Qwen3-32B-NVFP4"
|
||
precision: nvfp4
|
||
vram_minimum_gb: 22
|
||
description: "RedHatAI NVFP4 quantization"
|
||
|
||
redhat_nvfp4:
|
||
model_id: "RedHatAI/Qwen3-32B-NVFP4A16"
|
||
precision: nvfp4
|
||
label: "NVFP4 A16 (RedHat)"
|
||
vram_minimum_gb: 22
|
||
description: "RedHatAI NVFP4 with BF16 activations"
|
||
|
||
compatible_strategies:
|
||
- single_node_tp
|
||
- multi_node_tp
|
||
|
||
kv_offload_support:
|
||
offloading_cpu: verified
|
||
offloading_fs: verified
|
||
|
||
hardware_overrides:
|
||
xpu:
|
||
extra_args:
|
||
- "--gpu-memory-utilization"
|
||
- "0.90"
|
||
|
||
strategy_overrides: {}
|
||
|
||
guide: |
|
||
## Overview
|
||
|
||
Qwen3-32B is the flagship dense model in the Qwen3 series, with hybrid thinking / non-thinking modes. BF16 is the training precision and gives the best accuracy.
|
||
|
||
Toggle thinking mode per request with `enable_thinking` in `chat_template_kwargs`, or set the **Reasoning** feature to add the Qwen3 reasoning parser so `<think>` content is split into a separate field.
|
||
|
||
## Prerequisites
|
||
|
||
### NVIDIA
|
||
```bash
|
||
uv venv
|
||
source .venv/bin/activate
|
||
uv pip install -U vllm --torch-backend auto
|
||
```
|
||
|
||
### TPU (Trillium / v6e)
|
||
Use the official vLLM TPU image. The 32B model is served on a `v6e-4` slice (`--topology 2x2`, 4 chips, TP=4).
|
||
```bash
|
||
export HF_HOME=/dev/shm
|
||
export HF_TOKEN=<your HF token>
|
||
```
|
||
|
||
### Intel XPU (B60 / B70)
|
||
Use the official vLLM XPU image.
|
||
```bash
|
||
docker pull vllm/vllm-openai-xpu:latest
|
||
```
|
||
|
||
## Deployment Configurations
|
||
|
||
### TPU v6e (Trillium, 2x2 slice, TP=4)
|
||
Verified end-to-end on a 4-chip v6e slice.
|
||
```bash
|
||
export HF_HOME=/dev/shm
|
||
export HF_TOKEN=<your HF token>
|
||
export TP=4
|
||
export MAX_MODEL_LEN=4096
|
||
vllm serve Qwen/Qwen3-32B \
|
||
--seed 42 \
|
||
--gpu-memory-utilization 0.98 \
|
||
--max-num-batched-tokens 2048 \
|
||
--max-num-seqs 256 \
|
||
--tensor-parallel-size $TP \
|
||
--max-model-len $MAX_MODEL_LEN
|
||
```
|
||
|
||
### TPU v7 (Ironwood, single host, TP=2)
|
||
Verified end-to-end on a single Ironwood host (`tpu7x-standard-1t`, topology `1x1x1`). Served via the OpenAI API server entrypoint.
|
||
```bash
|
||
export HF_HOME=/data
|
||
export HUGGING_FACE_HUB_TOKEN=<your HF token>
|
||
export LAYOUT_Q_PROJ_AS_NDH=1
|
||
export USE_BATCHED_RPA_KERNEL=1
|
||
vllm serve Qwen/Qwen3-32B \
|
||
--host 0.0.0.0 \
|
||
--port 8000 \
|
||
--seed 42 \
|
||
--tensor-parallel-size 2 \
|
||
--data-parallel-size 1 \
|
||
--max-model-len 9216 \
|
||
--max-num-batched-tokens 4096 \
|
||
--max-num-seqs 135 \
|
||
--gpu-memory-utilization 0.98 \
|
||
--block-size 256 \
|
||
--kv-cache-dtype fp8 \
|
||
--no-enable-prefix-caching \
|
||
--async-scheduling
|
||
```
|
||
> Ironwood support ships in the nightly TPU image (e.g. `vllm/vllm-tpu:nightly-20260330-2f76400-8c0b626`).
|
||
|
||
### Single node (BF16, TP=2)
|
||
```bash
|
||
vllm serve Qwen/Qwen3-32B \
|
||
--host 0.0.0.0 \
|
||
--port 8000 \
|
||
--tensor-parallel-size 2
|
||
```
|
||
|
||
### Intel XPU (B60, B70)
|
||
|
||
Validated on 4× Intel Arc Pro B60 / B70 (B60 24 GB, B70 32 GB per card) with the
|
||
official vLLM XPU image `vllm/vllm-openai-xpu:latest`.
|
||
|
||
```bash
|
||
docker run --device /dev/dri \
|
||
-v /dev/dri/by-path:/dev/dri/by-path --shm-size=16g \
|
||
--privileged --ipc=host -p 8000:8000 \
|
||
-v ~/.cache/huggingface:/root/.cache/huggingface \
|
||
vllm/vllm-openai-xpu:latest \
|
||
Qwen/Qwen3-32B \
|
||
--gpu-memory-utilization 0.90 \
|
||
--tensor-parallel-size 4 \
|
||
--max-model-len 8192
|
||
```
|
||
|
||
## Configuration Tips
|
||
|
||
- `--max-model-len 4096` keeps memory in check on a 4-chip v6e slice; native context is 40K — raise it if you have headroom.
|
||
- vLLM uses 90% of device memory by default; the TPU recipe pushes `--gpu-memory-utilization 0.98` to maximize KV cache.
|
||
- `--max-num-batched-tokens` and `--max-num-seqs` tune batch throughput vs latency.
|
||
- For thinking mode Qwen recommends `temperature=0.6, top_p=0.95, top_k=20`; for non-thinking, `temperature=0.7, top_p=0.8`.
|
||
|
||
## Benchmarking
|
||
|
||
Launch the server with `--no-enable-prefix-caching` for consistent measurements.
|
||
|
||
```bash
|
||
vllm bench serve \
|
||
--host 0.0.0.0 \
|
||
--port 8000 \
|
||
--model Qwen/Qwen3-32B \
|
||
--dataset-name random \
|
||
--random-input-len 1024 \
|
||
--random-output-len 1024 \
|
||
--num-prompts 128
|
||
```
|
||
|
||
## References
|
||
|
||
- [Model card](https://huggingface.co/Qwen/Qwen3-32B)
|
||
- [Qwen3 on Trillium (v6e)](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/trillium/vLLM/Qwen3)
|
||
- [Qwen3-32B on Ironwood (v7)](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/ironwood/vLLM/Qwen3-32B)
|