diff --git a/docs/features/quantization/quark.md b/docs/features/quantization/quark.md index 8ed6b2ca60..8445288ff9 100644 --- a/docs/features/quantization/quark.md +++ b/docs/features/quantization/quark.md @@ -314,3 +314,82 @@ lm_eval --model vllm \ --tasks mmlu \ --batch_size auto ``` + +## Online Quantization + +All the workflows above are *offline* quantization: you run a script, write a new quantized checkpoint to disk, and later load it for serving. This produces the most accurate and smallest checkpoints because it can run accuracy-recovery algorithms (rotation, SmoothQuant, GPTQ, AWQ) during export, but it requires a separate quantization step and a second copy of the model on disk. + +*Online* quantization instead quantizes the weights at load time, directly from a high-precision checkpoint, and offers several advantages over the offline flow: + +- **No export step** — serve directly from the original `bf16`/`fp16` checkpoint; no separate quantization run before deployment. +- **No extra disk footprint** — nothing new is written to disk, so there is no second copy of the model to store or manage. +- **No calibration data** — activations are scaled dynamically at runtime, so no calibration dataset is needed. +- **Fast iteration** — switch schemes or per-layer selections instantly by changing a config, without re-exporting. + +### vLLM online quantization + +vLLM has [built-in online quantization](online.md) that skips the export step: it loads an ordinary high-precision (`bf16`) checkpoint and quantizes each layer's weights to schemes such as FP8 or MXFP4 at load time, without a pre-quantized checkpoint or calibration data. No new checkpoint is written to disk. See [Online Quantization](online.md) for the supported schemes and configuration. + +### Quark online quantization + +AMD Quark provides its own online path, exposed as the `quark_online` quantization backend, for users who want parity with Quark's offline export and Quark's per-layer mixed-precision configs. Like vLLM's built-in online quantization, it quantizes weights inside the weight-loading hook just before serving and writes no new checkpoint. + +Its quant math is aligned byte-for-byte with Quark's offline export, so what you validate online is what you get offline. Use it to *find* the scheme and per-layer selection you want. + +Compared to vLLM's built-in online quantization, the `quark_online` plugin adds: + +- **Flexible config parsing** — a terse config expands into Quark's verbose per-layer config, delegating all per-layer matching to a real `QuarkConfig`. +- **Per-layer / mixed schemes** — dispatch a different method per layer (e.g. MXFP4 experts with FP8 attention on an MoE model). Each online method subclasses the matching offline scheme, so a load-time quantized layer runs the identical inference kernel as an offline one. +- **Re-quantizing an already-quantized checkpoint** — an FP8 block-scale checkpoint (e.g. DeepSeek-R1) is dequantized and re-quantized to a target scheme layer-locally at load time, with no new checkpoint. + +The plugin ships in AMD Quark — no fork of vLLM, no patched checkpoint format; see the [Quark documentation](https://quark.docs.amd.com/latest/) for details. Three presets ship ready to use: + +| Preset key | Scheme | +| --- | --- | +| `ptpc_fp8` | FP8 E4M3, per-channel weight + dynamic per-token activation | +| `mxfp4` | MXFP4, per-group (group size 32) with E8M0 block scale | +| `linear_ptpc_fp8_moe_mxfp4` | Mixed: attention in FP8, MoE experts in MXFP4 | + +### Python API + +```python +from vllm import LLM, SamplingParams +from quark.online_quantization.vllm import HF_QUANTIZATION_CONFIGS + +llm = LLM( + model="Qwen/Qwen3-30B-A3B-Thinking-2507", + quantization="quark_online", + hf_overrides=HF_QUANTIZATION_CONFIGS["ptpc_fp8"], + enforce_eager=True, + tensor_parallel_size=1, +) +out = llm.generate(["The capital of France is"], SamplingParams(temperature=0.0, max_tokens=100)) +print(out) +``` + +### Native vLLM CLI + +```bash +export VLLM_PLUGINS="${VLLM_PLUGINS:-quark_online_quant}" +ONLINE_QUANT_CONFIG='{"online_quant_config": {"global_quant_config": "ptpc_fp8", "exclude_layer": ["lm_head"]}}' + +vllm serve Qwen/Qwen3-8B \ + --trust-remote-code \ + --tensor-parallel-size 1 \ + --additional-config "$ONLINE_QUANT_CONFIG" +``` + +### Re-quantizing an offline checkpoint + +No extra arguments — the same `hf_overrides` detects the checkpoint's existing config and merges it automatically: + +```python +llm = LLM( + model="deepseek-ai/DeepSeek-R1", # ships quant_method: "fp8" + quantization="quark_online", + hf_overrides=HF_QUANTIZATION_CONFIGS["ptpc_fp8"], + tensor_parallel_size=8, +) +``` + +The plugin can be disabled with `QUARK_DISABLE_VLLM_PLUGIN=1`, and it steps aside automatically if another platform plugin (e.g. ATOM) has already taken over the backend.