docs: clarify Gemma 4 assistant speculative decoding (#42180)
Signed-off-by: AbhiOnGithub <abhiOnGithub@users.noreply.github.com> Co-authored-by: AbhiOnGithub <abhiOnGithub@users.noreply.github.com>
This commit is contained in:
co-authored by
AbhiOnGithub
parent
00b0618a03
commit
27d3bac272
@@ -72,6 +72,17 @@ only apply to model-based methods such as `draft_model`, `mtp`, `eagle3`, and
|
||||
| `rejection_sample_method` | `string` | `strict` | `strict`, `probabilistic`, or `synthetic`. |
|
||||
| `synthetic_acceptance_rate` | `float` | `None` | Average acceptance rate to target when `rejection_sample_method` is `synthetic`. Valid range is `[0, 1]`. |
|
||||
|
||||
!!! note
|
||||
Gemma 4 assistant checkpoints are handled as Gemma 4 MTP speculators, not
|
||||
as generic draft models. Use `"method": "mtp"` with the assistant
|
||||
checkpoint in `model`, as shown in the [MTP guide](mtp.md#gemma-4-assistant-models).
|
||||
|
||||
If startup logs show `SpeculativeConfig(method='draft_model', ...)` for a
|
||||
Gemma 4 assistant checkpoint, the installed vLLM version does not include
|
||||
Gemma 4 MTP support for that path. Upgrade to a version that includes
|
||||
Gemma 4 MTP support instead of forcing the assistant checkpoint through
|
||||
generic draft-model speculative decoding.
|
||||
|
||||
### Method-specific keys
|
||||
|
||||
#### N-gram
|
||||
|
||||
@@ -9,6 +9,31 @@ MTP is useful when:
|
||||
- Your model natively supports MTP.
|
||||
- You want model-based speculative decoding with minimal extra configuration.
|
||||
|
||||
## Gemma 4 Assistant Models
|
||||
|
||||
Gemma 4 assistant checkpoints use vLLM's Gemma 4 MTP path. They are not generic
|
||||
draft models, even though they are passed through the `model` field in
|
||||
`--speculative-config`.
|
||||
|
||||
Use `"method": "mtp"` when serving Gemma 4 with an assistant checkpoint:
|
||||
|
||||
```bash
|
||||
vllm serve google/gemma-4-E2B-it \
|
||||
--tensor-parallel-size 1 \
|
||||
--max-model-len 8192 \
|
||||
--speculative-config '{"method":"mtp","model":"gg-hf-am/gemma-4-E2B-it-assistant","num_speculative_tokens":1}'
|
||||
```
|
||||
|
||||
The E2B, E4B, 26B-A4B, and 31B Gemma 4 IT assistant checkpoints are supported
|
||||
when their configuration uses `model_type: gemma4_assistant`. vLLM maps those
|
||||
checkpoints to `Gemma4MTPModel` internally and wires the assistant layers to
|
||||
share KV cache with the target model.
|
||||
|
||||
If an older vLLM release logs `SpeculativeConfig(method='draft_model', ...)`
|
||||
for a Gemma 4 assistant checkpoint, that release is treating the assistant as a
|
||||
generic draft model and may fail during initialization for multimodal Gemma 4
|
||||
targets. Upgrade to a version with Gemma 4 MTP support instead.
|
||||
|
||||
## Offline Example
|
||||
|
||||
```python
|
||||
|
||||
Reference in New Issue
Block a user