[Bugfix] Validate NIXL speculative config compatibility (#49230)
Signed-off-by: Tzu-Ling <tzulingk@nvidia.com> Co-authored-by: GPT-5.6 Sol <noreply@openai.com> Co-authored-by: Cursor Grok 4.5 <noreply@cursor.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
GPT-5.6 Sol
Cursor Grok 4.5
Claude Opus 4.8
parent
65cf1276a2
commit
952694e384
@@ -57,7 +57,9 @@ th:not(:first-child) {
|
||||
| Multimodal | ❔ | ❔ | ❔ | ❔ | ❔ | ❔ | ❔ |
|
||||
| Encoder-Decoder | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ |
|
||||
|
||||
<sup>1</sup> P and D instances must use the same speculation configuration.
|
||||
<sup>1</sup> P and D instances must use compatible speculation configurations.
|
||||
See [Configuration Notes](#configuration-notes) below for what must match and
|
||||
what may differ.
|
||||
|
||||
<sup>2</sup> Requires `FLASH_ATTN` or `FLASHINFER` backend **and** `HND` KV cache layout. Enable via `--kv-transfer-config '{"kv_connector_extra_config": {"enable_cross_layers_blocks": "True"}}'`.
|
||||
|
||||
@@ -79,6 +81,7 @@ By default, a **compatibility hash** is checked during handshake. P and D instan
|
||||
- Model (architecture, dtype, number of KV heads, head size, number of hidden layers)
|
||||
- Attention backend
|
||||
- KV cache dtype (`cache_dtype`)
|
||||
- EAGLE/MTP-style speculative method and draft-model configuration
|
||||
|
||||
!!! warning
|
||||
Disable the hash check with `--kv-transfer-config '{"kv_connector_extra_config": {"enforce_handshake_compat": false}}'` at your own risk.
|
||||
@@ -88,6 +91,10 @@ By default, a **compatibility hash** is checked during handshake. P and D instan
|
||||
- `tensor-parallel-size` (heterogeneous TP, subject to model restrictions above)
|
||||
- `block-size` (heterogeneous block size, subject to restrictions above)
|
||||
- Number of KV cache blocks (determined by available memory on each instance)
|
||||
- `num_speculative_tokens` (prefill and decode may use different draft depths)
|
||||
- Draft-model `attention_backend` (each instance auto-selects independently; the
|
||||
resulting KV block layout is validated at handshake time rather than via the
|
||||
compatibility hash)
|
||||
|
||||
### KV cache layout
|
||||
|
||||
|
||||
Reference in New Issue
Block a user