[Bugfix] Validate NIXL speculative config compatibility (#49230)

Signed-off-by: Tzu-Ling <tzulingk@nvidia.com>
Co-authored-by: GPT-5.6 Sol <noreply@openai.com>
Co-authored-by: Cursor Grok 4.5 <noreply@cursor.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Tzu-Ling Kan
2026-08-04 02:51:04 +08:00
committed by GitHub
co-authored by GPT-5.6 Sol Cursor Grok 4.5 Claude Opus 4.8
parent 65cf1276a2
commit 952694e384
3 changed files with 165 additions and 2 deletions
@@ -57,7 +57,9 @@ th:not(:first-child) {
| Multimodal | ❔ | ❔ | ❔ | ❔ | ❔ | ❔ | ❔ |
| Encoder-Decoder | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ |
<sup>1</sup> P and D instances must use the same speculation configuration.
<sup>1</sup> P and D instances must use compatible speculation configurations.
See [Configuration Notes](#configuration-notes) below for what must match and
what may differ.
<sup>2</sup> Requires `FLASH_ATTN` or `FLASHINFER` backend **and** `HND` KV cache layout. Enable via `--kv-transfer-config '{"kv_connector_extra_config": {"enable_cross_layers_blocks": "True"}}'`.
@@ -79,6 +81,7 @@ By default, a **compatibility hash** is checked during handshake. P and D instan
- Model (architecture, dtype, number of KV heads, head size, number of hidden layers)
- Attention backend
- KV cache dtype (`cache_dtype`)
- EAGLE/MTP-style speculative method and draft-model configuration
!!! warning
Disable the hash check with `--kv-transfer-config '{"kv_connector_extra_config": {"enforce_handshake_compat": false}}'` at your own risk.
@@ -88,6 +91,10 @@ By default, a **compatibility hash** is checked during handshake. P and D instan
- `tensor-parallel-size` (heterogeneous TP, subject to model restrictions above)
- `block-size` (heterogeneous block size, subject to restrictions above)
- Number of KV cache blocks (determined by available memory on each instance)
- `num_speculative_tokens` (prefill and decode may use different draft depths)
- Draft-model `attention_backend` (each instance auto-selects independently; the
resulting KV block layout is validated at handshake time rather than via the
compatibility hash)
### KV cache layout