Hidden states extraction improvements (#43805)

Signed-off-by: Fynn Schmitt-Ulms <fschmitt@redhat.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
This commit is contained in:
Fynn Schmitt-Ulms
2026-06-11 09:44:45 -04:00
committed by GitHub
co-authored by mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
parent ab3a1fd2e6
commit ebc6ef971a
7 changed files with 454 additions and 233 deletions
@@ -19,7 +19,6 @@ from vllm.distributed.kv_transfer.kv_connector.v1 import (
with tempfile.TemporaryDirectory() as tmpdir:
llm = LLM(
model="Qwen/Qwen3-8B",
enable_chunked_prefill=False,
speculative_config={
"method": "extract_hidden_states",
"num_speculative_tokens": 1,
@@ -59,17 +58,58 @@ For improved performance, it is recommended to use a RAM-mounted file system suc
```bash
vllm serve Qwen/Qwen3-8B \
--speculative_config '{"method": "extract_hidden_states", "num_speculative_tokens": 1, "draft_model_config": {"hf_config": {"eagle_aux_hidden_state_layer_ids": [1, 2, 3, 4]}}}' \
--kv_transfer_config '{"kv_connector": "ExampleHiddenStatesConnector", "kv_role": "kv_producer", "kv_connector_extra_config": {"shared_storage_path": "/dev/shm/hidden_states"}}' \
--no-enable-chunked-prefill
--kv_transfer_config '{"kv_connector": "ExampleHiddenStatesConnector", "kv_role": "kv_producer", "kv_connector_extra_config": {"shared_storage_path": "/dev/shm/hidden_states"}}'
```
## Per-Request Options
Both offline and online modes support per-request options via `kv_transfer_params`:
| Parameter | Default | Description |
| --- | --- | --- |
| `hidden_states_path` | Auto-generated | Custom file path for saving hidden states. If not set, files are saved to `<shared_storage_path>/<request_id>.safetensors`. Requires `allow_custom_save_path` to be enabled in the server config. |
| `include_output_tokens` | `False` | When `True`, save hidden states for both prompt and generated output tokens. When `False`, only prompt token hidden states are saved. |
### Offline usage
Pass per-request options via `extra_args` on `SamplingParams`:
```python
SamplingParams(
max_tokens=32,
extra_args={
"kv_transfer_params": {
"hidden_states_path": "/tmp/my_output.safetensors",
"include_output_tokens": True,
}
},
)
```
### Online usage
Pass `kv_transfer_params` as a top-level field in the API request:
```json
{
"model": "Qwen/Qwen3-8B",
"messages": [{"role": "user", "content": "Hello"}],
"max_tokens": 32,
"kv_transfer_params": {
"hidden_states_path": "/tmp/my_output.safetensors",
"include_output_tokens": true
}
}
```
## Configuration
The `kv_connector_extra_config` dict accepts these options:
The `kv_connector_extra_config` dict accepts these server-level options:
| Parameter | Default | Description |
| --- | --- | --- |
| `shared_storage_path` | `/tmp` | Directory where hidden state files are saved |
| `shared_storage_path` | `/tmp` | Directory where hidden state files are saved (used when `hidden_states_path` is not set per-request) |
| `allow_custom_save_path` | `False` | Allow API clients to specify custom file paths via `hidden_states_path`. When disabled, client-provided paths are ignored with a warning. Enable only with trusted clients — custom paths can write to arbitrary locations on the server. |
| `num_writer_threads` | `8` | Thread pool size for async disk writes |
| `use_synchronization_lock` | `True` | Use file locks so concurrent readers block until writes complete. Can be disabled for batch generation where synchronization is not needed. |