[Feature] Add first-class KV hints request envelope for programmatic KV management (#53423)

Signed-off-by: Karen Chung <karenc@nvidia.com>
This commit is contained in:
Karen Chung
2026-09-21 09:56:11 -07:00
committed by GitHub
parent 4f14516790
commit 986e217158
30 changed files with 340 additions and 18 deletions
+1
View File
@@ -194,6 +194,7 @@ impl ChatRequestProcessor {
add_special_tokens: request.add_special_tokens,
data_parallel_rank: request.data_parallel_rank,
session_id: request.session_id,
kv_hints: None,
reasoning_parser_kwargs,
lora_request: request.lora_request,
arrival_time: Some(arrival_time),