[Bugfix][Core] Reserve the KV null block when validating max_model_len (#47272)

Signed-off-by: hyungjun oh <92hyungjun@gmail.com>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
This commit is contained in:
92hyungjun
2026-08-20 02:49:23 +00:00
committed by GitHub
co-authored by Claude Nick Hill
parent c0233fcf01
commit 76fb6d210a
6 changed files with 109 additions and 16 deletions
@@ -332,11 +332,12 @@ def run_test(
):
spec_decoding = spec_config is not None
cache_arg: dict[str, Any] = (
# Force preemptions: with 32 blocks the cache holds at most a single
# max-length request, so the ~34 concurrent prompts contend and trigger
# preemption. (Prompts here are << max_model_len, so dropping
# max_model_len from 4096 to 512 doesn't change generation behavior.)
dict(num_gpu_blocks_override=32, max_model_len=512)
# Force preemptions: with 33 blocks (one is the reserved null block)
# the cache holds at most a single max-length request, so the ~34
# concurrent prompts contend and trigger preemption. (Prompts here are
# << max_model_len, so dropping max_model_len from 4096 to 512 doesn't
# change generation behavior.)
dict(num_gpu_blocks_override=33, max_model_len=512)
if test_preemption
else dict(gpu_memory_utilization=0.9, max_model_len=4096)
)