iHore/vLLM
8
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity
23,341 Commits 3 Branches 0 Tags
patches
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Nick Hill 65b2f405dc [Core] Simplify core kv-cache blocks initialization logic (#36521) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-03-10 20:20:02 +00:00
Nick Hill 2a68464c5b [Test] test_async_scheduling.py improvements (#36340) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-03-10 11:17:26 -07:00
Nick Hill ddbb0d230a [Model Runner V2] Fix mm input embeddings lookup (#36588) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-03-10 00:24:58 -07:00
Nick Hill 9efc3bdcd6 [Model Runner V2] Fix _compute_slot_mappings_kernel for chunked prefill (#36580) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-03-10 00:23:42 -07:00
Nick Hill b354686524 [Model Runner V2] Fix warmup for pipeline parallel (#36280) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-03-06 16:58:51 -08:00
Nick HillandMark McLoughlin 6a18d8789b [Core] Fix benign error log during normal shutdown (#36270) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Mark McLoughlin <markmc@redhat.com>
 2026-03-07 00:39:21 +00:00
Nick Hill 26bd43b52d Revert "[BugFix] Fix engine hanging after KV cache initialization fai… (#36262) 2026-03-06 08:28:09 -08:00
Nick Hill a73af584fe [Model Runner V2] Fix warmup for very small kvcache and/or blocksizes (#36176) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-03-05 14:48:10 -08:00
Nick Hill 16c472abe7 [Core] Move ray-specific WorkerWrapperBase methods to RayWorkerWrapper (#35328) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-03-05 12:11:59 +08:00
Nick Hill a3299c3d1d [Model Runner V2] Misc code simplification (#35941) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-03-04 15:26:35 -08:00
Nick Hill 417fd28fb1 [Model Runner V2] Fix pooling (#36019) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-03-04 10:53:17 -08:00
Nick Hill d15c3b90fc [Core] Move save_tensorized_model logic to Worker (#35825) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-03-03 15:31:59 -08:00
Nick Hill b1d9f5372d [Model Runner V2] Warmup kernels (#35172) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-02-27 10:43:30 -08:00
Nick Hill 1d897ff04f [Misc] Fill in some v1 CODEOWNERS gaps (#35524) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-02-27 09:34:37 -08:00
Nick Hill 876312f0b5 [Core] Fix gpu_worker.py pre-commit errors (#35312) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-02-27 07:54:24 -08:00
Nick Hill b6d5a17298 [Model Runner V2] Fix error-handling (#35063) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-02-26 11:00:19 -08:00
Nick Hill dbf0da817a [Core] Cleanup engine pause/sleep logic (#34528) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-02-24 19:33:34 -08:00
Nick HillandWoosuk Kwon 944ffb5968 [Model Runner V2][Minor] Remove redundant do_spec_decode field (#35039) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Woosuk Kwon <woosuk@inferact.ai>
 2026-02-22 16:18:04 -08:00
Nick HillandCyrus Leung 820d7815eb [Core] Minor structured-output related scheduler optimization (#34765) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>
 2026-02-21 01:38:28 -08:00
Nick Hill 40b2f1c3d9 [Model Runner V2] Minor CPU optimizations (#34856) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-02-19 16:05:37 -08:00
Nick Hill c50e105a88 [Model Runner V2] Avoid prepare prefill kernel launch overhead (#34780) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-02-18 00:49:21 -08:00
Nick Hill a49ea5a58f [Model Runner V2] A bit more PP simplification (#34766) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-02-17 21:39:07 -08:00
Nick Hill e09546cf05 [Frontend] Exploit tokenizers "new stream" in FastIncrementalDetokenizer (#34217) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-02-11 11:03:24 +01:00
Nick Hill 79504027ef [Misc] Bump fastsafetensors version for latest fixes (#34273) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-02-11 00:30:09 -08:00
Nick Hill e7e52781ff [ModelRunner V2][BugFix] Fix max_query_len calculation (#34167) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-02-09 21:47:17 +00:00
Nick HillandCyrus Leung d9bede0314 [BugFix] Fix fastsafetensors TP all procs using all GPUs (#34070) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>
 2026-02-09 15:15:46 +08:00
Nick Hill a96197f564 [Perf] Simplify DeepseekV32 tokenizer, ensure fast detokenization used (#33855) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-02-08 07:16:34 +00:00
Nick Hill 40218a82ba [ModelRunner V2] Revert token rank comparison difference for now (#34017) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-02-07 11:11:05 +08:00
Nick Hill add9f1fbd9 [Minor] Include StreamingInput in inputs package (#33856) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-02-05 04:38:20 +00:00
Nick Hill 52ee21021a [BugFix][Spec Decoding] Fix negative accepted tokens metric crash (#33729) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-02-03 23:34:41 +00:00
Nick Hill 61397891ce [Minor] Some code simplification in scheduler.py (#33597) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-02-03 15:00:00 +08:00
Nick Hill 876a16f4fb [ModelRunner V2] Fix spec decoding + logprobs (#33391) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-01-31 03:33:26 +00:00
Nick Hill 6bf3b46d78 [ModelRunner V2] Misc code simplification and cleanup (#33266) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-01-28 14:41:23 -08:00
Nick Hill 0cd259b2d8 [BugFix] Fix P/D with non-MoE DP (#33037) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-01-27 08:03:47 -08:00
Nick Hill 8518b30447 [Model Runner V2] Add KV Connector support (#32742) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-01-23 10:49:17 -08:00
Nick Hill 7fe255889e [Misc] Log vLLM logo when starting server (#32796) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-01-23 11:15:12 +08:00
Nick Hill bc14663e6a [Cleanup] Move scheduler get_routed_experts logic to separate method (#32706) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-01-22 10:46:00 -05:00
Nick Hill 098b2d66fe [Benchmark] Don't default to temperature==0 in vllm bench serve (#32723) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-01-22 10:03:15 +00:00
Nick Hill 24dc30f7ff [ModelRunner V2] Don't pin reused flashinfer tensors (#32799) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-01-21 13:17:43 -08:00
Nick Hill 9b693d023c [Misc] Omit "disable NCCL for DP sync" startup log when not applicable (#32707) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-01-21 17:03:39 +00:00
Nick Hill 6f067b1fb7 [Cleanup] Remove unused KVConnectorModelRunnerMixin methods (#32077) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-01-21 11:16:37 +08:00
Nick Hill c6bb5b5603 [BugFix] Fix engine crash caused by chat tools + response_format (#32127) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-01-13 10:33:14 +08:00
Nick Hill 9273a427b5 [Misc] Allow enabling NCCL for DP sync when async scheduling (#32197) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-01-13 02:03:08 +00:00
Nick Hill 3adffd5b90 [Misc] Enable async scheduling by default with spec decoding (#31998) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-01-09 23:09:34 +00:00
Nick Hill 29ce48221c [Cleanup] Remove obsolete spec decoding compatibility logic (#32003) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-01-09 05:44:18 +00:00
Nick Hill 11cec296dd [BugFix] Add spec-decode-incompatible request param validation (#31982) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-01-09 00:08:21 +00:00
Nick Hill a3d909ad2b [Misc] Tidy up some spec decode logic in GPUModelRunner (#31591) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-01-08 09:10:07 -08:00
Nick Hill 287b37cda4 [BugFix] Fix spec decoding edge case bugs (#31944) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-01-08 15:31:03 +08:00
Nick Hill 10ef65eded [BugFix] Fix bad words with speculative decoding (#31908) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-01-07 15:46:42 -05:00
Nick Hill 32f4e4db00 [Cleanup] Remove deprecated fields from CachedRequestData class (#31734) 
Signed-off-by: njhill <nickhill123@gmail.com>
 2026-01-05 21:07:14 +00:00
Nick Hill 43e3f8e4a9 [Misc] Various code simplifications (#31666) 
Signed-off-by: njhill <nickhill123@gmail.com>
 2026-01-04 18:35:56 -08:00
Nick Hill da436f868a [Minor] Small pooler output processing optimization (#31667) 
Signed-off-by: njhill <nickhill123@gmail.com>
 2026-01-04 18:33:12 -08:00
Nick Hill b53b89fdb3 [BugFix] Async scheduling: handle model forward errors more cleanly (#31611) 
Signed-off-by: njhill <nickhill123@gmail.com>
 2026-01-04 11:04:37 -08:00
Nick Hill bd877162eb [BugFix] Support online dense model DP without overhead (#30739) 
Signed-off-by: Nick Hill <nhill@redhat.com>
Signed-off-by: njhill <nickhill123@gmail.com>
 2026-01-02 23:36:38 +08:00
Nick Hill 6c2cfb62ff [BugFix] Fix async scheduling for pooling models (#31584) 
Signed-off-by: njhill <nickhill123@gmail.com>
 2025-12-31 14:48:51 -08:00
Nick Hill 3b312fb792 [Minor] Various small code cleanups/simplifications (#31508) 
Signed-off-by: njhill <nickhill123@gmail.com>
 2025-12-29 22:42:06 -08:00
Nick Hill e54ee3ea33 [Core] Deduplicate generate/encode logic in AsyncLLM (#31510) 
Signed-off-by: njhill <nickhill123@gmail.com>
 2025-12-30 10:42:45 +08:00
Nick Hill c2ff33cc8c [Core] Enable async scheduling by default (#27614) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2025-12-29 13:20:55 -07:00
Nick Hill 094fcce250 [BugFix] Re-fix async multimodal cpu tensor race condition (#31373) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Signed-off-by: njhill <nickhill123@gmail.com>
 2025-12-28 03:05:08 -08:00
Nick Hill 81786c8774 [BugFix] Fix async scheduling + reasoning with struct output (#31332) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2025-12-25 23:01:02 +00:00
Nick Hill 2ac85a4544 [BugFix] Fix logprobs with spec decode and modified logits (#30846) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-12-18 19:58:28 -08:00
Nick Hill 45c0526ac9 [BugFix] Handle errors when preprocessing added requests (#30895) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-12-19 01:29:11 +00:00
Nick Hill b0b77c4655 [BugFix] Fix spec decode + structured outputs + preemption edge case (#30916) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-12-18 12:59:55 -08:00
Nick Hill 686cbaac64 [Cleanup] Remove unused ModelRunner V1 InputBatch.num_tokens field (#30218) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-12-18 09:17:00 -08:00
Nick Hill 1cec5b7ea9 [Scheduer] Simplify stop checking for pooling models (#30591) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-12-13 09:45:26 +00:00
Nick Hill 947dfda9c2 [LMCache] Relax lmcache version requirement (#30425) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-12-11 18:18:47 -09:00
Nick Hill 6ccb7baeb1 [LMCache] Fix breakage due to new LMCache version (#30216) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-12-10 11:52:01 -08:00
Nick Hill d726a7b0ed [BugFix] Unblock use of LoRA with data parallel mode (#30220) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-12-08 12:21:05 +08:00
Nick Hill 4026ae31e9 [Misc] Move disable_nccl_for_dp_synchronization init logic into VllmConfig (#30161) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-12-05 20:59:04 -08:00
Nick Hill dc264bcea1 [BugFix] Eagerly abort cancelled final-step requests (#29987) 
Currently, when requests are cancelled while executing their final
step, "completion" is handled based on normal stop processing (e.g.
length or stop token), so the abort has no effect. This is typically
not a problem, but when a kv connector is involved it thinks the
request completed successfully rather than being aborted.

This is problematic for disaggregated prefill which will free kv
cache blocks if the request was aborted but not if it completed
successfully—since the cancelled request will never be sent to
the decode side, kv cache blocks remain pinned until the fall-back
timeout expires. The problem is exacerbated when many requests
are cancelled and/or there are large prefills whose forward pass
takes a long time (since the window is bigger).

This PR fixes the problem by processing pending aborts
immediately prior to processing model output each step; we process
only aborts, not new requests, since it's preferable for latency to
process model outputs before new incoming requests.

Fixes #26400.

Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-12-05 17:28:32 +00:00
Nick Hill 44822d7ff2 [BugFix] Preserve spec decoding uniform decode when scheduling (#29759) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-12-01 17:15:52 -08:00
Nick Hill 8e7a891602 [BugFix] Fix spec decoding max_tokens scheduling perf issue (#29542) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-11-28 20:52:23 +08:00
Nick Hill 4e57c6587f [Core] Support logprobs with spec decode + async scheduling (#29223) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-11-25 12:55:24 -08:00
Nick Hill db2906108a [Misc] Streamline unique id generation (#29375) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-11-25 08:30:11 +00:00
Nick Hill 7992324f23 [BugFix] Use unique ids for different transcription prompts (#29372) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-11-25 06:55:16 +00:00
Nick Hill a178a0b40b [BugFix] Fix duplicate id tool-call race condition (#29355) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-11-25 01:54:26 +00:00
Nick Hill 84371daf75 [Tests] Verify gpt_oss package is installed in harmony tests (#29336) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-11-24 22:04:31 +00:00
Nick Hill 7df331c66b [BugFix] Fix chunked prompt logprobs + preemption (#29071) 2025-11-22 16:07:18 -05:00
Nick Hill d44a63c6d6 [BugFix] Fix returned logprobs with spec decode + prefill chunking (#29216) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-11-22 22:41:25 +08:00
Nick Hill 9ccef8e333 [Misc] Colorize logs (#29017) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-11-19 19:26:04 -05:00
Nick Hill da8dadf68b [Minor] Rename ec_producer field to is_ec_producer (#28884) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-11-18 17:26:07 +00:00
Nick HillandCyrus Leung 439368496d [BugFix] Fix PP/async scheduling with pooling models (#28899) 
Signed-off-by: Nick Hill <nhill@redhat.com>
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>
 2025-11-18 00:20:45 -08:00
Nick Hill 5bdd155277 [CI] Fix async scheduling + spec decoding test flake (#28902) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-11-18 05:26:32 +00:00
Nick Hill 7765e5ba75 [BugFix] Fix PP performance and PP kv connector output regression (#28768) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-11-17 14:08:50 -08:00
Nick Hill 80b6080ddc [BugFix] Fix async scheduling + chunked prefill + preemption (#28787) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-11-17 06:46:46 +08:00
Nick Hill 637f292196 [CI] Fix broken pipeline (#28781) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-11-15 08:44:14 -08:00
Nick Hill ac86bff8cb Revert "[Core] Performance: Use list[np.ndarray] instead of list[list… (#28773) 2025-11-14 20:24:00 -08:00
Nick Hill 58e61e56b7 [Test] Rework e2e async scheduling tests (#28744) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-11-14 16:01:09 -08:00
Nick Hill bc3e43069a [BugFix] Fix multi-modal async scheduling race condition (#28706) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-11-14 01:11:13 -08:00
Nick Hill 327c0a9a23 [BugFix] Ensure EngineArgs.create_engine_config is idempotent (#28515) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-11-13 17:14:08 +00:00
Nick Hill 8832fff972 [BugFix] Fix mm_encoder_attn_backend arg type checking (#28599) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-11-13 03:06:03 +00:00
Nick Hill 289eb6c537 [Core] Simplify async KV output aggregation (#28327) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-11-09 09:44:13 -08:00
Nick Hill 67a2da890e [PerfFix] Avoid separate thread for MP executor shm spin (take 2) (#28319) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-11-07 22:11:03 +00:00
Nick Hill da786e339e [Core] Rework handling of async scheduling config (#28250) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-11-07 20:01:23 +00:00
Nick Hill 938a81692e [AsyncScheduling] Don't schedule past request max_tokens (#27922) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-11-04 17:06:28 +00:00
Nick Hill c9f66da8fd [PerfFix] Avoid separate thread for MP executor shm spin (#28012) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-11-04 08:33:55 -08:00
Nick Hill 5a0a6dfd55 [BugFix] Fix incorrect preallocated sampled_token_ids tensor size (#28025) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-11-04 07:38:16 -08:00
Nick Hill c2ed069b32 [BugFix] Fix mixed penalties batch with async scheduling (#27910) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-11-01 10:51:24 -07:00
Nick Hill 0cdbe7b744 [Core] Async scheduling + structured outputs compatibility (#26866) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-11-01 00:35:04 +00:00
Nick Hill 9e5bd3076e [Cleanup] Remove no-longer-used SpeculativeConfig.enable_chunked_prefill (#27826) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-10-31 10:57:45 -07:00