1848 Commits
Author SHA1 Message Date
7c9042c05a [Bugfix][MRV2] Fix speculative warmup deadlock with pipeline parallelism (#62129)
Signed-off-by: chenjr97 <chenjr97@mail2.sysu.edu.cn>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: vllm-agent <claw@inferact.ai>
2026-10-17 23:51:32 -07:00
Nick HillandClaude Opus 5.5 0c5dd18e7a [MRV2] Collect KV transfer results before the drafter runs (#62041)
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-17 10:27:57 -07:00
liuzhenweiandKunshang Ji 4e964b2623 [XPU] Fix Event init failure w/ external (#61743)
Signed-off-by: zhenwei-intel <zhenwei.liu@intel.com>
Co-authored-by: Kunshang Ji <kunshang.ji@intel.com>
2026-10-17 19:10:39 +08:00
61bb590877 [Bugfix][Spec Decode] Clear stale shared block-table rows (#61174)
Signed-off-by: Cheng Rui <286040359@qq.com>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Giancarlo Delfin <32987265+TheEpicDolphin@users.noreply.github.com>
2026-10-17 01:25:02 -07:00
c3de0b37e2 [Core][Spec Decode] Replace EAGLE group heuristics with explicit draft-layer tags (#61455)
Signed-off-by: Dakai An <dakaian108@gmail.com>
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai>
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 08:22:33 +00:00
7b6310c109 [Perf][Sampler] Speed up the small-batch split top-p pipeline (#61996)
Signed-off-by: Bowen Wang <bowen@inferact.ai>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: vllm-agent <claw@inferact.ai>
2026-10-16 21:22:11 -07:00
b544587408 [Core][Feat] Pluggable KVCacheConfigBuilder for platform/model-specific KV cache planning (#53558)
Signed-off-by: wangxiyuan <wangxiyuan1007@gmail.com>
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai>
2026-10-16 23:16:47 +00:00
cjackal 663e535640 [Perf][Spec Decode] Skip bonus-logit reduction for short drafts in rejection sampling (#61451)
Signed-off-by: cjackal <44624812+cjackal@users.noreply.github.com>
2026-10-16 12:42:48 -07:00
de7d70676d [Perf] Low-latency CPU all-reduce for DP metadata syncs (#61767)
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: hjjq <50634613+hjjq@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-15 15:29:13 -07:00
Stefano CastagnettaandKevin H. Luu 784d1fff0c [MRV2][Spec Decode] Quantized draft lm_head for drafters that share the target head (#59973)
Signed-off-by: Stefano Castagnetta <scastagnetta@nvidia.com>
Co-authored-by: Kevin H. Luu <khluu000@gmail.com>
2026-10-15 22:06:40 +00:00
d92a4f25f2 [watermarking] Add DFlash support to dual-key gumbel-max watermarking (#59554)
Signed-off-by: Shern Shiou Tan <shernshiou@gmail.com>
Signed-off-by: Simon Veitner <sveitner@redhat.com>
Signed-off-by: Raphael Rialland <raphael.rialland@mistral.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Simon Veitner <sveitner@redhat.com>
Co-authored-by: Raphael Rialland <raphael.rialland@mistral.ai>
Co-authored-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com>
2026-10-15 22:06:06 +00:00
DimensionSTPandKevin H. Luu 9edcc7fa7e [Bugfix][Spec Decode] Avoid reusing random draws across block verification rounds (#61161)
Signed-off-by: DimensionSTP <65501090+DimensionSTP@users.noreply.github.com>
Co-authored-by: Kevin H. Luu <khluu000@gmail.com>
2026-10-15 14:46:38 -07:00
d12ea8e309 [Bugfix][MRV2] Keep input_ids on later PP stages for raw-token models (#61016)
Signed-off-by: booth-algo <kevinlauofficial01@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Kevin H. Luu <khluu000@gmail.com>
2026-10-15 12:54:21 -07:00
mosafariuk 69425846f8 [Bugfix][CPU] Report the real cause when the worker outgrows its memory reservation (#59472)
Signed-off-by: Mo <mosafariuk@gmail.com>
2026-10-15 21:52:05 +08:00
Laith SakkaandNick Hill 9cede34a67 [Bugfix] Create cuBLAS handles before memory profiling to avoid OOM (#61044)
Signed-off-by: Laith Sakka <lsakka@meta.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
2026-10-15 08:59:22 -04:00
Harry MellorClaude Opus 5.5mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>Kevin H. Luu
b965ee564d [Chore] Use upstream Transformers DeepseekV4Config (#61392)
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Kevin H. Luu <khluu000@gmail.com>
2026-10-15 01:08:08 -07:00
Shanshan Shen f46021a7fa [ROCm][Perf] Enable DPA+ETP prefill compute/comm dual-batch-overlap for DeepSeek-V4 (#57773)
Signed-off-by: Shanshan Shen <87969357+shen-shanshan@users.noreply.github.com>
Signed-off-by: shen-shanshan <467638484@qq.com>
2026-10-15 11:16:48 +08:00
FeanorandTaneem Ibrahim 5d5c3d7c43 [Bugfix] Preserve padding token types for cross encoders (#58143)
Signed-off-by: zhaoye <772971548@qq.com>
Co-authored-by: Taneem Ibrahim <taneem.ibrahim@gmail.com>
2026-10-15 03:09:24 +00:00
bfdbfa1632 [Bugfix][PCP] Align MoE routing padding with gathered tokens (#60946)
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
Signed-off-by: wangyicong <wangyicong@bytedance.com>
Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: OpenAI Codex <noreply@openai.com>
2026-10-14 21:04:00 -04:00
Woosuk KwonandClaude Opus 5.5 8530aed2b7 [Spec Decode] Remove Medusa and MLP speculator support (#61701)
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 17:29:23 -07:00
Juntian LiuandClaude Opus 5.5 388a610dfb [Perf][DFlash] Capture context combine and anchor in the draft CUDA graph (#59511)
Signed-off-by: Juntian Liu <Juntianl777@gmail.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-14 15:40:52 -04:00
Ayushman Singh 9b5c6707a3 [Bugfix][MRV2] Capture DCP FULL cudagraphs with KV context (#61496)
Signed-off-by: Ayushman Singh <40520701+ayush1399@users.noreply.github.com>
2026-10-14 19:20:48 +00:00
Sandeep AgrawalandKevin H. Luu 21e207efa1 [Bugfix][MRV2][Spec Decode] Honor EAGLE3 single-state draft configuration (#61563)
Signed-off-by: Sandeep Agrawal <sandeepa@nvidia.com>
Co-authored-by: Kevin H. Luu <khluu000@gmail.com>
2026-10-14 12:09:06 -07:00
857ddac4e5 [Perf][MRV2] Skip empty draft-token resets on request admission (#61640)
Signed-off-by: Jakub Zakrzewski <jzakrzewski@nvidia.com>
Co-authored-by: OpenAI Codex <noreply@openai.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
2026-10-14 12:01:01 -07:00
qizixiClaude Opus 5.5mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
c1407a7e86 [Spec Decode][MRV2] Warm up top-k/top-p and greedy rejection kernels for speculative decoding (#59260)
Signed-off-by: zixi-qi <zixi@inferact.ai>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-10-14 14:59:21 -04:00
Sherif Waly be4ce2e505 [Bugfix] Share PCP-aware local GPU indexing for NUMA binding (#61237) 2026-10-14 10:54:03 +02:00
3ccc0a90b6 [KV Connector] Support sleep mode with MooncakeConnector over RDMA (#59625)
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Signed-off-by: Dakai An <dakaian108@gmail.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Dakai An <dakaian108@gmail.com>
Co-authored-by: Dakai An <77474977+andakai@users.noreply.github.com>
2026-10-14 04:18:47 +00:00
Almog TavorandItay Alroy 87e7dce3a7 [MRV2] Keep warmup out of the EPLB load stats (#56716)
Signed-off-by: almogtavor <almogtavor@gmail.com>
Co-authored-by: Itay Alroy <75032521+itayalroy@users.noreply.github.com>
2026-10-14 00:00:48 +00:00
Chaunceyandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> ad8d39cbab [Bugfix] Fix DSpark graph-padding handling in the DeepSeek-V4.1 MoE fused top-k (#59166)
Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-10-14 07:49:52 +08:00
de8291e24f [Core] Add lifecycle hooks for worker extensions (#59463)
Signed-off-by: Dakai An <dakaian108@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: aoshen02 <aoshen@inferact.ai>
2026-10-13 21:51:05 +08:00
df6c7ad031 [Core][Refactor] Move KV cache planning logic to kv_cache_planning.py (#61183)
Signed-off-by: Dakai An <dakaian108@gmail.com>
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai>
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 03:05:39 +00:00
Wentao Ye d0948f754d [Bug] Honor spec decode draft model config (#61240)
Signed-off-by: yewentao256 <zhyanwentao@126.com>
2026-10-12 15:13:23 -04:00
Woosuk KwonandClaude Opus 5.5 e1a5086256 [Attention] Remove cascade attention (#61126)
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 10:41:07 -07:00
Ilya Markov a281ad32f0 [EPLB] Add EPLB support to dspark draft models (DeepSeekV4) (#57549)
Signed-off-by: Markov Ilya <markovilya197@gmail.com>
2026-10-12 10:03:09 -04:00
Chauncey ac28877f0e [Bugfix] Fix out-of-bounds lookahead in adaptive block verification (#60945)
Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>
2026-10-12 09:21:35 -04:00
Giancarlo DelfinandRishi Puri 36f3972812 [Spec Decode] Skip metadata build for dflash/dspark when all model backends support it (#60692)
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
Co-authored-by: Rishi Puri <20074092+puririshi98@users.noreply.github.com>
2026-10-12 00:09:10 -07:00
Yan Ma f147949be8 [XPU] limit FA kernel block size to multipleOf 64 (#57297)
Signed-off-by: Yan Ma <yan.ma@intel.com>
2026-10-12 10:25:32 +08:00
dingdangmaoandCodex 124b89d5b7 [Bugfix][RecoverSSM] Correct aligned state indices at exact block boundaries (#59962)
Signed-off-by: apex-mochen <2756823972@qq.com>
Co-authored-by: Codex <noreply@openai.com>
2026-10-12 02:08:14 +00:00
Nick Hill 3fe917abb1 [Bugfix][MRV2] Fix FlashInfer attention and ReplaySSM warmups on MRV2 (#61013)
Signed-off-by: Nick Hill <nickhill123@gmail.com>
2026-10-11 16:04:41 -07:00
aoshen02andCodex 239a3fa3a8 [Sleep] Retain draft-private weights with existing parameter selector (#61046)
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: Codex <noreply@openai.com>
2026-10-11 23:30:56 +08:00
aoshen02 bd5bbce6e4 [RL] Release weight-transfer communicators without stopping inference (#61060)
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Signed-off-by: fh3 <a@b>
2026-10-11 21:15:14 +08:00
Xun Sun 26a488ad93 [Fast Start][Bugfix] Snapshot GPU memory after external weights are ready (#60966) 2026-10-11 07:10:04 +00:00
Nick Hill f8471164bf [MRV2] Detatch weight-reload logic from MRV1 (#60937) 2026-10-10 21:15:05 -07:00
f2376cdc4d [Fast Start] Preload the FlashInfer autotune table on the weight cache daemon (#60085)
Signed-off-by: liusy58 <mg21330037@smail.nju.edu.cn>
Signed-off-by: Isotr0py <Isotr0py@outlook.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Isotr0py <Isotr0py@outlook.com>
2026-10-10 15:48:06 +00:00
Matthew BonanniandClaude Opus 5.5 d5f356dae0 [HiSparse] Own GPU residency instead of editing core block-table rows (#60647)
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 21:25:44 -07:00
Misha GoinandClaude Opus 5.5 29e4d2d42e [Frontend][Build] Add vllm download-kernels for FlashInfer precompiled kernels (#58765)
Signed-off-by: mgoin <mgoin64@gmail.com>
Signed-off-by: Misha Goin <mgoin64@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 21:22:35 -07:00
DimensionSTPandOpenAI Codex 10cc2f6ae2 [Bugfix] Bound GDN speculative decode query widths (#55504)
Signed-off-by: DimensionSTP <65501090+DimensionSTP@users.noreply.github.com>
Co-authored-by: OpenAI Codex <noreply@openai.com>
2026-10-10 11:00:16 +08:00
Nick Hill 7d1d8660d6 [MRV2] Miscellaneous code structure updates (#60754)
Signed-off-by: Nick Hill <nickhill123@gmail.com>
2026-10-09 17:58:09 -07:00
ed7ae705a1 [Bugfix][MRV2] Use int64 index mappings to fix row-offset overflow (#60736)
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: vllmellm <vllm.ellm@embeddedllm.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-09 14:38:11 -07:00
02e55d2af3 [KV Cache] GLM-5.3-Flash: use the generic packed KV layout; CircularBufferSpec tail with speculative ring slots (#57169)
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai>
Signed-off-by: Dakai An <dakaian108@gmail.com>
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai>
Co-authored-by: OpenAI Codex <noreply@openai.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com>
2026-10-09 15:48:37 -04:00