7c9042c05a
[Bugfix][MRV2] Fix speculative warmup deadlock with pipeline parallelism ( #62129 )
...
Signed-off-by: chenjr97 <chenjr97@mail2.sysu.edu.cn >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: vllm-agent <claw@inferact.ai >
2026-10-17 23:51:32 -07:00
Nick Hill and Claude Opus 5.5
0c5dd18e7a
[MRV2] Collect KV transfer results before the drafter runs ( #62041 )
...
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com >
2026-10-17 10:27:57 -07:00
61bb590877
[Bugfix][Spec Decode] Clear stale shared block-table rows ( #61174 )
...
Signed-off-by: Cheng Rui <286040359@qq.com >
Co-authored-by: Codex <noreply@openai.com >
Co-authored-by: Giancarlo Delfin <32987265+TheEpicDolphin@users.noreply.github.com >
2026-10-17 01:25:02 -07:00
c3de0b37e2
[Core][Spec Decode] Replace EAGLE group heuristics with explicit draft-layer tags ( #61455 )
...
Signed-off-by: Dakai An <dakaian108@gmail.com >
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai >
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-17 08:22:33 +00:00
7b6310c109
[Perf][Sampler] Speed up the small-batch split top-p pipeline ( #61996 )
...
Signed-off-by: Bowen Wang <bowen@inferact.ai >
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: vllm-agent <claw@inferact.ai >
2026-10-16 21:22:11 -07:00
de7d70676d
[Perf] Low-latency CPU all-reduce for DP metadata syncs ( #61767 )
...
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: hjjq <50634613+hjjq@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com >
2026-10-15 15:29:13 -07:00
Stefano Castagnetta and Kevin H. Luu
784d1fff0c
[MRV2][Spec Decode] Quantized draft lm_head for drafters that share the target head ( #59973 )
...
Signed-off-by: Stefano Castagnetta <scastagnetta@nvidia.com >
Co-authored-by: Kevin H. Luu <khluu000@gmail.com >
2026-10-15 22:06:40 +00:00
49f944be3e
[Perf][HiSparse] Reuse pinned host pool after graph profiling ( #61668 )
...
Signed-off-by: Itay Etelis <92247226+Etelis@users.noreply.github.com >
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com >
Co-authored-by: Matthew Bonanni <mbonanni@redhat.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-15 12:33:03 -07:00
mosafariuk
69425846f8
[Bugfix][CPU] Report the real cause when the worker outgrows its memory reservation ( #59472 )
...
Signed-off-by: Mo <mosafariuk@gmail.com >
2026-10-15 21:52:05 +08:00
Shanshan Shen
f46021a7fa
[ROCm][Perf] Enable DPA+ETP prefill compute/comm dual-batch-overlap for DeepSeek-V4 ( #57773 )
...
Signed-off-by: Shanshan Shen <87969357+shen-shanshan@users.noreply.github.com >
Signed-off-by: shen-shanshan <467638484@qq.com >
2026-10-15 11:16:48 +08:00
bfdbfa1632
[Bugfix][PCP] Align MoE routing padding with gathered tokens ( #60946 )
...
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com >
Signed-off-by: wangyicong <wangyicong@bytedance.com >
Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com >
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com >
Co-authored-by: OpenAI Codex <noreply@openai.com >
2026-10-14 21:04:00 -04:00
Woosuk Kwon and Claude Opus 5.5
8530aed2b7
[Spec Decode] Remove Medusa and MLP speculator support ( #61701 )
...
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-14 17:29:23 -07:00
Sandeep Agrawal and Kevin H. Luu
21e207efa1
[Bugfix][MRV2][Spec Decode] Honor EAGLE3 single-state draft configuration ( #61563 )
...
Signed-off-by: Sandeep Agrawal <sandeepa@nvidia.com >
Co-authored-by: Kevin H. Luu <khluu000@gmail.com >
2026-10-14 12:09:06 -07:00
c1407a7e86
[Spec Decode][MRV2] Warm up top-k/top-p and greedy rejection kernels for speculative decoding ( #59260 )
...
Signed-off-by: zixi-qi <zixi@inferact.ai >
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-10-14 14:59:21 -04:00
Itay Etelis and Codex
efad400092
[HiSparse] Prefault the shared host pool in parallel ( #60869 )
...
Signed-off-by: Itay Etelis <92247226+Etelis@users.noreply.github.com >
Co-authored-by: Codex <noreply@openai.com >
2026-10-14 10:28:26 -04:00
Jiangyun Zhu and Claude Opus 5.5
badfcdbf91
[Perf][GDN] Internal prefill checkpoints for the FlashInfer GDN prefill backend ( #60050 )
...
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-14 14:49:46 +08:00
Almog Tavor and Itay Alroy
87e7dce3a7
[MRV2] Keep warmup out of the EPLB load stats ( #56716 )
...
Signed-off-by: almogtavor <almogtavor@gmail.com >
Co-authored-by: Itay Alroy <75032521+itayalroy@users.noreply.github.com >
2026-10-14 00:00:48 +00:00
de8291e24f
[Core] Add lifecycle hooks for worker extensions ( #59463 )
...
Signed-off-by: Dakai An <dakaian108@gmail.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: aoshen02 <aoshen@inferact.ai >
2026-10-13 21:51:05 +08:00
df6c7ad031
[Core][Refactor] Move KV cache planning logic to kv_cache_planning.py ( #61183 )
...
Signed-off-by: Dakai An <dakaian108@gmail.com >
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai >
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-13 03:05:39 +00:00
Wentao Ye
d0948f754d
[Bug] Honor spec decode draft model config ( #61240 )
...
Signed-off-by: yewentao256 <zhyanwentao@126.com >
2026-10-12 15:13:23 -04:00
Woosuk Kwon and Claude Opus 5.5
e1a5086256
[Attention] Remove cascade attention ( #61126 )
...
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-12 10:41:07 -07:00
Ilya Markov
a281ad32f0
[EPLB] Add EPLB support to dspark draft models (DeepSeekV4) ( #57549 )
...
Signed-off-by: Markov Ilya <markovilya197@gmail.com >
2026-10-12 10:03:09 -04:00
Giancarlo Delfin and Rishi Puri
36f3972812
[Spec Decode] Skip metadata build for dflash/dspark when all model backends support it ( #60692 )
...
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai >
Co-authored-by: Rishi Puri <20074092+puririshi98@users.noreply.github.com >
2026-10-12 00:09:10 -07:00
dingdangmao and Codex
124b89d5b7
[Bugfix][RecoverSSM] Correct aligned state indices at exact block boundaries ( #59962 )
...
Signed-off-by: apex-mochen <2756823972@qq.com >
Co-authored-by: Codex <noreply@openai.com >
2026-10-12 02:08:14 +00:00
Nick Hill
c059bee812
[MRV2] Switch CI tests still running on MRV1 unnecessarily ( #61014 )
...
Signed-off-by: Nick Hill <nickhill123@gmail.com >
2026-10-11 09:57:07 -07:00
aoshen02 and Codex
239a3fa3a8
[Sleep] Retain draft-private weights with existing parameter selector ( #61046 )
...
Signed-off-by: aoshen02 <aoshen@inferact.ai >
Co-authored-by: Codex <noreply@openai.com >
2026-10-11 23:30:56 +08:00
aoshen02
bd5bbce6e4
[RL] Release weight-transfer communicators without stopping inference ( #61060 )
...
Signed-off-by: aoshen02 <aoshen@inferact.ai >
Signed-off-by: fh3 <a@b>
2026-10-11 21:15:14 +08:00
Nick Hill
f8471164bf
[MRV2] Detatch weight-reload logic from MRV1 ( #60937 )
2026-10-10 21:15:05 -07:00
Matthew Bonanni and Claude Opus 5.5
d5f356dae0
[HiSparse] Own GPU residency instead of editing core block-table rows ( #60647 )
...
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-09 21:25:44 -07:00
DimensionSTP and OpenAI Codex
10cc2f6ae2
[Bugfix] Bound GDN speculative decode query widths ( #55504 )
...
Signed-off-by: DimensionSTP <65501090+DimensionSTP@users.noreply.github.com >
Co-authored-by: OpenAI Codex <noreply@openai.com >
2026-10-10 11:00:16 +08:00
Nick Hill
7d1d8660d6
[MRV2] Miscellaneous code structure updates ( #60754 )
...
Signed-off-by: Nick Hill <nickhill123@gmail.com >
2026-10-09 17:58:09 -07:00
Matthew Bonanni and Claude Opus 5.5
4c43254d52
[HiSparse] Report host-pool block residency as HiSparse metrics ( #59485 )
...
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-09 16:45:34 -06:00
ed7ae705a1
[Bugfix][MRV2] Use int64 index mappings to fix row-offset overflow ( #60736 )
...
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: vllmellm <vllm.ellm@embeddedllm.com >
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com >
2026-10-09 14:38:11 -07:00
02e55d2af3
[KV Cache] GLM-5.3-Flash: use the generic packed KV layout; CircularBufferSpec tail with speculative ring slots ( #57169 )
...
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai >
Signed-off-by: Dakai An <dakaian108@gmail.com >
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com >
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai >
Co-authored-by: OpenAI Codex <noreply@openai.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com >
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com >
Co-authored-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com >
2026-10-09 15:48:37 -04:00
zzaebok and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
702ce313f4
[Perf][Multimodal] Sort multimodal encoder inputs by modality ( #60196 )
...
Signed-off-by: Jaebok Lee <jaebok9541@naver.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-10-09 10:55:36 +08:00
Mia Ojeda and Lucas Wilkinson
4611c2c3d2
[BugFix][DiffusionGemma][CPU] Copy async output snapshots instead of aliasing reused buffers ( #59829 )
...
Signed-off-by: ojeda-e <eojeda@league.com >
Co-authored-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com >
2026-10-08 19:16:51 -07:00
Nishant Revur and Claude
581692cc0f
[Bugfix][Spec Decode] Fix multimodal embedding merge in Llama 4 and Mistral Large 3 EAGLE drafters ( #57633 )
...
Signed-off-by: Nishant Revur <kumrfni@amazon.com >
Co-authored-by: Claude <noreply@anthropic.com >
2026-10-09 01:44:50 +00:00
Woosuk Kwon
194bd72be7
[Compile] Remove torch.compile-based sequence parallelism and AsyncTP ( #60517 )
2026-10-08 08:28:23 -07:00
Matthew Bonanni and Claude Opus 5.5
4ed4c4433b
[Bugfix][HiSparse] Drop the unaccounted, write-only prefill mirror staging buffer ( #60429 )
...
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-08 14:00:49 +00:00
Giancarlo Delfin
6ee9be6b3c
[Spec Decode] Rename AR speculators to StandaloneAR / TargetDependentAR ( #60335 )
...
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai >
2026-10-07 14:01:59 -04:00
3d6853b32c
[Bugfix] Seed hybrid mamba state index with mamba_block_size on prefix-cache hits ( #55601 )
...
Signed-off-by: pondzikk <pondzik@gmail.com >
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com >
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com >
Co-authored-by: zjy0516 <riverclouds.zhu@qq.com >
Co-authored-by: ptorsten <1652174+ptorsten@users.noreply.github.com >
2026-10-07 17:09:50 +00:00
c83935b300
[Bugfix][Model Loader] Support reload_weights with runai_streamer load format ( #58124 )
...
Signed-off-by: Viktor Koukouliev <vkoukouliev@nvidia.com >
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com >
Co-authored-by: Isotr0py <Isotr0py@outlook.com >
2026-10-07 14:04:35 +00:00
8a26869bca
[Bugfix] Index Mamba align state copies by request slot in MRV2 ( #60210 )
...
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com >
Co-authored-by: Karl0007 <44745209+Karl0007@users.noreply.github.com >
Co-authored-by: Yongqin Wang <yongqinwang@roblox.com >
2026-10-07 02:32:46 +00:00
Miłosz Grunwald
703c29179c
[TEST][XPU][CI] Generalize model runner prefill tail classification tests to all platforms ( #60201 )
...
Signed-off-by: microslaw <milosz.grunwald@intel.com >
2026-10-07 08:59:20 +08:00
819852df6b
[ModelRunner V2] Speculative Decoding NGram GPU Implementations ( #40704 )
...
Signed-off-by: PatchouliTaisa <patchychen@tencent.com >
Signed-off-by: PatchouliTaisa <pyramkar@gmail.com >
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Signed-off-by: Xuanan Chen <xuananchenc@nvidia.com >
Co-authored-by: PatchouliTaisa <patchychen@tencent.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Xuanan Chen <xuananchenc@nvidia.com >
Co-authored-by: Giancarlo Delfin <32987265+TheEpicDolphin@users.noreply.github.com >
2026-10-06 13:32:59 -07:00
e37e51dd24
[Metrics] Expose cached prompt tokens by cache tier ( #56318 )
...
Signed-off-by: Cam Quilici <cjquilici@gmail.com >
Signed-off-by: Cam Quilici <cameron@semianalysis.com >
Co-authored-by: Cam Quilici <cameron@semianalysis.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai >
2026-10-06 14:52:05 -04:00
czhu-cohere and aoshen02
6fddc9b21e
[Profiler] Add per-session profiling controls to Python and Rust frontends ( #57875 )
...
Signed-off-by: czhu <conway.zhu@cohere.com >
Signed-off-by: Conway Zhu <conway.zhu@cohere.com >
Co-authored-by: aoshen02 <aoshen@inferact.ai >
2026-10-06 05:21:19 +00:00
eb2901b735
[Perf][DSv4.1] Run decoder replay layers in CUDA graphs and trim in PIECEWISE graphs ( #59532 )
...
Signed-off-by: Juntian Liu <Juntianl777@gmail.com >
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-10-05 22:11:45 -07:00
Tianyu Guo
30ab428235
[Bugfix][Attention] Restrict FlashInfer sparse MLA FULL graphs to decode ( #59751 )
...
Signed-off-by: Tianyu Guo <guoty@inferact.ai >
2026-10-06 04:13:23 +00:00
0eac152707
[Bugfix] Route every speculation-capable row through the speculative path so recurrent state stays consistent ( #56531 )
...
Signed-off-by: Vadim Gimpelson <vadim.gimpelson@gmail.com >
Co-authored-by: Cursor Agent <cursoragent@cursor.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-10-06 03:03:44 +04:00