MOVZX
c5ab42bc0f
Merge branch 'main' into patches
2026-10-18 16:04:36 +07:00
763e3f7c91
[Bugfix][Spec Decode] Disable dynamic speculative decoding for fixed-K proposers instead of crashing ( #46399 )
...
Signed-off-by: Ting Sun <suntcrick@gmail.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com >
2026-10-17 12:41:04 -04:00
MOVZX
e0fe5df731
Merge branch 'main' into patches
2026-10-17 08:06:56 +00:00
Wei Zhao
90d6d871d6
[Attention] Add CuTe DSL backend selection for MLA context prefill ( #62069 )
...
Signed-off-by: Wei Zhao <51183510+wzhao18@users.noreply.github.com >
Signed-off-by: wzhao18 <wzhao18.sz@gmail.com >
2026-10-17 01:21:55 -04:00
b544587408
[Core][Feat] Pluggable KVCacheConfigBuilder for platform/model-specific KV cache planning ( #53558 )
...
Signed-off-by: wangxiyuan <wangxiyuan1007@gmail.com >
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai >
2026-10-16 23:16:47 +00:00
MOVZX
be95edb7f6
Merge branch 'main' into patches
...
Signed-off-by: MOVZX <movzx@yahoo.com >
2026-10-16 17:35:42 +07:00
MOVZX and Hinata
18c06ed406
[Offloader] Stream multimodal towers from host RAM with bulk-copy
...
Add --offload-towers: place multimodal tower weights in pinned host
memory and bulk-copy each unit to the GPU only during its forward,
keeping towers out of VRAM between encodes. Unlike UVA zero-copy,
whole-tensor H2D transfers avoid re-reading operand tiles over PCIe
inside the GEMM inner loop. Tower streaming is independent of layer
group scheduling, so group_size may be 0 for tower-only offload.
Co-authored-by: Hinata <hinata@idihore.id >
Signed-off-by: MOVZX <movzx@yahoo.com >
2026-10-16 14:32:14 +07:00
maithilijoshi20 and Cyrus Leung
92dc3dd393
[Bugfix] Honor OPT token embedding width for prompt embeddings ( #61736 )
...
Signed-off-by: maithilijoshi20 <maithilij2003@gmail.com >
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk >
2026-10-16 04:41:50 +00:00
dd9e07a25f
[Feature] Add External LB Mode Support for Elastic EP Scaling ( #43202 )
...
Signed-off-by: lqh-hh <liuqihang6@huawei.com >
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com >
Co-authored-by: lqh-hh <liuqihang6@huawei.com >
Co-authored-by: Tyler Michael Smith <tlrmchlsmth@gmail.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
Co-authored-by: 2017403603 <705328646@qq.com >
Co-authored-by: Julien Mancuso <jmancuso@nvidia.com >
Co-authored-by: Itay Alroy <75032521+itayalroy@users.noreply.github.com >
Co-authored-by: Codex <noreply@openai.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-10-15 21:29:02 -04:00
Andreas Karatzas and Claude
29345f42a6
[ROCm][Qwen4Exp] Support HC and MoE sequence parallelism on ROCm ( #61471 )
...
Signed-off-by: Andreas Karatzas <akaratza@amd.com >
Co-authored-by: Claude <noreply@anthropic.com >
2026-10-15 20:11:17 -05:00
de7d70676d
[Perf] Low-latency CPU all-reduce for DP metadata syncs ( #61767 )
...
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: hjjq <50634613+hjjq@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com >
2026-10-15 15:29:13 -07:00
Stefano Castagnetta and Kevin H. Luu
784d1fff0c
[MRV2][Spec Decode] Quantized draft lm_head for drafters that share the target head ( #59973 )
...
Signed-off-by: Stefano Castagnetta <scastagnetta@nvidia.com >
Co-authored-by: Kevin H. Luu <khluu000@gmail.com >
2026-10-15 22:06:40 +00:00
d92a4f25f2
[watermarking] Add DFlash support to dual-key gumbel-max watermarking ( #59554 )
...
Signed-off-by: Shern Shiou Tan <shernshiou@gmail.com >
Signed-off-by: Simon Veitner <sveitner@redhat.com >
Signed-off-by: Raphael Rialland <raphael.rialland@mistral.ai >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: Simon Veitner <sveitner@redhat.com >
Co-authored-by: Raphael Rialland <raphael.rialland@mistral.ai >
Co-authored-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com >
2026-10-15 22:06:06 +00:00
e711723352
[ROCm] gfx1250 optimizations and model enablement ( #58800 )
...
Signed-off-by: jpvillam <juan.villamizar@amd.com >
Signed-off-by: Jaden Mathias <jaden.mathias@amd.com >
Signed-off-by: Juan Villamizar <jpvillam@amd.com >
Signed-off-by: jpvillam <jpvillam@amd.com >
Signed-off-by: Omar Muhammad <omar.muhammad@amd.com >
Signed-off-by: Daniel <danichan@amd.com >
Signed-off-by: Juan Villamizar <100237675+jpvillam-amd@users.noreply.github.com >
Co-authored-by: Jaden Mathias <jaden.mathias@amd.com >
Co-authored-by: jpvillam <jpvillam@amd.com >
Co-authored-by: Omar Muhammad <omar.muhammad@amd.com >
Co-authored-by: Daniel <danichan@amd.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
Co-authored-by: Daniel Chan <Daniel.Chan@amd.com >
2026-10-15 14:51:57 -07:00
7cd7982dfd
[Spec Decode] Don't reserve DFlash/DSpark draft slots in the scheduler token budget ( #59468 )
...
Signed-off-by: zixi-qi <zixi@inferact.ai >
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-10-16 01:23:58 +08:00
ed3050add4
[Core] Apply prefill_schedule_interval outside data parallelism ( #54627 )
...
Signed-off-by: lifulu <fululi12@amd.com >
Signed-off-by: lifulu <lifulu@users.noreply.github.com >
Co-authored-by: Cursor Agent <cursoragent@cursor.com >
Co-authored-by: lifulu <lifulu@users.noreply.github.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
2026-10-15 11:03:13 -05:00
MOVZX
76df77c04e
Merge branch 'main' into patches
...
Signed-off-by: MOVZX <movzx@yahoo.com >
2026-10-15 21:16:39 +07:00
24f632ca7e
[Bugfix] Engage indexer TP row sharding on total rows, not per-rank rows ( #61647 )
...
Signed-off-by: Zheng Cai <caizheng1993@gmail.com >
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com >
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com >
Co-authored-by: zjy0516 <riverclouds.zhu@qq.com >
2026-10-15 13:54:14 +00:00
Timofey and Codex
716c5225ae
[Model] Add AliceAI support ( #57963 )
...
Signed-off-by: timka <timka.byzov@yandex.ru >
Co-authored-by: Codex <noreply@openai.com >
2026-10-15 10:15:03 +00:00
Misha Goin and Claude Opus 5.5
1c4c319792
[Frontend] Advertise --enable-x/--disable-x for boolean CLI flags ( #61479 )
...
Signed-off-by: mgoin <mgoin64@gmail.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-15 01:08:02 -07:00
48f17b77bb
[Bugfix] Reject mismatched float16/bfloat16 KV cache dtype ( #60885 )
...
Signed-off-by: cipheraxat <cipheraxat@users.noreply.github.com >
Co-authored-by: cipheraxat <cipheraxat@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-15 08:06:59 +00:00
Shanshan Shen
f46021a7fa
[ROCm][Perf] Enable DPA+ETP prefill compute/comm dual-batch-overlap for DeepSeek-V4 ( #57773 )
...
Signed-off-by: Shanshan Shen <87969357+shen-shanshan@users.noreply.github.com >
Signed-off-by: shen-shanshan <467638484@qq.com >
2026-10-15 11:16:48 +08:00
Woosuk Kwon and Claude Opus 5.5
8530aed2b7
[Spec Decode] Remove Medusa and MLP speculator support ( #61701 )
...
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-14 17:29:23 -07:00
Luca Motz and Claude Opus 5.5
2d09fdcdc2
[Quantization] Auto-disable DeepGEMM for Kolibri 1 on SM12x ( #61274 )
...
Signed-off-by: Luca Motz <luca.motz@icloud.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-14 12:59:50 +00:00
Sherif Waly
be4ce2e505
[Bugfix] Share PCP-aware local GPU indexing for NUMA binding ( #61237 )
2026-10-14 10:54:03 +02:00
3ccc0a90b6
[KV Connector] Support sleep mode with MooncakeConnector over RDMA ( #59625 )
...
Signed-off-by: aoshen02 <aoshen@inferact.ai >
Signed-off-by: Dakai An <dakaian108@gmail.com >
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com >
Co-authored-by: Dakai An <dakaian108@gmail.com >
Co-authored-by: Dakai An <77474977+andakai@users.noreply.github.com >
2026-10-14 04:18:47 +00:00
c068db0e1d
[Bugfix][Compile] Replace Transformers AOT caching with graph caching ( #58918 )
...
Signed-off-by: fangyao fan <2281693949@qq.com >
Signed-off-by: Taimys <ffy2281693949@gmail.com >
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Co-authored-by: Taimys <ffy2281693949@gmail.com >
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
2026-10-13 19:17:29 +00:00
de8291e24f
[Core] Add lifecycle hooks for worker extensions ( #59463 )
...
Signed-off-by: Dakai An <dakaian108@gmail.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: aoshen02 <aoshen@inferact.ai >
2026-10-13 21:51:05 +08:00
2cc2e68d1c
[Core][Mamba] Remove leftovers of mamba_cache_mode="all" ( #60043 )
...
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com >
Signed-off-by: Jiangyun Zhu <zhujiangyun@inferact.ai >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-10-13 00:58:59 -07:00
Fangzhou Ai and Claude Opus 5.5
cc2a6eea1a
[ROCm][Perf][DSv4.1] Mono decode on every backbone layer ( #61082 )
...
Signed-off-by: Fangzhou Ai <fangzhou.ai@amd.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-13 10:16:50 +08:00
Maroon Ayoub and Claude
d6aca837c3
[Attention] Make the sparse MLA prefill workspace size configurable ( #61023 )
...
Signed-off-by: Maroon Ayoub <mayoub@redhat.com >
Co-authored-by: Claude <noreply@anthropic.com >
2026-10-12 14:56:51 -04:00
Woosuk Kwon and Claude Opus 5.5
e1a5086256
[Attention] Remove cascade attention ( #61126 )
...
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-12 10:41:07 -07:00
Harry Huang and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
c75cb63cd2
[Qwen4Exp] support Sequence Parallelism for Qwen3.8-Flash-Next ( #56322 )
...
Signed-off-by: huanghaoyan.hhy <huanghaoyan.hhy@alibaba-inc.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-10-12 20:56:44 +08:00
Tianyu Guo
258b97ff6b
[Bugfix][EC Connector] Size transfers by measured encoder output widths ( #57904 )
...
Signed-off-by: Tianyu Guo <guoty@inferact.ai >
2026-10-12 12:32:39 +00:00
Canlin Guo and Claude Opus 5.5
fe4df3df0e
[Bugfix][GLM-5.3-Flash] Restore MTP index sharing across draft iterations ( #61079 )
...
Signed-off-by: Canlin <canlinguosdu@gmail.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-12 05:41:52 +00:00
aoshen02 and Codex
239a3fa3a8
[Sleep] Retain draft-private weights with existing parameter selector ( #61046 )
...
Signed-off-by: aoshen02 <aoshen@inferact.ai >
Co-authored-by: Codex <noreply@openai.com >
2026-10-11 23:30:56 +08:00
SeongJun Lee and lesj0610
a98247ab4d
[Model] Drop the 4x M-RoPE cache headroom for Qwen3-VL, Qwen3.5 and Qwen3.8 ( #59853 )
...
Signed-off-by: lesj0610 <lesj0610@users.noreply.github.com >
Signed-off-by: lesj0610 <lesj0610@gmail.com >
Co-authored-by: lesj0610 <lesj0610@users.noreply.github.com >
2026-10-10 06:51:52 +00:00
Nick Hill
7d0b4e57aa
[ROCm] Remove MRV1 model exceptions ( #56360 )
...
Signed-off-by: Nick Hill <nickhill123@gmail.com >
2026-10-10 05:09:19 +00:00
Wei Zhao and Codex
03a7d3776f
[Perf] Support minimum KV splits configuration for TokenSpeed MLA decode ( #60345 )
...
Signed-off-by: Wei Zhao <51183510+wzhao18@users.noreply.github.com >
Signed-off-by: wzhao18 <wzhao18.sz@gmail.com >
Co-authored-by: Codex <noreply@openai.com >
2026-10-09 23:33:47 +00:00
04b02fc8a5
[Core] shm tensor arena for efficient cpu->gpu worker broadcast ( #51207 )
...
Signed-off-by: Bolin Sun <bolins@nvidia.com >
Signed-off-by: <bolins@nvidia.com >
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Claude <noreply@anthropic.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
2026-10-09 16:13:24 -07:00
04462732a4
[Bugfix] Reject max_num_scheduled_tokens=0 at config time instead of hanging ( #60714 )
...
Signed-off-by: Vadim Gimpelson <vadim.gimpelson@gmail.com >
Co-authored-by: Claude <noreply@anthropic.com >
Co-authored-by: Hui Li <40770106+untrall@users.noreply.github.com >
2026-10-09 19:15:10 +00:00
MOVZX
ac305c44cb
Merge branch 'main' into patches
...
Signed-off-by: MOVZX <movzx@yahoo.com >
2026-10-09 17:41:13 +00:00
Mikko Tukiainen and Cursor
81c993658e
[ROCm][Perf] Fuse main QK-norm/RoPE/gate and KV-cache write into the AMD QSA prepare launch ( #59732 )
...
Signed-off-by: Mikko Tukiainen <Mikko.Tukiainen@amd.com >
Co-authored-by: Cursor <cursoragent@cursor.com >
2026-10-09 21:14:49 +08:00
a6f43e57d6
[ROCm] Enable the cuMem CUDA-graph pool offload on ROCm ( #59523 )
...
Signed-off-by: aoshen02 <aoshen@inferact.ai >
Signed-off-by: indianspeedster <cspandey016@gmail.com >
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com >
Co-authored-by: indianspeedster <cspandey016@gmail.com >
2026-10-09 08:29:55 +00:00
d8fa463e2b
[Bugfix][Prefix caching] Fix inconsistent hybrid Mamba prefix cache boundaries ( #60533 )
...
Signed-off-by: Wei Zhao <weizha@aws-cmh-slurm-1-login-01.cm.cluster >
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com >
Co-authored-by: Wei Zhao <weizha@aws-cmh-slurm-1-login-01.cm.cluster >
Co-authored-by: Codex <noreply@openai.com >
Co-authored-by: zjy0516 <riverclouds.zhu@qq.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-10-08 18:33:48 +00:00
e41d5d5543
[Bugfix] Don't warn about missing watermark config for default requests ( #60412 )
...
Signed-off-by: Simon Veitner <sveitner@redhat.com >
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com >
Co-authored-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com >
2026-10-08 15:29:20 +00:00
Woosuk Kwon
194bd72be7
[Compile] Remove torch.compile-based sequence parallelism and AsyncTP ( #60517 )
2026-10-08 08:28:23 -07:00
Harry Mellor and Claude Opus 5.5
21d9758459
Remove Python 3.10 support now that it is EOL ( #60402 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-08 02:30:52 -07:00
ARAVINDHAN T and linitra24
e58f8dde7f
[Bugfix] Use FusedMoEWithLoRA for shared MoE LoRAs on 3D models ( #60098 )
...
Signed-off-by: ARAVINDHAN T <arvindhant01@gmail.com >
Co-authored-by: linitra24 <renshuang.zhou@daocloud.io >
2026-10-08 15:04:32 +08:00
f38c6778ce
[quantization] Add per-token NVFP4 quantization MoE support for ReLU2 ( #56740 )
...
Signed-off-by: Xuanteng Huang <xuantengh@nvidia.com >
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com >
Co-authored-by: OpenAI Codex <noreply@openai.com >
2026-10-07 22:01:09 +00:00
MOVZX
