MOVZX and Hinata
e61dcb9143
[Mini-vLLM] SM86-only cut: models, backends, Rust, tests
...
- Keep only Qwen3.5, Qwen3.5-MTP and Qwen4Exp model paths; trim the model registry to 41 entries
- Delete idle attention backends (MLA, FlashInfer, ROCm-only, quant backends) and dead GDN CuTeDSL kernels
- Remove inc/quark quant loaders; keep compressed-tensors and Marlin
- Drop the Rust frontend, pooling entrypoints, and idle test suites
- Rebuild kernels for sm_80;sm_86 and strip the extensions
- Verified: full import sweep clean (optional deps only) and a dummy-load Qwen3.5 engine run on RTX 3090 via the Triton GDN path
Co-authored-by: Hinata <hinata@idihore.id >
Signed-off-by: MOVZX <movzx@yahoo.com >
2026-10-18 19:19:56 +07:00
7b6310c109
[Perf][Sampler] Speed up the small-batch split top-p pipeline ( #61996 )
...
Signed-off-by: Bowen Wang <bowen@inferact.ai >
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: vllm-agent <claw@inferact.ai >
2026-10-16 21:22:11 -07:00
ce49c47cbb
[Bugfix][Sampler] Carry the boundary tie count across blocks in the split top-p mask kernel ( #62046 )
...
Signed-off-by: cipheraxat <cipheraxat@users.noreply.github.com >
Co-authored-by: cipheraxat <cipheraxat@users.noreply.github.com >
Co-authored-by: Bowen Wang <bowen@inferact.ai >
2026-10-17 02:26:57 +00:00
Wentao Ye
8e25bbca19
[Refactor] Remove dead code multiple places ( #61843 )
...
Signed-off-by: yewentao256 <zhyanwentao@126.com >
2026-10-16 15:01:07 +00:00
e711723352
[ROCm] gfx1250 optimizations and model enablement ( #58800 )
...
Signed-off-by: jpvillam <juan.villamizar@amd.com >
Signed-off-by: Jaden Mathias <jaden.mathias@amd.com >
Signed-off-by: Juan Villamizar <jpvillam@amd.com >
Signed-off-by: jpvillam <jpvillam@amd.com >
Signed-off-by: Omar Muhammad <omar.muhammad@amd.com >
Signed-off-by: Daniel <danichan@amd.com >
Signed-off-by: Juan Villamizar <100237675+jpvillam-amd@users.noreply.github.com >
Co-authored-by: Jaden Mathias <jaden.mathias@amd.com >
Co-authored-by: jpvillam <jpvillam@amd.com >
Co-authored-by: Omar Muhammad <omar.muhammad@amd.com >
Co-authored-by: Daniel <danichan@amd.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
Co-authored-by: Daniel Chan <Daniel.Chan@amd.com >
2026-10-15 14:51:57 -07:00
1484266938
[Perf] Improve _topk_topp_kernel() for masked logits ( #61213 )
...
Signed-off-by: Tomas Ruiz <tomas.ruiz.te@gmail.com >
Co-authored-by: Codex <noreply@openai.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
2026-10-12 16:01:11 -07:00
Rehan Khan and Bob
fd0216d3b5
[Bugfix][CPU] Fix fused Gumbel-max wrapping-window bias ( #60221 )
...
Signed-off-by: Rehan Khan <Rehan.Khan7@ibm.com >
Co-authored-by: Bob <bob@ibm.com >
2026-10-08 15:15:53 +08:00
Rehan Khan and Li, Jiang <jiang1.li@intel.com>
583637f126
[CPU][Perf] Add vectorized Sampler Kernel ( #53913 )
...
Signed-off-by: Rehan Khan <Rehan.Khan7@ibm.com >
Co-authored-by: Li, Jiang <jiang1.li@intel.com >
2026-09-30 15:16:28 +08:00
Misha Goin and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
1ee7f78e80
[Perf] Reduce redundant Triton sampler warmup specializations ( #58605 )
...
Signed-off-by: mgoin <mgoin64@gmail.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-09-29 21:23:27 +00:00
44bf011f44
[Dependency] Upgrade FlashInfer version to 0.7.0 ( #58069 )
...
Signed-off-by: wzhao18 <wzhao18.sz@gmail.com >
Co-authored-by: Codex <noreply@openai.com >
Co-authored-by: Yongye Zhu <zyy1102000@gmail.com >
Co-authored-by: Kevin H. Luu <khluu000@gmail.com >
2026-09-23 22:45:23 -07:00
Tony Lin and Flora Feng
e4340e41c9
[Feat][XPU] VLLM_BATCH_INVARIANT support for Dense/MoE models ( #55881 )
...
Signed-off-by: Tony Lin <tony.lin@intel.com >
Co-authored-by: Flora Feng <4florafeng@gmail.com >
2026-09-22 11:06:54 -04:00
liuzhenwei and Kunshang Ji
6654bcbfca
[MRV2][XPU] use xpu sample kernel in mrv2 sampler ( #57277 )
...
Signed-off-by: zhenwei-intel <zhenwei.liu@intel.com >
Co-authored-by: Kunshang Ji <kunshang.ji@intel.com >
2026-09-21 13:24:41 +08:00
Martín el Cheikh and Claude
2e50824766
[Bugfix] Fall back to native sampling when FlashInfer cannot target the GPU ( #48956 )
...
Signed-off-by: martin el cheikh <martinelcheikh@gmail.com >
Co-authored-by: Claude <noreply@anthropic.com >
2026-09-17 12:27:54 +00:00
Harry Mellor and Claude Opus 5
8c1557a79c
Add pydocstyle to the ruff rules ( #52136 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Signed-off-by: Harry Mellor <hej.mellor@gmail.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
2026-09-16 16:44:29 +00:00
aed894c190
[6/N][warmup][DSv4] Migrate sampling, and DFlash JIT kernels ( #56323 )
...
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com >
Signed-off-by: LopezCastroRoberto <rocastro@redhat.com >
Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
2026-09-12 17:24:24 -07:00
Nick Hill and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
bf95f58d10
[Core] Triton kernel for small-batch top-p only masking ( #54651 )
...
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-09-03 07:52:47 -07:00
MKQuantum
2f41c894e3
Fix seed loss when batch contains unseeded requests ( #51866 )
...
Signed-off-by: MKQuantum <121593006+MKQuantum@users.noreply.github.com >
2026-08-20 22:38:53 +00:00
wangxiyuan
cba06764d7
[Platform] Fill in the missing backend parameter for torch.compile ( #51781 )
...
Signed-off-by: wangxiyuan <wangxiyuan1007@gmail.com >
2026-08-19 10:31:19 +00:00
Nick Hill and Claude Opus 4.8
615d4cfade
[Core] Check for GPU<->CPU syncs during CI ( #43107 )
...
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com >
2026-08-14 17:42:05 -07:00
405bc86768
[Perf] Launch the top-k/top-p Triton sampler kernel with 8 warps ( #51507 )
...
Signed-off-by: BabyDrangoner <148877251+BabyDrangoner@users.noreply.github.com >
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
2026-08-10 10:55:35 -07:00
+1
4f1da84eb5
Enable gfx1250 ROCm architecture ( #46516 )
...
Signed-off-by: Douglas Lehr <Doug.Lehr@amd.com >
Signed-off-by: jpvillam <juan.villamizar@amd.com >
Signed-off-by: Daniel <danichan@amd.com >
Signed-off-by: Jaden Mathias <jaden.mathias@amd.com >
Signed-off-by: jpvillam <jpvillam@amd.com >
Signed-off-by: Juan Villamizar <100237675+jpvillam-amd@users.noreply.github.com >
Signed-off-by: Douglas Lehr <91553416+dllehr-amd@users.noreply.github.com >
Signed-off-by: Jaden Mathias <Jaden.Mathias@amd.com >
Co-authored-by: Douglas Lehr <Doug.Lehr@amd.com >
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com >
Co-authored-by: Daniel <danichan@amd.com >
Co-authored-by: Jaden Mathias <jaden.mathias@amd.com >
Co-authored-by: jpvillam <jpvillam@amd.com >
Co-authored-by: root <root@ctheliosr-1b114-f01-2.mnb.dcgpu >
Co-authored-by: Douglas Lehr <91553416+dllehr-amd@users.noreply.github.com >
Co-authored-by: dllehr <douglehr@amd.com >
Co-authored-by: Daniel Chan <Daniel.Chan@amd.com >
Co-authored-by: Andreas Karatzas <akaratza@amd.com >
2026-07-31 01:41:12 +00:00
Michael Goin and Nick Hill
ac36a7a1e7
[MRV2][Spec Decode] Avoid rejection sampler OOM by chunking ( #48630 )
...
Signed-off-by: mgoin <mgoin64@gmail.com >
Signed-off-by: Michael Goin <mgoin64@gmail.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
2026-07-23 18:13:13 +08:00
vllm-agent and vllm-agent
1134545b6f
Revert "[Sampler] Stop upcasting logits to fp32 in apply_sampling_params" ( #48641 ) ( #49033 )
...
Co-authored-by: vllm-agent <vllm-agent@users.noreply.github.com >
2026-07-21 09:36:45 +01:00
425c4eafb0
[Sampler] Stop upcasting logits to fp32 in apply_sampling_params ( #48641 )
...
Signed-off-by: mgoin <mgoin64@gmail.com >
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com >
2026-07-17 18:15:20 -07:00
3c43237233
[Bugfix][Model Runner V2][Spec Decode] Fix int32 offset overflow in sampler kernels ( #46560 )
...
Signed-off-by: xiaojun.wei <jessiewei747@gmail.com >
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com >
Co-authored-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com >
2026-06-24 11:00:57 -07:00
Nick Hill
7df3d7dada
[Core] Ensure memory is pinned prior to async h2d copy ( #45424 )
...
Signed-off-by: Nick Hill <nickhill123@gmail.com >
2026-06-20 20:02:24 -07:00
Andreas Karatzas
82a42234be
[ROCm][CI] Defer AITER sampler import and isolate server test PYTHONPATH ( #44823 )
...
Signed-off-by: Andreas Karatzas <akaratza@amd.com >
2026-06-10 08:56:11 +00:00
Chaojun Zhang
fa662b1a8b
[XPU] Cap topk/topp Triton BLOCK_SIZE to 4096 to fix Top-p mask difference failures ( #44470 )
...
Signed-off-by: Chaojun Zhang <chaojun.zhang@intel.com >
2026-06-08 09:36:51 +00:00
7fe7800fa4
[BUG] Fix FP64 Gumbel precision coverage ( #43150 )
...
Signed-off-by: tianyu-z <zhangtianyupro@gmail.com >
Signed-off-by: Tianyu Zhang <53099276+tianyu-z@users.noreply.github.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
2026-06-05 19:04:14 +08:00
Woosuk Kwon
6bad553f4e
[Minor] Remove FlashInfer version check in topk_topp_sampler ( #44442 )
...
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai >
2026-06-03 21:06:00 +00:00
Nick Hill
ec8d60bea8
[Model Runner V2] Use FlashInfer sampler ( #42472 )
2026-06-03 07:59:31 -07:00
Jongseok Park and Nick Hill
ca17b6b17d
[Perf] Apply single-pass min_larger finding and binary search in Triton Top-p path. ( #42191 )
...
Signed-off-by: js_park <cakeng@naver.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
2026-06-02 17:57:26 -07:00
Tianmu Li and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
94d3f4d205
[CPU Backend] CPU top-k and top-p sampling kernels using Triton ( #43633 )
...
Signed-off-by: Li, Tianmu <tianmu.li@intel.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-05-29 15:02:39 +08:00
JohnQinAMD and TJian
ed7fe831da
[ROCm] Enable the aiter top-k/top-p sampler by default ( #43331 )
...
Signed-off-by: John Qin <yanyuan.qin@amd.com >
Co-authored-by: TJian <tunjian.tan@embeddedllm.com >
2026-05-28 13:19:59 -05:00
71d810bbf4
[XPU] Ensure RNG offset alignment with PyTorch requirements in XPU sampler ( #43028 )
...
Signed-off-by: chaojun-zhang <chaojun.zhang@intel.com >
Signed-off-by: Chaojun Zhang <chaojun.zhang@intel.com >
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-05-26 02:01:30 +00:00
Xiaogang Zhou and xiaogang.zhou
d19db10974
[Bugfix] Fix native Triton top-k/top-p kernel assumes contiguous logi… ( #42739 )
...
Signed-off-by: xiaogang.zhou <xiaogang.zhou@bytedance.com >
Co-authored-by: xiaogang.zhou <xiaogang.zhou@bytedance.com >
2026-05-22 22:56:16 -07:00
Ofir Zafrir
8f16c4a5c0
[BugFix][CPU][Spec Decode] Fix Eagle implementation on CPU backend ( #42468 )
...
Signed-off-by: Ofir Zafrir <ofir.zafrir@intel.com >
2026-05-19 03:16:07 +00:00
bc03f280c8
[XPU] keep generator state of sycl kernel align with pytorch ( #41771 )
...
Signed-off-by: Yan Ma <yan.ma@intel.com >
Co-authored-by: Qiming Zhang <qiming1.zhang@intel.com >
Co-authored-by: Kunshang Ji <kunshang.ji@intel.com >
2026-05-12 11:44:47 +00:00
Nick Hill and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
bbee532988
[Perf][1/n] Eliminate various GPU<->CPU syncs ( #41429 )
...
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-05-11 20:36:03 +00:00
Kunshang Ji
2ceea42958
[XPU] use xpu topk topp sample kernel ( #39285 )
...
Signed-off-by: Kunshang Ji <kunshang.ji@intel.com >
2026-05-05 18:05:17 +08:00
Artem Perevedentsev
b92ef9ec5a
[Perf] Enable FlashInfer top-k/top-p sampler by default ( #40376 )
...
Signed-off-by: Artem Perevedentsev <aperevedents@nvidia.com >
2026-04-29 19:10:34 +04:00
Lucas Kabela and Luka Govedič
a8c6ee9b78
[Performance Improvement] Update batched_count_greater_than to handle batch size 1 without recompile ( #38933 )
...
Signed-off-by: Lucas Kabela <lucaskabela@meta.com >
Co-authored-by: Luka Govedič <ProExpertProg@users.noreply.github.com >
2026-04-09 23:51:31 +08:00
Michael Goin
51b2333be1
[Perf] Optimize top-k search in apply_top_k_top_p_triton sampler ( #37225 )
...
Signed-off-by: mgoin <mgoin64@gmail.com >
2026-03-17 11:35:17 -07:00
Kunshang Ji and Harry Mellor
16d2ad1d38
[Hardware] Replace torch.cuda.empty_cache with torch.accelerator.empty_cache ( #30681 )
...
Signed-off-by: Kunshang Ji <kunshang.ji@intel.com >
Signed-off-by: Kunshang Ji <jikunshang95@gmail.com >
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
2026-03-04 09:49:47 +00:00
Kunshang Ji
8ad54a991b
[Platform] Add current_platform.num_compute_units interface ( #35042 )
...
Signed-off-by: Kunshang Ji <kunshang.ji@intel.com >
Signed-off-by: Kunshang Ji <jikunshang95@gmail.com >
2026-02-24 22:22:49 -08:00
Kunshang Ji
92510edc32
remove cuda check in top_k_top_p_triton kernel ( #35011 )
...
Signed-off-by: Kunshang Ji <kunshang.ji@intel.com >
2026-02-24 22:22:31 -08:00
Xiao Li
30132cd144
Fix apply_top_k_top_p_triton called by non-cuda logits Tensor ( #35030 )
...
Signed-off-by: Xiao Li <ilx@meta.com >
2026-02-21 21:11:54 -08:00
c656ba3b4d
[Kernel] Triton-based Top-k and Top-p sampler kernels ( #33538 )
...
Signed-off-by: js_park <cakeng@naver.com >
Signed-off-by: Jongseok Park <37990712+cakeng@users.noreply.github.com >
Signed-off-by: Sunga Kim <sunga.kim@berkeley.edu >
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Sunga Kim <sunga.kim@berkeley.edu >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
2026-02-17 23:14:30 +00:00
Pleaplusone
77c16df31d
[ROCm][Bugfix] Disable hip sampler to fix deepseek's accuracy issue on ROCm ( #32413 )
...
Signed-off-by: ganyi <ygan@amd.com >
2026-01-15 16:35:47 +00:00
Nick Hill
10ef65eded
[BugFix] Fix bad words with speculative decoding ( #31908 )
...
Signed-off-by: Nick Hill <nickhill123@gmail.com >
2026-01-07 15:46:42 -05:00