MOVZX and Hinata
e61dcb9143
[Mini-vLLM] SM86-only cut: models, backends, Rust, tests
...
- Keep only Qwen3.5, Qwen3.5-MTP and Qwen4Exp model paths; trim the model registry to 41 entries
- Delete idle attention backends (MLA, FlashInfer, ROCm-only, quant backends) and dead GDN CuTeDSL kernels
- Remove inc/quark quant loaders; keep compressed-tensors and Marlin
- Drop the Rust frontend, pooling entrypoints, and idle test suites
- Rebuild kernels for sm_80;sm_86 and strip the extensions
- Verified: full import sweep clean (optional deps only) and a dummy-load Qwen3.5 engine run on RTX 3090 via the Triton GDN path
Co-authored-by: Hinata <hinata@idihore.id >
Signed-off-by: MOVZX <movzx@yahoo.com >
2026-10-18 19:19:56 +07:00
7b6310c109
[Perf][Sampler] Speed up the small-batch split top-p pipeline ( #61996 )
...
Signed-off-by: Bowen Wang <bowen@inferact.ai >
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: vllm-agent <claw@inferact.ai >
2026-10-16 21:22:11 -07:00
Wentao Ye
8e25bbca19
[Refactor] Remove dead code multiple places ( #61843 )
...
Signed-off-by: yewentao256 <zhyanwentao@126.com >
2026-10-16 15:01:07 +00:00
e711723352
[ROCm] gfx1250 optimizations and model enablement ( #58800 )
...
Signed-off-by: jpvillam <juan.villamizar@amd.com >
Signed-off-by: Jaden Mathias <jaden.mathias@amd.com >
Signed-off-by: Juan Villamizar <jpvillam@amd.com >
Signed-off-by: jpvillam <jpvillam@amd.com >
Signed-off-by: Omar Muhammad <omar.muhammad@amd.com >
Signed-off-by: Daniel <danichan@amd.com >
Signed-off-by: Juan Villamizar <100237675+jpvillam-amd@users.noreply.github.com >
Co-authored-by: Jaden Mathias <jaden.mathias@amd.com >
Co-authored-by: jpvillam <jpvillam@amd.com >
Co-authored-by: Omar Muhammad <omar.muhammad@amd.com >
Co-authored-by: Daniel <danichan@amd.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
Co-authored-by: Daniel Chan <Daniel.Chan@amd.com >
2026-10-15 14:51:57 -07:00
Rehan Khan and Bob
fd0216d3b5
[Bugfix][CPU] Fix fused Gumbel-max wrapping-window bias ( #60221 )
...
Signed-off-by: Rehan Khan <Rehan.Khan7@ibm.com >
Co-authored-by: Bob <bob@ibm.com >
2026-10-08 15:15:53 +08:00
Rehan Khan and Li, Jiang <jiang1.li@intel.com>
583637f126
[CPU][Perf] Add vectorized Sampler Kernel ( #53913 )
...
Signed-off-by: Rehan Khan <Rehan.Khan7@ibm.com >
Co-authored-by: Li, Jiang <jiang1.li@intel.com >
2026-09-30 15:16:28 +08:00
44bf011f44
[Dependency] Upgrade FlashInfer version to 0.7.0 ( #58069 )
...
Signed-off-by: wzhao18 <wzhao18.sz@gmail.com >
Co-authored-by: Codex <noreply@openai.com >
Co-authored-by: Yongye Zhu <zyy1102000@gmail.com >
Co-authored-by: Kevin H. Luu <khluu000@gmail.com >
2026-09-23 22:45:23 -07:00
Tony Lin and Flora Feng
e4340e41c9
[Feat][XPU] VLLM_BATCH_INVARIANT support for Dense/MoE models ( #55881 )
...
Signed-off-by: Tony Lin <tony.lin@intel.com >
Co-authored-by: Flora Feng <4florafeng@gmail.com >
2026-09-22 11:06:54 -04:00
liuzhenwei and Kunshang Ji
6654bcbfca
[MRV2][XPU] use xpu sample kernel in mrv2 sampler ( #57277 )
...
Signed-off-by: zhenwei-intel <zhenwei.liu@intel.com >
Co-authored-by: Kunshang Ji <kunshang.ji@intel.com >
2026-09-21 13:24:41 +08:00
Martín el Cheikh and Claude
2e50824766
[Bugfix] Fall back to native sampling when FlashInfer cannot target the GPU ( #48956 )
...
Signed-off-by: martin el cheikh <martinelcheikh@gmail.com >
Co-authored-by: Claude <noreply@anthropic.com >
2026-09-17 12:27:54 +00:00
Harry Mellor and Claude Opus 5
8c1557a79c
Add pydocstyle to the ruff rules ( #52136 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Signed-off-by: Harry Mellor <hej.mellor@gmail.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
2026-09-16 16:44:29 +00:00
aed894c190
[6/N][warmup][DSv4] Migrate sampling, and DFlash JIT kernels ( #56323 )
...
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com >
Signed-off-by: LopezCastroRoberto <rocastro@redhat.com >
Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
2026-09-12 17:24:24 -07:00
Nick Hill and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
bf95f58d10
[Core] Triton kernel for small-batch top-p only masking ( #54651 )
...
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-09-03 07:52:47 -07:00
MKQuantum
2f41c894e3
Fix seed loss when batch contains unseeded requests ( #51866 )
...
Signed-off-by: MKQuantum <121593006+MKQuantum@users.noreply.github.com >
2026-08-20 22:38:53 +00:00
wangxiyuan
cba06764d7
[Platform] Fill in the missing backend parameter for torch.compile ( #51781 )
...
Signed-off-by: wangxiyuan <wangxiyuan1007@gmail.com >
2026-08-19 10:31:19 +00:00
+1
4f1da84eb5
Enable gfx1250 ROCm architecture ( #46516 )
...
Signed-off-by: Douglas Lehr <Doug.Lehr@amd.com >
Signed-off-by: jpvillam <juan.villamizar@amd.com >
Signed-off-by: Daniel <danichan@amd.com >
Signed-off-by: Jaden Mathias <jaden.mathias@amd.com >
Signed-off-by: jpvillam <jpvillam@amd.com >
Signed-off-by: Juan Villamizar <100237675+jpvillam-amd@users.noreply.github.com >
Signed-off-by: Douglas Lehr <91553416+dllehr-amd@users.noreply.github.com >
Signed-off-by: Jaden Mathias <Jaden.Mathias@amd.com >
Co-authored-by: Douglas Lehr <Doug.Lehr@amd.com >
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com >
Co-authored-by: Daniel <danichan@amd.com >
Co-authored-by: Jaden Mathias <jaden.mathias@amd.com >
Co-authored-by: jpvillam <jpvillam@amd.com >
Co-authored-by: root <root@ctheliosr-1b114-f01-2.mnb.dcgpu >
Co-authored-by: Douglas Lehr <91553416+dllehr-amd@users.noreply.github.com >
Co-authored-by: dllehr <douglehr@amd.com >
Co-authored-by: Daniel Chan <Daniel.Chan@amd.com >
Co-authored-by: Andreas Karatzas <akaratza@amd.com >
2026-07-31 01:41:12 +00:00
Michael Goin and Nick Hill
ac36a7a1e7
[MRV2][Spec Decode] Avoid rejection sampler OOM by chunking ( #48630 )
...
Signed-off-by: mgoin <mgoin64@gmail.com >
Signed-off-by: Michael Goin <mgoin64@gmail.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
2026-07-23 18:13:13 +08:00
vllm-agent and vllm-agent
1134545b6f
Revert "[Sampler] Stop upcasting logits to fp32 in apply_sampling_params" ( #48641 ) ( #49033 )
...
Co-authored-by: vllm-agent <vllm-agent@users.noreply.github.com >
2026-07-21 09:36:45 +01:00
425c4eafb0
[Sampler] Stop upcasting logits to fp32 in apply_sampling_params ( #48641 )
...
Signed-off-by: mgoin <mgoin64@gmail.com >
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com >
2026-07-17 18:15:20 -07:00
Andreas Karatzas
82a42234be
[ROCm][CI] Defer AITER sampler import and isolate server test PYTHONPATH ( #44823 )
...
Signed-off-by: Andreas Karatzas <akaratza@amd.com >
2026-06-10 08:56:11 +00:00
7fe7800fa4
[BUG] Fix FP64 Gumbel precision coverage ( #43150 )
...
Signed-off-by: tianyu-z <zhangtianyupro@gmail.com >
Signed-off-by: Tianyu Zhang <53099276+tianyu-z@users.noreply.github.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
2026-06-05 19:04:14 +08:00
Woosuk Kwon
6bad553f4e
[Minor] Remove FlashInfer version check in topk_topp_sampler ( #44442 )
...
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai >
2026-06-03 21:06:00 +00:00
Nick Hill
ec8d60bea8
[Model Runner V2] Use FlashInfer sampler ( #42472 )
2026-06-03 07:59:31 -07:00
Tianmu Li and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
94d3f4d205
[CPU Backend] CPU top-k and top-p sampling kernels using Triton ( #43633 )
...
Signed-off-by: Li, Tianmu <tianmu.li@intel.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-05-29 15:02:39 +08:00
JohnQinAMD and TJian
ed7fe831da
[ROCm] Enable the aiter top-k/top-p sampler by default ( #43331 )
...
Signed-off-by: John Qin <yanyuan.qin@amd.com >
Co-authored-by: TJian <tunjian.tan@embeddedllm.com >
2026-05-28 13:19:59 -05:00
71d810bbf4
[XPU] Ensure RNG offset alignment with PyTorch requirements in XPU sampler ( #43028 )
...
Signed-off-by: chaojun-zhang <chaojun.zhang@intel.com >
Signed-off-by: Chaojun Zhang <chaojun.zhang@intel.com >
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-05-26 02:01:30 +00:00
Ofir Zafrir
8f16c4a5c0
[BugFix][CPU][Spec Decode] Fix Eagle implementation on CPU backend ( #42468 )
...
Signed-off-by: Ofir Zafrir <ofir.zafrir@intel.com >
2026-05-19 03:16:07 +00:00
bc03f280c8
[XPU] keep generator state of sycl kernel align with pytorch ( #41771 )
...
Signed-off-by: Yan Ma <yan.ma@intel.com >
Co-authored-by: Qiming Zhang <qiming1.zhang@intel.com >
Co-authored-by: Kunshang Ji <kunshang.ji@intel.com >
2026-05-12 11:44:47 +00:00
Kunshang Ji
2ceea42958
[XPU] use xpu topk topp sample kernel ( #39285 )
...
Signed-off-by: Kunshang Ji <kunshang.ji@intel.com >
2026-05-05 18:05:17 +08:00
Artem Perevedentsev
b92ef9ec5a
[Perf] Enable FlashInfer top-k/top-p sampler by default ( #40376 )
...
Signed-off-by: Artem Perevedentsev <aperevedents@nvidia.com >
2026-04-29 19:10:34 +04:00
Kunshang Ji
92510edc32
remove cuda check in top_k_top_p_triton kernel ( #35011 )
...
Signed-off-by: Kunshang Ji <kunshang.ji@intel.com >
2026-02-24 22:22:31 -08:00
Xiao Li
30132cd144
Fix apply_top_k_top_p_triton called by non-cuda logits Tensor ( #35030 )
...
Signed-off-by: Xiao Li <ilx@meta.com >
2026-02-21 21:11:54 -08:00
c656ba3b4d
[Kernel] Triton-based Top-k and Top-p sampler kernels ( #33538 )
...
Signed-off-by: js_park <cakeng@naver.com >
Signed-off-by: Jongseok Park <37990712+cakeng@users.noreply.github.com >
Signed-off-by: Sunga Kim <sunga.kim@berkeley.edu >
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Sunga Kim <sunga.kim@berkeley.edu >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
2026-02-17 23:14:30 +00:00
Pleaplusone
77c16df31d
[ROCm][Bugfix] Disable hip sampler to fix deepseek's accuracy issue on ROCm ( #32413 )
...
Signed-off-by: ganyi <ygan@amd.com >
2026-01-15 16:35:47 +00:00
Harry Mellor and Cyrus Leung
9eec282cb5
Guard FlashInfer sampler using the same check as FlashInfer attention backend ( #29415 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk >
2025-11-28 08:34:48 -08:00
Xiao Li
ed6ae1e36a
[AITER] [ROCm] Fix crash when loading llama4 model with old aiter version installed, fallback to forward_native implementation ( #29124 )
...
Signed-off-by: Xiao Li <ilx@meta.com >
2025-11-20 17:54:35 -08:00
vllmellm
0af3d4f0df
[FEAT] [AITER] [ROCm] integrate aiter sampling ops ( #26084 )
...
Signed-off-by: vllmellm <vllm.ellm@embeddedllm.com >
2025-11-18 17:28:34 +00:00
Zhang Xiangze
7bdb42b2f2
[CPU]Avoid repeated random sample compile ( #28260 )
...
Signed-off-by: Zhang Xiangze <Xiangze.Zhang@arm.com >
2025-11-07 11:03:57 +00:00
Wentao Ye
52efc34ebf
[Log] Optimize Startup Log ( #26740 )
...
Signed-off-by: yewentao256 <zhyanwentao@126.com >
2025-10-24 19:27:04 -04:00
Jee Jee Li
fec2b341ad
[Kernel] Lazy import FlashInfer ( #26977 )
2025-10-17 04:48:18 +00:00
Akash kaothalkar and Akash Kaothalkar
f7d318de2b
[Hardware][CPU][PowerPC]Disable torch.compile() in toptopk sampling ( #26987 )
...
Signed-off-by: Akash Kaothalkar <akash.kaothalkar@ibm.com >
Co-authored-by: Akash Kaothalkar <akash.kaothalkar@ibm.com >
2025-10-15 22:36:59 -07:00
Michael Goin
e66d787bce
Disable FlashInfer sampler by default ( #26859 )
...
Signed-off-by: mgoin <mgoin64@gmail.com >
2025-10-15 02:35:18 +00:00
ihb2032
4a61950f4d
[Hardware][CPU] Disable torch.compile for RISC-V to prevent APIError ( #26693 )
...
Signed-off-by: lyd1992 <liuyudong@iscas.ac.cn >
Signed-off-by: ihb2032 <1355790728@qq.com >
Signed-off-by: lyd1992 <liuyudong@iscas.ac.cn
2025-10-13 07:56:01 -07:00
Harry Mellor
8fcaaf6a16
Update Optional[x] -> x | None and Union[x, y] to x | y ( #26633 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
2025-10-12 09:51:31 -07:00
Harry Mellor
2f99f2f506
Tidy vllm/config/__init__.py to only add classes and functions ( #26405 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
2025-10-08 07:10:00 -07:00
Harry Mellor
d6953beb91
Convert formatting to use ruff instead of yapf + isort ( #26247 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
2025-10-05 07:06:22 -07:00
Russell Bryant
532a6cfccb
[ux] Switch a warning to debug about a pytorch fallback ( #23750 )
...
Signed-off-by: Russell Bryant <rbryant@redhat.com >
2025-09-25 14:38:16 +00:00
Li, Jiang
eb32335e35
[CPU] update torch 2.8 and fix missing fields in TorchSDPAMetadata ( #25652 )
...
Signed-off-by: jiang1.li <jiang1.li@intel.com >
2025-09-25 13:29:11 +00:00
Harry Mellor
aed16879a9
Move ModelConfig from config/__init__.py to config/model.py ( #25252 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
2025-09-19 16:22:33 +00:00
Woosuk Kwon
105d3d62ef
[TPU] Remove TopKTopPSampler dependency for TPU sampler ( #24391 )
...
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu >
2025-09-07 01:12:36 -07:00