MOVZX and Hinata
e61dcb9143
[Mini-vLLM] SM86-only cut: models, backends, Rust, tests
...
- Keep only Qwen3.5, Qwen3.5-MTP and Qwen4Exp model paths; trim the model registry to 41 entries
- Delete idle attention backends (MLA, FlashInfer, ROCm-only, quant backends) and dead GDN CuTeDSL kernels
- Remove inc/quark quant loaders; keep compressed-tensors and Marlin
- Drop the Rust frontend, pooling entrypoints, and idle test suites
- Rebuild kernels for sm_80;sm_86 and strip the extensions
- Verified: full import sweep clean (optional deps only) and a dummy-load Qwen3.5 engine run on RTX 3090 via the Triton GDN path
Co-authored-by: Hinata <hinata@idihore.id >
Signed-off-by: MOVZX <movzx@yahoo.com >
2026-10-18 19:19:56 +07:00
7b6310c109
[Perf][Sampler] Speed up the small-batch split top-p pipeline ( #61996 )
...
Signed-off-by: Bowen Wang <bowen@inferact.ai >
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: vllm-agent <claw@inferact.ai >
2026-10-16 21:22:11 -07:00
ce49c47cbb
[Bugfix][Sampler] Carry the boundary tie count across blocks in the split top-p mask kernel ( #62046 )
...
Signed-off-by: cipheraxat <cipheraxat@users.noreply.github.com >
Co-authored-by: cipheraxat <cipheraxat@users.noreply.github.com >
Co-authored-by: Bowen Wang <bowen@inferact.ai >
2026-10-17 02:26:57 +00:00
Wentao Ye
8e25bbca19
[Refactor] Remove dead code multiple places ( #61843 )
...
Signed-off-by: yewentao256 <zhyanwentao@126.com >
2026-10-16 15:01:07 +00:00
e711723352
[ROCm] gfx1250 optimizations and model enablement ( #58800 )
...
Signed-off-by: jpvillam <juan.villamizar@amd.com >
Signed-off-by: Jaden Mathias <jaden.mathias@amd.com >
Signed-off-by: Juan Villamizar <jpvillam@amd.com >
Signed-off-by: jpvillam <jpvillam@amd.com >
Signed-off-by: Omar Muhammad <omar.muhammad@amd.com >
Signed-off-by: Daniel <danichan@amd.com >
Signed-off-by: Juan Villamizar <100237675+jpvillam-amd@users.noreply.github.com >
Co-authored-by: Jaden Mathias <jaden.mathias@amd.com >
Co-authored-by: jpvillam <jpvillam@amd.com >
Co-authored-by: Omar Muhammad <omar.muhammad@amd.com >
Co-authored-by: Daniel <danichan@amd.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
Co-authored-by: Daniel Chan <Daniel.Chan@amd.com >
2026-10-15 14:51:57 -07:00
1484266938
[Perf] Improve _topk_topp_kernel() for masked logits ( #61213 )
...
Signed-off-by: Tomas Ruiz <tomas.ruiz.te@gmail.com >
Co-authored-by: Codex <noreply@openai.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
2026-10-12 16:01:11 -07:00
Rehan Khan and Bob
fd0216d3b5
[Bugfix][CPU] Fix fused Gumbel-max wrapping-window bias ( #60221 )
...
Signed-off-by: Rehan Khan <Rehan.Khan7@ibm.com >
Co-authored-by: Bob <bob@ibm.com >
2026-10-08 15:15:53 +08:00
Rehan Khan and Li, Jiang <jiang1.li@intel.com>
583637f126
[CPU][Perf] Add vectorized Sampler Kernel ( #53913 )
...
Signed-off-by: Rehan Khan <Rehan.Khan7@ibm.com >
Co-authored-by: Li, Jiang <jiang1.li@intel.com >
2026-09-30 15:16:28 +08:00
Misha Goin and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
1ee7f78e80
[Perf] Reduce redundant Triton sampler warmup specializations ( #58605 )
...
Signed-off-by: mgoin <mgoin64@gmail.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-09-29 21:23:27 +00:00
김재억
32fc82343f
[Bugfix][LogitsProcessor] Validate ':' separator in custom logits processor FQCN ( #56020 )
...
Signed-off-by: 100milliongold <gadian88@gmail.com >
2026-09-25 01:49:20 +00:00
44bf011f44
[Dependency] Upgrade FlashInfer version to 0.7.0 ( #58069 )
...
Signed-off-by: wzhao18 <wzhao18.sz@gmail.com >
Co-authored-by: Codex <noreply@openai.com >
Co-authored-by: Yongye Zhu <zyy1102000@gmail.com >
Co-authored-by: Kevin H. Luu <khluu000@gmail.com >
2026-09-23 22:45:23 -07:00
Tony Lin and Flora Feng
e4340e41c9
[Feat][XPU] VLLM_BATCH_INVARIANT support for Dense/MoE models ( #55881 )
...
Signed-off-by: Tony Lin <tony.lin@intel.com >
Co-authored-by: Flora Feng <4florafeng@gmail.com >
2026-09-22 11:06:54 -04:00
liuzhenwei and Kunshang Ji
6654bcbfca
[MRV2][XPU] use xpu sample kernel in mrv2 sampler ( #57277 )
...
Signed-off-by: zhenwei-intel <zhenwei.liu@intel.com >
Co-authored-by: Kunshang Ji <kunshang.ji@intel.com >
2026-09-21 13:24:41 +08:00
Martín el Cheikh and Claude
2e50824766
[Bugfix] Fall back to native sampling when FlashInfer cannot target the GPU ( #48956 )
...
Signed-off-by: martin el cheikh <martinelcheikh@gmail.com >
Co-authored-by: Claude <noreply@anthropic.com >
2026-09-17 12:27:54 +00:00
Harry Mellor and Claude Opus 5
651a88c09a
[CI] Ignore ruff D209, rejoin the docstrings it split, and silence incompatible-rule warnings ( #57212 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
2026-09-16 10:57:24 -07:00
Harry Mellor and Claude Opus 5
8c1557a79c
Add pydocstyle to the ruff rules ( #52136 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Signed-off-by: Harry Mellor <hej.mellor@gmail.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
2026-09-16 16:44:29 +00:00
Nick Hill
f2aad6aa70
[MRV2] Buffer util simplifications ( #56888 )
...
Signed-off-by: Nick Hill <nickhill123@gmail.com >
2026-09-15 20:36:51 +00:00
aed894c190
[6/N][warmup][DSv4] Migrate sampling, and DFlash JIT kernels ( #56323 )
...
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com >
Signed-off-by: LopezCastroRoberto <rocastro@redhat.com >
Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
2026-09-12 17:24:24 -07:00
Nick Hill and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
bf95f58d10
[Core] Triton kernel for small-batch top-p only masking ( #54651 )
...
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-09-03 07:52:47 -07:00
Artem Perevedentsev
39e276eaeb
[Structured Output] Let terminal grammars stop under min_tokens (II) ( #54218 )
...
Signed-off-by: Artem Perevedentsev <aperevedents@nvidia.com >
2026-08-31 20:39:27 +04:00
MKQuantum
2f41c894e3
Fix seed loss when batch contains unseeded requests ( #51866 )
...
Signed-off-by: MKQuantum <121593006+MKQuantum@users.noreply.github.com >
2026-08-20 22:38:53 +00:00
wangxiyuan
cba06764d7
[Platform] Fill in the missing backend parameter for torch.compile ( #51781 )
...
Signed-off-by: wangxiyuan <wangxiyuan1007@gmail.com >
2026-08-19 10:31:19 +00:00
Henry Su
c94cdd0ae0
[Bugfix][Sampling] Clear empty side on thinking-budget asymmetric SWAP ( #49613 )
...
Signed-off-by: Henry Su <henrysu4707@gmail.com >
2026-08-15 21:13:02 +00:00
Nick Hill and Claude Opus 4.8
615d4cfade
[Core] Check for GPU<->CPU syncs during CI ( #43107 )
...
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com >
2026-08-14 17:42:05 -07:00
405bc86768
[Perf] Launch the top-k/top-p Triton sampler kernel with 8 warps ( #51507 )
...
Signed-off-by: BabyDrangoner <148877251+BabyDrangoner@users.noreply.github.com >
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
2026-08-10 10:55:35 -07:00
+1
4f1da84eb5
Enable gfx1250 ROCm architecture ( #46516 )
...
Signed-off-by: Douglas Lehr <Doug.Lehr@amd.com >
Signed-off-by: jpvillam <juan.villamizar@amd.com >
Signed-off-by: Daniel <danichan@amd.com >
Signed-off-by: Jaden Mathias <jaden.mathias@amd.com >
Signed-off-by: jpvillam <jpvillam@amd.com >
Signed-off-by: Juan Villamizar <100237675+jpvillam-amd@users.noreply.github.com >
Signed-off-by: Douglas Lehr <91553416+dllehr-amd@users.noreply.github.com >
Signed-off-by: Jaden Mathias <Jaden.Mathias@amd.com >
Co-authored-by: Douglas Lehr <Doug.Lehr@amd.com >
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com >
Co-authored-by: Daniel <danichan@amd.com >
Co-authored-by: Jaden Mathias <jaden.mathias@amd.com >
Co-authored-by: jpvillam <jpvillam@amd.com >
Co-authored-by: root <root@ctheliosr-1b114-f01-2.mnb.dcgpu >
Co-authored-by: Douglas Lehr <91553416+dllehr-amd@users.noreply.github.com >
Co-authored-by: dllehr <douglehr@amd.com >
Co-authored-by: Daniel Chan <Daniel.Chan@amd.com >
Co-authored-by: Andreas Karatzas <akaratza@amd.com >
2026-07-31 01:41:12 +00:00
Zach Zhu
58f9659397
[Frontend][Core] Standardize request error handling with VLLMError hierarchy ( #49665 )
...
Signed-off-by: Zach Zhu <zzqshu@126.com >
2026-07-29 04:20:40 +00:00
Michael Goin and Nick Hill
ac36a7a1e7
[MRV2][Spec Decode] Avoid rejection sampler OOM by chunking ( #48630 )
...
Signed-off-by: mgoin <mgoin64@gmail.com >
Signed-off-by: Michael Goin <mgoin64@gmail.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
2026-07-23 18:13:13 +08:00
vllm-agent and vllm-agent
1134545b6f
Revert "[Sampler] Stop upcasting logits to fp32 in apply_sampling_params" ( #48641 ) ( #49033 )
...
Co-authored-by: vllm-agent <vllm-agent@users.noreply.github.com >
2026-07-21 09:36:45 +01:00
425c4eafb0
[Sampler] Stop upcasting logits to fp32 in apply_sampling_params ( #48641 )
...
Signed-off-by: mgoin <mgoin64@gmail.com >
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com >
2026-07-17 18:15:20 -07:00
41ea2dd44a
[Bugfix][V1/V2] Fix prompt_logprobs to respect logprobs_mode ( #47680 )
...
Signed-off-by: Wojciech Wais <wojciech.wais@gmail.com >
Signed-off-by: Federico Kamelhar <209537060+fede-kamel@users.noreply.github.com >
Signed-off-by: Allen Shen <aoshen@inferact.ai >
Co-authored-by: Wojciech Wais <wojciech.wais@gmail.com >
Co-authored-by: Federico Kamelhar <209537060+fede-kamel@users.noreply.github.com >
2026-07-17 21:58:59 +01:00
rishitdholakia13 and Cursor
32e632dfeb
[Reasoning] Optimize TPOT for thinking budget when used with speculative decoding ( #46662 )
...
Signed-off-by: rishitdholakia13 <rishit+github@cohere.com >
Co-authored-by: Cursor <cursoragent@cursor.com >
2026-07-14 18:55:40 +00:00
Ashwin Giridharan
ed908cf0a0
[Bugfix] Fix thinking_token_budget not enforced after natural </think> re-entry ( #45984 )
...
Signed-off-by: Ashwin Giridharan <girida@amazon.com >
2026-07-10 22:47:51 +00:00
cda05ee8c4
[Bugfix][Reasoning] Fix thinking_token_budget not enforced on re-entry after forced end ( #43757 )
...
Signed-off-by: Ashwin Giridharan <girida@amazon.com >
Signed-off-by: Cursor Agent <cursor-agent@cursor.com >
Co-authored-by: Cursor Agent <cursor-agent@cursor.com >
Co-authored-by: Simon Mo <simon.mo@hey.com >
2026-06-30 01:04:25 +00:00
3c43237233
[Bugfix][Model Runner V2][Spec Decode] Fix int32 offset overflow in sampler kernels ( #46560 )
...
Signed-off-by: xiaojun.wei <jessiewei747@gmail.com >
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com >
Co-authored-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com >
2026-06-24 11:00:57 -07:00
Walter Beller-Morales and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
7f99e80c3b
[Perf][ThinkingBudget] reduce search space for thinking tokens ( #46425 )
...
Signed-off-by: walterbm <walter.beller.morales@gmail.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-06-24 23:02:25 +08:00
Nick Hill
7ee4d22009
[Spec Decode] Reject placeholder (-1) draft tokens in rejection sampler ( #46533 )
...
Signed-off-by: Nick Hill <nickhill123@gmail.com >
2026-06-24 03:32:32 +00:00
Nick Hill
7df3d7dada
[Core] Ensure memory is pinned prior to async h2d copy ( #45424 )
...
Signed-off-by: Nick Hill <nickhill123@gmail.com >
2026-06-20 20:02:24 -07:00
Jonathan Mamou and Li, Jiang <jiang1.li@intel.com>
351c72d6e5
[CPU] Skip Triton kernel monkey-patches when Triton-CPU is available ( #44991 )
...
Signed-off-by: jmamou <jonathan.mamou@intel.com >
Co-authored-by: Li, Jiang <jiang1.li@intel.com >
2026-06-18 18:59:30 +08:00
Juan Pérez de Algaba
8a5cf1ccd6
[Security] Fix remote DoS via invalid recovered token reinjection ( #44744 )
...
Signed-off-by: jperezde <jperezde@redhat.com >
2026-06-10 02:31:43 -07:00
Andreas Karatzas
82a42234be
[ROCm][CI] Defer AITER sampler import and isolate server test PYTHONPATH ( #44823 )
...
Signed-off-by: Andreas Karatzas <akaratza@amd.com >
2026-06-10 08:56:11 +00:00
Chaojun Zhang
fa662b1a8b
[XPU] Cap topk/topp Triton BLOCK_SIZE to 4096 to fix Top-p mask difference failures ( #44470 )
...
Signed-off-by: Chaojun Zhang <chaojun.zhang@intel.com >
2026-06-08 09:36:51 +00:00
7fe7800fa4
[BUG] Fix FP64 Gumbel precision coverage ( #43150 )
...
Signed-off-by: tianyu-z <zhangtianyupro@gmail.com >
Signed-off-by: Tianyu Zhang <53099276+tianyu-z@users.noreply.github.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
2026-06-05 19:04:14 +08:00
Woosuk Kwon
6bad553f4e
[Minor] Remove FlashInfer version check in topk_topp_sampler ( #44442 )
...
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai >
2026-06-03 21:06:00 +00:00
Nick Hill
ec8d60bea8
[Model Runner V2] Use FlashInfer sampler ( #42472 )
2026-06-03 07:59:31 -07:00
Andreas Karatzas
53b88d1dfc
[CI] Reject out-of-vocabulary before they reach the GPU logprob path ( #44042 )
...
Signed-off-by: Andreas Karatzas <akaratza@amd.com >
2026-06-02 22:27:52 -05:00
Jongseok Park and Nick Hill
ca17b6b17d
[Perf] Apply single-pass min_larger finding and binary search in Triton Top-p path. ( #42191 )
...
Signed-off-by: js_park <cakeng@naver.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
2026-06-02 17:57:26 -07:00
Tianmu Li and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
94d3f4d205
[CPU Backend] CPU top-k and top-p sampling kernels using Triton ( #43633 )
...
Signed-off-by: Li, Tianmu <tianmu.li@intel.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-05-29 15:02:39 +08:00
JohnQinAMD and TJian
ed7fe831da
[ROCm] Enable the aiter top-k/top-p sampler by default ( #43331 )
...
Signed-off-by: John Qin <yanyuan.qin@amd.com >
Co-authored-by: TJian <tunjian.tan@embeddedllm.com >
2026-05-28 13:19:59 -05:00
71d810bbf4
[XPU] Ensure RNG offset alignment with PyTorch requirements in XPU sampler ( #43028 )
...
Signed-off-by: chaojun-zhang <chaojun.zhang@intel.com >
Signed-off-by: Chaojun Zhang <chaojun.zhang@intel.com >
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-05-26 02:01:30 +00:00