iHore/vLLM
8
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity
16,272 Commits 3 Branches 0 Tags
856ec4804a4e236bdd796e0954094a532c7d19bc
Commit Graph
5363 Commits
Author SHA1 Message Date
Hoang NguyenClaudemergify[bot] <37929162+mergify[bot]@users.noreply.github.com>Isotr0py
 		c3ad791e1a [Bugfix][Gemma 4] Clamp soft-token estimate to max_soft_tokens (#40796) 
Signed-off-by: Hoang Nguyen <118159510+hnt2601@users.noreply.github.com>
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Isotr0py <mozf@mail2.sysu.edu.cn>
 2026-05-02 06:34:59 +00:00
Luka Govedič d58c42e19c [vLLM IR] 2/N fused_add_rms_norm and maybe_inplace overload (#36823) 
Signed-off-by: Luka Govedič <lgovedic@redhat.com>
Signed-off-by: Luka Govedič <ProExpertProg@users.noreply.github.com>
 2026-05-01 23:41:15 -04:00
Ekagra Ranjan 3e49479c4b Limit concurrency on test_transcription_api_correctness.py (#41478) 
Signed-off-by: Ekagra Ranjan <3116519+ekagra-ranjan@users.noreply.github.com>
 2026-05-02 03:19:07 +00:00
John Calderon 964a4bc2a5 [MM][CG] Support ViT CG for Qwen2.5-VL (#40830) 
Signed-off-by: John Calderon <jcalderon@nvidia.com>
 2026-05-02 11:10:14 +08:00
Matthew BonanniMichael GoinLucas Wilkinsonmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 		f3fef12350 [Attention] Abstract the MLA prefill backends and eliminate cuDNN (#32623) 
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
Co-authored-by: Michael Goin <mgoin64@gmail.com>
Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-05-01 13:36:20 -04:00
Michael Goin 3ccc1ff495 [Eval][CI] Add basic mrcr eval to tests/evals/ (#40164) 
Signed-off-by: mgoin <mgoin64@gmail.com>
 2026-05-01 12:00:38 -04:00
vllmellmRita Brugarolasjunkang1991Luka GovedičTJian
 		529c671e80 [ROCm][FEAT] AITER Fused Allreduce + RMSNorm (#37646) 
Signed-off-by: vllmellm <vllm.ellm@embeddedllm.com>
Signed-off-by: Rita Brugarolas Brufau <rita.brugarolasbrufau@amd.com>
Signed-off-by: junkang1991 <junkangchow@gmail.com>
Co-authored-by: Rita Brugarolas <Rita.BrugarolasBrufau@amd.com>
Co-authored-by: junkang1991 <junkangchow@gmail.com>
Co-authored-by: Luka Govedič <ProExpertProg@users.noreply.github.com>
Co-authored-by: TJian <tunjian.tan@embeddedllm.com>
 2026-05-01 23:07:18 +08:00
sungsoo ha 4f7bde572a [Kernel] Pack output and LSE in DCP A2A (#41160) 2026-05-01 09:01:17 -04:00
Or Ozeri 2fa1f8ec00 [kv_offload+HMA][13/N]: Enable HMA support (#41445) 
This is the final PR in a series to enables HMA support for the
offloading connector. The connector advertises `SupportsHMA`
and is validated with unit tests and e2e tests.

Signed-off-by: Or Ozeri <oro@il.ibm.com>
 2026-05-01 12:30:03 +01:00
Seiji EicherMarwan SarieddineClaude
 		a3ec4a35f5 [Bugfix][Metrics] Fix RayPrometheusMetric.labels() returning shared labeled child (#40840) 
When vLLM runs with Ray Prometheus `vllm:request_success{finished_reason=...}`
only ever increments the repetition bucket regardless of the request's actual finish
reason; stop, length, abort, and error stay at zero. Root cause was `labels()` mutated
the wrapped Ray metric's default tags in place and returned self, so every `.labels(...)`
call on a given wrapper returned the same object. 

Co-authored-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
Signed-off-by: Seiji Eicher <seiji@anyscale.com>
 2026-05-01 08:43:39 +01:00
Bugen Zhaoandwang.yuqi a07642667d [Bugfix] Pass reasoning parser kwargs to structured output (#41199) 
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
Co-authored-by: wang.yuqi <yuqi.wang@daocloud.io>
 2026-04-30 23:38:02 -07:00
baonudesifeizhai c3868bbbe4 [compile] Add FlashInfer FP8 async TP fusion and preserve allreduce fusion ordering #27893 (#39505) 
Signed-off-by: baonudesifeizhai <baonudesifeizhai@gmail.com>
Signed-off-by: baonudesifeizhai <85092850+baonudesifeizhai@users.noreply.github.com>
Signed-off-by: roG0d <baonudesifeizhai@gmail.com>
 2026-05-01 05:08:34 +00:00
sychen52 947138b6c2 Add nvfp4 kv cache support (#40177) 
Signed-off-by: Shiyang Chen <shiychen@nvidia.com>
 2026-05-01 04:55:16 +00:00
Juhi Mittalandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> 6b6ac6c3c7 [Kernel][MoE] Support GELU on TRT-LLM NvFP4 fused MoE for Gemma4 (#41050) 
Signed-off-by: Juhi Mittal <juhim@nvidia.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-05-01 03:37:43 +00:00
Dong W 7198940b39 [Model] Add Moondream3 model support(only query and caption skills) (#32325) 
Signed-off-by: Dong Wang <dongw2019@gmail.com>
 2026-05-01 10:06:48 +08:00
Luis 🚀andAndrew Sansom 14043dfecd feat: Enable prompt_embeds Content Part Support in vLLM Chat Completions API (#40720) 
Signed-off-by: Luis Robaina <luis@protopia.ai>
Signed-off-by: Luis Robaina 🚀 <luisfabian1545@gmail.com>
Signed-off-by: LuisRobaina <luis@protopia.ai>
Co-authored-by: Andrew Sansom <qthequartermasterman@gmail.com>
 2026-05-01 10:05:55 +08:00
Andreas Karatzas 1adaa5056b [ROCm][CI] Add ROCm score absolute tolerance floor (#41341) 
Signed-off-by: Andreas Karatzas <akaratza@amd.com>
 2026-04-30 18:59:35 -07:00
Nick Hill dd5506a157 [Core] Simplify handling of scheduler_reserve_full_isl option (#41064) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-04-30 18:10:00 -07:00
Yongye ZhuandRoger Wang a3c83ff2fd Faster per-token fp8 group quant packed kernel for blackwell (#41326) 
Signed-off-by: Yongye Zhu <zyy1102000@gmail.com>
Co-authored-by: Roger Wang <hey@rogerw.io>
 2026-04-30 18:09:55 -07:00
fxmarty-amdandClaude 2917d6363a [NVFP4][Hopper/AMD Instinct] Add Triton kernels for NVFP4 dequantization and QDQ emulation (#40033) 
Signed-off-by: Felix Marty <Felix.Marty@amd.com>
Co-authored-by: Claude <noreply@anthropic.com>
 2026-04-30 17:35:48 -04:00
Stefano Castagnetta efb4cdf2b8 [CI/Build] Skip Prithvi/Terratorch model-registry tests when terratorch is missing (#41389) 
Signed-off-by: Stefano Castagnetta <scastagnetta@nvidia.com>
 2026-04-30 12:47:55 -07:00
Stefano Castagnettaandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> 92a7c121b6 [CI] Add MTP coverage: Qwen3.5 correctness + no-sync spec decode (#40472) 
Signed-off-by: Stefano Castagnetta <scastagnetta@nvidia.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-04-30 12:24:09 -07:00
Stefano Castagnetta 10558f5f46 [CI/Build] Skip terratorch + torchgeo while PyPI has lightning quarantined (#41377) 
Signed-off-by: Stefano Castagnetta <scastagnetta@nvidia.com>
 2026-04-30 07:59:07 -07:00
snadampal 3179e53135 [P/D] Prefill compute optimizations with bi-directional KV cache transfers between P and D nodes (#32553) 
Signed-off-by: Sunita Nadampalli <nadampal@amazon.com>
 2026-04-30 10:14:20 +00:00
Nicolò Lucchesi efdc95674d [KVConnector] MultiConnector SupportsHMA (#39571) 
Signed-off-by: NickLucche <nlucches@redhat.com>
 2026-04-30 02:10:50 -07:00
Chenxi Qianandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> 54146a9bf9 [Bugfix] correct h matrix layout in chunk_kda output kernel (#40956) 
Signed-off-by: ChenxiQian <chenxi.qian.cq@outlook.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-04-30 16:22:41 +08:00
Ekagra Ranjan a04e0cf3b8 Fix Cohere ASR after HF upgrade (#40582) 
Signed-off-by: Ekagra Ranjan <3116519+ekagra-ranjan@users.noreply.github.com>
 2026-04-29 23:39:04 -07:00
Dhruv SingalClaude Opus 4.6Dhruv SingalYour NameOpenCodeSimon Mo
 		cb1b02d0e8 [Frontend] Add VLLM_SKIP_MODEL_NAME_VALIDATION environment variable (#34676) 
Signed-off-by: Dhruv Singal <dhruvsingalabc@gmail.com>
Signed-off-by: Dhruv Singal <dsingal@Dhruvs-MacBook-Pro.local>
Signed-off-by: Your Name <you@example.com>
Signed-off-by: vLLM Assistant <assistant@vllm.ai>
Signed-off-by: Simon Mo <simon.mo@hey.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Dhruv Singal <dsingal@Dhruvs-MacBook-Pro.local>
Co-authored-by: Your Name <you@example.com>
Co-authored-by: OpenCode <noreply@openai.com>
Co-authored-by: Simon Mo <simon.mo@hey.com>
 2026-04-29 23:19:09 -07:00
Martin HickeyandOr Ozeri c42981d034 [Refactor][kv_offload] KV Offloading maintainability improvements (#40538) 
Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
 2026-04-30 05:55:31 +03:00
Wei Zhao 0ff1bf9bb1 [Bugfix] Fix failure to allocate KV blocks error (#41282) 
Signed-off-by: wzhao18 <wzhao18.sz@gmail.com>
 2026-04-29 18:44:07 -07:00
Nick Hill 18599bfdf2 [Ci][BugFix] Fix slow DP tests due to bad teardown logic (#41166) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-04-29 19:31:00 -04:00
Thien Tran 296741d025 [DSv4] Use cvt PTX for FP32->FP4 conversion (#41015) 
Signed-off-by: Thien Tran <gau.nernst@yahoo.com.sg>
 2026-04-29 16:16:40 -07:00
Hemanth Acharya 6841f5dc77 [ROCm] Add env flags to disable dynamic MXFP4 quant and enable AITER tuned GEMMs for Attention Projection Layers (#39987) 
Signed-off-by: Hemanth Acharya <heachary@amd.com>
 2026-04-29 16:07:46 -07:00
Rishi PuriClaudeStefano Castagnetta
 		ccfb620c62 Create tests/distributed/test_mnnvl_alltoall.py (#35241) 
Signed-off-by: Rishi Puri <riship@nvidia.com>
Signed-off-by: Claude <claude@anthropic.com>
Signed-off-by: Stefano Castagnetta <scastagnetta@nvidia.com>
Co-authored-by: Claude <claude@anthropic.com>
Co-authored-by: Stefano Castagnetta <scastagnetta@nvidia.com>
 2026-04-29 21:56:56 +00:00
Aaron HaoJunjie ZhangNick Hill
 		0335316a9b [BUG] Two phase pause to prevent deadlock (#39366) 
Signed-off-by: ahao-anyscale <ahao@anyscale.com>
Signed-off-by: Aaron Hao <ahao@anyscale.com>
Co-authored-by: Junjie Zhang <junj.jay.zhang@gmail.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
 2026-04-29 17:51:03 -04:00
Laith Sakka 6f20f81cbf Replace shape_invariants with simpler apprach in dynamic_arg_dims utilizing shape_id property. (#36194) 
Signed-off-by: Laith Sakka <lsakka@meta.com>
 2026-04-29 18:32:15 +00:00
danisereb d1a75e303d Fix timeout when using LoRA adapters with Nemotron Super (#40916) 
Signed-off-by: Daniel Serebrenik <daserebrenik@nvidia.com>
 2026-04-30 01:39:49 +08:00
Terrence Zhao 91a2d39014 [Models] Cohere MoE (#40817) 
Signed-off-by: Terrencezzj <terrence@cohere.ai>
 2026-04-29 15:54:54 +00:00
Artem Perevedentsev b92ef9ec5a [Perf] Enable FlashInfer top-k/top-p sampler by default (#40376) 
Signed-off-by: Artem Perevedentsev <aperevedents@nvidia.com>
 2026-04-29 19:10:34 +04:00
Tianmu LiClaudeLi, Jiang <jiang1.li@intel.com>
 		22524f7a92 [Feat] CPU fp8 attn for AMX/AVX-512 (#39445) 
Signed-off-by: Li, Tianmu <tianmu.li@intel.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Li, Jiang <jiang1.li@intel.com>
 2026-04-29 20:43:21 +08:00
Bugen Zhao 33f36d4260 [DSV4] Support max reasoning effort (#40982) 
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
 2026-04-29 11:03:47 +00:00
AlecandOpenAI Codex 3f1a4bb639 build: embed image provenance metadata in vLLM containers (#40653) 
Signed-off-by: Alec Flowers <aflowers@nvidia.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
 2026-04-29 03:07:41 -07:00
Chauncey 762022cafb [Bugfix] DSV32/V4 add missing type conversion for non-streaming tool calls (#41198) 
Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>
 2026-04-29 09:55:07 +00:00
Chauncey 3885d340a4 [Frontend]Responses API supports Tool/Function calling with streaming with named tool/function (#41110) 
Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>
 2026-04-29 09:11:27 +00:00
Chauncey 92879e12ba [CI] fix test_rotary_embedding_opcheck format error (#41202) 
Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>
 2026-04-29 00:32:37 -07:00
rishitdholakia13Nick HillCyrus Leung
 		68dd7db810 [Reasoning] Support for speculative decoding with thinking budget (#34668) 
Signed-off-by: rishitdholakia13 <rishit+github@cohere.com>
Signed-off-by: rishitdholakia13 <123388671+rishitdholakia13@users.noreply.github.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>
 2026-04-29 06:14:52 +00:00
Itay EtelisItay EtelisOr OzeriOr Ozeri
 		8a8c9b564e [KV Offload] Per-job store completion for CPU offloading connector (#39186) 
Signed-off-by: Itay Etelis <itay.etelis@ibm.com>
Signed-off-by: Itay Etelis <92247226+Etelis@users.noreply.github.com>
Co-authored-by: Itay Etelis <itay.etelis@ibm.com>
Co-authored-by: Or Ozeri <or@ozery.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
 2026-04-29 08:52:55 +03:00
Jee Jee Li a269744e9f [Bugfix] Fix rope (#41113) 
Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>
 2026-04-28 22:42:35 -07:00
Wei ZhaoandWei Zhao 8b49cf3a37 [Bugfix] Fix max_num_batched_token not captured in cuda graph (#40734) 
Signed-off-by: wzhao18 <wzhao18.sz@gmail.com>
Signed-off-by: Wei Zhao <51183510+wzhao18@users.noreply.github.com>
Co-authored-by: Wei Zhao (Engrg-Hardware 1) <weizha@login-bia02.bia.clusters.nvidia.com>
 2026-04-28 21:33:06 -07:00
liangel-02 7fd05e05ae uncomment flex backend for batch invariant mode (#40842) 
Signed-off-by: Angel Li <liangel@meta.com>
 2026-04-28 21:05:14 -07:00