iHore/vLLM
8
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity
21,722 Commits 3 Branches 0 Tags
382970ee6ca490aeaaaf4e32c53695b581ff61ba
Commit Graph
166 Commits
Author SHA1 Message Date
Misha Goin db7a24c230 [Perf] Add fused DFlash2 grouped convolution (#55960) 
Signed-off-by: mgoin <mgoin64@gmail.com>
 2026-09-17 21:38:59 +00:00
Doug Brown 2b02c6c29b [Model] Add Cohere2MoE Eagle3 auxiliary hidden states (#49819) 
Signed-off-by: Doug Brown <doug@github.douggo.com>
 2026-09-17 20:09:36 +00:00
Harry MellorandClaude Opus 5 651a88c09a [CI] Ignore ruff D209, rejoin the docstrings it split, and silence incompatible-rule warnings (#57212) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
 2026-09-16 10:57:24 -07:00
Harry MellorandClaude Opus 5 8c1557a79c Add pydocstyle to the ruff rules (#52136) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Signed-off-by: Harry Mellor <hej.mellor@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
 2026-09-16 16:44:29 +00:00
yzong-rh e6960af33b [Refactor] StructuredOutputManager x Speculative Decoding Refactor (#48200) 2026-09-15 09:48:49 -04:00
Giancarlo Delfin 3bb0a03f35 [Model Runner V2] Acceptance estimation for adaptive verification (#52228) 
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
 2026-09-14 15:58:48 -07:00
Matthew BonanniCodexLucas WilkinsonLucas WilkinsonOpenAI Codex
 		82a85dc1d2 [Attention] Add Triton/FlashInfer composite for multimodal prefix attention (#56305) 
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com>
Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
 2026-09-13 14:29:41 +00:00
Woosuk KwonandOpenAI Codex 2671fedfc7 [MRV2] Revert explicit Triton JIT warmup migration (#56654) 
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
Co-authored-by: OpenAI Codex <codex@openai.com>
 2026-09-13 04:07:56 +00:00
Roberto L. CastroLucas WilkinsonOpenAI Codex
 		aed894c190 [6/N][warmup][DSv4] Migrate sampling, and DFlash JIT kernels (#56323) 
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
Signed-off-by: LopezCastroRoberto <rocastro@redhat.com>
Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
 2026-09-12 17:24:24 -07:00
Raphaël RiallandOpenAI CodexSimon VeitnerTomas Ruiz
 		7ee8a6dd01 [watermarking] Dual-key gumbel-max watermarking for speculative decoding support (#56122) 
Signed-off-by: Raphael Rialland <raphael.rialland@mistral.ai>
Signed-off-by: Simon Veitner <sveitner@redhat.com>
Signed-off-by: Tomas Ruiz <tomas.ruiz.te@gmail.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
Co-authored-by: Simon Veitner <sveitner@redhat.com>
Co-authored-by: Tomas Ruiz <tomas.ruiz.te@gmail.com>
 2026-09-12 18:42:01 +00:00
Canlin GuoandCodex dca96bf97b [Bugfix][Spec Decode] Avoid fastsafetensors deadlock for PP draft models (#54416) 
Signed-off-by: Canlin <canlinguosdu@gmail.com>
Co-authored-by: Codex <noreply@openai.com>
 2026-09-12 14:22:30 +08:00
Giancarlo Delfin 9dcf6bf344 [BugFix] Fix DP token padding in dflash attention metadata (#56181) 
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
 2026-09-11 14:33:53 -07:00
Andreas KaratzasandOpenAI Codex 9dd969da09 [Model][ROCm] Enable DeepSeek V4 Vision (#55107) 
Signed-off-by: Andreas Karatzas <Andreas.Karatzas@amd.com>
Signed-off-by: Andreas Karatzas <akaratza@amd.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
 2026-09-11 09:17:30 -07:00
Andreas KaratzasOpenAI CodexLucas Wilkinson
 		d0dfe587d5 [Bugfix] Fall back to full decode graphs for noncompiled models (#55095) 
Signed-off-by: Andreas Karatzas <Andreas.Karatzas@amd.com>
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
Signed-off-by: Andreas Karatzas <akaratza@amd.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com>
 2026-09-11 01:56:05 -07:00
Lucas WilkinsonSummer YangOpenAI CodexOpenAI Codex
 		980c16c8e4 [PCP][Spec Decode] Adds PCP support for single-module MTP and replicated DSpark. (#56107) 
Signed-off-by: Summer Yang <girasoleyang@gmail.com>
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
Co-authored-by: Summer Yang <girasoleyang@gmail.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
Co-authored-by: OpenAI Codex <noreply@openai.com>
 2026-09-10 22:07:53 -07:00
Sy03Raphael RiallandZiyang ZhangCodexmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 		83990f5bcc [Bugfix][MRV2] Initialize DCP metadata after batch partitioning (#55212) 
Signed-off-by: Raphael Rialland <raphael.rialland@mistral.ai>
Signed-off-by: Ziyang Zhang <hafuhafu@qq.com>
Signed-off-by: Sy03 <1370724210@qq.com>
Co-authored-by: Raphael Rialland <raphael.rialland@mistral.ai>
Co-authored-by: Ziyang Zhang <hafuhafu@qq.com>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-09-10 06:50:09 +00:00
Andrii SkliarAndrii Skliarmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 		83fe99399e [Spec Decode] Add NVFP4 DSpark gathered top-k projection (#55713) 
Signed-off-by: Andrii Skliar <askliar@nvidia.com>
Co-authored-by: Andrii Skliar <askliar@nvidia.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-09-09 17:33:46 +00:00
Stefano CastagnettaNick Hillmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 		9c2d21046b [Bugfix][Spec Decode] Honour the draft's moe_backend on Model Runner V2 (#54788) 
Signed-off-by: Stefano Castagnetta <scastagnetta@nvidia.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-09-08 16:50:07 -07:00
Flora FengandWentao Ye 29f46ac475 [Perf] Eliminate full-history reasoning scans for structured outputs (#55223) 
Signed-off-by: sfeng33 <4florafeng@gmail.com>
Signed-off-by: Flora Feng <4florafeng@gmail.com>
Co-authored-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com>
 2026-09-08 14:07:16 -07:00
Stefano Castagnetta 2524051385 [Bugfix][Spec Decode] Honour the draft's attention_backend on Model Runner V2 (#54826) 
Signed-off-by: Stefano Castagnetta <scastagnetta@nvidia.com>
 2026-09-04 12:23:19 -07:00
Giancarlo Delfin fe755c8899 [Bugfix][Model Runner V2][Spec Decode] Decouple the draft's gumbel noise stream from the target's (#54282) 
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
 2026-08-29 09:58:29 -07:00
Rahul ChalamalaJanelle CaiDevin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>Devin
 		d3d79ffc1e [Bugfix][Spec Decode] Capture the widest uniform decode batch by default (#50488) 
Signed-off-by: Rahul Chalamala <22563365+rchalamala@users.noreply.github.com>
Signed-off-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Signed-off-by: Rahul Chalamala <rahul@modal.com>
Co-authored-by: Janelle Cai <janelle.cai@modal.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Devin <devin@cognition.ai>
 2026-08-28 22:38:13 -07:00
Summer Yang 6d4562c59b [Attention][DCP] Enable FlashInfer MLA for DSpark drafting (#54277) 
Signed-off-by: Summer Yang <girasoleyang@gmail.com>
 2026-08-28 21:11:28 -07:00
Qiming Zhang c01b50e390 [Test] Fix mock_current_vllm_config missing kernel_config in test_dflash2 (#54132) 
Signed-off-by: mayuyuace <qiming1.zhang@intel.com>
 2026-08-27 22:28:43 -07:00
Misha Goin 7de96050c5 [Spec Decode] Enable adaptive DSpark on SM100 sparse MLA (#52783) 
Signed-off-by: mgoin <mgoin64@gmail.com>
 2026-08-25 13:31:15 -04:00
Karan Shelar a9a17e7095 Dflash2 load fix (#53435) 2026-08-24 21:32:43 -07:00
Zihan Zhang b389ac2946 [Spec Decode] DFlash2: local convolution + candidate selector (#52816) 2026-08-20 22:27:22 -07:00
Chauncey c6e19b3be2 [Bugfix][Structured Output] Avoid spurious FSM errors after speculative reasoning end (#53046) 
Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>
 2026-08-21 10:01:06 +08:00
Matthew KotilaReed MeyersonClaude Opus 4.8
 		7cfb97e337 [Frontend][Core][Spec Decode] Per-request acceptance stats in OpenAI API responses (#48915) 
Signed-off-by: Matthew Kotila <7692737+matthewkotila@users.noreply.github.com>
Co-authored-by: Reed Meyerson <31574681+reed-meyerson@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-08-20 15:42:49 -07:00
Flora Feng 12f64b39d2 [Bugfix][Structured Output] Stop XGrammar token batches at termination (#52805) 
Signed-off-by: sfeng33 <4florafeng@gmail.com>
 2026-08-18 21:21:43 +00:00
Wei Zhao d1e3eee6fb [Spec decode] Support Kimi-K3 DCP with DSpark (#52188) 
Signed-off-by: wzhao18 <wzhao18.sz@gmail.com>
 2026-08-17 13:08:45 -07:00
oops-oomoops-oomClaude Opus 4.8
 		4d2a68d64d [Bugfix][Spec Decode][Structured Output] DSpark: fix the grammar bitmask mapping when the draft budget is zero (#52436) 
Signed-off-by: oops-oom <73481342@qq.com>
Co-authored-by: oops-oom <73481342@qq.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
 2026-08-16 11:09:23 +00:00
Gabriel Wu 97388c44f9 [Bugfix] Make DSV4 sparse MLA work end-to-end for plain decode, MTP, and DSpark (#51538) 2026-08-15 07:01:32 -07:00
Lucas WilkinsonOpenAI CodexClaude Opus 5Benjamin Chislettmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>Nick HillOpenAI Codex
 		7f7a32cfec [Spec Decode] DSpark confidence-scheduled verification (#47808) 
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
Signed-off-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com>
Signed-off-by: Benjamin Chislett <chislett.ben@gmail.com>
Signed-off-by: Lucas Wilkinson <wilkinson.lucas@gmail.com>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Benjamin Chislett <chislett.ben@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: OpenAI Codex <noreply@openai.com>
 2026-08-12 12:27:48 -07:00
Lucas WilkinsonandOpenAI Codex 02ac17851d [Refactor][MRV2] Unify uniform decode token count helper (#51917) 
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
 2026-08-12 05:58:43 +00:00
Nick HillRahul ChalamalaJanelle Cai
 		0a94d85a66 [Bugfix][MRV2] Require all requests to be decoding for uniform-decode dispatch (#51865) 
Signed-off-by: Rahul Chalamala <22563365+rchalamala@users.noreply.github.com>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Rahul Chalamala <22563365+rchalamala@users.noreply.github.com>
Co-authored-by: Janelle Cai <janelle.cai@modal.com>
 2026-08-11 17:15:06 -07:00
Giancarlo DelfinandNick Hill 27930df9c2 [Model Runner V2] Fix -1 placeholder draft token ids in rejection sam… (#50939) 
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
 2026-08-06 16:32:00 -07:00
Giancarlo Delfin 81bc196913 [Model Runner V2] Cache draft logits in model's LM head dtype (#50910) 
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
 2026-08-05 19:12:46 -07:00
gabriel-peracioandKimi K3 47a4e410ba [Bugfix][Spec Decode] Fix NaN handling in rejection sampler tl.argmax (#50183) 
Signed-off-by: Gabriel Peracio <gabriel@sigsonic.com>
Co-authored-by: Kimi K3 <noreply@moonshot.ai>
 2026-08-05 18:01:09 -07:00
Andrii SkliarandAndrii Skliar 5789897aa4 [Spec Decode] Add top-k DSpark Markov projection (#49969) 
Signed-off-by: Andrii Skliar <askliar@nvidia.com>
Co-authored-by: Andrii Skliar <askliar@nvidia.com>
 2026-08-04 12:52:53 +00:00
+13
Jiangyun ZhuIsotr0pyThien TranBugen ZhaoYongye ZhuZiming HuangRoger WangIsotr0pyaoshen02Woosuk KwonJee Jee Liaoshen02Summer YangKevin H. LuuBowen WanggnovackNick HillYifan QiaoxiaozhoupyRoy WangJeff MaCodexWei Zhao
 		aeeb36b1f1 [New model] Kimi K3 (#50000) 
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com>
Signed-off-by: Isotr0py <Isotr0py@outlook.com>
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
Signed-off-by: Jee Jee Li <jeejeelee@inferact.ai>
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai>
Signed-off-by: Yongye Zhu <zyy1102000@gmail.com>
Signed-off-by: wzhao18 <wzhao18.sz@gmail.com>
Co-authored-by: Isotr0py <Isotr0py@outlook.com>
Co-authored-by: Thien Tran <gau.nernst@yahoo.com.sg>
Co-authored-by: Bugen Zhao <i@bugenzhao.com>
Co-authored-by: Yongye Zhu <zyy1102000@gmail.com>
Co-authored-by: Ziming Huang <zelda.huanghuang@gmail.com>
Co-authored-by: Roger Wang <hey@rogerw.io>
Co-authored-by: Isotr0py <mozf@inferact.ai>
Co-authored-by: aoshen02 <aoshen02@users.noreply.github.com>
Co-authored-by: Woosuk Kwon <woosuk@inferact.ai>
Co-authored-by: Jee Jee Li <jeejeelee@inferact.ai>
Co-authored-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: Summer Yang <girasoleyang@gmail.com>
Co-authored-by: Kevin H. Luu <khluu000@gmail.com>
Co-authored-by: Bowen Wang <abmfy@icloud.com>
Co-authored-by: gnovack <novackgm@gmail.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai>
Co-authored-by: xiaozhoupy <peiyuanzhou1994@gmail.com>
Co-authored-by: Roy Wang <yasong.wang@inferact.ai>
Co-authored-by: Jeff (Junze) Ma <93145857+majunze2001@users.noreply.github.com>
Co-authored-by: Codex <codex@openai.com>
Co-authored-by: Wei Zhao <51183510+wzhao18@users.noreply.github.com>
 2026-07-30 03:49:58 -07:00
Jared Wenmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>Cyrus Leung
 		32e657e689 [BugFix] eagle draft max position embeddings (#49343) 
Signed-off-by: JaredforReal <w13431838023@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>
 2026-07-29 01:24:47 -07:00
Andreas Karatzas 5f89a03dcb [CI] Explicitly tear down speculative decode runners (#49910) 
Signed-off-by: Andreas Karatzas <akaratza@amd.com>
 2026-07-27 05:32:46 +00:00
Michael GoinandNick Hill ac36a7a1e7 [MRV2][Spec Decode] Avoid rejection sampler OOM by chunking (#48630) 
Signed-off-by: mgoin <mgoin64@gmail.com>
Signed-off-by: Michael Goin <mgoin64@gmail.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
 2026-07-23 18:13:13 +08:00
Michael GoinandCodex a7d00ec051 [Bugfix] DFlash fc sized wrong when num_target_layers != num_hidden_layers (#48524) 
Signed-off-by: mgoin <mgoin64@gmail.com>
Co-authored-by: Codex <noreply@openai.com>
 2026-07-21 14:42:52 -07:00
Oxana KorzhCursorokorzh-amdMatthew Bonanni
 		4ec199b66a [Bugfix][Spec-Decode] Populate draft seq_lens_cpu_upper_bound for spec-decode attention metadata (#44492) 
Signed-off-by: Oxana Korzh <okorzh@amd.com>
Signed-off-by: okorzh-amd <okorzh-amd@users.noreply.github.com>
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: okorzh-amd <okorzh-amd@users.noreply.github.com>
Co-authored-by: Matthew Bonanni <mbonanni@redhat.com>
 2026-07-20 20:50:58 +00:00
Michael GoinandClaude Fable 5 ecf4aa5ce2 [Bugfix] Fix FlashInfer non-causal draft attention (DFlash/DSpark) on Blackwell (#48167) 
Signed-off-by: mgoin <mgoin64@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
 2026-07-15 12:44:01 -07:00
Tyler Michael Smithmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>Claude Opus 4.6
 		93e2ab7111 Disable dynamic speculative decoding when DP is enabled (#45963) 
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-07-07 12:52:41 +00:00
yue.yuandBenjamin Chislett e7c9df9449 [Bugfix][Structured Output][Spec Decode] Constrain bitmask and trim grammar advance at the reasoning boundary (#44297) 
Signed-off-by: Allen.Yu <yuyue0225sc@163.com>
Signed-off-by: yue.yu <yuyue0225sc@163.com>
Co-authored-by: Benjamin Chislett <chislett.ben@gmail.com>
 2026-07-04 09:08:45 +00:00
Ekagra RanjanBenjamin ChislettNick Hill
 		07516fda67 [MRV2][SD] Make Dynamic SD comatible with Full Cuda Graphs (#45953) 
Signed-off-by: Ekagra Ranjan <3116519+ekagra-ranjan@users.noreply.github.com>
Co-authored-by: Benjamin Chislett <chislett.ben@gmail.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
 2026-07-03 23:58:27 -04:00