iHore/vLLM
8
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity
23,341 Commits 3 Branches 0 Tags
patches
Commit Graph
56 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Giancarlo DelfinandWoosuk Kwon 791f92135a [Attention] Set ALWAYS cudagraph support for FA4 (#60908) 
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
Co-authored-by: Woosuk Kwon <woosuk@inferact.ai>
 2026-10-13 16:22:52 -07:00
Giancarlo DelfinandRishi Puri 36f3972812 [Spec Decode] Skip metadata build for dflash/dspark when all model backends support it (#60692) 
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
Co-authored-by: Rishi Puri <20074092+puririshi98@users.noreply.github.com>
 2026-10-12 00:09:10 -07:00
Giancarlo Delfin ffdde24f54 [Docs] Document adaptive verification without a confidence head (#60913) 
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
 2026-10-09 13:54:40 -07:00
Giancarlo Delfin 6ee9be6b3c [Spec Decode] Rename AR speculators to StandaloneAR / TargetDependentAR (#60335) 
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
 2026-10-07 14:01:59 -04:00
Giancarlo Delfin fcf53d240c [Spec Decode] Remove eager metadata rebuild during MTP fused multi-step decode (#58463) 
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
 2026-10-06 12:17:30 -04:00
Giancarlo Delfin 88f3c601b3 [Docs] Add reviewer area of interest for TheEpicDolphin (#60092) 
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
 2026-10-05 18:35:19 +00:00
Giancarlo DelfinandWoosuk Kwon 3d5f4d4cd5 [Perf][Spec Decode] Avoid triton recompiles in the acceptance estimator (#57107) 
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
Co-authored-by: Woosuk Kwon <woosuk@inferact.ai>
 2026-09-28 13:46:25 -07:00
Giancarlo Delfin 85c1f58d50 [BugFix][Model Runner V2][Spec Decode] Fix decode instance's multi-layer MTP kv caches during P/D (#55055) 
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
 2026-09-15 17:31:25 -07:00
Giancarlo Delfin 3bb0a03f35 [Model Runner V2] Acceptance estimation for adaptive verification (#52228) 
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
 2026-09-14 15:58:48 -07:00
Giancarlo Delfin 9dcf6bf344 [BugFix] Fix DP token padding in dflash attention metadata (#56181) 
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
 2026-09-11 14:33:53 -07:00
Giancarlo Delfin facd9a74a1 [Model Runner V2][Spec Decode] Skip DP sync for all speculator uniform decodes (#54856) 2026-09-03 10:32:33 +00:00
Giancarlo Delfin fe755c8899 [Bugfix][Model Runner V2][Spec Decode] Decouple the draft's gumbel noise stream from the target's (#54282) 
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
 2026-08-29 09:58:29 -07:00
Giancarlo Delfin 5acc1c4e4b [Model Runner V2][Spec Decode] Skip DP sync before EAGLE/MTP draft prefill (#53694) 2026-08-26 20:59:35 -07:00
Giancarlo Delfin d154d90d6c [Model Runner V2] batch-sharded sample (#50465) 
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
 2026-08-24 15:49:57 -07:00
Giancarlo Delfin d4f4d3f40f [Model Runner V2][Spec Decode] Fix draft logits cache column stride in gumbel_sample (#53017) 2026-08-19 17:39:30 -07:00
Giancarlo DelfinandClaude Fable 5 f80b66f548 [Model Runner V2][Spec Decode] Add KV cache support for multi-layer MTP (#50062) 
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
 2026-08-13 16:22:29 -07:00
Giancarlo DelfinandWoosuk Kwon 3e174bb73c [Model Runner V2][MTP] Share topk index buffer between draft steps (#47352) 
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
Co-authored-by: Woosuk Kwon <woosuk@inferact.ai>
 2026-08-10 15:46:13 -07:00
Giancarlo DelfinandNick Hill 27930df9c2 [Model Runner V2] Fix -1 placeholder draft token ids in rejection sam… (#50939) 
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
 2026-08-06 16:32:00 -07:00
Giancarlo Delfin 81bc196913 [Model Runner V2] Cache draft logits in model's LM head dtype (#50910) 
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
 2026-08-05 19:12:46 -07:00
Giancarlo Delfin dec13a33b7 [Model Runner V2][Spec Decode] Add multi-layer MTP speculator (#48892) 
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
 2026-07-30 15:54:15 -07:00
Giancarlo Delfin 272abd5f48 [Tests][Spec Decode] Add gemma4 MTP acceptance rates test (#47920) 
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
 2026-07-27 16:15:54 -07:00
Giancarlo Delfin d63c8e9444 [BugFix][Spec Decode] Compact shared topk indices buffer after first MTP draft step (#47238) 2026-07-01 21:38:51 -07:00
Giancarlo Delfin 3cecee40f3 [Model Runner V2][Spec Decode] Fix stale values in idx_mapping from CG num reqs padding (#47066) 2026-06-30 11:25:32 -07:00
Giancarlo Delfin db808b3961 [Model Runner V2][Spec Decode] Implement block verification for rejection sampling (#46781) 
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
 2026-06-30 08:07:24 -07:00
Giancarlo Delfin b6caeb5a09 [Model Runner V2][Spec Decode] Use fp32 uniform threshold for acceptance (#46878) 2026-06-27 14:09:25 -07:00
Giancarlo Delfinandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> 02a1f23711 [DFlash] Fuse precompute kv per-layer rmsnorms (#46761) 
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-06-25 19:32:07 -07:00
Giancarlo Delfin 5314665bad [Model Runner V2][DFlash] Enable dflash attention backend selection (#46770) 
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
 2026-06-25 19:29:25 -07:00
Giancarlo Delfin c53994e134 [Model Runner V2][Spec Decode] Use log1p to compute residual during rejection sampling (#46665) 
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
 2026-06-25 23:46:10 +00:00
Giancarlo Delfin ef361de916 [Model Runer V2][DFlash] Fix lm head sharing for dflash (#46435) 
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
 2026-06-23 19:09:06 +00:00
Giancarlo Delfinandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> b5adb027ad [Models] Fix MiMo v2.x QKV TP sharding + FP4 support (#45200) 
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-06-15 15:13:34 +08:00
Giancarlo Delfin ceb0111a90 [Model Runner V2][Spec Decode] Add Gemma4 MTP support (#43241) 
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
 2026-06-04 00:51:06 +00:00
Giancarlo DelfinandWoosuk Kwon 91945b6e4a [Bug Fix][Model Runner V2][Spec Decode] Warmup & capture with different attention states for speculator prefill (#44253) 
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
Co-authored-by: Woosuk Kwon <woosuk@inferact.ai>
 2026-06-03 13:32:40 -07:00
Giancarlo Delfin 3b6a204789 [Model Runner V2][Bug Fix][DSV4] Ensure lazy attention state initializations happen during cudagraph capture (#42444) 
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
 2026-05-14 16:16:17 -07:00
Giancarlo Delfin fe5b4e0fe7 [Model Runner V2] Apply synthetic mode to probabilistic rejection sampler (#41035) 2026-05-12 13:37:03 -07:00
Giancarlo Delfin e1e4646b06 [Model Runner V2] Rebuild attn metadata between draft decode steps (#41162) 
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
 2026-05-05 00:44:55 +00:00
Giancarlo Delfin 4c7c69b4e0 [Model Runner V2] Skip attention metadata rebuild before draft prefill (#40410) 
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
 2026-04-27 15:38:05 -07:00
Giancarlo Delfin f5f9878514 [Model Runner V2] Fix rejection sampling acceptance rate gap vs MRV1 (#40651) 
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
 2026-04-26 19:12:08 -07:00
Giancarlo Delfin bf9a5ddb24 [MLA] Optimize mla indexer prepare uniform decode for MTP > 1 (#39458) 
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
 2026-04-16 16:27:51 -07:00
Giancarlo Delfin 343f65234b [Model Runner V2][BugFix] fix num_sampled dtype for probabilistic rej… (#39951) 
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
 2026-04-15 18:09:11 -07:00
Giancarlo Delfin 3bfe55a037 [Model Runner V2] Disable piecewise cudagraph mode fallback for eagle draft decodes (#39773) 
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
 2026-04-14 17:47:57 -07:00
Giancarlo Delfin ccf90ba784 [Model Runner V2] Add full cuda graph support for eagle prefill (#37588) 
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
 2026-04-13 16:01:24 -07:00
Giancarlo Delfin 5daf62271d [Model Runner V2] Fuse probabilistic rejection sample kernels (#38496) 
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
 2026-04-07 17:37:37 -07:00
Giancarlo Delfin 384e4d5f48 [Model Runner V2] Rebuild attention metadata before eagle decode full… (#38311) 
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
 2026-03-27 13:46:42 -07:00
Giancarlo Delfin c32e97602d [Model Runner V2] Enable forcing a specific acceptance rate during rejection sampling (#38045) 
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
 2026-03-26 13:38:12 -07:00
Giancarlo Delfin 8f4824b664 [Model Runner V2] Gather multimodal embeddings before draft model postprocess (#37932) 
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
 2026-03-23 18:14:13 -07:00
Giancarlo DelfinandWoosuk Kwon b3e846017d [Model Runner V2] Support multi-modal embeddings for spec decode model (#36097) 
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
Co-authored-by: Woosuk Kwon <woosuk@inferact.ai>
 2026-03-22 02:48:43 -07:00
Giancarlo Delfin dcee9be95a [Model Runner V2] Fix draft logits not populated during cudagraph replay (#37639) 
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
 2026-03-20 07:43:47 +00:00
Giancarlo Delfin 39474513f6 [Model Runner V2] fix draft attention metadata generation (#37364) 
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
 2026-03-19 21:05:15 -07:00
Giancarlo Delfin 053f3b6309 [Model Runner V2] Spec decode rejection sampler logprobs support (#37237) 
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
 2026-03-19 01:36:27 +00:00
Giancarlo Delfin 04244fd0e1 [Model Runner V2] Spec decode rejection sampler greedy support (#37238) 
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
 2026-03-18 15:59:03 -07:00
Giancarlo Delfin c77181e534 [Model Runner V2] Add probabilistic rejection sampling for spec decoding (#35461) 
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
 2026-03-11 14:04:32 -07:00
Giancarlo DelfinandNick Hill 6644796bf4 [V1][spec decode] return logprobs for spec decoding (#26060) 
Signed-off-by: Giancarlo Delfin <gdelfin@meta.com>
Co-authored-by: Nick Hill <nhill@redhat.com>
 2025-10-22 22:59:59 -07:00
Giancarlo Delfin d94e3026de [V1] Add tree drafting tests for eagle spec decoding (#22705) 
Signed-off-by: Giancarlo Delfin <gdelfin@meta.com>
 2025-08-13 04:11:28 -07:00
Giancarlo Delfin 469b3ffaaa [V1] port xformers backend to v1 (#21342) 
Signed-off-by: Giancarlo Delfin <gdelfin@meta.com>
 2025-08-05 10:04:46 -07:00
Giancarlo Delfin 5ea71ff46f [V1] reduce block size for tree attention correctness test to fix 'ou… (#22207) 
Signed-off-by: Giancarlo Delfin <gdelfin@meta.com>
 2025-08-04 19:11:06 -07:00
Giancarlo Delfin aa7012eb6d Add tree attention backend for v1 (part 1) (#20401) 
Signed-off-by: Giancarlo Delfin <gdelfin@meta.com>
 2025-08-03 22:13:26 -07:00