27 Commits
Author SHA1 Message Date
DimensionSTPandKevin H. Luu 9edcc7fa7e [Bugfix][Spec Decode] Avoid reusing random draws across block verification rounds (#61161)
Signed-off-by: DimensionSTP <65501090+DimensionSTP@users.noreply.github.com>
Co-authored-by: Kevin H. Luu <khluu000@gmail.com>
2026-10-15 14:46:38 -07:00
0f8b398158 [watermarking] add context deduplication support to speculative decoding (#56807)
Signed-off-by: Raphael Rialland <raphael.rialland@mistral.ai>
Signed-off-by: Simon Veitner <sveitner@redhat.com>
Co-authored-by: Simon Veitner <sveitner@redhat.com>
Co-authored-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com>
2026-09-30 12:22:10 -04:00
Wentao Ye fae5cbfc8b [Refactor] Remove dead code multiple places (#58002)
Signed-off-by: yewentao256 <zhyanwentao@126.com>
2026-09-22 16:36:03 -04:00
Jiang WuandJiang Wu 6af44e2887 [Bugfix][CPU] Fix DeepSeek V4.1 import without Triton (#57654)
Signed-off-by: Jiang Wu <245355730+JiangLLM@users.noreply.github.com>
Co-authored-by: Jiang Wu <245355730+JiangLLM@users.noreply.github.com>
2026-09-19 05:46:13 +00:00
freyfwtCodexmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>Jiangyun Zhu
18553c5c9f [Kernel] Use Murmur3 RNG for Gumbel sampling (#51367)
Signed-off-by: freyfwt <110148731+freyfwt@users.noreply.github.com>
Signed-off-by: freyfwt <freytian1996@gmail.com>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Jiangyun Zhu <riverclouds.zhu@qq.com>
2026-09-15 23:52:26 -07:00
Giancarlo Delfin fe755c8899 [Bugfix][Model Runner V2][Spec Decode] Decouple the draft's gumbel noise stream from the target's (#54282)
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
2026-08-29 09:58:29 -07:00
Zihan Zhang b389ac2946 [Spec Decode] DFlash2: local convolution + candidate selector (#52816) 2026-08-20 22:27:22 -07:00
Giancarlo Delfin d4f4d3f40f [Model Runner V2][Spec Decode] Fix draft logits cache column stride in gumbel_sample (#53017) 2026-08-19 17:39:30 -07:00
Giancarlo Delfin 81bc196913 [Model Runner V2] Cache draft logits in model's LM head dtype (#50910)
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
2026-08-05 19:12:46 -07:00
f5a8d73377 [Spec Decode] DSpark (#46995)
Signed-off-by: Benjamin Chislett <bchislett@nvidia.com>
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
Signed-off-by: mgoin <mgoin64@gmail.com>
Co-authored-by: Giancarlo Delfin <gdelfin@inferact.ai>
Co-authored-by: mgoin <mgoin64@gmail.com>
2026-07-01 08:30:24 -07:00
Giancarlo Delfin 3cecee40f3 [Model Runner V2][Spec Decode] Fix stale values in idx_mapping from CG num reqs padding (#47066) 2026-06-30 11:25:32 -07:00
Giancarlo Delfin b6caeb5a09 [Model Runner V2][Spec Decode] Use fp32 uniform threshold for acceptance (#46878) 2026-06-27 14:09:25 -07:00
3c43237233 [Bugfix][Model Runner V2][Spec Decode] Fix int32 offset overflow in sampler kernels (#46560)
Signed-off-by: xiaojun.wei <jessiewei747@gmail.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com>
2026-06-24 11:00:57 -07:00
Woosuk Kwon 16908e132e [MRV2] Make FP32 Gumbel sampling more accurate (#45996)
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
2026-06-18 19:42:09 +00:00
Benjamin ChislettandGiancarlo Delfin 0bae1d3848 [MRV2][Spec Decode] DFlash (#44586)
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
Signed-off-by: Benjamin Chislett <bchislett@nvidia.com>
Signed-off-by: Benjamin Chislett <chislett.ben@gmail.com>
Co-authored-by: Giancarlo Delfin <gdelfin@inferact.ai>
2026-06-10 08:47:46 -07:00
PatchyTISandPatchouliTaisa 0162596603 [Model Runner V2] FP32 gumbel sampling. (#41775)
Signed-off-by: PatchouliTaisa <patchychen@tencent.com>
Co-authored-by: PatchouliTaisa <patchychen@tencent.com>
2026-05-15 09:20:08 -07:00
Giancarlo Delfin e1e4646b06 [Model Runner V2] Rebuild attn metadata between draft decode steps (#41162)
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
2026-05-05 00:44:55 +00:00
Giancarlo Delfin 5daf62271d [Model Runner V2] Fuse probabilistic rejection sample kernels (#38496)
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
2026-04-07 17:37:37 -07:00
Woosuk Kwon a5e9d511de [MRV2] Use FP64 for Gumbel noise (#37798)
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
2026-03-22 12:28:10 -07:00
Nick Hill cd32d6f586 [Model Runner V2] Some code simplification (#36929)
Signed-off-by: Nick Hill <nickhill123@gmail.com>
2026-03-13 00:59:23 +00:00
Giancarlo Delfin c77181e534 [Model Runner V2] Add probabilistic rejection sampling for spec decoding (#35461)
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
2026-03-11 14:04:32 -07:00
Andy Lo 0a7165fd71 [ModelRunnerV2] Rename sampler functions and variables for clarity (#35459)
Signed-off-by: Andy Lo <andy@mistral.ai>
2026-03-02 19:48:56 -08:00
Woosuk Kwon 5fcb0cdd68 [Model Runner V2] Use FP32 for Gumbel Noise (#34854)
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
2026-02-18 17:07:37 -08:00
Woosuk Kwon 6c01ffb897 [Model Runner V2] Decouple temperature from penalties (#32629)
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
2026-01-19 19:13:24 -08:00
Woosuk Kwon 025a32f9ed [Model Runner V2] Remove async barrier (#32083)
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
2026-01-11 20:24:30 -08:00
Woosuk Kwon f223ed4181 [Model Runner V2] Fuse penalties and temperature into single kernel (#29720)
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
2025-11-29 02:29:16 -08:00
Woosuk Kwon 6afc0ffaf6 [Model Runner V2] Add sample/ directory and reorganize files (#29719)
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
2025-11-29 00:41:01 -08:00