iHore/vLLM
8
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity
8,162 Commits 3 Branches 0 Tags
3e36fcbee642f41278a4881c9e2bfbbd7c28e607
Commit Graph
63 Commits
Author SHA1 Message Date
Michael Goin 34a20c49b3 [Logs] Change flashinfer sampler logs to once (#21759) 
Signed-off-by: mgoin <mgoin64@gmail.com>
 2025-07-28 06:59:51 -07:00
Mengqing Cao f3a683b7c9 [Bugfix][Logprobs] Fix logprobs op to support more backend (#21591) 
Signed-off-by: MengqingCao <cmq0113@163.com>
 2025-07-25 05:53:07 -07:00
Lu Fang accac82928 [Sampler] Introduce logprobs mode for logging (#21398) 
Signed-off-by: Lu Fang <lufang@fb.com>
 2025-07-23 01:39:25 -07:00
Jialin Ouyang a32237665d [Core] Optimize update checks in LogitsProcessor (#21245) 
Signed-off-by: Jialin Ouyang <Jialin.Ouyang@gmail.com>
 2025-07-22 05:27:18 -07:00
Lu Fang 8d0a01a5f2 [v1][sampler] Inplace logprobs comparison to get the token rank (#21283) 
Signed-off-by: Lu Fang <lufang@fb.com>
 2025-07-21 13:47:47 -07:00
Maximilien de Bayser 5de8d9f111 Remove extra tensor on CPU (#20693) 
Signed-off-by: Max de Bayser <mbayser@br.ibm.com>
 2025-07-12 14:06:34 +08:00
Isotr0py 32c9be2200 [v1] Re-add fp32 support to v1 engine through FlexAttention (#19754) 
Signed-off-by: Isotr0py <2037008807@qq.com>
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>
 2025-07-05 09:41:10 +00:00
Jee Jee Li 1caca5a589 [Misc] Add SPDX-FileCopyrightText (#20428) 
Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>
 2025-07-04 07:40:42 +00:00
afeldman-nmandNick Hill 48fb076cbc [V1] LogitsProcessor programming model (#16728) 
Signed-off-by: Nick Hill <nhill@redhat.com>
Signed-off-by: Andrew Feldman <afeldman@neuralmagic.com>
Signed-off-by: Andrew Feldman <afeldman@redhat.com>
Co-authored-by: Nick Hill <nhill@redhat.com>
 2025-07-02 09:10:42 -07:00
afeldman-nm 19a53b2783 [V1] Decouple GPU and TPU InputBatch (#19778) 
Signed-off-by: Andrew Feldman <afeldman@redhat.com>
 2025-06-18 06:38:13 +00:00
Yu Guo 8267f9916f improve logits bias (#19041) 2025-06-06 19:59:25 +08:00
Simon Mo 02f0c7b220 [Misc] Add SPDX-FileCopyrightText (#19100) 
Signed-off-by: simon-mo <simon.mo@hey.com>
 2025-06-03 11:20:17 -07:00
Lukas Geiger e7523c2e03 [V1][Sampler] Improve performance of FlashInfer sampling by sampling logits instead of probs (#18608) 2025-05-26 11:49:36 -04:00
Bowen Wangandmgoin 7fdfa01530 [Sampler] Adapt to FlashInfer 0.2.3 sampler API (#15777) 
Signed-off-by: Bowen Wang <abmfy@icloud.com>
Co-authored-by: mgoin <mgoin64@gmail.com>
 2025-05-16 15:14:03 -07:00
Mengqing Cao f9bc5a0693 [Bugfix] Fix triton import with local TritonPlaceholder (#17446) 
Signed-off-by: Mengqing Cao <cmq0113@163.com>
 2025-05-06 17:53:09 +08:00
Nicolò Lucchesi 5941e0b7ea [TPU][V1] Add support for top-logprobs (#17072) 
Signed-off-by: NickLucche <nlucches@redhat.com>
 2025-05-05 14:20:15 -07:00
Harry Mellor d6484ef3c3 Add full API docs and improve the UX of navigating them (#17485) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2025-05-03 19:42:43 -07:00
Woosuk Kwon 41fb013d29 [V1][Spec Decode] Always use argmax for sampling draft tokens (#16899) 
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
 2025-04-23 14:57:43 -07:00
Nicolò Lucchesiandmgoin fa3bba2a53 [TPU][V1] Enable Top-P (#16843) 
Signed-off-by: NickLucche <nlucches@redhat.com>
Co-authored-by: mgoin <mgoin64@gmail.com>
 2025-04-22 00:46:07 +00:00
Nicolò LucchesiandHyesoo Yang eb5819b2d9 [V1][TPU] Enable Top K (#15489) 
Signed-off-by: NickLucche <nlucches@redhat.com>
Signed-off-by: Hyesoo Yang <hyeygit@gmail.com>
Co-authored-by: Hyesoo Yang <hyeygit@gmail.com>
 2025-04-17 18:18:11 +00:00
Ryan McConville 6c11ecf8d3 [Bugfix] Validate logit biases to prevent out of vocab ids crashing engine (#16529) 
Signed-off-by: Ryan McConville <ryan@ryanmcconville.com>
 2025-04-12 20:19:19 +00:00
Nicolò Lucchesi 3cc9af88ff [TPU][V1] Disable per-request seed/Generator (#16172) 
Signed-off-by: NickLucche <nlucches@redhat.com>
 2025-04-10 17:05:44 -04:00
Chengji Yao a454748544 [TPU][V1] Refine tpu_model_runner to mitigate future recompilation issues (#16275) 
Signed-off-by: Chengji Yao <chengjiyao@google.com>
 2025-04-09 18:51:51 -06:00
Hyesoo Yangandroot 1b84eff03a [V1][TPU] TPU-optimized top-p implementation (avoids scattering). (#15736) 
Signed-off-by: Hyesoo Yang <hyeygit@gmail.com>
Co-authored-by: root <root@t1v-n-822696b7-w-0.us-central2-b.c.tpu-prod-env-large-adhoc.internal>
 2025-04-02 17:18:08 -07:00
Brayden Zhong 6efb195a6e [V1] Fix: make sure k_index is int64 for apply_top_k_only (#15907) 
Signed-off-by: Brayden Zhong <b8zhong@uwaterloo.ca>
 2025-04-01 19:06:44 -07:00
Woosuk Kwon 2bc4be4e32 [V1][Minor] Simplify rejection sampler's parse_output (#15741) 
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
 2025-03-29 09:25:17 -07:00
Nicolò Lucchesi 4098b72210 [Bugfix][TPU][V1] Fix recompilation (#15553) 
Signed-off-by: NickLucche <nlucches@redhat.com>
 2025-03-27 19:15:06 +00:00
Cody YuandWoosuk Kwon 54aa619459 [V1] Refactor num_computed_tokens logic (#15307) 
Signed-off-by: Cody Yu <hao.yu.cody@gmail.com>
Co-authored-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
 2025-03-27 04:54:36 +00:00
Nick Hill 35fad35a48 [V1][Sampler] Faster top-k only implementation (#15478) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-03-26 10:56:47 -07:00
Nicolò Lucchesi a0dd7dcd49 [TPU][V1] Fix Sampler recompilation (#15309) 
Signed-off-by: NickLucche <nlucches@redhat.com>
 2025-03-25 16:43:54 -04:00
Woosuk Kwon 25f560a62c [V1][Spec Decode] Update target_logits in place for rejection sampling (#15427) 
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
 2025-03-24 21:04:41 -07:00
Woosuk Kwon 911c8eb000 [Minor][Spec Decode] Remove compiled_softmax (#15416) 
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
 2025-03-24 19:09:04 -07:00
Woosuk Kwon ebcebeeb6b [V1][Spec Decode] Enable spec decode for top-p & top-k sampling (#15063) 
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
 2025-03-24 17:16:46 -07:00
Chen1022andchenjincong 0893567db9 [V1][Minor] fix comments (#15392) 
Signed-off-by: chenjincong <chenjincong@baidu.com>
Signed-off-by: Chen-0210 <chenjincong11@gmail.com>
Co-authored-by: chenjincong <chenjincong@baidu.com>
 2025-03-24 08:45:32 -07:00
Hyesoo Yang 47195057e9 [V1][TPU] Speed up top-k on TPU by using torch.topk (#15242) 
Signed-off-by: Hyesoo Yang <hyeygit@gmail.com>
 2025-03-20 19:19:40 -07:00
Nicolò Lucchesi d8c6d7d6b5 [V1][TPU] Support V1 Sampler for ragged attention (#14227) 
Signed-off-by: NickLucche <nlucches@redhat.com>
 2025-03-19 21:00:39 -07:00
Woosuk Kwon 05ccd0aa35 [V1] Ensure using int64 for sampled token ids (#15065) 
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
 2025-03-18 23:52:19 -07:00
Chujie Zheng 027827cc1d fix long dtype in topk sampling (#15049) 2025-03-18 15:57:31 -07:00
Woosuk Kwon 99abb8b650 [V1][Spec Decode] Optimize Rejection Sampler with Triton Kernels (#14930) 
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
 2025-03-18 14:31:54 -07:00
Lily LiuandWoosuk Kwon 8d6cf89526 [V1] [Spec Decode] Support random sampling for spec decode (#13933) 
Co-authored-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
 2025-03-16 22:00:20 -07:00
DefTruth acaea3bb07 [Bugfix][V1] Fix flashinfer sampling (#14815) 2025-03-14 20:42:38 -07:00
Woosuk Kwon 32ef4983cd [V1] Temporarily disable FlashInfer Rejection Sampler (#14788) 
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
 2025-03-13 20:40:35 -07:00
22quinnandNick Hill eb8b5eb183 [V1] Support bad_words in sampler (#13376) 
Signed-off-by: 22quinn <33176974+22quinn@users.noreply.github.com>
Co-authored-by: Nick Hill <nhill@redhat.com>
 2025-03-08 14:50:26 -08:00
Harry Mellor cf069aa8aa Update deprecated Python 3.8 typing (#13971) 2025-03-02 17:34:51 -08:00
Lily Liu 5629f26df7 [V1][Spec Decode] Change Spec Decode Rejection Sampling API (#13729) 2025-02-25 18:14:48 -08:00
Lu Fang bb78fb318e [v1] Support allowed_token_ids in v1 Sampler (#13210) 
Signed-off-by: Lu Fang <lufang@fb.com>
 2025-02-22 14:13:05 +08:00
Nick Hill 31aa045c11 [V1][Sampler] Avoid an operation during temperature application (#13587) 2025-02-20 22:05:56 -08:00
Nick Hill 30172b4947 [V1] Optimize handling of sampling metadata and req_ids list (#13244) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-02-18 12:15:33 -08:00
Woosuk Kwon 69e1d23e1e [V1][BugFix] Clean up rejection sampler & Fix warning msg (#13362) 
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
 2025-02-16 12:25:29 -08:00
Lily Liu 80f63a3966 [V1][Spec Decode] Ngram Spec Decode (#12193) 
Signed-off-by: LiuXiaoxuanPKU <lilyliupku@gmail.com>
 2025-02-15 18:05:11 -08:00