iHore/vLLM
8
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity
4,381 Commits 3 Branches 0 Tags
44bbca78d71330909dbfdde232debdc73a4d5a81
Commit Graph
36 Commits
Author SHA1 Message Date
Ryan Nguyen fc542144c4 [Feature] Fix guided decoding blocking bitmask memcpy (#12563) 
**[Guided decoding performance optimization]** Sending the guided
decoding bitmask in xgrammar to the GPU
(`self.token_bitmask.to(scores.device)`) is a blocking operation that
prevents the CPU from pre-launching the sampler kernels. The CPU waits
until decode is complete, then copies the bitmask over. This PR changes
the operation to async via setting `non-blocking=True`.

(Current) The CPU is blocked on a `cudaStreamSynchronize` and only
pre-empts the sampling kernels after bitmask application. Below is the
Nsys profile for one decode phase from Llama 3.1 8B.

![image](https://github.com/user-attachments/assets/8997eae1-b822-4f52-beb8-ef19a7c6b824)

With the optimization, this is no longer the case:

![image](https://github.com/user-attachments/assets/6d5ea83f-f169-4f98-a8c1-41c719b3e1e7)

---------

Signed-off-by: Ryan N <ryan.nguyen@centml.ai>
 2025-01-31 15:37:30 -08:00
Cheng Kuan Yong Jason 06a760d6e8 [bugfix] catch xgrammar unsupported array constraints (#12210) 
Signed-off-by: Jason Cheng <jasoncky96@gmail.com>
 2025-01-20 16:42:02 -08:00
Michal Adamczyk 4e94951bb1 [BUGFIX] Move scores to float32 in case of running xgrammar on cpu (#12152) 
Signed-off-by: Michal Adamczyk <madamczyk@habana.ai>
 2025-01-19 11:12:05 +08:00
Michael Goin 74fa1d123c [Bugfix] Fix OpenAI parallel sampling when using xgrammar (#11637) 
Signed-off-by: mgoin <michael@neuralmagic.com>
 2024-12-31 03:43:54 +00:00
youkaichao b12e87f942 [platforms] enable platform plugins (#11602) 
Signed-off-by: youkaichao <youkaichao@gmail.com>
 2024-12-30 20:24:45 +08:00
Michael Goin 5bfb30a529 [Bugfix] Fix CFGGuide and use outlines for grammars that can't convert to GBNF (#11389) 
Signed-off-by: mgoin <michael@neuralmagic.com>
 2024-12-23 23:06:20 +08:00
Michael Goin a30482f054 [CI] Expand test_guided_generate to test all backends (#11313) 
Signed-off-by: mgoin <michael@neuralmagic.com>
 2024-12-19 04:00:38 +00:00
Wallas Henrique 8b79f9e107 [Bugfix] Fix guided decoding with tokenizer mode mistral (#11046) 2024-12-17 22:34:08 -08:00
Russell Bryant 48259264a4 [Core] Update outlines and increase its threadpool size (#11140) 
Signed-off-by: Russell Bryant <rbryant@redhat.com>
 2024-12-14 07:46:18 +00:00
Cody Yu 2c97eca1ff [Misc] Validate grammar and fail early (#11119) 2024-12-12 18:34:26 +00:00
Kevin H. Luu 9974fca047 [ci/build] Fix entrypoints test and pin outlines version (#11088) 2024-12-11 01:01:53 -08:00
Russell Bryant e739194926 [Core] Update to outlines >= 0.1.8 (#10576) 
Signed-off-by: Russell Bryant <rbryant@redhat.com>
 2024-12-10 12:08:16 -08:00
Jeff Cook e35879c276 [Bugfix] Fix xgrammar failing to read a vocab_size from LlavaConfig on PixtralHF. (#11043) 2024-12-10 14:54:22 +08:00
Michael Goin 8b59631855 [Core] Support Lark grammars for XGrammar (#10870) 
Signed-off-by: mgoin <michael@neuralmagic.com>
 2024-12-06 08:34:29 -07:00
Michael Goin 8d370e91cb [Bugfix] Fallback to outlines for complex json schemas (#10899) 
Signed-off-by: mgoin <michael@neuralmagic.com>
 2024-12-05 11:14:06 +08:00
Michael Goin 7090c27bb2 [Bugfix] Only require XGrammar on x86 (#10865) 
Signed-off-by: mgoin <michael@neuralmagic.com>
 2024-12-03 10:32:21 -08:00
Aaron Phamandmgoin 9323a3153b [Core][Performance] Add XGrammar support for guided decoding and set it as default (#10785) 
Signed-off-by: Aaron Pham <contact@aarnphm.xyz>
Signed-off-by: mgoin <michael@neuralmagic.com>
Co-authored-by: mgoin <michael@neuralmagic.com>
 2024-12-03 15:17:00 +08:00
Vasiliy Alekseev 07e981fdf4 [Frontend] Bad words sampling parameter (#9717) 
Signed-off-by: Vasily Alexeev <alvasian@yandex.ru>
 2024-10-26 16:29:38 +00:00
Travis Johnsonpavlo-rubanNick Hill
 		6567e13724 [Bugfix] Fix crash with llama 3.2 vision models and guided decoding (#9631) 
Signed-off-by: Travis Johnson <tsjohnso@us.ibm.com>
Co-authored-by: pavlo-ruban <pavlo.ruban@servicenow.com>
Co-authored-by: Nick Hill <nickhill@us.ibm.com>
 2024-10-25 15:42:56 -07:00
Joe RundeandNick Hill 062c89e7c9 [Frontend][Core] Move guided decoding params into sampling params (#8252) 
Signed-off-by: Joe Runde <Joseph.Runde@ibm.com>
Co-authored-by: Nick Hill <nickhill@us.ibm.com>
 2024-10-01 09:34:25 +08:00
Roger Wang 02c9afa2d0 Revert "[Misc][Bugfix] Disable guided decoding for mistral tokenizer" (#8593) 2024-09-19 04:14:28 +00:00
Aaron PhamandCyrus Leung 9d104b5beb [CI/Build] Update Ruff version (#8469) 
Signed-off-by: Aaron Pham <contact@aarnphm.xyz>
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>
 2024-09-18 11:00:56 +00:00
Roger Wang ee2bceaaa6 [Misc][Bugfix] Disable guided decoding for mistral tokenizer (#8521) 2024-09-16 22:22:45 -07:00
Kyle MisteleconstellateKyle Mistele
 		e02ce498be [Feature] OpenAI-Compatible Tools API + Streaming for Hermes & Mistral models (#5649) 
Co-authored-by: constellate <constellate@1-ai-appserver-staging.codereach.com>
Co-authored-by: Kyle Mistele <kyle@constellate.ai>
 2024-09-04 13:18:13 -07:00
youkaichao 7d9ffa2ae1 [misc][core] lazy import outlines (#7831) 2024-08-24 00:51:38 -07:00
Tyler Rockwood d81abefd2e [Frontend] add json_schema support from OpenAI protocol (#7654) 2024-08-23 23:07:24 -07:00
Yihuan BuandCyrus Leung 654bc5ca49 Support for guided decoding for offline LLM (#6878) 
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>
 2024-08-04 03:12:09 +00:00
Robert ShawJoe RundeJoe RundeNick HillSimon Mo
 		ed812a73fa [ Frontend ] Multiprocessing for OpenAI Server with zeromq (#6883) 
Signed-off-by: Joe Runde <Joseph.Runde@ibm.com>
Co-authored-by: Joe Runde <Joseph.Runde@ibm.com>
Co-authored-by: Joe Runde <joe@joerun.de>
Co-authored-by: Nick Hill <nickhill@us.ibm.com>
Co-authored-by: Simon Mo <simon.mo@hey.com>
 2024-08-02 18:27:28 -07:00
Eric 185ad31f37 [Bugfix] use diskcache in outlines _get_guide #5436 (#6203) 2024-07-08 11:23:24 -07:00
Breno FariaSimon MoBreno Faria
 		7b0a0dfb22 [Frontend][Core] Update Outlines Integration from FSM to Guide (#4109) 
Co-authored-by: Simon Mo <simon.mo@hey.com>
Co-authored-by: Breno Faria <breno.faria@intrafind.com>
 2024-06-05 16:49:12 -07:00
Breno Faria f775a07e30 [FRONTEND] OpenAI tools support named functions (#5032) 2024-06-03 18:25:29 -05:00
Robert Caulk c3845d82dc Allow user to define whitespace pattern for outlines (#4305) 2024-04-30 20:48:39 -07:00
SangBin Cho df29793dc7 [mypy][5/N] Support all typing on model executor (#4427) 2024-04-28 19:01:26 -07:00
Ayush RautwarandUbuntu 138485a82d [Bugfix] Add fix for JSON whitespace (#4189) 
Co-authored-by: Ubuntu <ubuntu@ip-172-31-13-147.ec2.internal>
 2024-04-19 20:49:22 -07:00
SangBin ChoandSangBin Cho 533d2a1f39 [Typing] Mypy typing part 2 (#4043) 
Co-authored-by: SangBin Cho <sangcho@sangcho-LT93GQWG9C.local>
 2024-04-17 17:28:43 -07:00
Noam GatandSimon Mo 05434764cd LM Format Enforcer Guided Decoding Support (#3868) 
Co-authored-by: Simon Mo <simon.mo@hey.com>
 2024-04-16 05:54:57 +00:00