iHore/vLLM
8
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity
4,774 Commits 3 Branches 0 Tags
db986c19ea35d7f3522a45d5205bf5d3ffab14e4
Commit Graph
42 Commits
Author SHA1 Message Date
Joe Runde bfbc0b32c6 [Frontend] Add backend-specific options for guided decoding (#13505) 
Signed-off-by: Joe Runde <Joseph.Runde@ibm.com>
 2025-02-20 15:07:58 -05:00
Michael Goin b53d79983c Add outlines fallback when JSON schema has enum (#13449) 
Signed-off-by: mgoin <mgoin64@gmail.com>
 2025-02-18 06:49:41 +00:00
Yu-Zhou c9e2d644e7 [Hardware][Gaudi][Bugfix] Fix error for guided decoding (#12317) 2025-02-14 04:36:49 -08:00
Russell Bryant 7734e9a291 [Core] choice-based structured output with xgrammar (#12632) 2025-02-14 04:36:05 -08:00
Yuan Tang 14ecab5be2 [Bugfix] Guided decoding falls back to outlines when fails to import xgrammar (#12976) 
Signed-off-by: Yuan Tang <terrytangyuan@gmail.com>
 2025-02-11 18:17:44 +00:00
Russell Bryant e489ad7a21 [Misc] Add SPDX-License-Identifier headers to python source files (#12628) 
- **Add SPDX license headers to python source files**
- **Check for SPDX headers using pre-commit**

commit 9d7ef44c3cfb72ca4c32e1c677d99259d10d4745
Author: Russell Bryant <rbryant@redhat.com>
Date:   Fri Jan 31 14:18:24 2025 -0500

    Add SPDX license headers to python source files
    
This commit adds SPDX license headers to python source files as
recommended to
the project by the Linux Foundation. These headers provide a concise way
that is
both human and machine readable for communicating license information
for each
source file. It helps avoid any ambiguity about the license of the code
and can
    also be easily used by tools to help manage license compliance.
    
The Linux Foundation runs license scans against the codebase to help
ensure
    we are in compliance with the licenses of the code we use, including
dependencies. Having these headers in place helps that tool do its job.
    
    More information can be found on the SPDX site:
    
    - https://spdx.dev/learn/handling-license-info/
    
    Signed-off-by: Russell Bryant <rbryant@redhat.com>

commit 5a1cf1cb3b80759131c73f6a9dddebccac039dea
Author: Russell Bryant <rbryant@redhat.com>
Date:   Fri Jan 31 14:36:32 2025 -0500

    Check for SPDX headers using pre-commit
    
    Signed-off-by: Russell Bryant <rbryant@redhat.com>

---------

Signed-off-by: Russell Bryant <rbryant@redhat.com>
 2025-02-02 11:58:18 -08:00
Ryan Nguyen fc542144c4 [Feature] Fix guided decoding blocking bitmask memcpy (#12563) 
**[Guided decoding performance optimization]** Sending the guided
decoding bitmask in xgrammar to the GPU
(`self.token_bitmask.to(scores.device)`) is a blocking operation that
prevents the CPU from pre-launching the sampler kernels. The CPU waits
until decode is complete, then copies the bitmask over. This PR changes
the operation to async via setting `non-blocking=True`.

(Current) The CPU is blocked on a `cudaStreamSynchronize` and only
pre-empts the sampling kernels after bitmask application. Below is the
Nsys profile for one decode phase from Llama 3.1 8B.

![image](https://github.com/user-attachments/assets/8997eae1-b822-4f52-beb8-ef19a7c6b824)

With the optimization, this is no longer the case:

![image](https://github.com/user-attachments/assets/6d5ea83f-f169-4f98-a8c1-41c719b3e1e7)

---------

Signed-off-by: Ryan N <ryan.nguyen@centml.ai>
 2025-01-31 15:37:30 -08:00
Cheng Kuan Yong Jason 06a760d6e8 [bugfix] catch xgrammar unsupported array constraints (#12210) 
Signed-off-by: Jason Cheng <jasoncky96@gmail.com>
 2025-01-20 16:42:02 -08:00
Michal Adamczyk 4e94951bb1 [BUGFIX] Move scores to float32 in case of running xgrammar on cpu (#12152) 
Signed-off-by: Michal Adamczyk <madamczyk@habana.ai>
 2025-01-19 11:12:05 +08:00
Michael Goin 74fa1d123c [Bugfix] Fix OpenAI parallel sampling when using xgrammar (#11637) 
Signed-off-by: mgoin <michael@neuralmagic.com>
 2024-12-31 03:43:54 +00:00
youkaichao b12e87f942 [platforms] enable platform plugins (#11602) 
Signed-off-by: youkaichao <youkaichao@gmail.com>
 2024-12-30 20:24:45 +08:00
Michael Goin 5bfb30a529 [Bugfix] Fix CFGGuide and use outlines for grammars that can't convert to GBNF (#11389) 
Signed-off-by: mgoin <michael@neuralmagic.com>
 2024-12-23 23:06:20 +08:00
Michael Goin a30482f054 [CI] Expand test_guided_generate to test all backends (#11313) 
Signed-off-by: mgoin <michael@neuralmagic.com>
 2024-12-19 04:00:38 +00:00
Wallas Henrique 8b79f9e107 [Bugfix] Fix guided decoding with tokenizer mode mistral (#11046) 2024-12-17 22:34:08 -08:00
Russell Bryant 48259264a4 [Core] Update outlines and increase its threadpool size (#11140) 
Signed-off-by: Russell Bryant <rbryant@redhat.com>
 2024-12-14 07:46:18 +00:00
Cody Yu 2c97eca1ff [Misc] Validate grammar and fail early (#11119) 2024-12-12 18:34:26 +00:00
Kevin H. Luu 9974fca047 [ci/build] Fix entrypoints test and pin outlines version (#11088) 2024-12-11 01:01:53 -08:00
Russell Bryant e739194926 [Core] Update to outlines >= 0.1.8 (#10576) 
Signed-off-by: Russell Bryant <rbryant@redhat.com>
 2024-12-10 12:08:16 -08:00
Jeff Cook e35879c276 [Bugfix] Fix xgrammar failing to read a vocab_size from LlavaConfig on PixtralHF. (#11043) 2024-12-10 14:54:22 +08:00
Michael Goin 8b59631855 [Core] Support Lark grammars for XGrammar (#10870) 
Signed-off-by: mgoin <michael@neuralmagic.com>
 2024-12-06 08:34:29 -07:00
Michael Goin 8d370e91cb [Bugfix] Fallback to outlines for complex json schemas (#10899) 
Signed-off-by: mgoin <michael@neuralmagic.com>
 2024-12-05 11:14:06 +08:00
Michael Goin 7090c27bb2 [Bugfix] Only require XGrammar on x86 (#10865) 
Signed-off-by: mgoin <michael@neuralmagic.com>
 2024-12-03 10:32:21 -08:00
Aaron Phamandmgoin 9323a3153b [Core][Performance] Add XGrammar support for guided decoding and set it as default (#10785) 
Signed-off-by: Aaron Pham <contact@aarnphm.xyz>
Signed-off-by: mgoin <michael@neuralmagic.com>
Co-authored-by: mgoin <michael@neuralmagic.com>
 2024-12-03 15:17:00 +08:00
Vasiliy Alekseev 07e981fdf4 [Frontend] Bad words sampling parameter (#9717) 
Signed-off-by: Vasily Alexeev <alvasian@yandex.ru>
 2024-10-26 16:29:38 +00:00
Travis Johnsonpavlo-rubanNick Hill
 		6567e13724 [Bugfix] Fix crash with llama 3.2 vision models and guided decoding (#9631) 
Signed-off-by: Travis Johnson <tsjohnso@us.ibm.com>
Co-authored-by: pavlo-ruban <pavlo.ruban@servicenow.com>
Co-authored-by: Nick Hill <nickhill@us.ibm.com>
 2024-10-25 15:42:56 -07:00
Joe RundeandNick Hill 062c89e7c9 [Frontend][Core] Move guided decoding params into sampling params (#8252) 
Signed-off-by: Joe Runde <Joseph.Runde@ibm.com>
Co-authored-by: Nick Hill <nickhill@us.ibm.com>
 2024-10-01 09:34:25 +08:00
Roger Wang 02c9afa2d0 Revert "[Misc][Bugfix] Disable guided decoding for mistral tokenizer" (#8593) 2024-09-19 04:14:28 +00:00
Aaron PhamandCyrus Leung 9d104b5beb [CI/Build] Update Ruff version (#8469) 
Signed-off-by: Aaron Pham <contact@aarnphm.xyz>
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>
 2024-09-18 11:00:56 +00:00
Roger Wang ee2bceaaa6 [Misc][Bugfix] Disable guided decoding for mistral tokenizer (#8521) 2024-09-16 22:22:45 -07:00
Kyle MisteleconstellateKyle Mistele
 		e02ce498be [Feature] OpenAI-Compatible Tools API + Streaming for Hermes & Mistral models (#5649) 
Co-authored-by: constellate <constellate@1-ai-appserver-staging.codereach.com>
Co-authored-by: Kyle Mistele <kyle@constellate.ai>
 2024-09-04 13:18:13 -07:00
youkaichao 7d9ffa2ae1 [misc][core] lazy import outlines (#7831) 2024-08-24 00:51:38 -07:00
Tyler Rockwood d81abefd2e [Frontend] add json_schema support from OpenAI protocol (#7654) 2024-08-23 23:07:24 -07:00
Yihuan BuandCyrus Leung 654bc5ca49 Support for guided decoding for offline LLM (#6878) 
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>
 2024-08-04 03:12:09 +00:00
Robert ShawJoe RundeJoe RundeNick HillSimon Mo
 		ed812a73fa [ Frontend ] Multiprocessing for OpenAI Server with zeromq (#6883) 
Signed-off-by: Joe Runde <Joseph.Runde@ibm.com>
Co-authored-by: Joe Runde <Joseph.Runde@ibm.com>
Co-authored-by: Joe Runde <joe@joerun.de>
Co-authored-by: Nick Hill <nickhill@us.ibm.com>
Co-authored-by: Simon Mo <simon.mo@hey.com>
 2024-08-02 18:27:28 -07:00
Eric 185ad31f37 [Bugfix] use diskcache in outlines _get_guide #5436 (#6203) 2024-07-08 11:23:24 -07:00
Breno FariaSimon MoBreno Faria
 		7b0a0dfb22 [Frontend][Core] Update Outlines Integration from FSM to Guide (#4109) 
Co-authored-by: Simon Mo <simon.mo@hey.com>
Co-authored-by: Breno Faria <breno.faria@intrafind.com>
 2024-06-05 16:49:12 -07:00
Breno Faria f775a07e30 [FRONTEND] OpenAI tools support named functions (#5032) 2024-06-03 18:25:29 -05:00
Robert Caulk c3845d82dc Allow user to define whitespace pattern for outlines (#4305) 2024-04-30 20:48:39 -07:00
SangBin Cho df29793dc7 [mypy][5/N] Support all typing on model executor (#4427) 2024-04-28 19:01:26 -07:00
Ayush RautwarandUbuntu 138485a82d [Bugfix] Add fix for JSON whitespace (#4189) 
Co-authored-by: Ubuntu <ubuntu@ip-172-31-13-147.ec2.internal>
 2024-04-19 20:49:22 -07:00
SangBin ChoandSangBin Cho 533d2a1f39 [Typing] Mypy typing part 2 (#4043) 
Co-authored-by: SangBin Cho <sangcho@sangcho-LT93GQWG9C.local>
 2024-04-17 17:28:43 -07:00
Noam GatandSimon Mo 05434764cd LM Format Enforcer Guided Decoding Support (#3868) 
Co-authored-by: Simon Mo <simon.mo@hey.com>
 2024-04-16 05:54:57 +00:00