iHore/vLLM
8
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity
5,995 Commits 3 Branches 0 Tags
3cde34a4a4bceb511f2f9fe2dade1da116eb7e8a
Commit Graph
59 Commits
Author SHA1 Message Date
Travis Johnson 3cde34a4a4 [Frontend] Support guidance:no-additional-properties for compatibility with xgrammar (#15949) 
Signed-off-by: Travis Johnson <tsjohnso@us.ibm.com>
 2025-04-23 18:34:41 +00:00
Russell Bryant dc1b4a6f13 [Core][V0] Enable regex support with xgrammar (#13228) 
Signed-off-by: Russell Bryant <rbryant@redhat.com>
 2025-04-14 10:13:38 +08:00
Russell Bryant 9665313c39 [V1] Set structured output backend to auto by default (#15724) 
Signed-off-by: Russell Bryant <rbryant@redhat.com>
 2025-04-10 17:53:26 +00:00
Russell Bryant cb84e45ac7 [Core] Upgrade to xgrammar 0.1.18, add cache size limit (#16283) 
Signed-off-by: Russell Bryant <rbryant@redhat.com>
 2025-04-08 19:13:22 -07:00
leon-seidel 24f1c01e0f [Bugfix][V0] XGrammar structured output supports Enum (#15878) 
Signed-off-by: Leon Seidel <leon.seidel@fau.de>
 2025-04-07 22:38:25 +00:00
Russell Bryant 14e53ed11f [V1] Fix json_object support with xgrammar (#15488) 
Signed-off-by: Russell Bryant <rbryant@redhat.com>
 2025-04-02 02:00:08 -07:00
Russell Bryant 7329ff5468 [V1] Support disable_any_whtespace for guidance backend (#15584) 
Signed-off-by: Russell Bryant <rbryant@redhat.com>
 2025-03-28 23:46:45 +08:00
Ce Gao 32b14baf8a [Refactor][Frontend] Keep all logic about reasoning into one class (#14428) 
Signed-off-by: Ce Gao <cegao@tensorchord.ai>
 2025-03-28 00:23:30 -07:00
Alex BrooksandJoe Runde 1711b929b6 [Model] Add Reasoning Parser for Granite Models (#14202) 
Signed-off-by: Alex-Brooks <Alex.brooks@ibm.com>
Co-authored-by: Joe Runde <joe@joerun.de>
 2025-03-26 14:28:07 +00:00
Russell BryantLoc HuynhMichal MoskalAaron Pham
 		1f16b7fe74 [Core][V0] Add guidance backend for structured output (#14589) 
Signed-off-by: Russell Bryant <rbryant@redhat.com>
Co-authored-by: Loc Huynh <lohuynh@microsoft.com>
Co-authored-by: Michal Moskal <michal@moskal.me>
Co-authored-by: Aaron Pham <contact@aarnphm.xyz>
 2025-03-19 21:33:51 -07:00
Aaron Phamrshaw@neuralmagic.com <rshaw@neuralmagic.com>Russell BryantRobert Shaw
 		c0efdd655b [Fix][Structured Output] using vocab_size to construct matcher (#14868) 
Signed-off-by: Russell Bryant <rbryant@redhat.com>
Signed-off-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com>
Signed-off-by: Aaron Pham <contact@aarnphm.xyz>
Co-authored-by: rshaw@neuralmagic.com <rshaw@neuralmagic.com>
Co-authored-by: Russell Bryant <rbryant@redhat.com>
Co-authored-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com>
 2025-03-17 11:42:45 -04:00
Russell Bryant 776dcec8fe Disable outlines cache by default (#14837) 2025-03-15 03:57:55 +00:00
Ce Gao f5f7f00cd9 [Bugfix][Structured Output] Support outlines engine with reasoning outputs for DeepSeek R1 (#14114) 2025-03-06 03:49:20 +00:00
Ce Gao bf33700ecd [v0][structured output] Support reasoning output (#12955) 
Signed-off-by: Ce Gao <cegao@tensorchord.ai>
 2025-03-02 14:49:42 -05:00
Wallas HenriqueandJoe Runde 4cb6fa0a9c [Bugfix] Backend option to disable xgrammar any_whitespace (#12744) 
Signed-off-by: Wallas Santos <wallashss@ibm.com>
Signed-off-by: Joe Runde <Joseph.Runde@ibm.com>
Co-authored-by: Joe Runde <Joseph.Runde@ibm.com>
 2025-02-26 10:52:34 -08:00
Seth Kimmel e206b54331 [v0][Core] Use xgrammar shared context to avoid copy overhead for offline engine (#13837) 
Signed-off-by: Seth Kimmel <seth.kimmel3@gmail.com>
 2025-02-26 14:58:24 +08:00
Russell Bryant aab392774b [Core] xgrammar: Expand list of unsupported jsonschema keywords (#13783) 
Signed-off-by: Russell Bryant <rbryant@redhat.com>
 2025-02-25 08:21:25 +00:00
Joe Runde bfbc0b32c6 [Frontend] Add backend-specific options for guided decoding (#13505) 
Signed-off-by: Joe Runde <Joseph.Runde@ibm.com>
 2025-02-20 15:07:58 -05:00
Michael Goin b53d79983c Add outlines fallback when JSON schema has enum (#13449) 
Signed-off-by: mgoin <mgoin64@gmail.com>
 2025-02-18 06:49:41 +00:00
Yu-Zhou c9e2d644e7 [Hardware][Gaudi][Bugfix] Fix error for guided decoding (#12317) 2025-02-14 04:36:49 -08:00
Russell Bryant 7734e9a291 [Core] choice-based structured output with xgrammar (#12632) 2025-02-14 04:36:05 -08:00
Yuan Tang 14ecab5be2 [Bugfix] Guided decoding falls back to outlines when fails to import xgrammar (#12976) 
Signed-off-by: Yuan Tang <terrytangyuan@gmail.com>
 2025-02-11 18:17:44 +00:00
Russell Bryant e489ad7a21 [Misc] Add SPDX-License-Identifier headers to python source files (#12628) 
- **Add SPDX license headers to python source files**
- **Check for SPDX headers using pre-commit**

commit 9d7ef44c3cfb72ca4c32e1c677d99259d10d4745
Author: Russell Bryant <rbryant@redhat.com>
Date:   Fri Jan 31 14:18:24 2025 -0500

    Add SPDX license headers to python source files
    
This commit adds SPDX license headers to python source files as
recommended to
the project by the Linux Foundation. These headers provide a concise way
that is
both human and machine readable for communicating license information
for each
source file. It helps avoid any ambiguity about the license of the code
and can
    also be easily used by tools to help manage license compliance.
    
The Linux Foundation runs license scans against the codebase to help
ensure
    we are in compliance with the licenses of the code we use, including
dependencies. Having these headers in place helps that tool do its job.
    
    More information can be found on the SPDX site:
    
    - https://spdx.dev/learn/handling-license-info/
    
    Signed-off-by: Russell Bryant <rbryant@redhat.com>

commit 5a1cf1cb3b80759131c73f6a9dddebccac039dea
Author: Russell Bryant <rbryant@redhat.com>
Date:   Fri Jan 31 14:36:32 2025 -0500

    Check for SPDX headers using pre-commit
    
    Signed-off-by: Russell Bryant <rbryant@redhat.com>

---------

Signed-off-by: Russell Bryant <rbryant@redhat.com>
 2025-02-02 11:58:18 -08:00
Ryan Nguyen fc542144c4 [Feature] Fix guided decoding blocking bitmask memcpy (#12563) 
**[Guided decoding performance optimization]** Sending the guided
decoding bitmask in xgrammar to the GPU
(`self.token_bitmask.to(scores.device)`) is a blocking operation that
prevents the CPU from pre-launching the sampler kernels. The CPU waits
until decode is complete, then copies the bitmask over. This PR changes
the operation to async via setting `non-blocking=True`.

(Current) The CPU is blocked on a `cudaStreamSynchronize` and only
pre-empts the sampling kernels after bitmask application. Below is the
Nsys profile for one decode phase from Llama 3.1 8B.

![image](https://github.com/user-attachments/assets/8997eae1-b822-4f52-beb8-ef19a7c6b824)

With the optimization, this is no longer the case:

![image](https://github.com/user-attachments/assets/6d5ea83f-f169-4f98-a8c1-41c719b3e1e7)

---------

Signed-off-by: Ryan N <ryan.nguyen@centml.ai>
 2025-01-31 15:37:30 -08:00
Cheng Kuan Yong Jason 06a760d6e8 [bugfix] catch xgrammar unsupported array constraints (#12210) 
Signed-off-by: Jason Cheng <jasoncky96@gmail.com>
 2025-01-20 16:42:02 -08:00
Michal Adamczyk 4e94951bb1 [BUGFIX] Move scores to float32 in case of running xgrammar on cpu (#12152) 
Signed-off-by: Michal Adamczyk <madamczyk@habana.ai>
 2025-01-19 11:12:05 +08:00
Michael Goin 74fa1d123c [Bugfix] Fix OpenAI parallel sampling when using xgrammar (#11637) 
Signed-off-by: mgoin <michael@neuralmagic.com>
 2024-12-31 03:43:54 +00:00
youkaichao b12e87f942 [platforms] enable platform plugins (#11602) 
Signed-off-by: youkaichao <youkaichao@gmail.com>
 2024-12-30 20:24:45 +08:00
Michael Goin 5bfb30a529 [Bugfix] Fix CFGGuide and use outlines for grammars that can't convert to GBNF (#11389) 
Signed-off-by: mgoin <michael@neuralmagic.com>
 2024-12-23 23:06:20 +08:00
Michael Goin a30482f054 [CI] Expand test_guided_generate to test all backends (#11313) 
Signed-off-by: mgoin <michael@neuralmagic.com>
 2024-12-19 04:00:38 +00:00
Wallas Henrique 8b79f9e107 [Bugfix] Fix guided decoding with tokenizer mode mistral (#11046) 2024-12-17 22:34:08 -08:00
Russell Bryant 48259264a4 [Core] Update outlines and increase its threadpool size (#11140) 
Signed-off-by: Russell Bryant <rbryant@redhat.com>
 2024-12-14 07:46:18 +00:00
Cody Yu 2c97eca1ff [Misc] Validate grammar and fail early (#11119) 2024-12-12 18:34:26 +00:00
Kevin H. Luu 9974fca047 [ci/build] Fix entrypoints test and pin outlines version (#11088) 2024-12-11 01:01:53 -08:00
Russell Bryant e739194926 [Core] Update to outlines >= 0.1.8 (#10576) 
Signed-off-by: Russell Bryant <rbryant@redhat.com>
 2024-12-10 12:08:16 -08:00
Jeff Cook e35879c276 [Bugfix] Fix xgrammar failing to read a vocab_size from LlavaConfig on PixtralHF. (#11043) 2024-12-10 14:54:22 +08:00
Michael Goin 8b59631855 [Core] Support Lark grammars for XGrammar (#10870) 
Signed-off-by: mgoin <michael@neuralmagic.com>
 2024-12-06 08:34:29 -07:00
Michael Goin 8d370e91cb [Bugfix] Fallback to outlines for complex json schemas (#10899) 
Signed-off-by: mgoin <michael@neuralmagic.com>
 2024-12-05 11:14:06 +08:00
Michael Goin 7090c27bb2 [Bugfix] Only require XGrammar on x86 (#10865) 
Signed-off-by: mgoin <michael@neuralmagic.com>
 2024-12-03 10:32:21 -08:00
Aaron Phamandmgoin 9323a3153b [Core][Performance] Add XGrammar support for guided decoding and set it as default (#10785) 
Signed-off-by: Aaron Pham <contact@aarnphm.xyz>
Signed-off-by: mgoin <michael@neuralmagic.com>
Co-authored-by: mgoin <michael@neuralmagic.com>
 2024-12-03 15:17:00 +08:00
Vasiliy Alekseev 07e981fdf4 [Frontend] Bad words sampling parameter (#9717) 
Signed-off-by: Vasily Alexeev <alvasian@yandex.ru>
 2024-10-26 16:29:38 +00:00
Travis Johnsonpavlo-rubanNick Hill
 		6567e13724 [Bugfix] Fix crash with llama 3.2 vision models and guided decoding (#9631) 
Signed-off-by: Travis Johnson <tsjohnso@us.ibm.com>
Co-authored-by: pavlo-ruban <pavlo.ruban@servicenow.com>
Co-authored-by: Nick Hill <nickhill@us.ibm.com>
 2024-10-25 15:42:56 -07:00
Joe RundeandNick Hill 062c89e7c9 [Frontend][Core] Move guided decoding params into sampling params (#8252) 
Signed-off-by: Joe Runde <Joseph.Runde@ibm.com>
Co-authored-by: Nick Hill <nickhill@us.ibm.com>
 2024-10-01 09:34:25 +08:00
Roger Wang 02c9afa2d0 Revert "[Misc][Bugfix] Disable guided decoding for mistral tokenizer" (#8593) 2024-09-19 04:14:28 +00:00
Aaron PhamandCyrus Leung 9d104b5beb [CI/Build] Update Ruff version (#8469) 
Signed-off-by: Aaron Pham <contact@aarnphm.xyz>
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>
 2024-09-18 11:00:56 +00:00
Roger Wang ee2bceaaa6 [Misc][Bugfix] Disable guided decoding for mistral tokenizer (#8521) 2024-09-16 22:22:45 -07:00
Kyle MisteleconstellateKyle Mistele
 		e02ce498be [Feature] OpenAI-Compatible Tools API + Streaming for Hermes & Mistral models (#5649) 
Co-authored-by: constellate <constellate@1-ai-appserver-staging.codereach.com>
Co-authored-by: Kyle Mistele <kyle@constellate.ai>
 2024-09-04 13:18:13 -07:00
youkaichao 7d9ffa2ae1 [misc][core] lazy import outlines (#7831) 2024-08-24 00:51:38 -07:00
Tyler Rockwood d81abefd2e [Frontend] add json_schema support from OpenAI protocol (#7654) 2024-08-23 23:07:24 -07:00
Yihuan BuandCyrus Leung 654bc5ca49 Support for guided decoding for offline LLM (#6878) 
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>
 2024-08-04 03:12:09 +00:00