Ryan Nguyen fc542144c4 [Feature] Fix guided decoding blocking bitmask memcpy (#12563) ...

**[Guided decoding performance optimization]** Sending the guided
decoding bitmask in xgrammar to the GPU
(`self.token_bitmask.to(scores.device)`) is a blocking operation that
prevents the CPU from pre-launching the sampler kernels. The CPU waits
until decode is complete, then copies the bitmask over. This PR changes
the operation to async via setting `non-blocking=True`.

(Current) The CPU is blocked on a `cudaStreamSynchronize` and only
pre-empts the sampling kernels after bitmask application. Below is the
Nsys profile for one decode phase from Llama 3.1 8B.

![image](https://github.com/user-attachments/assets/8997eae1-b822-4f52-beb8-ef19a7c6b824)

With the optimization, this is no longer the case:

![image](https://github.com/user-attachments/assets/6d5ea83f-f169-4f98-a8c1-41c719b3e1e7)

---------

Signed-off-by: Ryan N <ryan.nguyen@centml.ai>

2025-01-31 15:37:30 -08:00

..

__init__.py

[platforms] enable platform plugins (#11602)

2024-12-30 20:24:45 +08:00

guided_fields.py

[Frontend][Core] Move guided decoding params into sampling params (#8252)

2024-10-01 09:34:25 +08:00

lm_format_enforcer_decoding.py

[Frontend] Bad words sampling parameter (#9717)

2024-10-26 16:29:38 +00:00

outlines_decoding.py

[Core] Update outlines and increase its threadpool size (#11140)

2024-12-14 07:46:18 +00:00

outlines_logits_processors.py

[Bugfix] Fix CFGGuide and use outlines for grammars that can't convert to GBNF (#11389)

2024-12-23 23:06:20 +08:00

utils.py

[bugfix] catch xgrammar unsupported array constraints (#12210)

2025-01-20 16:42:02 -08:00

xgrammar_decoding.py

[Feature] Fix guided decoding blocking bitmask memcpy (#12563)

2025-01-31 15:37:30 -08:00