[MRV2][SD] Make Dynamic SD comatible with Full Cuda Graphs (#45953)
Signed-off-by: Ekagra Ranjan <3116519+ekagra-ranjan@users.noreply.github.com> Co-authored-by: Benjamin Chislett <chislett.ben@gmail.com> Co-authored-by: Nick Hill <nickhill123@gmail.com>
This commit is contained in:
co-authored by
Benjamin Chislett
Nick Hill
parent
67ff0ae30f
commit
07516fda67
@@ -71,8 +71,5 @@ VLLM_USE_V2_MODEL_RUNNER=0 vllm serve meta-llama/Llama-3.1-8B-Instruct \
|
||||
|
||||
## Limitations
|
||||
|
||||
* only tested with Eagle and Eagle-3. Other SD methods may or may not work out of the box
|
||||
* only usable with Model Runner V1
|
||||
* not compatible with full cuda graph so we force piece-wise cuda graph with this feature
|
||||
|
||||
We are working on enabling it on MRv2 with full cuda graph support.
|
||||
* Tested with Eagle, Eagle-3, and DFlash. Other SD methods may or may not work out of the box
|
||||
* Full Cudagraph only works with Model Runner V2. MRv1 only supports piece-wise cuda graph with this feature
|
||||
|
||||
Reference in New Issue
Block a user