[NVIDIA][test] Tests for flashinfer TRTLLM BF16 MoE (#33715)
Signed-off-by: Linda-Stadter <57756729+Linda-Stadter@users.noreply.github.com> Co-authored-by: Pavani Majety <pmajety@nvidia.com>
This commit is contained in:
@@ -5,3 +5,5 @@ num_fewshot: 5
|
||||
server_args: "--enforce-eager --max-model-len 8192 --tensor-parallel-size 2 --enable-expert-parallel"
|
||||
env:
|
||||
VLLM_USE_FLASHINFER_MOE_FP16: "1"
|
||||
VLLM_FLASHINFER_MOE_BACKEND: "throughput"
|
||||
|
||||
|
||||
@@ -5,3 +5,4 @@ num_fewshot: 5
|
||||
server_args: "--enforce-eager --max-model-len 8192 --tensor-parallel-size 2 --enable-expert-parallel"
|
||||
env:
|
||||
VLLM_USE_FLASHINFER_MOE_FP16: "1"
|
||||
VLLM_FLASHINFER_MOE_BACKEND: "throughput"
|
||||
|
||||
Reference in New Issue
Block a user