[CPU][Zen] Update zentorch to 2.14.0.1 & enable float16 on Venice and later (#61411)
Signed-off-by: Ganesh R <ganesh.r@amd.com> Co-authored-by: Kevin H. Luu <khluu000@gmail.com> Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>
This commit is contained in:
co-authored by
Kevin H. Luu
Cyrus Leung
parent
85c94c1706
commit
91ed2859c7
@@ -159,7 +159,7 @@ VLLM_USE_PRECOMPILED=1 VLLM_PRECOMPILED_WHEEL_VARIANT=cpu VLLM_TARGET_DEVICE=cpu
|
||||
### Which `dtype` should be used?
|
||||
|
||||
- Currently, vLLM CPU uses model default settings as `dtype`. However, due to unstable float16 support in torch CPU, it is recommended to explicitly set `dtype=bfloat16` if there are any performance or accuracy problem.
|
||||
- On AMD Zen CPUs (`ZenCpuPlatform`), `float16` is **not** supported. Only `bfloat16` and `float32` are accepted; models declared with `float16` are auto-downcast to `bfloat16` at model load time. See [AMD Zen optimizations](#amd-zen-optimizations).
|
||||
- On AMD Zen CPUs (`ZenCpuPlatform`), `float16` requires AVX512-FP16, available from Venice (6th Gen EPYC) onwards. On Turin (5th Gen EPYC) and earlier only `bfloat16` and `float32` are accepted, and models declared with `float16` are auto-downcast to `bfloat16` at model load time. See [AMD Zen optimizations](#amd-zen-optimizations).
|
||||
|
||||
### How to launch a vLLM service on CPU?
|
||||
|
||||
|
||||
@@ -273,7 +273,9 @@ Otherwise, vLLM falls back to the default `CpuPlatform` (oneDNN / sgl-kernel pat
|
||||
|
||||
### Supported dtypes
|
||||
|
||||
`float16` is **not** supported on `ZenCpuPlatform`. `ZenCpuPlatform.supported_dtypes` advertises only `bfloat16` and `float32`, so models declared with `torch_dtype=float16` are auto-downcast to `bfloat16` at load time with the standard `"Your device 'cpu' doesn't support torch.float16. Falling back to torch.bfloat16 for compatibility."` warning emitted from `vllm/config/model.py`.
|
||||
`ZenCpuPlatform.supported_dtypes` always advertises `bfloat16` and `float32`, and adds `float16` when `torch.ops.zentorch.zentorch_is_fp16_supported()` reports AVX512-FP16, which AMD ships from Venice (6th Gen EPYC) onwards. On Turin (5th Gen EPYC) and earlier, models declared with `torch_dtype=float16` are auto-downcast to `bfloat16` at load time with the standard `"Your device 'cpu' doesn't support torch.float16. Falling back to torch.bfloat16 for compatibility."` warning emitted from `vllm/config/model.py`.
|
||||
|
||||
`bfloat16` is listed first either way, so `--dtype auto` keeps resolving to it.
|
||||
|
||||
### Environment variables
|
||||
|
||||
|
||||
Reference in New Issue
Block a user