[ModelOpt] Redesign the LinearMethod classes using the generic QuantKey-driven method (#49381)
Signed-off-by: Juhi Mittal <juhim@nvidia.com> Signed-off-by: mgoin <mgoin64@gmail.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: mgoin <mgoin64@gmail.com>
This commit is contained in:
co-authored by
Claude Opus 4.8
mgoin
parent
4bf06be985
commit
7a977c0699
File diff suppressed because it is too large
Load Diff
@@ -38,6 +38,11 @@ from vllm.utils.platform_utils import get_device_name_as_file_name
|
||||
|
||||
logger = init_logger(__name__)
|
||||
|
||||
# Pre-fill value for scale parameters whose shards load independently. The
|
||||
# shards are combined with .max(), so an unloaded shard must never win; the
|
||||
# smallest representable float32 guarantees that.
|
||||
FP8_SCALE_SENTINEL = torch.finfo(torch.float32).min
|
||||
|
||||
|
||||
def is_fp8(x: torch.dtype | torch.Tensor) -> bool:
|
||||
if isinstance(x, torch.Tensor):
|
||||
|
||||
Reference in New Issue
Block a user