[Model] Add Inkling compressed-tensors dynamic FP8 support (#48876)
Signed-off-by: mgoin <mgoin64@gmail.com> Co-authored-by: mgoin <mgoin64@gmail.com> Co-authored-by: OpenAI Codex <noreply@openai.com>
This commit is contained in:
co-authored by
mgoin
OpenAI Codex
parent
54ab69b14e
commit
17a74b745b
@@ -589,6 +589,9 @@ class InklingMoE(nn.Module):
|
||||
param.data[lids] = vals.reshape(len(gids), *param.shape[1:]).to(
|
||||
param.device
|
||||
)
|
||||
elif key == "w2_weight_scale" and weight.shape[-1] == 1:
|
||||
# Per-output-channel scales are replicated across TP ranks.
|
||||
param.data[lids] = weight[gids].to(device=param.device, dtype=param.dtype)
|
||||
elif key.startswith("w13"):
|
||||
# Checkpoint w13 rows are interleaved [g0, u0, g1, u1, ...]; the
|
||||
# fused param layout is [w1(gate); w3(up)]. The TP-local rows form
|
||||
|
||||
Reference in New Issue
Block a user