[Feature][Model Loader] Apply model-owned MTP shard selection to RunAI ModelStreamer (#60759)

Signed-off-by: 浩韵 <zuohaiyu.zhy@alibaba-inc.com>
Co-authored-by: Kevin H. Luu <khluu000@gmail.com>
This commit is contained in:
Haiyu Zuo
2026-10-15 12:23:11 +08:00
committed by GitHub
co-authored by Kevin H. Luu
parent 833665cded
commit ff5a91a456
6 changed files with 724 additions and 13 deletions
@@ -51,6 +51,20 @@ vllm serve s3://core-llm/Llama-3-8b \
--load-format runai_streamer
```
## Model-specific checkpoint selection
For models exposing `is_unused_checkpoint_weight`, `runai_streamer` skips
safetensors files containing only unused weights before streaming. It uses the
same model-owned rule as the default loader, including MTP draft models with
this hook; it does not assume a particular MTP tensor prefix.
Object-storage checkpoints use `model.safetensors.index.json` and verify excluded
shards against their safetensors headers before streaming. Mixed shards remain
included. Missing or inconsistent metadata retains the original file list.
Models without a selection rule keep all files. Distributed streaming with
pipeline parallelism also retains the full file list. This optimization does
not change `runai_streamer_sharded`.
## Tunable parameters
You can tune parameters using `--model-loader-extra-config`: