MOVZX and Hinata
e61dcb9143
[Mini-vLLM] SM86-only cut: models, backends, Rust, tests
...
- Keep only Qwen3.5, Qwen3.5-MTP and Qwen4Exp model paths; trim the model registry to 41 entries
- Delete idle attention backends (MLA, FlashInfer, ROCm-only, quant backends) and dead GDN CuTeDSL kernels
- Remove inc/quark quant loaders; keep compressed-tensors and Marlin
- Drop the Rust frontend, pooling entrypoints, and idle test suites
- Rebuild kernels for sm_80;sm_86 and strip the extensions
- Verified: full import sweep clean (optional deps only) and a dummy-load Qwen3.5 engine run on RTX 3090 via the Triton GDN path
Co-authored-by: Hinata <hinata@idihore.id >
Signed-off-by: MOVZX <movzx@yahoo.com >
2026-10-18 19:19:56 +07:00
Haiyu Zuo and Kevin H. Luu
ff5a91a456
[Feature][Model Loader] Apply model-owned MTP shard selection to RunAI ModelStreamer ( #60759 )
...
Signed-off-by: 浩韵 <zuohaiyu.zhy@alibaba-inc.com >
Co-authored-by: Kevin H. Luu <khluu000@gmail.com >
2026-10-15 12:23:11 +08:00
amd-xavierwang and JartX
06265beb25
[ROCm] Enable fp8 query for Triton attention ( #60296 )
...
Signed-off-by: xawang <xawang@amd.com >
Co-authored-by: JartX <sagformas@epdcenter.es >
2026-10-14 15:03:20 -04:00
13dbf96093
[Model Loader] Apply EP weight filtering in InstantTensor ( #61167 )
...
Signed-off-by: Itay Etelis <92247226+Etelis@users.noreply.github.com >
Co-authored-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com >
Co-authored-by: Misha Goin <mgoin64@gmail.com >
2026-10-14 09:10:10 -07:00
Ronald and aoshen02
19aba3e4d5
[Bugfix][RL] Load attention sinks through weight_loader for layerwise reload ( #57797 )
...
Signed-off-by: Ronald1995 <ronaldautomobile@163.com >
Co-authored-by: aoshen02 <aoshen@inferact.ai >
2026-10-14 10:17:58 +00:00
189bbfb9f4
Add MoNe experts pruning support ( #50415 )
...
Signed-off-by: Ilya Boytsov <ieboytsov@nebius.com >
Signed-off-by: Misha Goin <mgoin64@gmail.com >
Signed-off-by: mgoin <mgoin64@gmail.com >
Co-authored-by: Ilya Boytsov <ieboytsov@nebius.com >
Co-authored-by: Codex <noreply@openai.com >
Co-authored-by: Misha Goin <mgoin64@gmail.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-12 23:18:43 +00:00
092c35dd15
[Model] Support Cloudflare/clef-omni on /v1/systemone ( #61064 )
...
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com >
Signed-off-by: Isotr0py <Isotr0py@outlook.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: Isotr0py <Isotr0py@outlook.com >
2026-10-12 21:27:34 +08:00
f2376cdc4d
[Fast Start] Preload the FlashInfer autotune table on the weight cache daemon ( #60085 )
...
Signed-off-by: liusy58 <mg21330037@smail.nju.edu.cn >
Signed-off-by: Isotr0py <Isotr0py@outlook.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
Co-authored-by: Isotr0py <Isotr0py@outlook.com >
2026-10-10 15:48:06 +00:00
02aa10539f
[Bugfix][ModelLoader] Make enable_weights_track check quantized layers ( #57108 )
...
Signed-off-by: siliangchen-amd <SiLiang.Chen@amd.com >
Co-authored-by: Cursor <cursoragent@cursor.com >
Co-authored-by: Shanshan Shen <467638484@qq.com >
2026-10-10 10:44:06 +08:00
Ning Xie and Simon Mo
a0f28d73a8
[sharded state loader] support pp in sharded state loader ( #52110 )
...
Signed-off-by: Andy Xie <andy.xning@gmail.com >
Co-authored-by: Simon Mo <simon.mo@hey.com >
2026-10-09 11:08:31 -07:00
9f3d70e081
[Quantization] Enable shared expert fusion compatibility with online shared_expert quantization (showcase: along Quark MXFP4 routed experts) ( #55686 )
...
Signed-off-by: Felix Marty <Felix.Marty@amd.com >
Signed-off-by: fxmarty-amd <felmarty@amd.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
Co-authored-by: Misha Goin <mgoin64@gmail.com >
2026-10-08 12:04:46 -04:00
Harry Mellor and Claude Opus 5.5
21d9758459
Remove Python 3.10 support now that it is EOL ( #60402 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-08 02:30:52 -07:00
Matt and Claude Opus 5.5
9dba339d07
[Bugfix][ROCm] Keep MLA derived weights in place on weight reload ( #60389 )
...
Signed-off-by: Matthew Wong <Matthew.Wong2@amd.com >
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com >
2026-10-07 20:38:06 -05:00
f38c6778ce
[quantization] Add per-token NVFP4 quantization MoE support for ReLU2 ( #56740 )
...
Signed-off-by: Xuanteng Huang <xuantengh@nvidia.com >
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com >
Co-authored-by: OpenAI Codex <noreply@openai.com >
2026-10-07 22:01:09 +00:00
e572e02d21
[UX] Skip checkpoint shards an MTP head does not load ( #60152 )
...
Signed-off-by: Robert Shaw <robshaw@redhat.com >
Co-authored-by: Robert Shaw <robshaw@redhat.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-07 15:16:19 -04:00
c83935b300
[Bugfix][Model Loader] Support reload_weights with runai_streamer load format ( #58124 )
...
Signed-off-by: Viktor Koukouliev <vkoukouliev@nvidia.com >
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com >
Co-authored-by: Isotr0py <Isotr0py@outlook.com >
2026-10-07 14:04:35 +00:00
Harry Mellor and Claude Opus 5.5
fc2c801ad9
Disable mypy arg-type and assignment checks in tests ( #59428 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-09-30 15:18:50 +00:00
42d9f8ccdb
support rl feature : weight checker ( #51350 )
...
Signed-off-by: shiyuan680 <yangcheng104@huawei.com >
Signed-off-by: Ronald1995 <ronaldautomobile@163.com >
Signed-off-by: aoshen02 <aoshen02@users.noreply.github.com >
Co-authored-by: shiyuan680 <yangcheng104@huawei.com >
Co-authored-by: Ronald1995 <ronaldautomobile@163.com >
Co-authored-by: DeepSeek Harness <noreply@deepseek.com >
Co-authored-by: aoshen02 <aoshen02@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: aoshen02 <aoshen@inferact.ai >
2026-09-30 03:13:43 +00:00
f5ed3d9c03
[Fast Start] Charge daemon-held weights against gpu_memory_utilization ( #57298 )
...
Signed-off-by: liusy58 <mg21330037@smail.nju.edu.cn >
Signed-off-by: Isotr0py <Isotr0py@outlook.com >
Signed-off-by: Xun Sun <UNIDY2002@outlook.com >
Co-authored-by: Isotr0py <Isotr0py@outlook.com >
Co-authored-by: Xun Sun <UNIDY2002@outlook.com >
2026-09-29 17:58:35 +00:00
af5b4857e1
[Fast Start] Support PP ( #55477 )
...
Signed-off-by: Isotr0py <Isotr0py@outlook.com >
Co-authored-by: Isotr0py <Isotr0py@outlook.com >
Co-authored-by: Xun Sun <UNIDY2002@outlook.com >
2026-09-29 09:37:43 +00:00
Xun Sun and Isotr0py
e7900156e1
[Fast Start] Add /health endpoint for the weight cache daemon ( #58552 )
...
Signed-off-by: Xun Sun <UNIDY2002@outlook.com >
Co-authored-by: Isotr0py <Isotr0py@outlook.com >
2026-09-27 17:48:54 +00:00
Isotr0py
24928a4d62
[UX][Frontend] Introduce vllm preload cli for fast restart ( #56680 )
...
Signed-off-by: Isotr0py <Isotr0py@outlook.com >
2026-09-25 06:56:39 +00:00
90a9515006
[Cleanup] Remove online quantization support in fp8.py in favor of online shorthands ( #53585 )
...
Signed-off-by: Felix Marty <Felix.Marty@amd.com >
Signed-off-by: mgoin <mgoin64@gmail.com >
Co-authored-by: mgoin <mgoin64@gmail.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-09-24 12:48:14 -07:00
Xun Sun
e773247f8a
[Fast Start] Wait for weight cache daemon readiness ( #58370 )
2026-09-24 19:18:04 +00:00
1291bdb85e
[EPD][Model Loader] Skip language-model checkpoint shards for --mm-encoder-only ( #58086 )
...
Signed-off-by: grYe99 <guorongye99@gmail.com >
Co-authored-by: grYe99 <guorongye99@gmail.com >
Co-authored-by: Cursor Agent <cursoragent@cursor.com >
2026-09-24 00:41:15 -07:00
siyu and Isotr0py
c024cfdb02
[Fast Start] Support data parallelism in the weight cache daemon ( #57386 )
...
Signed-off-by: liusy58 <mg21330037@smail.nju.edu.cn >
Signed-off-by: Isotr0py <Isotr0py@outlook.com >
Co-authored-by: Isotr0py <Isotr0py@outlook.com >
2026-09-22 18:19:40 +00:00
siyu and Isotr0py
c723a831a8
[Fast Start] Cache the MTP draft model in a separate daemon group ( #57312 )
...
Signed-off-by: liusy58 <mg21330037@smail.nju.edu.cn >
Signed-off-by: Isotr0py <Isotr0py@outlook.com >
Co-authored-by: Isotr0py <Isotr0py@outlook.com >
2026-09-22 03:13:38 +00:00
Misha Goin and ErinYin
e5fce7b56b
[Core][Kernel] Share persistent workspaces for Marlin and Humming ( #57421 )
...
Signed-off-by: mgoin <mgoin64@gmail.com >
Co-authored-by: ErinYin <liang.yin@daocloud.io >
2026-09-20 17:28:33 +08:00
b2f2bd71ba
[CI][Bugfix] Add tp_shard_with_padding to padded MoE reload test mock ( #57402 )
...
Signed-off-by: kimi-no-na-wa <kimi-no-na-wa@users.noreply.github.com >
Co-authored-by: kimi-no-na-wa <kimi-no-na-wa@users.noreply.github.com >
Co-authored-by: Misha Goin <mgoin64@gmail.com >
2026-09-17 13:02:24 -04:00
78fdf4efb8
[Bugfix][MoE] Convert FlashInfer BF16 weights in place ( #54699 )
...
Signed-off-by: Yuchen Wang <93700456+yuchenwang3@users.noreply.github.com >
Signed-off-by: Dakai An <dakaian108@gmail.com >
Co-authored-by: Dakai An <dakaian108@gmail.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
Co-authored-by: aoshen02 <aoshen@inferact.ai >
2026-09-17 22:00:04 +08:00
Harry Mellor and Claude Opus 5
651a88c09a
[CI] Ignore ruff D209, rejoin the docstrings it split, and silence incompatible-rule warnings ( #57212 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
2026-09-16 10:57:24 -07:00
Harry Mellor and Claude Opus 5
8c1557a79c
Add pydocstyle to the ruff rules ( #52136 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Signed-off-by: Harry Mellor <hej.mellor@gmail.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
2026-09-16 16:44:29 +00:00
aoshen02
eb61707851
[Bugfix] Fix stale HPC QK-norm weights after weight refit ( #56706 )
...
Signed-off-by: aoshen02 <aoshen02@users.noreply.github.com >
2026-09-14 21:34:41 -07:00
Isotr0py
dbf49dad11
[Fast Start] Use GPU uuid as socket folder identifier ( #56669 )
...
Signed-off-by: Isotr0py <Isotr0py@outlook.com >
2026-09-14 06:28:13 +00:00
siyu and Isotr0py
73fb19151f
[Fast Start] Support fp4 ( #55465 )
...
Signed-off-by: Isotr0py <Isotr0py@outlook.com >
Co-authored-by: Isotr0py <Isotr0py@outlook.com >
2026-09-10 13:22:17 +00:00
Qiming Zhang
49632fb3ac
[CI][XPU] Reject CUDA-IPC weight cache on non-CUDA/ROCm platforms ( #56010 )
...
Signed-off-by: mayuyuace <qiming1.zhang@intel.com >
2026-09-10 14:05:07 +08:00
Roderick Wu and mgoin
2c9d68f80b
[Quant][Kernel] Remove GPTQ Group/Dynamic Activation Ordering ( #54809 )
...
Signed-off-by: Roderick-Wu <roderickwu2003@gmail.com >
Signed-off-by: Roderick Wu <roderickwu2003@gmail.com >
Signed-off-by: mgoin <mgoin64@gmail.com >
Co-authored-by: mgoin <mgoin64@gmail.com >
2026-09-08 12:45:12 -07:00
a69e75b9b6
Fast Start ( #54921 )
...
Signed-off-by: Isotr0py <Isotr0py@outlook.com >
Co-authored-by: liusy58 <liusy58@smail.nju.edu.cn >
Co-authored-by: Isotr0py <Isotr0py@outlook.com >
2026-09-04 06:32:06 -07:00
Stellar鱼
5182f2705b
[CI/Build] Use file rendezvous for UniProc loader fixtures ( #52367 )
...
Signed-off-by: yu-xin-c <2182712990@qq.com >
2026-08-29 11:53:04 +08:00
Wyett
b26039b09f
Add routed expert loading for gpt-oss ( #52209 )
...
Signed-off-by: Wyett <wyettzeng@gmail.com >
Signed-off-by: wyett <wyettzeng@gmail.com >
2026-08-23 17:59:08 +08:00
Shuolei Wang and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
236f78cc5c
[Core][RL] Support sparse checkpoint updates through native weight loaders ( #50723 )
...
Signed-off-by: Shuolei Wang <shuoleiwang123@gmail.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-08-22 17:39:58 +00:00
Harry Mellor
f0c14b4f77
Fix weight tying ( #51665 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
2026-08-20 10:31:55 +00:00
aoshen02 and Codex
f2906b492e
[Bugfix] Avoid repeated layerwise reload warning scans ( #49505 )
...
Signed-off-by: aoshen02 <aoshen@inferact.ai >
Co-authored-by: Codex <noreply@openai.com >
2026-08-12 13:53:50 +08:00
Richard Zou
cb30f6f8dd
[Testing] Fix test_sharded_state_loader ( #51736 )
...
Signed-off-by: Richard Zou <zou3519@gmail.com >
2026-08-11 21:16:09 +00:00
aoshen02 and Roger Wang
99e62b802c
[Bugfix][Model Loader] Defer post-load attention weight processing ( #49519 )
...
Signed-off-by: aoshen02 <aoshen@inferact.ai >
Co-authored-by: Roger Wang <hey@rogerw.io >
2026-08-10 19:16:57 -07:00
aoshen02 and aoshen02
7153fd70f6
[Bugfix][MoE] Filter packed expert weights during EP loading ( #49558 )
...
Signed-off-by: aoshen02 <aoshen02@users.noreply.github.com >
Signed-off-by: aoshen02 <aoshen@inferact.ai >
Co-authored-by: aoshen02 <aoshen02@users.noreply.github.com >
2026-08-04 16:32:03 +00:00
Ryan Clark and Claude Fable 5
68fb303a41
[Bugfix] Preserve Marlin runtime tensor storage across weight reload ( #48438 )
...
Signed-off-by: Ryan Clark <ryanclark2k@gmail.com >
Co-authored-by: Claude Fable 5 <noreply@anthropic.com >
2026-07-30 14:50:03 -07:00
Andreas Karatzas and OpenAI Codex
30e333ca5f
[Bugfix] Shut down private Tensorizer engines ( #49840 )
...
Signed-off-by: Andreas Karatzas <akaratza@amd.com >
Co-authored-by: OpenAI Codex <noreply@openai.com >
2026-07-30 12:33:22 -07:00
Harry Mellor and Claude Opus 5
9321aff536
[Bugfix] Wait for the linear bias before layerwise online processing ( #49805 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
2026-07-25 17:56:32 +00:00
Harry Mellor
9a50464698
[CI] Stop flaky test from downloading model every time ( #49800 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
2026-07-25 13:29:36 +00:00
MOVZXandHinata
