MOVZX and Hinata
2066d3143f
[Mini-VLLM] Cut non-OpenAI surfaces, dead kernels, Mistral stack
...
- Remove Anthropic, Cohere, SageMaker, decisions, structured-decisions,
rl, render-launcher and batch entrypoints; keep scale-out token routes
- Drop the Swagger static bundle and the /docs, /redoc routes
- Delete device-keyed tuning configs for fused_moe, quant utils, mamba
select-state and helion; no sm86 entry ever existed for any of them
- Remove ten unused rope variants plus the fbgemm_fp8 and fp_quant
quantizers and their config, metrics and loader plumbing
- Cut the CPU device build: csrc/cpu, cpu_extension.cmake, cpu
requirements and the setup.py cpu target path
- Delete 29 environment variables nothing reads and 14 chat-template
fallbacks for models this fork no longer registers
- Remove nine dead CUDA ops (deepseek_v4, kimi_k3, minimax, muse_spark,
dsv3 gemm) with their kernels, bindings, wrappers and CMake entries
- Delete the connector test suites that targeted removed connectors
- Cut the Mistral tokenizer and parser stack, its tests, and the
mistral_common and ijson dependency pins
- Restore the Qwen4Exp registry rows and nvidia/mtp.py that an earlier
cut deleted while keeping the model tree
Co-authored-by: Hinata <hinata@idihore.id >
Signed-off-by: MOVZX <movzx@yahoo.com >
2026-10-19 17:10:56 +07:00
MOVZX and Hinata
f36850e389
[Mini-VLLM] Restore FlashInfer backend for FP8 KV & only probe quantization overrides that exist
...
FlashInfer is the only backend serving fp8_e4m3 KV cache on compute
capability 8.6; the SM86 cut removed it and the Triton fallback
refuses FP8 KV, crashing worker startup. Restore flashinfer.py and
the triton_flashinfer shim.
The SM86 cut removed the inc and deepseek_v4_fp8 loaders but
ModelConfig still probed them, crashing compressed-tensors
checkpoints with "Invalid quantization method: inc". Filter the
override probe list to the supported methods.
Co-authored-by: Hinata <hinata@idihore.id >
Signed-off-by: MOVZX <movzx@yahoo.com >
2026-10-18 23:31:30 +07:00
MOVZX and Hinata
e61dcb9143
[Mini-vLLM] SM86-only cut: models, backends, Rust, tests
...
- Keep only Qwen3.5, Qwen3.5-MTP and Qwen4Exp model paths; trim the model registry to 41 entries
- Delete idle attention backends (MLA, FlashInfer, ROCm-only, quant backends) and dead GDN CuTeDSL kernels
- Remove inc/quark quant loaders; keep compressed-tensors and Marlin
- Drop the Rust frontend, pooling entrypoints, and idle test suites
- Rebuild kernels for sm_80;sm_86 and strip the extensions
- Verified: full import sweep clean (optional deps only) and a dummy-load Qwen3.5 engine run on RTX 3090 via the Triton GDN path
Co-authored-by: Hinata <hinata@idihore.id >
Signed-off-by: MOVZX <movzx@yahoo.com >
2026-10-18 19:19:56 +07:00
b544587408
[Core][Feat] Pluggable KVCacheConfigBuilder for platform/model-specific KV cache planning ( #53558 )
...
Signed-off-by: wangxiyuan <wangxiyuan1007@gmail.com >
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai >
2026-10-16 23:16:47 +00:00
maithilijoshi20 and Cyrus Leung
92dc3dd393
[Bugfix] Honor OPT token embedding width for prompt embeddings ( #61736 )
...
Signed-off-by: maithilijoshi20 <maithilij2003@gmail.com >
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk >
2026-10-16 04:41:50 +00:00
Andreas Karatzas and Claude
29345f42a6
[ROCm][Qwen4Exp] Support HC and MoE sequence parallelism on ROCm ( #61471 )
...
Signed-off-by: Andreas Karatzas <akaratza@amd.com >
Co-authored-by: Claude <noreply@anthropic.com >
2026-10-15 20:11:17 -05:00
2cc2e68d1c
[Core][Mamba] Remove leftovers of mamba_cache_mode="all" ( #60043 )
...
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com >
Signed-off-by: Jiangyun Zhu <zhujiangyun@inferact.ai >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-10-13 00:58:59 -07:00
Woosuk Kwon and Claude Opus 5.5
e1a5086256
[Attention] Remove cascade attention ( #61126 )
...
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-12 10:41:07 -07:00
Harry Huang and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
c75cb63cd2
[Qwen4Exp] support Sequence Parallelism for Qwen3.8-Flash-Next ( #56322 )
...
Signed-off-by: huanghaoyan.hhy <huanghaoyan.hhy@alibaba-inc.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-10-12 20:56:44 +08:00
aoshen02 and Codex
239a3fa3a8
[Sleep] Retain draft-private weights with existing parameter selector ( #61046 )
...
Signed-off-by: aoshen02 <aoshen@inferact.ai >
Co-authored-by: Codex <noreply@openai.com >
2026-10-11 23:30:56 +08:00
SeongJun Lee and lesj0610
a98247ab4d
[Model] Drop the 4x M-RoPE cache headroom for Qwen3-VL, Qwen3.5 and Qwen3.8 ( #59853 )
...
Signed-off-by: lesj0610 <lesj0610@users.noreply.github.com >
Signed-off-by: lesj0610 <lesj0610@gmail.com >
Co-authored-by: lesj0610 <lesj0610@users.noreply.github.com >
2026-10-10 06:51:52 +00:00
34051ad714
feat: Add video support for the Transformers backend ( #57441 )
...
pre-commit / pre-run-check (push) Canceled after 0s
pre-commit / pre-commit (push) Canceled after 0s
Signed-off-by: Harshal Janjani <harshaljanjani@gmail.com >
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
2026-10-05 00:51:15 -07:00
aoshen02 and Claude Opus 5.5
5e56e9af2a
[Feature] Release the CUDA graph pool on sleep ( #59160 )
...
Signed-off-by: aoshen02 <aoshen@inferact.ai >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-04 01:13:10 +00:00
Harry Mellor
9e0b6fcf77
[CI] Bump Transformers version to 5.18.0 ( #59621 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
2026-10-04 00:42:35 +00:00
Stefano Castagnetta and Claude Opus 5
f0b4b36890
[Bugfix][Quantization] Detect NVFP4 in ModelOpt mixed-precision checkpoints ( #56050 )
...
Signed-off-by: Stefano Castagnetta <scastagnetta@nvidia.com >
Co-authored-by: Claude Opus 5 <noreply@anthropic.com >
2026-10-02 12:30:22 +00:00
Harry Mellor and Claude Opus 5
e3cae8d2ac
[Model] Use upstream GLM-5.3 and Qwen4-Exp configs and processor ( #57387 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
2026-10-02 02:25:18 +00:00
ae66f9a97c
[Hardware][PowerPC] Prioritize bfloat16 for auto dtype on PowerPC ( #58528 )
...
Signed-off-by: Akash kaothalkar <akash.kaothalkar@ibm.com >
Co-authored-by: Akash kaothalkar <akash.kaothalkar@ibm.com >
Co-authored-by: Antigravity <antigravity@google.com >
Co-authored-by: Li, Jiang <jiang1.li@intel.com >
2026-09-30 17:14:17 +08:00
Harry Mellor and Claude Opus 5.5
d636456b1f
[Frontend] Remove the slow tokenizer mode ( #58545 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com >
2026-09-24 08:48:28 -07:00
344fcc252a
[Bugfix] Resolve the Hub revision once per repo ( #56092 )
...
Signed-off-by: Wauplin <lucainp@gmail.com >
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-09-24 08:48:24 -07:00
a7d203efe5
fix(config): apply presence_penalty/frequency_penalty from override-generation-config ( #50769 )
...
Signed-off-by: Chenglun Hu <chenglunhu@gmail.com >
Signed-off-by: hclsys <chenglunhu@gmail.com >
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com >
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
2026-09-24 08:48:18 -07:00
Harry Mellor and Claude Opus 5.5
d881812500
[Chore] Use Transformers v5 names and drop redundant processor use_fast ( #58550 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com >
2026-09-24 14:14:10 +00:00
826e300cde
[Bugfix] Skip VllmConfig re-validation for with_hf_config submodel views ( #58212 )
...
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Kimi Code <noreply@moonshot.ai >
Co-authored-by: Roger Wang <rogerw@inferact.ai >
2026-09-23 00:50:17 +00:00
639461eda4
[RL][Sleep] Retain frozen weights across level-2 sleep ( #57891 )
...
Signed-off-by: aoshen02 <aoshen02@users.noreply.github.com >
Co-authored-by: aoshen02 <aoshen02@users.noreply.github.com >
Co-authored-by: Codex <noreply@openai.com >
2026-09-22 12:36:55 +08:00
Cyrus Leung
c8824a1e42
[MM] Move supports_multimodal_inputs and cache out of registry ( #57913 )
...
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk >
2026-09-21 07:43:07 -07:00
0b7f11a1ee
[AuxOutput] Add block-keyed storage for routed-expert outputs ( #45635 )
...
Signed-off-by: aoshen02 <aoshen@inferact.ai >
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: aoshen02 <aoshen@inferact.ai >
Co-authored-by: OpenAI Codex <codex@openai.com >
Co-authored-by: Roger Wang <hey@rogerw.io >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Codex <noreply@openai.com >
2026-09-21 10:25:05 +00:00
Harry Mellor and Claude Opus 5
651a88c09a
[CI] Ignore ruff D209, rejoin the docstrings it split, and silence incompatible-rule warnings ( #57212 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
2026-09-16 10:57:24 -07:00
Harry Mellor and Claude Opus 5
8c1557a79c
Add pydocstyle to the ruff rules ( #52136 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Signed-off-by: Harry Mellor <hej.mellor@gmail.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
2026-09-16 16:44:29 +00:00
Harry Mellor and Claude Opus 5
c191787a68
[Bugfix] Align vLLM YaRN with Transformers and stop re-scaling max_model_len ( #56446 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
2026-09-11 13:01:38 +00:00
Bugen Zhao and Isotr0py
b47b01cf33
[Model][Frontend] Support DeepSeek-V4.1-Flash in Rust and Python frontends ( #56208 )
...
Signed-off-by: Bugen Zhao <i@bugenzhao.com >
Co-authored-by: Isotr0py <Isotr0py@outlook.com >
2026-09-10 01:15:06 -07:00
Harry Mellor and Claude Opus 5
719284fe15
[Core][Model] Unify XD-RoPE into M-RoPE and derive the channel count ( #56078 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
2026-09-09 09:23:30 -07:00
Andreas Karatzas
18615ad1ee
[Bugfix] Handle null RoPE parameters for NoPE layers ( #55949 )
...
Signed-off-by: Andreas Karatzas <akaratza@amd.com >
2026-09-08 20:20:05 +00:00
Anshul Desai
da55494a12
docs: note that enforce_eager also disables torch.compile ( #55271 )
...
Signed-off-by: Anshul Desai <36732940+AnshulDesai@users.noreply.github.com >
2026-09-03 21:25:30 -07:00
Isotr0py
1356635d83
[New model][Multimodal] Add DeepSeek-V4-Flash-Vision-Exp support ( #54566 )
...
Signed-off-by: Isotr0py <Isotr0py@outlook.com >
2026-09-02 11:09:09 -07:00
4ac452ad98
[Core] Release NCCL communicator memory in sleep mode ( #51485 )
...
Signed-off-by: aoshen02 <aoshen02@users.noreply.github.com >
Co-authored-by: aoshen02 <aoshen02@users.noreply.github.com >
Co-authored-by: Claude Fable 5 <noreply@anthropic.com >
2026-09-01 14:31:16 +08:00
drakosha and Claude
9bf0d4717e
[Bugfix] Keep grid dims for XD-RoPE models on a prefix-cache hit ( #53456 )
...
Signed-off-by: Mikhail Kostryukov <mike@triptrack.net >
Co-authored-by: Claude <noreply@anthropic.com >
2026-08-25 01:14:32 +00:00
Harry Mellor
f0c14b4f77
Fix weight tying ( #51665 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
2026-08-20 10:31:55 +00:00
a1c5b1fd9f
[Core][V1] Support trace_decode_token_ids for deterministic decode replay ( #46701 )
...
Signed-off-by: ianzxliu <ianzxliu@tencent.com >
Signed-off-by: aoshen02 <aoshen@inferact.ai >
Signed-off-by: Zhixia Liu <liuzhixia.nju@gmail.com >
Co-authored-by: ianzxliu <ianzxliu@tencent.com >
Co-authored-by: Claude <noreply@anthropic.com >
Co-authored-by: aoshen02 <aoshen@inferact.ai >
Co-authored-by: Codex <noreply@openai.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
2026-08-20 15:12:43 +08:00
Egor and Claude
e9e1630e93
[Model] Support bidirectional (encoder-only) attention for DeepSeek e… ( #52948 )
...
Signed-off-by: Egor <erop001@gmail.com >
Signed-off-by: Egor <47443236+Lossfull@users.noreply.github.com >
Co-authored-by: Claude <noreply@anthropic.com >
2026-08-19 16:36:27 +00:00
Michał Ganczarenko and Wentao Ye
6948a43fbb
[Bugfix] Detect all attention-spelling variants in ModelConfig.is_hybrid ( #52161 )
...
Signed-off-by: Michal Ganczarenko <michal.ganczarenko@intel.com >
Signed-off-by: Michał Ganczarenko <michal.ganczarenko@intel.com >
Co-authored-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com >
2026-08-18 14:43:51 -04:00
Canlin Guo and Nick Hill
455edc022b
[ModelRunnerV2] Support prompt embeds ( #42963 )
...
Signed-off-by: gcanlin <canlinguosdu@gmail.com >
Signed-off-by: Canlin Guo <961750412@qq.com >
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
2026-08-17 21:58:50 +00:00
wangxiyuan and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
cdc4824a21
[Misc] Remove override_attention_dtype ( #48684 )
...
Signed-off-by: wangxiyuan <wangxiyuan1007@gmail.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-08-14 11:15:43 -04:00
50ba4bc6b2
[Feature] Mask Replay ( #49577 )
...
Signed-off-by: vx120 <893600387@qq.com >
Signed-off-by: vx120 <57470515+vx120@users.noreply.github.com >
Signed-off-by: aoshen02 <aoshen@inferact.ai >
Co-authored-by: aoshen02 <aoshen@inferact.ai >
Co-authored-by: Codex <noreply@openai.com >
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
2026-08-13 08:44:31 +00:00
Clinton Thomas and Lucas Bourtoule
90fd4a333f
Preserve revision pins in secondary artifact loaders ( #51446 )
...
Signed-off-by: Clinton Thomas <1033162+KernelClint@users.noreply.github.com >
Co-authored-by: Lucas Bourtoule <35483370+dhalf@users.noreply.github.com >
2026-08-11 04:33:10 +00:00
Harry Mellor
70b84f0bcb
Fix Gemma 4 for upcoming Transformers version ( #49797 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
2026-08-10 07:48:26 +00:00
Isotr0py
073c510c91
[Migration] Migrate bitsandbytes support to OOT plugin ( #43529 )
...
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn >
Signed-off-by: Isotr0py <Isotr0py@outlook.com >
2026-08-09 10:27:31 +08:00
music-dino and Harry Mellor
a231c5ceac
[Bugfix] Skip fetching revision for model when model and weights_model are different ( #51260 )
...
Signed-off-by: Dino Music <Dino.Music@amd.com >
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
2026-08-07 12:55:50 +00:00
wang.yuqi and Cyrus Leung
2fa490470d
[Model] Fused mm preprocess normalisation on the Device ( #50411 )
...
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io >
Signed-off-by: wang.yuqi <noooop@126.com >
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com >
2026-08-06 03:11:14 -07:00
Tianyu Guo
b30291cf0c
[EPD] Remove duplicate image preprocessing in EPD and enable preprocess on GPU ( #50390 )
...
Signed-off-by: Tianyu Guo <guoty@inferact.ai >
2026-08-05 15:59:54 -07:00
397094da17
Resolve revision to commit_hash once per model load, via huggingface_hub's resolve_revision ( #49990 )
...
Signed-off-by: Wauplin <lucainp@gmail.com >
Signed-off-by: Lucain <lucainp@gmail.com >
Signed-off-by: Lucain Pouget <lucain@huggingface.co >
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
2026-08-05 13:49:56 +00:00
aoshen02 and yihengz
6153dbe036
[R3] Unify routed expert shape configuration ( #50940 )
...
Signed-off-by: aoshen02 <aoshen@inferact.ai >
Co-authored-by: yihengz <yihengz@nvidia.com >
2026-08-04 20:20:41 -07:00