iHore/vLLM
8
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity
22,547 Commits 3 Branches 0 Tags
eb68fb9105eab33888ce252001a269a8c52c19a5
Commit Graph
173 Commits
Author SHA1 Message Date
MOVZX 2877121b2d Merge remote-tracking branch 'upstream/main' into patches 
Signed-off-by: MOVZX <movzx@yahoo.com>
 2026-10-03 21:59:03 +07:00
Harry MellorandClaude Opus 5 e3cae8d2ac [Model] Use upstream GLM-5.3 and Qwen4-Exp configs and processor (#57387) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
 2026-10-02 02:25:18 +00:00
Jiangyun ZhuClaude Opus 5.5Thomas Parnell
 		cb6458c5ce [Core] Remove deprecated mamba_cache_mode "all" (#58997) 
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Thomas Parnell <tpa@zurich.ibm.com>
 2026-10-01 02:59:37 -07:00
vx120Cuiping Chengaoshen02Claude Opus 5.5
 		70be6aae19 [Feature][Spec Decode] Support sampling mask replay for MRV2 MTP (#59359) 
Signed-off-by: Cuiping Cheng <chengcuipingswu@163.com>
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: Cuiping Cheng <chengcuipingswu@163.com>
Co-authored-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-01 13:51:39 +08:00
Andrii SkliarandAndrii Skliar deb148e730 [Mamba] Add FlashInfer ReplaySSM support for MTP (#52928) 
Signed-off-by: Andrii Skliar <askliar@nvidia.com>
Co-authored-by: Andrii Skliar <askliar@nvidia.com>
 2026-09-30 21:41:17 +00:00
Raphaël RiallandSimon VeitnerLucas Wilkinson
 		0f8b398158 [watermarking] add context deduplication support to speculative decoding (#56807) 
Signed-off-by: Raphael Rialland <raphael.rialland@mistral.ai>
Signed-off-by: Simon Veitner <sveitner@redhat.com>
Co-authored-by: Simon Veitner <sveitner@redhat.com>
Co-authored-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com>
 2026-09-30 12:22:10 -04:00
Lucas WilkinsonMatthew BonanniClaude Opus 5.5
 		90e13fc757 [Bugfix][HiSparse] Never allocate GPU pages without host backing (#59036) 
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
Co-authored-by: Matthew Bonanni <mbonanni@redhat.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-09-30 04:13:29 +00:00
jackandQwertyJack 79a0307d74 [Config] Remove Engram CUDA-alike device restrictions (#59171) 
Signed-off-by: QwertyJack <7554089+QwertyJack@users.noreply.github.com>
Co-authored-by: QwertyJack <7554089+QwertyJack@users.noreply.github.com>
 2026-09-29 12:42:02 +00:00
MOVZXandHinata ae0350c0de [Spec Decode][MRV2] RecoverSSM for Qwen GDN and PLE short conv (PR #58863) 
Intake of open upstream PR vllm-project/vllm#58863. Active only with
VLLM_USE_V2_MODEL_RUNNER=1; inert otherwise.

Speculative decoding on Gated DeltaNet snapshotted the full recurrent
state after every draft position and reserved one Mamba block per draft
position. On a 24 GB card that is reserved VRAM we never use, plus
per-step copy traffic.

RecoverSSM, the protocol Kimi-K3 KDA already uses, replaces this:
verify runs from a read-only checkpoint and stores a small per-token
replay record [correction (V), normalized key (K), gate (1)] in fp32;
commit replays only the accepted tokens and writes the recurrent state
once, plus the block-boundary state in align mode. The PLE dilated
short conv compacts its accepted window with the same plan.

Conflict in vllm/models/qwen4_exp/nvidia/mtp.py: our get_top_tokens()
work put LocalArgmaxMixin on Qwen4ExpMTP, the PR adds SupportsReplaySSM
to the same base list. Kept both.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
 2026-09-29 13:39:09 +07:00
MOVZX a63dca4afb Merge remote-tracking branch 'upstream/main' into patches 
Signed-off-by: MOVZX <movzx@yahoo.com>
 2026-09-29 12:54:18 +07:00
Ren YuzhouandClaude ec5e0c352f [Bugfix][Engram] Keep THP tables private when resolving shared memory (#59068) 
Signed-off-by: Ren Yuzhou <54501155+yuzhouo7@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
 2026-09-29 05:19:49 +00:00
Tyler Michael SmithandCodex 4406874e7c [Bugfix][FT] Pass stateless process-group timeouts explicitly (#58611) 
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
Co-authored-by: Codex <noreply@openai.com>
 2026-09-29 03:14:02 +00:00
MOVZXandHinata 43fbd6510b [Qwen4Exp] Serve PLE rows from the loader's mapped checkpoint shards (PR #58815) 
Adds an opt-in backend, --engram-config '{"host_file_gather": true}', that
serves Qwen4Exp PLE rows from the checkpoint files through the page cache
instead of allocating the 47.7 GiB table on GPU or in pinned host memory.

- Qwen4ExpPLEFileGatherEmbedding allocates no table rows; load_weights keeps
  the default loader's zero-copy safetensors views of each shard. At bind
  (post-load hook) every view must be file-backed per /proc/self/maps (HF
  cache symlinks work; heap/pinned/anonymous memory is rejected) and the
  shards must cover the whole table.
- Each step, Qwen4ExpModelState.prepare_inputs computes the PLE n-gram ids,
  copies them to pinned host memory, syncs the stream once, issues
  POSIX_FADV_WILLNEED over the coalesced page ranges of the step's rows,
  gathers the rows on the host, and copies them into a fixed device staging
  buffer. The forward reads only the staging buffer, so FULL CUDA graph
  capture is unaffected.
- Rows stay in the checkpoint dtype (FP8 table stays FP8, dequantized on the
  GPU); dummy loads serve zeros; weight reload is rejected; takes precedence
  over cpu_offload; rejected on ROCm and with elastic EP.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
 2026-09-27 09:54:42 +07:00
Jürgen SchmiedandClaude a80e41d690 [Mamba] Share RecoverSSM commit kernels and helpers across models 
RecoverSSM's commit-plan and conv-window compaction kernels are model independent.
Move them from models/kimi_k3 into model_executor/layers/mamba/ops/recoverssm.py
(Kimi-K3 keeps its private names as import aliases; kernel source unchanged), add
mamba/recoverssm_utils.py (activation predicate, argument validation) and the
generic spec-row/request mapping helpers to v1/attention/backends/recoverssm_metadata.py.

CacheConfig.use_kda_recoverssm becomes use_recoverssm: the switch now selects RecoverSSM
for any model whose recurrent layers implement it, not only KDA.

No functional change for Kimi-K3.

Signed-off-by: Jürgen Schmied <juergenschmied70@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
 2026-09-27 00:07:04 +02:00
qizixiOpenAI CodexClaude Opus 5.5Nick Hill
 		a4eb3f25d6 [Spec Decode] Enable Gemma4 DSpark adaptive verification with FlashInfer (#57263) 
Signed-off-by: zixi-qi <zixi@inferact.ai>
Co-authored-by: OpenAI Codex <noreply@openai.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
 2026-09-26 17:01:17 +08:00
Andrii SkliarAndrii SkliarCodexBenjamin Chislett
 		73a78e6f1f [SpecDecode] Add LiLiCorr drafter (#57934) 
Signed-off-by: Andrii Skliar <askliar@nvidia.com>
Signed-off-by: Andrii Skliar <andreyws96@gmail.com>
Co-authored-by: Andrii Skliar <askliar@nvidia.com>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Benjamin Chislett <chislett.ben@gmail.com>
 2026-09-25 10:08:28 -04:00
Mathew Oddenopencode+deepseek-v4-flash+vllmAndreas KaratzasOpenAI Codex
 		267eee54bf [Qwen4Exp][ROCm] PLE n-gram table CPU offload (#57497) 
Signed-off-by: Mathew Odden <modden@redhat.com>
Signed-off-by: Andreas Karatzas <akaratza@amd.com>
Co-authored-by: opencode+deepseek-v4-flash+vllm <opencode+deepseek-v4-flash+vllm@example.com>
Co-authored-by: Andreas Karatzas <akaratza@amd.com>
Co-authored-by: OpenAI Codex <noreply@openai.com>
 2026-09-24 23:28:20 -05:00
stefankoncarevicandAndreas Karatzas e33de821c0 [ROCm][CI] skip the ROCm MRV1 default where MRV1 cannot serve the config (#58535) 
Signed-off-by: Stefan Koncarevic <Stefan.Koncarevic@amd.com>
Co-authored-by: Andreas Karatzas <akaratza@amd.com>
 2026-09-24 15:36:36 -05:00
LucainClaude Opus 5Harry Mellormergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 		344fcc252a [Bugfix] Resolve the Hub revision once per repo (#56092) 
Signed-off-by: Wauplin <lucainp@gmail.com>
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-09-24 08:48:24 -07:00
Harry MellorandClaude Opus 5.5 d881812500 [Chore] Use Transformers v5 names and drop redundant processor use_fast (#58550) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
 2026-09-24 14:14:10 +00:00
Lio Einaudi 711fc55c10 [Perf] Use breakable CUDA graphs (no torch.compile) by default under VLLM_BATCH_INVARIANT so the tuned matmul configs see the runtime M (#57586) 
Signed-off-by: LioEinaudi <zhao3024667639@gmail.com>
 2026-09-23 14:39:38 -04:00
Nick HillKimi CodeRoger Wang
 		826e300cde [Bugfix] Skip VllmConfig re-validation for with_hf_config submodel views (#58212) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Kimi Code <noreply@moonshot.ai>
Co-authored-by: Roger Wang <rogerw@inferact.ai>
 2026-09-23 00:50:17 +00:00
Artem PerevedentsevandCNE Pierre FICHEPOIL 1ea7c63f4a [Bugfix][Structured Output] Disallow MRV1 + PP>1 + async sched + structured output (#56250) 
Signed-off-by: Artem Perevedentsev <aperevedents@nvidia.com>
Co-authored-by: CNE Pierre FICHEPOIL <pierre-1.fichepoil@gendarmerie.interieur.gouv.fr>
 2026-09-22 18:00:05 +04:00
Jeff (Junze) Ma 0961bbae28 [Spec Decode] Enable async scheduling for DFlash (#58065) 2026-09-21 22:05:44 -07:00
Nicolò Lucchesi 54020c3c3e [Engram] Drop redundant VLLM_PLE_CPU_OFFLOAD env var (#57937) 
Signed-off-by: NickLucche <nicolo.lucchesi@mistral.ai>
 2026-09-21 20:16:02 +02:00
Tony Lin dc479a2d83 [Bugfix] Fix external LB DP rank handling when replicas share nodes (#53743) 
Signed-off-by: Tony Lin <tony.lin@intel.com>
 2026-09-21 17:19:14 +02:00
Juntian LiuClaude Fable 5.1zjy0516
 		76ffb0d374 [Model][Engram] Share host tables across co-located DP replicas by default (#57651) 
Signed-off-by: Juntian Liu <Juntianl777@gmail.com>
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: zjy0516 <riverclouds.zhu@qq.com>
 2026-09-21 08:26:56 +00:00
JohnQinAMD 6dfd87f596 [ROCm][DSv4.1] Keep the Engram tables in host memory on ROCm (#57491) 2026-09-21 10:17:11 +08:00
David CheungClaudeNick Hill
 		5302d1fe40 [Model Runner V2] Support custom logits processors (#56497) 
Signed-off-by: David Cheung <d7cheung@gmail.com>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
 2026-09-19 17:46:49 +00:00
Laura Lopez-RealLaura López Realmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 		a85dd55177 Fix Laguna patch mutating flat RoPE parameters (#57189) 
Signed-off-by: Laura López Real <laulopezreal@users.noreply.github.com>
Co-authored-by: Laura López Real <laulopezreal@users.noreply.github.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-09-17 12:22:57 +00:00
Matt 2d8a5c5741 [ROCm][Bugfix] Reduce CUDA graph divergences (#57229) 
Signed-off-by: Matthew Wong <Matthew.Wong2@amd.com>
 2026-09-17 05:12:13 +00:00
kyleliang-nvKyle LiangOpenAI Codex
 		c58ff535f4 [Bugfix][Spec Decode] Fix EAGLE and dense draft startup with EP (#56930) 
Signed-off-by: Kyle Liang <kyleliang-nv@users.noreply.github.com>
Co-authored-by: Kyle Liang <kyleliang-nv@users.noreply.github.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
 2026-09-16 17:03:36 -07:00
Harry MellorandClaude Opus 5 651a88c09a [CI] Ignore ruff D209, rejoin the docstrings it split, and silence incompatible-rule warnings (#57212) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
 2026-09-16 10:57:24 -07:00
Harry MellorandClaude Opus 5 8c1557a79c Add pydocstyle to the ruff rules (#52136) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Signed-off-by: Harry Mellor <hej.mellor@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
 2026-09-16 16:44:29 +00:00
Nicolò Lucchesi df42d112ee [Config] Infer HiSparse attention config from HiSparseConnector (#57041) 
Signed-off-by: NickLucche <nicolo.lucchesi@mistral.ai>
 2026-09-15 19:44:26 +00:00
JooHo LeeAndreas KaratzasCyrus Leung
 		78e84261ab [Performance] Add Triton kernel for Gemma3n sparse GELU (#48498) 
Signed-off-by: BWAAEEEK <jooho414@gmail.com>
Co-authored-by: Andreas Karatzas <akaratza@amd.com>
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>
 2026-09-14 07:42:15 +00:00
Juntian LiuRoger Wangywang96 <>Woosuk KwonCodex
 		d2d649e674 [DS V4.1][Engram] Support async prefetch for offloaded engram lookups and engram DP sharding (#56512) 
Signed-off-by: Juntian Liu <Juntianl777@gmail.com>
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
Co-authored-by: Roger Wang <hey@rogerw.io>
Co-authored-by: ywang96 <>
Co-authored-by: Woosuk Kwon <woosuk@inferact.ai>
Co-authored-by: Codex <noreply@openai.com>
 2026-09-13 02:44:23 -07:00
Markus HartikainenandJee Jee Li c711f740b7 [ROCm] Auto-enable breakable CUDA graphs for DeepseekV41ForCausalLM (#56349) 
Signed-off-by: Markus Hartikainen <markus.hartikainen@amd.com>
Co-authored-by: Jee Jee Li <pandaleefree@gmail.com>
 2026-09-12 20:47:07 -07:00
Raphaël RiallandOpenAI CodexSimon VeitnerTomas Ruiz
 		7ee8a6dd01 [watermarking] Dual-key gumbel-max watermarking for speculative decoding support (#56122) 
Signed-off-by: Raphael Rialland <raphael.rialland@mistral.ai>
Signed-off-by: Simon Veitner <sveitner@redhat.com>
Signed-off-by: Tomas Ruiz <tomas.ruiz.te@gmail.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
Co-authored-by: Simon Veitner <sveitner@redhat.com>
Co-authored-by: Tomas Ruiz <tomas.ruiz.te@gmail.com>
 2026-09-12 18:42:01 +00:00
liuzhenwei 2f59050eda [XPU][CI] Skip ROCm test on non-ROCm platforms (#56555) 
Signed-off-by: zhenwei-intel <zhenwei.liu@intel.com>
 2026-09-12 21:26:55 +08:00
Matthew BonanniOpenAI CodexLucas WilkinsonLucas WilkinsonNicolò LucchesiOpenAI CodexClaude Opus 5Lucas Wilkinson
 		d43bb2f37f [3/N] HiSparse: host-resident sparse-MLA decode hot-buffering (#53781) 
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>
Signed-off-by: Matthew Bonanni <matthew@neuralmagic.com>
Signed-off-by: Nicolò Lucchesi <nicolo.lucchesi@mistral.ai>
Co-authored-by: OpenAI Codex <codex@openai.com>
Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com>
Co-authored-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>
Co-authored-by: Nicolò Lucchesi <nicolo.lucchesi@mistral.ai>
Co-authored-by: OpenAI Codex <noreply@openai.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com>
 2026-09-12 00:36:47 -04:00
Andreas KaratzasandOpenAI Codex 9dd969da09 [Model][ROCm] Enable DeepSeek V4 Vision (#55107) 
Signed-off-by: Andreas Karatzas <Andreas.Karatzas@amd.com>
Signed-off-by: Andreas Karatzas <akaratza@amd.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
 2026-09-11 09:17:30 -07:00
Harry MellorandClaude Opus 5 c191787a68 [Bugfix] Align vLLM YaRN with Transformers and stop re-scaling max_model_len (#56446) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
 2026-09-11 13:01:38 +00:00
Andreas KaratzasOpenAI CodexLucas Wilkinson
 		d0dfe587d5 [Bugfix] Fall back to full decode graphs for noncompiled models (#55095) 
Signed-off-by: Andreas Karatzas <Andreas.Karatzas@amd.com>
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
Signed-off-by: Andreas Karatzas <akaratza@amd.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com>
 2026-09-11 01:56:05 -07:00
Raphaël RiallandOpenAI Codexmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 		ea40bb9e90 [feature] Watermarked generation and detection (Gumbel-max algorithm) (#54053) 
Signed-off-by: Raphael Rialland <raphael.rialland@mistral.ai>
Co-authored-by: OpenAI Codex <codex@openai.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-09-09 21:00:15 -07:00
Harry Huangandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> 3116c5d06b [Qwen4Exp] Support UVA PLE-offload and Engram tensor parallelism (#54371) 
Signed-off-by: huanghaoyan.hhy <huanghaoyan.hhy@alibaba-inc.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-09-09 07:32:34 -07:00
Andreas Karatzas 18615ad1ee [Bugfix] Handle null RoPE parameters for NoPE layers (#55949) 
Signed-off-by: Andreas Karatzas <akaratza@amd.com>
 2026-09-08 20:20:05 +00:00
Thien Tran d9105ea800 [Qwen3.8-Flash-Next] Remove torch.compile for NVIDIA implementation (#55272) 
Signed-off-by: Thien Tran <gau.nernst@yahoo.com.sg>
 2026-09-06 21:57:56 -07:00
aoshen02Claude Fable 5.1vx120
 		560ef78bfe [Perf][Model Runner V2] Compact sampling masks on GPU instead of unpacking the full-vocab bitmask on CPU (#54901) 
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Signed-off-by: vx120 <893600387@qq.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: vx120 <893600387@qq.com>
 2026-09-03 19:03:36 -07:00
Nicolò Lucchesiandcjackal f81eb41934 [Bugfix] adjust_dcp_kv_cache_interleave_size for NixlConnector only (#54803) 
Signed-off-by: NickLucche <nicolo.lucchesi@mistral.ai>
Co-authored-by: cjackal <44624812+cjackal@users.noreply.github.com>
 2026-09-02 12:31:14 +02:00