iHore/vLLM
8
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity
22,022 Commits 3 Branches 0 Tags
5da84df6b7c824caefcac2976843ae25283d7f8b
Commit Graph
395 Commits
Author SHA1 Message Date
MOVZXandHinata 43fbd6510b [Qwen4Exp] Serve PLE rows from the loader's mapped checkpoint shards (PR #58815) 
Adds an opt-in backend, --engram-config '{"host_file_gather": true}', that
serves Qwen4Exp PLE rows from the checkpoint files through the page cache
instead of allocating the 47.7 GiB table on GPU or in pinned host memory.

- Qwen4ExpPLEFileGatherEmbedding allocates no table rows; load_weights keeps
  the default loader's zero-copy safetensors views of each shard. At bind
  (post-load hook) every view must be file-backed per /proc/self/maps (HF
  cache symlinks work; heap/pinned/anonymous memory is rejected) and the
  shards must cover the whole table.
- Each step, Qwen4ExpModelState.prepare_inputs computes the PLE n-gram ids,
  copies them to pinned host memory, syncs the stream once, issues
  POSIX_FADV_WILLNEED over the coalesced page ranges of the step's rows,
  gathers the rows on the host, and copies them into a fixed device staging
  buffer. The forward reads only the staging buffer, so FULL CUDA graph
  capture is unaffected.
- Rows stay in the checkpoint dtype (FP8 table stays FP8, dequantized on the
  GPU); dummy loads serve zeros; weight reload is rejected; takes precedence
  over cpu_offload; rejected on ROCm and with elastic EP.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
 2026-09-27 09:54:42 +07:00
Yousafandsfeng33 83db8c8390 [Feature][Frontend] Add granite_thinking_parser reasoning parser for Granite 4.2 (#55957) 
Signed-off-by: Yousaf shah <yousaf.shah@gmail.com>
Co-authored-by: sfeng33 <4florafeng@gmail.com>
 2026-09-25 20:43:08 +00:00
Andrii SkliarAndrii SkliarCodexBenjamin Chislett
 		73a78e6f1f [SpecDecode] Add LiLiCorr drafter (#57934) 
Signed-off-by: Andrii Skliar <askliar@nvidia.com>
Signed-off-by: Andrii Skliar <andreyws96@gmail.com>
Co-authored-by: Andrii Skliar <askliar@nvidia.com>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Benjamin Chislett <chislett.ben@gmail.com>
 2026-09-25 10:08:28 -04:00
Isotr0py 24928a4d62 [UX][Frontend] Introduce vllm preload cli for fast restart (#56680) 
Signed-off-by: Isotr0py <Isotr0py@outlook.com>
 2026-09-25 06:56:39 +00:00
fxmarty-amdmgoinmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 		90a9515006 [Cleanup] Remove online quantization support in fp8.py in favor of online shorthands (#53585) 
Signed-off-by: Felix Marty <Felix.Marty@amd.com>
Signed-off-by: mgoin <mgoin64@gmail.com>
Co-authored-by: mgoin <mgoin64@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-09-24 12:48:14 -07:00
Lio Einaudi 711fc55c10 [Perf] Use breakable CUDA graphs (no torch.compile) by default under VLLM_BATCH_INVARIANT so the tuned matmul configs see the runtime M (#57586) 
Signed-off-by: LioEinaudi <zhao3024667639@gmail.com>
 2026-09-23 14:39:38 -04:00
Tianyu GuoandCyrus Leung 3ee2906f47 [EPD] Support metadata-only audio inputs (#57887) 
Signed-off-by: Tianyu Guo <guoty@inferact.ai>
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>
 2026-09-23 09:03:50 +08:00
aoshen02aoshen02Codex
 		639461eda4 [RL][Sleep] Retain frozen weights across level-2 sleep (#57891) 
Signed-off-by: aoshen02 <aoshen02@users.noreply.github.com>
Co-authored-by: aoshen02 <aoshen02@users.noreply.github.com>
Co-authored-by: Codex <noreply@openai.com>
 2026-09-22 12:36:55 +08:00
Nils MattesonTaneem IbrahimelehayymAndreas Karatzasmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 		13058faa49 [Frontend] Add reusable TP1 initialized-engine snapshots (#51360) 
Signed-off-by: Nils Matteson <nilsmatteson@icloud.com>
Signed-off-by: Taneem Ibrahim <taneem.ibrahim@gmail.com>
Signed-off-by: Andreas Karatzas <Andreas.Karatzas@amd.com>
Co-authored-by: Taneem Ibrahim <taneem.ibrahim@gmail.com>
Co-authored-by: elehayym <52448798+Yuzu23@users.noreply.github.com>
Co-authored-by: Andreas Karatzas <akaratza@amd.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-09-21 23:18:49 +00:00
Nicolò Lucchesi fba33e39cb [Docs] Add an Engram feature page explaining Engram usage in vLLM (#57910) 
Signed-off-by: NickLucche <nicolo.lucchesi@mistral.ai>
 2026-09-21 17:15:28 +00:00
jiangkuaixue123andCodex 7ddd461a28 [Docs] Add ECMooncakeConnector usage example for EPD (#57742) 
Signed-off-by: jiangkuaixue123 <jiangxiaozhou111@163.com>
Co-authored-by: Codex <noreply@openai.com>
 2026-09-21 09:15:02 +08:00
roikoren755andmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> 49ee12d742 [Misc] Rename --enable-mamba-fine-grained-prefix-cache (#53945 follow-up) (#57382) 
Signed-off-by: Roi Koren <roik@nvidia.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-09-20 09:50:34 -07:00
David CheungClaudeNick Hill
 		5302d1fe40 [Model Runner V2] Support custom logits processors (#56497) 
Signed-off-by: David Cheung <d7cheung@gmail.com>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
 2026-09-19 17:46:49 +00:00
Nicolò Lucchesi 164dca8d1e [Nixl] Separate transport-failure metrics from KV expiry (#55854) 
Signed-off-by: NickLucche <nicolo.lucchesi@mistral.ai>
 2026-09-18 18:08:59 +02:00
lazymioClaude Opus 5Bugen Zhao
 		dee37d8911 [Frontend] Add --tool-strict-level for server-side control for structural tag activation (#56268) 
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Bugen Zhao <i@bugenzhao.com>
Signed-off-by: lazymio <mio@lazym.io>
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
 2026-09-18 04:32:39 +00:00
Dakai AnAlanFokCoaoshen02OpenAI Codex
 		092bdd6d57 [Frontend][Core] Add release_kv_cache_memory() API (#44890) 
Signed-off-by: Dakai An <dakaian108@gmail.com>
Signed-off-by: AlanFokCo <alanfok2868@gmail.com>
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Signed-off-by: Dakai An <77474977+andakai@users.noreply.github.com>
Co-authored-by: AlanFokCo <alanfok2868@gmail.com>
Co-authored-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: OpenAI Codex <noreply@openai.com>
 2026-09-18 07:12:12 +08:00
YoungandYoung b78fbab4be [Docs] Fix the typos in the document (#57190) 
Co-authored-by: Young <yangchh15@163.com>
 2026-09-17 15:52:10 +00:00
Zhou zihengTianyu Guojiangkuaixue123Codex
 		2743dc4f92 [Feature][Multimodal] Add cross-encoder caching to Mooncake P2P (#56242) 
Signed-off-by: Zhou ziheng <jiaranran2@gmail.com>
Co-authored-by: Tianyu Guo <guoty@inferact.ai>
Co-authored-by: jiangkuaixue123 <jiangxiaozhou111@163.com>
Co-authored-by: Codex <noreply@openai.com>
 2026-09-17 13:16:41 +08:00
xinnywinneandBenjamin Chislett fbf2c5e8be [Frontend] Add per-request metrics to Responses API (#55084) 
Signed-off-by: wxin <wxin@nvidia.com>
Co-authored-by: Benjamin Chislett <bchislett@nvidia.com>
 2026-09-16 22:52:05 +00:00
linitra24andJee Jee Li c81eb3e69c [LoRA] Support modules_to_save for sequence classification (#53555) 
Signed-off-by: linitra24 <Joy25810@foxmail.com>
Signed-off-by: linitra24 <renshuang.zhou@daocloud.io>
Co-authored-by: Jee Jee Li <pandaleefree@gmail.com>
 2026-09-16 07:15:42 -07:00
alberto 75dc588269 [KV Offload] Add per-request max_load_tokens control (#55885) 
Signed-off-by: Alberto Perdomo <aperdomo@redhat.com>
 2026-09-16 06:57:38 +03:00
qizixiClaudeYiliu DongCodex
 		0fefffc934 [KVConnector][NIXL] Support attention-HMA layouts in pipeline-parallel push prefill (#50494) 
Signed-off-by: zixi-qi <zixi@inferact.ai>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Yiliu Dong <91178480+qianlihuang@users.noreply.github.com>
Co-authored-by: Codex <noreply@openai.com>
 2026-09-15 15:43:56 -07:00
liranschour 17bc3e106a [KVConnector][P2P] Configurable unbound-store timeout and one-RTT rejection of a late fetch (#53453) 
Signed-off-by: Liran Schour <lirans@il.ibm.com>
 2026-09-15 05:38:18 +03:00
Netanel Haber d4ee7fe7a9 [Quantization] Select linear backends per quantization (#51204) 
Signed-off-by: Netanel Haber <58652339+netanel-haber@users.noreply.github.com>
 2026-09-14 16:24:18 -04:00
Nicolò LucchesiandCodex 1713a09491 [Rust Frontend] Forward per-request watermarking controls (#56338) 
Co-authored-by: Codex <noreply@openai.com>
Signed-off-by: NickLucche <nicolo.lucchesi@mistral.ai>
 2026-09-14 23:17:02 +08:00
Ganesh RCursorjiang1.li
 		a5f6f61a8a [CI][CPU] Add speculative-decoding coverage to CPU CI (#54934) 
Signed-off-by: R <Ganesh.R@amd.com>
Signed-off-by: jiang1.li <jiang1.li@intel.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: jiang1.li <jiang1.li@intel.com>
 2026-09-14 19:51:38 +08:00
Raphaël RiallandOpenAI CodexSimon VeitnerTomas Ruiz
 		7ee8a6dd01 [watermarking] Dual-key gumbel-max watermarking for speculative decoding support (#56122) 
Signed-off-by: Raphael Rialland <raphael.rialland@mistral.ai>
Signed-off-by: Simon Veitner <sveitner@redhat.com>
Signed-off-by: Tomas Ruiz <tomas.ruiz.te@gmail.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
Co-authored-by: Simon Veitner <sveitner@redhat.com>
Co-authored-by: Tomas Ruiz <tomas.ruiz.te@gmail.com>
 2026-09-12 18:42:01 +00:00
Walter Beller-MoralesandFlora Feng 756794a9a7 [Frontend] create unified Cohere parser (#56392) 
Signed-off-by: walterbm <walter.beller.morales@gmail.com>
Co-authored-by: Flora Feng <4florafeng@gmail.com>
 2026-09-12 04:04:46 -04:00
djramic 5392fbca2a [ROCm][Docker] Pin AINIC apt repo to snapshot 1.117.5-a-77 (#56459) 
Signed-off-by: Djordje Ramic <djoramic@amd.com>
 2026-09-11 14:14:16 +00:00
AlexHuangClaude Opus 5Or Ozeri
 		e83d57d8ae [Doc] Sync KV event medium terminology after #48123 (#51646) 
Signed-off-by: Alex <jihui.huang@daocloud.io>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
 2026-09-10 14:42:31 +03:00
Raphaël RiallandOpenAI Codexmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 		ea40bb9e90 [feature] Watermarked generation and detection (Gumbel-max algorithm) (#54053) 
Signed-off-by: Raphael Rialland <raphael.rialland@mistral.ai>
Co-authored-by: OpenAI Codex <codex@openai.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-09-09 21:00:15 -07:00
Ronen Schaffer 138d137b5b [Refactor][kv_offload]: rename block→chunk (#52615) 
Signed-off-by: Ronen Schaffer <ronen.schaffer@ibm.com>
 2026-09-09 17:45:49 +03:00
omerpaz95 a97dacb710 [Docs][EC Connector] CPU EC Connector usage Docs (#54522) 
Signed-off-by: omerpaz95 <omerpaz95@gmail.com>
 2026-09-08 17:43:08 -07:00
shikamd123 1a522b6949 [ROCm] [Docker] Upgrade default AINIC repo to ship libionic 54.0-187-1 (#54112) 
Signed-off-by: Shiksha Patel <shiksha.patel@amd.com>
 2026-09-08 19:21:18 +00:00
akshaverwzhao18Claude Opus 5roikoren755
 		263c4ff95f [Bugfix][Spec Decode] Cache the Mamba state at the block-grid position of EAGLE resume (#53945) 
Signed-off-by: wzhao18 <wzhao18.sz@gmail.com>
Signed-off-by: Adam Shaver <ashaver@nvidia.com>
Signed-off-by: akshaver <168006157+akshaver@users.noreply.github.com>
Co-authored-by: wzhao18 <wzhao18.sz@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: roikoren755 <26850796+roikoren755@users.noreply.github.com>
 2026-09-08 09:59:58 -04:00
Sean Westfall 58ad1f3b89 [Docs] Add OLMo 2 to batch-invariance tested models (#55691) 
Signed-off-by: Sean Westfall <sean.westfall@gmail.com>
 2026-09-07 09:56:49 +00:00
Andreas Karatzas f7f060d253 [Bugfix][Audio] Restore soundfile-first automatic decoding (#55642) 
Signed-off-by: Andreas Karatzas <Andreas.Karatzas@amd.com>
 2026-09-07 06:21:02 +00:00
fxmarty-amdCodexmgoin
 		5690b02c03 [Quantization] Support online quantization with partially pre-quantized checkpoints (#51392) 
Signed-off-by: Felix Marty <Felix.Marty@amd.com>
Signed-off-by: mgoin <mgoin64@gmail.com>
Co-authored-by: Codex <codex@openai.com>
Co-authored-by: mgoin <mgoin64@gmail.com>
 2026-09-04 11:46:06 -07:00
Jared Wensaltman155Isotr0py
 		6a039f465e [feat] add torchcodec as audio loader and implement selective audio backend (#51826) 
Signed-off-by: JaredforReal <w13431838023@gmail.com>
Co-authored-by: saltman155 <saltman155@outlook.com>
Co-authored-by: Isotr0py <Isotr0py@outlook.com>
 2026-09-04 16:16:10 +00:00
Tomasz Zielinskimergify[bot] <37929162+mergify[bot]@users.noreply.github.com>Claude Opus 5Harry Mellor
 		ffe3bb3c72 [Hardware][XPU] Register matmul and linear batch-invariant kernels for XPU (#49209) 
Signed-off-by: tzielinski-habana <tomasz.zielinski@intel.com>
Signed-off-by: Tomasz Zielinski <85164140+tzielinski-habana@users.noreply.github.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2026-09-02 13:31:38 +00:00
fxmarty-amd 7894394b07 [Online quantization] Add targeted online quantization configuration based on user patterns (#51285) 2026-09-02 08:55:15 -04:00
周峥andGuanyi Chen c866ba9d11 [KV Connector] Support heterogeneous TP sharing in Mooncake Store Connector (#53129) 
Signed-off-by: Guanyi Chen <939416532@qq.com>
Signed-off-by: z-zanez <zhouzh93@mail2.sysu.edu.cn>
Signed-off-by: Zheng Zhou <zhouzh93@mail2.sysu.edu.cn>
Signed-off-by: zane <zhouzh93@mail2.sysu.edu.cn>
Co-authored-by: Guanyi Chen <939416532@qq.com>
 2026-09-01 10:18:06 -07:00
Harjoth KharaClaudeHarry Mellor
 		40824284bc [Doc] Document FP8 GEMM kernel selection and Blackwell support (#49936) 
Signed-off-by: harjoth <harjoth.khara@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2026-09-01 13:22:40 +01:00
Jan Kwieciński ec32f669bb [Feature][MM_UUIDs] Allow empty video URLs when using multi-modal UUIDs (#54220) 
Signed-off-by: Jan Kwiecinski <jkwiecinski@nvidia.com>
 2026-09-01 05:55:27 +00:00
Lain 882ca8d696 [Kernel] add Flashinfer cutedsl w4a16 linear (#53014) 
Signed-off-by: Siyuan Fu <siyuanf@nvidia.com>
 2026-09-01 02:49:14 +00:00
macheroandClaude 87b9b5b8d9 [Test][VLM] Add batch-invariance tests for Qwen3-VL (#53531) 
Signed-off-by: machero <sr1123640465@126.com>
Co-authored-by: Claude <noreply@anthropic.com>
 2026-08-30 09:49:41 -04:00
Isotr0py 4fc943b867 [Multimodal] Deprecate PyAV video decoder backend (#54231) 
Signed-off-by: Isotr0py <Isotr0py@outlook.com>
 2026-08-29 15:13:28 +00:00
hangy-amd 31c579503d update quark docs to include online quantization (#52736) 
Signed-off-by: Hang Yang <hangy@amd.com>
 2026-08-28 18:23:06 +00:00
Paarth SharmaandWentao Ye a18dbe49ad [Doc] Add EXAONE-4.0-1.2B to batch invariance tested models (#53839) 
Signed-off-by: Cogniera <paarths376@gmail.com>
Signed-off-by: Paarth Sharma <paarths376@gmail.com>
Co-authored-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com>
 2026-08-27 18:38:03 +00:00
Shenglei FuClaudemergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 		f18d0ba90d [Doc] Add Granite 3.1 series to batch invariance tested models (#53650) 
Signed-off-by: Shenglei Fu <117230642+ShengleiFu@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-08-26 18:29:10 -04:00