iHore/vLLM
8
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity
22,490 Commits 3 Branches 0 Tags
4a3ebc6b5bfcfa8cef4873f7ff213bf6bec4fef0
Commit Graph
424 Commits
Author SHA1 Message Date
MOVZX f83ecfac16 Merge remote-tracking branch 'upstream/main' into patches 2026-10-05 08:52:31 +07:00
Shenglei FuClaude Opus 5.5Wentao Ye
 		1388100560 [Kernel][LoRA] Add deterministic split-K=8 LoRA shrink for batch invariance (#59377) 
Signed-off-by: Shenglei Fu <117230642+ShengleiFu@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com>
 2026-10-04 18:46:16 -04:00
Lio EinaudiYifan DaCodeBuddyClaude Fable 5.1Wentao Ye
 		89439db727 [CI][Docs] Fix decode/prefill consistency test prefix construction; add GLM-4-9B and Phi-4 to batch-invariant tested models (#53692) 
Signed-off-by: Yifan Da <1363818765@qq.com>
Signed-off-by: LioEinaudi <zhao3024667639@gmail.com>
Signed-off-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com>
Co-authored-by: Yifan Da <1363818765@qq.com>
Co-authored-by: CodeBuddy <noreply@codebuddy.dev>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com>
 2026-10-04 11:55:18 -04:00
aoshen02andClaude Opus 5.5 5e56e9af2a [Feature] Release the CUDA graph pool on sleep (#59160) 
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-04 01:13:10 +00:00
MOVZX 85c31d47ad Merge remote-tracking branch 'upstream/main' into patches 
Signed-off-by: MOVZX <movzx@yahoo.com>
 2026-10-03 22:06:10 +07:00
MOVZX 2877121b2d Merge remote-tracking branch 'upstream/main' into patches 
Signed-off-by: MOVZX <movzx@yahoo.com>
 2026-10-03 21:59:03 +07:00
Martin Hickey bc21cba967 [Frontend] Add output_mode to /inference/v1/generate (RFC #56851 Phase 1) (#58588) 
Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com>
 2026-10-03 19:38:25 +08:00
Nils Matteson 9fdb174678 [Bugfix] Avoid InfiniBand state in TP1 snapshots (#59699) 
Signed-off-by: Nils Matteson <nilsmatteson@icloud.com>
 2026-10-03 03:59:17 +00:00
yifanFenggClaude Opus 5.5mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>Wentao Ye
 		6e517b15c1 [Docs] Add ERNIE 4.5 to batch invariance tested models (#58476) 
Signed-off-by: yifanFengg <evanfengff@gmail.com>
Signed-off-by: yifanFengg <yifanfen@amazon.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com>
 2026-10-02 20:37:32 +00:00
Rajeev Warrierandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> 30e956fad3 [Docs] Fix stale W4A16 NVFP4 default kernel in ModelOpt docs (#59135) 
Signed-off-by: Rajeev Warrier <warrier.rajeev.95@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-10-02 13:21:43 +00:00
Teng MaandTeng Ma 1a9eaa3a68 [Doc] Document ECMooncakeConnector for EPD disaggregation (#59196) 
Signed-off-by: Teng Ma <stmatengss@gmail.com>
Co-authored-by: Teng Ma <stmatengss@users.noreply.github.com>
 2026-10-02 12:10:33 +08:00
Yoni GozlanCursorBugen ZhaoClaude Opus 5.5
 		9bd993564f [Frontend] Parse tool calls and reasoning from checkpoint response templates (#58604) 
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Bugen Zhao <i@bugenzhao.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Yoni Gozlan <yonigozlan@users.noreply.github.com>
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
 2026-10-02 02:33:00 +00:00
Flora Feng 785b990116 [Distributed] Enable custom all-reduce under VLLM_BATCH_INVARIANT (#58623) 
Signed-off-by: Flora Feng <4florafeng@gmail.com>
Signed-off-by: sfeng33 <4florafeng@gmail.com>
 2026-10-02 02:25:48 +00:00
Flora Feng a5105dba05 [Frontend] Port Step-3.5 parsers to the streaming parser engine (#59321) 
Signed-off-by: sfeng33 <4florafeng@gmail.com>
 2026-10-01 22:18:37 +00:00
Zijing Liu ea84051819 [KVConnector][NIXL] Count completion notifications that arrive after KV expiry (#58875) 
Signed-off-by: Zijing Liu <liuzijing2014@gmail.com>
 2026-10-01 21:30:58 +00:00
Sundri Lai bcee730b1a [Docs] Remove references to removed env vars (#59530) 
Signed-off-by: Sundri Lai <laiyanting.neu@gmail.com>
 2026-10-01 08:26:14 +00:00
Nils Matteson 51048b13ce [Docs] Clarify snapshot runtime image support (#59127) 
Signed-off-by: Nils Matteson <nilsmatteson@icloud.com>
 2026-09-30 20:31:21 +00:00
Raphaël RiallandSimon VeitnerLucas Wilkinson
 		0f8b398158 [watermarking] add context deduplication support to speculative decoding (#56807) 
Signed-off-by: Raphael Rialland <raphael.rialland@mistral.ai>
Signed-off-by: Simon Veitner <sveitner@redhat.com>
Co-authored-by: Simon Veitner <sveitner@redhat.com>
Co-authored-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com>
 2026-09-30 12:22:10 -04:00
yuvalluriaandClaude Sonnet 4.6 863475eb9c docs: add gemma-2, SmolLM2, Qwen2.5-Coder, Llama-3.2-1B to batch invariance tested models (#54441) 
Signed-off-by: Yuval Luria <yluria@redhat.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-09-30 11:13:23 +00:00
Isotr0py 31876d8ea1 [Bugfix] Fix TorchCodec audio IO correctness (#58364) 
Signed-off-by: Isotr0py <Isotr0py@outlook.com>
 2026-09-29 11:34:48 -05:00
Yixin DongCodexyuchuanBugen ZhaoClaude Opus 5.5
 		dfc8e0f3e2 [Frontend] Support strict MiMo-V2.6 tool calling (#58019) 
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: yuchuan <yuchuan.7streams@gmail.com>
Co-authored-by: Bugen Zhao <i@bugenzhao.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Ubospica <ubospica@gmail.com>
Signed-off-by: yuchuan <yuchuan.7streams@gmail.com>
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
 2026-09-29 12:47:47 +00:00
Andreas KaratzasandOpenAI Codex 3acddf01e7 [Bugfix][ROCm] Keep Mooncake bootstrap ports bound during startup (#58967) 
Signed-off-by: Andreas Karatzas <andreas.karatzas@protonmail.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
 2026-09-29 04:14:26 -05:00
MOVZX a63dca4afb Merge remote-tracking branch 'upstream/main' into patches 
Signed-off-by: MOVZX <movzx@yahoo.com>
 2026-09-29 12:54:18 +07:00
Shijie LyuShijie LyuClaude
 		c6843d397c [Bugfix][Frontend] Validate reused prompt token ids; render messages for media and echo (#55771) 
Signed-off-by: Shijie Lyu <sjlyu@amazon.com>
Co-authored-by: Shijie Lyu <sjlyu@amazon.com>
Co-authored-by: Claude <noreply@anthropic.com>
 2026-09-29 10:36:22 +08:00
qizixi 6d3ea3c2c9 [KVConnector][NIXL] Support packed MLA KV layouts in pipeline-parallel push prefill (#50499) 
Signed-off-by: zixi-qi <zixi@inferact.ai>
 2026-09-28 18:50:46 +02:00
linitra24andJee Jee Li 953f90d25f [LoRA] Support variable num_labels for sequence classification (#57766) 
Signed-off-by: linitra24 <renshuang.zhou@daocloud.io>
Co-authored-by: Jee Jee Li <pandaleefree@gmail.com>
 2026-09-28 14:58:35 +00:00
MOVZX 76af434852 Merge remote-tracking branch 'upstream/main' into patches 2026-09-28 12:06:52 +07:00
Chao Lei 386ac2573e [Bugfix][KV Connector] Retry Mooncake bootstrap registration on timeout (reopens #55763) (#58919) 2026-09-27 19:47:36 +00:00
MOVZXandHinata 43fbd6510b [Qwen4Exp] Serve PLE rows from the loader's mapped checkpoint shards (PR #58815) 
Adds an opt-in backend, --engram-config '{"host_file_gather": true}', that
serves Qwen4Exp PLE rows from the checkpoint files through the page cache
instead of allocating the 47.7 GiB table on GPU or in pinned host memory.

- Qwen4ExpPLEFileGatherEmbedding allocates no table rows; load_weights keeps
  the default loader's zero-copy safetensors views of each shard. At bind
  (post-load hook) every view must be file-backed per /proc/self/maps (HF
  cache symlinks work; heap/pinned/anonymous memory is rejected) and the
  shards must cover the whole table.
- Each step, Qwen4ExpModelState.prepare_inputs computes the PLE n-gram ids,
  copies them to pinned host memory, syncs the stream once, issues
  POSIX_FADV_WILLNEED over the coalesced page ranges of the step's rows,
  gathers the rows on the host, and copies them into a fixed device staging
  buffer. The forward reads only the staging buffer, so FULL CUDA graph
  capture is unaffected.
- Rows stay in the checkpoint dtype (FP8 table stays FP8, dequantized on the
  GPU); dummy loads serve zeros; weight reload is rejected; takes precedence
  over cpu_offload; rejected on ROCm and with elastic EP.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
 2026-09-27 09:54:42 +07:00
Lio Einaudiandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> 0c87a197b8 [Bugfix] Disable sequence parallelism / async TP under batch invariance and add a TP regression test (#56377) 
Signed-off-by: LioEinaudi <zhao3024667639@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-09-26 22:19:08 -04:00
Yousafandsfeng33 83db8c8390 [Feature][Frontend] Add granite_thinking_parser reasoning parser for Granite 4.2 (#55957) 
Signed-off-by: Yousaf shah <yousaf.shah@gmail.com>
Co-authored-by: sfeng33 <4florafeng@gmail.com>
 2026-09-25 20:43:08 +00:00
Andrii SkliarAndrii SkliarCodexBenjamin Chislett
 		73a78e6f1f [SpecDecode] Add LiLiCorr drafter (#57934) 
Signed-off-by: Andrii Skliar <askliar@nvidia.com>
Signed-off-by: Andrii Skliar <andreyws96@gmail.com>
Co-authored-by: Andrii Skliar <askliar@nvidia.com>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Benjamin Chislett <chislett.ben@gmail.com>
 2026-09-25 10:08:28 -04:00
Isotr0py 24928a4d62 [UX][Frontend] Introduce vllm preload cli for fast restart (#56680) 
Signed-off-by: Isotr0py <Isotr0py@outlook.com>
 2026-09-25 06:56:39 +00:00
fxmarty-amdmgoinmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 		90a9515006 [Cleanup] Remove online quantization support in fp8.py in favor of online shorthands (#53585) 
Signed-off-by: Felix Marty <Felix.Marty@amd.com>
Signed-off-by: mgoin <mgoin64@gmail.com>
Co-authored-by: mgoin <mgoin64@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-09-24 12:48:14 -07:00
Lio Einaudi 711fc55c10 [Perf] Use breakable CUDA graphs (no torch.compile) by default under VLLM_BATCH_INVARIANT so the tuned matmul configs see the runtime M (#57586) 
Signed-off-by: LioEinaudi <zhao3024667639@gmail.com>
 2026-09-23 14:39:38 -04:00
Tianyu GuoandCyrus Leung 3ee2906f47 [EPD] Support metadata-only audio inputs (#57887) 
Signed-off-by: Tianyu Guo <guoty@inferact.ai>
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>
 2026-09-23 09:03:50 +08:00
aoshen02aoshen02Codex
 		639461eda4 [RL][Sleep] Retain frozen weights across level-2 sleep (#57891) 
Signed-off-by: aoshen02 <aoshen02@users.noreply.github.com>
Co-authored-by: aoshen02 <aoshen02@users.noreply.github.com>
Co-authored-by: Codex <noreply@openai.com>
 2026-09-22 12:36:55 +08:00
Nils MattesonTaneem IbrahimelehayymAndreas Karatzasmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 		13058faa49 [Frontend] Add reusable TP1 initialized-engine snapshots (#51360) 
Signed-off-by: Nils Matteson <nilsmatteson@icloud.com>
Signed-off-by: Taneem Ibrahim <taneem.ibrahim@gmail.com>
Signed-off-by: Andreas Karatzas <Andreas.Karatzas@amd.com>
Co-authored-by: Taneem Ibrahim <taneem.ibrahim@gmail.com>
Co-authored-by: elehayym <52448798+Yuzu23@users.noreply.github.com>
Co-authored-by: Andreas Karatzas <akaratza@amd.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-09-21 23:18:49 +00:00
Nicolò Lucchesi fba33e39cb [Docs] Add an Engram feature page explaining Engram usage in vLLM (#57910) 
Signed-off-by: NickLucche <nicolo.lucchesi@mistral.ai>
 2026-09-21 17:15:28 +00:00
jiangkuaixue123andCodex 7ddd461a28 [Docs] Add ECMooncakeConnector usage example for EPD (#57742) 
Signed-off-by: jiangkuaixue123 <jiangxiaozhou111@163.com>
Co-authored-by: Codex <noreply@openai.com>
 2026-09-21 09:15:02 +08:00
roikoren755andmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> 49ee12d742 [Misc] Rename --enable-mamba-fine-grained-prefix-cache (#53945 follow-up) (#57382) 
Signed-off-by: Roi Koren <roik@nvidia.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-09-20 09:50:34 -07:00
David CheungClaudeNick Hill
 		5302d1fe40 [Model Runner V2] Support custom logits processors (#56497) 
Signed-off-by: David Cheung <d7cheung@gmail.com>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
 2026-09-19 17:46:49 +00:00
Nicolò Lucchesi 164dca8d1e [Nixl] Separate transport-failure metrics from KV expiry (#55854) 
Signed-off-by: NickLucche <nicolo.lucchesi@mistral.ai>
 2026-09-18 18:08:59 +02:00
lazymioClaude Opus 5Bugen Zhao
 		dee37d8911 [Frontend] Add --tool-strict-level for server-side control for structural tag activation (#56268) 
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Bugen Zhao <i@bugenzhao.com>
Signed-off-by: lazymio <mio@lazym.io>
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
 2026-09-18 04:32:39 +00:00
Dakai AnAlanFokCoaoshen02OpenAI Codex
 		092bdd6d57 [Frontend][Core] Add release_kv_cache_memory() API (#44890) 
Signed-off-by: Dakai An <dakaian108@gmail.com>
Signed-off-by: AlanFokCo <alanfok2868@gmail.com>
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Signed-off-by: Dakai An <77474977+andakai@users.noreply.github.com>
Co-authored-by: AlanFokCo <alanfok2868@gmail.com>
Co-authored-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: OpenAI Codex <noreply@openai.com>
 2026-09-18 07:12:12 +08:00
YoungandYoung b78fbab4be [Docs] Fix the typos in the document (#57190) 
Co-authored-by: Young <yangchh15@163.com>
 2026-09-17 15:52:10 +00:00
Zhou zihengTianyu Guojiangkuaixue123Codex
 		2743dc4f92 [Feature][Multimodal] Add cross-encoder caching to Mooncake P2P (#56242) 
Signed-off-by: Zhou ziheng <jiaranran2@gmail.com>
Co-authored-by: Tianyu Guo <guoty@inferact.ai>
Co-authored-by: jiangkuaixue123 <jiangxiaozhou111@163.com>
Co-authored-by: Codex <noreply@openai.com>
 2026-09-17 13:16:41 +08:00
xinnywinneandBenjamin Chislett fbf2c5e8be [Frontend] Add per-request metrics to Responses API (#55084) 
Signed-off-by: wxin <wxin@nvidia.com>
Co-authored-by: Benjamin Chislett <bchislett@nvidia.com>
 2026-09-16 22:52:05 +00:00
linitra24andJee Jee Li c81eb3e69c [LoRA] Support modules_to_save for sequence classification (#53555) 
Signed-off-by: linitra24 <Joy25810@foxmail.com>
Signed-off-by: linitra24 <renshuang.zhou@daocloud.io>
Co-authored-by: Jee Jee Li <pandaleefree@gmail.com>
 2026-09-16 07:15:42 -07:00
alberto 75dc588269 [KV Offload] Add per-request max_load_tokens control (#55885) 
Signed-off-by: Alberto Perdomo <aperdomo@redhat.com>
 2026-09-16 06:57:38 +03:00