iHore/vLLM
8
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity
20,269 Commits 3 Branches 0 Tags
e00a034ad1d7c8ccce28f135fa9380dd9110e525
Commit Graph
342 Commits
Author SHA1 Message Date
Luke Alonsoandmgoin 2740c817ff [Kernel] Add b12x FP4 MoE backend (#52018) 
Signed-off-by: Luke Alonso <lalonso@gmail.com>
Signed-off-by: mgoin <mgoin64@gmail.com>
Signed-off-by: Misha Goin <mgoin64@gmail.com>
Co-authored-by: mgoin <mgoin64@gmail.com>
 2026-08-21 08:04:36 -07:00
Matthew KotilaReed MeyersonClaude Opus 4.8
 		7cfb97e337 [Frontend][Core][Spec Decode] Per-request acceptance stats in OpenAI API responses (#48915) 
Signed-off-by: Matthew Kotila <7692737+matthewkotila@users.noreply.github.com>
Co-authored-by: Reed Meyerson <31574681+reed-meyerson@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-08-20 15:42:49 -07:00
SeonjinandMisha Goin bd8865a299 [Kernel] Add FlashInfer TRTLLM MXFP8 linear backend (#52204) 
Signed-off-by: seonjinn <sna@nvidia.com>
Signed-off-by: Misha Goin <mgoin64@gmail.com>
Co-authored-by: Misha Goin <mgoin64@gmail.com>
 2026-08-20 10:07:22 -04:00
Guanyi Chenandz-zanez bf2866f8bf [KV Connector] Add decode offloading to Mooncake Store consumers (#52466) 
Signed-off-by: z-zanez <zhouzh93@mail2.sysu.edu.cn>
Signed-off-by: Guanyi Chen <939416532@qq.com>
Co-authored-by: z-zanez <zhouzh93@mail2.sysu.edu.cn>
 2026-08-19 17:40:35 -07:00
Russell BryantandClaude Opus 4.8 ef47a897e2 [Core] Make prefix-cache NONE_HASH deterministic by default (#51875) 
Signed-off-by: Russell Bryant <rbryant@redhat.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-08-18 23:14:39 +00:00
Tzu-Ling KanandClaude Opus 4.8 653cc6faca [Bugfix][NIXL] Include transfer mode (push/pull) in the compatibility hash (#50620) 
Signed-off-by: Tzu-Ling <tzulingk@nvidia.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-08-14 12:11:22 +08:00
Luke Alonso 3c79b1a8bf [Kernel] Add B12X dense linear backends (#52016) 
Signed-off-by: Luke Alonso <lalonso@gmail.com>
 2026-08-13 19:48:33 -07:00
QWERQWERQWE86 443fa5aca0 [Doc] Fix stale rejection_sample_method and synthetic_acceptance_rate (#51611) 
Signed-off-by: qwerqwerqwe8688-jpg <xuuuuu2021@163.com>
 2026-08-13 10:56:27 +00:00
Lucas WilkinsonOpenAI CodexClaude Opus 5Benjamin Chislettmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>Nick HillOpenAI Codex
 		7f7a32cfec [Spec Decode] DSpark confidence-scheduled verification (#47808) 
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
Signed-off-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com>
Signed-off-by: Benjamin Chislett <chislett.ben@gmail.com>
Signed-off-by: Lucas Wilkinson <wilkinson.lucas@gmail.com>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Benjamin Chislett <chislett.ben@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: OpenAI Codex <noreply@openai.com>
 2026-08-12 12:27:48 -07:00
Kyungmin LeeandHarry Mellor 529d010351 [Doc] Fix typos in speculative decoding docs (#51500) 
Signed-off-by: lkm2835 <lkm2835@gmail.com>
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2026-08-11 09:17:05 +01:00
ChaunceyandOr Ozeri c423998642 [KV Offload] Emit self-describing events for partial recurrent blocks (#51243) 
Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
 2026-08-09 11:28:44 +03:00
Isotr0py 073c510c91 [Migration] Migrate bitsandbytes support to OOT plugin (#43529) 
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>
Signed-off-by: Isotr0py <Isotr0py@outlook.com>
 2026-08-09 10:27:31 +08:00
fxmarty-amd 3518110f2f [Online quantization] Add online MXFP4 quantization support (#49347) 
Signed-off-by: Felix Marty <Felix.Marty@amd.com>
 2026-08-07 12:39:46 -07:00
Silen NaihinDakai AnJiangyun Zhu
 		b50fdebce0 [Bugfix] Fix level-2 sleep/wake/reload with enable_lora=True (#39935) 
Signed-off-by: Silen Naihin <silen.naihin@gmail.com>
Co-authored-by: Dakai An <dakaian108@gmail.com>
Co-authored-by: Jiangyun Zhu <riverclouds.zhu@qq.com>
 2026-08-06 09:13:10 +08:00
LZWandRoger Wang d36f24b66a [KV Connector][Mooncake] Add tenant ID support to MooncakeStoreConnector (#48069) 
Signed-off-by: lzw <2994587066@qq.com>
Co-authored-by: Roger Wang <hey@rogerw.io>
 2026-08-05 17:31:18 -07:00
Zhewen LiZhewen LiClaude Opus 5
 		14e57ad47d [Docker][KVConnector] Install mooncake from official wheels instead of a custom build (#51067) 
Signed-off-by: Zhewen Li <zhewenli@inferact.ai>
Co-authored-by: Zhewen Li <zhewenli@inferact.ai>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
 2026-08-05 11:58:44 -07:00
Ronen SchafferandOr Ozeri 8543522ca7 [KV Offload] Support out-of-tree secondary tier managers via module_path (#51007) 
Signed-off-by: Ronen Schaffer <ronen.schaffer@ibm.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
 2026-08-05 20:30:53 +03:00
Tzu-Ling KanGPT-5.6 SolCursor Grok 4.5Claude Opus 4.8
 		952694e384 [Bugfix] Validate NIXL speculative config compatibility (#49230) 
Signed-off-by: Tzu-Ling <tzulingk@nvidia.com>
Co-authored-by: GPT-5.6 Sol <noreply@openai.com>
Co-authored-by: Cursor Grok 4.5 <noreply@cursor.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-08-04 02:51:04 +08:00
wangxiyuan dd11df04f3 [Misc] Remove deprecated calculate_kv_scales runtime KV scale calculation (#49389) 
Signed-off-by: wangxiyuan <wangxiyuan1007@gmail.com>
 2026-08-03 11:15:13 -04:00
Nicolò Lucchesi 8ba87e0181 [CI] Mooncake PD integration tests (#46844) 
Signed-off-by: NickLucche <nicolo.lucchesi@mistral.ai>
 2026-08-03 06:55:51 -07:00
Fede KamelharandHarry Mellor 005fa01756 docs: document reasoning_content output removal as a breaking client change (#50624) 
Signed-off-by: fede-kamel <fkamelhar@gmail.com>
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2026-08-03 13:45:23 +00:00
zofiaandKunshang Ji e481da9508 [XPU] [Linear] add torch as xpu linear backend (#49664) 
Signed-off-by: Zhu, Zufang <zufang.zhu@intel.com>
Co-authored-by: Kunshang Ji <kunshang.ji@intel.com>
 2026-08-03 14:30:21 +08:00
Nicolò Lucchesi 17beffd55f [Misc] Clarify mono audio requirement (#50141) 
Signed-off-by: NickLucche <nicolo.lucchesi@mistral.ai>
 2026-07-31 10:46:53 +00:00
yinfengLiu f1e8fd27ae [ROCm] Add AITER FP8 ViT encoder attention (#49937) 
Signed-off-by: Liuyinfeng01 <yinfeliu@amd.com>
 2026-07-30 06:22:10 +00:00
Brandon PelfreyandCyrus Leung 5c7a7f9462 [docs] Add documentation for pynvvideocodec video decoding backend (#49066) 
Signed-off-by: Brandon Pelfrey <bpelfrey@nvidia.com>
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>
 2026-07-29 17:23:04 +00:00
Seiji Eicher 6370e53f24 [Frontend] Reuse prefill token ids on the decode chat path for disaggregated serving (#48145) 
Signed-off-by: Seiji Eicher <seiji@anyscale.com>
 2026-07-29 10:02:57 +02:00
Philip Pesic dc1be79031 Add CachePolicyFactory for pluggable/external eviction policies (#49114) 
Signed-off-by: Philip Pesic <philippesic06@gmail.com>
 2026-07-29 07:34:55 +03:00
nvbfalk 60915c972c [Feature] Add VidCom2 video token pruning (#47750) 
Signed-off-by: Benedikt Falk <bfalk@nvidia.com>
 2026-07-28 03:20:17 +00:00
Harjoth KharaandClaude Opus 5 8de50e46d4 [Docs] Document NVFP4 GEMM kernel selection and Marlin weight-only fallback (#49376) 
Signed-off-by: harjoth <harjoth.khara@gmail.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
 2026-07-26 20:53:28 -07:00
liranschourOr OzeriOr Ozeri
 		da3a252fd1 [KVOffload][P2P] Generic P2P secondary tier: peer lookup and serving via ParentManager (#48021) 
Signed-off-by: Liran Schour <lirans@il.ibm.com>
Signed-off-by: liranschour <liranschour@users.noreply.github.com>
Co-authored-by: Or Ozeri <or@ozery.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
 2026-07-26 11:45:47 +03:00
Brandon PelfreyandOpenAI Codex 94682b79f4 [multimodal] Make PyNvVideoCodec decoder concurrency configurable (#49753) 
Signed-off-by: Brandon Pelfrey <bpelfrey@nvidia.com>
Co-authored-by: OpenAI Codex <noreply@openai.com>
 2026-07-24 23:31:41 -07:00
Andreas Karatzas 16aca639b7 [ROCm] Upgrade NIXL and UCX (#49251) 
Signed-off-by: Andreas Karatzas <Andreas.Karatzas@amd.com>
 2026-07-21 22:30:02 -05:00
Chang GuoandOpenAI Codex f007cceb42 [KV Offload] Support self-describing KV events with TieringOffloadingSpec (#48679) 
Signed-off-by: Change72 <changg@nvidia.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
 2026-07-20 16:41:58 +03:00
Itay EtelisandItay Etelis 5245c80564 [Doc] Document blocks_per_chunk in the KV offloading guide (#49100) 
Signed-off-by: Itay Etelis <itay.etelis@ibm.com>
Co-authored-by: Itay Etelis <itay.etelis@ibm.com>
 2026-07-20 09:48:43 +03:00
Chang GuoCodexClaude Fable 5
 		426d48bfa1 [KV Offload] Add optional tier locality to FS/OBJ KV events (#48281) 
Signed-off-by: Change72 <changg@nvidia.com>
Co-authored-by: Codex <codex@openai.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
 2026-07-17 10:31:52 +03:00
yuvalluriaYuval LuriaClaude Sonnet 4.5
 		ab0a20d151 [Docs] Add Phi-3.5-mini-instruct to batch invariance tested models (#46396) 
Signed-off-by: Yuval Luria <yuvalluria@users.noreply.github.com>
Co-authored-by: Yuval Luria <yuvalluria@users.noreply.github.com>
Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com>
 2026-07-16 18:05:58 -04:00
liranschourandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> 615834ee58 [KVOffload][P2P] Well-known default host/port env vars and per-DP-rank control port (#47636) 
Signed-off-by: Liran Schour <lirans@il.ibm.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-07-15 15:22:56 +03:00
ViranjanPagarClaude Opus 4.8Isotr0pyIsotr0pyRoger Wang
 		e23b19309b Deepstream video backend (#42424) 
Signed-off-by: Viranjan Pagar <vpagar@nvidia.com>
Signed-off-by: Isotr0py <Isotr0py@outlook.com>
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Isotr0py <mozf@mail2.sysu.edu.cn>
Co-authored-by: Isotr0py <Isotr0py@outlook.com>
Co-authored-by: Roger Wang <hey@rogerw.io>
 2026-07-10 02:23:30 -07:00
albertoandChauncey 5715fde12c [Feature][Parser] Support include_reasoning param for non-Harmony models (#44301) 
Signed-off-by: Alberto Perdomo <aperdomo@redhat.com>
Co-authored-by: Chauncey <chaunceyjiang@gmail.com>
 2026-07-10 15:34:02 +08:00
Chang GuoClaude Fable 5Or Ozeri
 		2d814a0082 [kv_offload] Emit tier-owned BlockStored events from FS/OBJ secondary tiers (#47923) 
Signed-off-by: Change72 <changg@nvidia.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
 2026-07-10 06:17:23 +03:00
Tyler Michael Smithmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>Claude Opus 4.6
 		93e2ab7111 Disable dynamic speculative decoding when DP is enabled (#45963) 
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-07-07 12:52:41 +00:00
Harry Mellor dd94484577 Bump Transformers version to 5.10.4 (#41359) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2026-07-07 05:13:28 -07:00
Nicolas HugClaudeIsotr0py
 		700e882eab Add TorchCodec as a video decoding backend (#46609) 
Signed-off-by: Nicolas Hug <contact@nicolas-hug.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Isotr0py <mozf@mail2.sysu.edu.cn>
 2026-07-06 19:58:51 -07:00
nv-nedelman-1Simon MoGPT-5.5
 		86db6c3070 [Frontend] add per-request timing metrics field to response body of Chat/Completions APIs (#46768) 
Signed-off-by: Nicholas Edelman <nedelman@nvidia.com>
Signed-off-by: Simon Mo <simon@inferact.ai>
Co-authored-by: Simon Mo <simon@inferact.ai>
Co-authored-by: GPT-5.5 <noreply@cursor.com>
 2026-07-06 17:48:29 -07:00
Zhenzhong Xuandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> d9c1767cd4 [INC][ARK] Direct Register Custom Op for ARK (#46361) 
Signed-off-by: Zhenzhong1 <zhenzhong.xu@intel.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-07-06 13:45:50 +08:00
Chaojun ZhangandCopilot 1a308c449c [XPU] Add W8A8 FP8 linear kernel with multi-granularity quant support (#43645) 
Signed-off-by: Chaojun,Zhang <chaojun.zhang@intel.com>
Signed-off-by: Chaojun Zhang <chaojun.zhang@intel.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-07-04 18:10:01 +08:00
Ekagra RanjanBenjamin ChislettNick Hill
 		07516fda67 [MRV2][SD] Make Dynamic SD comatible with Full Cuda Graphs (#45953) 
Signed-off-by: Ekagra Ranjan <3116519+ekagra-ranjan@users.noreply.github.com>
Co-authored-by: Benjamin Chislett <chislett.ben@gmail.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
 2026-07-03 23:58:27 -04:00
sychen52andmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> 67ff0ae30f Support nvfp4 kv with kv-cache-dtype-skip-layers sliding_window (#42890) 
Signed-off-by: Shiyang Chen <shiychen@nvidia.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-07-04 02:29:13 +00:00
WonderfulWan_DFBenjamin ChislettBenjamin Chislett
 		3af8789559 [Feature] Universal speculative decoding for heterogeneous vocabularies (TLI) (#38174) 
Signed-off-by: wan-danfeng <wandanfeng0802@gmail.com>
Signed-off-by: Wonderful <wandanfeng0802@gmail.com>
Co-authored-by: Wan_DF <wonderful199082@126.com>
Co-authored-by: Benjamin Chislett <chislett.ben@gmail.com>
Co-authored-by: Benjamin Chislett <bchislett@nvidia.com>
 2026-07-02 01:34:20 -07:00
liranschourOr OzeriOr Ozeri
 		bec232a914 Secondary tier implementation for PD disaggregation (#42285) 
Signed-off-by: Liran Schour <lirans@il.ibm.com>
Signed-off-by: liranschour <liranschour@users.noreply.github.com>
Co-authored-by: Or Ozeri <or@ozery.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
 2026-06-30 07:51:44 +03:00