iHore/vLLM
8
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity
20,845 Commits 3 Branches 0 Tags
78300cdabffbaaffe7625ed34412ec2aec75535c
Commit Graph
141 Commits
Author SHA1 Message Date
Ziming HuangandOpenAI Codex a26b71d868 [Bugfix][PD] Pad resumed speculative decode requests (#55126) 
Signed-off-by: Ziming Huang <48115868+ZeldaHuang@users.noreply.github.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
 2026-09-04 11:05:33 +08:00
aoshen02Claude Fable 5.1vx120
 		560ef78bfe [Perf][Model Runner V2] Compact sampling masks on GPU instead of unpacking the full-vocab bitmask on CPU (#54901) 
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Signed-off-by: vx120 <893600387@qq.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: vx120 <893600387@qq.com>
 2026-09-03 19:03:36 -07:00
Artem Perevedentsev 39e276eaeb [Structured Output] Let terminal grammars stop under min_tokens (II) (#54218) 
Signed-off-by: Artem Perevedentsev <aperevedents@nvidia.com>
 2026-08-31 20:39:27 +04:00
+5
Harry Huangzjy0516KungYorkThien TranYifan Qiaoandyluo7Isotr0pyandyluo7OpenAI Codexlishunyang12Vadim GimpelsonIsotr0pyLucas WilkinsonClaude Opus 5OpenAI Codex
 		e126687a9a [Model] Support Qwen3.8-Flash-Next (#53896) 
Signed-off-by: huanghaoyan.hhy <huanghaoyan.hhy@alibaba-inc.com>
Signed-off-by: andyluo7 <andy.luo@amd.com>
Signed-off-by: lishunyang12 <lishunyang03@gmail.com>
Signed-off-by: Harry Huang <huanghaoyan.hhy@alibaba-inc.com>
Signed-off-by: Thien Tran <gau.nernst@yahoo.com.sg>
Signed-off-by: Isotr0py <Isotr0py@outlook.com>
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai>
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com>
Co-authored-by: zjy0516 <riverclouds.zhu@qq.com>
Co-authored-by: KungYork <firelikest@163.com>
Co-authored-by: Thien Tran <gau.nernst@yahoo.com.sg>
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai>
Co-authored-by: andyluo7 <43718156+andyluo7@users.noreply.github.com>
Co-authored-by: Isotr0py <mozf@inferact.ai>
Co-authored-by: andyluo7 <andy.luo@amd.com>
Co-authored-by: OpenAI Codex <noreply@openai.com>
Co-authored-by: lishunyang12 <lishunyang03@gmail.com>
Co-authored-by: Vadim Gimpelson <156319763+vadiklyutiy@users.noreply.github.com>
Co-authored-by: Isotr0py <Isotr0py@outlook.com>
Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
 2026-08-31 05:57:56 +00:00
Summer YangandNick Hill 2aac565cae [Core][KV Connector] Start async KV loads after the forward launch when no sync loads are scheduled (#53333) 
Signed-off-by: Summer Yang <girasoleyang@gmail.com>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
 2026-08-28 21:58:00 +00:00
Matthew KotilaReed MeyersonClaude Opus 4.8
 		7cfb97e337 [Frontend][Core][Spec Decode] Per-request acceptance stats in OpenAI API responses (#48915) 
Signed-off-by: Matthew Kotila <7692737+matthewkotila@users.noreply.github.com>
Co-authored-by: Reed Meyerson <31574681+reed-meyerson@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-08-20 15:42:49 -07:00
Giancarlo DelfinandClaude Fable 5 f80b66f548 [Model Runner V2][Spec Decode] Add KV cache support for multi-layer MTP (#50062) 
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
 2026-08-13 16:22:29 -07:00
vx120aoshen02CodexClaude Opus 4.6OpenAI Codex
 		50ba4bc6b2 [Feature] Mask Replay (#49577) 
Signed-off-by: vx120 <893600387@qq.com>
Signed-off-by: vx120 <57470515+vx120@users.noreply.github.com>
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
 2026-08-13 08:44:31 +00:00
Wentao Ye 0914ed2e81 [Perf] Adaptive budget for spec scheduled input tokens, ~60% better Kimi K3 DSpark TTFT (#51725) 
Signed-off-by: yewentao256 <zhyanwentao@126.com>
 2026-08-10 20:52:21 -07:00
Jiangyun ZhuAkshat AnandCodex
 		243c63baf5 [V1][Scheduler] Apply Mamba alignment before encoder caps (#51603) 
Signed-off-by: Jiangyun Zhu <riverclouds.zhu@qq.com>
Co-authored-by: Akshat Anand <40275336+cipheraxat@users.noreply.github.com>
Co-authored-by: Codex <codex@openai.com>
 2026-08-10 16:37:17 +08:00
omerpaz95Nicolò Lucchesimergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 		61c1dd0966 [EC Connector] Call to EC Connector update_connector_output from scheduler (#49579) 
Signed-off-by: omerpaz95 <omerpaz95@gmail.com>
Signed-off-by: Nicolò Lucchesi <nicolo.lucchesi@gmail.com>
Co-authored-by: Nicolò Lucchesi <nicolo.lucchesi@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-08-10 14:25:57 +08:00
Yifan QiaoandOpenAI Codex d6941300fc [BugFix] Scope divergent hybrid cache hits to capable connectors (#50344) 
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai>
Co-authored-by: OpenAI Codex <codex@openai.com>
 2026-08-09 15:38:13 -07:00
Jeff (Junze) MaRoger WangYifan QiaoOpenAI Codex
 		58d3918e3e [BugFix] Preserve divergent FA hits with external Mamba state (#51468) 
Signed-off-by: Roger Wang <hey@rogerw.io>
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai>
Signed-off-by: Jeff Ma <jeffjma@umich.edu>
Co-authored-by: Roger Wang <hey@rogerw.io>
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai>
Co-authored-by: OpenAI Codex <codex@openai.com>
 2026-08-08 20:20:16 +08:00
Tianyu Guo dd856e48bb [Bugfix][EPD][Model Runner V2] Skip gather mm embeddings for encoder only instance (#51222) 
Signed-off-by: Tianyu Guo <guoty@inferact.ai>
 2026-08-06 20:40:47 -07:00
Tianyu Guo 50c51682a1 [Bugfix][EC Connector] Don't stop an encoder-instance request before its images are encoded (#50275) 
Signed-off-by: Tianyu Guo <guoty@inferact.ai>
 2026-08-05 05:53:58 +00:00
Nick HillChris EastwoodYifan Qiao
 		a0c092ee72 [BugFix] Fix num_output_placeholders preemption underflow (#48245) 
Signed-off-by: Chris Eastwood <chris.eastwood@pwn4g3.dev>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai>
Co-authored-by: Chris Eastwood <chris.eastwood@pwn4g3.dev>
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai>
 2026-07-29 06:41:54 -07:00
zhrrrandNick Hill 12213c6795 [Bugfix] handle grammar compilation failures to avoid engine crash (#47312) 
Signed-off-by: zhuhaoran <zhuhaoran.zhr@alibaba-inc.com>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
 2026-07-23 18:13:49 +08:00
Nick Hill 229e01e9e1 [BugFix] Handle per-group prefix-hit divergence for hybrid models with KV connector (#48425) 2026-07-22 17:19:11 -07:00
Seiji EicherLixi ZhangharjothkharaClaude Opus 4.8Nick Hill
 		8950394e0a [Bugfix] Prefix-cache metrics double-counted when a KV connector defers requests (#48860) 
Signed-off-by: Seiji Eicher <58963096+eicherseiji@users.noreply.github.com>
Signed-off-by: Seiji Eicher <seiji@anyscale.com>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Lixi Zhang <muxixibbb@gmail.com>
Co-authored-by: harjothkhara <harjoth.khara@gmail.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
 2026-07-21 15:02:03 +01:00
yzong-rh a287eb163f [Front-end] [Messages] Populate num_cache_creation_tokens (#48535) 
Signed-off-by: Yifan Zong <yzong@redhat.com>
 2026-07-18 13:04:35 -04:00
Max HuRoger WangShang Wang
 		5a65ba5f17 [Refactor] Move iteration logging to the frontend (#46647) 
Signed-off-by: maxyanghu <hyoung2991@gmail.com>
Co-authored-by: Roger Wang <hey@rogerw.io>
Co-authored-by: Shang Wang <shangw@nvidia.com>
 2026-07-15 17:59:05 -07:00
omerpaz95andOr Ozeri 8df14cfc8c [EC Connector] Add EC Transfer Params (#42433) 
Signed-off-by: omerpaz95 <omerpaz95@gmail.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
 2026-07-12 14:35:33 +03:00
Kyung Sub Lee (Daniel)andClaude Fable 5 c5b66233b2 [Bugfix][Spec Decode] Skip uniform spec-decode padding for diffusion models (#47464) 
Signed-off-by: kl527 <kl527@cornell.edu>
Signed-off-by: Kyung Sub Lee (Daniel) <66861800+kl527@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
 2026-07-07 09:25:12 +00:00
yue.yuandBenjamin Chislett e7c9df9449 [Bugfix][Structured Output][Spec Decode] Constrain bitmask and trim grammar advance at the reasoning boundary (#44297) 
Signed-off-by: Allen.Yu <yuyue0225sc@163.com>
Signed-off-by: yue.yu <yuyue0225sc@163.com>
Co-authored-by: Benjamin Chislett <chislett.ben@gmail.com>
 2026-07-04 09:08:45 +00:00
Nick Hill 658b54efe4 [ModelRunner V2] Update scheduler tests to cover MRV2 paths (#46771) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-06-26 09:36:31 -07:00
Yiliu DongNick HillJade ZhengGiancarlo DelfinZijing Liu
 		d490b98162 [Core] Avoid mixed length specdec batches via padding (#45237) 
Signed-off-by: Yiliu Dong <91178480+qianlihuang@users.noreply.github.com>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Jade Zheng <zheng.shoujian@outlook.com>
Co-authored-by: Giancarlo Delfin <gdelfin@inferact.ai>
Co-authored-by: Zijing Liu <liuzijing2014@gmail.com>
 2026-06-25 08:34:44 -07:00
Nick Hillandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> 6af0559ddb [Core][DP] Throttle prefills based on local prefill work (#46532) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-06-24 02:27:12 +00:00
Nick HillClaude Opus 4.8Roger Wang
 		3e6529cc0e [Bugfix][Spec Decode] Fix EAGLE drafter multimodal encoder cache misses (#46315) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Roger Wang <hey@rogerw.io>
 2026-06-22 18:14:02 +00:00
amanClaudeNick Hill
 		6e919960af [Perf] Skip/shrink all_token_ids copy in scheduler for non-async and V2 runner (#45840) 
Signed-off-by: amanchugh89 <amanchugh.89@gmail.com>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
 2026-06-20 22:36:57 +00:00
Nick Hill d8d95998dc [Core] Add prefill step cadence for better non-PD DP balancing (#44558) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-06-16 13:17:18 -07:00
llxNick HillJiangyun Zhu
 		d467a2a7f2 [Bugfix] Defer block freeing until in-flight steps finish under async scheduling + PD KV consumer (#45357) 
Signed-off-by: llx-08 <2596671364@qq.com>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Jiangyun Zhu <riverclouds.zhu@qq.com>
 2026-06-15 21:36:09 +00:00
Nick Hill 1a369783e9 [BugFix] Avoid prematurely freeing cached mm encoder outputs (#45347) 
Signed-off-by: Roger Wang <hey@rogerw.io>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-06-12 15:39:40 -07:00
Srinivas Krovvidiandsrinivas_oo7 9ff278b1d2 [Core][KV Connector] fix scheduler KV connector stats aggregation (#43877) 
Fixes scheduler-side KV connector stats collection so that:

1. update_connector_output() runs before scheduler-side stats are collected.
2. worker-side and scheduler-side KV connector stats are aggregated when both are present.
3. scheduler-only KV connector stats are still emitted when no worker-side stats exist.

Signed-off-by: srinivas_oo7 <sklinkedin0120@gmail.com>
Co-authored-by: srinivas_oo7 <sklinkedin0120@gmail.com>
 2026-06-12 14:51:55 +00:00
Nick HillandClaude Opus 4.8 4085ff7cb4 [Core] Add kvcache watermark to reduce preemptions (#44594) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-06-11 08:27:31 -07:00
Nick Hill 3b3d5287fa [BugFix] Resolve multiple async kv load deadlock (#44560) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-06-06 23:05:47 +00:00
omerpaz95andmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> d247a9dc13 [EC Connector] Non blocking EC Connector lookup (#41627) 
Signed-off-by: omerpaz95 <omerpaz95@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-06-02 08:48:25 +00:00
aoshen02aoshen524Claude Opus 4.7
 		1fc2cee50a [KVConnector][Mooncake] Wire reset_cache cascade end-to-end (#42694) 
Signed-off-by: aoshen524 <aoshen524@gmail.com>
Signed-off-by: Ao Shen <aoshen@inferact.ai>
Co-authored-by: aoshen524 <aoshen524@gmail.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-26 20:52:35 -07:00
Gabriel Wu 82536acc54 Keep scheduler alive for delayed KV connector frees (#43433) 
Signed-off-by: Zihua Wu <13583761+lucifer1004@users.noreply.github.com>
 2026-05-23 06:23:32 +00:00
Hongxin XuClaude Opus 4.6arlenxuJunjie Zhang
 		c7560af424 [RFC] Replace shared-memory routed experts with ModelRunnerOutput transfer and HTTP support (#39568) 
Signed-off-by: xhx1022 <1737006628@qq.com>
Signed-off-by: arlenxu <arlenxu@tencent.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: arlenxu <arlenxu@tencent.com>
Co-authored-by: Junjie Zhang <junj.jay.zhang@gmail.com>
 2026-05-14 14:12:30 +00:00
+7
Yifan QiaoYongye ZhuYongye ZhuSimon MoBugen ZhaoGiancarlo DelfinJee Jee LiNick HillRoger WangRoy WangWoosuk KwonyoukaichaoZhewen LiZijing LiukhluuqizixiZhewen Li
 		4d51588e23 [Feat] DeepSeek V4 Rebased (#40860) 
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai>
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
Signed-off-by: qizixi <zixi@inferact.ai>
Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>
Signed-off-by: Yongye Zhu <zyy1102000@gmail.com>
Co-authored-by: Yongye Zhu <zyy1102000@gmail.com>
Co-authored-by: Yongye Zhu <yongye@inferact.ai>
Co-authored-by: Simon Mo <simon@inferact.ai>
Co-authored-by: Bugen Zhao <i@bugenzhao.com>
Co-authored-by: Giancarlo Delfin <gdelfin@inferact.ai>
Co-authored-by: Jee Jee Li <pandaleefree@gmail.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Roger Wang <hey@rogerw.io>
Co-authored-by: Roy Wang <yasong.wang@inferact.ai>
Co-authored-by: Woosuk Kwon <woosuk@inferact.ai>
Co-authored-by: youkaichao <youkaichao@gmail.com>
Co-authored-by: Zhewen Li <jerven.vllm@gmail.com>
Co-authored-by: Zijing Liu <liuzijing2014@gmail.com>
Co-authored-by: khluu <khluu000@gmail.com>
Co-authored-by: qizixi <zixi@inferact.ai>
Co-authored-by: Zhewen Li <zhewenli@inferact.ai>
 2026-04-26 18:31:08 -07:00
mukesh-haiClaude Sonnet 4.6Mark McLoughlin
 		5c18b961d6 [Core][Metrics] expose waiting request breakdown via labeled metric (capacity/deferred) (#38435) 
Signed-off-by: Mukesh Baphna <mukesh@hippocraticai.com>
Signed-off-by: Mark McLoughlin <markmc@redhat.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: Mark McLoughlin <markmc@redhat.com>
 2026-04-13 15:30:55 -04:00
Walter Beller-Morales e69a265135 [Feat][Core] safely abort requests when FSM fails to advance (#38663) 
Signed-off-by: walterbm <walter.beller.morales@gmail.com>
 2026-04-06 08:00:16 -07:00
HarshRathvaandOr Ozeri 17b72fd1c8 Fix priority preemption regression test in scheduler (#37051) 
Signed-off-by: HarshRathva <harshrathvaai@gmail.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
 2026-04-01 06:36:12 +03:00
Wentao Ye 1bf2ddd0ee [Refactor] Rename WAITING_FOR_FSM to WAITING_FOR_STRUCTURED_OUTPUT_GRAMMAR (#38048) 
Signed-off-by: yewentao256 <zhyanwentao@126.com>
 2026-03-25 11:41:44 -04:00
Wentao YeandOr Ozeri a8ff2cca92 [Perf] Optimize scheduler overhead for PD disaggregation, around 5% E2E perf improvement (#35781) 
Signed-off-by: yewentao256 <zhyanwentao@126.com>
Signed-off-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
 2026-03-10 21:25:30 -07:00
lifandmcelrath 00b814ba5a [V0 Deprecation] Remove unused swap_space parameter (#36216) 
Signed-off-by: majiayu000 <1835304752@qq.com>
Co-authored-by: mcelrath
 2026-03-07 22:09:55 +08:00
Jiayi YanandHarry Mellor 6a895197fa [Bugfix][CI] fix typos (#34934) 
Signed-off-by: 1195343015 <1195343015@qq.com>
Signed-off-by: Jiayi Yan <66017932+1195343015@users.noreply.github.com>
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2026-03-05 17:05:46 +00:00
Qi Wang 6aa6ad8992 [BugFix] Fix implicit and incorrect assumption on ECConnector is_producer (#34783) 
Signed-off-by: Qi Wang <qiwa@nvidia.com>
 2026-03-04 15:01:30 +01:00
Ekagra RanjanandNicolò Lucchesi cd81cdb399 [Scheduler][ASR] Fix CrossAttn blocks per-request for Variable length encoder inputs (#31058) 
Signed-off-by: Ekagra Ranjan <3116519+ekagra-ranjan@users.noreply.github.com>
Co-authored-by: Nicolò Lucchesi <nlucches@redhat.com>
 2026-02-16 11:08:44 +00:00
Cyrus Leung ea5ff3a1f6 [Refactor] Simplify BOS/EOS token handling (#34435) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2026-02-12 18:18:24 -08:00