iHore/vLLM
8
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity
16,919 Commits 3 Branches 0 Tags
1806d1adfc9b598bc6eb94de38a330aaad04c291
Commit Graph
104 Commits
Author SHA1 Message Date
Gabriel Wu 82536acc54 Keep scheduler alive for delayed KV connector frees (#43433) 
Signed-off-by: Zihua Wu <13583761+lucifer1004@users.noreply.github.com>
 2026-05-23 06:23:32 +00:00
Hongxin XuClaude Opus 4.6arlenxuJunjie Zhang
 		c7560af424 [RFC] Replace shared-memory routed experts with ModelRunnerOutput transfer and HTTP support (#39568) 
Signed-off-by: xhx1022 <1737006628@qq.com>
Signed-off-by: arlenxu <arlenxu@tencent.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: arlenxu <arlenxu@tencent.com>
Co-authored-by: Junjie Zhang <junj.jay.zhang@gmail.com>
 2026-05-14 14:12:30 +00:00
+7
Yifan QiaoYongye ZhuYongye ZhuSimon MoBugen ZhaoGiancarlo DelfinJee Jee LiNick HillRoger WangRoy WangWoosuk KwonyoukaichaoZhewen LiZijing LiukhluuqizixiZhewen Li
 		4d51588e23 [Feat] DeepSeek V4 Rebased (#40860) 
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai>
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
Signed-off-by: qizixi <zixi@inferact.ai>
Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>
Signed-off-by: Yongye Zhu <zyy1102000@gmail.com>
Co-authored-by: Yongye Zhu <zyy1102000@gmail.com>
Co-authored-by: Yongye Zhu <yongye@inferact.ai>
Co-authored-by: Simon Mo <simon@inferact.ai>
Co-authored-by: Bugen Zhao <i@bugenzhao.com>
Co-authored-by: Giancarlo Delfin <gdelfin@inferact.ai>
Co-authored-by: Jee Jee Li <pandaleefree@gmail.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Roger Wang <hey@rogerw.io>
Co-authored-by: Roy Wang <yasong.wang@inferact.ai>
Co-authored-by: Woosuk Kwon <woosuk@inferact.ai>
Co-authored-by: youkaichao <youkaichao@gmail.com>
Co-authored-by: Zhewen Li <jerven.vllm@gmail.com>
Co-authored-by: Zijing Liu <liuzijing2014@gmail.com>
Co-authored-by: khluu <khluu000@gmail.com>
Co-authored-by: qizixi <zixi@inferact.ai>
Co-authored-by: Zhewen Li <zhewenli@inferact.ai>
 2026-04-26 18:31:08 -07:00
mukesh-haiClaude Sonnet 4.6Mark McLoughlin
 		5c18b961d6 [Core][Metrics] expose waiting request breakdown via labeled metric (capacity/deferred) (#38435) 
Signed-off-by: Mukesh Baphna <mukesh@hippocraticai.com>
Signed-off-by: Mark McLoughlin <markmc@redhat.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: Mark McLoughlin <markmc@redhat.com>
 2026-04-13 15:30:55 -04:00
Walter Beller-Morales e69a265135 [Feat][Core] safely abort requests when FSM fails to advance (#38663) 
Signed-off-by: walterbm <walter.beller.morales@gmail.com>
 2026-04-06 08:00:16 -07:00
HarshRathvaandOr Ozeri 17b72fd1c8 Fix priority preemption regression test in scheduler (#37051) 
Signed-off-by: HarshRathva <harshrathvaai@gmail.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
 2026-04-01 06:36:12 +03:00
Wentao Ye 1bf2ddd0ee [Refactor] Rename WAITING_FOR_FSM to WAITING_FOR_STRUCTURED_OUTPUT_GRAMMAR (#38048) 
Signed-off-by: yewentao256 <zhyanwentao@126.com>
 2026-03-25 11:41:44 -04:00
Wentao YeandOr Ozeri a8ff2cca92 [Perf] Optimize scheduler overhead for PD disaggregation, around 5% E2E perf improvement (#35781) 
Signed-off-by: yewentao256 <zhyanwentao@126.com>
Signed-off-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
 2026-03-10 21:25:30 -07:00
lifandmcelrath 00b814ba5a [V0 Deprecation] Remove unused swap_space parameter (#36216) 
Signed-off-by: majiayu000 <1835304752@qq.com>
Co-authored-by: mcelrath
 2026-03-07 22:09:55 +08:00
Jiayi YanandHarry Mellor 6a895197fa [Bugfix][CI] fix typos (#34934) 
Signed-off-by: 1195343015 <1195343015@qq.com>
Signed-off-by: Jiayi Yan <66017932+1195343015@users.noreply.github.com>
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2026-03-05 17:05:46 +00:00
Qi Wang 6aa6ad8992 [BugFix] Fix implicit and incorrect assumption on ECConnector is_producer (#34783) 
Signed-off-by: Qi Wang <qiwa@nvidia.com>
 2026-03-04 15:01:30 +01:00
Ekagra RanjanandNicolò Lucchesi cd81cdb399 [Scheduler][ASR] Fix CrossAttn blocks per-request for Variable length encoder inputs (#31058) 
Signed-off-by: Ekagra Ranjan <3116519+ekagra-ranjan@users.noreply.github.com>
Co-authored-by: Nicolò Lucchesi <nlucches@redhat.com>
 2026-02-16 11:08:44 +00:00
Cyrus Leung ea5ff3a1f6 [Refactor] Simplify BOS/EOS token handling (#34435) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2026-02-12 18:18:24 -08:00
Krish Gupta 748625cdaf [V1][BugFix] Fix EAGLE3 encoder cache miss with disable_chunked_mm_input (#34220) 
Signed-off-by: KrxGu <krishom70@gmail.com>
 2026-02-10 13:05:32 +00:00
Mark McLoughlin 2abd97592f [KV Connector][Metrics] Do not count local prefix cache hits in connector queries (#30522) 
Signed-off-by: Mark McLoughlin <markmc@redhat.com>
 2026-02-05 09:57:27 +02:00
Nick Hill fa4e0fb028 [Core] Don't schedule spec tokens with prefill chunks (#33652) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-02-04 23:40:22 +00:00
Or Ozeri 8e32690869 [KV Connector][BugFix] scheduler: Delay freeing blocks of aborted async loads (#32255) 
Fixes a not-yet-reported case where it was possible for blocks to be
freed by an abort before an async transfer completed, resulting
in corrupted KV data.

Signed-off-by: Or Ozeri <oro@il.ibm.com>
 2026-02-04 11:16:34 +00:00
Nick Hill 52ee21021a [BugFix][Spec Decoding] Fix negative accepted tokens metric crash (#33729) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-02-03 23:34:41 +00:00
Cyrus Leung c6e7404cc5 [Multimodal] Simplify MM input definitions (#33331) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2026-01-29 13:32:04 +00:00
Wentao YeandNick Hill 3e440786af [Feature] Fully support for async scheduling + PP, 30.8% E2E throughput improvement, 31.8% TPOT improvement (#32618) 
Signed-off-by: yewentao256 <zhyanwentao@126.com>
Signed-off-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
 2026-01-28 20:30:32 +00:00
Joshua DengRoger WangPatrick von PlatenNick Hill
 		91601ff478 [Feature] add session based streaming input support to v1 (#28973) 
Signed-off-by: Joshua Deng <joshuakdeng@gmail.com>
Signed-off-by: Patrick von Platen <patrick.v.platen@gmail.com>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Signed-off-by: Roger Wang <hey@rogerw.io>
Co-authored-by: Roger Wang <hey@rogerw.io>
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
 2026-01-24 12:06:28 -08:00
knlnguyen1802 378385b90c [EC Connector] Optimize remote cache check in scheduler (#32585) 
Signed-off-by: knlnguyen1802 <knlnguyen1802@gmail.com>
 2026-01-22 03:30:59 +00:00
Or Ozeri 2be765b68a [BugFix] scheduler: Fix ordering preserving of skipped requests (#32173) 
Signed-off-by: Or Ozeri <oro@il.ibm.com>
 2026-01-12 18:39:38 +00:00
Or Ozeri 028599739d [BugFix] scheduler: Fix resuming of preempted requests after async load (#31583) 
Signed-off-by: Or Ozeri <oro@il.ibm.com>
 2026-01-10 12:39:25 -08:00
Lumosis b634e619bb Decouple page_size_bytes calculation in AttentionSpec for TPU/RPA Compatibility. (#31635) 
Signed-off-by: Lihao Ran <imlihao.ran@gmail.com>
Signed-off-by: Lumosis <30372757+Lumosis@users.noreply.github.com>
 2026-01-08 09:00:07 +00:00
Nick Hill 32f4e4db00 [Cleanup] Remove deprecated fields from CachedRequestData class (#31734) 
Signed-off-by: njhill <nickhill123@gmail.com>
 2026-01-05 21:07:14 +00:00
Or Ozeri 4c6fd25880 kv_transfer: Rename the shared storage connectors (#30201) 
Signed-off-by: Or Ozeri <oro@il.ibm.com>
 2025-12-08 20:46:09 -08:00
Cyrus Leung e83b7e379c Revert "[Renderer] Separate out RendererConfig from ModelConfig (#30145)" (#30199) 2025-12-07 00:00:22 -08:00
Cyrus Leung 27f4c2fd46 [Renderer] Separate out RendererConfig from ModelConfig (#30145) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-12-06 23:15:42 -08:00
Harry Mellor 951445a52d Remove default values from InitVars so that they're not stored (#29859) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2025-12-02 12:16:37 +00:00
Zhuohan LiandNick Hill d0cd728907 [Core] Support reseting all running requests' KV while calling reset_prefix_cache (#28827) 
Signed-off-by: Zhuohan Li <zhuohan123@gmail.com>
Signed-off-by: Nick Hill <nhill@redhat.com>
Co-authored-by: Nick Hill <nhill@redhat.com>
 2025-12-02 02:25:05 +00:00
wang.yuqi 67fc16cd8c [Bugfix] If chunked_prefill is disabled, end the scheduling early. (#28911) 
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io>
 2025-11-25 16:06:09 +08:00
Cyrus Leung 5a4802588e [Misc] Further clean up chunked prefill and prefix caching init (#29186) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-11-22 19:34:15 +08:00
Mark McLoughlinandNicolò Lucchesi c6fa3895e9 [KV Connector] Fix async connector prefix cache metrics (#28585) 
Signed-off-by: Mark McLoughlin <markmc@redhat.com>
Co-authored-by: Nicolò Lucchesi <nlucches@redhat.com>
 2025-11-21 17:45:00 -05:00
Jialin Ouyang 30b9c67743 Revert "[Redo] #26368 (#28771)" (#29121) 
Signed-off-by: Jialin Ouyang <Jialin.Ouyang@gmail.com>
 2025-11-20 21:27:45 -08:00
Cyrus LeungandJialin Ouyang 98b4d389ed [Redo] #26368 (#28771) 
Signed-off-by: Jialin Ouyang <Jialin.Ouyang@gmail.com>
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Co-authored-by: Jialin Ouyang <Jialin.Ouyang@gmail.com>
 2025-11-14 22:47:41 -08:00
Nick Hill ac86bff8cb Revert "[Core] Performance: Use list[np.ndarray] instead of list[list… (#28773) 2025-11-14 20:24:00 -08:00
Jialin Ouyang 186352b270 [Core] Performance: Use list[np.ndarray] instead of list[list[int]] for output tokens for GC optimization (#26368) 
Signed-off-by: Jialin Ouyang <Jialin.Ouyang@gmail.com>
 2025-11-14 16:04:04 -08:00
Cyrus Leung e2741f6cbc [Chore] Rename SchedulerConfig.chunked_prefill_enabled (#28735) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-11-14 18:39:57 +00:00
Cyrus Leung 511a6b611d [Config] Clean up SchedulerConfig initialization (#28665) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-11-14 22:41:02 +08:00
Mark McLoughlin 6e25b1cddf [KV Connector] Test async mode in scheduler tests (#28550) 
Signed-off-by: Mark McLoughlin <markmc@redhat.com>
 2025-11-13 18:30:59 -05:00
Chenguang Zhengn00909098knlnguyen1802herotai214gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>Khuong LeKhuong Le
 		4ccffe561f [Core] Encoder separation for Encode-Prefill-Decode Disaggregation (#25233) 
Signed-off-by: n00909098 <nguyen.kha.long@huawei.com>
Signed-off-by: knlnguyen1802 <knlnguyen1802@gmail.com>
Signed-off-by: herotai214 <herotai214@gmail.com>
Signed-off-by: Khuong Le <khuong.le.manh@huawei.com>
Signed-off-by: Khuong Le <lemanhkhuong2611@gmail.com>
Co-authored-by: n00909098 <nguyen.kha.long@huawei.com>
Co-authored-by: knlnguyen1802 <knlnguyen1802@gmail.com>
Co-authored-by: herotai214 <herotai214@gmail.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: Khuong Le <khuong.le.manh@huawei.com>
Co-authored-by: Khuong Le <lemanhkhuong2611@gmail.com>
 2025-11-11 18:58:33 -08:00
Kuntai Du 86dca07d9b [Hybrid allocator + kv connector] revert connector test changes related to hybrid allocator (#28011) 
Signed-off-by: KuntaiDu <kuntai@uchicago.edu>
 2025-11-05 10:36:31 +00:00
Nick Hill 0cdbe7b744 [Core] Async scheduling + structured outputs compatibility (#26866) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-11-01 00:35:04 +00:00
Kuntai Du b853540388 [Core][Hybrid allocator + kv connector 1/n] Enable hybrid allocator + KV cache connector (#25712) 
Signed-off-by: KuntaiDu <kuntai@uchicago.edu>
Signed-off-by: Kuntai Du <kuntai@uchicago.edu>
 2025-10-24 23:34:18 -07:00
Tova Movshovitz 88afa11010 [Metrics] [KVConnector] Add connector prefix cache hit rate stats (#26245) 
Signed-off-by: tovam <tovam@pliops.com>
 2025-10-23 12:21:08 +02:00
Nick Hill 4aed506b65 [Core] Streamline some structured output related code (#26737) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-10-14 23:27:44 +00:00
Harry Mellor 8fcaaf6a16 Update Optional[x] -> x | None and Union[x, y] to x | y (#26633) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2025-10-12 09:51:31 -07:00
Harry Mellor 7c12763b24 Fix some typing issues found by mypy==1.18.2 (#26596) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2025-10-10 18:21:25 +00:00
Chen Zhang 606b00e80f [bugfix][DCP] fix block_size of hash in DCP prefix caching (#26296) 
Signed-off-by: Chen Zhang <zhangch99@outlook.com>
 2025-10-10 03:02:49 -07:00