iHore/vLLM
8
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity
6,768 Commits 3 Branches 0 Tags
6881107948c00a8564bc2fa85308f6fc2f065d64
Commit Graph
4568 Commits
Author SHA1 Message Date
Roger Wang 437b76ff59 [V1][Core] Fix memory issue with logits & sampling (#13721) 2025-02-24 06:10:06 -08:00
Nick Hill cbae7af552 [V1][BugFix] Fix engine core client shutdown hangs (#13298) 
Even though ZMQ context.destroy() is meant to close open sockets before terminating the context, it appears to be necessary to do this explicitly or else it can hang in the context.term() method.

Close zmq sockets explicitly before terminating context, make shutdown of client resource more robust, shut down engine core process prior to terminating zmq context.

Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-02-23 13:07:43 -08:00
youkaichao eb24dc4a45 [v1] torchrun compatibility (#13642) 
Signed-off-by: youkaichao <youkaichao@gmail.com>
 2025-02-23 22:47:24 +08:00
Nick Hill 5a2ba16f5c [Core][Distributed] Use IPC (domain socket) ZMQ socket for local comms (#13688) 2025-02-23 02:54:29 -08:00
Isotr0py ba5106e519 [LMM] Implement merged multimodal processor for whisper (#13278) 2025-02-23 01:46:03 -08:00
Kyle Sayers d5ca2110f1 [Quant] BaiChuan SupportsQuant (#13710) 2025-02-22 19:21:15 -08:00
Kevin H. Luu 2c5e637b57 [ci] Use env var to control whether to use S3 bucket in CI (#13634) 2025-02-22 19:19:45 -08:00
Andy Lo 322d2a27d6 [BugFix] Minor: logger import in attention backend (#13706) 
Signed-off-by: Andy Lo <andy@mistral.ai>
 2025-02-22 16:51:13 -08:00
Helena Kloosterman 382f66fb08 [Bugfix] Fix boolean conversion for OpenVINO env variable (#13615) 2025-02-22 08:04:12 -08:00
Gregory Shtrasberg c904fdddf6 [ROCm] Apply FP8 weights padding to values not divisible by 512 bytes on ROCm (#13231) 2025-02-22 05:54:38 -08:00
Sage Moore 558db8083c [V1][Kernel] Refactor the prefix_prefill kernel so that the caller no longer has to pass in the context lengths (#13095) 2025-02-22 05:25:41 -08:00
Kaixi Hou e109e598c7 [NVIDIA] Support nvfp4 cutlass gemm (#13571) 2025-02-22 05:24:05 -08:00
Keyun Tong 8db1b9d0a1 Support SSL Key Rotation in HTTP Server (#13495) 2025-02-22 05:17:44 -08:00
youkaichao 2382ad29d1 [ci] fix linter (#13701) 
Signed-off-by: youkaichao <youkaichao@gmail.com>
 2025-02-22 20:28:59 +08:00
youkaichao 3e472d882a [core] set up data parallel communication (#13591) 
Signed-off-by: youkaichao <youkaichao@gmail.com>
 2025-02-22 19:28:59 +08:00
Cyrus Leung 7f6bae561c [CI/Build] Fix pre-commit errors (#13696) 2025-02-22 00:31:26 -08:00
Jee Jee Li 105b8ce4c0 [Misc] Reduce LoRA-related static variable (#13166) 2025-02-22 00:21:30 -08:00
Mark McLoughlin 2cb8c1540e [Metrics] Add --show-hidden-metrics-for-version CLI arg (#13295) 2025-02-22 00:20:45 -08:00
Mark McLoughlin 1cd981da4f [V1][Metrics] Support vllm:cache_config_info (#13299) 2025-02-22 00:20:00 -08:00
Yu Chin Fabian Lim fca20841c2 Correction to TP logic for Mamba Mixer 2 when Num Groups not divisible by TP Size (#13660) 2025-02-22 00:19:10 -08:00
Jennifer ZhaoandRoger Wang da31b5333e [Bugfix] V1 Memory Profiling: V0 Sampler Integration without Rejection Sampler (#13594) 
Signed-off-by: Jennifer Zhao <7443418+JenZhao@users.noreply.github.com>
Co-authored-by: Roger Wang <ywang@roblox.com>
 2025-02-22 00:08:29 -08:00
Lu Fang bb78fb318e [v1] Support allowed_token_ids in v1 Sampler (#13210) 
Signed-off-by: Lu Fang <lufang@fb.com>
 2025-02-22 14:13:05 +08:00
Shane A 9a1f1da5d1 [Bugfix][Model] OLMo 2: split qkv correctly for GQA and MQA (#13687) 2025-02-21 22:07:45 -08:00
Gordon Wong 68d630a0c7 [ROCM] fix native attention function call (#13650) 2025-02-21 22:07:04 -08:00
Jun Duan 68d535ef44 [Misc] Capture and log the time of loading weights (#13666) 2025-02-21 22:06:34 -08:00
Robin c6ed93860f [Bugfix][API Server] Fix invalid usage of 'ge' and 'le' in port valid… (#13672) 2025-02-21 22:05:28 -08:00
Keyun Tong 0ffdf8ce0c [HTTP Server] Make model param optional in request (#13568) 2025-02-21 21:55:50 -08:00
Lucas WilkinsonPatrick Hornsimon-moTyler Michael Smith
 		288cc6c234 [Attention] MLA with chunked prefill (#12639) 
Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
Co-authored-by: Patrick Horn <patrick.horn@gmail.com>
Co-authored-by: simon-mo <xmo@berkeley.edu>
Co-authored-by: Tyler Michael Smith <tyler@neuralmagic.com>
 2025-02-21 15:30:12 -08:00
Isotr0py b2c3fc5d65 [Bugfix][CPU] Fix cpu all-reduce using native pytorch implementation (#13586) 2025-02-20 22:24:17 -08:00
Gabriel Marinho 1c3c975766 [FEATURE] Enables /score endpoint for embedding models (#12846) 2025-02-20 22:09:47 -08:00
Nick Hill 31aa045c11 [V1][Sampler] Avoid an operation during temperature application (#13587) 2025-02-20 22:05:56 -08:00
Roger Wang a30c093502 [Bugfix] Add mm_processor_kwargs to chat-related protocols (#13644) 2025-02-20 22:04:33 -08:00
Lingfan Yu 33170081f1 [Neuron][Kernel] Vectorize KV cache load in FlashPagedAttention to maximize DMA bandwidth (#13245) 
Signed-off-by: Lingfan Yu <lingfany@amazon.com>
 2025-02-20 17:45:45 -08:00
Michael Goin 71face8540 [Bugfix] Fix max_num_batched_tokens for MLA (#13620) 
Signed-off-by: mgoin <mgoin64@gmail.com>
 2025-02-20 17:45:20 -08:00
Joe Runde bfbc0b32c6 [Frontend] Add backend-specific options for guided decoding (#13505) 
Signed-off-by: Joe Runde <Joseph.Runde@ibm.com>
 2025-02-20 15:07:58 -05:00
ajayvohra2005 6a417b8600 fix neuron performance issue (#13589) 2025-02-20 10:59:36 -08:00
Woosuk Kwon d3ea50113c [V1][Minor] Print KV cache size in token counts (#13596) 
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
 2025-02-20 09:24:31 -08:00
chenxiaobing ed6e9075d3 [Bugfix] Fix deepseekv3 grouped topk error (#13474) 
Signed-off-by: Chen-XiaoBing <chenxb002@whu.edu.cn>
 2025-02-20 06:47:01 -08:00
Varun Sundar Rabindranath b69692a2d8 [Kernel] LoRA - Refactor sgmv kernels (#13110) 2025-02-20 07:28:06 -05:00
Yuan Tang 3738e6fa80 [API Server] Add port number range validation (#13506) 
Signed-off-by: Yuan Tang <terrytangyuan@gmail.com>
 2025-02-20 15:05:13 +08:00
Gregory Shtrasberg 0023cd2b9d [ROCm] MI300A compile targets deprecation (#13560) 2025-02-19 23:05:00 -08:00
燃 041e294716 [Misc] add mm_processor_kwargs to extra_body for Qwen2.5-VL (#13533) 2025-02-19 23:04:30 -08:00
Alex Brooks 9621667874 [Misc] Warn if the vLLM version can't be retrieved (#13501) 
Signed-off-by: Alex-Brooks <Alex.brooks@ibm.com>
 2025-02-20 06:24:48 +00:00
Simon Mo 8c755c3b6d [bugfix] spec decode worker get tp group only when initialized (#13578) 2025-02-20 04:46:28 +00:00
youkaichaoandcennn ba81163997 [core] add sleep and wake up endpoint and v1 support (#12987) 
Signed-off-by: youkaichao <youkaichao@gmail.com>
Signed-off-by: cennn <2523403608@qq.com>
Co-authored-by: cennn <2523403608@qq.com>
 2025-02-20 12:41:17 +08:00
Divakar Verma 0d243f2a54 [ROCm][MoE] mi300 mixtral8x7B perf for specific BS (#13577) 
Signed-off-by: Divakar Verma <divakar.verma@amd.com>
 2025-02-20 04:01:02 +00:00
Jee Jee Li 512368e34a [Misc] Qwen2.5 VL support LoRA (#13261) 2025-02-19 18:37:55 -08:00
Kevin H. LuuandEC2 Default User 473f51cfd9 [3/n][CI] Load Quantization test models with S3 (#13570) 
Signed-off-by: <>
Co-authored-by: EC2 Default User <ec2-user@ip-172-31-20-117.us-west-2.compute.internal>
 2025-02-20 10:12:30 +08:00
Nick Hill a4c402a756 [BugFix] Avoid error traceback in logs when V1 LLM terminates (#13565) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-02-20 00:49:01 +00:00
Isotr0py 550d97eb58 [Misc] Avoid calling unnecessary hf_list_repo_files for local model path (#13348) 
Signed-off-by: isotr0py <2037008807@qq.com>
 2025-02-19 18:57:48 +00:00