[New model] Kimi K3 (#50000)
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com> Signed-off-by: Isotr0py <Isotr0py@outlook.com> Signed-off-by: Bugen Zhao <i@bugenzhao.com> Signed-off-by: Jee Jee Li <jeejeelee@inferact.ai> Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai> Signed-off-by: Yongye Zhu <zyy1102000@gmail.com> Signed-off-by: wzhao18 <wzhao18.sz@gmail.com> Co-authored-by: Isotr0py <Isotr0py@outlook.com> Co-authored-by: Thien Tran <gau.nernst@yahoo.com.sg> Co-authored-by: Bugen Zhao <i@bugenzhao.com> Co-authored-by: Yongye Zhu <zyy1102000@gmail.com> Co-authored-by: Ziming Huang <zelda.huanghuang@gmail.com> Co-authored-by: Roger Wang <hey@rogerw.io> Co-authored-by: Isotr0py <mozf@inferact.ai> Co-authored-by: aoshen02 <aoshen02@users.noreply.github.com> Co-authored-by: Woosuk Kwon <woosuk@inferact.ai> Co-authored-by: Jee Jee Li <jeejeelee@inferact.ai> Co-authored-by: aoshen02 <aoshen@inferact.ai> Co-authored-by: Summer Yang <girasoleyang@gmail.com> Co-authored-by: Kevin H. Luu <khluu000@gmail.com> Co-authored-by: Bowen Wang <abmfy@icloud.com> Co-authored-by: gnovack <novackgm@gmail.com> Co-authored-by: Nick Hill <nickhill123@gmail.com> Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai> Co-authored-by: xiaozhoupy <peiyuanzhou1994@gmail.com> Co-authored-by: Roy Wang <yasong.wang@inferact.ai> Co-authored-by: Jeff (Junze) Ma <93145857+majunze2001@users.noreply.github.com> Co-authored-by: Codex <codex@openai.com> Co-authored-by: Wei Zhao <51183510+wzhao18@users.noreply.github.com>
This commit is contained in:
co-authored by
Isotr0py
Thien Tran
Bugen Zhao
Yongye Zhu
Ziming Huang
Roger Wang
Isotr0py
aoshen02
Woosuk Kwon
Jee Jee Li
aoshen02
Summer Yang
Kevin H. Luu
Bowen Wang
gnovack
Nick Hill
Yifan Qiao
xiaozhoupy
Roy Wang
Jeff Ma
Codex
Wei Zhao
parent
0c64be8873
commit
aeeb36b1f1
@@ -27,8 +27,9 @@ def maybe_execute_in_parallel(
|
||||
"""Run two functions potentially in parallel on separate CUDA streams.
|
||||
|
||||
When aux_stream is provided, fn0 runs on the current (default) stream and
|
||||
fn1 runs on aux_stream, synchronized via CUDA events. When aux_stream is
|
||||
None, both functions execute sequentially on the current stream.
|
||||
fn1 runs on aux_stream, synchronized via CUDA events. When aux_stream is
|
||||
None or a breakable CUDA graph capture is active, both functions execute
|
||||
sequentially on the current stream.
|
||||
|
||||
This design follows TensorRT-LLM's maybe_execute_in_parallel pattern
|
||||
(tensorrt_llm/_torch/modules/multi_stream_utils.py).
|
||||
@@ -39,11 +40,18 @@ def maybe_execute_in_parallel(
|
||||
event0: CUDA event recorded before fn0 so aux_stream can wait.
|
||||
event1: CUDA event recorded after fn1 so default stream can wait.
|
||||
aux_stream: The second CUDA stream for fn1.
|
||||
Multi-stream is disabled when aux_stream is None.
|
||||
Multi-stream is disabled when aux_stream is None or a breakable
|
||||
CUDA graph capture is active.
|
||||
|
||||
Returns:
|
||||
Tuple of (fn0_result, fn1_result).
|
||||
"""
|
||||
if aux_stream is not None:
|
||||
from vllm.compilation.breakable_cudagraph import BreakableCUDAGraphCapture
|
||||
|
||||
if BreakableCUDAGraphCapture.is_active():
|
||||
aux_stream = None
|
||||
|
||||
if aux_stream is not None:
|
||||
event0.record()
|
||||
result0 = fn0()
|
||||
|
||||
Reference in New Issue
Block a user