Files

T

Lucas WilkinsonandClaude 57bd0ed441 [5/N][KV-Cache Layout Refactor] Backend-published KV packing via customize_spec (#51704 )

Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
Co-authored-by: Claude <noreply@anthropic.com>

2026-08-14 09:47:55 -04:00

metrics

[Model Runner V2] Support num NaNs in logits (#30187 )

2025-12-09 10:00:49 -08:00

[MRV2][Multimodal] Enable encoder cuda graph for model runner v2 (#49852 )

2026-08-14 09:33:21 +00:00

model_states

[MRV2][Multimodal] Enable encoder cuda graph for model runner v2 (#49852 )

2026-08-14 09:33:21 +00:00

pool

[ModelRunner v2] Enable decoder token-wise pooling (#50931 )

2026-08-06 22:21:52 +00:00

sample

[Feature] Mask Replay (#49577 )

2026-08-13 08:44:31 +00:00

spec_decode

Add Muse Glimmer model support (#51655 )

2026-08-14 12:28:56 +08:00

__init__.py

GPU Model Runner V2 (#25266 )

2025-11-21 08:20:55 -08:00

async_utils.py

[Feature] Mask Replay (#49577 )

2026-08-13 08:44:31 +00:00

attn_utils.py

[5/N][KV-Cache Layout Refactor] Backend-published KV packing via customize_spec (#51704 )

2026-08-14 09:47:55 -04:00

block_table.py

[Model Runner v2] E/P/D disaggregation support (#38390 )

2026-08-04 15:52:23 +00:00

buffer_utils.py

[Core] Ensure memory is pinned prior to async h2d copy (#45424 )

2026-06-20 20:02:24 -07:00

cp_utils.py

[Model Runner V2] Minor simplification for DCP (#34786 )

2026-02-18 11:04:53 -08:00

cudagraph_utils.py

[Spec Decode] DSpark confidence-scheduled verification (#47808 )

2026-08-12 12:27:48 -07:00

dp_utils.py

[Spec Decode] DSpark confidence-scheduled verification (#47808 )

2026-08-12 12:27:48 -07:00

ec_connector.py

[Model Runner v2] E/P/D disaggregation support (#38390 )

2026-08-04 15:52:23 +00:00

eplb_utils.py

feat: extended EPLB support for Mistral Large 3 and additional MoE backends (#48355 )

2026-08-07 05:32:52 -07:00

input_batch.py

[Spec Decode] DSpark confidence-scheduled verification (#47808 )

2026-08-12 12:27:48 -07:00

kv_connector.py

[Core] Cleanup KVConnector handling with PP + fix MRV2 (#43732 )

2026-05-28 13:12:03 -07:00

lora_utils.py

[BugFix][MRV2] Don't create dummy requests longer than max_model_len (#49751 )

2026-07-27 02:04:20 +00:00

model_runner.py

[MRV2][Multimodal] Enable encoder cuda graph for model runner v2 (#49852 )

2026-08-14 09:33:21 +00:00

pcp_manager.py

[Bugfix][MRV2] Require all requests to be decoding for uniform-decode dispatch (#51865 )

2026-08-11 17:15:06 -07:00

pp_utils.py

[XPU] Fix PP accuracy on XPU device (#47253 )

2026-07-07 09:17:27 +08:00

README.md

GPU Model Runner V2 (#25266 )

2025-11-21 08:20:55 -08:00

shutdown.py

[MRV2] Add shutdown() method (#41297 )

2026-05-02 21:06:30 -07:00

states.py

[Model Runner V2][Spec Decode] Add multi-layer MTP speculator (#48892 )

2026-07-30 15:54:15 -07:00

structured_outputs.py

[CPU] Ship triton-cpu wheel and fix several hardcoded pin_memory=True (#52092 )

2026-08-13 14:40:53 +08:00

warmup.py

[Bugfix][MRV2] Reserve spec-decode lookahead blocks in V2 warmup (#51438 )

2026-08-08 23:58:43 +00:00

README.md

[Experimental] Model Runner V2

This directory contains the new model runner which is under active development. Ping Woosuk Kwon for any changes.