Files

T

Woosuk Kwon a0a5178ab4 [Model Runner V2] Use ModelState.prepare_attn() for cuda graph capture [5/N] (#35774 )

Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>

2026-03-02 20:06:27 -08:00

metrics

[Model Runner V2] Support num NaNs in logits (#30187 )

2025-12-09 10:00:49 -08:00

[Model Runner V2] Minor refactoring for EncoderRunner (#35628 )

2026-03-01 00:15:39 -08:00

model_states

[Model Runner V2] Minor refactoring for EncoderRunner (#35628 )

2026-03-01 00:15:39 -08:00

pool

[Model Runner V2] Support pooling models (#35120 )

2026-02-27 18:03:01 -08:00

sample

[ModelRunnerV2] Rename sampler functions and variables for clarity (#35459 )

2026-03-02 19:48:56 -08:00

spec_decode

[Model Runner V2] Use ModelState.prepare_attn() for cuda graph capture [5/N] (#35774 )

2026-03-02 20:06:27 -08:00

__init__.py

GPU Model Runner V2 (#25266 )

2025-11-21 08:20:55 -08:00

async_utils.py

[Model Runner V2] Support pooling models (#35120 )

2026-02-27 18:03:01 -08:00

attn_utils.py

[Model Runner V2] Support attention group (#35036 )

2026-02-21 16:42:53 -08:00

block_table.py

[Model Runner V2] Use block table apis for capture inputs (#35671 )

2026-03-01 10:31:13 -08:00

buffer_utils.py

[Model Runner V2] Minor CPU optimizations (#34856 )

2026-02-19 16:05:37 -08:00

cp_utils.py

[Model Runner V2] Minor simplification for DCP (#34786 )

2026-02-18 11:04:53 -08:00

cudagraph_utils.py

[Model Runner V2] Use ModelState.prepare_attn() for cuda graph capture [5/N] (#35774 )

2026-03-02 20:06:27 -08:00

dp_utils.py

[Model Runner V2] support piecewise & mixed cudagraph (#32771 )

2026-02-18 15:03:17 -08:00

input_batch.py

[Model Runner V2] Use ModelState.prepare_attn() for cuda graph capture [5/N] (#35774 )

2026-03-02 20:06:27 -08:00

kv_connector.py

[Spec Decode] Defer clearing KV connector metadata for EAGLE3 speculative decode + prefill / decode disagg setup (#34529 )

2026-02-22 08:08:32 -08:00

lora_utils.py

[ModelRunner V2] Misc minor simplifications and optimizations (#33467 )

2026-02-01 22:17:14 +00:00

model_runner.py

[Model Runner V2] Use ModelState.prepare_attn() for cuda graph capture [5/N] (#35774 )

2026-03-02 20:06:27 -08:00

pp_utils.py

[Model Runner V2] A bit more PP simplification (#34766 )

2026-02-17 21:39:07 -08:00

README.md

GPU Model Runner V2 (#25266 )

2025-11-21 08:20:55 -08:00

states.py

[Model Runner V2] Avoid prepare prefill kernel launch overhead (#34780 )

2026-02-18 00:49:21 -08:00

structured_outputs.py

[ModelRunner V2] Misc minor simplifications and optimizations (#33467 )

2026-02-01 22:17:14 +00:00

warmup.py

[Model Runner V2] Warmup kernels (#35172 )

2026-02-27 10:43:30 -08:00

README.md

[Experimental] Model Runner V2

This directory contains the new model runner which is under active development. Ping Woosuk Kwon for any changes.