iHore/vLLM

Fork 0

Files

T

History

8322d4e47f Enable Cross layers KV cache layout at NIXL Connector V2 (#33339 )

Signed-off-by: Liran Schour <lirans@il.ibm.com>
Signed-off-by: liranschour <liranschour@users.noreply.github.com>
Co-authored-by: Or Ozeri <or@ozery.com>
Co-authored-by: Nicolò Lucchesi <nicolo.lucchesi@gmail.com>
Co-authored-by: Nicolò Lucchesi <nlucches@redhat.com>

2026-02-05 02:17:02 -08:00

quantization

[UX] Use gguf repo_id:quant_type syntax for examples and docs (#33371 )

2026-01-31 12:14:54 +08:00

spec_decode

[Docs] Adding links and intro to Speculators and LLM Compressor (#32849 )

2026-01-29 14:12:35 -08:00

automatic_prefix_caching.md

[Docs] Reduce custom syntax used in docs (#27009 )

2025-10-16 20:05:34 -07:00

batch_invariance.md

[Feature] Enable TRITON_ATTN for Batch Invariance (#33688 )

2026-02-04 13:27:34 +08:00

custom_arguments.md

[Doc]: fix typos in various files (#28811 )

2025-11-16 14:30:06 +00:00

custom_logitsprocs.md

[Docs] Use relative md links instead of absolute html links for cross referencing (#31494 )

2025-12-29 13:33:44 +00:00

disagg_encoder.md

Fix various typos found in docs (#32212 )

2026-01-13 03:41:47 +00:00

disagg_prefill.md

[P/D] rework mooncake connector and introduce its bootstrap server (#31034 )

2026-02-03 08:08:25 -08:00

interleaved_thinking.md

[Frontend] supports interleaved thinking (#28531 )

2025-11-13 16:14:13 +08:00

lora.md

[Misc] HF Hub LoRA Resolver (#20320 )

2026-01-26 13:56:32 +00:00

mooncake_connector_usage.md

[P/D] rework mooncake connector and introduce its bootstrap server (#31034 )

2026-02-03 08:08:25 -08:00

multimodal_inputs.md

[Refactor] Use data parser for matching data items to multi-modal UUIDs (#32955 )

2026-01-26 15:00:28 +08:00

nixl_connector_usage.md

Enable Cross layers KV cache layout at NIXL Connector V2 (#33339 )

2026-02-05 02:17:02 -08:00

prompt_embeds.md

[Frontend] Require flag for loading text and image embeds (#27204 )

2025-10-22 15:52:02 +00:00

README.md

[Docs] Adding links and intro to Speculators and LLM Compressor (#32849 )

2026-01-29 14:12:35 -08:00

reasoning_outputs.md

[Frontend] Introduce Renderer for processing chat messages (using ModelConfig) (#30200 )

2026-01-22 12:44:22 +00:00

sleep_mode.md

[Core][AMD] Migrate fully transparent sleep mode to ROCm platform (#12695 )

2025-11-12 15:24:12 -08:00

structured_outputs.md

Fix various typos found in docs (#32212 )

2026-01-13 03:41:47 +00:00

tool_calling.md

[GLM-4.7] GLM Model support for GLM-Lite (#31386 )

2026-01-19 01:18:38 -08:00

README.md

Features

Compatibility Matrix

The tables below show mutually exclusive features and the support on some hardware.

The symbols used have the following meanings:

✅ = Full compatibility
🟠 = Partial compatibility
❌ = No compatibility
❔ = Unknown or TBD

!!! note Check the ❌ or 🟠 with links to see tracking issue for unsupported feature/hardware combination.

Feature x Feature

Feature	CP	APC	LoRA	SD	CUDA graph	pooling	enc-dec	logP	prmpt logP	async output	multi-step	mm	best-of	beam-search	prompt-embeds
CP	✅
APC	✅	✅
LoRA	✅	✅	✅
SD	✅	✅	❌	✅
CUDA graph	✅	✅	✅	✅	✅
pooling	🟠*	🟠*	✅	❌	✅	✅
enc-dec	❌	❌	❌	❌	✅	✅	✅
logP	✅	✅	✅	✅	✅	❌	✅	✅
prmpt logP	✅	✅	✅	✅	✅	❌	✅	✅	✅
async output	✅	✅	✅	❌	✅	❌	❌	✅	✅	✅
multi-step	❌	✅	❌	❌	✅	❌	❌	✅	✅	✅	✅
mm	✅	✅	🟠^{^}	❔	✅	✅	✅	✅	✅	✅	❔	✅
best-of	✅	✅	✅	❌	✅	❌	✅	✅	✅	❔	❌	✅	✅
beam-search	✅	✅	✅	❌	✅	❌	✅	✅	✅	❔	❌	❔	✅	✅
prompt-embeds	✅	✅	✅	❌	✅	❌	❌	✅	❌	❔	❔	❌	❔	❔	✅

* Chunked prefill and prefix caching are only applicable to last-token or all pooling with causal attention.
^{^} LoRA is only applicable to the language backbone of multimodal models.

Feature x Hardware

Feature	Volta	Turing	Ampere	Ada	Hopper	CPU	AMD	Intel GPU
CP	❌	✅	✅	✅	✅	✅	✅	✅
APC	❌	✅	✅	✅	✅	✅	✅	✅
LoRA	✅	✅	✅	✅	✅	✅	✅	✅
SD	✅	✅	✅	✅	✅	❌	✅	✅
CUDA graph	✅	✅	✅	✅	✅	❌	✅	❌
pooling	✅	✅	✅	✅	✅	✅	✅	✅
enc-dec	✅	✅	✅	✅	✅	✅	❌	✅
mm	✅	✅	✅	✅	✅	✅	✅	✅
prompt-embeds	✅	✅	✅	✅	✅	✅	❔	✅
logP	✅	✅	✅	✅	✅	✅	✅	✅
prmpt logP	✅	✅	✅	✅	✅	✅	✅	✅
async output	✅	✅	✅	✅	✅	❌	❌	✅
multi-step	✅	✅	✅	✅	✅	❌	✅	✅
best-of	✅	✅	✅	✅	✅	✅	✅	✅
beam-search	✅	✅	✅	✅	✅	✅	✅	✅

!!! note For information on feature support on Google TPU, please refer to the TPU-Inference Recommended Models and Features documentation.