iHore/vLLM

Fork 0

Files

T

History

MOVZXandHinata 43fbd6510b

[Qwen4Exp] Serve PLE rows from the loader's mapped checkpoint shards (PR #58815 )

Adds an opt-in backend, --engram-config '{"host_file_gather": true}', that
serves Qwen4Exp PLE rows from the checkpoint files through the page cache
instead of allocating the 47.7 GiB table on GPU or in pinned host memory.

- Qwen4ExpPLEFileGatherEmbedding allocates no table rows; load_weights keeps
  the default loader's zero-copy safetensors views of each shard. At bind
  (post-load hook) every view must be file-backed per /proc/self/maps (HF
  cache symlinks work; heap/pinned/anonymous memory is rejected) and the
  shards must cover the whole table.
- Each step, Qwen4ExpModelState.prepare_inputs computes the PLE n-gram ids,
  copies them to pinned host memory, syncs the stream once, issues
  POSIX_FADV_WILLNEED over the coalesced page ranges of the step's rows,
  gathers the rows on the host, and copies them into a fixed device staging
  buffer. The forward reads only the staging buffer, so FULL CUDA graph
  capture is unaffected.
- Rows stay in the checkpoint dtype (FP8 table stays FP8, dequantized on the
  GPU); dummy loads serve zeros; weight reload is rejected; takes precedence
  over cpu_offload; rejected on ROCm and with elastic EP.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>

2026-09-27 09:54:42 +07:00

quantization

[Cleanup] Remove online quantization support in fp8.py in favor of online shorthands (#53585 )

2026-09-24 12:48:14 -07:00

speculative_decoding

[SpecDecode] Add LiLiCorr drafter (#57934 )

2026-09-25 10:08:28 -04:00

automatic_prefix_caching.md

[Misc] Rename --enable-mamba-fine-grained-prefix-cache (#53945 follow-up) (#57382 )

2026-09-20 09:50:34 -07:00

batch_invariance.md

[Perf] Use breakable CUDA graphs (no torch.compile) by default under VLLM_BATCH_INVARIANT so the tuned matmul configs see the runtime M (#57586 )

2026-09-23 14:39:38 -04:00

context_extension.md

[Examples] Resettle features examples. (#40995 )

2026-04-28 00:33:41 -07:00

cross_encoder_cache.md

[Feature][Multimodal] Add cross-encoder caching to Mooncake P2P (#56242 )

2026-09-17 13:16:41 +08:00

custom_arguments.md

Forward Anthropic vllm_xargs to sampling params (#53308 )

2026-08-21 15:23:21 -07:00

custom_logitsprocs.md

[Model Runner V2] Support custom logits processors (#56497 )

2026-09-19 17:46:49 +00:00

disagg_encoder.md

[EPD] Support metadata-only audio inputs (#57887 )

2026-09-23 09:03:50 +08:00

disagg_prefill.md

[Frontend] Reuse prefill token ids on the decode chat path for disaggregated serving (#48145 )

2026-07-29 10:02:57 +02:00

ec_cpu_connector.md

[Docs][EC Connector] CPU EC Connector usage Docs (#54522 )

2026-09-08 17:43:08 -07:00

engram.md

[Qwen4Exp] Serve PLE rows from the loader's mapped checkpoint shards (PR #58815 )

2026-09-27 09:54:42 +07:00

index_cache.md

[Feature]: IndexCache support for DSA models (#37735 )

2026-04-29 15:15:35 -04:00

initialized_snapshots.md

[Frontend] Add reusable TP1 initialized-engine snapshots (#51360 )

2026-09-21 23:18:49 +00:00

interleaved_thinking.md

Allow markdownlint to run locally (#36398 )

2026-03-08 20:05:24 -07:00

kv_offloading_usage.md

[KV Offload] Add per-request max_load_tokens control (#55885 )

2026-09-16 06:57:38 +03:00

lora.md

[LoRA] Support modules_to_save for sequence classification (#53555 )

2026-09-16 07:15:42 -07:00

mooncake_connector_usage.md

[Docker][KVConnector] Install mooncake from official wheels instead of a custom build (#51067 )

2026-08-05 11:58:44 -07:00

mooncake_store_connector_usage.md

[KV Connector] Support heterogeneous TP sharing in Mooncake Store Connector (#53129 )

2026-09-01 10:18:06 -07:00

moriio_connector_usage.md

[ROCm][Docker] Pin AINIC apt repo to snapshot 1.117.5-a-77 (#56459 )

2026-09-11 14:14:16 +00:00

multimodal_inputs.md

[Bugfix][Audio] Restore soundfile-first automatic decoding (#55642 )

2026-09-07 06:21:02 +00:00

nixl_connector_compatibility.md

[KVConnector][NIXL] Support attention-HMA layouts in pipeline-parallel push prefill (#50494 )

2026-09-15 15:43:56 -07:00

nixl_connector_usage.md

[Nixl] Separate transport-failure metrics from KV expiry (#55854 )

2026-09-18 18:08:59 +02:00

per_request_metrics.md

[Frontend] Add per-request metrics to Responses API (#55084 )

2026-09-16 22:52:05 +00:00

preload.md

[UX][Frontend] Introduce vllm preload cli for fast restart (#56680 )

2026-09-25 06:56:39 +00:00

prompt_embeds.md

feat: Enable prompt_embeds Content Part Support in vLLM Chat Completions API (#40720 )

2026-05-01 10:05:55 +08:00

README.md

[CI][CPU] Add speculative-decoding coverage to CPU CI (#54934 )

2026-09-14 19:51:38 +08:00

reasoning_outputs.md

[Feature][Frontend] Add granite_thinking_parser reasoning parser for Granite 4.2 (#55957 )

2026-09-25 20:43:08 +00:00

sleep_mode.md

[RL][Sleep] Retain frozen weights across level-2 sleep (#57891 )

2026-09-22 12:36:55 +08:00

structured_outputs.md

[Docs] Reorganize online serving docs. (#41907 )

2026-05-19 14:43:18 +08:00

tool_calling.md

[Frontend] Add --tool-strict-level for server-side control for structural tag activation (#56268 )

2026-09-18 04:32:39 +00:00

watermarking.md

[Rust Frontend] Forward per-request watermarking controls (#56338 )

2026-09-14 23:17:02 +08:00

README.md

Features

Compatibility Matrix

The tables below show mutually exclusive features and the support on some hardware.

The symbols used have the following meanings:

✅ = Full compatibility
🟠 = Partial compatibility
❌ = No compatibility
❔ = Unknown or TBD

!!! note Check the ❌ or 🟠 with links to see tracking issue for unsupported feature/hardware combination.

Feature x Feature

Feature	CP	APC	LoRA	SD	CUDA graph	pooling	enc-dec	logP	prmpt logP	async output	multi-step	mm	best-of	beam-search	prompt-embeds
CP	✅
APC	✅	✅
LoRA	✅	✅	✅
SD	✅	✅	❌	✅
CUDA graph	✅	✅	✅	✅	✅
pooling	🟠*	🟠*	✅	❌	✅	✅
enc-dec	❌	❌	❌	❌	✅	✅	✅
logP	✅	✅	✅	✅	✅	❌	✅	✅
prmpt logP	✅	✅	✅	✅	✅	❌	✅	✅	✅
async output	✅	✅	✅	❌	✅	❌	❌	✅	✅	✅
multi-step	❌	✅	❌	❌	✅	❌	❌	✅	✅	✅	✅
mm	✅	✅	🟠^{^}	❔	✅	✅	✅	✅	✅	✅	❔	✅
best-of	✅	✅	✅	❌	✅	❌	✅	✅	✅	❔	❌	✅	✅
beam-search	✅	✅	✅	❌	✅	❌	✅	✅	✅	❔	❌	❔	✅	✅
prompt-embeds	✅	✅	✅	❌	✅	❌	❌	✅	❌	❔	❔	✅	❔	❔	✅

* Chunked prefill and prefix caching are only applicable to last-token or all pooling with causal attention.
^{^} LoRA is only applicable to the language backbone of multimodal models.

Feature x Hardware

Feature	Volta	Turing	Ampere	Ada	Hopper	CPU	AMD	Intel GPU
CP	❌	✅	✅	✅	✅	✅	✅	✅
APC	❌	✅	✅	✅	✅	✅	✅	✅
LoRA	✅	✅	✅	✅	✅	✅	✅	✅
SD	✅	✅	✅	✅	✅	✅	✅	✅
CUDA graph	✅	✅	✅	✅	✅	❌	✅	❌
pooling	✅	✅	✅	✅	✅	✅	✅	✅
enc-dec	✅	✅	✅	✅	✅	✅	❌	✅
mm	✅	✅	✅	✅	✅	✅	✅	✅
prompt-embeds	✅	✅	✅	✅	✅	✅	❔	✅
logP	✅	✅	✅	✅	✅	✅	✅	✅
prmpt logP	✅	✅	✅	✅	✅	✅	✅	✅
async output	✅	✅	✅	✅	✅	❌	❌	✅
multi-step	✅	✅	✅	✅	✅	❌	✅	✅
best-of	✅	✅	✅	✅	✅	✅	✅	✅
beam-search	✅	✅	✅	✅	✅	✅	✅	✅

!!! note For information on feature support on Google TPU, please refer to the TPU-Inference Recommended Models and Features documentation.