Compare commits

..
852 Commits
Author SHA1 Message Date
MOVZXandHinata 61fbe0b865 [Mini-VLLM] Remove unreferenced definitions found by static scan
A symbol scan over every tracked file found 123 top-level definitions in
vllm/ with zero references, including references inside strings. All of
them came from features this fork deleted earlier.

- Delete dead attention helpers: MLADCPManager, split_prefill_chunks, the
  reshape helpers, compute_varlen_chunk_metadata, the LINEAR backend's
  unused BailingLinearAttentionBackend, and the sparse flash-attn entry
  points.
- Delete the CPU and AMX op wrappers from _custom_ops.py: cpu_mla_decode,
  cpu_mla_extend, cpu_mla_get_scheduler_metadata,
  gather_mla_context_cache_cpu.
- Delete dead quantization helpers: awq_pack, gptq_pack,
  gptq_quantize_weights, get_and_maybe_dequant_weights,
  marlin_quant_fp8_torch, the rand_marlin_weight_*_like pair,
  ocp_mx_weight_dtype_and_rows, get_fp8_block_weight_scale,
  input_to_float8.
- Delete dead model helpers: resolve_visual_encoder_outputs,
  run_dp_sharded_vision_model, fast_topk, get_layer_index,
  is_pp_missing_parameter, skip_spec_layers, KimiK3MergedQKVGateLinear,
  selective_scan_fn, causal_conv1d_update_torch.
- Delete dead utils: format_kib, format_mib, chunk_list, prev_power_of_2,
  split_host_port, join_host_port, make_zmq_path, make_async_with_semaphore,
  is_meta_module, is_non_overlapping_and_dense, supports_xpu_graph,
  create_kv_caches_with_random*, and the unused has_* capability probes.
- Delete the unreachable deep-gemm mega-MoE warmup path, which imported
  vllm.models.deepseek_v4.nvidia.model and vllm.models.kimi_k3.nvidia.model,
  neither of which exists anymore.
- Drop the imports and the two now-empty guards left behind.

Verified statically: every changed file compiles, no unused import
remains, and no deleted name is referenced in vllm, tests or csrc. The
runtime gates still need the Python environment rebuilt.

Co-authored-by: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-19 21:39:56 +07:00
MOVZXandHinata 5aa2ac64f8 [Mini-VLLM] Remove LoRA, plugins, speech-to-text and scale-out surfaces
Production runs one static model behind the OpenAI-compatible API. Every
adapter path and non-OpenAI HTTP surface is unreachable there.

- Delete the LoRA stack: vllm/lora, LoRARequest on the engine wire, the
  scheduler, worker, cudagraph and MoE plumbing, LoRA metrics, the lora
  CLI group, and the LoRA tests.
- Drop the plugin loader in vllm/plugins. Platform resolution is now
  CUDA-only and the general/stat-logger plugin hooks are gone.
- Delete the speech-to-text and realtime entrypoints, the scale-out
  render routes, and vllm/config/speech_to_text.py. GenerationTask is
  generate-only.
- Cut DecodeBenchConnector and the disagg-prefill doc and image. The
  Multi, Offloading and Example connectors stay; production uses the
  CPU KV offload tier.
- Delete the ROCm-only fused KDA chunk kernel (2346 lines) with its CMake
  arch gate, declarations and op registrations. The CUDA build never
  defined the guard and no Python caller existed.
- Delete the orphan Mistral and Pixtral chat fixtures and examples/.
- Fix _check_model: removing the LoRA early return made every request
  fail with "The model ... does not exist. Valid aliases: ...".

Verified on an RTX 3090: import sweep reports 1242 modules with 7 known
optional-dependency failures; a live server answers chat, completions,
responses, tokenize and detokenize with zero log errors and 404 for the
removed surfaces; collection yields 8558 tests with the 3 pre-existing
errors; ruff adds no new finding; the incremental _C build succeeds.

Co-authored-by: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-19 21:06:19 +07:00
MOVZXandHinata 5f3c7bd8ab [Mini-VLLM] Remove the ROCm device path, debug routes and arch-gated tests
- Delete csrc/rocm, hipify, the _rocm_C extension and the setup.py rocm
  target path; drop the rocm requirement files
- Delete the aiter, rdna3, skinny-gemm and ROCm fused-MoE kernel modules
  and retarget the linear kernel registries away from them
- Reduce platforms/rocm.py to the GCN architecture predicates that live
  code still imports, and delete the RocmPlatform class and plugin
- Drop the VLLM_SERVER_DEV_MODE debug surface: reset, collective_rpc,
  server_info and sleep routes plus the log_error_stack flag
- Delete lm-format-enforcer, lark and otel-semantic-conventions-ai pins;
  the live server process maps none of them
- Delete tests that only build architectures this fork no longer
  registers, and test directories left without any test file
- Restore the tests/v1/kv_connector mock helpers that scheduler tests
  import at module level

Co-authored-by: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-19 19:31:07 +07:00
MOVZXandHinata 2066d3143f [Mini-VLLM] Cut non-OpenAI surfaces, dead kernels, Mistral stack
- Remove Anthropic, Cohere, SageMaker, decisions, structured-decisions,
  rl, render-launcher and batch entrypoints; keep scale-out token routes
- Drop the Swagger static bundle and the /docs, /redoc routes
- Delete device-keyed tuning configs for fused_moe, quant utils, mamba
  select-state and helion; no sm86 entry ever existed for any of them
- Remove ten unused rope variants plus the fbgemm_fp8 and fp_quant
  quantizers and their config, metrics and loader plumbing
- Cut the CPU device build: csrc/cpu, cpu_extension.cmake, cpu
  requirements and the setup.py cpu target path
- Delete 29 environment variables nothing reads and 14 chat-template
  fallbacks for models this fork no longer registers
- Remove nine dead CUDA ops (deepseek_v4, kimi_k3, minimax, muse_spark,
  dsv3 gemm) with their kernels, bindings, wrappers and CMake entries
- Delete the connector test suites that targeted removed connectors
- Cut the Mistral tokenizer and parser stack, its tests, and the
  mistral_common and ijson dependency pins
- Restore the Qwen4Exp registry rows and nvidia/mtp.py that an earlier
  cut deleted while keeping the model tree

Co-authored-by: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-19 17:10:56 +07:00
MOVZXandHinata bd70a61c20 [Mini-VLLM] Remove orphan subsystems: snapshot, CPU workers, dead decoders
- Delete the snapshot package, CLI command, and its launch tests
- Delete CPU workers, cpu_communicator, and the hw_agnostic layer package
- Delete the EC CPU connector, keeping the EC base glue intact
- Delete dead composite/triton_flashinfer backends and third_party/flashmla
- Delete dead video decoders, video-bound tests, and orphan utils
- Remove the deep_gemm_mega_moe option and dead allreduce fusion pair

Co-authored-by: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-19 09:27:06 +07:00
MOVZXandHinata 2b544b7ba0 [Mini-VLLM] Cut dead subsystems across serving, connectors, parsers
- Remove P/D transfer connectors: nixl, nixl_chat, mooncake x2, hf3fs,
  moriio, hisparse, example and hidden-states connectors
- Reduce weight_transfer to the base API used by model loading
- Remove kv_offload tiering, simple_kv_offload, ec cpu/example,
  snapshot command, engine snapshot server
- Remove transformers model fallback, dsv4 tokenizers, dead platforms
  cpu/tpu/xpu, flashmla third_party, dsa, megamoe/flydsl, dead attn
  ops, dead layers (short_conv, fused_embed_norm, indexer_topk,
  relu2 gemma_ar, lightning_attn, minimax_rms_norm), gemm_rs_ar
- Keep MoE LoRA op (imported eagerly by punica_gpu); drop fp8 variant
- Rebuild parser hubs around Qwen3 adapters only (36 dead adapters)
- Remove dead tests with the code they covered

Co-authored-by: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-19 02:19:03 +07:00
MOVZXandHinata 54a396586e [Mini-VLLM] Repair test suite after SM86 cut
- Restore example connector, test helpers, and dropped patch tests
- Guard MLA metadata import in the example connector
- Point the PLE test at common.ops.ple after upstream #60021 move
- Cut tests for removed features: tools, DRY, epd proxy, STT batch,
  diffusion scheduler, INC/Quark PLE cases, mamba-2 mixer

Co-authored-by: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-19 01:06:30 +07:00
MOVZXandHinata 0152af1336 [Mini-VLLM] Remove upstream CI, docs, and lint surfaces
- Delete .buildkite: fork runs no Buildkite pipelines
- Delete mkdocs.yaml, .readthedocs.yaml: docs/ is gone
- Delete rust-toolchain.toml: Rust frontend is gone
- Delete .pre-commit-config.yaml: all hooks call deleted tools/pre_commit
- Delete .dockerignore: docker/ is gone
- Drop three tests importing deleted quark and mooncake modules

Co-authored-by: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-19 00:43:46 +07:00
MOVZXandHinata e3af324d8a [Mini-VLLM] Drop upstream model recipes directory
- recipes/ feeds the public recipes.vllm.ai site only
- no runtime, build, or lint consumer remains in this tree
- remove its typos exclusion from pyproject.toml

Co-authored-by: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-19 00:39:59 +07:00
MOVZXandHinata 8dfaffb700 [Bugfix] Restore xpu_communicator for CUDA graph capture
parallel_state.graph_capture() imports xpu_communicator on CUDA too.
The SM86 cut removed the file and capture crashed at startup.

Co-authored-by: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-19 00:33:19 +07:00
MOVZXandHinata 81074e94a3 [Mini-vLLM] Cut dead subsystems: models, entrypoints, backends, utils, tests
- Drop dead model modules, keeping files live imports require
- Remove anthropic, cohere, codegen entrypoints and batch STT modes
- Delete unused attention backends, MoE experts, and NPU/XPU util leaves
- Strip dead third_party trees and ROCm leaves with no CUDA path
- Remove mooncake EC connectors: dependency missing since branch base
- Re-audit verified by full import sweep and Qwen3.5 smoke test

Co-authored-by: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-19 00:20:48 +07:00
MOVZXandHinata 6a95f62e2f [Bugfix] Load checkpoint KV-cache scales in the Qwen3.5 family
Checkpoints that ship calibrated k_scale/v_scale tensors (e.g.
unsloth/Qwen3.8-27B-NVFP4, which carries one pair per full-attention
layer) name them ...self_attn.{k,v}_scale, but the runtime parameters
live on the inner attention layer (...self_attn.attn.{k,v}_scale).
Qwen3_5Model.load_weights hands the iterable straight to
AutoWeightsLoader, which performs no scale-name remapping, so the
scales are silently dropped and --kv-cache-dtype fp8 runs at scale 1.0
with only a warning_once that fires even on success (the MTP draft
layer legitimately has no scales).

Route scale-suffixed names through maybe_remap_kv_scale_name first,
mirroring gemma4/gpt_oss. No behavior change for checkpoints without
scales.

Measured on RTX 5090 / Qwen3.8-27B NVFP4 / fp8 KV / MTP k=3 (GSM8K
greedy, n=300): 92.3% at scale 1.0 vs 93.7% with scales loaded
(bf16-KV control 94.5%).

Fixes #54623

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-18 17:19:34 +00:00
MOVZXandHinata f36850e389 [Mini-VLLM] Restore FlashInfer backend for FP8 KV & only probe quantization overrides that exist
FlashInfer is the only backend serving fp8_e4m3 KV cache on compute
capability 8.6; the SM86 cut removed it and the Triton fallback
refuses FP8 KV, crashing worker startup. Restore flashinfer.py and
the triton_flashinfer shim.

The SM86 cut removed the inc and deepseek_v4_fp8 loaders but
ModelConfig still probed them, crashing compressed-tensors
checkpoints with "Invalid quantization method: inc". Filter the
override probe list to the supported methods.

Co-authored-by: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-18 23:31:30 +07:00
MOVZXandHinata e61dcb9143 [Mini-vLLM] SM86-only cut: models, backends, Rust, tests
- Keep only Qwen3.5, Qwen3.5-MTP and Qwen4Exp model paths; trim the model registry to 41 entries
- Delete idle attention backends (MLA, FlashInfer, ROCm-only, quant backends) and dead GDN CuTeDSL kernels
- Remove inc/quark quant loaders; keep compressed-tensors and Marlin
- Drop the Rust frontend, pooling entrypoints, and idle test suites
- Rebuild kernels for sm_80;sm_86 and strip the extensions
- Verified: full import sweep clean (optional deps only) and a dummy-load Qwen3.5 engine run on RTX 3090 via the Triton GDN path

Co-authored-by: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-18 19:19:56 +07:00
MOVZX c5ab42bc0f Merge branch 'main' into patches 2026-10-18 16:04:36 +07:00
Louie Tsai 1b2a666391 [Doc][Recipe] Fix for redhat fp8 xeon support issue (#62245) 2026-10-18 16:46:56 +08:00
83c9a547bf [CI][Release] Fix release wheel builds: stop excluding tracked recipes/ from Docker context (#62221)
Signed-off-by: khluu <khluu000@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: vllm-agent <claw@inferact.ai>
2026-10-18 00:48:00 -07:00
b510857d37 [Bugfix][Build] Build fused GDN MTP decode for SM80-SM90 with CUDA < 13 (#60896)
Signed-off-by: kimseunghyun <incongnito12@gmail.com>
Signed-off-by: kimseunghyun <hellonypcteam@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Shengqi Chen <harry-chen@outlook.com>
Co-authored-by: kimseunghyun <hellonypcteam@gmail.com>
2026-10-18 15:14:06 +08:00
7c9042c05a [Bugfix][MRV2] Fix speculative warmup deadlock with pipeline parallelism (#62129)
Signed-off-by: chenjr97 <chenjr97@mail2.sysu.edu.cn>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: vllm-agent <claw@inferact.ai>
2026-10-17 23:51:32 -07:00
da9134bc89 [ROCm][CI] Give the MoE parallel-launch watchdog room for AITER JIT builds (#62161)
Signed-off-by: Andreas Karatzas <AndreasKaratzas@users.noreply.github.com>
Co-authored-by: Andreas Karatzas <AndreasKaratzas@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
2026-10-17 21:49:27 -05:00
b2456c622b [ROCm][CI] Make ROCm 10.0 the default image; keep ROCm 7.2 as -rocm72 (#58761)
Signed-off-by: Rohan Potdar <rohan.potdar@amd.com>
Signed-off-by: Rohan138 <rohanpotdar138@gmail.com>
Signed-off-by: Micah Williamson <micah.williamson@amd.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Micah Williamson <micah.williamson@amd.com>
Co-authored-by: Kevin H. Luu <khluu000@gmail.com>
Co-authored-by: Andreas Karatzas <akaratza@amd.com>
2026-10-17 21:43:59 -05:00
03ee2e134d [CI][ROCm] Fix order-dependent test_rocm_aiter_sampler_defers_import (CPU tensors hit Triton) (#62213)
Signed-off-by: khluu <khluu000@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Andreas Karatzas <akaratza@amd.com>
2026-10-18 01:10:08 +00:00
djramicandAndreas Karatzas 9c778e9ff8 [Bugfix][EPLB] Wait for pending expert buffer copies before NIXL READs (#62081)
Signed-off-by: Djordje Ramic <djoramic@amd.com>
Co-authored-by: Andreas Karatzas <akaratza@amd.com>
2026-10-17 18:18:02 -05:00
7879f3687b [NIXL] Drop unreachable producer-side notif counting in push mode (#62210)
Signed-off-by: David Cheung <d7cheung@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: vllm-agent <claw@inferact.ai>
2026-10-17 21:24:35 +00:00
Kevin H. LuuandClaude Opus 5.5 255f14599e [Bugfix][Rust Frontend] Fix macOS build: read inherited listener family from its address (#62177)
Signed-off-by: khluu <khluu000@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 11:56:28 -07:00
03e8300d2c [Bugfix] Enable inductor batch-invariant mode under VLLM_BATCH_INVARIANT (#61118)
Signed-off-by: Yifan Feng <evanfengff@gmail.com>
Signed-off-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com>
Co-authored-by: Kevin H. Luu <khluu000@gmail.com>
2026-10-17 14:53:06 -04:00
Afonso RibeiroandFlora Feng 5c1c903501 [Perf][Parser] Stream Qwen3 tool argument values incrementally (12 KB call: 2.1 s to 11 ms) (#62027)
Signed-off-by: Afonso Ribeiro <afonso.diamantino2@gmail.com>
Co-authored-by: Flora Feng <4florafeng@gmail.com>
2026-10-17 18:49:06 +00:00
Tyrone bafb263664 [Bugfix][Benchmarks] Name the real cause when the /tokenize alignment probe fails (#59888) 2026-10-17 14:47:52 -04:00
Nick HillandClaude Opus 5.5 0c5dd18e7a [MRV2] Collect KV transfer results before the drafter runs (#62041)
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-17 10:27:57 -07:00
Ting SUNmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>Wentao Ye
763e3f7c91 [Bugfix][Spec Decode] Disable dynamic speculative decoding for fixed-K proposers instead of crashing (#46399)
Signed-off-by: Ting Sun <suntcrick@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com>
2026-10-17 12:41:04 -04:00
AlecandClaude Opus 5.5 e744eacf68 [Rust Frontend] Validate bad_words limits during request lowering (#61933)
Signed-off-by: Alec Flowers <aflowers@nvidia.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 09:10:40 -07:00
c0b6f74bee [Benchmark] Add per-case kernel profiles to vllm bench forward (#61692)
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: vllm-agent <claw@inferact.ai>
2026-10-17 08:27:11 -07:00
siyuandXun Sun c61a1e73c6 [Feature] Skip MLA weight absorption when weights are pre-processed (#57320)
Co-authored-by: Xun Sun <UNIDY2002@outlook.com>
2026-10-17 23:10:04 +08:00
aoshen02andClaude Opus 5.5 237d99b4af Revert "[Docs] Add train-inference consistency agent skill" (#61551) (#62186)
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 13:54:30 +00:00
aoshen02andClaude Opus 5.5 2d8efee9a7 [Docs] Add train-inference consistency agent skill (#61551)
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 20:49:50 +08:00
Roy WangandClaude Opus 5.5 21e09ea959 [Doc] Sync DeepSeek-V4.1-Flash recipe with vllm-project/recipes#1136, #1137 (#62179)
Signed-off-by: esmeetu <jasonailu87@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 12:27:52 +00:00
837e87e1c5 [Feature][LoRA][MoE] Expert LoRA for FlashInfer TRTLLM MXFP4 experts (#61979)
Signed-off-by: aoshen524 <aoshen524@gmail.com>
Co-authored-by: aoshen524 <aoshen524@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 20:17:56 +08:00
liuzhenweiandKunshang Ji 4e964b2623 [XPU] Fix Event init failure w/ external (#61743)
Signed-off-by: zhenwei-intel <zhenwei.liu@intel.com>
Co-authored-by: Kunshang Ji <kunshang.ji@intel.com>
2026-10-17 19:10:39 +08:00
Vadim GimpelsonandClaude ec8b20fbf0 [Bugfix] Fix Gemma 4 encoder CUDA graph capture OOM at startup (#61365)
Signed-off-by: Vadim Gimpelson <vadim.gimpelson@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
2026-10-17 11:04:49 +00:00
2b85d9cb65 [Perf][Bugfix] Prefer MNNVL for single-node FlashInfer allreduce again (#61989)
Signed-off-by: khluu <khluu000@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Wei Zhao <51183510+wzhao18@users.noreply.github.com>
2026-10-17 02:55:28 -07:00
026f6ab268 [Model] DeepSeek-V4/V4.1 VL: stream language-model weights instead of sorting the checkpoint (#58857)
Signed-off-by: git bisector <gitbisector@gmail.com>
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai>
2026-10-17 09:37:56 +00:00
Roy WangandClaude Opus 5.5 b515648641 [Doc] Host model recipes in recipes/ (#61302)
Signed-off-by: esmeetu <jasonailu87@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 09:14:36 +00:00
f98ae6a9fd [Bugfix][Core] Keep resumable sessions that stop while preempted (#58778)
Signed-off-by: David Cheung <d7cheung@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Yifan Zong <yzong@redhat.com>
2026-10-17 09:08:14 +00:00
2067297b2f [ROCm][Perf] Use AITER tuned configs to select a4w4 gemm kernel instead of env var (#61278)
Signed-off-by: Andy Friedrich <afriedri@amd.com>
Signed-off-by: afriedri <afriedri@amd.com>
Co-authored-by: Simon Danielsson <70206058+simondanielsson@users.noreply.github.com>
Co-authored-by: Kevin H. Luu <khluu000@gmail.com>
Co-authored-by: Andreas Karatzas <akaratza@amd.com>
2026-10-17 04:00:28 -05:00
Aarushi JainandAndreas Karatzas 3bf25c5097 [CI][Bugfix] Ignore AMD CI webhook no-op builds in /amd-ci run and retry (#62087)
Co-authored-by: Andreas Karatzas <akaratza@amd.com>
2026-10-17 03:54:31 -05:00
Jee Jee Li e56b1603af [Kernel]Add norm fused kernel for Step-5 Preview and Muse Spark Open (#62132)
Signed-off-by: Jee Jee Li <jeejeelee@inferact.ai>
2026-10-17 01:53:00 -07:00
Ting SUN eb8de8d1a1 [Bugfix][OpenAI] Respect allowed_token_ids in beam search (#47630)
Signed-off-by: Ting Sun <suntcrick@gmail.com>
2026-10-17 08:50:40 +00:00
dskwe b05ede5616 [Bugfix][Frontend] Preserve token IDs in completion derender responses (#62114)
Signed-off-by: Yuan Li <dskwelmcy@163.com>
2026-10-17 08:27:48 +00:00
61bb590877 [Bugfix][Spec Decode] Clear stale shared block-table rows (#61174)
Signed-off-by: Cheng Rui <286040359@qq.com>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Giancarlo Delfin <32987265+TheEpicDolphin@users.noreply.github.com>
2026-10-17 01:25:02 -07:00
c3de0b37e2 [Core][Spec Decode] Replace EAGLE group heuristics with explicit draft-layer tags (#61455)
Signed-off-by: Dakai An <dakaian108@gmail.com>
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai>
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 08:22:33 +00:00
MOVZX e0fe5df731 Merge branch 'main' into patches 2026-10-17 08:06:56 +00:00
08a65e287e [Bugfix][Kernel] hadacore: don't load past the input on a partial last warp (#61944)
Signed-off-by: khluu <khluu000@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Jee Jee Li <pandaleefree@gmail.com>
Co-authored-by: vllm-agent <claw@inferact.ai>
2026-10-17 00:52:17 -07:00
znzrybandWoosuk Kwon ad2688ab30 [Bugfix] Validate draft-token capacity in forward benchmark (#61809)
Signed-off-by: znzryb <znzryb34@gmail.com>
Co-authored-by: Woosuk Kwon <woosuk@inferact.ai>
2026-10-17 00:51:36 -07:00
JovanandOpenAI Codex 5be4266f5c [Bugfix][Frontend] Preserve Cohere and Anthropic streaming errors (#61947)
Signed-off-by: Kevin <jia__kuan@163.com>
Co-authored-by: OpenAI Codex <noreply@openai.com>
2026-10-17 06:10:16 +00:00
Wei Zhao 90d6d871d6 [Attention] Add CuTe DSL backend selection for MLA context prefill (#62069)
Signed-off-by: Wei Zhao <51183510+wzhao18@users.noreply.github.com>
Signed-off-by: wzhao18 <wzhao18.sz@gmail.com>
2026-10-17 01:21:55 -04:00
Cy ChengandCy d74f4180cd [Bugfix][Frontend] Detokenize echoed prompts after truncation (#61876)
Signed-off-by: Cy <15118312+cycancode@users.noreply.github.com>
Co-authored-by: Cy <15118312+cycancode@users.noreply.github.com>
2026-10-17 05:20:38 +00:00
Jee Jee Li 39983a5e7a [Kernel] Add FP32 router GEMM support for Step-5 Preview and Muse Spark Open (#62126)
Signed-off-by: Jee Jee Li <jeejeelee@inferact.ai>
2026-10-16 22:17:55 -07:00
41b8c15809 [Feature][MoRIIO] Support hybrid WRITE transfers with DSpark (#60544)
Signed-off-by: Yichao Zhu <Yichao.Zhu@amd.com>
Signed-off-by: tjtanaa <tunjian.tan@embeddedllm.com>
Co-authored-by: OpenAI Codex <noreply@openai.com>
Co-authored-by: tjtanaa <tunjian.tan@embeddedllm.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-17 04:57:58 +00:00
Micah WilliamsonandCursor a0e7137c4b [CI][ROCm][Bugfix] DeepSeek V4.1: don't pass expert intermediate padding to AITER (#61960)
Signed-off-by: Micah Williamson <micah.williamson@amd.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-10-17 04:25:39 +00:00
7b6310c109 [Perf][Sampler] Speed up the small-batch split top-p pipeline (#61996)
Signed-off-by: Bowen Wang <bowen@inferact.ai>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: vllm-agent <claw@inferact.ai>
2026-10-16 21:22:11 -07:00
Fadi Arafehandvllm-agent 556ac15eb9 [Bugfix] Remove extra args from CPU's compute slot mapping kernel (#61992)
Signed-off-by: Fadi Arafeh <fadi.arafeh@arm.com>
Co-authored-by: vllm-agent <claw@inferact.ai>
2026-10-17 12:11:33 +08:00
Gabriel WuandClaude Opus 5.5 dcd07967ca [CPU] Keep the hybrid block size across the engine core's config rerun (#61805)
Signed-off-by: Zihua Wu <13583761+lucifer1004@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 12:06:03 +08:00
ce49c47cbb [Bugfix][Sampler] Carry the boundary tie count across blocks in the split top-p mask kernel (#62046)
Signed-off-by: cipheraxat <cipheraxat@users.noreply.github.com>
Co-authored-by: cipheraxat <cipheraxat@users.noreply.github.com>
Co-authored-by: Bowen Wang <bowen@inferact.ai>
2026-10-17 02:26:57 +00:00
Sherif Walyandvllm-agent ce8853d27a [Bugfix][KV Offload] Allow explicit SimpleCPUOffload block counts (#61674)
Signed-off-by: Sherif Waly <sherif.waly@mistral.ai>
Co-authored-by: vllm-agent <claw@inferact.ai>
2026-10-17 02:11:12 +00:00
71c20f73f2 [FlashInfer] Migrate autotune warmup to flashinfer.autotune_v2 managed cache (#55671)
Signed-off-by: Yang Xu <yanxu@nvidia.com>
Signed-off-by: Zihua Wu <13583761+lucifer1004@users.noreply.github.com>
Signed-off-by: YangXu <yanxu@nvidia.com>
Co-authored-by: Zihua Wu <13583761+lucifer1004@users.noreply.github.com>
Co-authored-by: Kimi Code <noreply@moonshot.cn>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Misha Goin <mgoin64@gmail.com>
Co-authored-by: vllm-agent <claw@inferact.ai>
2026-10-17 02:08:28 +00:00
Shuolei Wangandaoshen02 b3a7e2ff6e [RL][Frontend] Return weight-version spans on OpenAI completions and chat (#61347)
Signed-off-by: Shuolei Wang <shuoleiwang123@gmail.com>
Co-authored-by: aoshen02 <aoshen@inferact.ai>
2026-10-17 02:08:07 +00:00
6b6a6c5c98 [RL] Support per-parameter source layouts for NCCL M2N (#60934)
Signed-off-by: pkousha <pkousha@nvidia.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
Co-authored-by: aoshen02 <aoshen@inferact.ai>
2026-10-16 21:41:17 -04:00
Harry MellorandClaude Opus 5.5 f8112704d5 [Chore] Use upstream Transformers InklingProcessor (#61883)
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 01:04:54 +00:00
Omar MuhammadNicholas SusantoDouglas Lehrmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
f019e3e105 [ROCm] attn res + sigmoid_mul + conv fusions (#52968)
Signed-off-by: Omar Muhammad <omar.muhammad@amd.com>
Signed-off-by: Nicholas Susanto <nicholas.susanto@amd.com>
Co-authored-by: Nicholas Susanto <nicholas.susanto@amd.com>
Co-authored-by: Douglas Lehr <91553416+dllehr-amd@users.noreply.github.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-10-17 00:54:54 +00:00
Vadim GimpelsonandClaude 783380af39 [Bugfix] Fix NaN output of the Mamba2 chunk scan kernel on sm90 (#61718)
Signed-off-by: Vadim Gimpelson <vadim.gimpelson@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
2026-10-16 17:24:51 -07:00
Wei ZhaoandCodex be25977532 [CI][Tests] Account for FP8 KV-cache rounding in MLA reference (#62067)
Signed-off-by: wzhao18 <wzhao18.sz@gmail.com>
Co-authored-by: Codex <noreply@openai.com>
2026-10-16 17:10:08 -07:00
523d74e2ac [Bugfix][LoRA] Fix garbage/NaN lora_expand output on SM90 with fp32 i… (#61906)
Signed-off-by: BeichenHuang <polarishuang0.0@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: aoshen02 <aoshen@inferact.ai>
2026-10-16 19:42:30 -04:00
a94c315711 [DP] Engines own wave wake-ups; drop the front-end FIRST_REQ (#59559)
Signed-off-by: fh3 <a@b>
Co-authored-by: fh3 <a@b>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 19:41:54 -04:00
b544587408 [Core][Feat] Pluggable KVCacheConfigBuilder for platform/model-specific KV cache planning (#53558)
Signed-off-by: wangxiyuan <wangxiyuan1007@gmail.com>
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai>
2026-10-16 23:16:47 +00:00
alpandmottopanikeiku 2268d83bd9 [Bugfix][Multimodal] Honor RGB PNG transparency when loading images (#61986)
Signed-off-by: mottopanikeiku <176798723+mottopanikeiku@users.noreply.github.com>
Co-authored-by: mottopanikeiku <176798723+mottopanikeiku@users.noreply.github.com>
2026-10-16 23:09:12 +00:00
28e92eeff2 [ROCm][Perf][Kimi-K3] Use AITER Gluon chunk_kda for KDA prefill (#59552)
Signed-off-by: Omar Muhammad <omar.muhammad@amd.com>
Signed-off-by: Nicholas Susanto <nicholas.susanto@amd.com>
Co-authored-by: Nicholas Susanto <nicholas.susanto@amd.com>
Co-authored-by: Douglas Lehr <91553416+dllehr-amd@users.noreply.github.com>
Co-authored-by: Kevin H. Luu <khluu000@gmail.com>
2026-10-16 22:58:56 +00:00
Afonso RibeiroandFlora Feng b088eba944 [Perf][Parser] Stream MiniMax M2 tool argument values incrementally (16 KB call: 5.9 s to 26 ms) (#62035)
Signed-off-by: Afonso Ribeiro <afonso.diamantino2@gmail.com>
Co-authored-by: Flora Feng <4florafeng@gmail.com>
2026-10-16 22:46:02 +00:00
596e530675 [Kernel] Add router GEMM kernels for (5120, 256) and BF16 output to ll_bf16_gemm (#62042)
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
Co-authored-by: Lain <fusiyuan2000@gmail.com>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: vllm-agent <claw@inferact.ai>
2026-10-16 15:39:46 -07:00
153a17166e [Frontend] Add the Beam reasoning and tool parsers (#62052)
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
Co-authored-by: Bugen Zhao <i@bugenzhao.com>
Co-authored-by: Yoni Gozlan <74535834+yonigozlan@users.noreply.github.com>
Co-authored-by: neilkg <33635204+neilkg@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 14:08:29 -07:00
Dan BlanaruandClaude Opus 5.5 9731f30a03 [Bugfix] Keep VLLM_RAY_BUNDLE_INDICES out of the compile cache key (#61211)
Signed-off-by: DanBlanaru <48605845+DanBlanaru@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 20:59:48 +00:00
68ebea789a [Perf][MoE] Skip padded work in SM100 FP4 DeepGEMM experts (#59044)
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com>
2026-10-16 16:20:52 -04:00
afriedrimergify[bot] <37929162+mergify[bot]@users.noreply.github.com>Claude Opus 5.5
cc2e4fecf2 [ROCm] [fusion] support ROCm aiter fusion of per tensor quant + rmsnorm (#52831)
Signed-off-by: Andy Friedrich <afriedri@amd.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 15:19:07 -05:00
9eb7fb9157 [Build] Remove unused tools/install_deepgemm.sh (#61265)
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Misha Goin <mgoin64@gmail.com>
Co-authored-by: Kevin H. Luu <khluu000@gmail.com>
2026-10-16 15:57:49 -04:00
cjackal 663e535640 [Perf][Spec Decode] Skip bonus-logit reduction for short drafts in rejection sampling (#61451)
Signed-off-by: cjackal <44624812+cjackal@users.noreply.github.com>
2026-10-16 12:42:48 -07:00
Oxana KorzhandClaude Opus 5.5 5bf3d9f665 [ROCm][Kimi-K3] Support DSpark adaptive verification on MI355X (#59824)
Signed-off-by: Oxana Korzh <okorzh@amd.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-16 13:53:49 -05:00
djramic 307c5844ad [ROCm][CI] Enable batch invariance for the ngram scheduling accuracy test (#62048)
Signed-off-by: Djordje Ramic <djoramic@amd.com>
2026-10-16 18:36:14 +00:00
Guan-Ming Chiuandvllm-agent 3dc6def360 [Attention][Spec Decode] Enable varlen Mamba2 decode for adaptive verification (#53036)
Signed-off-by: Guan-Ming (Wesley) Chiu <105915352+guan404ming@users.noreply.github.com>
Co-authored-by: vllm-agent <claw@inferact.ai>
2026-10-16 11:23:59 -07:00
djramic e7a5bdb454 [ROCm][CI] Move Spec Decode Speculators + MTP to MI355 (#62053)
Signed-off-by: Djordje Ramic <djoramic@amd.com>
2026-10-16 18:22:38 +00:00
alberto ae8f41238c [Bugfix][Parser] Honor a prompt-closed reasoning block in non-streaming parse and reasoning token counts (#61857)
Signed-off-by: Alberto Perdomo <aperdomo@redhat.com>
2026-10-16 12:54:42 -04:00
cffc0cfa31 [CI] Fix B200 FusedMoE Layer Kernels hang on NVSHMEM IB setup (#61695)
Signed-off-by: mgoin <mgoin64@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Kevin H. Luu <khluu000@gmail.com>
2026-10-16 12:00:12 -04:00
Andreas KaratzasandClaude 706791d516 [CI][AMD] Move 17 test groups from MI300 to MI355 and test large DeepSeek/Nemotron evals on AITER (#61955)
Signed-off-by: Andreas Karatzas <akaratza@amd.com>
Co-authored-by: Claude <noreply@anthropic.com>
2026-10-16 10:58:59 -05:00
Sawyer BowermanandCodex 76081b9ee4 [Bugfix][Model] Honor quantization config in Inkling MTP (#61921)
Signed-off-by: Sawyer Bowerman <sbowerma@redhat.com>
Co-authored-by: Codex <noreply@openai.com>
2026-10-16 15:49:26 +00:00
bnellnm 1a5b187066 Add @bnellnm to CODEOWNERS (#62025)
Signed-off-by: Bill Nell <bnell@redhat.com>
2026-10-16 11:16:03 -04:00
Wentao Ye 8e25bbca19 [Refactor] Remove dead code multiple places (#61843)
Signed-off-by: yewentao256 <zhyanwentao@126.com>
2026-10-16 15:01:07 +00:00
8a6c0449b3 [Bugfix][ROCm][GLM-5.2] Stop forcing skip_final_all_reduce on AMD DSA model path (#58589)
Signed-off-by: avininjamay8 <Avinash.Paul@amd.com>
Signed-off-by: avininjamay8 <avpaul@amd.com>
Co-authored-by: avininjamay8 <avpaul@amd.com>
Co-authored-by: Kevin H. Luu <khluu000@gmail.com>
Co-authored-by: vllm-agent <claw@inferact.ai>
2026-10-16 21:38:55 +08:00
Ayushman Singh de7eeaa87b [Perf] Share Triton JIT warmup compilation pool across registered kernels (#61422) 2026-10-16 08:59:42 -04:00
MOVZX be95edb7f6 Merge branch 'main' into patches
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-16 17:35:42 +07:00
Martin Hickey 64175d3827 [Bugfix][Frontend] Keep the parser's tool call IDs in batch derender (#61852)
Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com>
2026-10-16 10:27:17 +00:00
alpandmottopanikeiku 4ab99ff136 [Bugfix][Frontend] Honor the constructor chat template in offline scoring (#61990)
Signed-off-by: mottopanikeiku <176798723+mottopanikeiku@users.noreply.github.com>
Co-authored-by: mottopanikeiku <176798723+mottopanikeiku@users.noreply.github.com>
2026-10-16 10:20:46 +00:00
024c5188bf [Frontend] Resolve logprobs on the parsed streaming derender path (#60437)
Signed-off-by: Shimi Bandiel <shimib@google.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: wang.yuqi <yuqi.wang@daocloud.io>
Co-authored-by: Kevin H. Luu <khluu000@gmail.com>
2026-10-16 18:04:15 +08:00
Hexiang Wang 9a16d43ef0 [Bugfix][Mooncake] Fix MultiConnector stats and proxy SSE responses (#61218)
Signed-off-by: whx-sjtu <xiaowang990929@gmail.com>
Signed-off-by: Nicolò Lucchesi <nicolo.lucchesi@gmail.com>
2026-10-16 17:54:48 +08:00
MOVZXandHinata 8dc54f35dd [Offloader] Stream every multimodal tower parameter to host RAM
Qwen3_VisionTransformer derived its compute device from
patch_embed.proj.weight and fed pos_embed.weight straight into the
position-embedding interpolation kernel, so the last two anchor weights
had to stay in VRAM for a tower to be streamable at all.

Record the device the tower was built on before the offloader moves
anything, and fall back to it when no weight is on the accelerator.
Move the interpolation table to that device per encode, which costs one
8.4 MiB host-to-device copy for a 456 MiB tower. Towers now leave no
parameters in VRAM between encodes; only small buffers such as the fused
input norm and rotary inv_freq remain.

Also let a tower restrict streaming through tower_offload_children, and
warn when a tower resolves its device property to another device type,
so a tower that reads an anchor weight reports the cause instead of a
Triton pointer error.

Co-authored-by: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-16 16:07:53 +07:00
zofia 07f3857fb9 [XPU][Linear][MXFP4] enable torch as mxfp4 linear backend on xpu (#51965)
Signed-off-by: Zhu, Zufang <zufang.zhu@intel.com>
Signed-off-by: zofia <110436990+zufangzhu@users.noreply.github.com>
2026-10-16 17:07:05 +08:00
MOVZXandHinata 18c06ed406 [Offloader] Stream multimodal towers from host RAM with bulk-copy
Add --offload-towers: place multimodal tower weights in pinned host
memory and bulk-copy each unit to the GPU only during its forward,
keeping towers out of VRAM between encodes. Unlike UVA zero-copy,
whole-tensor H2D transfers avoid re-reading operand tiles over PCIe
inside the GEMM inner loop. Tower streaming is independent of layer
group scheduling, so group_size may be 0 for tower-only offload.

Co-authored-by: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-16 14:32:14 +07:00
fa75d3456d [Feature][PCP] Add DeepSeek-V4.1-Flash support (#59857)
Signed-off-by: wangyicong <wangyicong@bytedance.com>
Co-authored-by: OpenAI Codex <noreply@openai.com>
Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Kevin H. Luu <khluu000@gmail.com>
2026-10-15 23:17:51 -07:00
Chaojun Zhangandvllm-agent 074c83a09a [CI][Intel] Skip ROCm-only test_topk_gating_enabled_by_cdna on XPU (#61932)
Signed-off-by: Chaojun Zhang <chaojun.zhang@intel.com>
Co-authored-by: vllm-agent <claw@inferact.ai>
2026-10-16 13:44:32 +08:00
Wei Zhao e365792997 [Dependency] Upgrade TokenSpeed MLA to 2.0.3 (#61689)
Signed-off-by: Wei Zhao <51183510+wzhao18@users.noreply.github.com>
2026-10-15 22:28:59 -07:00
dskwe 49cb2bbf71 [Bugfix] Honor code_revision in config-based tokenizer loading (#61611)
Signed-off-by: Yuan Li <dskwelmcy@163.com>
2026-10-16 05:21:12 +00:00
4ac1f8d2ac [Bugfix][Structured Output] Mask grammar-invalid implicit reasoning ends (#61574)
Signed-off-by: Maroon Ayoub <mayoub@redhat.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com>
2026-10-16 12:53:41 +08:00
maithilijoshi20andCyrus Leung 92dc3dd393 [Bugfix] Honor OPT token embedding width for prompt embeddings (#61736)
Signed-off-by: maithilijoshi20 <maithilij2003@gmail.com>
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>
2026-10-16 04:41:50 +00:00
Andreas KaratzasandClaude b466fbacd2 [ROCm][Perf] Any-shape low-latency MoE router GEMM on MI355 (#61895)
Signed-off-by: Andreas Karatzas <akaratza@amd.com>
Co-authored-by: Claude <noreply@anthropic.com>
2026-10-15 23:22:31 -05:00
642c18d64c [Perf] Kimi K3 Specialized Decode: CuTeDSL Optimized GB300 (1.5-2x) (#61867)
Signed-off-by: Benjamin Chislett <bchislett@nvidia.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: vllm-agent <claw@inferact.ai>
2026-10-15 22:50:41 -04:00
3ddc6702a8 [XPU][Docker] Bump nixl to 1.5.0 (#60356)
Signed-off-by: Sandeep Maddipatla <sandeep.maddipatla@intel.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Kevin H. Luu <khluu000@gmail.com>
2026-10-16 10:35:29 +08:00
Shuolei Wang 05ad0a8ace [Perf][DSv4.1] Split fused_wqa_wkv by token across TP ranks on SM90 (#58872)
Signed-off-by: Shuolei Wang <shuoleiwang123@gmail.com>
2026-10-16 02:31:34 +00:00
fangyuchulqh-hhTyler Michael SmithOpenAI Codex2017403603Julien MancusoItay AlroyCodexmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
dd9e07a25f [Feature] Add External LB Mode Support for Elastic EP Scaling (#43202)
Signed-off-by: lqh-hh <liuqihang6@huawei.com>
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
Co-authored-by: lqh-hh <liuqihang6@huawei.com>
Co-authored-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
Co-authored-by: 2017403603 <705328646@qq.com>
Co-authored-by: Julien Mancuso <jmancuso@nvidia.com>
Co-authored-by: Itay Alroy <75032521+itayalroy@users.noreply.github.com>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-10-15 21:29:02 -04:00
MonishverandKevin H. Luu fb3fec970e [Bugfix][Kernel] silu_and_mul_quant: run the vectorized loop (#61716)
Signed-off-by: Monishver Chandrasekaran <monishverchandrasekaran@gmail.com>
Co-authored-by: Kevin H. Luu <khluu000@gmail.com>
2026-10-15 18:16:53 -07:00
Andreas KaratzasandClaude 29345f42a6 [ROCm][Qwen4Exp] Support HC and MoE sequence parallelism on ROCm (#61471)
Signed-off-by: Andreas Karatzas <akaratza@amd.com>
Co-authored-by: Claude <noreply@anthropic.com>
2026-10-15 20:11:17 -05:00
Kevin H. LuuandClaude Opus 5.5 a956a8126e [Bugfix][MoE] Keep the gate's hash_indices_table out of RoutedExperts parameters (#61930)
Signed-off-by: khluu <khluu000@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 18:00:17 -07:00
Jee Jee Li bee95149e2 Update DeepGEMM commit (#61744)
Signed-off-by: Jee Jee Li <jeejeelee@inferact.ai>
2026-10-15 17:54:40 -07:00
Tessa PhamandBenjamin Chislett 8f93c64866 [Bugfix] remove duplicate hash-routing capability method (#61924)
Signed-off-by: Tessa Pham <tepham@redhat.com>
Co-authored-by: Benjamin Chislett <bchislett@nvidia.com>
2026-10-15 17:40:23 -07:00
a8be1500ba [Bugfix][MoE] Restore TP-sharded routed-expert capture for DP=1 (#61388)
Signed-off-by: Ronald1995 <ronaldautomobile@163.com>
Signed-off-by: Ao Shen <281812371+aoshen02@users.noreply.github.com>
Signed-off-by: Ao Shen <aoshen@inferact.ai>
Co-authored-by: Ao Shen <281812371+aoshen02@users.noreply.github.com>
Co-authored-by: OpenAI Codex <noreply@openai.com>
Co-authored-by: Ao Shen <aoshen@inferact.ai>
2026-10-16 00:21:24 +00:00
Jack HuandDouglas Lehr ef38e8d006 [Model][Hardware][AMD][Kernel] Part 2/2 -> Enable e2e QK Norm + RoPE + KV Cache runtime fusion for Qwen3-30B-A3B on ROCM_ATTN (#42755)
Signed-off-by: Jack Hu <Jack.Hu@amd.com>
Signed-off-by: Jack Hu <jingxujaster@hotmail.com>
Signed-off-by: Jack Hu <jack.hu@amd.com>
Co-authored-by: Douglas Lehr <91553416+dllehr-amd@users.noreply.github.com>
2026-10-15 17:10:14 -07:00
Anerudhan GopalandOpenAI Codex af28c171f5 [Dependency] Upgrade FlashInfer to 0.7.2 and restore KDA auto backend (#61913)
Signed-off-by: Anerudhan Gopal <agopal@nvidia.com>
Co-authored-by: OpenAI Codex <noreply@openai.com>
2026-10-15 17:04:24 -07:00
Juntian LiuandClaude Opus 5.5 19b0efce30 [Perf][Core] Skip the null head of sliding-window block tables (#61724)
Signed-off-by: Juntian Liu <Juntianl777@gmail.com>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 16:14:51 -07:00
Woosuk KwonandClaude Opus 5.5 4a89316efa [Core] Drop the Ray driver-node GPU pin left over from the legacy executor (#61713)
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 16:00:18 -07:00
3f1e57225a [ROCm][Perf][Kimi-K3] Use AITER Gluon KDA kernels for decode and spec decode (#57392)
Signed-off-by: omuhamma <omar.muhammad@amd.com>
Signed-off-by: Nicholas Susanto <nicholas.susanto@amd.com>
Co-authored-by: Nicholas Susanto <nicholas.susanto@amd.com>
Co-authored-by: Simon Danielsson <70206058+simondanielsson@users.noreply.github.com>
2026-10-15 17:54:24 -05:00
de7d70676d [Perf] Low-latency CPU all-reduce for DP metadata syncs (#61767)
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: hjjq <50634613+hjjq@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-15 15:29:13 -07:00
Stefano CastagnettaandKevin H. Luu 784d1fff0c [MRV2][Spec Decode] Quantized draft lm_head for drafters that share the target head (#59973)
Signed-off-by: Stefano Castagnetta <scastagnetta@nvidia.com>
Co-authored-by: Kevin H. Luu <khluu000@gmail.com>
2026-10-15 22:06:40 +00:00
d92a4f25f2 [watermarking] Add DFlash support to dual-key gumbel-max watermarking (#59554)
Signed-off-by: Shern Shiou Tan <shernshiou@gmail.com>
Signed-off-by: Simon Veitner <sveitner@redhat.com>
Signed-off-by: Raphael Rialland <raphael.rialland@mistral.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Simon Veitner <sveitner@redhat.com>
Co-authored-by: Raphael Rialland <raphael.rialland@mistral.ai>
Co-authored-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com>
2026-10-15 22:06:06 +00:00
e711723352 [ROCm] gfx1250 optimizations and model enablement (#58800)
Signed-off-by: jpvillam <juan.villamizar@amd.com>
Signed-off-by: Jaden Mathias <jaden.mathias@amd.com>
Signed-off-by: Juan Villamizar <jpvillam@amd.com>
Signed-off-by: jpvillam <jpvillam@amd.com>
Signed-off-by: Omar Muhammad <omar.muhammad@amd.com>
Signed-off-by: Daniel <danichan@amd.com>
Signed-off-by: Juan Villamizar <100237675+jpvillam-amd@users.noreply.github.com>
Co-authored-by: Jaden Mathias <jaden.mathias@amd.com>
Co-authored-by: jpvillam <jpvillam@amd.com>
Co-authored-by: Omar Muhammad <omar.muhammad@amd.com>
Co-authored-by: Daniel <danichan@amd.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Daniel Chan <Daniel.Chan@amd.com>
2026-10-15 14:51:57 -07:00
Thang NguyenandClaude Opus 5.5 9b99ccb56b [CI] Shard the AMD mirrors of the DP, RayExecutorV2 and batch invariance steps automatically (#61861)
Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 16:48:19 -05:00
DimensionSTPandKevin H. Luu 9edcc7fa7e [Bugfix][Spec Decode] Avoid reusing random draws across block verification rounds (#61161)
Signed-off-by: DimensionSTP <65501090+DimensionSTP@users.noreply.github.com>
Co-authored-by: Kevin H. Luu <khluu000@gmail.com>
2026-10-15 14:46:38 -07:00
Mikko Tukiainen 0c25849292 [ROCm][Perf][Bugfix] Probe the preshuffled FP4 GEMM config by logical K (#59406)
Signed-off-by: Mikko Tukiainen <Mikko.Tukiainen@amd.com>
2026-10-15 21:11:33 +00:00
f8460ef236 [Kernel][DSv4.1] Bump FlashMLA to the 2026.09.30 release and its interleaved V4.1 KV cache (#61371)
Signed-off-by: juntianl <juntianl@inferact.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Kimi Code <noreply@moonshot.ai>
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai>
2026-10-15 13:58:25 -07:00
33212db2c2 [KV Connector][NIXL] Reject NVFP4 KV cache with heterogeneous TP head split (#61865)
Signed-off-by: NickLucche <nicolo.lucchesi@mistral.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Kevin H. Luu <khluu000@gmail.com>
2026-10-15 13:57:37 -07:00
Thang NguyenandClaude Opus 5.5 c414bf0295 [CI] Shard four AMD mirrors automatically (#61817)
Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 13:57:35 -07:00
Wentao Ye 54d9b1cb48 [GLM5.3 Perf] Inplace update draft decode metadata, 1.7% E2E Throughput Improvement (#61873)
Signed-off-by: yewentao256 <zhyanwentao@126.com>
2026-10-15 20:20:12 +00:00
SeongJun LeeandWei Zhao ee8c47961b [CI] Use an ungated model as the attention test config default (#61732)
Signed-off-by: lesj0610 <lesj0610@gmail.com>
Co-authored-by: Wei Zhao <51183510+wzhao18@users.noreply.github.com>
2026-10-15 20:19:09 +00:00
21f4b6dc84 [KV Offload] Split the CPU offload region across shm files (#59546)
Signed-off-by: Itay Etelis <92247226+Etelis@users.noreply.github.com>
Co-authored-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Kevin H. Luu <khluu000@gmail.com>
2026-10-15 20:17:40 +00:00
Thang NguyenandClaude Opus 5.5 16532876e7 [CI] Shard four more AMD mirrors automatically (#61860)
Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 15:13:51 -05:00
d12ea8e309 [Bugfix][MRV2] Keep input_ids on later PP stages for raw-token models (#61016)
Signed-off-by: booth-algo <kevinlauofficial01@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Kevin H. Luu <khluu000@gmail.com>
2026-10-15 12:54:21 -07:00
d486219fc2 [Bugfix][Core] Uncount stop-trimmed spec tokens from num_computed_tokens (#58668)
Signed-off-by: David Cheung <d7cheung@gmail.com>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
2026-10-15 12:50:06 -07:00
49f944be3e [Perf][HiSparse] Reuse pinned host pool after graph profiling (#61668)
Signed-off-by: Itay Etelis <92247226+Etelis@users.noreply.github.com>
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
Co-authored-by: Matthew Bonanni <mbonanni@redhat.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 12:33:03 -07:00
Rohan PotdarandClaude Opus 5.5 105605b8e9 [ROCm] Support DeepSeek-V4 hash routing in AITER Triton W4A16 MoE (#60282)
Signed-off-by: Rohan Potdar <rohan.potdar@amd.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 14:10:12 -05:00
Thien Tran 82b9ccaa79 [Perf][MoE] Skip the TRTLLM fp8-block activation-scale transpose copy (#61773)
Signed-off-by: Thien Tran <gau.nernst@yahoo.com.sg>
2026-10-15 12:09:07 -07:00
4d58fa254a [MoE] Unify DeepGEMM MegaMoE experts behind one feature-matrix layer; add NVFP4 (#61538)
Signed-off-by: Yongye Zhu <zyy1102000@gmail.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Kevin H. Luu <khluu000@gmail.com>
2026-10-15 19:02:01 +00:00
Misha GoinandClaude Opus 5.5 4d9a8d6a41 [Bugfix] Make Humming DeviceInfo lookups traceable under torch.compile (#61858)
Signed-off-by: mgoin <mgoin64@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 11:12:21 -07:00
Mikko TukiainenandCursor 47fd76e4ce [ROCm][Perf] Route softmax MoE top-k, including fused shared-expert sigmoid, through AITER topk_gating (#56226)
Signed-off-by: Mikko Tukiainen <Mikko.Tukiainen@amd.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-10-15 17:53:18 +00:00
4f49b55c1f [MoE] Shape-based padding for FlashInfer TRT-LLM MXFP8 experts (gated and non-gated) (#61293)
Signed-off-by: Jhao-Ting Chen <jhaotingc@nvidia.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com>
2026-10-15 13:44:11 -04:00
qizixiClaude Opus 5.5mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
7cd7982dfd [Spec Decode] Don't reserve DFlash/DSpark draft slots in the scheduler token budget (#59468)
Signed-off-by: zixi-qi <zixi@inferact.ai>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-10-16 01:23:58 +08:00
bzsuniandFlora Feng 338dab077c [Bugfix] Fix Apertus streaming scientific notation (#61812)
Signed-off-by: bzsuni <bingzhe.sun@daocloud.io>
Co-authored-by: Flora Feng <4florafeng@gmail.com>
2026-10-15 16:49:57 +00:00
957d512e96 [Bugfix][Frontend] Preserve streaming chunk boundaries in RequestOutputCollector (#58941)
Signed-off-by: David Cheung <d7cheung@gmail.com>
Co-authored-by: Benedikt Falk <bfalk@nvidia.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
2026-10-15 16:38:28 +00:00
Jovan a9e90b9d1d [Bugfix][Frontend] Reject Cohere REQUIRED tool choice without tools (#61825)
Signed-off-by: Kevin <jia__kuan@163.com>
2026-10-15 16:37:08 +00:00
d6f0b7a416 [CI] Skip exited descendants when finding the worker in the FT e2e test (#61648)
Signed-off-by: Oxana Korzh <okorzh@amd.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Andreas Karatzas <akaratza@amd.com>
2026-10-15 16:08:14 +00:00
ed3050add4 [Core] Apply prefill_schedule_interval outside data parallelism (#54627)
Signed-off-by: lifulu <fululi12@amd.com>
Signed-off-by: lifulu <lifulu@users.noreply.github.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: lifulu <lifulu@users.noreply.github.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-10-15 11:03:13 -05:00
c825e1d235 [Perf] Fuse the mamba align block-table gather into one Triton kernel, 7 kernels -> 1, 6x speedup on all shapes for this kernel (#58737)
Signed-off-by: lifulu <fululi12@amd.com>
Signed-off-by: lifulu <lifulu@users.noreply.github.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: lifulu <lifulu@users.noreply.github.com>
2026-10-15 11:02:53 -05:00
MOVZXandHinata 5d6754ff6b fix: keep the refusal-direction projection dynamo-safe
With VLLM_ABLITERATION set, boot crashed: dynamo traces
abliteration.register from Qwen3NextGatedDeltaNet.__init__, and
buf.copy_(r) fails on the meta device during the compile-time model
construction pass. Disable tracing for register and guard the copy so
an unmaterializable buffer turns projection off with a warning.

Log activation at registration instead of inside project_: project_
runs in the compiled region, and a logger call there aborts dynamo.
The width guard uses the same escape hatch.

Also project the embedding that Qwen3NextMultiTokenPredictor builds
for itself, which the first pass missed, and add tests for the
2560-value Flash-Next direction against a 5120 model and for a
half-width activation.

Co-authored-by: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-15 22:56:06 +07:00
CharlesWuandYichao Zhu a97d19d792 [UMBP][1/10] Add shared connector, runtime contracts, and request lifecycle (#61789)
Signed-off-by: CharlesWu <Charles.Wu@amd.com>
Co-authored-by: Yichao Zhu <Yichao.Zhu@amd.com>
2026-10-15 15:12:39 +00:00
3dcfaeb4e8 [Frontend] Apply each endpoint's top_logprobs cut in derender (#59518)
Signed-off-by: Yin Li <kevinli.ai.work@gmail.com>
Co-authored-by: Yin Li <kevinli.ai.work@gmail.com>
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>
2026-10-15 15:04:04 +00:00
MOVZXandHinata 0076adc78b [Qwen3.5] Project the refusal direction out of the residual stream at runtime
orcarouter/Qwen3.8-27B-Uncensored removes refusal by editing weights. The
Swift release we serve is AutoRound W8A16, so re-editing weights means
re-quantizing. Project one unit direction r out of every residual-stream
writer output instead: x <- x - r (r . x), which equals the weight edit
W' = W - r(r^T W) on every writer feeding that stream (Arditi et al. 2024).

Sites: attention output for both the full-attention and GDN branches, the
MLP output, and token embeddings, in Qwen3NextDecoderLayer.forward and
Qwen3NextModel.embed_input_ids, which Qwen3_5DecoderLayer and Qwen3_5Model
inherit but whose base constructors they skip, so both register there too.
The MTP drafter reuses Qwen3_5DecoderLayer and calls its own
embed_input_ids, so the draft path is covered at the same width, and
_maybe_share_embeddings at PP=1 hands the draft the target embedding
module. Sequence parallelism shards dim 0 only, so the last dim is
hidden_size at every site.

VLLM_ABLITERATION names a JSON file with key refusal_direction, a
safetensors tensor, or a bare name resolved under
/opt/qwen38/abliteration then /data/abliteration. Empty or 0 is off.
register() always creates the buffer _abliteration_r, zeros when off, so
one compiled graph serves ON and OFF and toggling the switch
(declared in envs.py, hence in compile_factors) invalidates the compile
cache. project_() is an in-place rank-1 update with no full-size
temporary, and skips tensors whose width is not hidden_size.

Co-authored-by: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-15 21:40:16 +07:00
Matthias Gehre 011fa68ca9 [ROCm][Perf] Pad fp16/bf16 GEMM weight row stride for L2 cache (#55090)
Signed-off-by: Matthias Gehre <matthias.gehre@amd.com>
2026-10-15 14:26:32 +00:00
MOVZX 76df77c04e Merge branch 'main' into patches
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-15 21:16:39 +07:00
Ronaldandaoshen02 d12d5460a5 [Bugfix][MoE] Keep routing rows during DeepSeek V4.1 decoder replay (#61643)
Signed-off-by: Ronald1995 <ronaldautomobile@163.com>
Co-authored-by: aoshen02 <aoshen@inferact.ai>
2026-10-15 14:02:06 +00:00
24f632ca7e [Bugfix] Engage indexer TP row sharding on total rows, not per-rank rows (#61647)
Signed-off-by: Zheng Cai <caizheng1993@gmail.com>
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com>
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Co-authored-by: zjy0516 <riverclouds.zhu@qq.com>
2026-10-15 13:54:14 +00:00
mosafariuk 69425846f8 [Bugfix][CPU] Report the real cause when the worker outgrows its memory reservation (#59472)
Signed-off-by: Mo <mosafariuk@gmail.com>
2026-10-15 21:52:05 +08:00
b7b20b1b8f [Bugfix][Frontend] Leave the final stop token out of chat derender text (#61757)
Signed-off-by: khluu <khluu000@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>
2026-10-15 21:51:34 +08:00
littlecircle0730andShanshan Shen 71536bd5f8 [MM][CG] Support ViT full CUDA graph for DeepSeek-VL2 (#46005)
Signed-off-by: littlecircle0730 <littlecircle0730@gmail.com>
Signed-off-by: littlecircle0730 <43994952+littlecircle0730@users.noreply.github.com>
Co-authored-by: Shanshan Shen <467638484@qq.com>
2026-10-15 21:38:52 +08:00
0951fc183c [ROCm][DSv4.1] Disambiguate ROCm attention class names (#58689)
Signed-off-by: Tuukka Sarvi <tuukka.sarvi@amd.com>
Co-authored-by: Shanshan Shen <467638484@qq.com>
Co-authored-by: Kevin H. Luu <khluu000@gmail.com>
2026-10-15 21:38:39 +08:00
1463501553 [Bugfix] Terminate streaming sessions on ABORT/ERROR and prevent zombie sessions (#58942)
Signed-off-by: David Cheung <d7cheung@gmail.com>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Benedikt Falk <bfalk@nvidia.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
2026-10-15 13:32:19 +00:00
Xunzhuo 766c837bd5 [Model] Support every Decision 3.0 size (d3-flash, d3-mini, d3-nano, d3-lite, d3-edge) (#61778)
Signed-off-by: Xunzhuo Liu <xunzhuo.liu@amd.com>
2026-10-15 13:29:24 +00:00
Laith SakkaandNick Hill 9cede34a67 [Bugfix] Create cuBLAS handles before memory profiling to avoid OOM (#61044)
Signed-off-by: Laith Sakka <lsakka@meta.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
2026-10-15 08:59:22 -04:00
Martin Hickey 252be28c18 [Frontend] Replace return_mm_kwargs with an mm_payload enum on render requests (#61427)
Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com>
2026-10-15 12:41:46 +00:00
Martin Hickey c6b772cf6b [Bugfix][Frontend] Emit kv/ec transfer params at the top level of render output (#61241)
Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com>
2026-10-15 12:34:15 +00:00
Micah Williamson e8e1e96196 [ROCm][DSv4] AITER MegaMoE Fixed-Slot Dispatch For DeepSeek-V4 (#61262)
Signed-off-by: Micah Williamson <micah.williamson@amd.com>
2026-10-15 19:10:59 +08:00
8b27d27402 [Bugfix][KV Offload] Always attempt NIXL agent removal during peer teardown (#60279)
Signed-off-by: nej1gotnochill <divyanshlalotra2@gmail.com>
Co-authored-by: Codebuff <noreply@codebuff.com>
Co-authored-by: Kevin H. Luu <khluu000@gmail.com>
2026-10-15 14:07:23 +03:00
Talor Abramovich 81e42c9987 [Feature][Frontend] Control the log level of request JSON body using env var (#60630)
Signed-off-by: talora <talora@nvidia.com>
2026-10-15 11:07:19 +00:00
jimmy-adams 365aaa77bd [Bugfix][DSv4] Remap Quark MTP layer_quant_config onto draft prefixes (#56054)
Signed-off-by: jimmy-adams <41593649+jimmy-adams@users.noreply.github.com>
2026-10-15 18:51:57 +08:00
e10c10a644 [Core] Add PROMOTION_LATENCY histogram metric for tiering offload (#53910)
Signed-off-by: Wen Zhou <wenzhou@redhat.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: OpenAI Codex <noreply@openai.com>
Co-authored-by: Kevin H. Luu <khluu000@gmail.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
2026-10-15 13:47:41 +03:00
Pei LiandKevin H. Luu 6fe7401fac [Perf] Read VLLM_COMPUTE_NANS_IN_LOGITS once per IterationStats (#61710)
Signed-off-by: Pei Li <peili@nvidia.com>
Co-authored-by: Kevin H. Luu <khluu000@gmail.com>
2026-10-15 18:35:54 +08:00
TimofeyandCodex 716c5225ae [Model] Add AliceAI support (#57963)
Signed-off-by: timka <timka.byzov@yandex.ru>
Co-authored-by: Codex <noreply@openai.com>
2026-10-15 10:15:03 +00:00
7020600b1d [KV Connector][NIXL] Register packed MLA KV rows as a single region (#61074)
Signed-off-by: NickLucche <nicolo.lucchesi@mistral.ai>
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai>
Co-authored-by: Codex <noreply@openai.com>
2026-10-15 09:18:15 +00:00
Flora FengandChauncey 80dd705c1d [Frontend] Make finish_reason and Responses item status truthful (#59324)
Signed-off-by: Flora Feng <4florafeng@gmail.com>
Signed-off-by: sfeng33 <4florafeng@gmail.com>
Co-authored-by: Chauncey <chaunceyjiang@gmail.com>
2026-10-15 10:12:02 +01:00
Denis KhachyanandClaude Code 427915bed2 [Build][CPU] Install Run:ai Model Streamer in vllm-openai-cpu image (#61382)
Signed-off-by: Denis Khachyan <khachyanda@gmail.com>
Co-authored-by: Claude Code <noreply@anthropic.com>
2026-10-15 09:11:51 +00:00
Kevin H. LuuandClaude Opus 5.5 7f730b4cea [CI] Run 4xH100 steps nightly only (#61782)
Signed-off-by: khluu <khluu000@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 01:32:43 -07:00
wang.yuqi 0f0bc4dbb1 [Test] Split the chat_utils tests. (#61765)
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io>
2026-10-15 08:26:22 +00:00
Harry MellorClaude Opus 5.5mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>Kevin H. Luu
b965ee564d [Chore] Use upstream Transformers DeepseekV4Config (#61392)
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Kevin H. Luu <khluu000@gmail.com>
2026-10-15 01:08:08 -07:00
Misha GoinandClaude Opus 5.5 1c4c319792 [Frontend] Advertise --enable-x/--disable-x for boolean CLI flags (#61479)
Signed-off-by: mgoin <mgoin64@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 01:08:02 -07:00
48f17b77bb [Bugfix] Reject mismatched float16/bfloat16 KV cache dtype (#60885)
Signed-off-by: cipheraxat <cipheraxat@users.noreply.github.com>
Co-authored-by: cipheraxat <cipheraxat@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 08:06:59 +00:00
fc8e08a2a6 [Bugfix][KV Offload] Do not let mark_miss() flip a pending or in-flight async lookup probe (#59096)
Signed-off-by: Sohom Chakraborty <16609933+sohom-cs@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Kevin H. Luu <khluu000@gmail.com>
2026-10-15 10:57:46 +03:00
Bugen ZhaoandClaude Opus 5.5 8fa5ae844a [Frontend] Let an unset --tool-strict-level fall back to the tool parser's default (#61328)
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 15:48:38 +08:00
91ed2859c7 [CPU][Zen] Update zentorch to 2.14.0.1 & enable float16 on Venice and later (#61411)
Signed-off-by: Ganesh R <ganesh.r@amd.com>
Co-authored-by: Kevin H. Luu <khluu000@gmail.com>
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>
2026-10-15 00:23:50 -07:00
85c94c1706 [Tests][XPU] Skip modelopt_fp4 online MoE re-quantization test on XPU (#61723)
Signed-off-by: Chaojun Zhang <chaojun.zhang@intel.com>
Co-authored-by: Kevin H. Luu <khluu000@gmail.com>
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>
2026-10-15 00:23:45 -07:00
Kevin H. LuuandClaude Opus 5.5 8f111b22ee [CI] Run 4xB200 steps nightly only (#61541)
Signed-off-by: khluu <khluu000@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 00:08:48 -07:00
Kevin H. LuuandClaude Opus 5.5 32feecd91f [CI] Pin DeepSeek-V4.1-Flash-NVFP4 eval to its FP8-Engram revision (#61763)
Signed-off-by: khluu <khluu000@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 00:05:18 -07:00
Chaojun ZhangandKevin H. Luu ab593678b5 [CI][XPU] Ignore test_reload.py on Intel CI (#61740)
Signed-off-by: Chaojun Zhang <chaojun.zhang@intel.com>
Co-authored-by: Kevin H. Luu <khluu000@gmail.com>
2026-10-15 15:03:34 +08:00
a9fe30e9f6 [Bugfix][Multimodal] Recover PyNvVideoCodec decoder slots after failures (#59597)
Signed-off-by: Daniel Afrimi <dafrimi@nvidia.com>
Signed-off-by: danielafrimi <45691845+danielafrimi@users.noreply.github.com>
Signed-off-by: Daniel Afrimi <dafrimi@oci-hsg-cs-001-login-02.cm.cluster>
Co-authored-by: OpenAI Codex <noreply@openai.com>
Co-authored-by: Daniel Afrimi <dafrimi@oci-hsg-cs-001-login-02.cm.cluster>
Co-authored-by: tomeras91 <57313761+tomeras91@users.noreply.github.com>
2026-10-15 09:17:19 +03:00
liranschourandKevin H. Luu c5f8adc0b1 [KV Offload] Expose OffloadingManager.lock()/unlock() and service the tiering control plane off the scheduler thread (#58168)
Signed-off-by: Liran Schour <lirans@il.ibm.com>
Co-authored-by: Kevin H. Luu <khluu000@gmail.com>
2026-10-15 08:50:56 +03:00
HarshilandCyrus Leung c54dca96bd [Bugfix][Benchmark] Fix probe latency and duration in bench serve (#60511)
Signed-off-by: QHarshil <harshil_c@hotmail.com>
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>
2026-10-15 05:50:30 +00:00
5341b5ba7d [KV Offload] Pin only each worker's own offload slots (#59544)
Signed-off-by: Itay Etelis <92247226+Etelis@users.noreply.github.com>
Co-authored-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Kevin H. Luu <khluu000@gmail.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
2026-10-15 08:49:42 +03:00
6605802904 [Perf][KV Connector][Mooncake] Send the KV layout once per peer (#59825)
Signed-off-by: staryxchen <staryxchen@tencent.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Jiangyun Zhu <riverclouds.zhu@qq.com>
2026-10-15 05:36:31 +00:00
a400804ce4 [CI] Relax MTP sharded sampling cross-boot noise bound (#57026)
Signed-off-by: Kevin Luu <51931015+khluu@users.noreply.github.com>
Signed-off-by: khluu <khluu000@gmail.com>
Co-authored-by: OpenAI Codex <noreply@openai.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 22:31:45 -07:00
Liqian LinCodexRobert ShawItay Etelismergify[bot] <37929162+mergify[bot]@users.noreply.github.com>Or OzeriKevin H. Luu
3f1bcfd82b [Bugfix][KV Offload] Bound unanswered P2P lookup waits (#56836)
Signed-off-by: jacklin78911-collab <jacklin78911@gmail.com>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com>
Co-authored-by: Itay Etelis <92247226+Etelis@users.noreply.github.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
Co-authored-by: Kevin H. Luu <khluu000@gmail.com>
2026-10-15 08:23:34 +03:00
81e1c3f183 [XPU][MLA] Integrate XPU MLA support into the existing FLASH_ATTN_MLA backend (#58297)
Signed-off-by: baodii <di.bao@intel.com>
Signed-off-by: Zhouxiaoya25 <xiaoya.zhou@intel.com>
Signed-off-by: ZhouXiaoya25 <xiaoya.zhou@intel.com>
Co-authored-by: baodii <di.bao@intel.com>
Co-authored-by: Kunshang Ji <kunshang.ji@intel.com>
2026-10-15 12:47:37 +08:00
Vivaan SharmaandKevin H. Luu 96b5b0ee4b [Model] Enable torch.compile and CUDA graphs for EmbeddingGemma2 (#61572)
Signed-off-by: Vivaan Sharma <99265792+Vivaansharma2003@users.noreply.github.com>
Co-authored-by: Kevin H. Luu <khluu000@gmail.com>
2026-10-15 12:32:39 +08:00
Haiyu ZuoandKevin H. Luu ff5a91a456 [Feature][Model Loader] Apply model-owned MTP shard selection to RunAI ModelStreamer (#60759)
Signed-off-by: 浩韵 <zuohaiyu.zhy@alibaba-inc.com>
Co-authored-by: Kevin H. Luu <khluu000@gmail.com>
2026-10-15 12:23:11 +08:00
Chaitanya Sri Krishna LollaandCursor 833665cded [ROCm][DI] Enable MoRI FP4 dispatch for mxfp4 w4a4 moe (#59596)
Signed-off-by: lcskrishna <lollachaitanya@gmail.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-10-14 23:20:14 -05:00
Tuukka SarviandKevin H. Luu 10d2c6083d [ROCm][DSv4.1] Enable DSpark adaptive verification (#58151)
Signed-off-by: Tuukka Sarvi <tuukka.sarvi@amd.com>
Co-authored-by: Kevin H. Luu <khluu000@gmail.com>
2026-10-15 12:05:43 +08:00
Shanshan Shen f46021a7fa [ROCm][Perf] Enable DPA+ETP prefill compute/comm dual-batch-overlap for DeepSeek-V4 (#57773)
Signed-off-by: Shanshan Shen <87969357+shen-shanshan@users.noreply.github.com>
Signed-off-by: shen-shanshan <467638484@qq.com>
2026-10-15 11:16:48 +08:00
FeanorandTaneem Ibrahim 5d5c3d7c43 [Bugfix] Preserve padding token types for cross encoders (#58143)
Signed-off-by: zhaoye <772971548@qq.com>
Co-authored-by: Taneem Ibrahim <taneem.ibrahim@gmail.com>
2026-10-15 03:09:24 +00:00
429a3da665 [Bugfix][ROCm][Disagg] Allow MoRIIO HMA per-layer block_len (#58585)
Signed-off-by: avininjamay8 <Avinash.Paul@amd.com>
Signed-off-by: avininjamay8 <avpaul@amd.com>
Co-authored-by: avininjamay8 <avpaul@amd.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Shanshan Shen <467638484@qq.com>
2026-10-15 09:35:31 +08:00
KaiandKunshang Ji de2b869ee9 [Bugfix][Kernel] Register int8 scaled_mm as a custom op (#57331)
Signed-off-by: Kai <limkc.reply@gmail.com>
Co-authored-by: Kunshang Ji <kunshang.ji@intel.com>
2026-10-15 09:27:49 +08:00
Aarushi Jain 2abd479aee [ROCm][CI] Add AMD job for DeepSeek-V4.1 Mono Decode Numerics (#61720)
Signed-off-by: aarushjain29 <aarushi.jain2@amd.com>
2026-10-15 01:23:25 +00:00
Woosuk KwonandClaude Opus 5.5 e57bb4ace7 [Core] Remove the legacy Ray compiled-graph executor (#61712)
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 18:13:27 -07:00
bfdbfa1632 [Bugfix][PCP] Align MoE routing padding with gathered tokens (#60946)
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
Signed-off-by: wangyicong <wangyicong@bytedance.com>
Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: OpenAI Codex <noreply@openai.com>
2026-10-14 21:04:00 -04:00
Woosuk KwonandClaude Opus 5.5 8530aed2b7 [Spec Decode] Remove Medusa and MLP speculator support (#61701)
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 17:29:23 -07:00
josh-schell-amdandClaude Opus 5.5 273a04964e [ROCm] Bump AITER to v0.1.25 and relax the DSv4.1 mono MoE tolerance (#61697)
Signed-off-by: josh schell <josh.schell@amd.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-14 23:50:24 +00:00
Aarushi JainandAndreas Karatzas 7840bede98 [ROCm][CI] Add AMD jobs for test groups that only exist in tes… (#59678)
Co-authored-by: Andreas Karatzas <akaratza@amd.com>
2026-10-14 23:00:44 +00:00
Kevin H. Luu 4b61f300c0 [CI] Fix /ci run reporting a failed merge on fork PRs (#61684)
Signed-off-by: khluu <khluu000@gmail.com>
2026-10-14 15:26:15 -07:00
Micah Williamson a4d553a46a Revert "[Build] Update deep_gemm git tag for locality_domain update" (#61703) 2026-10-14 14:57:44 -07:00
Rohan PotdarandClaude Opus 5.5 9a648b4649 [ROCm][Kimi-K3] Don't read W_UK_T before weight loading in the MLA wrapper (#61654)
Signed-off-by: Rohan138 <rohanpotdar138@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 21:44:55 +00:00
Ben BrowningandFlora Feng 4c5ea86f53 [Bugfix][Parser] Stream Gemma4 string tool arguments incrementally (#61494)
Signed-off-by: Ben Browning <56071+bbrowning@users.noreply.github.com>
Co-authored-by: Flora Feng <4florafeng@gmail.com>
2026-10-14 21:25:41 +00:00
984c242ede [Quantization] Support compressed-tensors MXFP4 MoE with FP8 activations via Mxfp4MoEMethod (#59815)
Signed-off-by: mgoin <mgoin64@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Kevin H. Luu <khluu000@gmail.com>
2026-10-14 13:33:35 -07:00
fe98d90a9d [Dependency] Update Humming and remove NVFP4 scale workaround (#61556)
Signed-off-by: jinzhen.ljz <jinzhen.ljz@antgroup.com>
Co-authored-by: Codex <codex@openai.com>
Co-authored-by: Misha Goin <mgoin64@gmail.com>
Co-authored-by: Kevin H. Luu <khluu000@gmail.com>
2026-10-14 13:32:36 -07:00
Woosuk KwonandClaude Opus 5.5 2e8a5930f6 [Benchmark] Remove the temporary bench forward dummy loader (#61676)
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 13:30:18 -07:00
Juntian Liu 7c3f7e5500 [Perf][DSv4.1] Length-aware candidate block selection in the decode indexer (#61039)
Signed-off-by: Juntian Liu <Juntianl777@gmail.com>
2026-10-14 20:15:58 +00:00
rasmithandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> 2622ee21aa [ROCm][BugFix] Don't use AITER rms_norm as an IR OP on gfx90a when AITER always on. (#59290)
Signed-off-by: Randall Smith <Randall.Smith@amd.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-10-14 12:48:37 -07:00
alpandmottopanikeiku e73fa00981 [Bugfix][Frontend] Apply the JSON schema in Cohere Chat v2 response_format (#60586)
Signed-off-by: mottopanikeiku <176798723+mottopanikeiku@users.noreply.github.com>
Co-authored-by: mottopanikeiku <176798723+mottopanikeiku@users.noreply.github.com>
2026-10-14 19:42:20 +00:00
Chris LeonardandShengqi Chen ded7ae35e5 [Build] Update deep_gemm git tag for locality_domain update (#61491)
Signed-off-by: Chris Leonard <chleonar@redhat.com>
Signed-off-by: Shengqi Chen <harry-chen@outlook.com>
Co-authored-by: Shengqi Chen <harry-chen@outlook.com>
2026-10-14 15:41:16 -04:00
Juntian LiuandClaude Opus 5.5 388a610dfb [Perf][DFlash] Capture context combine and anchor in the draft CUDA graph (#59511)
Signed-off-by: Juntian Liu <Juntianl777@gmail.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-14 15:40:52 -04:00
Wei ZhaoandJee Jee Li 4975fda882 [Dependency] Upgrade FlashInfer to 0.7.1 (#60948)
Signed-off-by: wzhao18 <wzhao18.sz@gmail.com>
Co-authored-by: Jee Jee Li <pandaleefree@gmail.com>
2026-10-14 12:31:08 -07:00
Ayushman Singh 9b5c6707a3 [Bugfix][MRV2] Capture DCP FULL cudagraphs with KV context (#61496)
Signed-off-by: Ayushman Singh <40520701+ayush1399@users.noreply.github.com>
2026-10-14 19:20:48 +00:00
Stefano CastagnettaandMisha Goin 222e00fc44 [CI/Build] Fix test_deepseek_v41_mxfp8_scale_loading after #57797 (#61678)
Signed-off-by: Stefano Castagnetta <scastagnetta@nvidia.com>
Co-authored-by: Misha Goin <mgoin64@gmail.com>
2026-10-14 12:17:59 -07:00
Andreas KaratzasandClaude e96c822660 [Bugfix][ROCm][MiniMax-M3] Reject unbuilt index-head counts in the AITER indexer (#60990)
Signed-off-by: Andreas Karatzas <akaratza@amd.com>
Co-authored-by: Claude <noreply@anthropic.com>
2026-10-14 14:13:48 -05:00
Sandeep AgrawalandKevin H. Luu 21e207efa1 [Bugfix][MRV2][Spec Decode] Honor EAGLE3 single-state draft configuration (#61563)
Signed-off-by: Sandeep Agrawal <sandeepa@nvidia.com>
Co-authored-by: Kevin H. Luu <khluu000@gmail.com>
2026-10-14 12:09:06 -07:00
amd-xavierwangandJartX 06265beb25 [ROCm] Enable fp8 query for Triton attention (#60296)
Signed-off-by: xawang <xawang@amd.com>
Co-authored-by: JartX <sagformas@epdcenter.es>
2026-10-14 15:03:20 -04:00
857ddac4e5 [Perf][MRV2] Skip empty draft-token resets on request admission (#61640)
Signed-off-by: Jakub Zakrzewski <jzakrzewski@nvidia.com>
Co-authored-by: OpenAI Codex <noreply@openai.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
2026-10-14 12:01:01 -07:00
qizixiClaude Opus 5.5mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
c1407a7e86 [Spec Decode][MRV2] Warm up top-k/top-p and greedy rejection kernels for speculative decoding (#59260)
Signed-off-by: zixi-qi <zixi@inferact.ai>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-10-14 14:59:21 -04:00
simon-veitner-redhatandClaude ae94e9c9e8 [Bugfix] Don't crash DeepGEMM MegaMoE warmup when get_bf16_mega_gate_config is missing (#61657)
Signed-off-by: Simon Veitner <sveitner@redhat.com>
Co-authored-by: Claude <noreply@anthropic.com>
2026-10-14 18:58:35 +00:00
Simon Danielsson 3f729dafcc [Perf][ROCm][GLM-5.3-Flash] AITER prefill indexer topk (#60753)
Signed-off-by: simondanielsson <simon.danielsson99@hotmail.com>
2026-10-14 13:48:56 -05:00
lucas-santos-amdandDouglas Lehr e534024e49 [ROCm][Bugfix][MLA] Fix FP8 MLA prefill unpad for interleave-padded heads (#60907)
Signed-off-by: Lucas Santos <Lucas.Santos@amd.com>
Co-authored-by: Douglas Lehr <91553416+dllehr-amd@users.noreply.github.com>
2026-10-14 14:34:28 -04:00
d1d8e6c096 [Perf][Nemotron-Parse] Do image rescale/normalize on the device (#60500)
Signed-off-by: DiegoCao <DiegoCao@users.noreply.github.com>
Co-authored-by: DiegoCao <DiegoCao@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 18:33:22 +00:00
Kevin H. LuuandOpenAI Codex 0b6d07cbe2 [CI/Build] Use FileStore for DeepEP test rendezvous (#56595)
Signed-off-by: Kevin Luu <51931015+khluu@users.noreply.github.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
2026-10-14 14:29:12 -04:00
Robert ShawandClaude d3f6b2a7b3 [Frontend] Clean Client Facing 500 Report (#58043)
Co-authored-by: Claude <noreply@anthropic.com>
2026-10-14 14:27:58 -04:00
Ganesh RandCyrus Leung 47232f3c7f [Bugfix][CI] Fix renderers tests broken by tool name removal from warn logs (#61652)
Signed-off-by: Ganesh R <ganesh.r@amd.com>
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>
2026-10-14 18:14:58 +00:00
Rohan PotdarandClaude Opus 5.5 52a8cc5316 [ROCm][CI] Add dtype to the KDA metadata test's fake model config (#61650)
Signed-off-by: Rohan138 <rohanpotdar138@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 18:04:14 +00:00
Woosuk KwonandClaude Opus 5.5 9d3dfa2886 [Benchmark] Add vllm bench forward: exact-shape forward timing (#61339)
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 10:52:26 -07:00
TaimysandOpenAI Codex 2c74fd1873 [Model Runner V2] Enable batch-sharded sampling for GPT-OSS (#61257)
Co-authored-by: OpenAI Codex <codex@openai.com>
2026-10-14 10:11:11 -07:00
Xunzhuo 89cab55086 [Model] Support Decision 3.0 decision models (vllm-sr/d3) (#61565)
Signed-off-by: Xunzhuo Liu <xunzhuo.liu@amd.com>
2026-10-14 17:10:55 +00:00
zexplorerhjandCodex 3895a8babe [Bugfix] Fix TP loading of small block-FP8 MLA gates in BailingMoeV3 (#61642)
Signed-off-by: zexplorerhj <zhjoneson@163.com>
Co-authored-by: Codex <noreply@openai.com>
2026-10-14 10:09:23 -07:00
SeongJun LeeandWei Zhao e2e3e2b834 [CI] Use an ungated model in the DGX Spark XQA test (#61416)
Signed-off-by: lesj0610 <lesj0610@gmail.com>
Co-authored-by: Wei Zhao <51183510+wzhao18@users.noreply.github.com>
2026-10-14 17:04:07 +00:00
Harry MellorandClaude Opus 5.5 3e851a7803 [Bugfix][Attention] Make the query contiguous for TRTLLM ragged MLA prefill (#61634)
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 16:38:52 +00:00
Zheng GongandCursor 355553ac22 [ROCm][Perf] Drop two dead per-step ops from the AITER sparse MLA metadata build (#58354)
Signed-off-by: ZhengGong-amd <zgong@amd.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-10-14 11:15:51 -05:00
13dbf96093 [Model Loader] Apply EP weight filtering in InstantTensor (#61167)
Signed-off-by: Itay Etelis <92247226+Etelis@users.noreply.github.com>
Co-authored-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com>
Co-authored-by: Misha Goin <mgoin64@gmail.com>
2026-10-14 09:10:10 -07:00
Nick HillandIlia Yastrebov eaf25f675a [NIXL] Prepare KV transfer descriptors as NIXL 1.5 strided lists (#61032)
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Ilia Yastrebov <iyastrebov@nvidia.com>
2026-10-14 18:07:24 +02:00
Nan ZhangandRobert Shaw 9e2b579aab [Model] Support EAGLE3/DFlash aux hidden states for GraniteMoe and GraniteMoeShared (#61110)
Signed-off-by: ZN1010 <neilzhang1010@126.com>
Co-authored-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com>
2026-10-14 12:01:14 -04:00
e2aabc8b15 [Bugfix][KV Offload][P2P] Keep store pins and AbortAck until NIXL transfers stop (#59790)
Signed-off-by: apex-mochen <2756823972@qq.com>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Kevin H. Luu <khluu000@gmail.com>
2026-10-14 18:31:31 +03:00
Juan Pérez de AlgabaandOpenAI Codex 5d88c034f6 [Security] Reject local verifier references in Speculators metadata (#61587)
Signed-off-by: Juan Pérez de Algaba <jperezde@redhat.com>
Co-authored-by: OpenAI Codex <noreply@openai.com>
2026-10-14 15:30:20 +00:00
bf6c80c7e8 [Misc] Remove tool name from warn logs (#59294)
Signed-off-by: Benjamin Merkel <benjamin.merkel@tngtech.com>
Co-authored-by: Benjamin Merkel <benjamin.merkel@tngtech.com>
Co-authored-by: Chauncey <chaunceyjiang@gmail.com>
2026-10-14 08:03:11 -07:00
Itay EtelisandCodex efad400092 [HiSparse] Prefault the shared host pool in parallel (#60869)
Signed-off-by: Itay Etelis <92247226+Etelis@users.noreply.github.com>
Co-authored-by: Codex <noreply@openai.com>
2026-10-14 10:28:26 -04:00
Thuongvu HoCodexmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>Or Ozeri
02653fa22c [Bugfix][KV Offload] Fix inaccurate proportional store dropping (#59140)
Signed-off-by: Thuongvu Ho <thuongvuho@gmail.com>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
2026-10-14 17:05:21 +03:00
Ashish RaoandVadim Gimpelson b4b7eca0ef [Bugfix][Quantization] Clamp FP8 scales before UE8M0 rounding (#58032)
Signed-off-by: Ashish Rao <asrao@nvidia.com>
Co-authored-by: Vadim Gimpelson <156319763+vadiklyutiy@users.noreply.github.com>
2026-10-14 13:56:42 +00:00
7115b70c01 [Bugfix][KV Offload][P2P] Drop a timed-out store job's blocks from the round's parked supply (#59329)
Signed-off-by: Sohom Chakraborty <16609933+sohom-cs@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
2026-10-14 13:22:28 +00:00
Luca MotzandClaude Opus 5.5 2d09fdcdc2 [Quantization] Auto-disable DeepGEMM for Kolibri 1 on SM12x (#61274)
Signed-off-by: Luca Motz <luca.motz@icloud.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 12:59:50 +00:00
3723a1ea96 [Bugfix][Transformers backend] Fix CUDA graph capture for dynamic-rope models (#58064)
Signed-off-by: YongfengX <erkang040804@gmail.com>
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Kevin H. Luu <khluu000@gmail.com>
2026-10-14 05:56:54 -07:00
hclandHarry Mellor 2631afd3fe [Bugfix][Platform] Fall back to torch when device-level NVML memory is unsupported (GB10) (#57378)
Signed-off-by: hclsys <chenglunhu@gmail.com>
Signed-off-by: Chenglun Hu <chenglunhu@gmail.com>
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
2026-10-14 12:45:29 +00:00
Kapil AryaandOr Ozeri a68dbbad05 [Bugfix][KV Offload] Retain offload mappings after unsafe KVCR startup failure (#57500)
Signed-off-by: Kapil Arya <kapila@nvidia.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
2026-10-14 15:43:04 +03:00
775fefa804 [Metrics][KV Offload] Add vllm:kv_offload_config_info - generic info metrics for offloading the offloading connector (#56867)
Signed-off-by: Amir Friedman <Amir.Friedman1@ibm.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
2026-10-14 12:34:37 +00:00
ssingh231219 3a5a6b11a3 [Perf][Model] Batch PaddleOCR-VL vision encoder and projector (#61458)
Signed-off-by: ssingh231219 <simran.singh@artificialy.com>
2026-10-14 19:59:44 +08:00
d39c2fb05a [Bugfix][Gemma4] Quantize audio towers independently (#59935)
Signed-off-by: sudhanshu shukla <sudhanshu112233shukla@users.noreply.github.com>
Signed-off-by: Isotr0py <Isotr0py@outlook.com>
Co-authored-by: sudhanshu shukla <sudhanshu112233shukla@users.noreply.github.com>
Co-authored-by: Isotr0py <Isotr0py@outlook.com>
2026-10-14 11:27:44 +00:00
Lazurite 23fbc2ec2f [Model] Support precomputed image embeddings for LightOnOCR (#61585)
Signed-off-by: Rapisurazurite <43494437+Rapisurazurite@users.noreply.github.com>
2026-10-14 10:54:09 +00:00
xuebwang 81503550f3 [ROCm] Disable AITER FP8/FP4 BMM for MLA by default (#53123)
Signed-off-by: xuebwang-amd <xuebwang@amd.com>
2026-10-14 10:46:02 +00:00
linshi wangandOr Ozeri c67c69e657 [Bugfix][KV Offload] Track allocated block IDs when num_external_tokens is 0 (#47046)
Signed-off-by: linshi-w <linshiw.edu@gmail.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
2026-10-14 13:34:10 +03:00
wang.yuqi 6ecb39eaa7 [Docs] Update structured decisions docs. (#61588)
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io>
2026-10-14 10:28:22 +00:00
Ronaldandaoshen02 19aba3e4d5 [Bugfix][RL] Load attention sinks through weight_loader for layerwise reload (#57797)
Signed-off-by: Ronald1995 <ronaldautomobile@163.com>
Co-authored-by: aoshen02 <aoshen@inferact.ai>
2026-10-14 10:17:58 +00:00
Mikko TukiainenCursormergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
f9b65c30c7 [ROCm][Perf] Make AITER FlyDSL GDN prefill default on gfx942/gfx950 (#60645)
Signed-off-by: Mikko Tukiainen <Mikko.Tukiainen@amd.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-10-14 10:11:57 +00:00
3f466c852c [Bugfix][Parser] Recover genuine tool calls after unclosed Qwen3 <function= opener in prose (#59928)
Signed-off-by: Abhay Joshi <105213625+abhayjoshi201@users.noreply.github.com>
Signed-off-by: Artem Perevedentsev <aperevedents@nvidia.com>
Co-authored-by: Gemini <noreply@google.com>
Co-authored-by: Artem Perevedentsev <aperevedents@nvidia.com>
2026-10-14 13:01:25 +03:00
8a8e2baa02 [Bugfix][Rust Frontend] Inject VLLM_USE_RUST_FRONTEND=1 into vllm-rs managed engine envs (#61586)
Signed-off-by: Isotr0py <Isotr0py@outlook.com>
Co-authored-by: Kimi Code <noreply@moonshot.cn>
Co-authored-by: Bugen Zhao <i@bugenzhao.com>
2026-10-14 17:09:08 +08:00
Sherif Waly be4ce2e505 [Bugfix] Share PCP-aware local GPU indexing for NUMA binding (#61237) 2026-10-14 10:54:03 +02:00
Thien TranandKimi Code 937d6c19df [Perf][GDN] Skip FLA chunk metadata when FlashInfer GDN prefill is active (#61569)
Signed-off-by: Thien Tran <gau.nernst@yahoo.com.sg>
Co-authored-by: Kimi Code <noreply@moonshot.cn>
2026-10-14 01:46:54 -07:00
Kevin H. LuuandMisha Goin d36158f919 [CI] Merge the target branch into stale PRs on /ci run (#61592)
Signed-off-by: khluu <khluu000@gmail.com>
Co-authored-by: Misha Goin <mgoin64@gmail.com>
2026-10-14 01:43:03 -07:00
Artem Perevedentsev bb1438a93f [CI] Fix Unlimited-OCR loading after upstream model_type rename (#61590)
Signed-off-by: Artem Perevedentsev <aperevedents@nvidia.com>
2026-10-14 01:24:24 -07:00
467b024f21 [Rust Frontend] Report vllm-proto schema version in ServerInfo (#61533)
Signed-off-by: Alec Flowers <aflowers@nvidia.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Bugen Zhao <i@bugenzhao.com>
2026-10-14 16:21:48 +08:00
aa0c433a3e [Frontend] Accept --reasoning-parser none as the unset default (#61525)
Signed-off-by: Alec Flowers <aflowers@nvidia.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Bugen Zhao <i@bugenzhao.com>
2026-10-14 15:50:35 +08:00
c9bfa574ce [Online quantization] Extend re-quantization API to MOE (showcase NVFP4 modelopt/CT/quark to any online MOE method) (#60276)
Signed-off-by: Felix Marty <Felix.Marty@amd.com>
Signed-off-by: Tan Pin Siang <tanpinsiang@gmail.com>
Co-authored-by: vllmellm <vllm.ellm@embeddedllm.com>
Co-authored-by: Tan Pin Siang <tanpinsiang@gmail.com>
2026-10-14 15:49:37 +08:00
ChaunceyandCyrus Leung e1726d9142 [Frontend] Add cache salt and tracing to openai decisions (#61550)
Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>
2026-10-14 00:48:44 -07:00
zofia 4dc5add6ea [XPU] [Linear] [MXFP8] enable torch as mxfp8 linear backend (#51933)
Signed-off-by: Zhu, Zufang <zufang.zhu@intel.com>
Signed-off-by: zofia <110436990+zufangzhu@users.noreply.github.com>
2026-10-14 00:48:38 -07:00
Ronald ec8a372566 [ROCm][Bugfix] Pin AITER back to v0.1.24.post1 for DSv4.1 mono decode numerics (#61559)
Signed-off-by: Ronald1995 <ronaldautomobile@163.com>
2026-10-14 00:19:38 -07:00
30e79c3a76 [Core] Update PyTorch to 2.14.0, torchvision to 0.29.0, triton to 3.8.0 (release channel) (#52183)
Signed-off-by: Andrey Talman <atalman@users.noreply.github.com>
Signed-off-by: Andrey Talman <atalman@fb.com>
Co-authored-by: Andrey Talman <atalman@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 00:17:59 -07:00
ec317a4618 [CI] Keep vLLM log propagation on through the first LLM init in caplog_vllm (#61535)
Signed-off-by: khluu <khluu000@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Bugen Zhao <i@bugenzhao.com>
2026-10-14 15:10:35 +08:00
Jiangyun ZhuandClaude Opus 5.5 badfcdbf91 [Perf][GDN] Internal prefill checkpoints for the FlashInfer GDN prefill backend (#60050)
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 14:49:46 +08:00
liranschourItay Etelismergify[bot] <37929162+mergify[bot]@users.noreply.github.com>Or Ozeri
a2fb482732 [KVConnector][P2P] Register peers off the scheduler thread (#57139)
Signed-off-by: Liran Schour <lirans@il.ibm.com>
Co-authored-by: Itay Etelis <92247226+Etelis@users.noreply.github.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
2026-10-14 08:21:57 +03:00
77154ca611 [Perf][Model][Qwen4Exp] QSA: decode four FP8 e4m3 bytes per register below sm_89 (#60992)
Signed-off-by: kimseunghyun <hellonypcteam@gmail.com>
Co-authored-by: kimseunghyun <hellonypcteam@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Thien Tran <gau.nernst@yahoo.com.sg>
2026-10-14 13:06:16 +08:00
Akshan Krithick 02d69c663a [Bugfix][Benchmark] Preserve SSE output across transport chunk boundaries (#51172)
Signed-off-by: Akshan Krithick <akshankrithick305@gmail.com>
2026-10-13 21:58:58 -07:00
a78d050eff [Bugfix] Bump tokenspeed-mla to 2.0.2 to fix nanobind conflict with Triton 3.9 (#61125)
Signed-off-by: Warren Deng <warrdeng@meta.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Codex <noreply@openai.com>
2026-10-13 21:28:07 -07:00
3ccc0a90b6 [KV Connector] Support sleep mode with MooncakeConnector over RDMA (#59625)
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Signed-off-by: Dakai An <dakaian108@gmail.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Dakai An <dakaian108@gmail.com>
Co-authored-by: Dakai An <77474977+andakai@users.noreply.github.com>
2026-10-14 04:18:47 +00:00
Lanze LiuandNing Xie f4b8d43a68 [Bugfix] Keep force-killing the process tree when a child exits mid-walk (#59984)
Signed-off-by: Lanze Liu <lanzetech@gmail.com>
Co-authored-by: Ning Xie <andy.xning@gmail.com>
2026-10-13 20:33:18 -07:00
Shanshan Shen 2521446ac8 [ROCm][Perf] Enqueue the default chain first in CSA/HCA at small batch sizes for DeepSeek-V4 (#61378)
Signed-off-by: shen-shanshan <467638484@qq.com>
2026-10-14 10:57:26 +08:00
2308b6f00b [Perf][MoE] Cheaper triton_kernels 3.8 routing build in the OAI Triton MoE (#58897)
Signed-off-by: Zheng Gong <zgong@amd.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Shanshan Shen <467638484@qq.com>
2026-10-14 10:48:05 +08:00
xaguilar-amd a1f51a0226 [ROCm][MLA] Route gfx942 DCP causal multi-token verify to Triton split-KV decode (#58555)
Signed-off-by: Xavier Aguilar <xavier.aguilarfruto@amd.com>
2026-10-14 10:17:15 +08:00
John CalderonandArtem Perevedentsev 449c05a0ca [CI][Tests] Upgrade torch version for Rubin prerelease (#61468)
Signed-off-by: John Calderon <jcalderon@nvidia.com>
Co-authored-by: Artem Perevedentsev <aperevedents@nvidia.com>
2026-10-14 10:10:00 +08:00
Kevin H. LuuandCodex d5e62627a3 [CI] Shard chat APIs, model evals and model tests automatically (#61475)
Signed-off-by: Kevin Luu <51931015+khluu@users.noreply.github.com>
Co-authored-by: Codex <noreply@openai.com>
2026-10-13 18:16:27 -07:00
Stefano CastagnettaandKartikey Vyas 4e0681ed56 [Quantization] Support packed NVFP4 Qwen4Exp PLE embeddings (#56273)
Signed-off-by: Stefano Castagnetta <scastagnetta@nvidia.com>
Co-authored-by: Kartikey Vyas <kartikey@artificialanalysis.ai>
2026-10-14 08:24:55 +08:00
Almog TavorandItay Alroy 87e7dce3a7 [MRV2] Keep warmup out of the EPLB load stats (#56716)
Signed-off-by: almogtavor <almogtavor@gmail.com>
Co-authored-by: Itay Alroy <75032521+itayalroy@users.noreply.github.com>
2026-10-14 00:00:48 +00:00
Ryan RockandMicah Williamson f43d7e1a0f [CI][AMD] Gate Aiter MHA FA on dtype (#60679)
Signed-off-by: Ryan Rock <ryan.rock@amd.com>
Co-authored-by: Micah Williamson <micah.williamson@amd.com>
2026-10-13 18:55:19 -05:00
josh-schell-amdandMicah Williamson 7a9d3db896 [ROCm] Bump ROCm base images to AITER v1.25.0 (#61230)
Signed-off-by: josh schell <josh.schell@amd.com>
Co-authored-by: Micah Williamson <micah.williamson@amd.com>
2026-10-13 18:55:09 -05:00
c923e7769b [Feat] Add text-only OpenAI Decisions API MVP (#60465)
Signed-off-by: Francisco Javier Arceo <4163062+franciscojavierarceo@users.noreply.github.com>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Misha Goin <mgoin64@gmail.com>
2026-10-14 07:51:08 +08:00
Chaunceyandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> ad8d39cbab [Bugfix] Fix DSpark graph-padding handling in the DeepSeek-V4.1 MoE fused top-k (#59166)
Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-10-14 07:49:52 +08:00
Giancarlo DelfinandWoosuk Kwon 791f92135a [Attention] Set ALWAYS cudagraph support for FA4 (#60908)
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
Co-authored-by: Woosuk Kwon <woosuk@inferact.ai>
2026-10-13 16:22:52 -07:00
Kevin H. LuuandCodex 6a03954b45 [CI] Shard quantization, pooling, multimodal APIs, core kernels and PCP eval automatically (#61477)
Signed-off-by: Kevin Luu <51931015+khluu@users.noreply.github.com>
Co-authored-by: Codex <noreply@openai.com>
2026-10-13 16:15:30 -07:00
Kevin H. LuuandCodex 0da66a6e46 [CI] Shard batch invariance, Kimi prefix cache, EP eval and generate APIs automatically (#61474)
Signed-off-by: Kevin Luu <51931015+khluu@users.noreply.github.com>
Co-authored-by: Codex <noreply@openai.com>
2026-10-13 16:14:18 -07:00
8082b62f32 [Mooncake] Allow Mamba cache mode none for capacity-only workers (#60712)
Signed-off-by: Wei Zhao <51183510+wzhao18@users.noreply.github.com>
Signed-off-by: wzhao18 <wzhao18.sz@gmail.com>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 23:13:58 +00:00
Tim002andThien Tran 96fc2da315 [Perf][Qwen4Exp] Add SM121 TP=2 merged QSA projection plans (#61437)
Signed-off-by: TimRen01 <HolaTim@hotmail.com>
Co-authored-by: Thien Tran <gau.nernst@yahoo.com.sg>
2026-10-13 22:58:44 +00:00
Nick HillandClaude Opus 5.5 78b32b3fdc Revert "[Build] DeepGEMM migration to torch Stable ABI (#59482)" (#61510)
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-13 15:33:50 -07:00
Wenbin ChenandMisha Goin 6463d04601 Fix write-after-write hazard in xpu KV RoPE triton kernel (#46685)
Signed-off-by: Wenbin Chen <wenbin.chen@intel.com>
Co-authored-by: Misha Goin <mgoin64@gmail.com>
2026-10-13 22:17:14 +00:00
698e79f3bd [Model] Kolibri 1 via the Transformers modeling backend (#60827)
Signed-off-by: Luca Motz <luca.motz@icloud.com>
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
2026-10-13 22:17:52 +01:00
Anirudh Ravichandran e90158d912 [Bugfix][MoE] Clamp BLOCK_SIZE_K to the FP8 quantization block in the batched Triton MoE (#60733) 2026-10-13 17:02:17 -04:00
Matthew BonanniandClaude Opus 5.5 292da21f9d [HiSparse] Read GPU-resident KV directly for mixed and BF16 prefills (#60417)
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 16:40:20 -04:00
Wentao Ye c19c263ac8 [GLM 5.3 Perf] Handle two elements at the same time for per token group quant, 2.8% E2E ITL improvement (#60927)
Signed-off-by: yewentao256 <zhyanwentao@126.com>
2026-10-13 13:06:41 -07:00
Andreas KaratzasandClaude 67516b6c42 [Bugfix][ROCm][Kimi-K3] Stage fused KDA decode weights after loading (#61207)
Signed-off-by: Andreas Karatzas <akaratza@amd.com>
Co-authored-by: Claude <noreply@anthropic.com>
2026-10-13 14:55:57 -05:00
djramic 93d0bd9d50 [ROCm][CI] Drop the -inf padding check from the AITER MQA logits test (#61420)
Signed-off-by: Djordje Ramic <djoramic@amd.com>
2026-10-13 15:49:30 -04:00
Itay EtelisRobert Shawmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
a31054f5ef [NIXL] Bump CUDA 12 NIXL to 1.5.0 (#60428)
Signed-off-by: Itay Etelis <92247226+Etelis@users.noreply.github.com>
Co-authored-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-10-13 12:47:41 -07:00
68fc395554 [Bugfix] Keep GateLinear batch invariant under VLLM_BATCH_INVARIANT (#59985)
Signed-off-by: yifanFengg <evanfengff@gmail.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com>
2026-10-13 15:43:54 -04:00
yifanFenggandClaude Fable 5.1 b33cb81237 [Bugfix] Keep compressed-tensors wNa16 linear layers batch invariant (#60524)
Signed-off-by: yifanFengg <evanfengff@gmail.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-13 15:43:40 -04:00
Wentao Ye 2f18d13cf0 [GLM5.3 Perf] Enable fp8 kv cache for GLM5.3 Flash by default, 9.8% E2E Throughput Improvement (#61228)
Signed-off-by: yewentao256 <zhyanwentao@126.com>
2026-10-13 15:29:17 -04:00
Itay EtelisRobert ShawCodexmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
483e0c24c7 [KV Offload] Split replicated-layout pre-faulting across workers (#59545)
Signed-off-by: Itay Etelis <92247226+Etelis@users.noreply.github.com>
Co-authored-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-10-13 19:18:56 +00:00
c068db0e1d [Bugfix][Compile] Replace Transformers AOT caching with graph caching (#58918)
Signed-off-by: fangyao fan <2281693949@qq.com>
Signed-off-by: Taimys <ffy2281693949@gmail.com>
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Taimys <ffy2281693949@gmail.com>
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
2026-10-13 19:17:29 +00:00
Anish ShanbhagandOpenAI Codex 3f550129e8 [Perf][Attention][MiniMax-M3] Select CUTLASS MSA by EAGLE3 decode shape (#55838)
Signed-off-by: Anish Shanbhag <ashanbhag@nvidia.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
2026-10-13 15:05:28 -04:00
GuanLuoandsfeng33 6d1daa9fe5 [Bugfix][Frontend] Expose completion response schemas in OpenAPI (#60926)
Signed-off-by: Guan Luo <gluo@nvidia.com>
Signed-off-by: sfeng33 <4florafeng@gmail.com>
Co-authored-by: sfeng33 <4florafeng@gmail.com>
2026-10-13 18:54:06 +00:00
c5bda658d2 [Model][MoE] Load MegaMoE experts directly into the DeepGEMM layout (#60096)
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Signed-off-by: Aaron Hao <hao-aaron@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Aaron Hao <hao-aaron@users.noreply.github.com>
2026-10-13 11:52:08 -07:00
Matthew BonanniandClaude Opus 5.5 624a5ddafd [Attention] Fetch MLA prefill workspace at use instead of init (#60654)
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 17:25:03 +00:00
e83222396f [ROCm][Perf] W4A16: speed up skinny GEMM decode, including speculative batches (#57199)
Signed-off-by: Robert Esclapez Garcia <robert.garcia@amd.com>
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Co-authored-by: Douglas Lehr <91553416+dllehr-amd@users.noreply.github.com>
2026-10-13 17:21:30 +00:00
Roberto L. CastroandClaude Opus 5.5 7fad7db330 [Perf][Feat] Add generic cuteDSL LL FP32 router (GEMM) (#44343)
Signed-off-by: LopezCastroRoberto <rocastro@redhat.com>
Signed-off-by: Roberto L. Castro <38211239+LopezCastroRoberto@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 12:38:46 -04:00
6849ca696a [Quantization] Add ModelOpt IQ2_XS, IQ2_XXS, and Q8_0 support via b12x (#59604)
Signed-off-by: Andrii Skliar <askliar@nvidia.com>
Signed-off-by: Luke Alonso <lalonso@gmail.com>
Co-authored-by: Andrii Skliar <askliar@nvidia.com>
Co-authored-by: Luke Alonso <lalonso@gmail.com>
2026-10-13 09:06:33 -07:00
Woosuk KwonandClaude Opus 5.5 ac71b17517 [Model Loader] Keep norm layers neutral under dummy weight init (#61323)
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 08:54:24 -07:00
Taksh Kothari 12b005eefa [Bugfix] Don't split a multimodal item that a prefill chunk starts exactly on (#52412)
Signed-off-by: Taksh <takshkothari09@gmail.com>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
2026-10-13 15:32:38 +00:00
Elvir Crnčević b9b401b2ef [Kernel] Fuse DeepEP-v2 decode globalize+align into one launch (#53911)
Signed-off-by: Elvir Crncevic <elvircrn@gmail.com>
2026-10-13 11:10:38 -04:00
wang.yuqi 669f85e686 [Misc] Break chat_utils into more maintainable modules. (#60972)
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io>
2026-10-13 15:08:07 +00:00
Rishabh Agarwal f395a6b1ad [ROCM][Bugfix] Deferring to TRITON_ATTN backend for Prefix lm models instead of ROCM_ATTN and fixed minor bug in Gemma3 Multimodal code (#46754)
Signed-off-by: GodRishUniverse <rishabh.agarwal@amd.com>
Signed-off-by: Rishabh Agarwal <risagarw@amd.com>
Signed-off-by: GodRishUniverse <risagarw@amd.com>
2026-10-13 10:03:48 -05:00
Chris LeonardandShengqi Chen f5fefbffe5 [Build] DeepGEMM migration to torch Stable ABI (#59482)
Signed-off-by: Chris Leonard <chleonar@redhat.com>
Signed-off-by: Shengqi Chen <harry-chen@outlook.com>
Co-authored-by: Shengqi Chen <harry-chen@outlook.com>
2026-10-13 08:01:44 -07:00
Jiangyun ZhuandClaude Opus 5.5 61d9c777bf [CI] Run the Clef pooler test on GPU (#61409)
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 14:42:53 +00:00
de8291e24f [Core] Add lifecycle hooks for worker extensions (#59463)
Signed-off-by: Dakai An <dakaian108@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: aoshen02 <aoshen@inferact.ai>
2026-10-13 21:51:05 +08:00
Thien Tran 0fae39a014 [Perf] per_token_group_quant_8bit: multiply by reciprocal instead of dividing per element (#61343)
Signed-off-by: Thien Tran <gau.nernst@yahoo.com.sg>
2026-10-13 21:37:10 +08:00
31e90ac370 [Attention] Enable FA4 sparse MQA decode on Blackwell (#55866)
Signed-off-by: Simon Veitner <sveitner@redhat.com>
Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com>
Co-authored-by: Matthew Bonanni <mbonanni@redhat.com>
2026-10-13 09:31:37 -04:00
Nathan SalaandShaun Kotek - Nvidia 0838cd6593 [Bugfix] Load and apply Nemotron Super final vision LayerNorm (#61401)
Signed-off-by: Shaun Kotek <skotek@nvidia.com>
Co-authored-by: Shaun Kotek - Nvidia <skotek@nvidia.com>
2026-10-13 12:42:29 +00:00
Hexiang Wang 7342dc0317 [MoRIIO] Require pending remote prefill for READ cache hits (#60638)
Signed-off-by: whx-sjtu <xiaowang990929@gmail.com>
2026-10-13 20:19:53 +08:00
253e02b95e [Perf] Warm up DeepGEMM MegaMoE kernels before serving (#60468)
Signed-off-by: Simon Veitner <sveitner@redhat.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Roberto L. Castro <38211239+LopezCastroRoberto@users.noreply.github.com>
2026-10-13 20:17:48 +08:00
Aditya Ganesh Kumar d9a543c500 [Bugfix][Frontend] Forward --enable-log-outputs to the Cohere Chat v2 (#61089)
Signed-off-by: Aditya Ganesh Kumar <adty910@gmail.com>
2026-10-13 12:01:59 +00:00
818f70c295 [Bugfix] Mask special added tokens in xgrammar JSON, regex and EBNF grammars (#60749)
Signed-off-by: Vadim Gimpelson <vadim.gimpelson@gmail.com>
Signed-off-by: Artem Perevedentsev <aperevedents@nvidia.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Artem Perevedentsev <aperevedents@nvidia.com>
2026-10-13 14:38:23 +03:00
Vadim GimpelsonandClaude cd7517fb17 [Bugfix] Fix ReplaySSM ring shape assertion when TP does not divide Mamba2 n_groups (#60704)
Signed-off-by: Vadim Gimpelson <vadim.gimpelson@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
2026-10-13 11:31:53 +00:00
Linze Shi e6e84f350c [Feature][Pooling] Add offline fixed-length late chunking (#60164)
Signed-off-by: Linze-Shi <linzeshi0@gmail.com>
2026-10-13 11:16:21 +00:00
noone-zhhhandIsotr0py 426f3bd36b [Bugfix][Model][Rust Frontend] Qwen3_ASR: Handle ragged Qwen audio batches from Rust frontend (#60979)
Signed-off-by: zh98 <workallday.zhh@gmail.com>
Co-authored-by: Isotr0py <Isotr0py@outlook.com>
2026-10-13 10:49:39 +00:00
samayp01andArtem Perevedentsev 54db759eea [Bugfix][Structured Output] Reject structural_tag and unhandled specs in backend validation (#58862)
Signed-off-by: Samay <samay.piduru@gmail.com>
Co-authored-by: Artem Perevedentsev <aperevedents@nvidia.com>
2026-10-13 13:27:34 +03:00
Juan Pérez de AlgabaandOpenAI Codex 7173c64e2e [Security] Bound decoded and downloaded media per request (#61226)
Signed-off-by: juanpas97 <juan.perezalgaba@gmail.com>
Signed-off-by: Juan Pérez de Algaba <jperezde@redhat.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
2026-10-13 10:17:14 +00:00
JackRao123andClaude Opus 5.5 877f6df889 [Bugfix][DSv4] Don't bake the C128 compressor skip into CUDA graphs (#59709)
Signed-off-by: Jack Rao <jack.rao@baseten.co>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 10:15:11 +00:00
c7a305ec10 [Bugfix][Mamba] Store conv_kernel - 1 inputs in Mamba2 prefill checkpoints (#60797)
Signed-off-by: Sungsoo Ha <sungsooh@nvidia.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: tomeras91 <57313761+tomeras91@users.noreply.github.com>
2026-10-13 10:09:17 +00:00
Kevin H. LuuandClaude Opus 5.5 206e45d470 [CI] Shard V1 Attention automatically instead of by fixed pytest shards (#61200)
Signed-off-by: khluu <khluu000@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 02:24:06 -07:00
Thien TranandOpenAI Codex 656d0d4506 [Perf] Tune Triton GDN decode tiles on SM100-family GPUs (#61355)
Signed-off-by: Thien Tran <gau.nernst@yahoo.com.sg>
Co-authored-by: OpenAI Codex <noreply@openai.com>
2026-10-13 17:01:12 +08:00
Vadim GimpelsonandClaude 183625d4f9 [Bugfix] Fall back to auto for unquantized MoE layers when --moe-backend is marlin (#60219)
Signed-off-by: Vadim Gimpelson <vadim.gimpelson@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
2026-10-13 08:48:13 +00:00
Harry MellorandClaude Opus 5 6f11c220b5 [Transformers backend] Fuse MoE routers that score after the top-k (#57410)
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-10-13 08:41:23 +00:00
stefankoncarevic 24aa1191f4 [ROCm][Bugfix] Run the AITER AR+RMSNorm fallback test on the platform… (#61350)
Signed-off-by: Stefan Koncarevic <stefan.koncarevic@amd.com>
Signed-off-by: Stefan Koncarevic <Stefan.Koncarevic@amd.com>
2026-10-13 08:35:57 +00:00
Jiangyun Zhu ff305934f1 [Tests][Mamba] Wait for accepted counts in fake MTP proposer (#61342)
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com>
2026-10-13 16:15:47 +08:00
zhouyou 3ad096faf8 [Bugfix][EC Connector] Don't create hash directories on cache-miss lookups in ECExampleConnector (#60826)
Signed-off-by: zhouyou9505 <zhouyou9505@gmail.com>
2026-10-13 15:59:48 +08:00
Jiangyun ZhuClaude Opus 5.5mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2cc2e68d1c [Core][Mamba] Remove leftovers of mamba_cache_mode="all" (#60043)
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com>
Signed-off-by: Jiangyun Zhu <zhujiangyun@inferact.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-10-13 00:58:59 -07:00
bc3ddb198a [Model][Qwen4Exp] FP8 QSA main KV cache on Ampere (sm_80, sm_86) (#60602)
Signed-off-by: kimseunghyun <hellonypcteam@gmail.com>
Signed-off-by: kimseunghyun <incongnito12@gmail.com>
Signed-off-by: Thien Tran <gau.nernst@yahoo.com.sg>
Co-authored-by: kimseunghyun <hellonypcteam@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Thien Tran <gau.nernst@yahoo.com.sg>
2026-10-13 15:30:19 +08:00
Kevin H. LuuandClaude Opus 5.5 4e4eab73ed [CI] Shard Model Executor and L4 Pipeline + Context Parallelism automatically (#61202)
Signed-off-by: khluu <khluu000@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 23:57:32 -07:00
e2ad0ce5f5 [Bugfix][LoRA] Slice expert LoRA by the unpadded per-rank size under TP (#61235)
Signed-off-by: aoshen524 <aoshen524@gmail.com>
Co-authored-by: aoshen524 <aoshen524@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: linitra24 <renshuang.zhou@daocloud.io>
2026-10-13 14:25:16 +08:00
Yongye ZhuandClaude Opus 5.5 a6009ea85b [Kernel][Build] Bump DeepGEMM to the 26/09/30 release and adapt the sparse indexer (#61149)
Signed-off-by: Yongye Zhu <zyy1102000@gmail.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-12 23:24:43 -07:00
Nicolò LucchesiandClaude Opus 5.5 86b1c2ef15 [Attention][DCP] Support a2a DCP comm backend for FlashMLA sparse (#61285)
Signed-off-by: NickLucche <nicolo.lucchesi@mistral.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 13:51:22 +08:00
380638afbb [Model] Add Clef-Flash support (#61308)
Signed-off-by: Tianyu Guo <guoty@inferact.ai>
Signed-off-by: Isotr0py <Isotr0py@outlook.com>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Isotr0py <Isotr0py@outlook.com>
2026-10-13 05:46:37 +00:00
Sriram Kumarmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>Douglas LehrAndreas KaratzasSimon Danielsson
0d2689e0df [ROCm][Perf] Skip redundant sparse index remap on non-indexer layers (#51309)
Signed-off-by: Sriram Kumar <sriramkumar.kishorekumar@amd.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Douglas Lehr <91553416+dllehr-amd@users.noreply.github.com>
Co-authored-by: Andreas Karatzas <akaratza@amd.com>
Co-authored-by: Simon Danielsson <70206058+simondanielsson@users.noreply.github.com>
2026-10-13 05:25:56 +00:00
Sriram KumarDouglas Lehrmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2ce8012fa0 [ROCm][Perf] Skip cleaning sparse prefill MQA logits (#51314)
Signed-off-by: Sriram Kumar <sriramkumar.kishorekumar@amd.com>
Co-authored-by: Douglas Lehr <91553416+dllehr-amd@users.noreply.github.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-10-13 00:21:52 -05:00
Divakar VermaandClaude Sonnet 5 f7ba007b5e [ROCm][MoE] Split AiterExperts by weight-quantization key (#60322)
Signed-off-by: Divakar Verma <divakar.verma@amd.com>
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
2026-10-13 00:13:33 -05:00
b337523c11 [XPU] [Test] Extend spawn-safe sitecustomize monkeypatch to XPU (#57911)
Signed-off-by: Chaojun Zhang <chaojun.zhang@intel.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: xiangdong <40376367+zxd1997066@users.noreply.github.com>
2026-10-13 13:12:10 +08:00
Mookandcjackal 5555a2d8f7 [Bugfix][Model] Fix EXAONE-4.5 image inputs: ViT received un-normalised pixels (#61104)
Signed-off-by: godmook <cmoh4135@naver.com>
Co-authored-by: cjackal <44624812+cjackal@users.noreply.github.com>
2026-10-13 13:02:12 +08:00
Yoni GozlanandBugen Zhao 69d00bc264 [Bugfix] Accept whole-number floats for integer tool arguments in the response parser (#61297)
Signed-off-by: Yoni Gozlan <74535834+yonigozlan@users.noreply.github.com>
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
Co-authored-by: Bugen Zhao <i@bugenzhao.com>
2026-10-13 12:55:20 +08:00
dskweandCyrus Leung 28d679e762 [Bugfix] Preserve independent tokenizer code_revision in snapshot resolution (#61272)
Signed-off-by: Yuan Li <dskwelmcy@163.com>
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>
2026-10-13 04:23:45 +00:00
Yoni GozlanandBugen Zhao 657a2e9417 [Rust Frontend] Accept whole-number floats for integer tool arguments (#61299)
Signed-off-by: Yoni Gozlan <74535834+yonigozlan@users.noreply.github.com>
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
Co-authored-by: Bugen Zhao <i@bugenzhao.com>
2026-10-13 12:22:48 +08:00
RAHUL KUMARandClaude Sonnet 5.5 c5422f4c46 [Test][Renderer] Cover the reasoning -> reasoning_content mirror in chat message parsing (#56129)
Signed-off-by: pctablet505 <55033230+pctablet505@users.noreply.github.com>
Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-13 03:58:32 +00:00
df6c7ad031 [Core][Refactor] Move KV cache planning logic to kv_cache_planning.py (#61183)
Signed-off-by: Dakai An <dakaian108@gmail.com>
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai>
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 03:05:39 +00:00
frida-andersson c077dc663d [ROCm][Perf][DSv4.1] Enable decoder SWA bounded replay (#59039)
Signed-off-by: Frida Andersson <fanderss@amd.com>
2026-10-13 02:19:14 +00:00
xiangdongandKunshang Ji 69d2b6f82d [CI][XPU] Standardize test job labels for intel jobs (#51755)
Signed-off-by: zengxian <xiangdong.zeng@intel.com>
Signed-off-by: xiangdong <40376367+zxd1997066@users.noreply.github.com>
Co-authored-by: Kunshang Ji <kunshang.ji@intel.com>
2026-10-13 10:18:25 +08:00
Fangzhou AiandClaude Opus 5.5 cc2a6eea1a [ROCm][Perf][DSv4.1] Mono decode on every backbone layer (#61082)
Signed-off-by: Fangzhou Ai <fangzhou.ai@amd.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 10:16:50 +08:00
Woosuk KwonandClaude Opus 5.5 f47792af7f [Model Loader] Make dummy weight init dtype-aware (#61134)
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 18:33:26 -07:00
Hexiang WangOpenAI CodexClaude CodeFangzhou Aimergify[bot] <37929162+mergify[bot]@users.noreply.github.com>Simon Danielsson
c6566bb9f6 [ROCm][MLA] Keep DCP prefill context in FP8 through AllGather (#59070)
Signed-off-by: whx-sjtu <xiaowang990929@gmail.com>
Co-authored-by: OpenAI Codex <noreply@openai.com>
Co-authored-by: Claude Code <noreply@anthropic.com>
Co-authored-by: Fangzhou Ai <31551580+Fangzhou-Ai@users.noreply.github.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Simon Danielsson <70206058+simondanielsson@users.noreply.github.com>
2026-10-13 09:20:38 +08:00
VenkyOpenAI GPT-6 AstraDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>Claude Opus 5.5
98f7211162 [Attention][Spec Decode] Multi-query Triton split-K for spec-decode verify, with empty-segment/padding hardening (#56148)
Signed-off-by: venkywonka <23023424+venkywonka@users.noreply.github.com>
Signed-off-by: Venky <23023424+venkywonka@users.noreply.github.com>
Co-authored-by: OpenAI GPT-6 Astra <noreply@openai.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-13 04:37:56 +04:00
189bbfb9f4 Add MoNe experts pruning support (#50415)
Signed-off-by: Ilya Boytsov <ieboytsov@nebius.com>
Signed-off-by: Misha Goin <mgoin64@gmail.com>
Signed-off-by: mgoin <mgoin64@gmail.com>
Co-authored-by: Ilya Boytsov <ieboytsov@nebius.com>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Misha Goin <mgoin64@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 23:18:43 +00:00
1484266938 [Perf] Improve _topk_topp_kernel() for masked logits (#61213)
Signed-off-by: Tomas Ruiz <tomas.ruiz.te@gmail.com>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
2026-10-12 16:01:11 -07:00
1bc6e072b0 [Rust Frontend] Add per-request timing metrics to chat and completions (#59314)
Signed-off-by: wxin <wxin@nvidia.com>
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
Co-authored-by: Bugen Zhao <i@bugenzhao.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 15:39:10 -07:00
Akshay ViswanathanAkshay ViswanathanCursorSimon Danielssonmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>Hongxia Yang
802feb6770 [Bugfix][KVConnector] Fix MoRI-IO WRITE-mode requests stranded in WAITING_FOR_REMOTE_KVS (#57705)
Signed-off-by: Akshay Viswanathan <4953343+akshayv@users.noreply.github.com>
Co-authored-by: Akshay Viswanathan <4953343+akshayv@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Simon Danielsson <70206058+simondanielsson@users.noreply.github.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Hongxia Yang <62075498+hongxiayang@users.noreply.github.com>
2026-10-12 22:29:31 +00:00
Maroon AyoubandFlora Feng 44f0ddfd7b [Bugfix][Parser] Stop re-parsing the whole GLM tool argument per chunk (#61253)
Signed-off-by: Maroon Ayoub <mayoub@redhat.com>
Co-authored-by: Flora Feng <4florafeng@gmail.com>
2026-10-12 21:28:35 +00:00
Almog TavorandRobert Shaw 69d54eb078 [Docs] Fix broken llm-d.ai links on the llm-d integration page (#61022)
Signed-off-by: Almog Tavor <almogtavor@gmail.com>
Co-authored-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com>
2026-10-12 21:26:43 +00:00
Matthew BonanniandClaude Opus 5.5 fe341a53dd [HiSparse] Report host pool in KV cache capacity logs (#60285)
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 20:49:49 +00:00
deedc7be00 [Perf][Humming] Skip the redundant MoE input copy before the w13 GEMM (#59340)
Signed-off-by: lesj0610 <lesj0610@users.noreply.github.com>
Co-authored-by: lesj0610 <lesj0610@users.noreply.github.com>
Co-authored-by: Misha Goin <mgoin64@gmail.com>
2026-10-12 20:24:12 +00:00
Andrey Talman 654b725ea9 [CI] Pace the CRCR nightly report under the relay rate limit (#61269)
Signed-off-by: Andrey Talman <atalman@fb.com>
2026-10-12 13:23:53 -07:00
Mikko Tukiainen abad32a718 [ROCm][Perf] Fix Gemma AR+RMS prefill fallback (#58547)
Signed-off-by: Mikko Tukiainen <Mikko.Tukiainen@amd.com>
Signed-off-by: Mikko Tukiainen <mikko.tukiainen@amd.com>
2026-10-12 19:53:08 +00:00
Jiangyun Zhu fabaecbfa5 [Misc] Expand @ZJY0516 CODEOWNERS to Mamba and KV cache (#61181)
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com>
2026-10-12 19:32:25 +00:00
Martin Hickey 7039c7f17b [Bugfix][Rust Frontend] Omit unset sampling params in render output (#61220)
Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com>
2026-10-12 19:31:03 +00:00
ztysthandOpenAI Codex 1bdd3fa2d5 [Bugfix][Core] Check multimodal chunk boundaries before encoder cache reuse (#59721)
Signed-off-by: ztysth <rnk.liulw@gmail.com>
Signed-off-by: ztysth <166926442+ztysth@users.noreply.github.com>
Co-authored-by: OpenAI Codex <noreply@openai.com>
2026-10-12 19:25:35 +00:00
Martin Hickey 08582942d8 [Bugfix][Frontend] Honor request_id and X-Request-Id in chat and completions render (#61236)
Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com>
2026-10-12 19:17:12 +00:00
Wentao Ye d0948f754d [Bug] Honor spec decode draft model config (#61240)
Signed-off-by: yewentao256 <zhyanwentao@126.com>
2026-10-12 15:13:23 -04:00
Maroon AyoubandClaude d6aca837c3 [Attention] Make the sparse MLA prefill workspace size configurable (#61023)
Signed-off-by: Maroon Ayoub <mayoub@redhat.com>
Co-authored-by: Claude <noreply@anthropic.com>
2026-10-12 14:56:51 -04:00
SeongJun LeeandNing Xie 56e3b081ae [Bugfix][weight loader] Stop warning when a parameter only widens the weight (#60020)
Signed-off-by: lesj0610 <lesj0610@gmail.com>
Co-authored-by: Ning Xie <andy.xning@gmail.com>
2026-10-12 14:33:31 -04:00
fululi12Claude Opus 5mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>lifulu
8055293353 [ROCm][MLA] Cut ~21x host dispatch from the MLA metadata build (#58381)
Signed-off-by: lifulu <fululi12@amd.com>
Signed-off-by: lifulu <lifulu@users.noreply.github.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: lifulu <lifulu@users.noreply.github.com>
2026-10-12 13:11:20 -05:00
c9e62f13c2 [CI][ROCm] Add AITER nightly overlay scripts (#60883)
Signed-off-by: josh schell <josh.schell@amd.com>
Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-authored-by: Micah Williamson <micah.williamson@amd.com>
2026-10-12 13:04:37 -05:00
Woosuk KwonandClaude Opus 5.5 e1a5086256 [Attention] Remove cascade attention (#61126)
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 10:41:07 -07:00
5b0b8f251e [Bugfix][HiSparse] Fix TP group hang on steps without scheduled tokens (#60941)
Signed-off-by: Maroon Ayoub <mayoub@redhat.com>
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
Co-authored-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com>
Co-authored-by: Matthew Bonanni <mbonanni@redhat.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 17:38:12 +00:00
a7b7bc1fbe [Metrics] Count finished requests by watermark status (#58715)
Signed-off-by: Simon Veitner <sveitner@redhat.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com>
2026-10-12 17:21:22 +00:00
Misha GoinandClaude Opus 5.5 45bb710eca [CI] Add nightly Python 3.14 smoke test (#60456)
Signed-off-by: mgoin <mgoin64@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 10:11:26 -07:00
Laith Sakkaandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> 12e6a585ae [Fix][Compile] Declare group_shape in the static FP8 quant fusion … (#61108)
Signed-off-by: Laith Sakka <lsakka@meta.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-10-13 01:09:57 +08:00
Chester c599b5971f [Bugfix][Frontend] Return all logprobs when completions requests logprobs=-1 (#55717)
Signed-off-by: Chester <42577861+superdiaodiao@users.noreply.github.com>
2026-10-12 17:05:55 +00:00
haotiaa 6e6e86078a [Bugfix][Platform] Don't resolve current_platform during import vllm (#60766)
Signed-off-by: Haotian An <haotiaa@amazon.com>
2026-10-12 17:05:34 +00:00
fa39582756 [Frontend] Serve LiquidAI/d1-3B on /v1/systemone (#60676)
Signed-off-by: mgoin <mgoin64@gmail.com>
Co-authored-by: Simon Veitner <sveitner@redhat.com>
Co-authored-by: Claude <noreply@anthropic.com>
2026-10-12 13:00:47 -04:00
Misha GoinandClaude d79b52b182 [Misc] Consolidate csrc shared headers and prune dead code (#60721)
Signed-off-by: mgoin <mgoin64@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
2026-10-12 09:46:22 -07:00
e0317b28ce [Feature] Support batch invariance on ROCm (#52231)
Signed-off-by: Matthew Wong <Matthew.Wong2@amd.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Andreas Karatzas <akaratza@amd.com>
2026-10-12 10:53:26 -05:00
TJian 50b21a2667 [Perf] [Feat] [ROCm] Add densemha support to ROCm AITER Sparse MLA (#49263)
Signed-off-by: tjtanaa <tunjian.tan@embeddedllm.com>
2026-10-12 15:50:28 +00:00
Nick Riasanovsky a686c8e1b7 [Triton] Migrate Triton kernels off tl.make_block_ptr for triton-main compat (#60502)
Signed-off-by: Nick Riasanovsky <njriasan@meta.com>
2026-10-12 11:38:43 -04:00
Shang WangandClaude Opus 5.5 ffec221276 [Bugfix][DSv4.1] Match decoder-replay input_ids dtype for FlashInfer MegaMoE (#61041)
Signed-off-by: Shang Wang <samshang.wang@mail.utoronto.ca>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 14:23:00 +00:00
Hoai VoandClaude Opus 5.5 f3cb10b690 [Model][LoRA] Enable LoRA support for BERT and RoBERTa embedding models (#61033)
Signed-off-by: hoaivoke <vokehoai@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 14:09:53 +00:00
Shang WangandClaude Opus 5.5 716961cf1f [Bugfix] Drop enable_in_kernel_fc2_reduce from FlashInfer CuteDSL MegaMoE config (#59900)
Signed-off-by: Shang Wang <samshang.wang@mail.utoronto.ca>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 09:05:09 -05:00
Ilya Markov a281ad32f0 [EPLB] Add EPLB support to dspark draft models (DeepSeekV4) (#57549)
Signed-off-by: Markov Ilya <markovilya197@gmail.com>
2026-10-12 10:03:09 -04:00
Ilya Markov b3ee560ae8 [Bugfix][MoE] Apply expert_map remapping for CUTLASS FP4 MOE expert parallel (#59042)
Signed-off-by: Markov Ilya <markovilya197@gmail.com>
2026-10-12 10:02:56 -04:00
bzsuni f022975964 [Bugfix] Reject beam search in completion render (#61212)
Signed-off-by: bzsuni <bingzhe.sun@daocloud.io>
2026-10-12 13:49:46 +00:00
4c36d21280 [Bugfix][LoRA] Apply PEFT rank_pattern / alpha_pattern scaling per module (#59801)
Signed-off-by: suraj <surajsharan85@gmail.com>
Co-authored-by: Vadim Gimpelson <156319763+vadiklyutiy@users.noreply.github.com>
Co-authored-by: linitra24 <renshuang.zhou@daocloud.io>
2026-10-12 21:49:30 +08:00
3facf822b7 [Performance] Load HF tokenizers and processors from a local snapshot (#60825)
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Kevin H. Luu <khluu000@gmail.com>
2026-10-12 13:45:54 +00:00
092c35dd15 [Model] Support Cloudflare/clef-omni on /v1/systemone (#61064)
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com>
Signed-off-by: Isotr0py <Isotr0py@outlook.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Isotr0py <Isotr0py@outlook.com>
2026-10-12 21:27:34 +08:00
DimensionSTPandArtem Perevedentsev 314ab8cffd [Bugfix][Structured Output] Drain compilation tasks before releasing the backend (#61047)
Signed-off-by: DimensionSTP <65501090+DimensionSTP@users.noreply.github.com>
Co-authored-by: Artem Perevedentsev <aperevedents@nvidia.com>
2026-10-12 16:25:36 +03:00
Chauncey ac28877f0e [Bugfix] Fix out-of-bounds lookahead in adaptive block verification (#60945)
Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>
2026-10-12 09:21:35 -04:00
stefankoncarevic 8c175f403a [CI][ROCm] Compare greedy output only in the AITER attn+quant test (#61185)
Signed-off-by: Stefan Koncarevic <Stefan.Koncarevic@amd.com>
2026-10-12 08:20:35 -05:00
Rohan PotdarandClaude Opus 5.5 7900d2eabc [Test] Accept tied tokens at the k-th value in the Triton top-k test (#60726)
Signed-off-by: Rohan138 <rohanpotdar138@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 08:18:44 -05:00
Harry Huangandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> c75cb63cd2 [Qwen4Exp] support Sequence Parallelism for Qwen3.8-Flash-Next (#56322)
Signed-off-by: huanghaoyan.hhy <huanghaoyan.hhy@alibaba-inc.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-10-12 20:56:44 +08:00
Tianyu Guo 258b97ff6b [Bugfix][EC Connector] Size transfers by measured encoder output widths (#57904)
Signed-off-by: Tianyu Guo <guoty@inferact.ai>
2026-10-12 12:32:39 +00:00
Liana KolevaandBugen Zhao aa1d44df16 [Bugfix][Rust Frontend] Alias qwen3_xml to qwen3_coder and add qwen3_json for the Qwen JSON format (#60136)
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
Co-authored-by: Bugen Zhao <i@bugenzhao.com>
2026-10-12 20:20:53 +08:00
8d7b93fec2 [Frontend] Report stop_reason on /inference/v1/generate and pass it through derender (#60251)
Signed-off-by: Hyeonki Hong <hhk7734@gmail.com>
Signed-off-by: Shimi Bandiel <shimib@google.com>
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
Co-authored-by: Hyeonki Hong <hhk7734@gmail.com>
Co-authored-by: Mimir <mimir@nousresearch.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Bugen Zhao <i@bugenzhao.com>
2026-10-12 19:37:53 +08:00
32b05addc1 [Bugfix][Rust Frontend] Restore DP wave delivery after XPUB reconnects (#60249)
Signed-off-by: reidliu41 <reid201711@gmail.com>
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
Co-authored-by: Bugen Zhao <i@bugenzhao.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 19:12:43 +08:00
ReidandBugen Zhao f5cda48c90 [Bugfix][Rust Frontend] Export corrupted request metric (#59937)
Signed-off-by: reidliu41 <reid201711@gmail.com>
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
Co-authored-by: Bugen Zhao <i@bugenzhao.com>
2026-10-12 19:12:39 +08:00
3b9e462774 [Bugfix][Rust Frontend] Correct DP routing after request removal (#59631)
Signed-off-by: 0xsensei <prblmslvr.aditya@gmail.com>
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
Co-authored-by: Bugen Zhao <i@bugenzhao.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 19:12:35 +08:00
21445806bd [Rust Frontend] Support logprob_token_ids on OpenAI endpoints (#59549)
Signed-off-by: Wang Zupeng <zupenwang@gmail.com>
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
Co-authored-by: OpenAI Codex <noreply@openai.com>
Co-authored-by: Bugen Zhao <i@bugenzhao.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 19:12:30 +08:00
Thang NguyenandClaude Opus 5.5 ee8f7ea449 [CI] Enroll a fourth batch of five steps in automatic sharding (#60858)
Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 03:46:07 -07:00
MatthewCYM f7a3a7265f [Bugfix][Multimodal] Fix Qwen3-ASR audio preprocessing parity (#57146)
Signed-off-by: MatthewCYM <chenyiming0v0@gmail.com>
2026-10-12 10:42:51 +00:00
Jimmy WangandJuan Pérez de Algaba 2534a96fcf [Docs] List additional unauthenticated endpoints (#61081)
Signed-off-by: Jimmy Wang <53066820+JimmyWang0417@users.noreply.github.com>
Co-authored-by: Juan Pérez de Algaba <124347725+jperezdealgaba@users.noreply.github.com>
2026-10-12 11:47:52 +02:00
Bugen ZhaoandClaude Opus 5.5 5a09062632 [Rust Frontend] Compose answer constraints into parser-owned output grammars (#60781)
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 09:43:48 +00:00
Bugen ZhaoClaude Opus 5.5mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
0466bd36ea [Rust Frontend] Build the Kimi K3 full-output grammar from its parser mode (#59190)
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-10-12 09:43:47 +00:00
Shijin ZhangandChauncey 83773ecdf3 [Frontend] Support CustomToolCall in the non-Harmony Responses API (#57322)
Signed-off-by: Shijin Zhang <75300765+Dovis01@users.noreply.github.com>
Co-authored-by: Chauncey <chaunceyjiang@gmail.com>
2026-10-12 17:22:55 +08:00
zhangchengzhucufe-devandKunshang Ji a33d46b471 [Bugfix][XPU] Forward softcap and alibi_slopes in xpu_ops.flash_attn_varlen_func (#58936)
Signed-off-by: zhangchengzhucufe-dev <zhangchengzhucufe@gmail.com>
Signed-off-by: Kunshang Ji <kunshang.ji@intel.com>
Co-authored-by: Kunshang Ji <kunshang.ji@intel.com>
2026-10-12 17:10:55 +08:00
Yan Ma e05201aa2e [XPU] refine memory release for sleep mode (#61158)
Signed-off-by: Yan Ma <yan.ma@intel.com>
2026-10-12 16:59:38 +08:00
Thuongvu HoandNicolò Lucchesi 99b44c7a42 [Metrics] Add admission rejection metrics (#55812)
Signed-off-by: Thuongvu Ho <thuongvuho@gmail.com>
Co-authored-by: Nicolò Lucchesi <nicolo.lucchesi@mistral.ai>
2026-10-12 08:31:07 +00:00
Cyrus Leung 5bed289707 [DX] Update CODEOWNERS for entrypoints (#60984)
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
2026-10-12 08:06:47 +00:00
4163a80142 [Rust Frontend] Return weight-version spans on final outputs (#61070)
Signed-off-by: Shuolei Wang <shuoleiwang123@gmail.com>
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
Co-authored-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: Bugen Zhao <i@bugenzhao.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 15:56:59 +08:00
Ting SUN f4114e068e [Bugfix][Frontend] Preserve token IDs in chat derender responses (#61065)
Signed-off-by: Ting Sun <suntcrick@gmail.com>
2026-10-12 07:55:06 +00:00
AMD-yanfeiwangandaoshen02 e3bbaf2369 [ROCm][DSv4] Fuse the shared expert into AITER MegaMoE (#61004)
Signed-off-by: Yanfei Wang <yanfei.wang@amd.com>
Co-authored-by: aoshen02 <aoshen@inferact.ai>
2026-10-12 15:53:16 +08:00
Giancarlo DelfinandRishi Puri 36f3972812 [Spec Decode] Skip metadata build for dflash/dspark when all model backends support it (#60692)
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
Co-authored-by: Rishi Puri <20074092+puririshi98@users.noreply.github.com>
2026-10-12 00:09:10 -07:00
Zhenzhong XuandKunshang Ji 6a70913590 [XPU] bump up auto-round-lib to 0.16.0 (#60976)
Signed-off-by: Zhenzhong1 <zhenzhong.xu@intel.com>
Co-authored-by: Kunshang Ji <kunshang.ji@intel.com>
2026-10-12 14:47:59 +08:00
74f4404f58 [Bugfix][Distributed] Own SHM out-of-band buffers before releasing read slots (#61000)
Signed-off-by: Hubert Marek <163992334+hubert-marek@users.noreply.github.com>
Signed-off-by: Ao Shen <aoshen@inferact.ai>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Ao Shen <aoshen@inferact.ai>
2026-10-12 01:44:12 -04:00
Canlin GuoandClaude Opus 5.5 fe4df3df0e [Bugfix][GLM-5.3-Flash] Restore MTP index sharing across draft iterations (#61079)
Signed-off-by: Canlin <canlinguosdu@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 05:41:52 +00:00
d4e92ccc37 [Bugfix][Frontend] Honor endpoint plugin route overrides (#58839)
Signed-off-by: 子华 <huaxi.shx@alibaba-inc.com>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>
2026-10-12 03:58:15 +00:00
liuzhenweiandKunshang Ji bfa7e5b7af [XPU][CI] Enable XPU graph in graph tests (#60784)
Signed-off-by: zhenwei-intel <zhenwei.liu@intel.com>
Co-authored-by: Kunshang Ji <kunshang.ji@intel.com>
2026-10-12 10:39:40 +08:00
Yan Ma f147949be8 [XPU] limit FA kernel block size to multipleOf 64 (#57297)
Signed-off-by: Yan Ma <yan.ma@intel.com>
2026-10-12 10:25:32 +08:00
dingdangmaoandCodex 124b89d5b7 [Bugfix][RecoverSSM] Correct aligned state indices at exact block boundaries (#59962)
Signed-off-by: apex-mochen <2756823972@qq.com>
Co-authored-by: Codex <noreply@openai.com>
2026-10-12 02:08:14 +00:00
Michał GanczarenkoandClaude Opus 5.5 03091e28b8 [Bugfix] Read Ling quantization_config from top-level config for Bailing V3 VL (#59026)
Signed-off-by: Michal Ganczarenko <michal.ganczarenko@intel.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 09:25:29 +08:00
Laith Sakka cc9b0f097d [Fix] MiniMax: fix CUDA IPC handle exchange on cuda-bindings >= 13.4 (#60917)
Signed-off-by: Laith Sakka <lsakka@meta.com>
2026-10-11 20:38:56 -04:00
Nick Hill 3fe917abb1 [Bugfix][MRV2] Fix FlashInfer attention and ReplaySSM warmups on MRV2 (#61013)
Signed-off-by: Nick Hill <nickhill123@gmail.com>
2026-10-11 16:04:41 -07:00
Jared WenClaude Opus 5.5mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
f26d9f432c [Bugfix][GLM-5.3-Flash] Honor --mamba-ssm-cache-dtype for the KDA recurrent state (#59147)
Signed-off-by: Jared Wen <jaredwen@inferact.ai>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-10-11 17:55:00 +00:00
Nick Hill c059bee812 [MRV2] Switch CI tests still running on MRV1 unnecessarily (#61014)
Signed-off-by: Nick Hill <nickhill123@gmail.com>
2026-10-11 09:57:07 -07:00
Ting SUNandCyrus Leung b99fdbe1f8 [Bugfix] Respect the configured offline chat template (#61061)
Signed-off-by: Ting Sun <suntcrick@gmail.com>
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>
2026-10-11 16:27:21 +00:00
93d2cab35e [Security] Fix P2P ZMQ socket exhaustion via unbounded peer sessions (#51504)
Signed-off-by: jperezde <jperezde@redhat.com>
Signed-off-by: Juan Pérez de Algaba <jperezde@redhat.com>
Co-authored-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com>
Co-authored-by: Itay Etelis <92247226+Etelis@users.noreply.github.com>
2026-10-11 11:39:11 -04:00
aoshen02andCodex 239a3fa3a8 [Sleep] Retain draft-private weights with existing parameter selector (#61046)
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: Codex <noreply@openai.com>
2026-10-11 23:30:56 +08:00
3e8e1364ff [Perf][GLM-5.3-Flash] Shard long-context indexer prefill rows across TP (#54951)
Signed-off-by: Zheng Cai <8370601+zigzagcai@users.noreply.github.com>
Signed-off-by: Zheng Cai <caizheng1993@gmail.com>
Signed-off-by: Codex <noreply@openai.com>
Co-authored-by: root <root@rjob-fb29567b88fec0a9-88c9a4c797eb2ec1-0.caizheng.ailab-sys.svc.pjlab.local>
Co-authored-by: OpenAI Codex <noreply@openai.com>
Co-authored-by: zigzagcai <zigzagcai@users.noreply.github.com>
Co-authored-by: Claude Code <noreply@anthropic.com>
Co-authored-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com>
2026-10-11 10:43:37 -04:00
aoshen02 bd5bbce6e4 [RL] Release weight-transfer communicators without stopping inference (#61060)
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Signed-off-by: fh3 <a@b>
2026-10-11 21:15:14 +08:00
tc-mbandCyrus Leung c85616efc6 [Model] Support MiniCPM-V 4.7 (#58674)
Signed-off-by: tc-mb <tianchi_cai@icloud.com>
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>
2026-10-11 10:21:10 +00:00
Subhash Polisetti f425a21208 [Bugfix][Frontend] Support scalar priority and wire priority into LLM.chat() (#57243)
Signed-off-by: subhash polisetti <subhashr161347@gmail.com>
2026-10-11 10:03:33 +00:00
2339bd39d6 [Bugfix][Mooncake] Use the EAGLE replay boundary when retaining sliding-window tails (#61012)
Signed-off-by: Stu Cao <stu@primeintellect.ai>
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai>
Co-authored-by: Codex <noreply@openai.com>
2026-10-11 09:15:18 +00:00
aoshen02andOpenAI Codex 1789f09588 [Frontend] Explain disabled scale-out endpoints with opt-in hints (#61042)
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: OpenAI Codex <noreply@openai.com>
2026-10-11 08:58:07 +00:00
Juan Pérez de AlgabaandCyrus Leung fcae5c9887 [Security] Restrict default CORS origins (#61009)
Signed-off-by: Juan Pérez de Algaba <jperezde@redhat.com>
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>
2026-10-11 08:26:02 +00:00
Xun Sun 26a488ad93 [Fast Start][Bugfix] Snapshot GPU memory after external weights are ready (#60966) 2026-10-11 07:10:04 +00:00
ef42b4385f [Bugfix] Reclaim cached GPU memory during weight loading (#57516)
Signed-off-by: Debanshu Das <debanshu@google.com>
Signed-off-by: Isotr0py <Isotr0py@outlook.com>
Co-authored-by: Isotr0py <Isotr0py@outlook.com>
Co-authored-by: Kimi Code <noreply@moonshot.cn>
2026-10-11 07:03:06 +00:00
Bruno VolpatoandCodex 61f084b5f3 [Bugfix][Multimodal] Fix per-video cache option reuse (#55203)
Signed-off-by: bvolpato <brunocvcunha@gmail.com>
Co-authored-by: Codex <noreply@openai.com>
2026-10-11 06:21:55 +00:00
Yongye ZhuandClaude Opus 5.5 2eb491e18b [Perf][MiniMax-M3] MSA prefill attends with the E4M3 query; bump fmha_sm100 to 968061a (#60684)
Signed-off-by: Yongye Zhu <zyy1102000@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 13:40:45 +08:00
Nick Hill f8471164bf [MRV2] Detatch weight-reload logic from MRV1 (#60937) 2026-10-10 21:15:05 -07:00
bzsuni 01abe975b5 [Bugfix][KV Connector] Fix speculative block reservations across KV cache groups (#59880)
Signed-off-by: bzsuni <bingzhe.sun@daocloud.io>
2026-10-10 20:55:41 -07:00
Shuolei Wangandaoshen02 b89852aad0 [RL][Frontend] Return token-level weight-version spans on final outputs (#59582)
Signed-off-by: Shuolei Wang <shuoleiwang123@gmail.com>
Signed-off-by: Shuolei Wang <948904026@qq.com>
Co-authored-by: aoshen02 <aoshen@inferact.ai>
2026-10-11 11:33:29 +08:00
3c062d391d [Bugfix] Support 0-dim tensors in packed weight transfer (#52235)
Signed-off-by: subhash polisetti <subhashr161347@gmail.com>
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: Codex <noreply@openai.com>
2026-10-11 01:41:06 +00:00
Wentao Ye 276fbcff27 [Deprecation] Remove scheduled deprecated items (#60923)
Signed-off-by: yewentao256 <zhyanwentao@126.com>
2026-10-10 15:45:33 -04:00
Yufeng He 187a0eb98a [Bugfix][Frontend] Forward --enable-log-outputs to the Anthropic handler (#60982)
Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
2026-10-10 16:36:03 +00:00
f2376cdc4d [Fast Start] Preload the FlashInfer autotune table on the weight cache daemon (#60085)
Signed-off-by: liusy58 <mg21330037@smail.nju.edu.cn>
Signed-off-by: Isotr0py <Isotr0py@outlook.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Isotr0py <Isotr0py@outlook.com>
2026-10-10 15:48:06 +00:00
Wentao Ye 3709632ff2 [Refactor] Remove dead code multiple places (#60120)
Signed-off-by: yewentao256 <zhyanwentao@126.com>
2026-10-10 15:41:10 +00:00
Wentao Ye 838526c7e5 [Compile] Fix sampler compile warning (#60680)
Signed-off-by: yewentao256 <zhyanwentao@126.com>
2026-10-10 08:09:45 -07:00
Mook bf16c9c139 [Bugfix][Model] K-EXAONE: restore loading of pre-2.0 configs and apply RoPE to the MTP layer (#60187)
Signed-off-by: godmook <cmoh4135@naver.com>
Signed-off-by: Mook <68294499+Godmook@users.noreply.github.com>
2026-10-10 15:05:55 +00:00
Tony Lin 2bea22002d [BUG] Fix KV offload shared-memory slot selection using worker rank (#58497)
Signed-off-by: Tony Lin <tony.lin@intel.com>
2026-10-10 20:25:11 +08:00
Kay YanandCodex c41b2639e2 [Bugfix][Responses API] Complete cache-write token accounting (#60973)
Signed-off-by: Kay Yan <kay.yan@daocloud.io>
Co-authored-by: Codex <noreply@openai.com>
2026-10-10 09:08:36 +00:00
SeongJun Leeandlesj0610 a98247ab4d [Model] Drop the 4x M-RoPE cache headroom for Qwen3-VL, Qwen3.5 and Qwen3.8 (#59853)
Signed-off-by: lesj0610 <lesj0610@users.noreply.github.com>
Signed-off-by: lesj0610 <lesj0610@gmail.com>
Co-authored-by: lesj0610 <lesj0610@users.noreply.github.com>
2026-10-10 06:51:52 +00:00
wang.yuqi c1692c85f1 [Misc] mv chat_utils.py to renderers. (#60522)
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io>
2026-10-10 06:50:34 +00:00
SeongJun LeeandIsotr0py 699c892dd5 [Perf] Skip the M-RoPE warmup when every holder of the RoPE fuses it (#60246)
Signed-off-by: lesj0610 <lesj0610@gmail.com>
Co-authored-by: Isotr0py <Isotr0py@outlook.com>
2026-10-10 06:47:39 +00:00
liuzhenweiandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> ff440c7805 [BreakableCG] Add eager break for Mamba-family ops (#57881)
Signed-off-by: zhenwei-intel <zhenwei.liu@intel.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-10-10 14:06:57 +08:00
Andreas KaratzasandClaude a205fb3177 [Bugfix][Model] Restore AXK1 config defaults dropped with the vendored config (#60954)
Signed-off-by: Andreas Karatzas <akaratza@amd.com>
Co-authored-by: Claude <noreply@anthropic.com>
2026-10-10 05:54:52 +00:00
devpramodandClaude Opus 5.5 46fb84c2d4 [Model] Support EAGLE3/DFlash aux hidden states for Granite (#60855)
Signed-off-by: Pai, Pramod <pramod.pai@intel.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-10 13:34:10 +08:00
Chauncey fa5cfd48d5 [Refactor] Deduplicate trace header handling in BaseServing (#60828)
Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>
2026-10-10 05:27:58 +00:00
Nick Hill 7d0b4e57aa [ROCm] Remove MRV1 model exceptions (#56360)
Signed-off-by: Nick Hill <nickhill123@gmail.com>
2026-10-10 05:09:19 +00:00
8ea646b827 [Rust Frontend][gRPC] Remove GenerateStream producer task so disconnects abort requests (#60922)
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Bugen Zhao <i@bugenzhao.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Alec Flowers <aflowers@nvidia.com>
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
2026-10-10 04:59:35 +00:00
linitra24 364bc81a9e [Bugfix][LoRA] Fix GPT-OSS gate/up interleaving with mixed MoE LoRA formats (#60812)
Signed-off-by: linitra24 <renshuang.zhou@daocloud.io>
2026-10-10 12:52:51 +08:00
Puneet Sharma 5ca26880ad Optimize ppc64le vLLM runtime image size (#48074)
Signed-off-by: puneetsharma21 <puneet.sharma21@ibm.com>
2026-10-10 12:39:58 +08:00
Guancheng FuandKunshang Ji acec1b1d84 [Bugfix][XPU] Remove stale W4A8 weight after repacking (#57511)
Signed-off-by: Guancheng Fu <guancheng.fu@intel.com>
Co-authored-by: Kunshang Ji <kunshang.ji@intel.com>
2026-10-10 04:27:35 +00:00
Matthew BonanniandClaude Opus 5.5 d5f356dae0 [HiSparse] Own GPU residency instead of editing core block-table rows (#60647)
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 21:25:44 -07:00
SeongJun LeeandMisha Goin 9bd074a996 [Attention] Decode an NVFP4 KV cache with XQA on SM12x (#60452)
Signed-off-by: lesj0610 <lesj0610@gmail.com>
Co-authored-by: Misha Goin <mgoin64@gmail.com>
2026-10-09 22:24:00 -06:00
Misha GoinandClaude Opus 5.5 29e4d2d42e [Frontend][Build] Add vllm download-kernels for FlashInfer precompiled kernels (#58765)
Signed-off-by: mgoin <mgoin64@gmail.com>
Signed-off-by: Misha Goin <mgoin64@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 21:22:35 -07:00
Miłosz GrunwaldandKunshang Ji be0823357f [TEST][XPU][CI] Fix ci uva kernel test instability 2 (#60404)
Signed-off-by: microslaw <milosz.grunwald@intel.com>
Co-authored-by: Kunshang Ji <kunshang.ji@intel.com>
2026-10-10 12:09:13 +08:00
Giulio De Pasquale f1a3c60558 [Feature][Attention] Cover a whole spec-verify in one DiffKV split-KV program (#59085)
Signed-off-by: Giulio De Pasquale <git@depasquale.giugl.io>
2026-10-10 11:55:45 +08:00
2d69083c4a [ROCm][MXFP4] Align the AITER W4A4 MoE intermediate size to 128 (#55104)
Signed-off-by: xiaofei.zheng <xiaofei.zheng@amd.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Fangzhou Ai <31551580+Fangzhou-Ai@users.noreply.github.com>
2026-10-10 11:14:01 +08:00
DimensionSTPandOpenAI Codex 10cc2f6ae2 [Bugfix] Bound GDN speculative decode query widths (#55504)
Signed-off-by: DimensionSTP <65501090+DimensionSTP@users.noreply.github.com>
Co-authored-by: OpenAI Codex <noreply@openai.com>
2026-10-10 11:00:16 +08:00
02aa10539f [Bugfix][ModelLoader] Make enable_weights_track check quantized layers (#57108)
Signed-off-by: siliangchen-amd <SiLiang.Chen@amd.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Shanshan Shen <467638484@qq.com>
2026-10-10 10:44:06 +08:00
aoshen02andClaude Opus 5.5 1dfe9fbdae [Core] Release FlashInfer allreduce workspace with --enable-nccl-comm-suspend (#59360)
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 02:26:40 +00:00
Fangzhou AiandClaude Opus 5.5 193922d6ff [ROCm][Perf][DSv4.1] Mono decode layer for DeepSeek-V4.1-Flash on MI355X (#60397)
Signed-off-by: Fangzhou Ai <fangzhou.ai@amd.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 20:53:18 -05:00
3c802af3cf [Feat][Frontend] structured decisions: DiffusionGemma read strategy (#60466)
Signed-off-by: Matt Mastracci <matthew@mastracci.com>
Signed-off-by: Simon Veitner <sveitner@redhat.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Simon Veitner <sveitner@redhat.com>
Co-authored-by: Misha Goin <mgoin64@gmail.com>
2026-10-10 09:33:22 +08:00
Nick Hill 7d1d8660d6 [MRV2] Miscellaneous code structure updates (#60754)
Signed-off-by: Nick Hill <nickhill123@gmail.com>
2026-10-09 17:58:09 -07:00
9f1c783886 [Test] Allow one fp8 step on K bytes in the kpool decode fuzz test (#60725)
Signed-off-by: Rohan138 <rohanpotdar138@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Andreas Karatzas <akaratza@amd.com>
2026-10-10 00:26:53 +00:00
f8e6e365ae [Bugfix][Mooncake] Save the EAGLE attention proof with hybrid Mamba partial tails (#60695)
Signed-off-by: Dao Le <Dao007forever@gmail.com>
Signed-off-by: Dao007forever <daole@inferact.ai>
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com>
Co-authored-by: Wei Zhao <51183510+wzhao18@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: zjy0516 <riverclouds.zhu@qq.com>
2026-10-10 00:08:08 +00:00
Wei ZhaoandCodex 03a7d3776f [Perf] Support minimum KV splits configuration for TokenSpeed MLA decode (#60345)
Signed-off-by: Wei Zhao <51183510+wzhao18@users.noreply.github.com>
Signed-off-by: wzhao18 <wzhao18.sz@gmail.com>
Co-authored-by: Codex <noreply@openai.com>
2026-10-09 23:33:47 +00:00
04b02fc8a5 [Core] shm tensor arena for efficient cpu->gpu worker broadcast (#51207)
Signed-off-by: Bolin Sun <bolins@nvidia.com>
Signed-off-by: <bolins@nvidia.com>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
2026-10-09 16:13:24 -07:00
Rohan PotdarandClaude Opus 5.5 fa94ad4143 [Test] Allow one bf16 ulp in the fused mHC post/pre residual check (#60724)
Signed-off-by: Rohan138 <rohanpotdar138@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 17:54:06 -05:00
Matthew BonanniandClaude Opus 5.5 4c43254d52 [HiSparse] Report host-pool block residency as HiSparse metrics (#59485)
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 16:45:34 -06:00
Kevin H. LuuandClaude Opus 5.5 783ac12071 [CI] Accept whitespace around /ci commands and reply instead of going silent (#60925)
Signed-off-by: khluu <khluu000@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 15:43:37 -07:00
ed7ae705a1 [Bugfix][MRV2] Use int64 index mappings to fix row-offset overflow (#60736)
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: vllmellm <vllm.ellm@embeddedllm.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-09 14:38:11 -07:00
51144dd83e [Bugfix][Spec Decode] Trim token_ids/logprobs left past a stop string under speculative decoding (#47616)
Signed-off-by: Ting Sun <suntcrick@gmail.com>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 14:22:31 -07:00
Bugen ZhaoandClaude Opus 5.5 35e920d671 [Rust Frontend] Support --stream-interval and per-request stream_interval (#60589)
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 14:10:42 -07:00
Giancarlo Delfin ffdde24f54 [Docs] Document adaptive verification without a confidence head (#60913)
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
2026-10-09 13:54:40 -07:00
Kai HoangandFlora Feng e0cb3efce0 [Bugfix][Structured Output] Decode choice specs as JSON in outlines and LMFE (#60748)
Signed-off-by: hpdkhoa <hpdkhoa2311@gmail.com>
Co-authored-by: Flora Feng <4florafeng@gmail.com>
2026-10-09 20:52:08 +00:00
02e55d2af3 [KV Cache] GLM-5.3-Flash: use the generic packed KV layout; CircularBufferSpec tail with speculative ring slots (#57169)
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai>
Signed-off-by: Dakai An <dakaian108@gmail.com>
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai>
Co-authored-by: OpenAI Codex <noreply@openai.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com>
2026-10-09 15:48:37 -04:00
Misha GoinandClaude 3766a67a32 [WideEP] Explain how to fix a too-old NCCL for DeepEP v2 (#60449)
Signed-off-by: mgoin <mgoin64@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
2026-10-09 12:34:58 -07:00
04462732a4 [Bugfix] Reject max_num_scheduled_tokens=0 at config time instead of hanging (#60714)
Signed-off-by: Vadim Gimpelson <vadim.gimpelson@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Hui Li <40770106+untrall@users.noreply.github.com>
2026-10-09 19:15:10 +00:00
Willow LopezandArtem Perevedentsev ad8a6065f4 [Bugfix] Validate guidance tokenizer compatibility before EngineCore (#52298)
Signed-off-by: Oxygen56 <jiangth99@163.com>
Signed-off-by: Artem Perevedentsev <aperevedents@nvidia.com>
Co-authored-by: Artem Perevedentsev <aperevedents@nvidia.com>
2026-10-09 21:59:02 +03:00
Misha GoinandClaude 1dc60dd367 [CI] Pin pydantic in the mypy pre-commit hook (#60894)
Signed-off-by: mgoin <mgoin64@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
2026-10-09 11:42:41 -07:00
Ning XieandSimon Mo a0f28d73a8 [sharded state loader] support pp in sharded state loader (#52110)
Signed-off-by: Andy Xie <andy.xning@gmail.com>
Co-authored-by: Simon Mo <simon.mo@hey.com>
2026-10-09 11:08:31 -07:00
premsurawutandClaude Opus 5.5 c811943137 [Bugfix][Frontend] Reject beam search with echo and logprobs in /v1/completions (#60876)
Signed-off-by: surawut.j <surawut.jirasaktavee@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 18:06:26 +00:00
Bugen ZhaoandClaude Opus 5.5 5dc9bcd8cd [Rust Frontend] Pass reasoning controls through render -> generate (#60546)
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 11:04:49 -07:00
Bugen ZhaoandClaude Opus 5.5 ce4603cad6 [Bugfix][Rust Frontend] Pass through raw characters in byte-level decode (#59861)
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 11:04:00 -07:00
Bugen ZhaoandClaude Opus 5.5 9a585c62aa [Rust Frontend] Box rarely-set fields of per-token output types (#60588)
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 11:01:16 -07:00
MOVZX ac305c44cb Merge branch 'main' into patches
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 17:41:13 +00:00
1a3a7a8f01 [Bugfix] Treat max_tokens as output upper bound, not input-space reservation (#57035)
Signed-off-by: YuvrajSinghBhadoria2 <245782664+YuvrajSinghBhadoria2@users.noreply.github.com>
Signed-off-by: YuvrajSinghBhadoria2 <yuvrajsinghbhado2030@gmail.com>
Signed-off-by: Yuvraj Singh Bhadoria <yuvrajsinghbhado2030@gmail.com>
Co-authored-by: YuvrajSinghBhadoria2 <245782664+YuvrajSinghBhadoria2@users.noreply.github.com>
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-10-10 01:15:22 +08:00
Andrey TalmanandAndrey Talman 89ac8d971b [CI][Test] Allow 10% ngram acceptance drop under preemption on CUDA in test_async_scheduling_accuracy (#60715)
pre-commit / pre-run-check (push) Canceled after 0s
pre-commit / pre-commit (push) Canceled after 0s
Signed-off-by: Andrey Talman <atalman@users.noreply.github.com>
Co-authored-by: Andrey Talman <atalman@users.noreply.github.com>
2026-10-09 16:59:17 +00:00
stefankoncarevic b74233860d [CI][ROCm] Wait for VRAM to settle before building LLM in test_full_graph (#60872)
Signed-off-by: Stefan Koncarevic <Stefan.Koncarevic@amd.com>
2026-10-09 16:48:43 +00:00
Taewoong KimandCyrus Leung 75e3b17c05 [Frontend] Add cache salt support to generative scoring (#60841)
Signed-off-by: Taewoong Kim <ktw2172@gmail.com>
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>
2026-10-09 16:07:50 +00:00
90ba2f34b5 [Build] Find compatible published wheels for precompiled installs (#58871)
Signed-off-by: greg pereira <grpereir@redhat.com>
Signed-off-by: mgoin <mgoin64@gmail.com>
Co-authored-by: mgoin <mgoin64@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 08:05:54 -07:00
Abhijit Roy 970dc63478 [Docs] Add Giga-Embeddings bidirectional Qwen3 usage to embed docs (#60854)
Signed-off-by: Abhijit <abroy@redhat.com>
2026-10-09 14:41:38 +00:00
Taneem Ibrahim a6b39f0262 [Bugfix] Clean up async client sockets after event loop closure (#57757)
Signed-off-by: Taneem Ibrahim <taneem.ibrahim@gmail.com>
2026-10-09 10:32:47 -04:00
Warren Dengandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> fc39317392 [Bugfix][Kernel] Remove invalid NC from Kimi-K3 KDA autotune keys (#60446)
Signed-off-by: Warren Deng <warrdeng@meta.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-10-09 14:25:41 +00:00
Juan Pérez de Algabaandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> 3d52578e13 [Security] Do not retain Phi-4 MM audio attention matrices (#60614)
Signed-off-by: Juan Pérez de Algaba <jperezde@redhat.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-10-09 22:23:35 +08:00
Mikko TukiainenandCursor 81c993658e [ROCm][Perf] Fuse main QK-norm/RoPE/gate and KV-cache write into the AMD QSA prepare launch (#59732)
Signed-off-by: Mikko Tukiainen <Mikko.Tukiainen@amd.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-10-09 21:14:49 +08:00
Li, Jiang 9adba10f3f [Bugfix] Fix pre-commit check (#60840)
Signed-off-by: jiang1.li <jiang1.li@intel.com>
2026-10-09 12:48:32 +00:00
Suraj Sharanandlinitra24 1ee69dde9c [Bugfix][LoRA] Reject PEFT adapter features vLLM does not implement (#60225)
Signed-off-by: suraj <surajsharan85@gmail.com>
Co-authored-by: linitra24 <renshuang.zhou@daocloud.io>
2026-10-09 16:39:58 +04:00
Yan Ma c17fadcfdc [XPU] fix fused_input_norm for MM (#59865)
Signed-off-by: Yan Ma <yan.ma@intel.com>
2026-10-09 20:16:35 +08:00
YiSheng5andKunshang Ji 0cc0460870 [XPU]Fix the accuracy issue when meet topk_ids=-1 on DP+EP scenarios (#57787)
Signed-off-by: yisheng <yi.sheng@intel.com>
Co-authored-by: Kunshang Ji <kunshang.ji@intel.com>
2026-10-09 20:01:54 +08:00
Louie Tsai 196483e25b [Tools] Recipes sweep workflow (#57307)
Signed-off-by: louie-tsai <louie.tsai@intel.com>
2026-10-09 19:44:09 +08:00
Praneeth Reddy AandLi, Jiang <jiang1.li@intel.com> 1c47ebfc57 [Docs] Fix VLLM_CPU_KVCACHE_SPACE default and units (#60053)
Signed-off-by: Praneeth <praneethreddyarikatla@gmail.com>
Co-authored-by: Li, Jiang <jiang1.li@intel.com>
2026-10-09 19:43:55 +08:00
Chen ChengandIsotr0py d095e61c0b [Bugfix][Spec Decode] Fix GLM-5.3-Flash DFlash2/EAGLE3 aux hidden-state capture with the upstream config (#60763)
Signed-off-by: Chen Cheng <85005591+ischencheng@users.noreply.github.com>
Signed-off-by: Isotr0py <Isotr0py@outlook.com>
Co-authored-by: Isotr0py <Isotr0py@outlook.com>
2026-10-09 11:35:48 +00:00
wang.yuqi 7368fadebd [CI Failure] Move MTEB tests to h200_35gb (#60817)
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io>
2026-10-09 11:34:38 +00:00
Luca MotzandClaude Opus 5.5 83ed0c7c09 [Bugfix][NIXL] Fix receive post-process for different P/D block sizes (#58860)
Signed-off-by: Luca Motz <luca.motz@icloud.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 19:30:11 +08:00
Shreyansh JainandArtem Perevedentsev 28d76f3bad [Bugfix][Structured Output] Reject unsupported regex inside groups (#60359)
Signed-off-by: Shreyansh Jain <shreyanshjain646@gmail.com>
Signed-off-by: Artem Perevedentsev <aperevedents@nvidia.com>
Co-authored-by: Artem Perevedentsev <aperevedents@nvidia.com>
2026-10-09 14:16:13 +03:00
Chaojun Zhang b027ac8312 [XPU] Support register KV offload mmap region as pinned host memory on XPU (#51956)
Signed-off-by: Chaojun Zhang <chaojun.zhang@intel.com>
2026-10-09 18:58:07 +08:00
5cadabfafd [Docs] Add Qwen2.5-Coder-7B-Instruct to batch invariance tested models (#46368)
Signed-off-by: Yuval Luria <yluria@redhat.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
2026-10-09 10:41:29 +00:00
Yan Ma 4e318ac563 [XPU] fix online fp8_per_channel quantization (#56027)
Signed-off-by: Yan Ma <yan.ma@intel.com>
2026-10-09 18:38:01 +08:00
Thang NguyenandClaude Opus 5.5 efd141047d [CI] Enroll a third batch of five steps in automatic sharding (#60757)
Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 03:26:47 -07:00
Thang NguyenandClaude Opus 5.5 b8bd335754 [CI] Enroll five more steps in automatic sharding (#60662)
Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 03:26:27 -07:00
Warren Deng 59ffd73e85 [Bugfix][Kernel] Restore FP fusion for indexer Q quantization (#60716)
Signed-off-by: Warren Deng <warrdeng@meta.com>
2026-10-09 09:46:41 +00:00
Damien LaineandCyrus Leung 2186b89a9a [Bugfix][Model] Voxtral realtime: fix boot OOM and engine crash at max_model_len (#45022)
Signed-off-by: Damien Laine <damien.laine@gmail.com>
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>
2026-10-09 17:40:29 +08:00
Harry MellorandClaude Opus 5.5 2c99ee9333 [Chore] Remove vendored code made redundant by Transformers 5.16.1 (#60623)
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 09:29:03 +00:00
Chao Lei bc713cf8cc [Bugfix]MooncakeConnect add query timeout and retry (#55923) 2026-10-09 09:19:33 +00:00
Taewoong KimandChauncey 18c19eaf23 [Frontend] Add cache salt and tracing to structured decisions (#60786)
Signed-off-by: Taewoong Kim <ktw2172@gmail.com>
Co-authored-by: Chauncey <chaunceyjiang@gmail.com>
2026-10-09 08:53:50 +00:00
a6f43e57d6 [ROCm] Enable the cuMem CUDA-graph pool offload on ROCm (#59523)
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Signed-off-by: indianspeedster <cspandey016@gmail.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: indianspeedster <cspandey016@gmail.com>
2026-10-09 08:29:55 +00:00
The Anh Nguyen bacbbe1878 [Bugfix][NIXL] Guard record_transfer against None telemetry (#57734)
Signed-off-by: The Anh Nguyen <ntheanh201@gmail.com>
2026-10-09 08:22:18 +00:00
Sriram Kumar 560fae64cd [ROCm][Bugfix] Pass per-sequence context lengths to the AITER paged MQA-logits kernels (#59420)
Signed-off-by: Sriram Kumar <sriramkumar.kishorekumar@amd.com>
2026-10-09 08:16:20 +00:00
Maroon AyoubandClaude 8a03ccf7b3 [Frontend] Add return_mm_kwargs to render requests (#60728)
Signed-off-by: Maroon Ayoub <mayoub@redhat.com>
Co-authored-by: Claude <noreply@anthropic.com>
2026-10-09 08:14:13 +00:00
f43e4c39e3 [Structured Output] Allow propertyNames with additionalProperties on xgrammar (#57594)
Signed-off-by: errmakov <ide404@gmail.com>
Signed-off-by: Artem Perevedentsev <aperevedents@nvidia.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Artem Perevedentsev <aperevedents@nvidia.com>
2026-10-09 10:49:47 +03:00
Juntian LiuClaude Opus 5.5mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
efaca9547d [Perf][DSv4.1] Exchange packed FP8 Engram rows across DP and feed wkv MXFP8 (#59923)
Signed-off-by: Juntian Liu <Juntianl777@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-10-09 15:30:18 +08:00
82e89e9944 [Perf][Qwen4Exp] Retune H200 M=4 merged QSA LL-GEMM plan (#60776)
Signed-off-by: zigzagcai <zigzagcai@users.noreply.github.com>
Co-authored-by: zigzagcai <zigzagcai@users.noreply.github.com>
Co-authored-by: Claude Code <noreply@anthropic.com>
2026-10-09 07:18:50 +00:00
19c0bed584 [Fusion] Support 3D activation quantization fusion and enable it for Pixtral (#59612)
Signed-off-by: Jakub Zakrzewski <jzakrzewski@nvidia.com>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Misha Goin <mgoin64@gmail.com>
2026-10-09 15:12:03 +08:00
dd30786fef [CI] Enable Zen CPU image build on AMD CI (#60016)
Signed-off-by: Chinmay Kulkarni <Chinmay.Kulkarni@amd.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Micah Williamson <micah.williamson@amd.com>
2026-10-09 01:40:21 -05:00
Paweł MańczakandKunshang Ji f0a5f111f2 [XPU] Add tuned Mamba SSU configs for Intel Arc Pro B60 (#56765)
Signed-off-by: pmanczak <pawel.manczak@intel.com>
Co-authored-by: Kunshang Ji <kunshang.ji@intel.com>
2026-10-09 14:29:29 +08:00
ad73a4740d [Perf][DSv4.1] Trim the last KV-source layer in decoder replay (#59894)
Signed-off-by: Canlin <canlinguosdu@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Juntian Liu <Juntianl777@gmail.com>
2026-10-09 06:07:49 +00:00
8cbd5d0300 [Frontend] structured decisions: noul and score question types (#60696)
Signed-off-by: Matt Mastracci <matthew@mastracci.com>
Co-authored-by: DeepSeek <noreply@deepseek.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Misha Goin <mgoin64@gmail.com>
2026-10-09 04:48:16 +00:00
Rohan PotdarandClaude Opus 5.5 88a419649e [ROCm][Kimi-K3] Use vLLM's FP8 max in the fused AttnRes quantization (#60717)
Signed-off-by: Rohan138 <rohanpotdar138@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 04:45:13 +00:00
fa312090cb [Frontend] Accept each connection on the least loaded API worker (#60723)
Signed-off-by: hao-aaron <aaronhao@inferact.ai>
Co-authored-by: aoshen02 <aoshen02@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 04:40:13 +00:00
Rajeev Warrier 81d9c45e4b [Docs] [Multimodal] Note that per-request media_io_kwargs needs --trust-request-mm… (#60598)
Signed-off-by: Rajeev Warrier <warrier.rajeev.95@gmail.com>
2026-10-09 04:23:07 +00:00
e31b823bf5 [Bugfix][Tool Parser] Use LongCat tags in the LongCat structural tag (#60747)
Signed-off-by: hpdkhoa <hpdkhoa2311@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Flora Feng <4florafeng@gmail.com>
2026-10-08 21:20:54 -07:00
9824159d94 [Bugfix] Fix InternLM2 tool parser dropping characters from streamed arguments (#60709)
Signed-off-by: Vadim Gimpelson <vadim.gimpelson@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Flora Feng <4florafeng@gmail.com>
2026-10-08 21:20:49 -07:00
Vineeth Sai VarikuntlaandFlora Feng 112a282915 [Bugfix][Frontend] Keep streamed completion text_offset in step with tokens (#60668)
Signed-off-by: Vineeth Sai <vineethsai4444@gmail.com>
Co-authored-by: Flora Feng <4florafeng@gmail.com>
2026-10-08 21:20:43 -07:00
51ef239768 [Rust Frontend] Carry reasoning controls over gRPC (#59836)
Signed-off-by: Alec Flowers <aflowers@nvidia.com>
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
Co-authored-by: Bugen Zhao <i@bugenzhao.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 11:11:19 +08:00
Rémi Delhaye a1cd0ec650 [Model] Add zerank-2 reranker (#60415)
Signed-off-by: rdlhio <contact@rdlh.io>
2026-10-09 03:01:51 +00:00
zzaebokandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> 702ce313f4 [Perf][Multimodal] Sort multimodal encoder inputs by modality (#60196)
Signed-off-by: Jaebok Lee <jaebok9541@naver.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-10-09 10:55:36 +08:00
Tiezhen WANGandKevin H. Luu 295c062db8 Remove sudo from dockerfile command (#60488)
Signed-off-by: Tiezhen WANG <38108242+xianbaoqian@users.noreply.github.com>
Co-authored-by: Kevin H. Luu <khluu000@gmail.com>
2026-10-08 19:24:56 -07:00
Mia OjedaandLucas Wilkinson 4611c2c3d2 [BugFix][DiffusionGemma][CPU] Copy async output snapshots instead of aliasing reused buffers (#59829)
Signed-off-by: ojeda-e <eojeda@league.com>
Co-authored-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com>
2026-10-08 19:16:51 -07:00
201e537d98 [Bugfix][LoRA] Validate incompatible MoE adapter formats (#59924)
Signed-off-by: Brian Weston <brianweston@gmail.com>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Jee Jee Li <pandaleefree@gmail.com>
Co-authored-by: linitra24 <renshuang.zhou@daocloud.io>
2026-10-09 09:49:13 +08:00
Nishant RevurandClaude 581692cc0f [Bugfix][Spec Decode] Fix multimodal embedding merge in Llama 4 and Mistral Large 3 EAGLE drafters (#57633)
Signed-off-by: Nishant Revur <kumrfni@amazon.com>
Co-authored-by: Claude <noreply@anthropic.com>
2026-10-09 01:44:50 +00:00
DGlan 4274ae956c [Bugfix][EC Connector] Release shared memory on NIXL setup failure (#59875)
Signed-off-by: DGlan <162426117+DGlan@users.noreply.github.com>
2026-10-08 17:38:16 -07:00
ab905a885d [CI] Run GH200 test on the shared arm64 CI image (#60253)
Signed-off-by: mgoin <mgoin64@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Kevin H. Luu <khluu000@gmail.com>
2026-10-08 15:22:20 -07:00
KamranandCodex 8d10c38b73 [Perf] Reduce fallback decode CPU by 69% at 32K (#60497)
Signed-off-by: spa5k <79936503+spa5k@users.noreply.github.com>
Co-authored-by: Codex <codex@openai.com>
2026-10-08 21:42:56 +00:00
Yifan QiaoandClaude Opus 5.5 240785b82c [Bugfix] Keep the GLM-5.3 kpool tail and Qwen4 QSA ring out of the null block (#59528)
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 20:44:04 +00:00
sungbin1015andFlora Feng 2ddde19e8e [Frontend] Support min_p in the Responses API (#48084)
Signed-off-by: sungbin1015 <sbin@solbox.com>
Co-authored-by: Flora Feng <4florafeng@gmail.com>
2026-10-08 20:38:17 +00:00
Divakar Verma c5c1161382 [ROCm][Build] Fail fast when setup.py develop deps aren't pip-preinstalled (#59447)
Signed-off-by: Divakar Verma <divakar.verma@amd.com>
2026-10-08 15:28:38 -05:00
ae9adec905 [Frontend] Expose /v1/systemone without an opt-in flag (#60677)
Signed-off-by: Francisco Javier Arceo <4163062+franciscojavierarceo@users.noreply.github.com>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>
2026-10-08 20:21:05 +00:00
72d59adc2c [Bugfix][KV Connector][NIXL] Transfer the whole PLE short-conv page in disaggregated P/D (#59997)
Signed-off-by: Alberto Massidda <amassidda@nvidia.com>
Co-authored-by: Alberto Massidda <amassidda@nvidia.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Artem Perevedentsev <aperevedents@nvidia.com>
2026-10-08 23:58:13 +04:00
Matthew BonanniandClaude Opus 5.5 1afb029839 [Bugfix][HiSparse] Route by swap-row capacity, not the index-group workspace (#60682)
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 19:45:44 +00:00
c23ca06b6d [Bugfix][Structured Output] Feed the token that implicitly ends reasoning to the grammar (#59627)
Signed-off-by: Simon Veitner <sveitner@redhat.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Misha Goin <mgoin64@gmail.com>
Co-authored-by: Yifan Zong <263738792+yzong-rh@users.noreply.github.com>
2026-10-08 19:38:33 +00:00
dingdangmaoandFlora Feng bab0ac61f8 [Bugfix][Frontend] Forward the reasoning wiring to the engine for batch chat completions (#59263)
Signed-off-by: apex-mochen <2756823972@qq.com>
Signed-off-by: sfeng33 <4florafeng@gmail.com>
Co-authored-by: Flora Feng <4florafeng@gmail.com>
2026-10-08 19:32:46 +00:00
Misha GoinandClaude e3a29aa9b3 [Bugfix][MoE] Size the permute scratch from its inputs (#60447)
Signed-off-by: mgoin <mgoin64@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
2026-10-08 15:32:07 -04:00
SeongJun LeeandMisha Goin fff10d2072 [Attention] Allow an NVFP4 KV cache on SM90 with fa2 (#60620)
Signed-off-by: lesj0610 <lesj0610@gmail.com>
Co-authored-by: Misha Goin <mgoin64@gmail.com>
2026-10-08 12:16:22 -07:00
Kevin LiandMisha Goin fc494822ea [Test][Quantization] Gate CUTLASS W4A8 kernel tests to SM90 (#58878)
Signed-off-by: Kevin Li <kevinbean99@gmail.com>
Co-authored-by: Misha Goin <mgoin64@gmail.com>
2026-10-08 19:14:35 +00:00
Divakar VermaandAndreas Karatzas 18751bc6a7 [ROCm][CI] Add coverage for AiterExperts token padding: inf/nan garbage rows (#59334)
Signed-off-by: Divakar Verma <divakar.verma@amd.com>
Co-authored-by: Andreas Karatzas <akaratza@amd.com>
2026-10-08 19:11:45 +00:00
Aarushi JainandAndreas Karatzas 9bbc581560 [Bugfix] Close MessageQueue zmq resources via finalizer to avoid exit (#60472)
Signed-off-by: aarushjain29 <aarushi.jain2@amd.com>
Co-authored-by: Andreas Karatzas <akaratza@amd.com>
2026-10-08 13:53:00 -05:00
Yeonwoo SungandFlora Feng 56e6da2c47 [Bugfix][Frontend] Honor stop_token_ids on the Responses API (#56592)
Signed-off-by: YeonwooSung <neos960518@gmail.com>
Signed-off-by: sfeng33 <4florafeng@gmail.com>
Co-authored-by: Flora Feng <4florafeng@gmail.com>
2026-10-08 18:47:30 +00:00
Ning XieandSimon Mo c4a5ff5c18 [Frontend] shutdown engine core and its sub processes asap when initializing (#52299)
Signed-off-by: Andy Xie <andy.xning@gmail.com>
Co-authored-by: Simon Mo <simon.mo@hey.com>
2026-10-08 11:46:01 -07:00
Shijin Zhang 2274e86842 [Spec Decode][Model] Support DFlash2 draft models with GLM-5.3-Flash (#56983)
Signed-off-by: Shijin Zhang <75300765+Dovis01@users.noreply.github.com>
2026-10-08 14:41:10 -04:00
Misha GoinandClaude 30c7405890 [MoE] MXFP4 oracle: add CUTLASS W4A4 backend and BF16 activation fallback (#60079)
Signed-off-by: mgoin <mgoin64@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
2026-10-08 11:39:09 -07:00
Wei ZhaoWei ZhaoCodexzjy0516Claude Opus 5.5mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
d8fa463e2b [Bugfix][Prefix caching] Fix inconsistent hybrid Mamba prefix cache boundaries (#60533)
Signed-off-by: Wei Zhao <weizha@aws-cmh-slurm-1-login-01.cm.cluster>
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com>
Co-authored-by: Wei Zhao <weizha@aws-cmh-slurm-1-login-01.cm.cluster>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: zjy0516 <riverclouds.zhu@qq.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-10-08 18:33:48 +00:00
kyleliang-nvKyle LiangOpenAI Codexmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
1664c4f444 [Bugfix][MiniMax-M3] Guard sparse decode sentinel indices (#54597)
Signed-off-by: Kyle Liang <kylliang@nvidia.com>
Signed-off-by: Kyle Liang <kyleliang-nv@users.noreply.github.com>
Co-authored-by: Kyle Liang <kyleliang-nv@users.noreply.github.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-10-08 18:27:57 +00:00
TialoandMisha Goin 972195f4f5 [CI][Build] build macos wheels using abi3 (#60312)
Co-authored-by: Misha Goin <mgoin64@gmail.com>
2026-10-08 14:19:20 -04:00
Francisco Javier ArceoandMisha Goin 705e124c05 [Bugfix] Followup to #59299: preserve Unicode in structured decision state prompts (#60651)
Signed-off-by: Francisco Javier Arceo <4163062+franciscojavierarceo@users.noreply.github.com>
Co-authored-by: Misha Goin <mgoin64@gmail.com>
2026-10-08 11:00:03 -07:00
Doug Smith 242e4213fc [Doc] Require GCC 13 for CUDA source builds (#60259)
Signed-off-by: Doug Smith <dosmith@redhat.com>
2026-10-08 09:43:39 -07:00
Elvir CrnčevićandMisha Goin 8674a794c7 [Perf] Enable Kimi K3 shared expert sharding with DeepEP v2 (#56875)
Co-authored-by: Misha Goin <mgoin64@gmail.com>
2026-10-08 16:42:15 +00:00
stefankoncarevic f4dde3132c [CI][ROCm] Reuse AITER's prebuilt MoE module in the modular-kernel harness (#60633)
Signed-off-by: Stefan Koncarevic <stefan.koncarevic@amd.com>
2026-10-08 11:18:05 -05:00
9f3d70e081 [Quantization] Enable shared expert fusion compatibility with online shared_expert quantization (showcase: along Quark MXFP4 routed experts) (#55686)
Signed-off-by: Felix Marty <Felix.Marty@amd.com>
Signed-off-by: fxmarty-amd <felmarty@amd.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
Co-authored-by: Misha Goin <mgoin64@gmail.com>
2026-10-08 12:04:46 -04:00
Mikko TukiainenandCursor e2414ec1db [ROCm][Qwen4Exp] Use fused PLE Triton kernels on the AMD backend (#60021)
Signed-off-by: Mikko Tukiainen <Mikko.Tukiainen@amd.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-10-08 15:47:56 +00:00
Justin BarlowandHarry Mellor 0fc0cefb4e [Bugfix] Don't int-sort attention-type names in --kv-cache-dtype-skip-layers for packed KV dtypes (#60364)
Signed-off-by: lobo235 <7@netlobo.com>
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
2026-10-08 16:34:55 +01:00
e41d5d5543 [Bugfix] Don't warn about missing watermark config for default requests (#60412)
Signed-off-by: Simon Veitner <sveitner@redhat.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com>
2026-10-08 15:29:20 +00:00
Haokai MaandClaude 0751f5272f [Bugfix] Scope deprecated CLI argument warnings to the parser that defines them (#59917)
Signed-off-by: Tyler Ma <158780265+TylerMa-debugg@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
2026-10-08 15:29:04 +00:00
Woosuk Kwon 194bd72be7 [Compile] Remove torch.compile-based sequence parallelism and AsyncTP (#60517) 2026-10-08 08:28:23 -07:00
wangxiyuanandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> 072d49bb6e [Platform] Support platform-specific MLA prefill backend selection (#56959)
Signed-off-by: wangxiyuan <wangxiyuan1007@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-10-08 15:25:21 +00:00
fd928e5f3c [Bugfix] Check batch-invariance support for every attention backend (#60271)
Signed-off-by: yifanFengg <evanfengff@gmail.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com>
2026-10-08 11:19:57 -04:00
JartX 994b8b77be [Bugfix][ROCm] RDNA3 W4A16 MoE: handle the N-first CT weight layout (#60388)
Signed-off-by: JartX <sagformas@epdcenter.es>
2026-10-08 15:06:39 +00:00
kliuae d80ddaac73 [ROCm][Perf] Kimi-K3 Store only the current rank's shards in latent MoE up-proj (#59591)
Signed-off-by: kliuae <kuanfu.liu@embeddedllm.com>
2026-10-08 14:55:20 +00:00
d33872dfeb [Feat] Structured decisions endpoint (/v1/systemone) (#59299)
Signed-off-by: Matt Mastracci <matthew@mastracci.com>
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
Signed-off-by: Francisco Javier Arceo <4163062+franciscojavierarceo@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com>
Co-authored-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com>
Co-authored-by: Francisco Javier Arceo <4163062+franciscojavierarceo@users.noreply.github.com>
Co-authored-by: Chauncey <chaunceyjiang@gmail.com>
2026-10-08 14:48:12 +00:00
yhcheong (vllmellm) 273e099693 [Bugfix][ROCm] Select page-aligned kernel blocks for pooled indexers so the block table addresses storage pages (GLM-5.3-Flash) (#59412)
Signed-off-by: vllmellm <vllm.ellm@embeddedllm.com>
2026-10-08 14:44:58 +00:00
zzaebok 2f7f05c256 [Bugfix][MoRIIO] Prevent remote KV reload after decoder preemption (#53250)
Signed-off-by: Jaebok Lee <jaebok9541@naver.com>
2026-10-08 22:28:16 +08:00
daa9085143 [KV Connector][NIXL] Register HiSparse host pool under single TP Rank (#60161)
Signed-off-by: Pravein Govindan Kannan <pravein.govindan.kannan@ibm.com>
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
Co-authored-by: Matthew Bonanni <mbonanni@redhat.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 07:25:02 -07:00
Matthew BonanniandClaude Opus 5.5 4ed4c4433b [Bugfix][HiSparse] Drop the unaccounted, write-only prefill mirror staging buffer (#60429)
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 14:00:49 +00:00
Vineeth Sai VarikuntlaandHarry Mellor 194da61de1 [Bugfix] Reject negative --device-ids indices instead of selecting the wrong device (#49053)
Signed-off-by: Vineeth Sai <vineethsai4444@gmail.com>
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
2026-10-08 13:01:25 +00:00
458ba2edf8 [ROCm][Kimi-K3] Fuse AttnRes output with per-token FP8 input quantization (#59069)
Signed-off-by: whx-sjtu <xiaowang990929@gmail.com>
Co-authored-by: OpenAI Codex <noreply@openai.com>
Co-authored-by: Fangzhou Ai <31551580+Fangzhou-Ai@users.noreply.github.com>
2026-10-08 12:01:47 +00:00
73c742b09c [Bugfix] Make disable_any_whitespace actually disable whitespace on xgrammar (#58067)
Signed-off-by: Stu Cao <stu@primeintellect.ai>
Signed-off-by: Artem Perevedentsev <aperevedents@nvidia.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Artem Perevedentsev <aperevedents@nvidia.com>
2026-10-08 14:41:14 +03:00
devshah-cohereandBugen Zhao b49c921a27 [Bugfix][Rust Frontend] Always include token_ids in completion stream choices (#60286)
Co-authored-by: Bugen Zhao <i@bugenzhao.com>
Signed-off-by: devshah-cohere <dev.shah@cohere.com>
2026-10-08 19:11:55 +08:00
Andy Lo ba77c4c130 [Bugfix][Attention] Preserve small FP8 softmax weights in Triton attention (#60156)
Signed-off-by: Andy Lo <andy@mistral.ai>
2026-10-08 12:35:22 +02:00
Nicolò Lucchesi e161a0798c [Perf][Core] Reduce free KV cache queue overhead (#60269)
Signed-off-by: NickLucche <nicolo.lucchesi@mistral.ai>
2026-10-08 10:17:11 +00:00
wang.yuqi 459764104d [Frontend] Move offline_utils.py to entrypoints/common. (#58052)
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io>
2026-10-08 09:56:05 +00:00
Muhammad Usman MateenandMisha Goin e73895a5df [Bugfix] Make suppress_stdout redirect fd 1 instead of sys.stdout.fileno() (#59336)
Signed-off-by: usmanmateen <usmanmateen255@gmail.com>
Co-authored-by: Misha Goin <mgoin64@gmail.com>
2026-10-08 09:52:50 +00:00
Chaojun Zhang dfaabf91ac [XPU] [CI] Fix stale ignore path for weight_transfer_metrics tests on Intel CI (#60558)
Signed-off-by: Chaojun Zhang <chaojun.zhang@intel.com>
2026-10-08 17:48:47 +08:00
Kevin H. LuuandClaude Opus 5.5 24be8c0bca [CI] Remove dead in-tree TPU (torch_xla) CI scripts (#60591)
Signed-off-by: khluu <khluu000@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 02:42:07 -07:00
Jiangyun ZhuandClaude Opus 5.5 e265b77982 [CI][Mamba] Use well-conditioned prompts for FlashInfer ReplaySSM parity tests (#60515)
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 17:39:56 +08:00
7816afdb25 [Bugfix][Core] Drain async KV loads in pause(mode="wait"), and pause the drain test only once a request is in flight (#60572)
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Signed-off-by: Stefan Koncarevic <Stefan.Koncarevic@amd.com>
Co-authored-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 09:33:41 +00:00
Kevin H. LuuandClaude Opus 5.5 c4cd88d9fe [CI] Make (H100) Distributed DP + EP optional (#60540)
Signed-off-by: khluu <khluu000@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 02:31:13 -07:00
Harry MellorandClaude Opus 5.5 21d9758459 Remove Python 3.10 support now that it is EOL (#60402)
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 02:30:52 -07:00
liuzhenweiandKunshang Ji 0004536dd0 [XPU][Graph] Replace boolean-mask assignment with torch.where (#60529)
Signed-off-by: zhenwei-intel <zhenwei.liu@intel.com>
Co-authored-by: Kunshang Ji <kunshang.ji@intel.com>
2026-10-08 17:13:48 +08:00
Nicolò Lucchesi 70bdce6133 [Model][GLM-5.3-Flash] Warm up the DCP top-k merge kernels for the kpool indexer (#60032)
Signed-off-by: NickLucche <nicolo.lucchesi@mistral.ai>
2026-10-08 11:05:52 +02:00
7d47ac2ddb [CI] Enroll five steps in automatic sharding (#60492)
Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: khluu <khluu000@gmail.com>
2026-10-08 01:52:59 -07:00
Paweł Mańczak 4cf1307e72 [XPU] Add tuned Mamba SSU configs for 2 more B70 shapes (#57565)
Signed-off-by: pmanczak <pawel.manczak@intel.com>
2026-10-08 16:36:05 +08:00
Ting SUN 0f112d1800 [Bugfix][Pooling] Preserve cache_salt for chunked embeddings (#47696)
Signed-off-by: Ting Sun <suntcrick@gmail.com>
2026-10-08 07:33:28 +00:00
Alec 29ae81fe83 [Rust Frontend] Preserve explicit zero sampling values in gRPC requests (#60115)
Signed-off-by: Alec Flowers <aflowers@nvidia.com>
2026-10-08 07:27:47 +00:00
Rehan KhanandBob fd0216d3b5 [Bugfix][CPU] Fix fused Gumbel-max wrapping-window bias (#60221)
Signed-off-by: Rehan Khan <Rehan.Khan7@ibm.com>
Co-authored-by: Bob <bob@ibm.com>
2026-10-08 15:15:53 +08:00
ARAVINDHAN Tandlinitra24 e58f8dde7f [Bugfix] Use FusedMoEWithLoRA for shared MoE LoRAs on 3D models (#60098)
Signed-off-by: ARAVINDHAN T <arvindhant01@gmail.com>
Co-authored-by: linitra24 <renshuang.zhou@daocloud.io>
2026-10-08 15:04:32 +08:00
xiangdong 032d39e9b4 [XPU][CI] Split Intel jobs to improve B50 utilization (#58817)
Signed-off-by: zengxian <xiangdong.zeng@intel.com>
Signed-off-by: xiangdong <40376367+zxd1997066@users.noreply.github.com>
2026-10-08 14:53:37 +08:00
stefankoncarevicandAndreas Karatzas 814301b268 [Bugfix][Quant] Read safetensors headers of checkpoints with other file names (#60411)
Signed-off-by: Stefan Koncarevic <Stefan.Koncarevic@amd.com>
Co-authored-by: Andreas Karatzas <akaratza@amd.com>
2026-10-08 01:38:41 -05:00
Vadim GimpelsonandClaude ee406adf0c [Bugfix] Fix ZeroDivisionError in Qwen3-VL video sampling when the source fps is unknown (#60501)
Signed-off-by: Vadim Gimpelson <vadim.gimpelson@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
2026-10-08 06:28:44 +00:00
f226af0e11 [Bugfix] Eliminate frontend ZMQ port TOCTOU with inherited listeners (#54113)
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 06:25:27 +00:00
JohnQinAMDClaude Opus 5Ren Yuzhoumergify[bot] <37929162+mergify[bot]@users.noreply.github.com>Douglas Lehr
fb2ac82458 [ROCm][DSv4.1][Perf] Replace the Torch prefill combine with the shared combine_topk_swa_indices kernel (#56638)
Signed-off-by: JohnQinAMD <yanyuan.qin@amd.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Ren Yuzhou <54501155+yuzhouo7@users.noreply.github.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Douglas Lehr <91553416+dllehr-amd@users.noreply.github.com>
2026-10-08 14:15:18 +08:00
Shanshan Shen 95307ac6d4 [Doc] Update ViT CUDA graph design doc (#60519)
Signed-off-by: Shanshan Shen <87969357+shen-shanshan@users.noreply.github.com>
2026-10-08 14:00:58 +08:00
Jiahan Chang (Cyrus) 816173ca9d [Kernel] Remove WAR for LL CuTeDSL BF16 GEMM (#60434)
Signed-off-by: jiahanc <173873397+jiahanc@users.noreply.github.com>
2026-10-08 05:38:07 +00:00
81198e97ba [Refactor] Abstract DeepGemm MegaMOE backend (#58053)
Signed-off-by: Hui Shi <kalinshi@tencent.com>
Co-authored-by: LuneExplorer <liyihan23@mails.tsinghua.edu.cn>
Co-authored-by: ArmageddonKnight <bojian@cs.toronto.edu>
2026-10-08 05:05:39 +00:00
4ac0179f8a [ROCm][Perf] Use native merge_attn_states kernel on gfx942/gfx950 (#60159)
Signed-off-by: mghanimi <mghanimi@amd.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Douglas Lehr <91553416+dllehr-amd@users.noreply.github.com>
2026-10-08 04:49:49 +00:00
775792f2d5 [Bugfix] Reland #53020: tie lm_head.weight for Nemotron Parse when checkpoint omits it (#60272)
Signed-off-by: Anish Kumar <aniskumar@nvidia.com>
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-08 04:40:55 +00:00
4187ade7b8 [Frontend] Consolidate rl entrypoints. (#57849)
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io>
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 21:37:11 -07:00
Hyukjoon LeeClaude Opus 5.5mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>Summer Yang
1fbbfa1657 [Bugfix][Kimi-K3] Declare max_tp_shards on the DSpark MLA KV cache spec (#59733)
Signed-off-by: hyukjlee <hyukjlee@amd.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Summer Yang <girasoleyang@gmail.com>
2026-10-08 00:33:32 -04:00
MattandClaude Opus 5.5 5dd4a628dd [ROCm][Bugfix] Map sleep-mode cuMem memory for the host (#60367)
Signed-off-by: Matthew Wong <Matthew.Wong2@amd.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-08 04:26:12 +00:00
Isotr0pyandShanshan Shen 8352b2427f [Multimodal] Optimize encoder cudagraph padding logic (#58925)
Signed-off-by: Isotr0py <Isotr0py@outlook.com>
Co-authored-by: Shanshan Shen <467638484@qq.com>
2026-10-08 03:38:09 +00:00
Simon DanielssonandDouglas Lehr 87d9996abe [ROCm][Perf][GLM-5.3-Flash] BF16 splitk kernel for sparse MLA decode (#58584)
Signed-off-by: simondanielsson <simon.danielsson99@hotmail.com>
Co-authored-by: Douglas Lehr <91553416+dllehr-amd@users.noreply.github.com>
2026-10-07 21:37:51 -05:00
bnkaiteamandFlora Feng 3722645539 [Bugfix][Frontend] Accept successful 2xx responses for batch output uploads (#60507)
Signed-off-by: bnkaiteam <bnkaiteam@gmail.com>
Co-authored-by: Flora Feng <4florafeng@gmail.com>
2026-10-08 02:32:35 +00:00
caff2b44cb [BugFix] Pick the DP world-group port at bind time via the coordination store (#51018)
Signed-off-by: aoshen02 <aoshen02@users.noreply.github.com>
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: aoshen02 <aoshen02@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Codex <noreply@openai.com>
2026-10-08 02:30:35 +00:00
wangxiyuanandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> a4b14e57b6 [Platform] Allow BackendEnum dynamic register (#41091)
Signed-off-by: wangxiyuan <wangxiyuan1007@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-10-08 09:56:19 +08:00
MattandClaude Opus 5.5 9dba339d07 [Bugfix][ROCm] Keep MLA derived weights in place on weight reload (#60389)
Signed-off-by: Matthew Wong <Matthew.Wong2@amd.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-07 20:38:06 -05:00
Nicolò Lucchesi 8733a21d55 [Core] Bound execute_dummy_batch RPC waits by the execute-model timeout (#60398)
Signed-off-by: NickLucche <nicolo.lucchesi@mistral.ai>
2026-10-08 01:22:04 +00:00
Junyi Yaoandsfeng33 fdfc171a74 [Bugfix][ToolParser] Stream every tool call when one delta completes several (#60354)
Signed-off-by: Junyi Yao <j.yao@wustl.edu>
Signed-off-by: sfeng33 <4florafeng@gmail.com>
Co-authored-by: sfeng33 <4florafeng@gmail.com>
2026-10-08 01:07:35 +00:00
2dd887426e [Bugfix][Frontend] Do not modify caller params in LLM.score() (#59959)
Signed-off-by: 叮当猫 <2756823972@qq.com>
Signed-off-by: apex-mochen <2756823972@qq.com>
Co-authored-by: DeepSeek Harness <noreply@deepseek.com>
Co-authored-by: Taneem Ibrahim <taneem.ibrahim@gmail.com>
Co-authored-by: Codex <noreply@openai.com>
2026-10-08 09:03:08 +08:00
9e3e37cb3c [TEST][XPU][CI] disable tests of unsupported quantization (#60383)
Signed-off-by: microslaw <you@example.com>
Co-authored-by: microslaw <you@example.com>
Co-authored-by: Kunshang Ji <kunshang.ji@intel.com>
2026-10-08 08:56:28 +08:00
Kang NahoonandFlora Feng 1995c0fcd0 [Bugfix][Frontend] Apply output_config and thinking in Anthropic count_tokens (#59180)
Signed-off-by: Kang Nahoon <ejrtks102020@gmail.com>
Co-authored-by: Flora Feng <4florafeng@gmail.com>
2026-10-08 00:24:20 +00:00
Kevin H. LuuandSherlock 1283362e76 [CI] Add headroom and selection coverage for H200 fast lanes (#56775)
Signed-off-by: Kevin Luu <51931015+khluu@users.noreply.github.com>
Co-authored-by: Sherlock <sherlock@raft.ai>
2026-10-07 17:17:56 -07:00
Nicolò Lucchesi a58bdd0d4e [Perf] JIT monitor: hook TileLang JITImpl.compile instead of __call__ (#58685)
Signed-off-by: NickLucche <nicolo.lucchesi@mistral.ai>
2026-10-08 00:11:59 +00:00
8faac3dd2a [Bugfix] [Spec Decode] DeepSeek V4/V4.1 NVFP4: keep ignored MTP/DSpark experts on MXFP4 (#59655)
Signed-off-by: Shobhit Verma <shobhitv@nvidia.com>
Signed-off-by: v-shobhit <161510941+v-shobhit@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Benjamin Chislett <chislett.ben@gmail.com>
2026-10-08 00:10:32 +00:00
YueshenZandsfeng33 ed825aafc0 [Bugfix][Frontend] Fix Responses namespace aliases in named tool calling (#59723)
Signed-off-by: Shen <312058264+YueshenZ@users.noreply.github.com>
Signed-off-by: sfeng33 <4florafeng@gmail.com>
Co-authored-by: sfeng33 <4florafeng@gmail.com>
2026-10-08 00:07:52 +00:00
www6vandsfeng33 38dc8ee500 [Bugfix] Validate tool-parser/tokenizer compatibility at startup (#59749)
Signed-off-by: www6v <www6v@126.com>
Signed-off-by: sfeng33 <4florafeng@gmail.com>
Co-authored-by: sfeng33 <4florafeng@gmail.com>
2026-10-07 23:46:41 +00:00
Dhairyashil R Gandsfeng33 d6fe5dca68 [Bugfix][Parser] Emit buffered post-reasoning text when no tool parser is configured (#58911)
Signed-off-by: Dhairyashil R G <18738735+dhairyashilRG@users.noreply.github.com>
Signed-off-by: sfeng33 <4florafeng@gmail.com>
Co-authored-by: sfeng33 <4florafeng@gmail.com>
2026-10-07 23:21:51 +00:00
Artem Perevedentsev 4ac5a36a8e [CI/Build][NVIDIA] Build NIXL with NIXL EP from source in Rubin image (#60090)
Signed-off-by: Artem Perevedentsev <aperevedents@nvidia.com>
2026-10-08 02:44:17 +04:00
ef63c23d35 [Bugfix][DSv4.1] Skip SWA bounded replay for KV loads that carry the window (#59197)
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 15:28:42 -07:00
Nicolò LucchesiandMistral Vibe d169c5a903 [Docs] Add disagg PD flow-example guidance to /pr-checklist skill (#60111)
Signed-off-by: NickLucche <nicolo.lucchesi@mistral.ai>
Co-authored-by: Mistral Vibe <noreply@mistral.ai>
2026-10-07 22:16:05 +00:00
f38c6778ce [quantization] Add per-token NVFP4 quantization MoE support for ReLU2 (#56740)
Signed-off-by: Xuanteng Huang <xuantengh@nvidia.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: OpenAI Codex <noreply@openai.com>
2026-10-07 22:01:09 +00:00
b803edbc9f [Model] Streamline EmbeddingGemma 2 config resolution and harmonize Triton prefill attention (#60289)
Signed-off-by: Luciano Martins <lucianommartins@users.noreply.github.com>
Co-authored-by: Luciano Martins <lucianommartins@users.noreply.github.com>
Co-authored-by: Matthew Bonanni <mbonanni@redhat.com>
2026-10-07 14:56:40 -07:00
Simon Danielsson 947dd62c07 [CI] Fix failing test_deepseek_v4_rocm_dspark* tests (#60462)
Signed-off-by: simondanielsson <simon.danielsson99@hotmail.com>
2026-10-07 14:18:19 -07:00
Thang NguyenandKimi b47ec440bc [CI] Stamp provenance labels on the torch-nightly image so Initialized Snapshot E2E can run (#59487)
Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
Co-authored-by: Kimi <noreply@moonshot.cn>
2026-10-07 14:15:39 -07:00
Misha GoinandClaude Opus 5.5 a982c81e3e [Bugfix] Limit GlmMoeDsaForCausalLM fp8 KV cache default to SM100 (#60281)
Signed-off-by: mgoin <mgoin64@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 13:37:13 -07:00
Rui 08567505b0 [Bugfix][KV Offload] Restore KVCR construction through the secondary-tier factory (#58088) 2026-10-07 21:36:57 +02:00
e572e02d21 [UX] Skip checkpoint shards an MTP head does not load (#60152)
Signed-off-by: Robert Shaw <robshaw@redhat.com>
Co-authored-by: Robert Shaw <robshaw@redhat.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 15:16:19 -04:00
Moein Khazraee 17c70c8953 [KV Offload] Update KVCR adapter to pool-and-index API (#59899) 2026-10-07 21:14:07 +02:00
b9767299db [Bugfix] Respect Inductor deterministic mode in combo-kernel defaults (#59074)
Signed-off-by: cheesecake <farzanaman99@gmail.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 20:08:11 +01:00
Jinzhen LinandKevin H. Luu 32fbfa15e8 [Bugfix][Quantization] Accept folded NVFP4 scales for Humming A16 (#60337)
Signed-off-by: jinzhen.ljz <jinzhen.ljz@antgroup.com>
Co-authored-by: Kevin H. Luu <khluu000@gmail.com>
2026-10-07 11:04:43 -07:00
Giancarlo Delfin 6ee9be6b3c [Spec Decode] Rename AR speculators to StandaloneAR / TargetDependentAR (#60335)
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
2026-10-07 14:01:59 -04:00
fba462e008 [Bugfix]Fix FlashInfer warmup crash: autotune(tuning_buckets=...) must round M up, not down (Invalid MXFP8 split-K tactic with spec-decode) (#58165)
Signed-off-by: Yongye Zhu <zyy1102000@gmail.com>
Co-authored-by: Yongye Zhu <zyy1102000@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 14:01:21 -04:00
554340f3d3 [Attention] Support NVFP4 KV cache on SM8x and SM12x with FlashInfer (#46963)
Signed-off-by: lesj0610 <lesj0610@users.noreply.github.com>
Signed-off-by: mgoin <mgoin64@gmail.com>
Co-authored-by: lesj0610 <lesj0610@users.noreply.github.com>
Co-authored-by: mgoin <mgoin64@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
2026-10-07 13:24:21 -04:00
3d6853b32c [Bugfix] Seed hybrid mamba state index with mamba_block_size on prefix-cache hits (#55601)
Signed-off-by: pondzikk <pondzik@gmail.com>
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: zjy0516 <riverclouds.zhu@qq.com>
Co-authored-by: ptorsten <1652174+ptorsten@users.noreply.github.com>
2026-10-07 17:09:50 +00:00
Harry MellorandClaude Opus 5.5 a8260cbc5c [CI] Bump Transformers version to 5.19.0 (#60381)
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 10:01:41 -07:00
d64a18832b [ROCm][Triton] Migrating MiniMax-M3 kernels from make_block_ptr to tensor descriptors (#59285)
Signed-off-by: jpvillam <juan.villamizar@amd.com>
Signed-off-by: Daniel <danichan@amd.com>
Co-authored-by: jpvillam <juan.villamizar@amd.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Rohan Potdar <rohan.potdar@amd.com>
Co-authored-by: Daniel <danichan@amd.com>
2026-10-07 11:49:59 -05:00
MohitAMDandCursor 240ee8de9e [Bugfix][KV Connector][ROCm] Fix MoRI-IO KV block-offset for speculative decoding (MTP) (#55053)
Signed-off-by: Mohit Deopujari <Mohit.Deopujari@amd.com>
Signed-off-by: MohitAMD <mohit.deopujari@amd.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-10-07 11:33:30 -05:00
Andy Lo c741bfca70 [Docs] Add review areas for andylolu2 (#60416)
Signed-off-by: Andy Lo <andy@mistral.ai>
2026-10-07 16:05:29 +00:00
pmanczak f7f62d1916 [XPU][MoE] Tune Triton fused MoE for Intel XPU (#53065) 2026-10-08 00:02:13 +08:00
xaguilar-amdandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> 9367d8b969 [Bugfix][MLA] Scope DSpark non-causal capability to builder layers (#57992)
Signed-off-by: Xavier Aguilar <xavier.aguilarfruto@amd.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-10-07 10:22:32 -05:00
stefankoncarevic 71f9a74920 [CI][ROCm] Re-enable the DSv4.1 decoder replay graph test on ROCm (#60392)
Signed-off-by: Stefan Koncarevic <Stefan.Koncarevic@amd.com>
2026-10-07 09:57:52 -05:00
Alec 14e3902d51 [Frontend][Rust] Add gRPC forbidden token sequences and cache usage (#59837)
Signed-off-by: Alec Flowers <aflowers@nvidia.com>
2026-10-07 07:46:44 -07:00
Mikko TukiainenandCursor 548597367e [ROCm][Perf] Add AITER FlyDSL GDN prefill backend (#57560)
Signed-off-by: Mikko Tukiainen <Mikko.Tukiainen@amd.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-10-07 09:44:22 -05:00
Micah WilliamsonandFangzhou Ai f6e3944e31 [ROCm][DSv4] AITER MegaMoEV2 Integration For DeepSeek V4 (#59685)
Signed-off-by: Micah Williamson <micah.williamson@amd.com>
Co-authored-by: Fangzhou Ai <31551580+Fangzhou-Ai@users.noreply.github.com>
2026-10-07 09:35:53 -05:00
c83935b300 [Bugfix][Model Loader] Support reload_weights with runai_streamer load format (#58124)
Signed-off-by: Viktor Koukouliev <vkoukouliev@nvidia.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Isotr0py <Isotr0py@outlook.com>
2026-10-07 14:04:35 +00:00
00CC 5281e49908 [ROCm][Perf] Enable medium-skinny dispatch in RDNAHybridW4A16LinearKernel (#52619)
Signed-off-by: tangzzycc <3081129260@qq.com>
2026-10-07 13:21:43 +00:00
Divakar VermaandAndreas Karatzas efb8d85f24 [ROCm][CI] Add FP32 rounding slack to accuracy test near-tie upper bounds (#60292)
Signed-off-by: Divakar Verma <divakar.verma@amd.com>
Co-authored-by: Andreas Karatzas <akaratza@amd.com>
2026-10-07 08:19:22 -05:00
2a54f6b625 [ROCm][Perf][GLM-5.3-Flash] Enable AITER fused shared experts on gfx950 (#59221)
Signed-off-by: Jin Tao <jin.tao@amd.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Simon Danielsson <70206058+simondanielsson@users.noreply.github.com>
2026-10-07 20:53:40 +08:00
d9503dc503 [Bugfix][Qwen4Exp] Derive the QSA head_dim from the unsharded head count (#59945)
Signed-off-by: lesj0610 <lesj0610@godoiksan.org>
Co-authored-by: lesj0610 <lesj0610@godoiksan.org>
Co-authored-by: Thien Tran <gau.nernst@yahoo.com.sg>
2026-10-07 12:24:40 +00:00
stefankoncarevic 3ca00a8261 [CI] Skip EmbeddingGemma2 pooling tests needing transformers >= 5.19.0 (#60371)
Signed-off-by: Stefan Koncarevic <stefan.koncarevic@amd.com>
2026-10-07 10:22:31 +00:00
SeongJun Lee 6b75dfb8a1 [Misc] Report a missing DeepSelect extension only when it is requested (#60242)
Signed-off-by: lesj0610 <lesj0610@gmail.com>
2026-10-07 09:53:17 +00:00
Kevin H. LuuandClaude Opus 5.5 02d8a94e61 [CI][ROCm] Skip DSv4.1 decoder replay graph test on ROCm (#60378)
Signed-off-by: khluu <khluu000@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 02:38:31 -07:00
df417f780a [Bugfix][MoRIIO] Exclude synchronous READ destinations from KV zeroing (#59164)
Signed-off-by: Yichao Zhu <Yichao.Zhu@amd.com>
Signed-off-by: whx-sjtu <xiaowang990929@gmail.com>
Signed-off-by: Hexiang Wang <56632993+whx-sjtu@users.noreply.github.com>
Co-authored-by: Yichao Zhu <Yichao.Zhu@amd.com>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai>
2026-10-07 16:35:56 +08:00
42ecc8eebe [CI][RL] Consolidate RL entrypoint tests under tests/entrypoints/rl (#59948)
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io>
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: wang.yuqi <yuqi.wang@daocloud.io>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 08:29:09 +00:00
Bugen ZhaoandClaude Opus 5.5 bc8dd4ad27 [Rust Frontend] Share SchemaRoot between argument coercion and grammars (#59408)
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 08:28:27 +00:00
stefankoncarevic 90f4936d9e [CI/Build] Widen the ragged prefill scoring tolerance, drop batch invariance (#60192)
Signed-off-by: Stefan Koncarevic <Stefan.Koncarevic@amd.com>
2026-10-07 16:21:47 +08:00
Bugen ZhaoandClaude Opus 5.5 7436a7f119 [Rust Frontend] Backtrack from safe text at a complete marker (#59563)
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 07:45:15 +00:00
e6fc81bc78 [Bugfix][Scheduler] Let pooling chunked prefill use full context (#48039)
Signed-off-by: Tommy <tran.tommy@hotmail.com>
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
2026-10-07 07:32:00 +00:00
cbee82324c [Docs][RL] Move pause/resume and batch invariance examples to examples/rl (#59949)
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io>
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: wang.yuqi <yuqi.wang@daocloud.io>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 06:22:10 +00:00
SrirajandClaude 43b4aaea3e [Bugfix][Frontend] Return 4xx for client errors in /cohere/v2/chat (#60309)
Signed-off-by: sricursion <sricursion@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
2026-10-07 04:55:34 +00:00
Turner JabbourandClaude Sonnet 5 aecb717f10 [CI] Allowlist-shrink batch 2: wire 9 tests/models/ files into Buildkite, fix 2 stale bugs found (#59050)
Signed-off-by: Turner Jabbour <doubleujabbour@gmail.com>
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
2026-10-06 21:16:53 -07:00
Mohammad HijjawiandCyrus Leung ef014bdf06 [Bugfix][Frontend] Return logprobs in Cohere Chat v2 responses (#59072)
Signed-off-by: Mohammad Hijjawi <mohammad.hijjawi1997@gmail.com>
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>
2026-10-07 04:13:52 +00:00
Hyukjoon LeeandClaude Opus 5.5 2342343746 [ROCm][DCP] Default MLA DCP verify to the round-robin asm decode where it applies (#59965)
Signed-off-by: hyukjlee <hyukjlee@amd.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 03:10:54 +00:00
8a26869bca [Bugfix] Index Mamba align state copies by request slot in MRV2 (#60210)
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com>
Co-authored-by: Karl0007 <44745209+Karl0007@users.noreply.github.com>
Co-authored-by: Yongqin Wang <yongqinwang@roblox.com>
2026-10-07 02:32:46 +00:00
956960f85f [Perf][MRV2] Avoid stream syncs in NonUvaBuffer host-to-device copies (#60215)
Signed-off-by: cheesecake <farzanaman99@gmail.com>
Co-authored-by: William Joyner <133391209+joynerwk03@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 01:51:12 +00:00
Robert ShawandClaude Code 283d76d56a [Bugfix] Widen ep_gather output offset to int64 (#58373)
Signed-off-by: Robert Shaw <robertgshaw2@gmail.com>
Co-authored-by: Claude Code <noreply@anthropic.com>
2026-10-07 01:45:04 +00:00
fe51194e2d [Bugfix][MiniMax-M3] Fence decode indexer shared-memory stage reuse (#60194)
Signed-off-by: Vladimir Anisimov <vanisimov@nvidia.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: OpenAI Codex <noreply@openai.com>
Co-authored-by: Thien Tran <gau.nernst@yahoo.com.sg>
2026-10-07 01:35:12 +00:00
Kevin H. LuuandClaude Opus 5.5 4b4eb6277a [CI] Remove AMD mirror of CPU-only Rust frontend cargo steps (#60323)
Signed-off-by: khluu <khluu000@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 18:30:52 -07:00
Micah WilliamsonandKevin H. Luu 13f8c453ee [CI][Releases] Trigger perf-eval Through Release Pipeline (#52213)
Signed-off-by: Micah Williamson <micah.williamson@amd.com>
Co-authored-by: Kevin H. Luu <khluu000@gmail.com>
2026-10-06 18:27:27 -07:00
Miłosz Grunwald 703c29179c [TEST][XPU][CI] Generalize model runner prefill tail classification tests to all platforms (#60201)
Signed-off-by: microslaw <milosz.grunwald@intel.com>
2026-10-07 08:59:20 +08:00
SrirajandFlora Feng db6e3cd8c4 [Bugfix][Frontend] Log the full request body when max_log_len is unset (#60310)
Signed-off-by: sricursion <sricursion@gmail.com>
Co-authored-by: Flora Feng <4florafeng@gmail.com>
2026-10-07 00:42:17 +00:00
jxjandOpenAI Codex 6c2925fe5f [Bugfix][Core] Preserve speculative metrics when coalescing outputs (#55708)
Signed-off-by: git-jxj <65210887+git-jxj@users.noreply.github.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
2026-10-07 08:35:14 +08:00
SrirajandFlora Feng d77cf81156 [Bugfix][Frontend] Honor default penalties in batch chat completions (#60307)
Signed-off-by: sricursion <sricursion@gmail.com>
Co-authored-by: Flora Feng <4florafeng@gmail.com>
2026-10-07 00:21:00 +00:00
Tyler Michael SmithCodexmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
5ab4e4445b [Perf][MoE] Skip padded work in block-FP8 DeepGEMM experts (#59128)
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-10-06 18:49:57 -05:00
Cheese CakeandClaude Opus 5.5 b6d8e8afd9 [Bugfix][MRV2][Spec Decode] Accept num_speculative_tokens in NgramGPUSpeculator.propose (#60300)
Signed-off-by: cheesecake <farzanaman99@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 15:02:04 -07:00
ae19af883a [CI] Require transformers 5.19 for the EmbeddingGemma2 registry test (#60295)
Signed-off-by: khluu <khluu000@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Andreas Karatzas <akaratza@amd.com>
2026-10-06 14:50:10 -07:00
aditi-amd 385d86a6cf [Attention] Add UltraQuant 4-bit KV cache backend (FlyDSL D=256) (#57057)
Signed-off-by: Aditi Rana <aditi.rana@amd.com>
2026-10-06 20:42:52 +00:00
819852df6b [ModelRunner V2] Speculative Decoding NGram GPU Implementations (#40704)
Signed-off-by: PatchouliTaisa <patchychen@tencent.com>
Signed-off-by: PatchouliTaisa <pyramkar@gmail.com>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Signed-off-by: Xuanan Chen <xuananchenc@nvidia.com>
Co-authored-by: PatchouliTaisa <patchychen@tencent.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Xuanan Chen <xuananchenc@nvidia.com>
Co-authored-by: Giancarlo Delfin <32987265+TheEpicDolphin@users.noreply.github.com>
2026-10-06 13:32:59 -07:00
Benjamin ChislettandClaude Opus 5.5 dac35c5b66 [Misc] Add chat-completion options to gsm8k_eval.py CLI (#60114)
Signed-off-by: Benjamin Chislett <bchislett@nvidia.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 20:13:44 +00:00
PatrykSafferandMisha Goin e43db1f5e2 Pad to 64 not 128 Q heads in flashMLA sparse (#60029)
Signed-off-by: Patryk Saffer <patryk.saffer@mistral.ai>
Co-authored-by: Misha Goin <mgoin64@gmail.com>
2026-10-06 16:03:05 -04:00
d1f3d8b870 [Quantization] Prefer Humming before Marlin backends on SM90 (#56997)
Signed-off-by: jinzhen.ljz <jinzhen.ljz@antgroup.com>
Signed-off-by: mgoin <mgoin64@gmail.com>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: mgoin <mgoin64@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 12:31:35 -07:00
Roger WangandKimi Code bb87d227d4 [Bugfix][Model] Fix mypy safe-super error in EmbeddingGemma2Model (#60280)
Signed-off-by: Roger Wang <hey@rogerw.io>
Co-authored-by: Kimi Code <noreply@moonshot.cn>
2026-10-06 15:10:00 -04:00
e37e51dd24 [Metrics] Expose cached prompt tokens by cache tier (#56318)
Signed-off-by: Cam Quilici <cjquilici@gmail.com>
Signed-off-by: Cam Quilici <cameron@semianalysis.com>
Co-authored-by: Cam Quilici <cameron@semianalysis.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai>
2026-10-06 14:52:05 -04:00
Nishant RevurandClaude ad0f67a7ec [Bugfix][Spec Decode] Keep heterogeneous-vocab draft models on Model Runner V1 (#59541)
Signed-off-by: Nishant Revur <kumrfni@amazon.com>
Co-authored-by: Claude <noreply@anthropic.com>
2026-10-06 18:03:44 +00:00
3403e0f176 [Bugfix] Persist FlashInfer autotune cache per rank to fix the rank-0-only cache-hit deadlock (#57635)
Signed-off-by: Madhav Jivrajani <madhav.jiv@gmail.com>
Signed-off-by: NickLucche <nicolo.lucchesi@mistral.ai>
Co-authored-by: NickLucche <nicolo.lucchesi@mistral.ai>
Co-authored-by: Wei Zhao <51183510+wzhao18@users.noreply.github.com>
2026-10-06 17:51:01 +00:00
Luciano MartinsandLuciano Martins 02b83919aa [Model] Support EmbeddingGemma2 multimodal pooling architecture (#60254)
Signed-off-by: Luciano Martins <lucianommartins@users.noreply.github.com>
Co-authored-by: Luciano Martins <lucianommartins@users.noreply.github.com>
2026-10-06 13:43:38 -04:00
7b665f7a58 [Frontend] Integer token IDs for generate output logprobs (GenerateLogProbs) (#58181)
Signed-off-by: Yin Li <kevinli.ai.work@gmail.com>
Co-authored-by: Yin Li <kevinli.ai.work@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
2026-10-06 17:22:47 +00:00
Matthew BonanniandClaude Opus 5.5 e82b80099a [Perf][HiSparse] Cache per-request residency instead of rescanning every page each step (#60083)
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 13:04:55 -04:00
Wentao Ye fba31f31a9 [GLM5.3 Perf] Switch to fp8 kv cache by default for GLM, 2.3%~5.5% E2E Throughput Improvement (#60140)
Signed-off-by: yewentao256 <zhyanwentao@126.com>
2026-10-06 12:54:20 -04:00
f8d92cae41 [Quantization] Layer re-quantization for linear layers through online quantization API (MXFP8 -> FP8 PTPC showcase) (#55684)
Signed-off-by: Felix Marty <Felix.Marty@amd.com>
Signed-off-by: Tan Pin Siang <tanpinsiang@gmail.com>
Co-authored-by: vllmellm <vllm.ellm@embeddedllm.com>
Co-authored-by: Tan Pin Siang <tanpinsiang@gmail.com>
Co-authored-by: Misha Goin <mgoin64@gmail.com>
2026-10-06 11:38:18 -05:00
eky-amdandClaude Fable 5 29f955cce0 [ROCm] Fuse MLA dual RMSNorm + FP8 group quant for DeepSeek-R1 (#54857)
Signed-off-by: Ethan Ky <ethan.ky@amd.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-10-06 11:26:38 -05:00
stefankoncarevic 4516896626 [CI/Build] Warm the DP engines before measuring load balance in test_load (#60241)
Signed-off-by: Stefan Koncarevic <Stefan.Koncarevic@amd.com>
2026-10-06 11:23:37 -05:00
Giancarlo Delfin fcf53d240c [Spec Decode] Remove eager metadata rebuild during MTP fused multi-step decode (#58463)
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
2026-10-06 12:17:30 -04:00
SeongJun Lee 4ea0c28bc4 [Bugfix] Warn only on per-request do_normalize/do_rescale overrides (#60240)
Signed-off-by: lesj0610 <lesj0610@gmail.com>
2026-10-06 15:55:27 +00:00
Jeff-Tseng-dev aad4579bbe Upgrade tpu-inference to v0.31.0 (#60190) 2026-10-06 15:44:56 +00:00
Chris LeonardandShengqi Chen cc73cca9c8 [Build] Migrate vendored DeepGEMM from pybind to TORCH_LIBRARY (abi3) (#48962)
Signed-off-by: Chris Leonard <chleonar@redhat.com>
Co-authored-by: Shengqi Chen <harry-chen@outlook.com>
2026-10-06 08:39:04 -07:00
ovidiusm cbe1f9740d [NIXL] Bump NIXL version to 1.5.0 (#56907)
Signed-off-by: Ovidiu Mara <ovidium@nvidia.com>
2026-10-06 16:55:03 +02:00
Bugen ZhaoandClaude Opus 5.5 0719975cee [CI][Rust Frontend] Pass --locked to cargo binstall (#60182)
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 22:53:17 +08:00
8bd7737a03 [Docs] Clarify that vLLM does not isolate tenants sharing a server (#59811)
Signed-off-by: SunnyR <sunny.rangnani@chainguard.dev>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Juan Pérez de Algaba <124347725+jperezdealgaba@users.noreply.github.com>
2026-10-06 22:39:59 +08:00
Samuel NordmannandMisha Goin 4a99ca40d6 [Perf][MoE] Eliminate staging from NCCL symmetric reduce-scatter (#49194)
Signed-off-by: samnordmann <snordmann@nvidia.com>
Co-authored-by: Misha Goin <mgoin64@gmail.com>
2026-10-06 07:37:01 -07:00
Jiangyun ZhuandClaude Opus 5.5 bf049ddb8f [Docs] Run pr-checklist before opening a PR (#60235)
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 14:12:18 +00:00
Matthias Gehre b267fe4881 [ROCm][Perf] W4A16: pad gfx11 weight and activation strides (#56301)
Signed-off-by: Matthias Gehre <matthias.gehre@amd.com>
2026-10-06 08:22:33 -05:00
Artem PerevedentsevandShashank Varma 049507aa76 [Bugfix][Structured Output] Flag multi-branch allOf as unsupported for xgrammar (#59061)
Signed-off-by: Artem Perevedentsev <aperevedents@nvidia.com>
Co-authored-by: Shashank Varma <324153016+shashankvarma499@users.noreply.github.com>
2026-10-06 17:01:04 +04:00
Roy Wang 1a86733c7b [Docs] Add model recipes skill (#57786) 2026-10-06 20:40:10 +08:00
Rodrigo Garcia 3faf214051 [vllm-bench, feature] Added support for Mooncake-style, timed-traces replay. (#55937) 2026-10-06 20:39:35 +08:00
Sherif Waly 25385102fe [Bugfix][KV Offload] Honor speculative cacheability in SimpleCPU (#60071)
Signed-off-by: Sherif Waly <sherif.waly@mistral.ai>
2026-10-06 14:25:08 +02:00
Wentao Ye 3e182185aa [Deprecation] Deprecate sonet dataset as scheduled (#60129)
Signed-off-by: yewentao256 <zhyanwentao@126.com>
2026-10-06 08:15:12 -04:00
Mia Ojeda 3313825983 [Bugfix][DiffusionGemma] Read the causal mask as int32 and compile the sample step (#59992)
Signed-off-by: ojeda-e <eojeda@league.com>
2026-10-06 19:25:51 +08:00
09e0ce3d82 [Bugfix][Frontend] Apply model-default reasoning parser in GPU-less render server (#54835)
Signed-off-by: Shimi Bandiel <shimib@google.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>
2026-10-06 18:46:24 +08:00
zhangchengzhucufe-dev 68088ed392 Fix grammar in BlockPool.reset_prefix_cache docstring: "to invalid" → "to invalidate" (#59886)
Signed-off-by: zhangchengzhucufe-dev <zhangchengzhucufe@gmail.com>
2026-10-06 10:19:54 +00:00
Louie Tsai 888074b42d [CPU][Recipes] Detect model head constraints for automatic tensor parallel selection in vLLM Recipes Tool (#60170)
Signed-off-by: louie-tsai <louie.tsai@intel.com>
2026-10-06 18:15:40 +08:00
Bugen ZhaoandClaude Opus 5.5 6bbad6acdf [Rust Frontend] Pass tool defer_loading through to chat templates (#60203)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
2026-10-06 18:00:15 +08:00
aoshen02andClaude Opus 5.5 31e2443c90 [Model] LongCat-Flash: scale MLA norms while loading and drop the post-load sweep (#60080)
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 08:47:02 +00:00
roikoren755andzjy0516 e11962fc1c [Feat][Mamba2] Enable internal prefill checkpoints (#57329)
Signed-off-by: Roi Koren <roik@nvidia.com>
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com>
Co-authored-by: zjy0516 <riverclouds.zhu@qq.com>
2026-10-06 07:53:51 +00:00
edfccc9469 [ROCM][Perf] Add ROCM_SEGMENTED_ATTN attention-backend works across RDNA3/3.5/4 (#59132)
Signed-off-by: root <jeffaw99@hotmail.com>
Signed-off-by: big-yellow-duck <jeffaw99@hotmail.com>
Co-authored-by: Tan Pin Siang <1716735+tanpinsiang@users.noreply.github.com>
Co-authored-by: vllmellm <190700713+vllmellm@users.noreply.github.com>
Co-authored-by: Codex <noreply@openai.com>
2026-10-06 07:53:47 +00:00
LostFox11LostFox11CodexYangShuai52mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
15c1cc12ac [Bugfix][EPLB] Use group-local ranks for torch P2P transfers (#55804)
Signed-off-by: LostFox11 <wangziyue17@huawei.com>
Co-authored-by: LostFox11 <wangziyue17@huawei.com>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: YangShuai52 <178856220+YangShuai52@users.noreply.github.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-10-06 07:44:57 +00:00
Tarun Kumar 6ad2e06c4a fix: exclude flashinfer-jit-cache from wheel install_requires (#60052)
Signed-off-by: Tarun Kumar <takumar@redhat.com>
2026-10-06 06:31:13 +00:00
sungbin1015andClaude 1e5d0ea888 [CI/Build] Fix incorrect CLI arg validation tests in test_cli_args.py (#46779)
Signed-off-by: sungbin1015 <sbin@solbox.com>
Co-authored-by: Claude <noreply@anthropic.com>
2026-10-06 05:58:47 +00:00
40c86ae1ce [Kernel] Enable OAI Triton MXFP4 MoE on SM12x (#58877)
Signed-off-by: lazymio <mio@lazym.io>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Shengqi Chen <harry-chen@outlook.com>
2026-10-06 01:49:13 -04:00
Bugen ZhaoandClaude Opus 5.5 b3082d3044 [Rust Frontend] Remove the PyO3 tool-parser bridge (#59744)
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 05:46:27 +00:00
Bugen ZhaoandClaude Opus 5.5 5145e35e50 [Frontend] Port the MiniMax M3 tool parser to the parser engine (#59743)
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 05:46:27 +00:00
czhu-cohereandaoshen02 6fddc9b21e [Profiler] Add per-session profiling controls to Python and Rust frontends (#57875)
Signed-off-by: czhu <conway.zhu@cohere.com>
Signed-off-by: Conway Zhu <conway.zhu@cohere.com>
Co-authored-by: aoshen02 <aoshen@inferact.ai>
2026-10-06 05:21:19 +00:00
aoshen02andClaude Opus 5.5 9a1f6fd4cb [Model][DeepSeek-V4.1] Permute mega-attention wq_b / wo_a while loading (#60064)
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 05:20:09 +00:00
Chinmay-Kulkarni-AMDandCursor Agent 533d81b4b0 [ZenCPU][Test] Add zentorch/torch version pin consistency test (#52888)
Signed-off-by: Chinmay Kulkarni <Chinmay.Kulkarni@amd.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
2026-10-06 13:12:19 +08:00
Juntian LiuClaude Opus 5.5mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
eb2901b735 [Perf][DSv4.1] Run decoder replay layers in CUDA graphs and trim in PIECEWISE graphs (#59532)
Signed-off-by: Juntian Liu <Juntianl777@gmail.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-10-05 22:11:45 -07:00
21d93d0d8c [Attention][Model] FP8 KV cache for Triton DiffKV and MiMo-V2.6-Flash (#58128)
Signed-off-by: Luca Motz <luca.motz@icloud.com>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com>
2026-10-05 21:18:29 -07:00
Tianyu Guo 30ab428235 [Bugfix][Attention] Restrict FlashInfer sparse MLA FULL graphs to decode (#59751)
Signed-off-by: Tianyu Guo <guoty@inferact.ai>
2026-10-06 04:13:23 +00:00
lzhan011andFlora Feng a3e0243b1c [Bugfix][Parser] Mistral pre-v11: don't fail requests on unexpected tool call JSON (#54844)
Signed-off-by: lzhan011 <35493221+lzhan011@users.noreply.github.com>
Signed-off-by: Flora Feng <4florafeng@gmail.com>
Co-authored-by: Flora Feng <4florafeng@gmail.com>
2026-10-06 03:37:30 +00:00
Tianyu GuoandOpenAI Codex 8c417a2dae [Bugfix][Model] Fix Qwen3-Omni interleaved M-RoPE boundary positions (#59842)
Signed-off-by: Tianyu Guo <guoty@inferact.ai>
Co-authored-by: OpenAI Codex <codex@openai.com>
2026-10-06 03:35:53 +00:00
Tianyu Guo 48eb44b719 [Bugfix][KVConnector][NIXL] Fix TP handshake test after per-region FA desc flags (#60157)
Signed-off-by: Tianyu Guo <guoty@inferact.ai>
2026-10-06 03:19:41 +00:00
6fa67afed9 [Bugfix][Core] Retire stale Mamba spec blocks in prefill checkpoint steps (#59759)
Signed-off-by: Simon Veitner <sveitner@redhat.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com>
2026-10-06 11:01:19 +08:00
2b32d9ac6a [Bugfix][Core] Drop stale block hashes when a session truncates tokens (#59103)
Signed-off-by: David Cheung <d7cheung@gmail.com>
Co-authored-by: Brian Woods <me@brian.biz>
Co-authored-by: Change72 <changg@nvidia.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-10-06 02:51:03 +00:00
b0b23a48d4 [Bugfix] Fix Qwen4Exp FP8 PLE pinned lookup kernel failing to compile below SM89 (#60142)
Signed-off-by: Vadim Gimpelson <vadim.gimpelson@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Thien Tran <gau.nernst@yahoo.com.sg>
2026-10-06 01:56:44 +00:00
ahmed xijiaatOpenAI CodexNick HillYannik Hintereggermergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
9465e1c783 [Bugfix][NIXL] Recover locally invalidated pull peer metadata (#55471)
Signed-off-by: xijiade.aihemaiti <3146335281@qq.com>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Yannik Hinteregger <37209495+YannikHinteregger@users.noreply.github.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-10-05 18:46:53 -07:00
lzhan011andFlora Feng 9bf4e51030 [Bugfix][Frontend] Malformed $defs in a tool schema should be a 400, not a 500 (#54850)
Signed-off-by: lzhan011 <35493221+lzhan011@users.noreply.github.com>
Signed-off-by: Flora Feng <4florafeng@gmail.com>
Co-authored-by: Flora Feng <4florafeng@gmail.com>
2026-10-06 01:37:20 +00:00
amd-xavierwangandTJian efd001756b [ROCm] Disable AITER attention query quantization on RDNA3 (#57750)
Signed-off-by: xawang <xawang@amd.com>
Co-authored-by: TJian <tunjian.tan@embeddedllm.com>
2026-10-06 01:37:18 +00:00
2537629eb2 [Bugfix][KVConnector][NIXL] Size per-region replicate flags by each region's block count (#60107)
Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Matthew Bonanni <mbonanni@redhat.com>
2026-10-06 01:27:50 +00:00
0e8f8ecba8 [Bugfix][KVConnector][NIXL] Return NumPy descriptors from mixed-memory local registration (#60108)
Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Matthew Bonanni <mbonanni@redhat.com>
2026-10-06 01:23:20 +00:00
4e198cef83 [TEST][XPU][CI] Fix ci uva kernel test instability (#60041)
Signed-off-by: Your Name <you@example.com>
Co-authored-by: Your Name <you@example.com>
Co-authored-by: Kunshang Ji <kunshang.ji@intel.com>
2026-10-06 09:19:01 +08:00
JohnQinAMDClaude Opus 5mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
97bd8c74eb [ROCm][DSv4.1][Perf] One-pass dequantize and a gather-sized grid for the compressed K cache (#56720)
Signed-off-by: JohnQinAMD <yanyuan.qin@amd.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-10-05 17:57:46 -07:00
b2863c0de9 [Kimi-K3] Use the fused K/V pack kernel on the decode-context-parallel prefill path (#56881)
Signed-off-by: Brandon Zhang <31413216+brandonfzhang@users.noreply.github.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Wei Zhao <51183510+wzhao18@users.noreply.github.com>
2026-10-06 00:13:55 +00:00
Mai Hoàng ĐứcandFlora Feng 5cd8a2626b [Bugfix][Structured Output] Cap JSON schema nesting to prevent 500s and API-server stalls (#60036)
Signed-off-by: Mazukiri <maihoangduc04@gmail.com>
Signed-off-by: sfeng33 <4florafeng@gmail.com>
Co-authored-by: Flora Feng <4florafeng@gmail.com>
2026-10-06 00:06:59 +00:00
Aarushi JainandAndreas Karatzas fa423b614c [CI][ROCm] Wait for engine teardown between LM Eval models (#60100)
Signed-off-by: aarushjain29 <aarushi.jain2@amd.com>
Co-authored-by: Andreas Karatzas <akaratza@amd.com>
2026-10-05 23:39:59 +00:00
MasonandYongye Zhu 6dd813e0b7 [Bugfix][MoE] Allow MiniMax2 routing in TRTLLM BF16 monolithic backend (#59571)
Signed-off-by: Mason Pitt <bradley.b.pitt@gmail.com>
Co-authored-by: Yongye Zhu <zyy1102000@gmail.com>
2026-10-05 16:24:14 -07:00
Vadim GimpelsonCursor Agentmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
0eac152707 [Bugfix] Route every speculation-capable row through the speculative path so recurrent state stays consistent (#56531)
Signed-off-by: Vadim Gimpelson <vadim.gimpelson@gmail.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-10-06 03:03:44 +04:00
Ting SUNandFlora Feng 4a705c7b54 [Bugfix][Frontend] Preserve logprobs when top_logprobs is null (#47838)
Signed-off-by: Ting Sun <suntcrick@gmail.com>
Signed-off-by: Flora Feng <4florafeng@gmail.com>
Co-authored-by: Flora Feng <4florafeng@gmail.com>
2026-10-05 22:43:31 +00:00
Kevin H. LuuandClaude Opus 5.5 89a00995c9 [CI] Read each profiler round's trace as it stops in test_gpu_profiler (#59769)
Signed-off-by: khluu <khluu000@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 15:25:11 -07:00
RomeJayandFlora Feng 2e06395fd3 [Bugfix][Frontend] Build tool-call grammars from the tools the prompt renders (#59879)
Signed-off-by: Rongjie Jiang <j1870831863@outlook.com>
Signed-off-by: sfeng33 <4florafeng@gmail.com>
Co-authored-by: Flora Feng <4florafeng@gmail.com>
2026-10-05 22:01:03 +00:00
Martin HickeyandFlora Feng 3fb1cd796b [Frontend] Extract parse and assembly out of batch chat derender (#60073)
Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com>
Co-authored-by: Flora Feng <4florafeng@gmail.com>
2026-10-05 21:57:45 +00:00
2e3154aa18 [HiSparse] Log steady-state max concurrency and expose host-tier utilization gauges (#58949)
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Matthew Bonanni <mbonanni@redhat.com>
2026-10-05 14:20:32 -07:00
Matthew BonanniandClaude Opus 5.5 8542c3eff5 [CI] Fix test_mixed_warmup_gate after #57053 (#60116)
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 21:17:00 +00:00
Yida WengandFlora Feng 18c8a65edd [BugFix][Frontend] Pass reasoning_ended through render → generate (#60059) (#60062)
Signed-off-by: Yida Weng <84164537+YidaWeng@users.noreply.github.com>
Signed-off-by: Flora Feng <4florafeng@gmail.com>
Co-authored-by: Flora Feng <4florafeng@gmail.com>
2026-10-05 20:39:25 +00:00
Wentao Ye 74c5cbcd7b [Bugfix][MRV2][Spec Decode] Honor dynamic K in autoregressive speculators, 1.2~1.3x kernel perf improvement (#57053)
Signed-off-by: yewentao256 <zhyanwentao@126.com>
2026-10-05 14:54:11 -04:00
Raphaël RiallandandSimon Veitner c32495d58f [watermark] compatibility validation (#56801)
Signed-off-by: Raphael Rialland <raphael.rialland@mistral.ai>
Signed-off-by: Simon Veitner <sveitner@redhat.com>
Co-authored-by: Simon Veitner <sveitner@redhat.com>
2026-10-05 14:37:13 -04:00
Giancarlo Delfin 88f3c601b3 [Docs] Add reviewer area of interest for TheEpicDolphin (#60092)
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
2026-10-05 18:35:19 +00:00
aoshen02andClaude Opus 5.5 d09ff77637 [Bugfix] Give each data-parallel engine its own global RNG streams (#59788)
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 17:38:18 +00:00
877ddcc456 [Bugfix][Spec Decode] Reserve the bonus KV slot for fill-in DSpark (#59105)
Signed-off-by: Evgeny Savinov <notime.sea@gmail.com>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Tomas Ruiz <tomas.ruiz.te@gmail.com>
2026-10-05 17:35:51 +00:00
akii96 182247257f [ROCm][Perf] Use a stacked hipBLASLt GEMM for the bf16x3 router (#52668)
Signed-off-by: Aakif Nawaz <aakif.nawaz@amd.com>
2026-10-05 17:28:07 +00:00
Matthew BonanniandClaude Opus 5.5 54d93af9fb [Bugfix][HiSparse] Reject cudagraph_mode=FULL at startup (#59688)
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 12:42:27 -04:00
Sohaib Ahmed 64cb683842 [Model] Add Nemotron 3.5 ASR transcription support (#59827)
Signed-off-by: Sohaib-Ahmed21 <sohaibahmed1919@gmail.com>
2026-10-05 16:28:09 +00:00
stefankoncarevicandAndreas Karatzas 87954c5b04 [CI/Build] Declare the video modality on the CohereCompass reference model (#60063)
Signed-off-by: Stefan Koncarevic <Stefan.Koncarevic@amd.com>
Co-authored-by: Andreas Karatzas <akaratza@amd.com>
2026-10-05 16:25:42 +00:00
Mikko TukiainenandCursor d23af12cb1 [ROCm][Perf] Reach the fused QSA pre-indexer from the AMD path (#57947)
Signed-off-by: mjkvaak-amd <mjkvaak-amd@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-10-05 16:22:29 +00:00
bzsuniandcjackal 528772a4bf [Bugfix][Core] Retain encoder cache references for repeated multimodal inputs (#59942)
Signed-off-by: bzsuni <bingzhe.sun@daocloud.io>
Signed-off-by: cjackal <44624812+cjackal@users.noreply.github.com>
Co-authored-by: cjackal <44624812+cjackal@users.noreply.github.com>
2026-10-05 15:52:46 +00:00
charlesli640 edde9d2ce4 Remove redundant dependency requirement for TPU (#59977)
Signed-off-by: Charles Li <licharles@google.com>
2026-10-05 08:45:37 -07:00
stefankoncarevic f2d8fbf459 [CI/Build] Run the prefill token scoring test under batch invariance (#60056)
Signed-off-by: Stefan Koncarevic <Stefan.Koncarevic@amd.com>
2026-10-05 10:32:33 -05:00
Matthew BonanniandClaude Opus 5.5 9e931a0c6c [CI] Add CODEOWNERS for HiSparse (#60060)
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 15:18:29 +00:00
60932a6401 [Bugfix][Structured Output] Preserve literal values in Guidance disable_additional_properties (#58709)
Signed-off-by: Abhay Joshi <105213625+abhayjoshi201@users.noreply.github.com>
Co-authored-by: Antigravity <noreply@google.com>
Co-authored-by: Artem Perevedentsev <aperevedents@nvidia.com>
2026-10-05 18:12:40 +03:00
Nicolò Lucchesi 30d4032363 [Model][GLM-5.3-Flash] Support DCP for the kpool sparse indexer (#59211)
Signed-off-by: NickLucche <nicolo.lucchesi@mistral.ai>
2026-10-05 16:44:14 +02:00
Jee Jee Li f9c9e8ac24 [LoRA] Remove tensorizer (#60024)
Signed-off-by: Jee Jee Li <jeejeelee@inferact.ai>
2026-10-05 07:01:56 -07:00
Abhay ShuklaandClaude Opus 5.5 d3547f9d03 [Test] Re-enable Voxtral HF reference test on Transformers v5 (#59771)
Signed-off-by: AbhayShuklaIIT <abhay.shukla@live.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 13:43:49 +00:00
cjackal 0e468adb43 [Model] Extend device-side mm normalization to Kimi K2.5 / K3 (#59278)
Signed-off-by: cjackal <44624812+cjackal@users.noreply.github.com>
2026-10-05 21:28:13 +08:00
Nicolò LucchesiandLeoyzen ff53f32409 [Attention][MLA] Support fp8_ds_mla KV cache for NoPE-512 models on SM90 (#59246)
Signed-off-by: NickLucche <nicolo.lucchesi@mistral.ai>
Co-authored-by: Leoyzen <leoyzen@gmail.com>
2026-10-05 13:12:00 +00:00
Vadim GimpelsonandClaude d0d6e5f3a2 [Bugfix] Fix Mamba page size AssertionError with spec decoding on GraniteMoeHybrid, FalconH1 and Zamba2 (#59975)
Signed-off-by: Vadim Gimpelson <vadim.gimpelson@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
2026-10-05 12:49:47 +00:00
roikoren755andmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> 51eeb0c58f [Bugfix] Load stacked expert weights for non-gated MoE (#59031)
Signed-off-by: Roi Koren <roik@nvidia.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-10-05 15:26:55 +03:00
Stefano CastagnettaandThien Tran 4ff028d77e [Bugfix][Qwen4Exp] Honor --kv-cache-dtype-skip-layers in QSA attention (#60023)
Signed-off-by: Stefano Castagnetta <scastagnetta@nvidia.com>
Co-authored-by: Thien Tran <gau.nernst@yahoo.com.sg>
2026-10-05 12:13:30 +00:00
Stefano CastagnettaandThien Tran 55b80221ed [Perf][Qwen4Exp] Keep the HC up projection on the skinny GEMM path (#60027)
Signed-off-by: Stefano Castagnetta <scastagnetta@nvidia.com>
Co-authored-by: Thien Tran <gau.nernst@yahoo.com.sg>
2026-10-05 20:09:36 +08:00
Juan Pérez de Algaba 710ac56e69 [Security] Restrict Pillow image formats on untrusted media paths (#60022)
Signed-off-by: Juan Pérez de Algaba <jperezde@redhat.com>
2026-10-05 10:40:15 +00:00
Mikko TukiainenCursormergify[bot] <37929162+mergify[bot]@users.noreply.github.com>TJian
edca360f13 [Bugfix][Qwen4Exp] Load PLE tables unquantized under Quark checkpoints (#59443)
Signed-off-by: Mikko Tukiainen <Mikko.Tukiainen@amd.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: TJian <tunjian.tan@embeddedllm.com>
2026-10-05 10:40:02 +00:00
c04c79e50d [Performance] Add SM121 TP=2 skinny-GEMM plans (#59632)
Signed-off-by: sudhanshu shukla <sudhanshu112233shukla@users.noreply.github.com>
Co-authored-by: sudhanshu shukla <sudhanshu112233shukla@users.noreply.github.com>
Co-authored-by: Thien Tran <gau.nernst@yahoo.com.sg>
2026-10-05 18:27:51 +08:00
Maroon Ayoub b1401e0aa7 [Bugfix][NIXL] Count heartbeats as remote engine activity (#59873)
Signed-off-by: Maroon Ayoub <mayoub@redhat.com>
2026-10-05 09:36:13 +00:00
bcsdhjewmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>Codex
ae53b06898 [Bugfix][GLM-5.3-Flash] Keep batch x heads out of gridDim.z in the GLM-5.3-Flash fused recurrent KDA kernel (#56974)
Signed-off-by: NolenLiang <nliang@nvidia.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Codex <noreply@openai.com>
2026-10-05 09:31:45 +00:00
Ian Eavesandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> b1f229fb75 [Model] Declare SupportsEagle3 on Qwen3ASRForConditionalGeneration (#52824)
Signed-off-by: Ian Eaves <ian.k.eaves@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-10-05 09:23:47 +00:00
Jee Jee Li e3af5bf3ce [LoRA] Code cleanup (#60017)
Signed-off-by: Jee Jee Li <jeejeelee@inferact.ai>
2026-10-05 09:15:53 +00:00
aoshen02andClaude Opus 5.5 4a30c4cad0 [Model][DeepSeek-V4] Make MegaMoE shared-expert finalize independent of linear post-load order (#59927)
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 08:39:49 +00:00
6367 changed files with 56836 additions and 1750072 deletions
@@ -0,0 +1,115 @@
---
name: model-microbenchmark
description: Time a vLLM model's forward pass on exact batch shapes (decode, prefill, chunked prefill, prefix hit, spec-decode verify, or mixes) with `vllm bench forward`, single-GPU or DP/TP/EP across nodes. Use for per-step model latency, layout/parallelism comparisons, and checking a model-level optimization. Not for single kernels (use kernel-microbenchmark) or end-to-end serving (use `vllm bench serve`).
---
# Model Microbenchmark (`vllm bench forward`)
Builds the batch without the scheduler, runs it through MRV2's real
`execute_model`, and times the forward as a one-shot CUDA graph (prefill
included). Sampling, the speculator, and CPU step overhead are excluded, so
expect results somewhat below a serving step.
## Usage
```bash
vllm bench forward --model deepseek-ai/DeepSeek-V4-Flash \
--case decode=64*1@8192 --case prefill=8192 --case mix=4096,32*1@8192
# DP/TP/EP: one process per GPU (multi-node: add --nnodes/--node-rank/--master-addr)
torchrun --nproc-per-node=4 -m vllm.entrypoints.cli.main bench forward \
--model ... -dp 4 --enable-expert-parallel --case decode=128*1@100000
```
Requests: `[N*]Q[@K][c][:D]`, comma-separated, per DP rank.
| Form | Meaning |
| --- | --- |
| `8192` | fresh prefill, 8192 tokens |
| `32*1@8192` | 32 decodes, KV length 8192 (KV length **includes** the query) |
| `4@4096:3` | spec-decode verify, 3 draft tokens |
| `2048@10240` | prefill after an 8192-token prefix-cache hit |
| `2048@10240c` | chunked prefill: the prefix came from earlier chunks |
Verify cases require `--speculative-config` with `num_speculative_tokens`
at least as large as the case's draft count; insufficient capacity is rejected.
Hit vs chunk only differs on models with SWA bounded replay (DSV4.1): a hit
also recomputes the last window (128 tokens) of the prefix.
Or `--shapes file.json|yaml`:
`{"cases": [{"name": ..., "requests": [{"query_len", "kv_len", "num_draft_tokens", "chunked", "count"}]}]}`.
Key flags (plus all `EngineArgs`):
- `--kv-len-jitter 0.5` (default): prefixes drawn within ±50%, same on every
DP rank, capped at `max_model_len`. Use `0` for exact lengths.
- `--routing auto|real|balanced`: `auto` = balanced with dummy weights.
- `--iters 20 --samples 5`, `--output-json out.json` (per-rank latency stats,
`samples_us`, graph-vs-eager check).
- Defaults: `--load-format dummy`, prefix caching off, no torch.compile.
Read `median_us` (each replay is already the max over ranks).
## Kernel breakdown
Add `--profile-dir DIR` to see where the time goes. After timing, each case is
replayed 3 more times under `torch.profiler`; every rank writes
`DIR/<case>.rank<r>.kernels.json` (GPU time per kernel per forward, sorted)
and `DIR/<case>.rank<r>.trace.json.gz` (open in Perfetto). Rank 0's top
kernels are printed and included in `--output-json`.
```bash
python -m vllm.benchmarks.forward.profile DIR/decode.rank0.kernels.json
# Before/after a change, same case: per-kernel deltas, largest first.
python -m vllm.benchmarks.forward.profile base/decode.rank0.kernels.json new/decode.rank0.kernels.json
```
- Kernels are grouped by name without template or function arguments; the
JSON keeps one full name per group in `example`.
- Only the timed forward is included: the in-graph KV restore and rank
rendezvous are excluded (`pre_forward_us`). `span_us` (first to last
forward kernel) should be close to `median_us`, a bit higher from profiler
overhead; `busy_us` can exceed it when streams overlap.
- Compare the same rank across runs: under EP, ranks differ (e.g. collective
wait time shows up inside communication kernels).
For Nsight Systems (NVTX, NCCL, CPU side, GPU metrics), skip `--profile-dir`:
the two can't share CUPTI. Each case's 3 profiled replays run between
`cudaProfilerStart`/`Stop` in an NVTX range `forward/<case>`, so a capture range
records only them, one report per case (and rank):
```bash
nsys profile --capture-range=cudaProfilerApi --capture-range-end=repeat \
--cuda-graph-trace=node -o prof vllm bench forward ... # prof.1, prof.2, ...
torchrun --nproc-per-node=4 --no-python nsys profile ... -o prof_rank%q{RANK} \
python -m vllm.entrypoints.cli.main bench forward ...
nsys stats --report cuda_gpu_kern_sum --format csv -o kern prof.1.nsys-rep
```
Keep `--capture-range-end=repeat`: nsys's default stops after the first case
and leaves the bench hanging. `nsys stats` also counts the KV restore copies
that the JSON summary excludes.
## Caveats
- **Dummy weights lie for data-dependent kernels.** Dummy gates route every
token to the same experts (hence balanced routing). Cross-check suspicious
kernels with real weights.
- **torch.compile models (e.g. Qwen3/3.5) look slow**: no compile here. Use
them for correctness only; trust perf for models under `vllm/models/`.
- `sample spread > 0.5%` warns of unstable timing. DSV4.1 DEP prefill is
inherently ~0.5–1% (MegaMoE cross-rank sync, host engram reads).
- `graph_vs_eager` may be nonzero for small batches where models branch on
`is_current_stream_capturing()`; that's expected, not a bug.
- KV cache is zero-initialized (`--kv-init zeros`); random KV was within 3%.
- Uniform DP only: every rank gets the same case; no idle ranks.
- CUDA only: it hangs on ROCm (untested cause: likely the in-graph timing
events).
- Large prefills/prefix hits need room: raise `--max-num-batched-tokens` /
`--max-model-len` / `--gpu-memory-utilization` if build fails on KV blocks.
- Startup cost is JIT/autotune, not the bench: install FlashInfer jit-cache
and cubin; DeepGEMM warmup is skipped (each case's eager pass compiles its
own shapes).
- To compare against serving decode: `vllm bench latency` with 65- vs
1-token outputs, divided by 64, with a FULL decode graph.
+157
View File
@@ -0,0 +1,157 @@
---
name: model-recipes
description: Find vLLM deployment recipes and configs for LLMs and Hugging Face models by model and hardware. Use for recipe lookup, creation, and updates.
---
# Model Recipes
Model serving recipes are maintained in this repository's
[recipes/models/ directory](../../../recipes/models). Omni (image, video,
audio) recipes stay in vllm-project/recipes for now (moving to vllm-omni),
and Ascend NPU content is inline in these recipes (moving to vllm-ascend
overlays). The published catalog, rendered by
[vllm-project/recipes](https://github.com/vllm-project/recipes), is at
[recipes.vllm.ai](https://recipes.vllm.ai).
## Usage notes
1. Recipe lookup returns commands and deployment guidance without launching
`vllm serve`.
2. Use published configurations or the recipes repository's command generator
for supported selections. Return commands as published, including their
flags and parallelism settings.
3. Recipes are starting points for deployment and may not be tuned for your
workload. Check each recipe for its verified configurations, and use
benchmarks of your workload to guide further tuning.
4. To create or update a recipe, follow the
[authoring workflow below](#create-or-update-a-recipe).
## Find a recipe
Read recipes from the published JSON on recipes.vllm.ai. It is generated from
the source YAML with variant, hardware, and strategy settings already
resolved; the YAML contains comments and conditional overrides, so do not use
it for lookup. When presenting results, link the website page. Read the source
YAML only when creating or updating a recipe.
Each recipe has a page at `https://recipes.vllm.ai/<org>/<repo>` and JSON at
the same path plus `.json`. Paths use the full Hugging Face ID, preserving
case; do not substitute `meta.slug` or a display title. For example,
`Qwen/Qwen3.8-27B` maps to
[recipes.vllm.ai/Qwen/Qwen3.8-27B](https://recipes.vllm.ai/Qwen/Qwen3.8-27B).
Recipe JSON is often over 20 KB, mostly the `guide`. Instead of printing it,
read only what the request needs: each step below returns the paths for the
next, so stop once the request is answered. Commands need `curl` and `jq`.
[scripts/read_recipe.sh](scripts/read_recipe.sh) is in this skill's directory;
the examples use its path from the vLLM repository root.
### Search the catalog
Skip this step if the exact HF ID is known. Otherwise search by model ID,
family, or provider using case-insensitive keywords. Each
whitespace-separated word must occur in the ID, title, or provider:
```bash
curl -fsSL https://recipes.vllm.ai/models.json |
jq --arg q 'qwen 27b' '
($q | ascii_downcase | [scan("\\S+")]) as $words | .[] |
([.hf_id, .title, .provider] | join(" ") | ascii_downcase) as $text |
select(all($words[]; . as $word | $text | contains($word))) |
{hf_id, json, url, derived_from}'
```
`json` is the recipe data path; `url` is its page path. Prefix both with
`https://recipes.vllm.ai`. A `derived_from` entry is a variant of a parent
recipe: fetch its own `json` to get that checkpoint's command. Its page may
belong to the parent, so link the returned `url` rather than assuming every
checkpoint has a separate page.
If there is no exact match, search the family and inspect `variants` before
concluding the recipe is missing.
### Summarize the recipe
Save the recipe JSON to a file and summarize it:
```bash
read_recipe=.agents/skills/model-recipes/scripts/read_recipe.sh
recipe=$(mktemp)
curl -fsSL https://recipes.vllm.ai/Qwen/Qwen3.8-27B.json -o "$recipe"
"$read_recipe" "$recipe"
```
`default` is the recommended hardware, strategy, and variant; `hardware` maps
each hardware ID to its command path; `recipe_verified_hardware` lists hardware
where the recipe as a whole was verified end to end. For capability
requirements, such as tool calling or text-to-video, check `features` and
`omni_tasks`, then confirm in the guide.
Some recipes, including Omni, lack `hardware`; skip hardware selection and
use the guide's launch and task-specific sections.
### Select hardware and strategy
Match a hardware ID to the user's GPU and fetch its path from `hardware`,
prefixed with `https://recipes.vllm.ai`, rather than constructing a URL. That
response is the command object itself: single-node objects carry `command`,
and multi-node objects carry `head_command` and `worker_commands` instead, so
list its keys before selecting fields.
Check `hardware_profile` to confirm that the selected configuration matches
the user's GPU model and specifications. Its `gpu_count` describes the profile,
not necessarily the number of GPUs used by the command. Check the command's
parallelism settings, `node_count`, and all `worker_commands` against the
user's available topology, and compare the variant's `vram_minimum_gb`, an
estimate, with the total memory of the GPUs the command uses; a configuration
below that minimum is not a working option unless the guide says it was
tested. Ask when the user's hardware is unclear. If no
published command or guide command fits, report the mismatch and return the
closest one unchanged; list any adjustments the user's setup would need
separately and mark them unverified instead of editing the command.
For another strategy, follow `alternatives` in the same response. These are
static files; query parameters do not change their configuration. For other
feature/topology selections, use the recipes repository's command synthesis.
### Read deployment guidance
Read the guide sections that apply to the selection, such as prerequisites,
launch notes for the selected hardware or variant, troubleshooting, and known
limitations. Pass entries from `guide_sections` or prefixes of their text.
The script prints every matching section with its subsections and fails when
a section matches nothing:
```bash
"$read_recipe" "$recipe" prerequisites troubleshooting
```
Query other fields with `jq` on the saved file, such as `features.<name>`,
`hardware_overrides`, or `omni.tasks` for request examples.
### Return the result
Return the website link, selected checkpoint and hardware, the requested
command, and relevant prerequisites and caveats from the guide. Pick one
command and copy it, including its Docker form, exactly as the command object
or guide section gives it; do not combine commands from different sections or
add, drop, or reorder flags. List optional changes separately.
For each command, state its source, either a generated hardware or strategy
configuration or a named guide section, and whether that source says it was
tested. A `recipe_verified_hardware` entry covers the recipe on that
hardware, not every variant, strategy, or guide command, so it does not make a
command verified. Generated configurations are not automatically verified.
## Create or update a recipe
Recipes cover official or widely used checkpoints, with configurations verified
end to end on real hardware. Do not add recipes for arbitrary community
uploads, untested hardware claims, or non-standard flags that only work for one
setup. Before opening a pull request, confirm that the user ran the
configuration and can share the results.
Edit the YAML under [recipes/models/](../../../recipes/models) in this
repository, following the recipes repository's
[add-recipe skill](https://github.com/vllm-project/recipes/blob/main/.agents/skills/add-recipe/SKILL.md)
for the schema. Validate and preview it from a checkout of
`vllm-project/recipes` pointed at this directory, as
[recipes/README.md](../../../recipes/README.md) describes.
@@ -0,0 +1,4 @@
interface:
display_name: "Model Recipes"
short_description: "Find vLLM deployment recipes and configs for LLMs and Hugging Face models by model and hardware."
default_prompt: "Use $model-recipes to find a recipe for my model and hardware, and return its commands and deployment caveats."
+66
View File
@@ -0,0 +1,66 @@
#!/bin/bash
# Usage: read_recipe.sh <recipe.json> [guide-section...]
# Without sections, print a compact summary of a recipes.vllm.ai recipe JSON.
# With sections, print each guide section whose heading starts with one of
# them (case-insensitive), including its subsections.
set -euo pipefail
if [ $# -lt 1 ]; then
echo "Usage: $0 <recipe.json> [guide-section...]" >&2
exit 1
fi
# Print "<level>\t<heading text>\t<line>" for guide lines, marking headings
# outside code fences with their level and other lines with 0.
guide_lines() {
jq -r '.guide // ""' "$1" | awk '
/^[[:space:]]*(```|~~~)/ { fence = !fence }
!fence && match($0, /^#+ /) {
print RLENGTH - 1 "\t" substr($0, RLENGTH + 1) "\t" $0
next
}
{ print "0\t\t" $0 }'
}
if [ $# -gt 1 ]; then
recipe=$1
shift
status=0
for section in "$@"; do
guide_lines "$recipe" | awk -F '\t' -v h="$section" '
BEGIN { h = tolower(h); sub(/^#+[[:space:]]*/, "", h) }
$1 > 0 && lvl && $1 <= lvl { lvl = 0 }
$1 > 1 && !lvl && index(tolower($2), h) == 1 { lvl = $1; found = 1 }
lvl { sub(/^[^\t]*\t[^\t]*\t/, ""); print }
END {
if (!found) {
print "No guide section starts with: " h > "/dev/stderr"
exit 1
}
}' || status=1
done
exit $status
fi
sections=$(guide_lines "$1" | awk -F '\t' '$1 == 2 || $1 == 3 { print $3 }' |
jq -R . | jq -s .)
jq --argjson sections "$sections" '{
hf_id,
title: .meta.title,
tasks: .meta.tasks,
recipe_verified_hardware: .meta.hardware,
omni_tasks: (.omni.tasks // null |
if . then map(objects |= {id, label, endpoint, model_id, description})
else null end),
variants: (.variants // {} | map_values(
{model_id, precision, vram_minimum_gb, supported_hardware} |
with_entries(select(.value != null)))),
features: (.features // {} | keys),
opt_in_features,
default: (.recommended_command // null |
if . then {hardware, strategy, variant} else null end),
hardware: .recommended_command.by_hardware,
guide_sections: $sections
} | with_entries(select(.value != null))' "$1"
+3 -1
View File
@@ -1,6 +1,6 @@
---
name: pr-checklist
description: Prepare a vLLM change for human PR review or re-review, or analyze an open pull request. Use to check design fit, behavioral coverage, performance evidence, diff quality, and closure of previous feedback before requesting maintainer attention.
description: Prepare a vLLM change for human PR review or re-review, or analyze an open pull request. Use before opening a PR, drafts included (e.g. `gh pr create`), or pushing review fixes, to check design fit, behavioral coverage, performance evidence, diff quality, and closure of previous feedback.
---
# PR Checklist
@@ -181,6 +181,8 @@ Summarize model evaluations and benchmarks in compact tables with the configurat
When fixing bugs or issues, provide a clear root-cause analysis. Explain what caused the problem, how it was identified, and why the chosen solution effectively addresses it. A patch which resolves the issue without a proper root-cause analysis may lead to recurring problems and increased maintenance overhead.
For fixes that involve disaggregated prefill/decode (PD) logic, such as the NIXL connector (`vllm/distributed/kv_transfer/kv_connector/v1/nixl/`), make a best effort to include a minimal example of the flow being changed and/or the initial flow that caused the failure. A short numbered sequence (or diagram) of a request's path across the P and D instances, e.g. scheduler admission, prefill execution, KV transfer registration and write, decode-side read, token generation, lets reviewers trace the root cause and the fix without reconstructing the distributed flow from the diff. Keep the example minimal: only the steps and components involved in the change. See the [disaggregated prefill](../../../docs/features/disagg_prefill.md), [NIXL KV push](../../../docs/design/nixl_kv_push_connector.md), and [NIXL KV cache lease](../../../docs/design/nixl_kv_cache_lease.md) docs for the flows involved.
#### 4.4: Implementation Details
Provide a detailed explanation of the implementation, including design decisions, trade-offs, and any relevant technical details. Highlight any non-obvious choices, tradeoffs, potential limitations, and areas where maintainer attention is required.
@@ -0,0 +1,190 @@
---
name: transformers-upgrade
description: "Playbook for changing vLLM's Transformers version bounds. Use when bumping the pinned or maximum Transformers version, raising the minimum Transformers version, or deleting vendored configs, processors, or compat shims that upstream Transformers now provides."
---
# Upgrading Transformers
The supported range lives in `requirements/common.txt`. CI pins one exact
version in `requirements/test/*.in` and the compiled `requirements/test/*.txt`.
A version change is one of two jobs, and they usually come as a pair:
- **Bump the pin**: a new Transformers release becomes the CI version and
the upper bound. vLLM grows compat code for it.
- **Raise the floor**: the minimum moves up. That compat code, and every
vendored config or processor upstream now ships, becomes dead weight.
Every floor raise ends with an **unvendoring pass**.
## Bump the pin
1. Edit the pin line in each `requirements/test/*.in` and `*.txt` and the
upper bound in `requirements/common.txt`. Past bumps edit only the pin
line in the compiled files; see `git log -- requirements/common.txt`.
2. Read the release notes for config validation changes, renamed auto-mapping
formats, and removed attributes; these are what break vLLM.
3. Search `tests/` for `min_transformers_version` and skip markers naming the
new version, and un-gate them.
4. Run the suites in [Testing](#testing) and fix what breaks. Mark every
shim with a `# Transformers >= X` comment so the next floor raise finds it.
Done when the [Testing](#testing) suites pass on the new pin and every test
waiting on this version is un-gated.
## Raise the floor (unvendoring pass)
### 1. Inventory
Inventory candidates with the floor version installed (no vLLM install needed):
```bash
uv run --no-project --with transformers==<floor> \
python .agents/skills/transformers-upgrade/scripts/find_upstreamed.py
```
It lists:
- vendored configs whose `model_type` upstream now owns;
- `_CONFIG_REGISTRY` overrides of upstream types;
- registered processors that collide with upstream class names or models;
- version-gated code at or below the floor;
- code that reads the installed Transformers version, whatever its threshold;
- patches citing a Transformers issue or PR (with whether a referenced PR is
in the floor release) or a future minimum version.
It finds candidates by name only. Every hit still needs a verdict.
### 2. Port each candidate to upstream
Every hit is deleted by default. A vendored copy is a liability: it drifts
from upstream, hides upstream fixes, and shadows the native class. Port the
vLLM model code to the upstream schema instead, renaming attribute reads and
deriving values from upstream fields.
Fan candidates out to parallel subagents. Each one compares the vendored and
upstream classes, then greps every attribute the vLLM model code reads.
[Upstreaming hazards](#upstreaming-hazards) lists the usual fixes.
Keep a vendored class only when upstream cannot load a checkpoint vLLM must
keep serving, and record why in the PR description. A different upstream
schema, a model-code rewrite, or a needed eval is work for the PR, not a
reason to keep.
### 3. Prove it on real checkpoints
For every checkpoint in `tests/models/registry.py` that uses a ported
config, dump what vLLM builds before and after the change, then diff the two.
Run both dumps; each catches what the other misses:
- `dump_configs.py`: the resolved config, attribute by attribute.
- `dump_model.py`: the model built on the meta device with no GPU or weights.
It records ModelConfig sizes, every parameter shape, and every module's
scalar attributes (`head_dim`, `num_kv_heads`, `scaling`, ...). Missing
fields that are read with defaults only show up here.
```bash
git worktree add --detach <scratch>/vllm-base upstream/main
S=.agents/skills/transformers-upgrade/scripts
for dump in dump_configs dump_model; do
PYTHONPATH=<scratch>/vllm-base .venv/bin/python $S/$dump.py before.jsonl <repos>
PYTHONPATH=$PWD .venv/bin/python $S/$dump.py after.jsonl <repos>
.venv/bin/python $S/diff_dumps.py before.jsonl after.jsonl
done
```
Add `--trust-remote-code` for checkpoints whose `config.json` has an
`auto_map`, and dump both ways.
How to read the diffs:
- **A `<MISSING>` config field the model reads:** this crashes or silently
falls back to a default. Port the model code to the upstream field.
- **New upstream-only defaults** (`attention_bias=False`,
`pad_token_id=None`): usually benign. A model diff of 0 confirms it.
- **Any model diff:** this is a numerics change, and the PR needs a model
eval. For example, A.X-K1 gained the checkpoint's YaRN rope scaling, which
the vendored class had been dropping; it shows up as
`layers.*.self_attn.scaling (x61)`.
### 4. Delete
For each deleted config:
- Delete the file.
- Remove its entries from `_CLASS_TO_MODULE` and `__all__` in
`vllm/transformers_utils/configs/__init__.py`.
- Remove its entries from `_CONFIG_REGISTRY` in
`vllm/transformers_utils/config.py`.
- Switch every importer, including tests, to `from transformers import ...`.
- Remove the `source_file_dependencies` lines naming the file from
`.buildkite/`.
Deleted processors follow the same steps in
`vllm/transformers_utils/processors/__init__.py`.
Before merging, rebase and grep again for the old field names: PRs that land
in between still read them.
Done when every line of the inventory output is deleted, or kept with a
reason that meets the bar in step 2. The PR also carries the config diff
results and evals for any changed values.
## Upstreaming hazards
Upstream configs are `@strict` dataclasses. Checkpoint keys upstream does not
declare survive as plain attributes, but only when the checkpoint sets them.
Defaults that only vLLM sets, and values vLLM derives, vanish.
| Hazard | Signal | Fix |
| --- | --- | --- |
| Upstream consumes legacy keys | `__post_init__` pops keys the model reads (`deepseek_v4` `compress_ratios`) | Read the upstream replacement (`layer_types`, `mlp_layer_types`, per-layer `rope_parameters`) |
| Renamed or dropped fields | Diff shows `<MISSING>` for a field the model reads | Read the upstream name; derive dropped flags from what implies them (`enable_moe_block` from `num_experts`) |
| Undeclared field without a default | `config.x` raises `AttributeError` for configs that omit `x`, e.g. minimal test configs (A.X-K1 `scoring_func`). Dumps of real checkpoints miss it | Upstream ignoring `x` usually means vLLM can stop reading it; if vLLM truly needs it, read `getattr(config, "x", <old default>)` |
| Top-level config stops forwarding | The vendored top-level config forwarded reads to `text_config`; upstream does not. `getattr(config, "x", default)` silently takes the default, e.g. on MTP draft configs from `SpeculativeConfig.hf_config_override`, which the dumps do not build | Read from `get_text_config()`; grep `getattr` reads of every field, including speculative-decoding paths |
| Type check on the config | `ctx.get_hf_config(SomeConfig)` relies on the vendored base class | Accept the upstream class too, e.g. a `Cosmos3ProcessingInfo` overriding `get_hf_config` |
| Remote code takes over | Checkpoint `config.json` has `auto_map`; with `--trust-remote-code`, the hub class replaces the native one once vLLM stops registering its own | Check the remote class loads; if not, register the upstream class for that `model_type` |
| Read-only upstream fields | Properties or derived fields break subclasses and `hf_overrides` (`nemotron_h` `hybrid_override_pattern`) | Override the source field (`layers_block_type`) instead |
| Behavioural rewrite | Upstream derives fields differently (`diffusion_gemma` `sliding_window`) | Upstream matches the reference implementation; verify with an eval |
| Original checkpoint format | Upstream targets `-hf` conversions; vLLM serves the originals (`qwen3_asr`, `qianfan_ocr`) | Keep only if upstream cannot load the original at all |
| Processor output contract | Model code consumes keys upstream does not emit | Port the model to upstream's keys |
| Name collision | A registered vendored processor shares an upstream class name and hijacks `get_processor` (`InternVLProcessor`) | Drop the registry entry |
Version gates from the inventory need a decision:
- **Code** only reachable below the floor is deleted. This covers:
- comparisons against `TRANSFORMERS_VERSION`;
- branches labelled `# Transformers < X`;
- fallbacks for fields that upstream configs now always set, e.g.
`mlp_layer_types`, `per_layer_config` or legacy `layer_types` names.
A `skipif` whose condition is now always true becomes an unconditional
`skip` with the same reason.
- **`min_transformers_version`** at or below the floor is removed.
- **`max_transformers_version`** below the floor means CI never tests that
model; report it rather than deleting it silently.
- **Patches citing a Transformers issue or PR:**
- Delete the patch when its fix is in the floor release.
- For an issue, find the PR that closed it and check that PR. A closed
issue is not a fixed one: issues get closed while the bug is still in
the release.
- Re-run the patched behaviour before deleting. A fix in the release does
not always cover vLLM's path: Emu3's processor gained BOS upstream, but
vLLM tokenizes without the processor.
- Links that only explain a design choice (e.g. a float32 RoPE note) are
not patches; leave them.
## Testing
```bash
.venv/bin/python -m pytest tests/transformers_utils tests/config tests/tokenizers_
.venv/bin/python -m pytest tests/models/multimodal/processing/test_common.py -k "<models>"
```
Add `tests/models/test_initialization.py -k "<archs>"` where a GPU is
available, and a model eval for any model whose resolved config changed.
Models in `vllm/models/` are high priority: whenever their config or model
code changes, run GSM8k (`tests/evals/gsm8k/`) on `main` and the branch,
even when the dumps show no diff, and put the score in the PR. FlashInfer
tactics afresh on every server start, which moved DeepSeek-V4-Flash GSM8k by
about 1 point; pass `--kernel-config '{"enable_flashinfer_autotune": false}'`
to both servers so greedy outputs are comparable.
@@ -0,0 +1,4 @@
interface:
display_name: "Transformers Upgrade"
short_description: "Bump vLLM's Transformers version and delete vendored code upstream now provides."
default_prompt: "Use $transformers-upgrade to raise the minimum Transformers version and remove vendored configs, processors, and shims it makes redundant."
@@ -0,0 +1,51 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
"""Diff two dump_configs.py or dump_model.py outputs, per repo.
python diff_dumps.py before.jsonl after.jsonl
"""
import json
import sys
from collections import Counter
import regex as re
IGNORED = {"transformers_version", "_name_or_path", "architectures", "auto_map"}
def load(path: str) -> dict[str, dict]:
with open(path) as f:
return {record["repo"]: record for record in map(json.loads, f)}
def diff(before: dict, after: dict, prefix: str = ""):
for key in sorted(set(before) | set(after)):
if key in IGNORED:
continue
old, new = before.get(key, "<MISSING>"), after.get(key, "<MISSING>")
if isinstance(old, dict) and isinstance(new, dict) and key != "rope_parameters":
yield from diff(old, new, f"{prefix}{key}.")
elif old != new:
yield f"{prefix}{key}", json.dumps(old), json.dumps(new)
def collapse(diffs) -> Counter:
"""Merge diffs that differ only by layer index, e.g. `layers.*.scaling`."""
return Counter(
(re.sub(r"\.\d+\.", ".*.", path), old, new) for path, old, new in diffs
)
before, after = load(sys.argv[1]), load(sys.argv[2])
for repo, old in before.items():
new = after.get(repo, {"error": "missing from after"})
if "error" in old or "error" in new:
print(f"== {repo}\n before: {old.get('error', 'ok')}")
print(f" after: {new.get('error', 'ok')}")
continue
diffs = collapse(diff(old["cfg"], new["cfg"]))
print(f"== {repo}: {old['cls']} -> {new['cls']} ({sum(diffs.values())} diffs)")
for (path, old_value, new_value), count in diffs.items():
times = f" (x{count})" if count > 1 else ""
print(f" {path}{times}: before={old_value} after={new_value}")
@@ -0,0 +1,30 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
"""Dump the config vLLM resolves for each checkpoint, one JSON line per repo.
Run once per vLLM tree (before and after a change), then compare the two
outputs with diff_dumps.py:
PYTHONPATH=<vllm tree> .venv/bin/python dump_configs.py out.jsonl <repo>...
Pass --trust-remote-code to resolve checkpoints the way users who set it do.
"""
import json
import sys
from vllm.transformers_utils.config import get_config
out, *args = sys.argv[1:]
trust_remote_code = "--trust-remote-code" in args
with open(out, "w") as f:
for repo in (arg for arg in args if arg != "--trust-remote-code"):
try:
config = get_config(repo, trust_remote_code=trust_remote_code)
cls = f"{type(config).__module__}.{type(config).__name__}"
record = {"repo": repo, "cls": cls, "cfg": config.to_dict()}
except Exception as e:
error = f"{type(e).__name__}: {e}".splitlines()[0]
record = {"repo": repo, "error": error}
f.write(json.dumps(record, default=str, sort_keys=True) + "\n")
f.flush()
@@ -0,0 +1,114 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
"""Build each model on the meta device and dump what vLLM constructs.
Records ModelConfig-derived sizes, every parameter shape, and scalar
attributes of every module (head_dim, num_kv_heads, sliding_window, ...).
Needs no GPU and no weights. Run once per vLLM tree, then compare the two
outputs with diff_dumps.py:
PYTHONPATH=<vllm tree> .venv/bin/python dump_model.py out.jsonl <repo>...
Pass --trust-remote-code to resolve checkpoints the way users who set it do.
"""
import json
import socket
import sys
import torch
from vllm.config import set_current_vllm_config
from vllm.config.vllm import VllmConfig
from vllm.distributed import (
init_distributed_environment,
initialize_model_parallel,
model_parallel_is_initialized,
)
from vllm.engine.arg_utils import EngineArgs
from vllm.model_executor.model_loader.utils import initialize_model
from vllm.utils.torch_utils import set_default_torch_dtype
# Construction only: skip model runner checks that need a GPU or Triton
VllmConfig._get_v1_model_runner_unsupported_features = lambda self: [] # type: ignore[method-assign]
# Model code that moves submodules to a device must leave them on meta
_module_to = torch.nn.Module.to
def _meta_safe_to(self, *args, **kwargs):
kwargs.pop("device", None)
args = tuple(a for a in args if not isinstance(a, (torch.device, str, int)))
return _module_to(self, *args, **kwargs)
torch.nn.Module.to = _meta_safe_to # type: ignore[method-assign]
SCALARS = (bool, int, float, str)
def module_attrs(model: torch.nn.Module) -> dict[str, dict]:
attrs = {}
for name, module in model.named_modules():
scalars = {
key: value
for key, value in vars(module).items()
if not key.startswith("_") and isinstance(value, SCALARS)
}
if scalars:
attrs[name or "<root>"] = scalars
return attrs
def dump(repo: str, trust_remote_code: bool) -> dict:
args = EngineArgs(
model=repo,
load_format="dummy",
enforce_eager=True,
max_model_len=4096,
trust_remote_code=trust_remote_code,
)
vllm_config = args.create_engine_config()
model_config = vllm_config.model_config
with set_current_vllm_config(vllm_config):
if not model_parallel_is_initialized():
with socket.socket() as sock:
sock.bind(("", 0))
port = sock.getsockname()[1]
init_distributed_environment(1, 0, f"tcp://127.0.0.1:{port}", 0, "gloo")
initialize_model_parallel(1, 1)
with set_default_torch_dtype(model_config.dtype), torch.device("meta"):
model = initialize_model(vllm_config=vllm_config)
return {
"repo": repo,
"cls": type(model).__name__,
"cfg": {
"model_config": {
"head_size": model_config.get_head_size(),
"total_num_kv_heads": model_config.get_total_num_kv_heads(),
"sliding_window": model_config.get_sliding_window(),
"num_layers": model_config.get_num_layers(vllm_config.parallel_config),
},
"params": {n: list(p.shape) for n, p in model.named_parameters()},
"modules": module_attrs(model),
},
}
def main() -> None:
out, *args = sys.argv[1:]
trust_remote_code = "--trust-remote-code" in args
repos = [arg for arg in args if arg != "--trust-remote-code"]
with open(out, "w") as f:
for repo in repos:
try:
record = dump(repo, trust_remote_code)
except Exception as e:
error = f"{type(e).__name__}: {e}".splitlines()[0]
record = {"repo": repo, "error": error}
f.write(json.dumps(record, default=str, sort_keys=True) + "\n")
f.flush()
if __name__ == "__main__":
main()
@@ -0,0 +1,147 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
"""List vendored vLLM code that the installed Transformers may make redundant.
Run with the Transformers version under test, from the vLLM repo root:
uv run --no-project --with transformers==<version> \
python .agents/skills/transformers-upgrade/scripts/find_upstreamed.py
Imports nothing from vLLM, so no vLLM install is needed. Uses the `gh` CLI, if
available, to check whether referenced Transformers PRs are in this release.
"""
import json
import subprocess
import sys
from pathlib import Path
import regex as re
import transformers
from packaging.version import Version
from transformers.models.auto.configuration_auto import CONFIG_MAPPING_NAMES
from transformers.models.auto.processing_auto import PROCESSOR_MAPPING_NAMES
ROOT = Path(sys.argv[1] if len(sys.argv) > 1 else ".")
FLOOR = Version(transformers.__version__)
UPSTREAM_NAMES = set(dir(transformers))
MODEL_TYPE = re.compile(r'model_type\s*(?::\s*str\s*)?=\s*"([^"]+)"')
REGISTRY_KEY = re.compile(r'^\s+(?:\*\*\{)?"?([\w-]+)"?(?:=|": )"(\w+)"', re.M)
CLASS_KEY = re.compile(r'^\s+"(\w+)": ', re.M)
UPSTREAM_REF = re.compile(
r"huggingface/transformers(?:/(?:issues|pull)/|#)(\d+)"
r"|(?:once|until|when) [Tt]ransformers [\d.]+ is the minimum"
)
VERSIONED = re.compile(
r"(?:min_transformers_version|max_transformers_version|check_version|Version)"
r'\(?\s*=?\s*"([45]\.\d+(?:\.\d+)?)[^"]*"'
r"|[Tt]ransformers\s*(?:v|[<>]=?\s*)([45]\.\d+(?:\.\d+)?|4)\b"
)
READS_VERSION = re.compile(
r"transformers import __version__|transformers\.__version__"
r"|version\([\"']transformers[\"']\)|TRANSFORMERS_VERSION|check_version\("
)
def section(title: str) -> None:
print(f"\n## {title}")
def vendored_configs() -> None:
section(f"Vendored configs whose model_type is upstream in {FLOOR}")
files = sorted(ROOT.glob("vllm/transformers_utils/configs/*.py"))
files += sorted(ROOT.glob("vllm/models/*/config*.py"))
files += sorted(ROOT.glob("vllm/models/*/configs/*.py"))
for path in files:
types = sorted(set(MODEL_TYPE.findall(path.read_text())))
hits = [t for t in types if t in CONFIG_MAPPING_NAMES]
if hits:
upstream = ", ".join(f"{t} -> {CONFIG_MAPPING_NAMES[t]}" for t in hits)
print(f"- {path.relative_to(ROOT)}: {upstream}")
registry = (ROOT / "vllm/transformers_utils/config.py").read_text()
start = registry.index("_CONFIG_REGISTRY")
body = registry[start : registry.index("\n)\n", start)]
section("_CONFIG_REGISTRY entries that override an upstream model_type")
for model_type, cls in REGISTRY_KEY.findall(body):
if model_type in CONFIG_MAPPING_NAMES:
print(f"- {model_type}: vLLM {cls} over {CONFIG_MAPPING_NAMES[model_type]}")
def vendored_processors() -> None:
init = (ROOT / "vllm/transformers_utils/processors/__init__.py").read_text()
names = sorted(set(CLASS_KEY.findall(init)))
section("Registered processor names that collide with upstream classes")
for name in names:
if name in UPSTREAM_NAMES:
print(f"- {name}")
section("Processor files named after a model_type with an upstream processor")
for path in sorted(ROOT.glob("vllm/transformers_utils/processors/*.py")):
if path.stem in PROCESSOR_MAPPING_NAMES:
upstream = PROCESSOR_MAPPING_NAMES[path.stem]
print(f"- {path.relative_to(ROOT)} -> {upstream}")
def version_gates() -> None:
files = sorted([*ROOT.glob("vllm/**/*.py"), *ROOT.glob("tests/**/*.py")])
section(f"Version-gated code at or below {FLOOR} (review each)")
for path in files:
for lineno, line in enumerate(path.read_text().splitlines(), 1):
for match in VERSIONED.finditer(line):
version = Version(match.group(1) or match.group(2))
if version <= FLOOR:
print(f"- {path.relative_to(ROOT)}:{lineno}: {line.strip()}")
break
section("Code that reads the installed Transformers version (review each)")
for path in files:
for lineno, line in enumerate(path.read_text().splitlines(), 1):
if READS_VERSION.search(line):
print(f"- {path.relative_to(ROOT)}:{lineno}: {line.strip()}")
def upstream_status(number: str) -> str:
"""Whether Transformers issue/PR `number` is resolved in this release."""
def gh(path: str) -> dict | None:
try:
out = subprocess.run(["gh", "api", path], capture_output=True, text=True)
except FileNotFoundError:
return None
return json.loads(out.stdout) if out.returncode == 0 else None
repo = "repos/huggingface/transformers"
if (issue := gh(f"{repo}/issues/{number}")) is None:
return "status unknown"
if "pull_request" not in issue:
return f"issue {issue['state']}, check which PR fixed it"
pr = gh(f"{repo}/pulls/{number}")
if pr is None or not pr["merged_at"]:
return "PR not merged"
compare = gh(f"{repo}/compare/v{FLOOR}...{pr['merge_commit_sha']}")
included = compare and compare["status"] in ("behind", "identical")
return f"PR merged, {'in' if included else 'not in'} v{FLOOR}"
def upstream_refs() -> None:
section("Patches referencing Transformers issues/PRs or a future minimum")
files = sorted([*ROOT.glob("vllm/**/*.py"), *ROOT.glob("tests/**/*.py")])
statuses: dict[str, str] = {}
for path in files:
for lineno, line in enumerate(path.read_text().splitlines(), 1):
if (match := UPSTREAM_REF.search(line)) is None:
continue
note = ""
if number := match.group(1):
if number not in statuses:
statuses[number] = upstream_status(number)
note = f" [#{number}: {statuses[number]}]"
print(f"- {path.relative_to(ROOT)}:{lineno}:{note} {line.strip()}")
if __name__ == "__main__":
print(f"# Transformers {FLOOR}")
vendored_configs()
vendored_processors()
version_gates()
upstream_refs()
View File
-166
View File
@@ -1,166 +0,0 @@
#!/bin/bash
# =============================================================================
# cluster.sh — single, generic config for the vLLM disaggregated (P/D) launcher.
# -----------------------------------------------------------------------------
# This script is *sourced* by vllm_disagg.sh.
#
# Every value is `${VAR:-default}`, so the environment always wins:
# environment variable > built-in default below
#
# So you can override anything inline:
# PREFILL_IP=10.0.0.1 DECODE_IP=10.0.0.2 ./vllm_disagg.sh prefill
#
# Site-specific values (model dir, IPs, NIC list, partition) are the defaults
# in the "site defaults" sections below — edit those for a new cluster.
# Model-SPECIFIC perf flags live in models.yaml, NOT here.
# =============================================================================
_CLUSTER_SH_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
# ----------------------------------------------------------------- model / mode
# MODEL_NAME indexes into models.yaml; MODEL_DIR is the parent dir holding it.
# MODEL_PATH (resolved by the launcher) = ${MODEL_DIR}/${MODEL_NAME}. [site]
export MODEL_NAME="${MODEL_NAME:-DeepSeek-V3}"
export MODEL_DIR="${MODEL_DIR:-/data/models2}"
# Shared NFS root (5 TB, visible on every node): model weights + per-run logs. [site]
export SHARED_MOUNT="${SHARED_MOUNT:-/data}"
# Parallelism mode (launcher derives PARALLEL_MODE tp|ep from this):
# WIDE_EP_MODE=0 tp : each node is an independent TP server (TP8, 1P1D)
# WIDE_EP_MODE=1 ep : data-parallel + expert-parallel across xP/yD nodes (wideep)
export WIDE_EP_MODE="${WIDE_EP_MODE:-0}"
# ----------------------------------------------------------------- topology
# xP prefill nodes + yD decode nodes. IPADDRS is the ordered, comma-separated
# node list (prefill IPs first, then decode IPs). NODE_RANK is this node's global
# 0-based rank; under SLURM it defaults to $SLURM_PROCID. Leave IPADDRS empty to
# use the PREFILL_IP/DECODE_IP fallback defaults below (1P1D only).
export xP="${xP:-1}"
export yD="${yD:-1}"
export IPADDRS="${IPADDRS:-}"
export NODE_RANK="${NODE_RANK:-${SLURM_PROCID:-}}"
export PREFILL_IP="${PREFILL_IP:-10.0.0.1}"
export DECODE_IP="${DECODE_IP:-10.0.0.2}"
# Per-node GPU count and TP degree
export GPUS_PER_NODE="${GPUS_PER_NODE:-8}"
export TP_SIZE="${TP_SIZE:-${GPUS_PER_NODE}}"
# ----------------------------------------------------------------- ports
# TP-mode (WIDE_EP_MODE=0) server ports.
export PREFILL_PORT="${PREFILL_PORT:-8100}"
export DECODE_PORT="${DECODE_PORT:-8200}"
# EP-mode (WIDE_EP_MODE=1) ports: API serve port, DP RPC port, KV transfer port, and
# per-node MoRIIO local ping port (must differ from PROXY_PING_PORT).
export SERVE_PORT="${SERVE_PORT:-20005}"
export RPC_PORT="${RPC_PORT:-13345}"
export KV_PORT="${KV_PORT:-9711}"
export LOCAL_PING_PORT="${LOCAL_PING_PORT:-61555}"
# MoRIIO proxy: HTTP port clients/benchmark hit, plus the connector control ports.
# PROXY_PING_PORT MUST be 36367 — the proxy hardcodes its zmq service-discovery
# socket on that port; prefill/decode register to PROXY_IP:PROXY_PING_PORT.
export PROXY_IP="${PROXY_IP:-${PREFILL_IP}}"
export PROXY_PORT="${PROXY_PORT:-10001}"
export PROXY_PING_PORT="${PROXY_PING_PORT:-36367}"
export HANDSHAKE_PORT="${HANDSHAKE_PORT:-6301}"
export NOTIFY_PORT="${NOTIFY_PORT:-61005}"
export PROXY_SCRIPT="${PROXY_SCRIPT:-/app/vllm/examples/disaggregated/disaggregated_serving/moriio_toy_proxy_server.py}"
# MoRIIO KV transfer direction (injected into --kv-transfer-config by the launcher):
# 0 -> omit read_mode (default; MoRIIO write mode: prefill pushes to decode)
# 1 -> "read_mode": true (decode pulls KV from prefill; matches upstream disagg)
export MORIIO_READ_MODE="${MORIIO_READ_MODE:-0}"
# ----------------------------------------------------------------- router / gateway
# Selection for client (bench/accuracy) traffic:
# proxy -> the in-container MoRIIO proxy started by the launcher
# vllm-router -> an external `vllm/vllm-router` container started by the SLURM job
# on the rank-0 node (default)
# Both use the SAME MoRIIO discovery mechanism (prefill/decode register to
# PROXY_IP:PROXY_PING_PORT=36367); only the client HTTP front door differs.
export ROUTER_TYPE="${ROUTER_TYPE:-vllm-router}"
export ROUTER_PORT="${ROUTER_PORT:-30000}"
export ROUTER_POLICY="${ROUTER_POLICY:-round_robin}"
export VLLM_ROUTER_IMAGE="${VLLM_ROUTER_IMAGE:-vllm/vllm-router:nightly}"
# Single client-facing port bench/accuracy target: the router port when routing,
# else the proxy port. Env override always wins.
if [[ "${ROUTER_TYPE}" == "vllm-router" ]]; then
export GATEWAY_PORT="${GATEWAY_PORT:-${ROUTER_PORT}}"
else
export GATEWAY_PORT="${GATEWAY_PORT:-${PROXY_PORT}}"
fi
# Where per-run logs / benchmark results are written. A $SLURM_JOB_ID subdir is
# appended so each CI run is self-scoped (falls back to 'local' off-SLURM). [site]
_LOG_BASE="${LOG_BASE:-/data/${USER:-$(id -un)}/disagg_logs}"
export LOG_PATH="${LOG_PATH:-${_LOG_BASE}/${SLURM_JOB_ID:-local}}"
# ----------------------------------------------------------------- vLLM runtime
# Engine/platform/transport-level env (NOT model-specific). Model-architecture
# AITER kernel toggles live in models.yaml under each model's `env:` block.
export VLLM_USE_V1="${VLLM_USE_V1:-1}"
export HSA_NO_SCRATCH_RECLAIM="${HSA_NO_SCRATCH_RECLAIM:-1}"
#export HF_HUB_OFFLINE="${HF_HUB_OFFLINE:-1}"
#export TRANSFORMERS_OFFLINE="${TRANSFORMERS_OFFLINE:-1}"
#
#export HOME=/tmp
export HF_HOME="${HF_HOME:-/tmp/hf_home}"
export XDG_CACHE_HOME="${XDG_CACHE_HOME:-/tmp/.cache}"
export VLLM_ENGINE_READY_TIMEOUT_S="${VLLM_ENGINE_READY_TIMEOUT_S:-3600}"
# ----------------------------------------------------------------- scale
# DP/EP group formation timeout across nodes (seconds).
export DISTRIBUTED_TIMEOUT_SECONDS="${DISTRIBUTED_TIMEOUT_SECONDS:-7200}"
# ----------------------------------------------------------------- RDMA / NCCL
# AMD Pensando AINIC RoCE fabric: 8 NICs exposed as ionic_0..7 (netdevs eth2..9),
# each rail on its own /24. GID index 1 + traffic class 104
_IB_DEVICES="${IB_DEVICES:-ionic_0,ionic_1,ionic_2,ionic_3,ionic_4,ionic_5,ionic_6,ionic_7}"
_IB_GID_INDEX="${NCCL_IB_GID_INDEX:-1}"
export IB_DEVICES="${IB_DEVICES:-${_IB_DEVICES}}"
export NCCL_IB_HCA="${NCCL_IB_HCA:-${_IB_DEVICES}}"
export NCCL_IB_GID_INDEX="${NCCL_IB_GID_INDEX:-${_IB_GID_INDEX}}"
export NCCL_IB_DISABLE="${NCCL_IB_DISABLE:-0}"
# In-box RCCL net transport (no external plugin); pin bootstrap to the VPC iface.
export NCCL_NET_PLUGIN="${NCCL_NET_PLUGIN:-none}"
export NCCL_SOCKET_IFNAME="${NCCL_SOCKET_IFNAME:-eth0}"
export GLOO_SOCKET_IFNAME="${GLOO_SOCKET_IFNAME:-${NCCL_SOCKET_IFNAME}}"
export NCCL_CROSS_NIC="${NCCL_CROSS_NIC:-0}"
export NCCL_PXN_DISABLE="${NCCL_PXN_DISABLE:-0}"
export NCCL_NET_DISABLE_INTRA="${NCCL_NET_DISABLE_INTRA:-1}"
export NCCL_IB_TC="${NCCL_IB_TC:-104}"
export NCCL_IB_FIFO_TC="${NCCL_IB_FIFO_TC:-192}"
export NCCL_IB_QPS_PER_CONNECTION="${NCCL_IB_QPS_PER_CONNECTION:-1}"
export NCCL_IB_TIMEOUT="${NCCL_IB_TIMEOUT:-22}"
export NCCL_IB_RETRY_CNT="${NCCL_IB_RETRY_CNT:-12}"
# MoRI uses the same NIC set as NCCL.
export MORI_RDMA_DEVICES="${MORI_RDMA_DEVICES:-${_IB_DEVICES}}"
export MORI_IB_GID_INDEX="${MORI_IB_GID_INDEX:-${_IB_GID_INDEX}}"
export MORI_SHMEM_HEAP_SIZE="${MORI_SHMEM_HEAP_SIZE:-16G}"
# Pin to gfx950 to avoid jit compilation failures with other archs on this cluster.
export MORI_GPU_ARCHS="gfx950"
# ----------------------------------------------------------------- benchmark
export BENCHMARK_COMBINATIONS="${BENCHMARK_COMBINATIONS:-1024/128 2048/128}"
export BENCHMARK_CON="${BENCHMARK_CON:-32 64}"
export NUM_PROMPTS_FACTOR="${NUM_PROMPTS_FACTOR:-2}"
export BENCHMARK_MIN_PROMPTS="${BENCHMARK_MIN_PROMPTS:-32}"
# ----------------------------------------------------------------- accuracy
# `accuracy` role runs lm_eval (local-completions backend) against the proxy.
export ACCURACY_TASKS="${ACCURACY_TASKS:-gsm8k}"
export ACCURACY_NUM_CONCURRENT="${ACCURACY_NUM_CONCURRENT:-64}"
export ACCURACY_MAX_RETRIES="${ACCURACY_MAX_RETRIES:-3}"
export ACCURACY_METRIC="${ACCURACY_METRIC:-exact_match}"
export ACCURACY_THRESHOLD="${ACCURACY_THRESHOLD:-0.90}"
# ----------------------------------------------------------------- SLURM (submit)
# Used by run-slurm-disagg-test.sh on the login node (harmless to export here). [site]
export SLURM_PARTITION="${SLURM_PARTITION:-default}"
-121
View File
@@ -1,121 +0,0 @@
# =============================================================================
# Model catalog for the vLLM disaggregated (P/D) inference CI.
# -----------------------------------------------------------------------------
# Loaded by vllm_disagg.sh for the selected MODEL_NAME. WIDE_EP_MODE selects the
# flag set (0 -> tp keys, 1 -> ep keys):
# tp -> base_flags + prefill.tp / decode.tp
# ep -> base_flags + prefill.ep / decode.ep
#
# Keep ONLY model-specific perf flags here. The launcher owns everything
# topological / mode-structural:
# - tp: --host/--port, --tensor-parallel-size, --kv-transfer-config
# - ep: --tp 1, --data-parallel-size[-local], --data-parallel-address/-rpc-port,
# --data-parallel-start-rank/--headless (children), --enable-expert-parallel,
# --all2all-backend mori, --no-enable-prefix-caching,
# --api-server-count + --kv-transfer-config (masters only)
# Do NOT put any of those in this file.
#
# Format: a top-level `models:` list. The launcher selects the entry whose
# `model:` matches MODEL_NAME. Each entry:
# model : name (must match MODEL_NAME; also the dir under MODEL_DIR)
# env : model/arch-specific environment variables. The launcher
# exports these BEFORE `vllm serve`, but only if they are
# not already set, so precedence is:
# caller/inline env > models.yaml env:
# (Transport/fabric/engine env stays in cluster.sh,
# e.g. MORI_RDMA_DEVICES, NCCL_IB_HCA, VLLM_USE_V1.)
# base_flags : always applied (both roles, both modes)
# prefill/decode.<tp|ep> : role + mode specific perf flags
# experimental_flags : optional extra flags (omit if empty)
# =============================================================================
models:
- model: DeepSeek-V3
env:
VLLM_ROCM_USE_AITER: "1"
VLLM_ROCM_USE_AITER_MLA: "1"
VLLM_ROCM_USE_AITER_MOE: "1"
VLLM_ROCM_USE_AITER_RMSNORM: "1"
VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS: "0"
VLLM_ROCM_USE_AITER_FP8BMM: "false"
base_flags: "--trust-remote-code --kv-cache-dtype fp8"
prefill:
tp: "--gpu-memory-utilization 0.85"
ep: "--gpu-memory-utilization 0.85 --enforce-eager"
decode:
tp: "--gpu-memory-utilization 0.85"
ep: '--gpu-memory-utilization 0.75 --compilation-config {"cudagraph_mode":"FULL_DECODE_ONLY","custom_ops":["+quant_fp8"]}'
- model: MiniMax-M3-MXFP8
env:
VLLM_ROCM_USE_AITER: "1"
VLLM_ROCM_USE_AITER_MOE: "1"
VLLM_ROCM_USE_AITER_RMSNORM: "1"
VLLM_USE_BREAKABLE_CUDAGRAPH: "0"
VLLM_ROCM_QUICK_REDUCE_QUANTIZATION: "INT6"
VLLM_KV_CACHE_LAYOUT: "HND"
base_flags: "--trust-remote-code --attention-backend TRITON_ATTN --block-size 128 --language-model-only --kv-cache-dtype fp8"
prefill:
tp: "--gpu-memory-utilization 0.85 --enforce-eager"
ep: "--gpu-memory-utilization 0.85 --enforce-eager"
decode:
tp: "--gpu-memory-utilization 0.85"
ep: '--gpu-memory-utilization 0.75 --compilation-config {"cudagraph_mode":"FULL_DECODE_ONLY","custom_ops":["+quant_fp8"]}'
- model: DeepSeek-R1-MXFP4
env:
VLLM_ROCM_USE_AITER: "1"
VLLM_ROCM_USE_AITER_MLA: "1"
VLLM_ROCM_USE_AITER_MOE: "1"
VLLM_ROCM_USE_AITER_RMSNORM: "1"
VLLM_ROCM_USE_AITER_FP8BMM: "false"
base_flags: "--trust-remote-code --kv-cache-dtype fp8"
prefill:
tp: "--gpu-memory-utilization 0.85"
ep: "--gpu-memory-utilization 0.85 --enforce-eager"
decode:
tp: "--gpu-memory-utilization 0.85"
ep: '--gpu-memory-utilization 0.75 --compilation-config {"cudagraph_mode":"FULL_DECODE_ONLY","custom_ops":["+quant_fp8"]}'
- model: Kimi-K2.5-MXFP4
env:
VLLM_ROCM_USE_AITER: "1"
VLLM_ROCM_USE_AITER_MOE: "1"
VLLM_ROCM_USE_AITER_RMSNORM: "1"
base_flags: "--trust-remote-code"
prefill:
tp: "--gpu-memory-utilization 0.85"
ep: "--gpu-memory-utilization 0.85 --enforce-eager"
decode:
tp: "--gpu-memory-utilization 0.85"
ep: '--gpu-memory-utilization 0.75 --compilation-config {"cudagraph_mode":"FULL_DECODE_ONLY","custom_ops":["+quant_fp8"]}'
- model: Kimi-K2.6-MXFP4
env:
VLLM_ROCM_USE_AITER: "1"
AMDGCN_USE_BUFFER_OPS: "1"
VLLM_ROCM_USE_AITER_MLA: "1"
VLLM_ROCM_QUICK_REDUCE_QUANTIZATION: "INT4"
VLLM_ROCM_USE_SKINNY_GEMM: "0"
base_flags: "--trust-remote-code --kv-cache-dtype fp8 --mm-encoder-tp-mode data --block-size 1 --attention-backend ROCM_AITER_MLA"
prefill:
tp: "--gpu-memory-utilization 0.9"
ep: "--gpu-memory-utilization 0.85 --enforce-eager"
decode:
tp: "--gpu-memory-utilization 0.9"
ep: '--gpu-memory-utilization 0.75 --compilation-config {"cudagraph_mode":"FULL_DECODE_ONLY","custom_ops":["+quant_fp8"]}'
- model: GLM-5.2-FP8
env:
VLLM_ROCM_USE_AITER: "1"
VLLM_ROCM_USE_AITER_MLA: "1"
VLLM_ROCM_USE_AITER_MOE: "1"
VLLM_ROCM_USE_AITER_RMSNORM: "1"
VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS: "0"
base_flags: "--trust-remote-code --kv-cache-dtype fp8"
prefill:
tp: "--gpu-memory-utilization 0.85"
ep: "--gpu-memory-utilization 0.85 --enforce-eager"
decode:
tp: "--gpu-memory-utilization 0.85"
ep: '--gpu-memory-utilization 0.85 --compilation-config {"cudagraph_mode":"PIECEWISE"}'
-319
View File
@@ -1,319 +0,0 @@
# Native Buildkite pipeline — disaggregated PD SLURM tests (MoRIIO).
#
# ROUTER_TYPE must be set explicitly on every step: run-slurm-disagg-test.sh
# defaults to `vllm-router`
#
steps:
#-------------------------------------- mi350 · disagg 1P1D (TP8, 2 nodes) · SLURM --------------------------------------#
- label: "DeepSeek-V3-PD-1P1D-TP8-MoRIIO-proxy"
agents:
queue: amd_mi350_ainic
timeout_in_minutes: 180
concurrency: 2
concurrency_group: "amd-disagg/mi350/2node"
commands:
- MODEL_NAME=DeepSeek-V3 NODES=2 GPUS_PER_NODE=8 WIDE_EP_MODE=0 MORIIO_READ_MODE=0 RUN_AFTER_HEALTH=accuracy ROUTER_TYPE=proxy WAIT=1 bash .buildkite/amd-disagg/run-slurm-disagg-test.sh
- label: "DeepSeek-V3-PD-1P1D-TP8-MoRIIO-vllm-router"
agents:
queue: amd_mi350_ainic
timeout_in_minutes: 180
concurrency: 2
concurrency_group: "amd-disagg/mi350/2node"
commands:
- MODEL_NAME=DeepSeek-V3 NODES=2 GPUS_PER_NODE=8 WIDE_EP_MODE=0 MORIIO_READ_MODE=0 RUN_AFTER_HEALTH=accuracy ROUTER_TYPE=vllm-router WAIT=1 bash .buildkite/amd-disagg/run-slurm-disagg-test.sh
- label: "DeepSeek-R1-MXFP4-PD-1P1D-TP8-MoRIIO-proxy"
agents:
queue: amd_mi350_ainic
timeout_in_minutes: 180
concurrency: 2
concurrency_group: "amd-disagg/mi350/2node"
commands:
- MODEL_NAME=DeepSeek-R1-MXFP4 NODES=2 GPUS_PER_NODE=8 WIDE_EP_MODE=0 MORIIO_READ_MODE=0 RUN_AFTER_HEALTH=accuracy ROUTER_TYPE=proxy WAIT=1 bash .buildkite/amd-disagg/run-slurm-disagg-test.sh
- label: "DeepSeek-R1-MXFP4-PD-1P1D-TP8-MoRIIO-vllm-router"
agents:
queue: amd_mi350_ainic
timeout_in_minutes: 180
concurrency: 2
concurrency_group: "amd-disagg/mi350/2node"
commands:
- MODEL_NAME=DeepSeek-R1-MXFP4 NODES=2 GPUS_PER_NODE=8 WIDE_EP_MODE=0 MORIIO_READ_MODE=0 RUN_AFTER_HEALTH=accuracy ROUTER_TYPE=vllm-router WAIT=1 bash .buildkite/amd-disagg/run-slurm-disagg-test.sh
- label: "Kimi-K2.5-MXFP4-PD-1P1D-TP8-MoRIIO-proxy"
agents:
queue: amd_mi350_ainic
timeout_in_minutes: 180
concurrency: 2
concurrency_group: "amd-disagg/mi350/2node"
commands:
- MODEL_NAME=Kimi-K2.5-MXFP4 NODES=2 GPUS_PER_NODE=8 WIDE_EP_MODE=0 MORIIO_READ_MODE=0 RUN_AFTER_HEALTH=accuracy ROUTER_TYPE=proxy WAIT=1 bash .buildkite/amd-disagg/run-slurm-disagg-test.sh
- label: "Kimi-K2.5-MXFP4-PD-1P1D-TP8-MoRIIO-vllm-router"
agents:
queue: amd_mi350_ainic
timeout_in_minutes: 180
concurrency: 2
concurrency_group: "amd-disagg/mi350/2node"
commands:
- MODEL_NAME=Kimi-K2.5-MXFP4 NODES=2 GPUS_PER_NODE=8 WIDE_EP_MODE=0 MORIIO_READ_MODE=0 RUN_AFTER_HEALTH=accuracy ROUTER_TYPE=vllm-router WAIT=1 bash .buildkite/amd-disagg/run-slurm-disagg-test.sh
- label: "Kimi-K2.6-MXFP4-PD-1P1D-TP8-MoRIIO-proxy"
agents:
queue: amd_mi350_ainic
timeout_in_minutes: 180
concurrency: 2
concurrency_group: "amd-disagg/mi350/2node"
commands:
- MODEL_NAME=Kimi-K2.6-MXFP4 NODES=2 GPUS_PER_NODE=8 WIDE_EP_MODE=0 MORIIO_READ_MODE=0 RUN_AFTER_HEALTH=accuracy ROUTER_TYPE=proxy WAIT=1 bash .buildkite/amd-disagg/run-slurm-disagg-test.sh
- label: "Kimi-K2.6-MXFP4-PD-1P1D-TP8-MoRIIO-vllm-router"
agents:
queue: amd_mi350_ainic
timeout_in_minutes: 180
concurrency: 2
concurrency_group: "amd-disagg/mi350/2node"
commands:
- MODEL_NAME=Kimi-K2.6-MXFP4 NODES=2 GPUS_PER_NODE=8 WIDE_EP_MODE=0 MORIIO_READ_MODE=0 RUN_AFTER_HEALTH=accuracy ROUTER_TYPE=vllm-router WAIT=1 bash .buildkite/amd-disagg/run-slurm-disagg-test.sh
- label: "MiniMax-M3-MXFP8-PD-1P1D-TP8-MoRIIO-proxy"
agents:
queue: amd_mi350_ainic
timeout_in_minutes: 180
concurrency: 2
concurrency_group: "amd-disagg/mi350/2node"
commands:
- MODEL_NAME=MiniMax-M3-MXFP8 NODES=2 GPUS_PER_NODE=8 WIDE_EP_MODE=0 MORIIO_READ_MODE=0 RUN_AFTER_HEALTH=accuracy ROUTER_TYPE=proxy WAIT=1 bash .buildkite/amd-disagg/run-slurm-disagg-test.sh
- label: "MiniMax-M3-MXFP8-PD-1P1D-TP8-MoRIIO-vllm-router"
agents:
queue: amd_mi350_ainic
timeout_in_minutes: 180
concurrency: 2
concurrency_group: "amd-disagg/mi350/2node"
commands:
- MODEL_NAME=MiniMax-M3-MXFP8 NODES=2 GPUS_PER_NODE=8 WIDE_EP_MODE=0 MORIIO_READ_MODE=0 RUN_AFTER_HEALTH=accuracy ROUTER_TYPE=vllm-router WAIT=1 bash .buildkite/amd-disagg/run-slurm-disagg-test.sh
- label: "GLM-5.2-FP8-PD-1P1D-TP8-MoRIIO-proxy"
agents:
queue: amd_mi350_ainic
timeout_in_minutes: 180
concurrency: 2
concurrency_group: "amd-disagg/mi350/2node"
commands:
- MODEL_NAME=GLM-5.2-FP8 NODES=2 GPUS_PER_NODE=8 WIDE_EP_MODE=0 MORIIO_READ_MODE=0 RUN_AFTER_HEALTH=accuracy ROUTER_TYPE=proxy WAIT=1 bash .buildkite/amd-disagg/run-slurm-disagg-test.sh
- label: "GLM-5.2-FP8-PD-1P1D-TP8-MoRIIO-vllm-router"
agents:
queue: amd_mi350_ainic
timeout_in_minutes: 180
concurrency: 2
concurrency_group: "amd-disagg/mi350/2node"
commands:
- MODEL_NAME=GLM-5.2-FP8 NODES=2 GPUS_PER_NODE=8 WIDE_EP_MODE=0 MORIIO_READ_MODE=0 RUN_AFTER_HEALTH=accuracy ROUTER_TYPE=vllm-router WAIT=1 bash .buildkite/amd-disagg/run-slurm-disagg-test.sh
#-------------------------------------- mi350 · disagg 2P2D (TP8, 4 nodes) · SLURM --------------------------------------#
- label: "DeepSeek-V3-PD-2P2D-TP8-MoRIIO-proxy"
agents:
queue: amd_mi350_ainic
timeout_in_minutes: 180
concurrency: 2
concurrency_group: "amd-disagg/mi350/4node"
commands:
- MODEL_NAME=DeepSeek-V3 xP=2 yD=2 NODES=4 GPUS_PER_NODE=8 WIDE_EP_MODE=0 MORIIO_READ_MODE=0 RUN_AFTER_HEALTH=accuracy ROUTER_TYPE=proxy WAIT=1 bash .buildkite/amd-disagg/run-slurm-disagg-test.sh
- label: "DeepSeek-V3-PD-2P2D-TP8-MoRIIO-vllm-router"
agents:
queue: amd_mi350_ainic
timeout_in_minutes: 180
concurrency: 2
concurrency_group: "amd-disagg/mi350/4node"
commands:
- MODEL_NAME=DeepSeek-V3 xP=2 yD=2 NODES=4 GPUS_PER_NODE=8 WIDE_EP_MODE=0 MORIIO_READ_MODE=0 RUN_AFTER_HEALTH=accuracy ROUTER_TYPE=vllm-router WAIT=1 bash .buildkite/amd-disagg/run-slurm-disagg-test.sh
- label: "DeepSeek-R1-MXFP4-PD-2P2D-TP8-MoRIIO-proxy"
agents:
queue: amd_mi350_ainic
timeout_in_minutes: 180
concurrency: 2
concurrency_group: "amd-disagg/mi350/4node"
commands:
- MODEL_NAME=DeepSeek-R1-MXFP4 xP=2 yD=2 NODES=4 GPUS_PER_NODE=8 WIDE_EP_MODE=0 MORIIO_READ_MODE=0 RUN_AFTER_HEALTH=accuracy ROUTER_TYPE=proxy WAIT=1 bash .buildkite/amd-disagg/run-slurm-disagg-test.sh
- label: "DeepSeek-R1-MXFP4-PD-2P2D-TP8-MoRIIO-vllm-router"
agents:
queue: amd_mi350_ainic
timeout_in_minutes: 180
concurrency: 2
concurrency_group: "amd-disagg/mi350/4node"
commands:
- MODEL_NAME=DeepSeek-R1-MXFP4 xP=2 yD=2 NODES=4 GPUS_PER_NODE=8 WIDE_EP_MODE=0 MORIIO_READ_MODE=0 RUN_AFTER_HEALTH=accuracy ROUTER_TYPE=vllm-router WAIT=1 bash .buildkite/amd-disagg/run-slurm-disagg-test.sh
- label: "Kimi-K2.5-MXFP4-PD-2P2D-TP8-MoRIIO-proxy"
agents:
queue: amd_mi350_ainic
timeout_in_minutes: 180
concurrency: 2
concurrency_group: "amd-disagg/mi350/4node"
commands:
- MODEL_NAME=Kimi-K2.5-MXFP4 xP=2 yD=2 NODES=4 GPUS_PER_NODE=8 WIDE_EP_MODE=0 MORIIO_READ_MODE=0 RUN_AFTER_HEALTH=accuracy ROUTER_TYPE=proxy WAIT=1 bash .buildkite/amd-disagg/run-slurm-disagg-test.sh
- label: "Kimi-K2.5-MXFP4-PD-2P2D-TP8-MoRIIO-vllm-router"
agents:
queue: amd_mi350_ainic
timeout_in_minutes: 180
concurrency: 2
concurrency_group: "amd-disagg/mi350/4node"
commands:
- MODEL_NAME=Kimi-K2.5-MXFP4 xP=2 yD=2 NODES=4 GPUS_PER_NODE=8 WIDE_EP_MODE=0 MORIIO_READ_MODE=0 RUN_AFTER_HEALTH=accuracy ROUTER_TYPE=vllm-router WAIT=1 bash .buildkite/amd-disagg/run-slurm-disagg-test.sh
- label: "Kimi-K2.6-MXFP4-PD-2P2D-TP8-MoRIIO-proxy"
agents:
queue: amd_mi350_ainic
timeout_in_minutes: 180
concurrency: 2
concurrency_group: "amd-disagg/mi350/4node"
commands:
- MODEL_NAME=Kimi-K2.6-MXFP4 xP=2 yD=2 NODES=4 GPUS_PER_NODE=8 WIDE_EP_MODE=0 MORIIO_READ_MODE=0 RUN_AFTER_HEALTH=accuracy ROUTER_TYPE=proxy WAIT=1 bash .buildkite/amd-disagg/run-slurm-disagg-test.sh
- label: "Kimi-K2.6-MXFP4-PD-2P2D-TP8-MoRIIO-vllm-router"
agents:
queue: amd_mi350_ainic
timeout_in_minutes: 180
concurrency: 2
concurrency_group: "amd-disagg/mi350/4node"
commands:
- MODEL_NAME=Kimi-K2.6-MXFP4 xP=2 yD=2 NODES=4 GPUS_PER_NODE=8 WIDE_EP_MODE=0 MORIIO_READ_MODE=0 RUN_AFTER_HEALTH=accuracy ROUTER_TYPE=vllm-router WAIT=1 bash .buildkite/amd-disagg/run-slurm-disagg-test.sh
- label: "MiniMax-M3-MXFP8-PD-2P2D-TP8-MoRIIO-proxy"
agents:
queue: amd_mi350_ainic
timeout_in_minutes: 180
concurrency: 2
concurrency_group: "amd-disagg/mi350/4node"
commands:
- MODEL_NAME=MiniMax-M3-MXFP8 xP=2 yD=2 NODES=4 GPUS_PER_NODE=8 WIDE_EP_MODE=0 MORIIO_READ_MODE=0 RUN_AFTER_HEALTH=accuracy ROUTER_TYPE=proxy WAIT=1 bash .buildkite/amd-disagg/run-slurm-disagg-test.sh
- label: "MiniMax-M3-MXFP8-PD-2P2D-TP8-MoRIIO-vllm-router"
agents:
queue: amd_mi350_ainic
timeout_in_minutes: 180
concurrency: 2
concurrency_group: "amd-disagg/mi350/4node"
commands:
- MODEL_NAME=MiniMax-M3-MXFP8 xP=2 yD=2 NODES=4 GPUS_PER_NODE=8 WIDE_EP_MODE=0 MORIIO_READ_MODE=0 RUN_AFTER_HEALTH=accuracy ROUTER_TYPE=vllm-router WAIT=1 bash .buildkite/amd-disagg/run-slurm-disagg-test.sh
#-------------------------------------- mi350 · disagg wide-EP intranode (EP8/DP8, 2 nodes) · SLURM --------------------------------------#
- label: "DeepSeek-V3-PD-1P1D-EP8/DP8-WideEP-MoRIIO-proxy"
agents:
queue: amd_mi350_ainic
timeout_in_minutes: 180
concurrency: 2
concurrency_group: "amd-disagg/mi350/2node"
commands:
- MODEL_NAME=DeepSeek-V3 NODES=2 GPUS_PER_NODE=8 WIDE_EP_MODE=1 MORIIO_READ_MODE=0 RUN_AFTER_HEALTH=accuracy ROUTER_TYPE=proxy WAIT=1 bash .buildkite/amd-disagg/run-slurm-disagg-test.sh
- label: "DeepSeek-V3-PD-1P1D-EP8/DP8-WideEP-MoRIIO-vllm-router"
agents:
queue: amd_mi350_ainic
timeout_in_minutes: 180
concurrency: 2
concurrency_group: "amd-disagg/mi350/2node"
commands:
- MODEL_NAME=DeepSeek-V3 NODES=2 GPUS_PER_NODE=8 WIDE_EP_MODE=1 MORIIO_READ_MODE=0 RUN_AFTER_HEALTH=accuracy ROUTER_TYPE=vllm-router WAIT=1 bash .buildkite/amd-disagg/run-slurm-disagg-test.sh
- label: "DeepSeek-R1-MXFP4-PD-1P1D-EP8/DP8-WideEP-MoRIIO-proxy"
agents:
queue: amd_mi350_ainic
timeout_in_minutes: 180
concurrency: 2
concurrency_group: "amd-disagg/mi350/2node"
commands:
- MODEL_NAME=DeepSeek-R1-MXFP4 NODES=2 GPUS_PER_NODE=8 WIDE_EP_MODE=1 MORIIO_READ_MODE=0 RUN_AFTER_HEALTH=accuracy ROUTER_TYPE=proxy WAIT=1 bash .buildkite/amd-disagg/run-slurm-disagg-test.sh
- label: "DeepSeek-R1-MXFP4-PD-1P1D-EP8/DP8-WideEP-MoRIIO-vllm-router"
agents:
queue: amd_mi350_ainic
timeout_in_minutes: 180
concurrency: 2
concurrency_group: "amd-disagg/mi350/2node"
commands:
- MODEL_NAME=DeepSeek-R1-MXFP4 NODES=2 GPUS_PER_NODE=8 WIDE_EP_MODE=1 MORIIO_READ_MODE=0 RUN_AFTER_HEALTH=accuracy ROUTER_TYPE=vllm-router WAIT=1 bash .buildkite/amd-disagg/run-slurm-disagg-test.sh
- label: "Kimi-K2.5-MXFP4-PD-1P1D-EP8/DP8-WideEP-MoRIIO-proxy"
agents:
queue: amd_mi350_ainic
timeout_in_minutes: 180
concurrency: 2
concurrency_group: "amd-disagg/mi350/2node"
commands:
- MODEL_NAME=Kimi-K2.5-MXFP4 NODES=2 GPUS_PER_NODE=8 WIDE_EP_MODE=1 MORIIO_READ_MODE=0 RUN_AFTER_HEALTH=accuracy ROUTER_TYPE=proxy WAIT=1 bash .buildkite/amd-disagg/run-slurm-disagg-test.sh
- label: "Kimi-K2.5-MXFP4-PD-1P1D-EP8/DP8-WideEP-MoRIIO-vllm-router"
agents:
queue: amd_mi350_ainic
timeout_in_minutes: 180
concurrency: 2
concurrency_group: "amd-disagg/mi350/2node"
commands:
- MODEL_NAME=Kimi-K2.5-MXFP4 NODES=2 GPUS_PER_NODE=8 WIDE_EP_MODE=1 MORIIO_READ_MODE=0 RUN_AFTER_HEALTH=accuracy ROUTER_TYPE=vllm-router WAIT=1 bash .buildkite/amd-disagg/run-slurm-disagg-test.sh
- label: "Kimi-K2.6-MXFP4-PD-1P1D-EP8/DP8-WideEP-MoRIIO-proxy"
agents:
queue: amd_mi350_ainic
timeout_in_minutes: 180
concurrency: 2
concurrency_group: "amd-disagg/mi350/2node"
commands:
- MODEL_NAME=Kimi-K2.6-MXFP4 NODES=2 GPUS_PER_NODE=8 WIDE_EP_MODE=1 MORIIO_READ_MODE=0 RUN_AFTER_HEALTH=accuracy ROUTER_TYPE=proxy WAIT=1 bash .buildkite/amd-disagg/run-slurm-disagg-test.sh
- label: "Kimi-K2.6-MXFP4-PD-1P1D-EP8/DP8-WideEP-MoRIIO-vllm-router"
agents:
queue: amd_mi350_ainic
timeout_in_minutes: 180
concurrency: 2
concurrency_group: "amd-disagg/mi350/2node"
commands:
- MODEL_NAME=Kimi-K2.6-MXFP4 NODES=2 GPUS_PER_NODE=8 WIDE_EP_MODE=1 MORIIO_READ_MODE=0 RUN_AFTER_HEALTH=accuracy ROUTER_TYPE=vllm-router WAIT=1 bash .buildkite/amd-disagg/run-slurm-disagg-test.sh
- label: "MiniMax-M3-MXFP8-PD-1P1D-EP8/DP8-WideEP-MoRIIO-proxy"
agents:
queue: amd_mi350_ainic
timeout_in_minutes: 180
concurrency: 2
concurrency_group: "amd-disagg/mi350/2node"
commands:
- MODEL_NAME=MiniMax-M3-MXFP8 NODES=2 GPUS_PER_NODE=8 WIDE_EP_MODE=1 MORIIO_READ_MODE=0 RUN_AFTER_HEALTH=accuracy ROUTER_TYPE=proxy WAIT=1 bash .buildkite/amd-disagg/run-slurm-disagg-test.sh
- label: "MiniMax-M3-MXFP8-PD-1P1D-EP8/DP8-WideEP-MoRIIO-vllm-router"
agents:
queue: amd_mi350_ainic
timeout_in_minutes: 180
concurrency: 2
concurrency_group: "amd-disagg/mi350/2node"
commands:
- MODEL_NAME=MiniMax-M3-MXFP8 NODES=2 GPUS_PER_NODE=8 WIDE_EP_MODE=1 MORIIO_READ_MODE=0 RUN_AFTER_HEALTH=accuracy ROUTER_TYPE=vllm-router WAIT=1 bash .buildkite/amd-disagg/run-slurm-disagg-test.sh
- label: "GLM-5.2-FP8-PD-1P1D-EP8/DP8-WideEP-MoRIIO-proxy"
agents:
queue: amd_mi350_ainic
timeout_in_minutes: 180
concurrency: 2
concurrency_group: "amd-disagg/mi350/2node"
commands:
- MODEL_NAME=GLM-5.2-FP8 NODES=2 GPUS_PER_NODE=8 WIDE_EP_MODE=1 MORIIO_READ_MODE=0 RUN_AFTER_HEALTH=accuracy ROUTER_TYPE=proxy WAIT=1 bash .buildkite/amd-disagg/run-slurm-disagg-test.sh
- label: "GLM-5.2-FP8-PD-1P1D-EP8/DP8-WideEP-MoRIIO-vllm-router"
agents:
queue: amd_mi350_ainic
timeout_in_minutes: 180
concurrency: 2
concurrency_group: "amd-disagg/mi350/2node"
commands:
- MODEL_NAME=GLM-5.2-FP8 NODES=2 GPUS_PER_NODE=8 WIDE_EP_MODE=1 MORIIO_READ_MODE=0 RUN_AFTER_HEALTH=accuracy ROUTER_TYPE=vllm-router WAIT=1 bash .buildkite/amd-disagg/run-slurm-disagg-test.sh
@@ -1,338 +0,0 @@
#!/bin/bash
# =============================================================================
# run-slurm-disagg-test.sh — foreground (login-node) submitter for the disagg
# P/D gate. The single command the Buildkite step runs.
# -----------------------------------------------------------------------------
# Thin glue: submits run_xPyD_disagg.slurm. By default (WAIT=0, Spur-safe) it
# fire-and-forgets and returns 0 on a good submit; with WAIT=1 it polls the
# scheduler and exits with the job's exit code so a CI step pass/fails.
# stdout carries ONLY the SLURM job info
# (image/nodes line, job id, log path); after sbatch nothing else is written to
# stdout — the full job log persists at the --output path and pass/fail
# diagnostics go to stderr. The actual work lives in run_xPyD_disagg.slurm: it
# selects nodes,
# fans out one container per node via a single srun, and hands off to
# vllm_disagg.sh (rank-based prefill/decode self-select).
#
# Default target: 1P1D TP8 (NODES=2), nightly image, accuracy gate.
#
# Spur usage (fire-and-forget; the default here):
# bash run-slurm-disagg-test.sh # 1P1D TP8
# WIDE_EP_MODE=1 bash run-slurm-disagg-test.sh # 1P1D wide-EP
# NODES=4 WIDE_EP_MODE=1 xP=2 yD=2 bash run-slurm-disagg-test.sh # 2P2D EP
# Prints the job id + NFS log paths and returns immediately (does NOT poll
# squeue/sacct, which hang on Spur). Track the run via the printed `tail -f`.
#
# Classic-Slurm / CI usage (block until done, exit with the gate's pass/fail):
# WAIT=1 bash run-slurm-disagg-test.sh
# =============================================================================
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
JOB_SCRIPT="${JOB_SCRIPT:-${SCRIPT_DIR}/run_xPyD_disagg.slurm}"
# ---- knobs (override from the Buildkite step env) --------------------------
# Defaults tuned for the Spur AMD MI350X cluster
IMAGE="${IMAGE:-vllm/vllm-openai-rocm:nightly}"
NODES="${NODES:-2}"
GPUS_PER_NODE="${GPUS_PER_NODE:-8}"
PARTITION="${SLURM_PARTITION:-}"
TIME_LIMIT="${SLURM_TIME_LIMIT:-03:00:00}"
WIDE_EP_MODE="${WIDE_EP_MODE:-0}" # 0 -> 1P1D TP8 (default); 1 -> wide-EP
xP="${xP:-1}"
yD="${yD:-1}"
RUN_AFTER_HEALTH="${RUN_AFTER_HEALTH:-accuracy}"
HEALTH_TIMEOUT_S="${HEALTH_TIMEOUT_S:-3600}" # P/D bring-up budget; +900s grace must fit the CI step timeout
SHARED_MOUNT="${SHARED_MOUNT:-/data}"
LOG_ROOT="${LOG_ROOT:-${SHARED_MOUNT}/${USER:-$(whoami)}/disagg_logs}"
DRY_RUN="${DRY_RUN:-0}"
MORIIO_READ_MODE="${MORIIO_READ_MODE:-0}"
WAIT="${WAIT:-0}"
# ROUTER Type - defaults to vllm-router
ROUTER_TYPE="${ROUTER_TYPE:-vllm-router}"
ROUTER_PORT="${ROUTER_PORT:-30000}"
VLLM_ROUTER_IMAGE="${VLLM_ROUTER_IMAGE:-vllm/vllm-router:nightly}"
# Dry-run only validates wiring; cap its walltime low so it never holds the queue.
[[ "${DRY_RUN}" == "1" ]] && TIME_LIMIT="${SLURM_TIME_LIMIT:-00:10:00}"
mkdir -p "${LOG_ROOT}"
# Spur - sbatch scheduler.
DISAGG_SCRIPTS_STAGE="${DISAGG_SCRIPTS_STAGE:-/data/scratch/buildkite-agent}"
STAGED_DIR="${DISAGG_SCRIPTS_STAGE}/${BUILDKITE_COMMIT:-local}"
mkdir -p "${STAGED_DIR}"
cp -rL --no-preserve=ownership,timestamps "${SCRIPT_DIR}/." "${STAGED_DIR}/"
chmod -R u+rwX "${STAGED_DIR}" 2>/dev/null || true
export DISAGG_SCRIPTS_DIR="${STAGED_DIR}"
echo "[slurm-submit] staged scripts for compute nodes: ${DISAGG_SCRIPTS_DIR}" >&2
export IMAGE MODEL_NAME WIDE_EP_MODE xP yD GPUS_PER_NODE RUN_AFTER_HEALTH HEALTH_TIMEOUT_S
export SHARED_MOUNT LOG_ROOT DRY_RUN MORIIO_READ_MODE
export ROUTER_TYPE ROUTER_PORT VLLM_ROUTER_IMAGE
# Model selection.
[[ -n "${MODEL_NAME:-}" ]] && export MODEL_NAME
[[ -n "${MODEL_DIR:-}" ]] && export MODEL_DIR
[[ -n "${PARTITION}" ]] && echo "[slurm-submit] NOTE: PARTITION='${PARTITION}' ignored (Spur sbatch has no --partition)" >&2
SUBMIT_SCRIPT="${JOB_SCRIPT}"
FILE_NODES="$(grep -oE '^#SBATCH[[:space:]]+--nodes=[0-9]+' "${JOB_SCRIPT}" | grep -oE '[0-9]+' | head -n1)"
if [[ -n "${NODES}" && -n "${FILE_NODES}" && "${NODES}" != "${FILE_NODES}" ]]; then
SUBMIT_SCRIPT="/tmp/$(basename "${JOB_SCRIPT%.slurm}")-n${NODES}-$$.slurm"
sed -E "s/^#SBATCH([[:space:]]+)--nodes=[0-9]+/#SBATCH\\1--nodes=${NODES}/" "${JOB_SCRIPT}" > "${SUBMIT_SCRIPT}"
echo "[slurm-submit] node count ${FILE_NODES}->${NODES}: submitting patched copy ${SUBMIT_SCRIPT}" >&2
fi
# Job name comes from the script's #SBATCH --job-name (SLURM sets SLURM_JOB_NAME
JOB_NAME="$(grep -oE '^#SBATCH[[:space:]]+--job-name=[^[:space:]]+' "${JOB_SCRIPT}" | sed -E 's/.*--job-name=//' | head -n1)"
JOB_NAME="${JOB_NAME:-vllm-disagg-pd}"
echo "[slurm-submit] image=${IMAGE} nodes=${NODES} gpus/node=${GPUS_PER_NODE} mode=$([[ ${WIDE_EP_MODE} == 0 ]] && echo tp || echo ep) router=${ROUTER_TYPE} walltime=${TIME_LIMIT}"
SUBMIT_OUT="$(sbatch --time="${TIME_LIMIT}" "${SUBMIT_SCRIPT}")"
echo "${SUBMIT_OUT}"
# "Submitted batch job 114" -> 114 (last integer on the line).
JOB_ID="$(printf '%s\n' "${SUBMIT_OUT}" | grep -oE '[0-9]+' | tail -n1 || true)"
if [[ -z "${JOB_ID}" ]]; then
echo "[slurm-submit] ERROR: could not parse a job id from sbatch output above" >&2
exit 1
fi
echo "[slurm-submit] submitted job ${JOB_ID}"
# The real per-job log is written to NFS from inside the job body by
# run_xPyD_disagg.slurm (`exec > "$LOG_ROOT/${SLURM_JOB_NAME}-${SLURM_JOB_ID}.log"`).
# SLURM_JOB_NAME resolves to our --job-name, so we can compute the path here. The
# per-role server/proxy/bench logs land under ${LOG_ROOT}/${JOB_ID}/.
LOG_FILE="${LOG_ROOT}/${JOB_NAME}-${JOB_ID}.log"
LOG_DIR="${LOG_ROOT}/${JOB_ID}"
echo "[slurm-submit] job log: ${LOG_FILE}"
echo "[slurm-submit] role logs: ${LOG_DIR}/"
# --- Spur-safe by default: fire-and-forget --------------------------------------
if [[ "${WAIT}" != "1" ]]; then
echo "[slurm-submit] submitted (WAIT=0, not polling). Track with:" >&2
echo " tail -f ${LOG_FILE}" >&2
echo " grep -aE '(PASS|FAIL): |exact_match' ${LOG_FILE}" >&2
exit 0
fi
# --- WAIT=1: phase-aware poll of the NFS job log (scontrol-based) -------------
# Detection is phased so failures surface fast instead of waiting out the full
# walltime:
# 1) submitted -> running : scontrol catches infra/scheduler failures
# (NODE_FAIL/BOOT_FAIL/CANCELLED/TIMEOUT/...) within
# one poll, or a stuck-PENDING/never-started job.
# 2) running -> healthy : advance once every endpoint reports healthy (+ the
# vllm-router, when used); fail on bring-up errors or
# the health budget.
# 3) healthy -> verdict : PASS/FAIL from the accuracy gate, capped.
# scontrol is the scheduler authority on this cluster
echo "[slurm-submit] WAIT=1: phase-aware poll of ${LOG_FILE} (timeout ${TIME_LIMIT})" >&2
_h=0 _m=0 _s=0
IFS=: read -r _h _m _s <<< "${TIME_LIMIT}"
WAIT_DEADLINE=$(( $(date +%s) + 10#${_h}*3600 + 10#${_m}*60 + 10#${_s:-0} ))
unset -v _h _m _s
# Per-phase budgets (all overridable from the Buildkite step env).
POLL_INTERVAL="${POLL_INTERVAL:-20}"
SUBMIT_GRACE_S="${SUBMIT_GRACE_S:-900}" # reach RUNNING within 15m
PENDING_MAX_S="${PENDING_MAX_S:-1800}" # tolerate 30m queued
HEALTH_PHASE_TIMEOUT_S="${HEALTH_PHASE_TIMEOUT_S:-$(( HEALTH_TIMEOUT_S + 900 ))}"
WORKLOAD_TIMEOUT_S="${WORKLOAD_TIMEOUT_S:-1800}" # accuracy/bench cap
COMPLETED_GRACE_S="${COMPLETED_GRACE_S:-60}" # verdict lag after job exit
SENTINEL="${LOG_DIR}/.disagg_done"
# scontrol field extractor (authoritative here; timeout-guarded so a momentary
# scheduler stall can't wedge the poll). Returns the value or "".
job_field() { # $1=jobid $2=field -> value | ""
timeout 15 scontrol show job "$1" 2>/dev/null \
| grep -oE "$2=[^ ]+" | head -n1 | cut -d= -f2- || true
}
have() { grep -aqE "$1" "${LOG_FILE}" 2>/dev/null; }
# The job's own verdict sources, in authority order: the .disagg_done sentinel
# (rank-0 rc, written by a single writer) and then the gate line in the log.
# Sets STATE/RC/REASON and returns 0 when a verdict exists, 1 when it does not.
read_verdict() {
if [[ -f "${SENTINEL}" ]]; then
RC="$(tr -dc '0-9' < "${SENTINEL}" 2>/dev/null || true)"; RC="${RC:-1}"
if [[ "${RC}" == "0" ]]; then STATE="COMPLETED"; else STATE="FAILED"; fi
REASON="sentinel"; return 0
fi
if have '(PASS|FAIL): '; then
if have 'FAIL: '; then STATE="FAILED"; RC=1; else STATE="COMPLETED"; RC=0; fi
REASON="gate"; return 0
fi
return 1
}
# Never let the job outlive this poller. Cancelling a Buildkite build kills the
# agent's bootstrap, and without this the sbatch job keeps its whole allocation
# until the walltime expires. Only armed on the WAIT=1 path — under WAIT=0,
# leaving the job running is the point.
CANCEL_GRACE_S="${CANCEL_GRACE_S:-120}"
_CLEANED=0
# shellcheck disable=SC2329 # invoked from cleanup_job, which the traps below call
job_active() {
case "$(job_field "${JOB_ID}" JobState)" in
RUNNING|PENDING|COMPLETING|CONFIGURING|SUSPENDED|REQUEUED) return 0 ;;
*) return 1 ;;
esac
}
# shellcheck disable=SC2329 # invoked from the traps below
cleanup_job() {
[[ "${_CLEANED}" == "1" || -z "${JOB_ID:-}" ]] && return 0
_CLEANED=1
# A job that reported its own verdict is mid-teardown; give it a bounded
# moment to finish. Timeouts and infra failures get cancelled immediately —
# there is nothing to wait for and the nodes should come back now.
if [[ "${REASON:-}" == "sentinel" || "${REASON:-}" == "gate" || "${STATE:-}" == "COMPLETED" ]]; then
local deadline=$(( $(date +%s) + CANCEL_GRACE_S ))
while (( $(date +%s) < deadline )) && job_active; do sleep 5; done
fi
if job_active; then
echo "[slurm-submit] cleanup: cancelling job ${JOB_ID}" >&2
scancel "${JOB_ID}" 2>/dev/null \
|| echo "[slurm-submit] WARN: scancel ${JOB_ID} failed; job may still hold nodes" >&2
fi
return 0
}
trap cleanup_job EXIT
trap 'cleanup_job; exit 130' INT
trap 'cleanup_job; exit 143' TERM
trap 'cleanup_job; exit 129' HUP
STATE=""
RC=1
REASON=""
PHASE="submitted"
T_PHASE=$(date +%s)
while [[ $(date +%s) -lt ${WAIT_DEADLINE} ]]; do
NOW=$(date +%s)
# (1) Ultimate authority: terminal sentinel (holds rank-0 rc), then the
# explicit accuracy gate line. Honored regardless of phase.
read_verdict && break
# (2) Scheduler state via scontrol: drives phase transitions and catches
# infra/scheduler failures fast.
ST="$(job_field "${JOB_ID}" JobState)"
case "${ST}" in
RUNNING|COMPLETING)
if [[ "${PHASE}" == "submitted" ]]; then PHASE="bringup"; T_PHASE="${NOW}"; fi
;;
NODE_FAIL|BOOT_FAIL|CANCELLED|TIMEOUT|OUT_OF_MEMORY|DEADLINE|PREEMPTED)
STATE="infra-${ST}"; RC=1
REASON="scontrol Reason=$(job_field "${JOB_ID}" Reason)"; break
;;
FAILED)
# Ambiguous (infra crash vs a legit accuracy exit=1). A gate/sentinel
# line would have won above, so classify by the phase we reached.
case "${PHASE}" in
submitted) STATE="infra-FAILED" ;;
bringup) STATE="server-failed" ;;
*) STATE="workload-failed" ;;
esac
RC=1; REASON="scontrol JobState=FAILED phase=${PHASE}"; break
;;
COMPLETED)
_vd=$(( NOW + COMPLETED_GRACE_S ))
_found=0
while :; do
if read_verdict; then _found=1; break; fi
(( $(date +%s) >= _vd )) && break
sleep 5
done
if (( _found == 1 )); then
echo "[slurm-submit] verdict arrived after JobState=COMPLETED (${REASON})" >&2
else
STATE="completed-no-verdict"; RC=1
REASON="COMPLETED with no sentinel/gate within ${COMPLETED_GRACE_S}s (a walltime kill reports COMPLETED here)"
fi
break
;;
PENDING|CONFIGURING|RESV_DEL_HOLD|REQUEUED)
if (( NOW - T_PHASE > PENDING_MAX_S )); then
STATE="infra-stuck-${ST}"; RC=1; REASON="queued > ${PENDING_MAX_S}s"; break
fi
;;
"")
# scontrol lost the job (purged past MinJobAge) with no log verdict:
# rely on the terminal markers above + the per-phase deadlines below.
:
;;
esac
# (3) Log-driven phase progress + per-phase deadlines. Fast-fails hangs where
# the job is still RUNNING (so scontrol won't help) but bring-up/workload
# is stuck.
case "${PHASE}" in
submitted)
# Job body appearing in the log is a second "it started" signal for
# when scontrol is briefly empty.
if have 'Selected node IPs|health-gate:|\[disagg-pd\]'; then
PHASE="bringup"; T_PHASE="${NOW}"
elif [[ -z "${ST}" ]] && (( NOW - T_PHASE > SUBMIT_GRACE_S )); then
# scontrol can't confirm the job exists (empty state) AND no log
# output within the grace window -> treat as a lost/failed launch.
# NB: a genuinely-queued job reports PENDING via scontrol above and
# is governed by PENDING_MAX_S (default 30m), not this grace window.
STATE="infra-nostart"; RC=1; REASON="no scheduler state or log within ${SUBMIT_GRACE_S}s"; break
fi
;;
bringup)
if have 'FAIL:|TIMEOUT waiting for|exited while waiting|Traceback \(most recent'; then
STATE="server-failed"; RC=1; REASON="bring-up failure in log"; break
fi
NEED="$(grep -aoE 'waiting on [0-9]+' "${LOG_FILE}" 2>/dev/null | grep -oE '[0-9]+' | tail -n1 || true)"
GOT="$(grep -acE '\] healthy: ' "${LOG_FILE}" 2>/dev/null || true)"; GOT="${GOT:-0}"
ROUTER_OK=1
if [[ "${ROUTER_TYPE}" == "vllm-router" ]]; then
if ! have 'vllm-router healthy on'; then ROUTER_OK=0; fi
fi
if [[ -n "${NEED}" ]] && (( GOT >= NEED )) && (( ROUTER_OK == 1 )); then
PHASE="workload"; T_PHASE="${NOW}"
elif (( NOW - T_PHASE > HEALTH_PHASE_TIMEOUT_S )); then
STATE="bringup-timeout"; RC=1; REASON="no healthy within ${HEALTH_PHASE_TIMEOUT_S}s"; break
fi
;;
workload)
# PASS/FAIL handled at the top; only enforce the cap here.
if (( NOW - T_PHASE > WORKLOAD_TIMEOUT_S )); then
STATE="workload-timeout"; RC=1; REASON="no verdict within ${WORKLOAD_TIMEOUT_S}s"; break
fi
;;
esac
sleep "${POLL_INTERVAL}"
done
if [[ -z "${STATE}" ]]; then
echo "[slurm-submit] WARN: no verdict before ${TIME_LIMIT}; failing" >&2
STATE="deadline"; RC=1; REASON="poll deadline"
fi
# Pre-flight logic.
PF_LINES=()
if compgen -G "${LOG_DIR}/preflight_NODE*.log" >/dev/null 2>&1; then
cat -- "${LOG_DIR}"/preflight_NODE*.log >&2 || true
mapfile -t PF_LINES < <(grep -h '^PREFLIGHT-REJECTED: ' "${LOG_DIR}"/preflight_NODE*.log 2>/dev/null || true)
fi
if (( ${#PF_LINES[@]} > 0 )) && (( RC != 0 )); then
STATE="preflight-rejected"
REASON="${PF_LINES[0]#PREFLIGHT-REJECTED: }"
if (( ${#PF_LINES[@]} > 1 )); then
REASON="${REASON} [+$(( ${#PF_LINES[@]} - 1 )) more node(s), see ${LOG_DIR}/]"
fi
fi
# Surface the accuracy gate verdict (if any) from the job log — to stderr.
GATE_LINE=$(grep -aE '(PASS|FAIL): ' "${LOG_FILE}" 2>/dev/null | tail -n1 || true)
[[ -n "${GATE_LINE}" ]] && echo "[slurm-submit] gate: ${GATE_LINE}" >&2
echo "[slurm-submit] job ${JOB_ID} finished: state=${STATE:-unknown} phase=${PHASE} exit=${RC} reason=${REASON:-}" >&2
exit "${RC}"
-535
View File
@@ -1,535 +0,0 @@
#!/bin/bash
#SBATCH --job-name=vllm-disagg-pd
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=1
#SBATCH --gres=gpu:8
#SBATCH --exclusive
#SBATCH --chdir=/tmp
#SBATCH --output=/tmp/spur-%j.out
#SBATCH --error=/tmp/spur-%j.err
set -euo pipefail
# ---- inputs (env > default; forwarded by run-slurm-disagg-test.sh via --export)
IMAGE="${IMAGE:-vllm/vllm-openai-rocm:nightly}"
WIDE_EP_MODE="${WIDE_EP_MODE:-0}"
xP="${xP:-1}"
yD="${yD:-1}"
GPUS_PER_NODE="${GPUS_PER_NODE:-8}"
SHARED_MOUNT="${SHARED_MOUNT:-/data}"
LOG_ROOT="${LOG_ROOT:-/data/${USER:-$(id -un)}/disagg_logs}"
RUN_AFTER_HEALTH="${RUN_AFTER_HEALTH:-accuracy}" # bench | accuracy
DRY_RUN="${DRY_RUN:-0}"
MORIIO_READ_MODE="${MORIIO_READ_MODE:-0}" # 0 -> prefill writes (default); 1 -> decode reads KV (read_mode)
# ---- router / gateway (host-side; also forwarded into the container) --------
# vllm-router -> external `vllm/vllm-router` container started on the rank-0 node (default)
# proxy -> in-container MoRIIO proxy (no extra container)
ROUTER_TYPE="${ROUTER_TYPE:-vllm-router}"
ROUTER_PORT="${ROUTER_PORT:-30000}"
ROUTER_POLICY="${ROUTER_POLICY:-round_robin}"
VLLM_ROUTER_IMAGE="${VLLM_ROUTER_IMAGE:-vllm/vllm-router:nightly}"
PROXY_PING_PORT="${PROXY_PING_PORT:-36367}" # must equal vllm-router --vllm-discovery-address port
PROXY_PORT="${PROXY_PORT:-10001}"
# Intra-node data-parallel size the router uses to route the MoRIIO KV-notify to
# the right DP rank. Required for wideEP DP (else decode hangs: "remote blocks
# never arrived"). EP -> GPUS_PER_NODE (matches --data-parallel-size-local); TP -> 1.
if [[ "${WIDE_EP_MODE}" == "1" ]]; then
ROUTER_DP_LOCAL="${ROUTER_DP_LOCAL:-${GPUS_PER_NODE}}"
else
ROUTER_DP_LOCAL="${ROUTER_DP_LOCAL:-1}"
fi
if [[ "${ROUTER_TYPE}" == "vllm-router" ]]; then
GATEWAY_PORT="${GATEWAY_PORT:-${ROUTER_PORT}}"
else
GATEWAY_PORT="${GATEWAY_PORT:-${PROXY_PORT}}"
fi
# This line is a recommended template from infra side.
exec > "$LOG_ROOT/${SLURM_JOB_NAME}-${SLURM_JOB_ID}.log" 2>&1
# Where the disagg scripts live ON THE HOST (bind-mounted into the container).
# Spur runs the job body from /tmp, so $(pwd)/SLURM_SUBMIT_DIR aren't reliable;
# default to the known host scripts dir. Override via env if you relocate it.
DISAGG_SCRIPTS_DIR="${DISAGG_SCRIPTS_DIR:-${SLURM_SUBMIT_DIR:-$(pwd)}}"
echo "DISAGG SCRIPTS DIR is ----"
echo "$DISAGG_SCRIPTS_DIR"
# In-container path the scripts are mounted to (and sourced from).
CONTAINER_SCRIPTS="${CONTAINER_SCRIPTS:-/vllm-workspace/.buildkite/amd-disagg}"
CLUSTER_ENV_IN="${CLUSTER_ENV_IN:-${CONTAINER_SCRIPTS}/cluster.sh}"
# Health endpoints (TP-mode ports from cluster.sh defaults).
PREFILL_PORT="${PREFILL_PORT:-8100}"
DECODE_PORT="${DECODE_PORT:-8200}"
HEALTH_TIMEOUT_S="${HEALTH_TIMEOUT_S:-3600}"
# EP-mode ports, mirrored from cluster.sh purely so the preflight below can test
# them host-side; the in-container values still come from cluster.sh.
SERVE_PORT="${SERVE_PORT:-20005}"
RPC_PORT="${RPC_PORT:-13345}"
KV_PORT="${KV_PORT:-9711}"
LOCAL_PING_PORT="${LOCAL_PING_PORT:-61555}"
# vllm-router binds this for Prometheus in addition to ROUTER_PORT, and panics
# at startup if it is taken.
ROUTER_METRICS_PORT="${ROUTER_METRICS_PORT:-29000}"
# Node preflight (see preflight_node): 0 disables, VRAM% above which a GPU counts
# as busy, and how long to let a reaped container's VRAM drain before judging.
PREFLIGHT="${PREFLIGHT:-1}"
PREFLIGHT_MAX_VRAM_PCT="${PREFLIGHT_MAX_VRAM_PCT:-10}"
PREFLIGHT_GPU_SETTLE_S="${PREFLIGHT_GPU_SETTLE_S:-30}"
NUM_NODES=$((xP + yD))
echo "Calculated NUM_NODES: $NUM_NODES (xP=$xP + yD=$yD) WIDE_EP_MODE=$WIDE_EP_MODE"
# ------------------------
# Extract NUM_NODES from SLURM allocation and select nodes
# ------------------------
echo "Original SLURM allocation:"
echo "SLURM_JOB_NODELIST: $SLURM_JOB_NODELIST"
echo "SLURM_NNODES: $SLURM_NNODES"
echo "SLURM_NTASKS: $SLURM_NTASKS"
# scontrol is available on classic Slurm but not in the Spur job body. Use it to
# expand compressed ranges when present, else split the (already comma-separated,
# full-hostname) SLURM_JOB_NODELIST on commas.
if command -v scontrol >/dev/null 2>&1; then
FULL_NODELIST=$(scontrol show hostnames "$SLURM_JOB_NODELIST")
else
FULL_NODELIST=$(echo "$SLURM_JOB_NODELIST" | tr ',' '\n')
fi
AVAILABLE_COUNT=$(echo "$FULL_NODELIST" | wc -l)
if (( AVAILABLE_COUNT < NUM_NODES )); then
echo "ERROR: Need ${NUM_NODES} nodes (xP=${xP} + yD=${yD}) but only ${AVAILABLE_COUNT} allocated." >&2
exit 1
fi
# Pick the first NUM_NODES nodes and order them to match how IPs get assigned:
# * classic Slurm: sort alphabetically -- srun assigns PROCID=0 to the
# alphabetically-first hostname, so IPS[] (built per node below) lines up.
# * Spur: preserve RANK order (SPUR_NODELIST), so the hostnames pair 1:1 with
# the rank-ordered IPs in SPUR_PEER_NODES; an alphabetical sort here would
# desync MASTER_NODE / MASTER_ADDR / IPADDRS from the actual ranks.
# IPADDRS[0..xP-1] = prefill, IPADDRS[xP..] = decode, so this order must match.
if command -v srun >/dev/null 2>&1; then
SELECTED_NODES=$(echo "$FULL_NODELIST" | head -n "$NUM_NODES" | sort)
elif [[ -n "${SPUR_NODELIST:-}" ]]; then
SELECTED_NODES=$(echo "${SPUR_NODELIST}" | tr ',' '\n' | sed '/^$/d' | head -n "$NUM_NODES")
else
SELECTED_NODES=$(echo "$FULL_NODELIST" | head -n "$NUM_NODES")
fi
SELECTED_NODELIST_STR=$(echo "$SELECTED_NODES" | tr '\n' ',' | sed 's/,$//')
# Create new nodelist in SLURM format
# This is a simplified approach - for complex ranges, you might need more sophisticated parsing
NEW_SLURM_NODELIST=$(echo "$SELECTED_NODES" | paste -sd, | sed 's/,/,/g')
# Update SLURM environment variables
export SLURM_NNODES=$NUM_NODES
export SLURM_NTASKS=$NUM_NODES
export SLURM_JOB_NUM_NODES=$NUM_NODES
export SLURM_NPROCS=$NUM_NODES
export SLURM_JOB_NODELIST="$NEW_SLURM_NODELIST"
export SLURM_NODELIST="$NEW_SLURM_NODELIST"
# Keep other SLURM variables as they were or set defaults
export SLURM_TASKS_PER_NODE="1(x$NUM_NODES)"
export SLURM_SUBMIT_DIR="${SLURM_SUBMIT_DIR:-${HOME}}"
export SLURM_CLUSTER_NAME="${SLURM_CLUSTER_NAME:-vllm-di-ci}"
export SLURM_JOB_CPUS_PER_NODE="${SLURM_JOB_CPUS_PER_NODE:-1}"
export SLURM_JOB_PARTITION="${SLURM_JOB_PARTITION:-amd-rccl}"
export SLURM_JOBID="${SLURM_JOBID:-$SLURM_JOB_ID}"
export SLURM_JOB_QOS="${SLURM_JOB_QOS:-normal}"
export SLURM_JOB_ACCOUNT="${SLURM_JOB_ACCOUNT:-amd-rccl}"
export SLURM_NTASKS_PER_NODE=1
export SLURM_SUBMIT_HOST="${SLURM_SUBMIT_HOST:-$(hostname)}"
export SLURM_JOB_ID="${SLURM_JOB_ID:-}"
export SLURM_CONF="${SLURM_CONF:-/etc/slurm/slurm.conf}"
export SLURM_JOB_NAME="${SLURM_JOB_NAME:-${xP}p${yD}d_bench-serving}"
echo ""
echo "Updated SLURM Environment Variables:"
echo "SLURM_JOB_ID: $SLURM_JOB_ID"
echo "SLURM_JOB_NODELIST: $SLURM_JOB_NODELIST"
echo "SLURM_NNODES: $SLURM_NNODES"
echo "SLURM_NTASKS: $SLURM_NTASKS"
echo "SLURM_TASKS_PER_NODE: $SLURM_TASKS_PER_NODE"
echo "SLURM_JOB_CPUS_PER_NODE: $SLURM_JOB_CPUS_PER_NODE"
echo "SLURM_JOB_PARTITION: $SLURM_JOB_PARTITION"
echo "SLURM_JOB_NUM_NODES: $SLURM_JOB_NUM_NODES"
echo "SLURM_JOBID: $SLURM_JOBID"
echo "SLURM_JOB_QOS: $SLURM_JOB_QOS"
echo "SLURM_NODELIST: $SLURM_NODELIST"
echo "SLURM_JOB_ACCOUNT: $SLURM_JOB_ACCOUNT"
echo "SLURM_NPROCS: $SLURM_NPROCS"
echo "SLURM_SUBMIT_HOST: $SLURM_SUBMIT_HOST"
echo "SLURM_CONF: $SLURM_CONF"
echo "SLURM_JOB_NAME: $SLURM_JOB_NAME"
echo "SLURM_NTASKS_PER_NODE: $SLURM_NTASKS_PER_NODE"
echo "SLURM_SUBMIT_DIR: $SLURM_SUBMIT_DIR"
echo "SLURM_CLUSTER_NAME: $SLURM_CLUSTER_NAME"
echo "ulimit: $(ulimit -a)"
echo ""
echo "Selected nodes for execution:"
echo "$SELECTED_NODES"
echo ""
# Node information
MASTER_NODE=$(echo "$SELECTED_NODES" | head -n 1)
MASTER_PORT=39566 # Choose an open port
IPS=()
if command -v srun >/dev/null 2>&1; then
# Classic Slurm: resolve each node's IP via srun (original behavior).
for NODE in $SELECTED_NODES; do
IP=$(srun --nodes=1 --ntasks=1 --time=00:20:00 --nodelist="$NODE" bash -c 'hostname -I' | awk 'NR==1 {print $1}')
IPS+=("$IP")
done
else
# Spur job body (no srun): use the rank-ordered IPs Spur provides in
# SPUR_PEER_NODES ("ip:port,ip:port,..."). These are the cross-node-routable
# bootstrap addresses; hostnames resolve to public IPs and the local node to
# 127.0.1.1, so DNS is not usable here.
mapfile -t IPS < <(echo "${SPUR_PEER_NODES:-}" | tr ',' '\n' | cut -d: -f1 | sed '/^$/d')
if (( ${#IPS[@]} == 0 )); then
echo "ERROR: no srun and SPUR_PEER_NODES is empty; cannot determine node IPs." >&2
exit 1
fi
fi
MASTER_ADDR="${IPS[0]}"
IPADDRS="${IPS[*]}"
IPADDRS="${IPADDRS// /,}"
echo "Selected node IPs: ${IPADDRS}"
NNODES=$NUM_NODES
echo "MASTER_NODE is ${MASTER_NODE}"
echo "MASTER_ADDR is ${MASTER_ADDR}"
echo "MASTER_PORT is ${MASTER_PORT}"
echo "NNODES is ${NNODES}"
# Per-job log folder (MAD-style): all per-role server/proxy/bench logs
export LOG_PATH="${LOG_ROOT}/${SLURM_JOB_ID}"
mkdir -p "$LOG_PATH"
echo "[disagg-pd] LOG_PATH=$LOG_PATH"
# ----------------------------------------------------------------------------
# Launch one container per node via a single srun and hand off to
# vllm_disagg.sh node (rank-based self-select).
# ----------------------------------------------------------------------------
# IPADDRS is the ordered node IP list (prefill IPs first, then decode). IPS[]
# was built in SELECTED_NODES (alphabetical) order — the same order srun assigns
# SLURM_PROCID — so the rank->role mapping lines up:
# rank 0 -> prefill master + MoRIIO proxy + orchestrator
# rank 1 .. xP-1 -> prefill
# rank xP .. end -> decode
export IPADDRS
export WIDE_EP_MODE xP yD GPUS_PER_NODE RUN_AFTER_HEALTH HEALTH_TIMEOUT_S DRY_RUN MORIIO_READ_MODE
export IMAGE SHARED_MOUNT CONTAINER_SCRIPTS DISAGG_SCRIPTS_DIR
export CLUSTER_ENV="${CLUSTER_ENV_IN}"
export DOCKER_CONT_NAME="vllm_disagg_${MODEL_NAME:-model}_${SLURM_JOB_ID}"
# Router env: forwarded into the main container (cluster.sh honors these) and used
# host-side to launch the external router container on rank 0.
export ROUTER_TYPE ROUTER_PORT ROUTER_POLICY VLLM_ROUTER_IMAGE PROXY_PING_PORT GATEWAY_PORT ROUTER_DP_LOCAL
export ROUTER_CONT_NAME="vllm_router_${SLURM_JOB_ID}"
# Consumed by preflight_node on each node (srun forwards the environment).
export PREFILL_PORT DECODE_PORT PROXY_PORT ROUTER_METRICS_PORT
export SERVE_PORT RPC_PORT KV_PORT LOCAL_PING_PORT
export PREFLIGHT PREFLIGHT_MAX_VRAM_PCT PREFLIGHT_GPU_SETTLE_S
echo "[disagg-pd] IPADDRS=${IPADDRS}"
echo "[disagg-pd] launching ${NUM_NODES} containers (1/node) -> vllm_disagg.sh node"
# Refuse a node that is not actually free. $1 = this node's global rank.
#
# `#SBATCH --exclusive` only keeps out other SLURM jobs. A plain `docker run` on
# a node is invisible to the scheduler, so we get allocated nodes whose GPUs and
# ports are already in use. Without this the job pulls the image, loads the model
# and dies minutes later inside vLLM ("Free memory on device cuda:N ... is less
# than desired GPU memory utilization"); worse, on rank 0 a foreign router
# already listening on ROUTER_PORT can satisfy the health gate and take the
# accuracy traffic. Called after the stale-container reap, with a settle window
# so VRAM the reap is still releasing is not counted against us.
preflight_node() {
local rank="$1"
if [ "${PREFLIGHT:-1}" != "1" ]; then
echo "Rank $rank preflight: skipped (PREFLIGHT=0)"
return 0
fi
local host max_pct settle problems=""
local -a issues=() # one short tag per fault, for the one-line summary
host="$(hostname)"
max_pct="${PREFLIGHT_MAX_VRAM_PCT:-10}"
settle="${PREFLIGHT_GPU_SETTLE_S:-30}"
# GPUs: vLLM asks for a fraction of TOTAL, so memory already resident is
# subtracted from what it can get and pushes it under its own threshold.
if command -v rocm-smi >/dev/null 2>&1; then
local csv busy nbusy ncards deadline
deadline=$(( $(date +%s) + settle ))
while :; do
csv="$(rocm-smi --showmeminfo vram --csv 2>/dev/null || true)"
busy="$(printf '%s\n' "$csv" |
awk -F, -v m="$max_pct" '/^card/ && $2+0 > 0 {
pct = $3 * 100 / $2
if (pct > m) printf "%s=%d%%(%.0fGiB) ", $1, pct, $3/1073741824
}' || true)"
if [ -z "$busy" ]; then break; fi
if [ "$(date +%s)" -ge "$deadline" ]; then break; fi
sleep 5
done
if [ -n "$busy" ]; then
nbusy="$(printf '%s' "$busy" | tr ' ' '\n' | grep -c 'card' || true)"
ncards="$(printf '%s\n' "$csv" | grep -c '^card' || true)"
issues+=("${nbusy}/${ncards} GPUs hold VRAM above ${max_pct}%")
problems="${problems} GPU VRAM in use above ${max_pct}%: ${busy}"$'\n'
fi
else
echo "Rank $rank preflight: no rocm-smi on $host, skipping GPU check"
fi
# Ports this rank will bind. Every container here is --network host, so a
# foreign listener either collides or silently serves our traffic.
local -a want=()
if [ "${WIDE_EP_MODE:-0}" = "1" ]; then
want+=("$SERVE_PORT" "$RPC_PORT" "$KV_PORT" "$LOCAL_PING_PORT")
elif [ "$rank" -lt "${xP:-1}" ]; then
want+=("$PREFILL_PORT")
else
want+=("$DECODE_PORT")
fi
if [ "$rank" = "0" ]; then
want+=("$PROXY_PING_PORT")
if [ "$ROUTER_TYPE" = "vllm-router" ]; then
want+=("$ROUTER_PORT" "$ROUTER_METRICS_PORT")
else
want+=("$PROXY_PORT")
fi
fi
if command -v ss >/dev/null 2>&1; then
local bound taken="" p
bound="$(ss -lnt 2>/dev/null | awk '{ n = split($4, a, ":"); print a[n] }' | sort -u || true)"
for p in "${want[@]}"; do
if printf '%s\n' "$bound" | grep -qx -- "$p"; then
taken="${taken}${p} "
fi
done
if [ -n "$taken" ]; then
issues+=("ports already bound: ${taken% }")
problems="${problems} ports already bound: ${taken}"$'\n'
fi
else
echo "Rank $rank preflight: no ss on $host, skipping port check"
fi
if [ -n "$problems" ]; then
mkdir -p "$LOG_PATH" 2>/dev/null || true
local summary
summary="$(printf '%s; ' "${issues[@]}")"
summary="${summary%; }"
# Every node writes the shared job log concurrently, so this diagnostic can
# be overwritten there. Keep a single-writer copy per node as well, or the
# job fails in seconds with no surviving explanation. The submitter reads
# the PREFLIGHT-REJECTED line back out of it to use as the failure reason,
# so keep that prefix in sync with run-slurm-disagg-test.sh. It must not
# contain "FAIL:" — that is the accuracy gate's pattern.
{
echo "PREFLIGHT-REJECTED: rank ${rank} on ${host}: ${summary}"
printf '%s' "$problems"
echo " containers here: $(docker ps --format '{{.Names}} ({{.Image}})' 2>/dev/null | paste -sd'; ' - || true)"
echo " --exclusive does not exclude non-SLURM workloads; clear the node or exclude it from the allocation."
echo " Set PREFLIGHT=0 to submit anyway."
} | tee -a "$LOG_PATH/preflight_NODE${rank}.log" 2>/dev/null || true
# Unblock the other ranks: they park on this sentinel, and the submitter
# polls it. Without it a peer waits out HEALTH_TIMEOUT_S for an endpoint
# that is never coming.
echo 1 > "$LOG_PATH/.disagg_done" 2>/dev/null || true
return 1
fi
echo "Rank $rank preflight ok on $host: GPUs idle, ports free (${want[*]})"
return 0
}
# Per-node container launch, auto-dispatched by srun availability:
# Launch this node's disagg container and hand off to vllm_disagg.sh. $1 = this
# node's global rank. Relies on the env exported above (forwarded by srun on
# classic Slurm; already in scope when called directly on Spur).
run_node_container() {
local rank="$1"
set -o pipefail # propagate docker run exit code through the tee pipe
echo "Rank $rank on $(hostname)"
local -a stale=()
mapfile -t stale < <(docker ps -aq --filter name=vllm_disagg_ --filter name=vllm_router_ 2>/dev/null || true)
if (( ${#stale[@]} > 0 )); then
echo "Reaping stale containers on $(hostname): ${stale[*]}"
docker stop -t 30 "${stale[@]}" >/dev/null 2>&1 || true
docker rm "${stale[@]}" >/dev/null 2>&1 || true
fi
preflight_node "$rank" || return 1
# Bind-mount the host AMD Pensando (ionic) RoCE userspace provider into the
# container IF present: the image ships an ABI-incompatible copy and the NICs
# are invisible without the host libs. Each entry is added only when the host
# file exists, so this is a no-op on non-ionic (e.g. mlx5) clusters.
local IONIC_MOUNTS=()
local host_ionic
host_ionic=$(readlink -f /usr/lib/x86_64-linux-gnu/libionic.so.1 2>/dev/null || true)
[ -n "$host_ionic" ] && IONIC_MOUNTS+=(-v "$host_ionic:/usr/lib/x86_64-linux-gnu/libionic.so.1:ro")
[ -f /usr/lib/x86_64-linux-gnu/libibverbs/libionic-rdmav34.so ] && \
IONIC_MOUNTS+=(-v "/usr/lib/x86_64-linux-gnu/libibverbs/libionic-rdmav34.so:/usr/lib/x86_64-linux-gnu/libibverbs/libionic-rdmav34.so:ro")
[ -f /etc/libibverbs.d/ionic.driver ] && \
IONIC_MOUNTS+=(-v "/etc/libibverbs.d/ionic.driver:/etc/libibverbs.d/ionic.driver:ro")
echo "Rank $rank ionic mounts: ${IONIC_MOUNTS[*]:-<none>}"
# Pull latest container.
if [ "$DRY_RUN" != "1" ]; then
echo "Rank $rank pulling image on $(hostname): $IMAGE"
docker pull "$IMAGE" || echo "WARN: docker pull failed on $(hostname) for $IMAGE (using local copy if present)"
if [ "$ROUTER_TYPE" = "vllm-router" ] && [ "$rank" = "0" ]; then
echo "Rank $rank pulling router image on $(hostname): $VLLM_ROUTER_IMAGE"
docker pull "$VLLM_ROUTER_IMAGE" || echo "WARN: docker pull failed on $(hostname) for $VLLM_ROUTER_IMAGE (using local copy if present)"
fi
fi
# Start the external vLLM router as a SEPARATE container on the rank-0 node.
if [ "$ROUTER_TYPE" = "vllm-router" ] && [ "$rank" = "0" ] && [ "$DRY_RUN" = "1" ]; then
echo "DRY_RUN: would start vllm-router on $(hostname): $VLLM_ROUTER_IMAGE (discovery :$PROXY_PING_PORT, port :$ROUTER_PORT, policy $ROUTER_POLICY, dp_local=$ROUTER_DP_LOCAL)"
elif [ "$ROUTER_TYPE" = "vllm-router" ] && [ "$rank" = "0" ]; then
mkdir -p "$LOG_PATH" 2>/dev/null || true
{
echo "Starting vllm-router on $(hostname): $VLLM_ROUTER_IMAGE (discovery :$PROXY_PING_PORT, port :$ROUTER_PORT, policy $ROUTER_POLICY, dp_local=$ROUTER_DP_LOCAL)"
docker run -d \
--name "$ROUTER_CONT_NAME" \
--network host \
--user "$(id -u):$(id -g)" \
-v "$LOG_PATH":/run_logs \
-v "$SHARED_MOUNT":"$SHARED_MOUNT" \
-v "/data/$(id -un):/workspace" \
-e HOME=/workspace \
-e USER="$(id -un)" \
-e LOGNAME="$(id -un)" \
"$VLLM_ROUTER_IMAGE" \
bash -lc "exec vllm-router --vllm-pd-disaggregation --kv-connector moriio --vllm-discovery-address 0.0.0.0:$PROXY_PING_PORT --port $ROUTER_PORT --host 0.0.0.0 --intra-node-data-parallel-size $ROUTER_DP_LOCAL --policy $ROUTER_POLICY --prefill-policy $ROUTER_POLICY --decode-policy $ROUTER_POLICY --log-level info >> $LOG_PATH/vllm_router.log 2>&1" \
&& echo "vllm-router container started ok" \
|| echo "WARN: failed to start vllm-router container on $(hostname)"
} 2>&1 | tee -a "$LOG_PATH/vllm_router_start.log"
fi
docker run --rm \
--user "$(id -u):$(id -g)" \
--device /dev/dri \
--device /dev/kfd \
--device /dev/infiniband \
--network host \
--ipc host \
--group-add "$(getent group video | cut -d: -f3)" \
--group-add "$(getent group render | cut -d: -f3)" \
--cap-add SYS_PTRACE \
--cap-add IPC_LOCK \
--security-opt seccomp=unconfined \
--shm-size 64G \
--ulimit nofile=1048576:1048576 \
-v "$SHARED_MOUNT":"$SHARED_MOUNT" \
-v "$DISAGG_SCRIPTS_DIR":"$CONTAINER_SCRIPTS" \
-v "/data/$(id -un):/workspace" \
"${IONIC_MOUNTS[@]}" \
--ulimit memlock=-1:-1 \
-e HF_TOKEN \
-e HOME=/workspace \
-e USER="$(id -un)" \
-e LOGNAME="$(id -un)" \
-e TRITON_CACHE_DIR=/tmp/triton_cache \
-e TORCHINDUCTOR_CACHE_DIR=/tmp/inductor_cache \
-e VLLM_CACHE_ROOT=/tmp/vllm_cache \
-e AITER_JIT_DIR=/tmp/aiter_jit \
-e FLYDSL_RUNTIME_CACHE_DIR=/tmp/flydsl_cache \
-e SLURM_JOB_ID="$SLURM_JOB_ID" \
-e SLURM_JOB_NODELIST="$SLURM_JOB_NODELIST" \
-e NODE_RANK="$rank" \
-e IPADDRS="$IPADDRS" \
-e WIDE_EP_MODE="$WIDE_EP_MODE" \
-e xP="$xP" -e yD="$yD" \
-e GPUS_PER_NODE="$GPUS_PER_NODE" \
-e CLUSTER_ENV="$CLUSTER_ENV" \
-e MODEL_NAME="${MODEL_NAME:-}" \
-e MODEL_DIR="${MODEL_DIR:-}" \
-e LOG_PATH="$LOG_PATH" \
-e RUN_AFTER_HEALTH="$RUN_AFTER_HEALTH" \
-e HEALTH_TIMEOUT_S="$HEALTH_TIMEOUT_S" \
-e DRY_RUN="$DRY_RUN" \
-e MORIIO_READ_MODE="$MORIIO_READ_MODE" \
-e ROUTER_TYPE="$ROUTER_TYPE" \
-e ROUTER_PORT="$ROUTER_PORT" \
-e ROUTER_POLICY="$ROUTER_POLICY" \
-e GATEWAY_PORT="$GATEWAY_PORT" \
--name "$DOCKER_CONT_NAME" \
--entrypoint /bin/bash \
"$IMAGE" -c "bash $CONTAINER_SCRIPTS/vllm_disagg.sh node" \
2>&1 | tee "$LOG_PATH/main_NODE${rank}.log"
local node_rc=${PIPESTATUS[0]} # docker run exit code (not tee), for rank-0 pass/fail
# Stop the external router (rank-0 node) once the main container returns.
if [ "$ROUTER_TYPE" = "vllm-router" ] && [ "$rank" = "0" ]; then
docker stop "$ROUTER_CONT_NAME" 2>/dev/null || true; docker rm "$ROUTER_CONT_NAME" 2>/dev/null || true;
fi
return "$node_rc"
}
# Teardown of the main + external router containers on every node ($1 = docker
# stop timeout). `docker run` containers outlive the client that started them,
# so on scancel the srun tasks die but the containers keep the GPUs until the
# walltime expires unless they are stopped explicitly.
teardown_containers() {
export TEARDOWN_STOP_T="${1:-30}"
if command -v srun >/dev/null 2>&1; then
# shellcheck disable=SC2016 # must expand on the remote node, not here
srun --nodelist="$SELECTED_NODELIST_STR" \
--nodes="$NUM_NODES" --ntasks="$NUM_NODES" --ntasks-per-node=1 \
bash -c 'docker stop -t "${TEARDOWN_STOP_T:-30}" "$DOCKER_CONT_NAME" >/dev/null 2>&1 || true; docker rm "$DOCKER_CONT_NAME" >/dev/null 2>&1 || true
[ "$ROUTER_TYPE" = "vllm-router" ] && { docker stop -t "${TEARDOWN_STOP_T:-30}" "$ROUTER_CONT_NAME" >/dev/null 2>&1 || true; docker rm "$ROUTER_CONT_NAME" >/dev/null 2>&1 || true; }' || true
else
docker stop -t "$TEARDOWN_STOP_T" "$DOCKER_CONT_NAME" >/dev/null 2>&1 || true
docker rm "$DOCKER_CONT_NAME" >/dev/null 2>&1 || true
if [ "$ROUTER_TYPE" = "vllm-router" ]; then
docker stop -t "$TEARDOWN_STOP_T" "$ROUTER_CONT_NAME" >/dev/null 2>&1 || true
docker rm "$ROUTER_CONT_NAME" >/dev/null 2>&1 || true
fi
fi
return 0
}
# SLURM sends SIGTERM then SIGKILL after KillWait (30s by default), so the trap
# uses a short stop timeout to fit inside that window.
trap 'echo "[disagg-pd] signal received, tearing down containers"; teardown_containers 10; exit 143' TERM INT
RC=0
if command -v srun >/dev/null 2>&1; then
# classic Slurm: fan out one task/node; ship the fn and call it with the
# srun-supplied per-task SLURM_PROCID (escaped so the remote shell expands it).
srun --label \
--nodelist="$SELECTED_NODELIST_STR" \
--nodes="$NUM_NODES" --ntasks="$NUM_NODES" --ntasks-per-node=1 \
bash -c "$(declare -f preflight_node run_node_container); run_node_container \"\$SLURM_PROCID\"" || RC=$?
else
# Spur: the body already runs once per node; this node's rank is
# SPUR_TASK_OFFSET (fall back to NODE_RANK, then 0 for a single-node run).
run_node_container "${SPUR_TASK_OFFSET:-${NODE_RANK:-0}}" || RC=$?
fi
teardown_containers 30
echo "[disagg-pd] launch finished rc=${RC}; logs in ${LOG_PATH}"
exit "${RC}"
-635
View File
@@ -1,635 +0,0 @@
#!/bin/bash
# =============================================================================
# vLLM disaggregated (P/D) launcher (rank-based / SLURM-native).
# -----------------------------------------------------------------------------
#
# rank 0 .. xP-1 -> prefill (rank 0 is also the orchestrator + proxy)
# rank xP .. xP+yD-1 -> decode
#
# Rank 0 additionally: starts the MoRIIO proxy, health-gates every server,
# runs the post-health workload (RUN_AFTER_HEALTH=bench|accuracy|none), then
# drops a shared-FS completion sentinel so the other ranks shut down and the
# single `srun` returns. Rank 0's exit code is the run's pass/fail.
#
# Explicit per-role invocation is also supported (granular for manual use):
# proxy | prefill | decode | bench | accuracy.
#
# ---- LAUNCH IT (both supported) --------------------------------------------
# 1) Single srun (SLURM provides the rank) — what run_xPyD_disagg.slurm does:
# srun --nodes=$((xP+yD)) --ntasks-per-node=1 \
# bash .../vllm_disagg.sh node
# (IPADDRS must be ordered to match ranks: prefill IPs first, then decode,
# in the SAME order srun assigns PROCID — sort nodes alphabetically.)
#
# 2) Manual, one shell per node (rank from NODE_RANK; share LOG_PATH/FS):
# IPADDRS=ipP0,ipD0 xP=1 yD=1 NODE_RANK=0 bash vllm_disagg.sh node
# IPADDRS=ipP0,ipD0 xP=1 yD=1 NODE_RANK=1 bash vllm_disagg.sh node
#
# 3) Manual, fully granular (per-role):
# bash vllm_disagg.sh proxy
# bash vllm_disagg.sh prefill
# NODE_RANK=1 bash vllm_disagg.sh decode
# bash vllm_disagg.sh bench
#
# Cluster config: cluster.sh (sourced). Model flags: models.yaml.
# Parallelism: WIDE_EP_MODE=0 tp (independent TP servers) | 1 ep (DP+EP groups).
# =============================================================================
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
_PROXY_IP_OVERRIDE="${PROXY_IP:-}"
log() { echo "[vllm_disagg] $*"; }
die() { echo "ERROR: $*" >&2; exit 1; }
usage() {
echo "Usage: $0 [node|proxy|prefill|decode|bench|accuracy] [--wide-ep-mode 0|1]" >&2
echo " (no role => 'node': derive role from \$SLURM_PROCID/\$NODE_RANK)" >&2
exit 1
}
# ----------------------------------------------------------------- arg parsing
# Sets globals: ROLE (default 'node'), _WIDE_EP_MODE_OVERRIDE.
parse_args() {
ROLE="${1:-node}"
shift || true
_WIDE_EP_MODE_OVERRIDE=""
while [[ $# -gt 0 ]]; do
case "$1" in
--wide-ep-mode) _WIDE_EP_MODE_OVERRIDE="${2:-}"; shift 2 ;;
--wide-ep-mode=*) _WIDE_EP_MODE_OVERRIDE="${1#*=}"; shift ;;
-h|--help) usage ;;
*) die "Unknown argument: $1" ;;
esac
done
}
# ----------------------------------------------------------------- load config
# Sources cluster.sh and resolves WIDE_EP_MODE -> PARALLEL_MODE (tp|ep).
load_config() {
CLUSTER_ENV="${CLUSTER_ENV:-${SCRIPT_DIR}/cluster.sh}"
[[ -f "${CLUSTER_ENV}" ]] || die "cluster env file not found: ${CLUSTER_ENV}"
# shellcheck disable=SC1090
source "${CLUSTER_ENV}"
# Topology fanout normally arrives from cluster.sh; restate the defaults so
# the rank arithmetic is defined even when it does not.
: "${xP:=1}" "${yD:=1}"
[[ -n "${_WIDE_EP_MODE_OVERRIDE}" ]] && WIDE_EP_MODE="${_WIDE_EP_MODE_OVERRIDE}"
WIDE_EP_MODE="${WIDE_EP_MODE:-0}"
case "${WIDE_EP_MODE}" in
0) PARALLEL_MODE="tp" ;;
1) PARALLEL_MODE="ep" ;;
*) die "WIDE_EP_MODE must be 0 (tp) or 1 (ep); got '${WIDE_EP_MODE}'" ;;
esac
# node-mode orchestration knobs
RUN_AFTER_HEALTH="${RUN_AFTER_HEALTH:-accuracy}" # bench | accuracy | none
HEALTH_TIMEOUT_S="${HEALTH_TIMEOUT_S:-3600}"
# MoRIIO KV transfer direction. 0 (default)
MORIIO_READ_MODE="${MORIIO_READ_MODE:-0}"
}
# Emits the `read_mode` KV-config fragment (leading comma + newline) when read
# mode is enabled, else nothing (leaving MoRIIO's write-mode default).
moriio_read_mode_kv() {
case "${MORIIO_READ_MODE:-0}" in
1|true|True|TRUE|yes|on) printf ',\n "read_mode": true' ;;
*) : ;;
esac
}
# ----------------------------------------------------------------- topology
# Resolves MODEL_PATH, MODELS_YAML, the IP array, and the master/proxy addresses.
# IP_ARRAY[0..xP-1] = prefill nodes, [xP..xP+yD-1] = decode nodes.
resolve_topology() {
MODEL_PATH="${MODEL_DIR%/}/${MODEL_NAME}"
MODELS_YAML="${MODELS_YAML:-${SCRIPT_DIR}/models.yaml}"
mkdir -p "${LOG_PATH}" 2>/dev/null || true
[[ -z "${IPADDRS}" ]] && IPADDRS="${PREFILL_IP},${DECODE_IP}" # 1P1D fallback
IFS=',' read -ra IP_ARRAY <<< "${IPADDRS}"
PREFILL_MASTER_ADDR="${IP_ARRAY[0]}"
DECODE_MASTER_ADDR="${IP_ARRAY[$xP]:-${IP_ARRAY[-1]}}"
# Proxy is co-located on the prefill master (rank 0) unless the CALLER
# explicitly set PROXY_IP. We use the pre-cluster.sh override (captured at the
# top) — not the live PROXY_IP, which cluster.sh always populates with the
# static PREFILL_IP fallback and would otherwise mask the real master.
PROXY_IP="${_PROXY_IP_OVERRIDE:-${PREFILL_MASTER_ADDR}}"
}
# ============================================================================
# Non-server roles
# ============================================================================
run_proxy() {
[[ -f "${PROXY_SCRIPT}" ]] || die "proxy script not found: ${PROXY_SCRIPT}"
log "proxy: http=:${PROXY_PORT} discovery=:${PROXY_PING_PORT} (${PROXY_SCRIPT})"
exec python3 "${PROXY_SCRIPT}" --port "${PROXY_PORT}"
}
# Start the proxy in the BACKGROUND (node-mode rank 0). Sets PROXY_PID.
start_proxy_bg() {
[[ -f "${PROXY_SCRIPT}" ]] || die "proxy script not found: ${PROXY_SCRIPT}"
local plog="${LOG_PATH}/proxy_$(date +%Y%m%d_%H%M%S).log"
log "proxy(bg): http=:${PROXY_PORT} discovery=:${PROXY_PING_PORT} log=${plog}"
python3 "${PROXY_SCRIPT}" --port "${PROXY_PORT}" >"${plog}" 2>&1 &
PROXY_PID=$!
}
run_bench() {
local base_url="http://127.0.0.1:${GATEWAY_PORT:-${PROXY_PORT}}"
local result_dir="${LOG_PATH}/bench_$(date +%Y%m%d_%H%M%S)"
mkdir -p "${result_dir}"
log "bench -> ${base_url} model=${MODEL_PATH} wide_ep_mode=${WIDE_EP_MODE}(${PARALLEL_MODE})"
log "combinations='${BENCHMARK_COMBINATIONS}' concurrency='${BENCHMARK_CON}'"
local combo isl osl con nump logf
for combo in ${BENCHMARK_COMBINATIONS}; do
isl="${combo%%/*}"; osl="${combo##*/}"
for con in ${BENCHMARK_CON}; do
nump=$(( NUM_PROMPTS_FACTOR * con ))
(( nump < BENCHMARK_MIN_PROMPTS )) && nump="${BENCHMARK_MIN_PROMPTS}"
logf="${result_dir}/bench_${isl}_${osl}_con${con}.log"
log "ISL=${isl} OSL=${osl} CON=${con} num_prompts=${nump} -> ${logf}"
vllm bench serve \
--backend openai \
--base-url "${base_url}" \
--endpoint /v1/completions \
--model "${MODEL_PATH}" \
--dataset-name random \
--random-input-len "${isl}" \
--random-output-len "${osl}" \
--num-prompts "${nump}" \
--max-concurrency "${con}" \
--percentile-metrics ttft,tpot,itl,e2el \
--ignore-eos \
2>&1 | tee "${logf}"
done
done
log "bench complete. Results in ${result_dir}"
}
run_accuracy() {
# lm_eval isn't in the stock vLLM image; install on demand (rank 0 only, and
# only when accuracy is actually selected). Override with ACCURACY_PIP_SPEC.
if ! command -v lm_eval >/dev/null 2>&1; then
log "lm_eval not found — installing ${ACCURACY_PIP_SPEC:-lm_eval[api]}"
python3 -m pip install --no-cache-dir "${ACCURACY_PIP_SPEC:-lm_eval[api]}"
fi
local base_url="http://127.0.0.1:${GATEWAY_PORT:-${PROXY_PORT}}/v1/completions"
local ts; ts="$(date +%Y%m%d_%H%M%S)"
local logf="${LOG_PATH}/accuracy_${MODEL_NAME}_${PARALLEL_MODE}_${ts}.log"
local outdir="${LOG_PATH}/lm_eval_${MODEL_NAME}_${PARALLEL_MODE}_${ts}"
log "accuracy -> ${base_url} model=${MODEL_PATH} tasks=${ACCURACY_TASKS} wide_ep_mode=${WIDE_EP_MODE}(${PARALLEL_MODE})"
log "log=${logf} results=${outdir}"
# --output_path makes lm_eval persist a results_*.json for scraping.
local eval_rc=0
( set -o pipefail
# Model weights are already served (loaded offline in a separate process);
# allow the Hub only here so lm_eval can fetch the eval dataset (e.g. gsm8k).
export HF_HUB_OFFLINE=0 HF_DATASETS_OFFLINE=0
python3 -m lm_eval --model local-completions \
--tasks "${ACCURACY_TASKS}" \
--model_args "model=${MODEL_PATH},base_url=${base_url},num_concurrent=${ACCURACY_NUM_CONCURRENT},max_retries=${ACCURACY_MAX_RETRIES},tokenized_requests=False,trust_remote_code=True,timeout=${ACCURACY_TIMEOUT:-3600}" \
--output_path "${outdir}" \
2>&1 | tee "${logf}"
) || eval_rc=$?
if (( eval_rc != 0 )); then
log "FAIL: lm_eval exited rc=${eval_rc} (no usable score) — Log: ${logf}"
return "${eval_rc}"
fi
# Gate on the score. Find the results JSON lm_eval just wrote, pull the max
# value of ACCURACY_METRIC across filters/tasks, and compare to the threshold.
local results_json=""
results_json="$(find "${outdir}" -name 'results_*.json' -type f 2>/dev/null | sort | tail -n1)"
if [[ -z "${results_json}" ]]; then
log "FAIL: no results_*.json under ${outdir} — cannot evaluate threshold"
return 1
fi
local score=""
score="$(python3 - "${results_json}" "${ACCURACY_METRIC}" <<'PY'
import json, sys
path, metric = sys.argv[1], sys.argv[2]
with open(path) as f:
data = json.load(f)
best = None
for task, metrics in (data.get("results") or {}).items():
for key, val in metrics.items():
if isinstance(val, bool) or not isinstance(val, (int, float)):
continue
if key.split(",")[0] == metric:
best = val if best is None else max(best, val)
print("" if best is None else "%.6f" % best)
PY
)" || score=""
if [[ -z "${score}" ]]; then
log "FAIL: could not parse metric '${ACCURACY_METRIC}' from ${results_json}"
return 1
fi
if awk -v s="${score}" -v t="${ACCURACY_THRESHOLD}" 'BEGIN{exit !(s+0 >= t+0)}'; then
log "PASS: ${ACCURACY_METRIC}=${score} >= threshold=${ACCURACY_THRESHOLD} (tasks=${ACCURACY_TASKS}). Log: ${logf}"
return 0
fi
log "FAIL: ${ACCURACY_METRIC}=${score} < threshold=${ACCURACY_THRESHOLD} (tasks=${ACCURACY_TASKS}). Log: ${logf}"
return 1
}
# ============================================================================
# Server roles (prefill / decode) and their shared helpers
# ============================================================================
# Read model-specific flags from models.yaml (mode + role aware) into globals:
# MODEL_BASE_FLAGS / MODEL_ROLE_FLAGS / MODEL_EXPERIMENTAL_FLAGS -> MODEL_CONFIG
# Also exports the model's env: block (only if not already set; caller env wins).
load_model_flags() {
[[ -f "${MODELS_YAML}" ]] || die "models.yaml not found: ${MODELS_YAML}"
export MODELS_YAML MODEL_NAME PARALLEL_MODE ROLE
eval "$(python3 - <<'PY'
import os, shlex, sys, yaml
path = os.environ["MODELS_YAML"]; name = os.environ["MODEL_NAME"]
mode = os.environ["PARALLEL_MODE"]; role = os.environ["ROLE"]
with open(path, "r", encoding="utf-8") as f:
doc = yaml.safe_load(f) or {}
# Preferred form: top-level `models:` list of {model: <name>, ...} entries.
# Back-compat: a top-level mapping keyed by model name.
entries = doc.get("models") if isinstance(doc, dict) else doc
cfg = None
if isinstance(entries, list):
cfg = next((e for e in entries if isinstance(e, dict) and e.get("model") == name), None)
elif isinstance(doc, dict):
cfg = doc.get(name)
if cfg is None:
print(f'echo "ERROR: model {name} not found in {path}" >&2; exit 1'); sys.exit(0)
role_cfg = cfg.get(role, {}) or {}
def q(v): return shlex.quote(str(v if v is not None else ""))
exports = {
"MODEL_BASE_FLAGS": cfg.get("base_flags", "") or "",
"MODEL_ROLE_FLAGS": role_cfg.get(mode, "") or "",
"MODEL_EXPERIMENTAL_FLAGS": cfg.get("experimental_flags", "") or "",
}
for k, v in exports.items(): print(f"{k}={q(v)}")
# Model-specific env: exported only if not already set (caller env wins).
for k, v in (cfg.get("env", {}) or {}).items():
if k not in os.environ:
print(f"export {k}={q(v)}")
PY
)"
MODEL_CONFIG="${MODEL_BASE_FLAGS} ${MODEL_ROLE_FLAGS} ${MODEL_EXPERIMENTAL_FLAGS}"
}
# Build the TP-mode (WIDE_EP_MODE=0) serve command into the global CMD array.
# Each node is an independent TP server registered to the proxy.
build_tp_cmd() {
local port="${TP_PORT}"
[[ -n "${HOST_IP}" ]] && CMD+=(--host "${HOST_IP}")
CMD+=(--port "${port}" --tensor-parallel-size "${TP_SIZE}")
local _mc; read -ra _mc <<< "${MODEL_CONFIG}"; CMD+=("${_mc[@]}")
local kv_cfg read_mode_kv
read_mode_kv="$(moriio_read_mode_kv)"
kv_cfg=$(cat <<EOF
{
"kv_connector": "MoRIIOConnector",
"kv_role": "${KV_ROLE}",
"kv_connector_extra_config": {
"proxy_ip": "${PROXY_IP}",
"proxy_ping_port": "${PROXY_PING_PORT}",
"http_port": "${port}",
"handshake_port": "${HANDSHAKE_PORT}",
"notify_port": "${NOTIFY_PORT}"${read_mode_kv}
}
}
EOF
)
CMD+=(--kv-transfer-config "${kv_cfg}")
log "role=${ROLE} mode=tp host=${HOST_IP:-0.0.0.0}:${port} kv_role=${KV_ROLE} tp=${TP_SIZE} read_mode=${MORIIO_READ_MODE}"
}
# Build the EP-mode (WIDE_EP_MODE=1) serve command into the global CMD array.
# DP + expert parallel; masters expose the API server + KV transfer, children
# join headless.
build_ep_cmd() {
# prefill -> mori_high_throughput (paired with --enforce-eager in models.yaml)
# decode -> mori_low_latency (paired with CUDA graphs in models.yaml)
local default_backend="mori_low_latency"
[[ "${ROLE}" == "prefill" ]] && default_backend="mori_high_throughput"
local backend="${ALL2ALL_BACKEND:-${default_backend}}"
export VLLM_ALL2ALL_BACKEND="${VLLM_ALL2ALL_BACKEND:-${backend}}"
local port="${SERVE_PORT}"
CMD+=(
-tp 1
--data-parallel-size "${DP_GROUP_SIZE}"
--data-parallel-size-local "${GPUS_PER_NODE}"
--data-parallel-address "${DP_MASTER_ADDR}"
--data-parallel-rpc-port "${RPC_PORT}"
--enable-expert-parallel
--all2all-backend "${backend}"
--port "${port}"
--no-enable-prefix-caching
--distributed-timeout-seconds "${DISTRIBUTED_TIMEOUT_SECONDS}"
)
local _mc; read -ra _mc <<< "${MODEL_CONFIG}"; CMD+=("${_mc[@]}")
if [[ "${IS_MASTER}" == "1" ]]; then
CMD+=(--api-server-count="${GPUS_PER_NODE}")
local kv_cfg read_mode_kv
read_mode_kv="$(moriio_read_mode_kv)"
kv_cfg=$(cat <<EOF
{
"kv_connector": "MoRIIOConnector",
"kv_role": "${KV_ROLE}",
"kv_port": "${KV_PORT}",
"kv_connector_extra_config": {
"proxy_ip": "${PROXY_IP}",
"proxy_port": "${PROXY_PORT}",
"proxy_ping_port": "${PROXY_PING_PORT}",
"http_port": "${port}",
"local_ping_port": "${LOCAL_PING_PORT}",
"handshake_port": "${HANDSHAKE_PORT}",
"notify_port": "${NOTIFY_PORT}"${read_mode_kv}
}
}
EOF
)
CMD+=(--kv-transfer-config "${kv_cfg}")
log "role=${ROLE} mode=ep MASTER node_rank=${NODE_RANK} dp_size=${DP_GROUP_SIZE} dp_addr=${DP_MASTER_ADDR} kv_role=${KV_ROLE} all2all=${backend} read_mode=${MORIIO_READ_MODE}"
else
CMD+=(--data-parallel-start-rank "${DP_START_RANK}" --headless)
log "role=${ROLE} mode=ep CHILD node_rank=${NODE_RANK} dp_size=${DP_GROUP_SIZE} dp_start_rank=${DP_START_RANK} dp_addr=${DP_MASTER_ADDR} all2all=${backend}"
fi
}
# Set the prefill topology globals for the current NODE_RANK (no execution).
configure_prefill() {
: "${NODE_RANK:=0}" # manual default: prefill master
KV_ROLE="kv_producer"
DP_GROUP_SIZE=$(( xP * GPUS_PER_NODE ))
DP_MASTER_ADDR="${PREFILL_MASTER_ADDR}"
DP_START_RANK=$(( NODE_RANK * GPUS_PER_NODE ))
[[ "${NODE_RANK}" -eq 0 ]] && IS_MASTER=1 || IS_MASTER=0
TP_PORT="${PREFILL_PORT}"
}
# Set the decode topology globals for the current NODE_RANK (no execution).
configure_decode() {
: "${NODE_RANK:=${xP}}" # manual default: decode master
KV_ROLE="kv_consumer"
DP_GROUP_SIZE=$(( yD * GPUS_PER_NODE ))
DP_MASTER_ADDR="${DECODE_MASTER_ADDR}"
DP_START_RANK=$(( (NODE_RANK - xP) * GPUS_PER_NODE ))
[[ "${NODE_RANK}" -eq "${xP}" ]] && IS_MASTER=1 || IS_MASTER=0
TP_PORT="${DECODE_PORT}"
}
# Build the serve command (model flags + CMD array + LOGF). No exec.
# Expects role globals set by configure_prefill/configure_decode.
build_server_cmd() {
HOST_IP="${IP_ARRAY[$NODE_RANK]:-}" # own IP for --host binding, if known
load_model_flags
LOGF="${LOG_PATH}/${ROLE}_${PARALLEL_MODE}_node${NODE_RANK}_$(date +%Y%m%d_%H%M%S).log"
CMD=(vllm serve "${MODEL_PATH}")
if [[ "${WIDE_EP_MODE}" == "0" ]]; then build_tp_cmd; else build_ep_cmd; fi
log "model=${MODEL_PATH}"
}
# Foreground server run (explicit prefill/decode roles — original behavior).
run_server_fg() {
build_server_cmd
log "log=${LOGF}"
"${CMD[@]}" 2>&1 | tee "${LOGF}"
}
run_prefill() { configure_prefill; run_server_fg; }
run_decode() { configure_decode; run_server_fg; }
# ============================================================================
# node mode — SLURM-native rank-based self-select + rank-0 orchestration
# ============================================================================
# Endpoints to health-check, one per line as "ip:port", depending on mode.
# tp: every prefill node (:PREFILL_PORT) and every decode node (:DECODE_PORT)
# ep: only the masters expose an API (prefill rank 0, decode rank xP) :SERVE_PORT
health_endpoints() {
local i
if [[ "${WIDE_EP_MODE}" == "0" ]]; then
for (( i=0; i<xP; i++ )); do echo "${IP_ARRAY[$i]}:${PREFILL_PORT}"; done
for (( i=xP; i<xP+yD; i++ )); do echo "${IP_ARRAY[$i]}:${DECODE_PORT}"; done
else
echo "${IP_ARRAY[0]}:${SERVE_PORT}"
echo "${IP_ARRAY[$xP]}:${SERVE_PORT}"
fi
}
# Block until every endpoint's /health is OK, or timeout, or our local server dies.
wait_all_healthy() {
local deadline=$(( $(date +%s) + HEALTH_TIMEOUT_S ))
local eps; mapfile -t eps < <(health_endpoints)
log "health-gate: waiting on ${#eps[@]} endpoint(s): ${eps[*]}"
local ep
for ep in "${eps[@]}"; do
until /usr/bin/curl -sf "http://${ep}/health" >/dev/null 2>&1; do
(( $(date +%s) >= deadline )) && { log "TIMEOUT waiting for ${ep}"; return 1; }
kill -0 "${SERVER_PID}" 2>/dev/null || { log "local server (pid ${SERVER_PID}) exited while waiting"; return 1; }
sleep 10
done
log "healthy: ${ep}"
done
return 0
}
run_workload() {
case "${RUN_AFTER_HEALTH}" in
bench) run_bench ;;
accuracy) run_accuracy ;;
none|"") log "RUN_AFTER_HEALTH=none — skipping workload" ;;
*) die "unknown RUN_AFTER_HEALTH='${RUN_AFTER_HEALTH}'" ;;
esac
}
# rank 0: proxy + health-gate + workload, then write the completion sentinel.
orchestrate_master() {
local sentinel="$1" rc=0
# Front door: the proxy runs in-container (started here); the vllm-router
# runs as a SEPARATE container started by the SLURM job on this (rank-0) node,
# so in that mode we don't start anything here.
PROXY_PID=""
if [[ "${ROUTER_TYPE:-vllm-router}" == "vllm-router" ]]; then
log "ROUTER_TYPE=vllm-router: external router expected on gateway :${GATEWAY_PORT:-${ROUTER_PORT}} (not starting proxy)"
else
start_proxy_bg
fi
if wait_all_healthy; then
# If using the external vllm-router, wait for it to become ready before
# running the workload. The router starts as a separate container on this
# (rank-0) node; give it up to 300s to bind port ROUTER_PORT.
if [[ "${ROUTER_TYPE:-vllm-router}" == "vllm-router" ]]; then
local router_deadline=$(( $(date +%s) + 300 ))
log "waiting for vllm-router on :${ROUTER_PORT} (up to 300s)"
until /usr/bin/curl -sf "http://127.0.0.1:${ROUTER_PORT}/health" >/dev/null 2>&1; do
if (( $(date +%s) >= router_deadline )); then
log "FAIL: vllm-router not ready on :${ROUTER_PORT} after 300s (bring-up)"
rc=1
break
fi
sleep 5
done
(( rc == 0 )) && log "vllm-router healthy on :${ROUTER_PORT}"
fi
if (( rc == 0 )); then
set +e
run_workload
rc=$?
set -e
fi
else
log "FAIL: health-gate did not pass — server bring-up failed"
rc=1
fi
echo "${rc}" > "${sentinel}" 2>/dev/null || true
log "master: workload rc=${rc}; sentinel=${sentinel}; tearing down local proxy+server"
[[ -n "${PROXY_PID:-}" ]] && kill "${PROXY_PID}" 2>/dev/null || true
kill "${SERVER_PID}" 2>/dev/null || true
wait "${SERVER_PID}" 2>/dev/null || true
return "${rc}"
}
# non-master ranks: keep serving until the master signals completion (sentinel)
# or our local server dies unexpectedly.
watch_until_done() {
local sentinel="$1" rc=0
log "rank ${NODE_RANK} serving; waiting for completion sentinel (${sentinel})"
while :; do
[[ -f "${sentinel}" ]] && { log "completion sentinel seen; shutting down"; break; }
if ! kill -0 "${SERVER_PID}" 2>/dev/null; then
log "local server (pid ${SERVER_PID}) exited before completion"; rc=1; break
fi
sleep 10
done
kill "${SERVER_PID}" 2>/dev/null || true
wait "${SERVER_PID}" 2>/dev/null || true
return "${rc}"
}
run_node() {
NODE_RANK="${NODE_RANK:-${SLURM_PROCID:-0}}"
local total=$(( xP + yD ))
(( NODE_RANK >= 0 && NODE_RANK < total )) \
|| die "NODE_RANK=${NODE_RANK} out of range [0,${total}) (xP=${xP} yD=${yD})"
if (( NODE_RANK < xP )); then ROLE="prefill"; configure_prefill
else ROLE="decode"; configure_decode
fi
# Shared-FS completion sentinel (LOG_PATH is shared & per-run/per-job).
local sentinel="${LOG_PATH}/.disagg_done"
log "node mode: rank=${NODE_RANK}/${total} role=${ROLE} master=${IS_MASTER} wide_ep=${WIDE_EP_MODE}(${PARALLEL_MODE})"
log "topology: xP=${xP} yD=${yD} gpus/node=${GPUS_PER_NODE} IPADDRS=${IPADDRS}"
# DRY_RUN: resolve+print the plan (role, serve cmd, endpoints) and exit 0.
# Validates wiring without launching vLLM / the proxy / any workload.
if [[ "${DRY_RUN:-0}" == "1" ]]; then
build_server_cmd
log "DRY_RUN: server cmd: ${CMD[*]}"
if (( NODE_RANK == 0 )); then
local eps; mapfile -t eps < <(health_endpoints)
log "DRY_RUN: health endpoints: ${eps[*]}"
log "DRY_RUN: post-health workload: ${RUN_AFTER_HEALTH}"
log "DRY_RUN: sentinel: ${sentinel}"
fi
log "DRY_RUN complete (rank ${NODE_RANK}) — no processes launched"
exit 0
fi
# aiter and flydsl keep runtime JIT/kernel caches inside the aiter install
# tree (…/aiter/jit and …/aiter/jit/flydsl_cache), which is read-only for the
# non-root (--user) container. AITER_JIT_DIR / FLYDSL_RUNTIME_CACHE_DIR
# redirect those to node-local /tmp (both are sanctioned overrides: aiter only
# points flydsl at its bundled cache when FLYDSL_RUNTIME_CACHE_DIR is unset).
# Seed each once from the image's bundled copy so we keep the prebuilt
# modules/kernels (fast, no NFS, no from-source rebuild / MLIR recompile).
local _aiter_src
_aiter_src="$(python3 -c 'import os,aiter; print(os.path.join(os.path.dirname(aiter.__file__),"jit"))' 2>/dev/null || true)"
if [[ -n "${AITER_JIT_DIR:-}" && ! -e "${AITER_JIT_DIR}/.seeded" ]]; then
if [[ -n "${_aiter_src}" && -d "${_aiter_src}" ]]; then
mkdir -p "${AITER_JIT_DIR}"
cp -a "${_aiter_src}/." "${AITER_JIT_DIR}/" 2>/dev/null || true
touch "${AITER_JIT_DIR}/.seeded" 2>/dev/null || true
log "seeded AITER_JIT_DIR=${AITER_JIT_DIR} from ${_aiter_src}"
else
log "WARN: could not locate aiter jit dir to seed AITER_JIT_DIR=${AITER_JIT_DIR}; aiter may rebuild from source"
fi
fi
if [[ -n "${FLYDSL_RUNTIME_CACHE_DIR:-}" && ! -e "${FLYDSL_RUNTIME_CACHE_DIR}/.seeded" ]]; then
mkdir -p "${FLYDSL_RUNTIME_CACHE_DIR}"
if [[ -n "${_aiter_src}" && -d "${_aiter_src}/flydsl_cache" ]]; then
cp -a "${_aiter_src}/flydsl_cache/." "${FLYDSL_RUNTIME_CACHE_DIR}/" 2>/dev/null || true
log "seeded FLYDSL_RUNTIME_CACHE_DIR=${FLYDSL_RUNTIME_CACHE_DIR} from ${_aiter_src}/flydsl_cache"
else
log "FLYDSL_RUNTIME_CACHE_DIR=${FLYDSL_RUNTIME_CACHE_DIR} set; no bundled flydsl_cache to seed (will JIT-compile at runtime)"
fi
touch "${FLYDSL_RUNTIME_CACHE_DIR}/.seeded" 2>/dev/null || true
fi
# mori JIT-compiles its shmem/all2all kernels to ~/.mori/jit/<arch>_<nic>/ on
# first EP use. HOME here is /workspace, bind-mounted to persistent+shared NFS
# (/data/$USER), so a wrong-arch cache from an earlier run/build survives and
# gets reloaded even after MORI_GPU_ARCHS changes -- an existing cache dir
# wins over the arch setting. Reloading a gfx942 shmem_kernels.hsaco on gfx950
# hardware fails with "device kernel image is invalid" and every EP worker
# dies at init (surfacing async as a torch.tensor HIP error). Guard: on EP
# runs, if the cache has no build for our target arch, purge it so mori
# recompiles for MORI_GPU_ARCHS. A matching arch dir is kept (fast reload).
if [[ "${WIDE_EP_MODE}" == "1" && -n "${MORI_GPU_ARCHS:-}" ]]; then
local _mori_jit="${HOME:-/workspace}/.mori/jit"
if [[ -d "${_mori_jit}" ]] && ! compgen -G "${_mori_jit}/${MORI_GPU_ARCHS}_*" >/dev/null 2>&1; then
log "purging stale mori jit cache ${_mori_jit} (no ${MORI_GPU_ARCHS}_* build present; forcing recompile for ${MORI_GPU_ARCHS})"
rm -rf "${_mori_jit}" 2>/dev/null || true
fi
fi
# Start this node's server in the background (full server log -> LOGF; this
# shell's stdout stays free for orchestration logs the srun/CI captures).
build_server_cmd
"${CMD[@]}" >"${LOGF}" 2>&1 &
SERVER_PID=$!
log "server started pid=${SERVER_PID} log=${LOGF}"
local rc=0
if (( NODE_RANK == 0 )); then
orchestrate_master "${sentinel}" || rc=$?
# Canonical, human-readable end-of-run verdict for the login-node poller
# and CI logs (the .disagg_done sentinel remains the machine authority).
log "VERDICT: $( (( rc == 0 )) && echo PASS || echo FAIL) rc=${rc}"
else
watch_until_done "${sentinel}" || rc=$?
fi
log "node rank=${NODE_RANK} exiting rc=${rc}"
exit "${rc}"
}
# ============================================================================
main() {
parse_args "$@"
load_config
resolve_topology
case "${ROLE}" in
node) run_node ;;
proxy) run_proxy ;;
bench) run_bench ;;
accuracy) run_accuracy ;;
prefill) run_prefill ;;
decode) run_decode ;;
*) usage ;;
esac
}
main "$@"
-104
View File
@@ -1,104 +0,0 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
"""Audit vLLM compiled libraries for PyTorch stable ABI compliance."""
import fnmatch
import sys
from pathlib import Path
from torch_abi_audit import inspect_package
from torch_abi_audit.report import ExtensionReport, PackageReport
# Temporary allowlist of extensions not yet on the stable ABI.
# Shrink and remove over time.
ALLOWED_UNSTABLE_LIBRARIES: tuple[str, ...] = (
# ROCm-only legacy extensions. Most ROCm ops have moved to
# _C_stable_libtorch, but these two modules still bind libtorch directly.
"_C.abi3.so",
"_rocm_C.abi3.so",
"third_party/deep_gemm/_C*.so",
)
def _relative_path(lib: ExtensionReport, package_root: Path) -> str:
try:
return lib.path.relative_to(package_root).as_posix()
except ValueError:
return lib.path.name
def _is_torch_unstable(lib: ExtensionReport) -> bool:
return lib.error is None and lib.torch.uses_torch and not lib.torch.stable
def _matches_allowlist(rel_path: str, patterns: tuple[str, ...]) -> bool:
return any(fnmatch.fnmatch(rel_path, pattern) for pattern in patterns)
def _iter_libs(report: PackageReport) -> tuple[ExtensionReport, ...]:
return (*report.extensions, *report.bundled_libs)
def _collect_unstable(report: PackageReport) -> list[str]:
return sorted(
_relative_path(lib, report.root)
for lib in _iter_libs(report)
if _is_torch_unstable(lib)
)
def _find_stale_allowlist_entries(
report: PackageReport, patterns: tuple[str, ...]
) -> list[str]:
"""Allowlist patterns that match a built library which is no longer unstable."""
stale: list[str] = []
for pattern in patterns:
for lib in _iter_libs(report):
if lib.error is not None:
continue
if not fnmatch.fnmatch(_relative_path(lib, report.root), pattern):
continue
if not _is_torch_unstable(lib):
stale.append(pattern)
break
return stale
def check_torch_abi(
package: str = "vllm",
patterns: tuple[str, ...] = ALLOWED_UNSTABLE_LIBRARIES,
) -> int:
report = inspect_package(package)
if report.error:
print(f"error: failed to inspect {package!r}: {report.error}", file=sys.stderr)
return 2
unstable = _collect_unstable(report)
unexpected = [
rel_path for rel_path in unstable if not _matches_allowlist(rel_path, patterns)
]
stale = _find_stale_allowlist_entries(report, patterns)
if unexpected or stale:
if unexpected:
print(
"Not allowed: torch-unstable libraries outside "
f"ALLOWED_UNSTABLE_LIBRARIES: {', '.join(unexpected)}",
file=sys.stderr,
)
if stale:
print(
"Not allowed: stale ALLOWED_UNSTABLE_LIBRARIES entries: "
f"{', '.join(stale)}",
file=sys.stderr,
)
return 1
print("Torch stable ABI check passed.")
return 0
if __name__ == "__main__":
print(">>> Auditing vLLM extension modules for PyTorch stable ABI compliance")
sys.exit(check_torch_abi())
-53
View File
@@ -1,53 +0,0 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
import os
import sys
import zipfile
# Read the VLLM_MAX_SIZE_MB environment variable, defaulting to 500 MiB
# Note that we have 800 MiB quota, please use it wisely.
# See https://github.com/pypi/support/issues/6326 .
# Please also sync the value with the one in Dockerfile.
VLLM_MAX_SIZE_MB = int(os.environ.get("VLLM_MAX_SIZE_MB", 500))
def print_top_10_largest_files(zip_file):
"""Print the top 10 largest files in the given zip file."""
with zipfile.ZipFile(zip_file, "r") as z:
file_sizes = [(f, z.getinfo(f).file_size) for f in z.namelist()]
file_sizes.sort(key=lambda x: x[1], reverse=True)
for f, size in file_sizes[:10]:
print(f"{f}: {size / (1024 * 1024):.2f} MBs uncompressed.")
def check_wheel_size(directory):
"""Check the size of .whl files in the given directory."""
for root, _, files in os.walk(directory):
for file_name in files:
if file_name.endswith(".whl"):
wheel_path = os.path.join(root, file_name)
wheel_size_mb = os.path.getsize(wheel_path) / (1024 * 1024)
if wheel_size_mb > VLLM_MAX_SIZE_MB:
print(
f"Not allowed: Wheel {wheel_path} is larger "
f"({wheel_size_mb:.2f} MB) than the limit "
f"({VLLM_MAX_SIZE_MB} MB)."
)
print_top_10_largest_files(wheel_path)
return 1
else:
print(
f"Wheel {wheel_path} is within the allowed size "
f"({wheel_size_mb:.2f} MB)."
)
return 0
if __name__ == "__main__":
if len(sys.argv) < 2:
print("Usage: python check-wheel-size.py <directory>")
sys.exit(1)
directory = sys.argv[1]
sys.exit(check_wheel_size(directory))
-27
View File
@@ -1,27 +0,0 @@
name: vllm_ci
job_dirs:
- ".buildkite/image_build"
- ".buildkite/test_areas"
- ".buildkite/hardware_tests"
run_all_patterns:
- "docker/Dockerfile"
- "CMakeLists.txt"
- "requirements/common.txt"
- "requirements/cuda.txt"
- "requirements/kv_connectors.txt"
- "requirements/build/cuda.txt"
- "requirements/test/cuda.txt"
- "setup.py"
- "csrc/"
- "cmake/"
- ".buildkite/check-torch-abi.py"
run_all_exclude_patterns:
- "docker/Dockerfile."
- "csrc/cpu/"
- "csrc/rocm/"
- "cmake/hipify.py"
- "cmake/cpu_extension.cmake"
registries: public.ecr.aws/q9t5s3a7
repositories:
main: "vllm-ci-postmerge-repo"
premerge: "vllm-ci-test-repo"
-23
View File
@@ -1,23 +0,0 @@
name: vllm_intel_ci
job_dirs:
- ".buildkite/intel_jobs"
run_all_patterns:
- ".buildkite/ci_config_intel.yaml"
- ".buildkite/scripts/hardware_ci/run-intel-test.sh"
- "docker/Dockerfile"
- "docker/Dockerfile.xpu"
- "CMakeLists.txt"
- "requirements/common.txt"
- "requirements/xpu.txt"
- "setup.py"
- "csrc/"
- "cmake/"
run_all_exclude_patterns:
- "csrc/cpu/"
- "csrc/rocm/"
- "cmake/hipify.py"
- "cmake/cpu_extension.cmake"
registries: public.ecr.aws/q9t5s3a7
repositories:
main: "vllm-ci-test-repo"
premerge: "vllm-ci-test-repo"
-36
View File
@@ -1,36 +0,0 @@
name: vllm_rocm_ci
job_dirs:
- ".buildkite/hardware_tests"
run_all_patterns:
- ".dockerignore"
- "docker/Dockerfile.rocm"
- "docker/Dockerfile.rocm_base"
- "docker/ci-rocm.hcl"
- "docker/docker-bake-rocm.hcl"
- ".buildkite/hardware_tests/amd.yaml"
- ".buildkite/ci_config_rocm.yaml"
- ".buildkite/scripts/ci-bake-rocm.sh"
- ".buildkite/scripts/rocm/"
- ".buildkite/scripts/hardware_ci/run-amd-test.py"
- ".buildkite/scripts/hardware_ci/run-amd-test.sh"
- "CMakeLists.txt"
- "requirements/common.txt"
- "requirements/rocm.txt"
- "requirements/build/rust.txt"
- "requirements/build/rocm.txt"
- "requirements/test/rocm.txt"
- "pyproject.toml"
- "setup.py"
- "tools/build_rust.sh"
- "rust/"
- "rust-toolchain.toml"
- "tools/build_rust.py"
- "tools/install_torchcodec_rocm.sh"
- "tests/vllm_test_utils/"
- "vllm/envs.py"
- "vllm/__init__.py"
- "csrc/"
- "cmake/"
run_all_exclude_patterns:
- "csrc/cpu/"
- "cmake/cpu_extension.cmake"
-88
View File
@@ -1,88 +0,0 @@
group: Hardware - AMD Build
# ROCm image flow:
# 1. Select the content-matching ROCm base, building only on a cache miss.
# 2. Build ci_base from either the stable base or the newly selected base.
# 3. Build the build-scoped ROCm CI image and smoke-test it before GPU jobs run.
steps:
- label: "AMD: :docker: refresh ROCm base"
key: refresh-rocm-base-amd
depends_on: []
device: amd_cpu
no_plugin: true
commands:
- bash .buildkite/scripts/rocm/refresh-base-image.sh
concurrency: 1
concurrency_group: "vllm/rocm/base-build/$BUILDKITE_COMMIT"
env:
DOCKER_BUILDKIT: "1"
BUILDKIT_PROGRESS: "tty"
TERM: "xterm-256color"
retry:
automatic:
- exit_status: 1 # Transient Docker/BuildKit failure
limit: 1
- exit_status: -1 # Agent was lost
limit: 1
- exit_status: -10 # Agent was lost
limit: 1
# Ensure ci_base is up-to-date before building the test image.
# Compares a content hash of ci_base-affecting files against the remote
# image label. If hashes match the build is skipped (< 30 s); if they
# differ ci_base is rebuilt and pushed automatically.
- label: "AMD: :docker: ensure ci_base"
key: ensure-ci-base-amd
soft_fail: false
depends_on:
- refresh-rocm-base-amd
device: amd_cpu
no_plugin: true
commands:
- bash .buildkite/scripts/rocm/build-ci-base.sh
concurrency: 1
concurrency_group: "vllm/rocm/ci-base-build/$BUILDKITE_COMMIT"
env:
DOCKER_BUILDKIT: "1"
BUILDKIT_PROGRESS: "tty"
TERM: "xterm-256color"
VLLM_BAKE_FILE: "docker/docker-bake-rocm.hcl"
PYTORCH_ROCM_ARCH: "gfx90a;gfx942;gfx950"
REMOTE_VLLM: "0"
retry:
automatic:
- exit_status: 1 # Transient Docker/BuildKit failure
limit: 1
- exit_status: 2 # Exact image found but registry handoff failed
limit: 1
- exit_status: -1 # Agent was lost
limit: 1
- exit_status: -10 # Agent was lost
limit: 1
- label: "AMD: :docker: build test image and artifacts"
key: image-build-amd
soft_fail: false
depends_on:
- ensure-ci-base-amd
device: amd_cpu
no_plugin: true
commands:
- bash .buildkite/scripts/rocm/build-test-image.sh
- bash .buildkite/scripts/rocm/smoke-test-image.sh
env:
DOCKER_BUILDKIT: "1"
BUILDKIT_PROGRESS: "tty"
TERM: "xterm-256color"
VLLM_BAKE_FILE: "docker/docker-bake-rocm.hcl"
PYTORCH_ROCM_ARCH: "gfx90a;gfx942;gfx950"
IMAGE_TAG: "rocm/vllm-ci:build-$BUILDKITE_BUILD_ID"
REMOTE_VLLM: "0"
retry:
automatic:
- exit_status: 1 # Transient Docker/BuildKit failure
limit: 1
- exit_status: -1 # Agent was lost
limit: 1
- exit_status: -10 # Agent was lost
limit: 1
-105
View File
@@ -1,105 +0,0 @@
group: Hardware
depends_on: ~
steps:
- label: "Ascend NPU Test"
key: ascend-npu-test
soft_fail: true
no_plugin: true
device: ascend_npu
timeout_in_minutes: 30
source_file_dependencies:
- vllm/
- .buildkite/scripts/hardware_ci/run-npu-test.sh
- "!vllm/platforms/cuda.py"
- "!vllm/platforms/xpu.py"
- "!vllm/platforms/tpu.py"
- "!vllm/platforms/cpu.py"
- "!vllm/platforms/zen_cpu.py"
- "!vllm/cute_utils/"
- "!vllm/tilelang_utils/"
- "!vllm/device_allocator/"
- "!vllm/vllm_flash_attn/"
- "!vllm/kernels/"
- "!vllm/model_executor/kernels/"
- "!vllm/engine/"
- "!vllm/entrypoints/"
- "!vllm/ray/"
- "!vllm/v1/attention/backends/flash_attn.py"
- "!vllm/v1/attention/backends/flash_attn_diffkv.py"
- "!vllm/v1/attention/backends/flashinfer.py"
- "!vllm/v1/attention/backends/triton_attn.py"
- "!vllm/v1/attention/backends/triton_attn_diffkv.py"
- "!vllm/v1/attention/backends/rocm_attn.py"
- "!vllm/v1/attention/backends/rocm_aiter_fa.py"
- "!vllm/v1/attention/backends/rocm_aiter_unified_attn.py"
- "!vllm/v1/attention/backends/flex_attention.py"
- "!vllm/v1/attention/backends/hpc_attn.py"
- "!vllm/v1/attention/backends/mamba1_attn.py"
- "!vllm/v1/attention/backends/mamba2_attn.py"
- "!vllm/v1/attention/backends/mamba_attn.py"
- "!vllm/v1/attention/backends/short_conv_attn.py"
- "!vllm/v1/attention/backends/turboquant_attn.py"
- "!vllm/v1/attention/backends/cpu_attn.py"
- "!vllm/v1/attention/backends/fa_utils.py"
- "!vllm/distributed/device_communicators/pynccl.py"
- "!vllm/distributed/device_communicators/pynccl_wrapper.py"
- "!vllm/distributed/device_communicators/pynccl_allocator.py"
- "!vllm/distributed/device_communicators/xpu_communicator.py"
- "!vllm/distributed/device_communicators/cpu_communicator.py"
- "!vllm/distributed/device_communicators/cuda_wrapper.py"
- "!vllm/distributed/device_communicators/flashinfer_all_reduce.py"
- "!vllm/distributed/device_communicators/aiter_custom_all_reduce.py"
- "!vllm/distributed/device_communicators/custom_all_reduce.py"
- "!vllm/distributed/device_communicators/quick_all_reduce.py"
- "!vllm/distributed/device_communicators/all2all.py"
- "!vllm/distributed/device_communicators/all_reduce_utils.py"
- "!vllm/distributed/device_communicators/shm_object_storage.py"
- "!vllm/distributed/device_communicators/symm_mem.py"
- "!vllm/distributed/device_communicators/mnnvl_compat.py"
- "!vllm/distributed/device_communicators/ray_communicator.py"
- "!vllm/third_party/flashmla/"
- "!vllm/third_party/pynvml.py"
- "!vllm/_custom_ops.py"
- "!vllm/_xpu_ops.py"
- "!vllm/_aiter_ops"
- "!vllm/collect_env.py"
- "!vllm/connections.py"
- "!vllm/env_override.py"
- "!vllm/logits_process.py"
- "!vllm/logprobs.py"
- "!vllm/model_inspection.py"
- "!vllm/outputs.py"
- "!vllm/scalar_type.py"
- "!vllm/scripts.py"
- "!vllm/version.py"
- "!vllm/assets/"
- "!vllm/benchmarks/"
- "!vllm/inputs/"
- "!vllm/ir/"
- "!vllm/parser/"
- "!vllm/plugins/"
- "!vllm/reasoning/"
- "!vllm/renderers/"
- "!vllm/tokenizers/"
- "!vllm/tool_parsers/"
- "!vllm/tracing/"
- "!vllm/usage/"
- "!tests/"
- "!benchmarks/"
- "!docs/"
- "!examples/"
- "!scripts/"
- "!vllm/model_executor/layers/quantization/auto_awq.py"
- "!vllm/model_executor/layers/quantization/auto_gptq.py"
- "!vllm/model_executor/layers/quantization/inc/"
- "!vllm/model_executor/layers/quantization/awq_triton.py"
- "!vllm/model_executor/layers/quantization/experts_int8.py"
- "!vllm/model_executor/layers/quantization/fbgemm_fp8.py"
- "!vllm/model_executor/layers/quantization/fp_quant.py"
- "!vllm/model_executor/layers/quantization/humming.py"
- "!vllm/model_executor/layers/quantization/moe_wna16.py"
- "!vllm/model_executor/layers/quantization/qutlass_utils.py"
- "!vllm/model_executor/layers/quantization/quark/"
- "!vllm/model_executor/layers/quantization/turboquant/"
commands:
- bash .buildkite/scripts/hardware_ci/run-npu-test.sh
-230
View File
@@ -1,230 +0,0 @@
group: CPU
depends_on: []
steps:
- label: CPU-Kernel Tests Shard %N
key: cpu-kernel-tests
depends_on: []
device: intel_cpu
no_plugin: true
source_file_dependencies:
- csrc/cpu/
- cmake/cpu_extension.cmake
- CMakeLists.txt
- vllm/_custom_ops.py
- tests/kernels/attention/test_cpu_attn.py
- tests/v1/attention/test_group_head_counts.py
- tests/kernels/attention/test_amx_mla.py
- vllm/v1/attention/backends/mla/amx_mla.py
- vllm/model_executor/layers/attention/mla_attention.py
- tests/kernels/test_deepseek_v4_cpu_kernels.py
- tests/kernels/moe/test_cpu_fused_moe.py
- tests/kernels/moe/test_cpu_quant_fused_moe.py
- tests/kernels/moe/test_zen_cpu_int8_moe.py
- vllm/model_executor/layers/fused_moe/experts/cpu_moe.py
- tests/kernels/test_onednn.py
- tests/kernels/test_awq_int4_to_int8.py
- tests/kernels/quantization/test_cpu_fp8_scaled_mm.py
- tests/kernels/quantization/test_zen_da8w4.py
- vllm/model_executor/kernels/linear/mixed_precision/zentorch.py
- vllm/model_executor/layers/fused_moe/experts/zentorch_moe.py
- tests/kernels/mamba/cpu/test_cpu_gdn_ops.py
- tests/kernels/mamba/test_cpu_short_conv.py
- tests/kernels/mamba/test_causal_conv1d.py
- tests/kernels/mamba/test_mamba_ssm.py
- vllm/v1/sample/ops/topk_topp_triton.py
- vllm/v1/sample/ops/topk_topp_sampler.py
- tests/v1/sample/test_topk_topp_sampler.py
commands:
- |
bash .buildkite/scripts/hardware_ci/run-cpu-test.sh 30m "
pytest -x -v -s --num-shards=$$BUILDKITE_PARALLEL_JOB_COUNT --shard-id=$$BUILDKITE_PARALLEL_JOB \
tests/kernels/attention/test_cpu_attn.py \
tests/v1/attention/test_group_head_counts.py \
tests/kernels/attention/test_amx_mla.py \
tests/kernels/test_deepseek_v4_cpu_kernels.py \
tests/kernels/moe/test_cpu_fused_moe.py \
tests/kernels/moe/test_cpu_quant_fused_moe.py \
tests/kernels/moe/test_zen_cpu_int8_moe.py \
tests/kernels/mamba/test_cpu_short_conv.py \
tests/kernels/test_onednn.py \
tests/kernels/test_awq_int4_to_int8.py \
tests/kernels/quantization/test_cpu_fp8_scaled_mm.py \
tests/kernels/quantization/test_zen_da8w4.py \
tests/kernels/mamba/cpu/test_cpu_gdn_ops.py \
tests/kernels/mamba/test_causal_conv1d.py \
tests/kernels/mamba/test_mamba_ssm.py \
tests/v1/sample/test_topk_topp_sampler.py::TestTritonTopkTopp"
parallelism: 2
# Speculative decoding under a CPU platform. The unit tests already pass on
# CPU but no CPU job runs them; the e2e tests gate the engine path itself,
# which unit tests cannot reach.
- label: CPU-Spec Decode Tests
key: cpu-spec-decode-tests
depends_on: []
soft_fail: true
device: intel_cpu
no_plugin: true
source_file_dependencies:
- vllm/v1/spec_decode/
- vllm/v1/worker/cpu/
- vllm/v1/worker/cpu_model_runner.py
- vllm/v1/worker/gpu/spec_decode/
- vllm/v1/attention/backends/cpu_attn.py
- vllm/config/speculative.py
- vllm/model_executor/models/qwen3_5.py
- vllm/model_executor/models/qwen3_5_mtp.py
- vllm/model_executor/models/llama_eagle3.py
- vllm/model_executor/layers/mamba/gdn/qwen_gdn_linear_attn.py
- vllm/model_executor/layers/mamba/mamba_mixer2.py
- vllm/model_executor/layers/mamba/ops/cpu/
- tests/v1/e2e/test_cpu_spec_decode.py
- tests/v1/spec_decode/
- tests/v1/worker/
commands:
- |
bash .buildkite/scripts/hardware_ci/run-cpu-test.sh 30m "
pytest -x -v -s \
tests/v1/worker/test_gpu_autoregressive_speculator.py \
tests/v1/worker/test_gpu_extract_hidden_states_speculator.py \
tests/v1/worker/test_gpu_batch_ordering.py \
tests/v1/worker/test_gpu_rejection_sampler_chunking.py \
tests/v1/worker/test_eagle3_aux_hidden_states_pp.py \
tests/v1/worker/test_spec_decode_embed_sharing_pp.py \
tests/v1/spec_decode/test_adaptive_verification.py \
tests/v1/spec_decode/test_dflash2.py \
tests/v1/spec_decode/test_dflash_causality.py \
tests/v1/spec_decode/test_eagle_draft_attn_metadata.py \
tests/v1/spec_decode/test_draft_attention_backend_override.py \
tests/v1/spec_decode/test_draft_moe_backend_override.py \
tests/v1/spec_decode/test_ngram.py \
tests/v1/spec_decode/test_llm_base_proposer_sampling.py
unset VLLM_CPU_CI_ENV
pytest -x -v -s tests/v1/e2e/test_cpu_spec_decode.py"
# Note: SDE can't be downloaded from CI host because of AWS WAF
# - label: CPU-Compatibility Tests
# depends_on: []
# device: intel_cpu
# no_plugin: true
# source_file_dependencies:
# - cmake/cpu_extension.cmake
# - setup.py
# - vllm/platforms/cpu.py
# commands:
# - |
# bash .buildkite/scripts/hardware_ci/run-cpu-test.sh 20m "
# bash .buildkite/scripts/hardware_ci/run-cpu-compatibility-test.sh"
- label: CPU-Language Generation and Pooling Model Tests Shard %N
key: cpu-language-generation-and-pooling-model-tests
depends_on: []
device: intel_cpu
no_plugin: true
source_file_dependencies:
- csrc/cpu/
- vllm/
- tests/models/language/generation/
- tests/models/language/pooling/
- tests/v1/e2e/test_cpu_linear_attn_chunked_prefix.py
commands:
- |
bash .buildkite/scripts/hardware_ci/run-cpu-test.sh 25m "
pytest -x -v -s tests/models/language/generation tests/models/language/pooling tests/v1/e2e/test_cpu_linear_attn_chunked_prefix.py -m cpu_model --num-shards=$$BUILDKITE_PARALLEL_JOB_COUNT --shard-id=$$BUILDKITE_PARALLEL_JOB"
parallelism: 3
- label: CPU-Quantization Model Tests
key: cpu-quantization-model-tests
depends_on: []
device: intel_cpu
no_plugin: true
source_file_dependencies:
- csrc/cpu/
- vllm/model_executor/layers/quantization/auto_gptq.py
- vllm/model_executor/layers/quantization/compressed_tensors/schemes/compressed_tensors_w8a8_int8.py
- vllm/model_executor/kernels/linear/mixed_precision/cpu.py
- vllm/model_executor/kernels/linear/scaled_mm/cpu.py
- vllm/model_executor/layers/fused_moe/experts/cpu_moe.py
- tests/quantization/test_compressed_tensors.py
- tests/quantization/test_cpu_wna16.py
- tests/quantization/test_cpu_w8a8.py
commands:
- |
bash .buildkite/scripts/hardware_ci/run-cpu-test.sh 45m "
pytest -x -v -s tests/quantization/test_compressed_tensors.py::test_compressed_tensors_w8a8_logprobs
pytest -x -v -s tests/quantization/test_cpu_wna16.py
pytest -x -v -s tests/quantization/test_cpu_w8a8.py"
- label: CPU-Distributed Tests (PP+TP)
key: cpu-distributed-tests-pp-tp
depends_on: []
device: intel_cpu
no_plugin: true
source_file_dependencies: &cpu_distributed_deps
- csrc/cpu/shm.cpp
- vllm/v1/worker/cpu_worker.py
- vllm/v1/worker/gpu_worker.py
- vllm/v1/worker/cpu_model_runner.py
- vllm/v1/worker/gpu_model_runner.py
- vllm/platforms/cpu.py
- vllm/distributed/parallel_state.py
- vllm/distributed/device_communicators/cpu_communicator.py
- .buildkite/scripts/hardware_ci/run-cpu-distributed-smoke-test.sh
commands:
- |
bash .buildkite/scripts/hardware_ci/run-cpu-test.sh 10m "
bash .buildkite/scripts/hardware_ci/run-cpu-distributed-smoke-test.sh tp_pp"
- label: CPU-Distributed Tests (DP+TP)
key: cpu-distributed-tests-dp-tp
depends_on: []
device: intel_cpu
no_plugin: true
source_file_dependencies: *cpu_distributed_deps
commands:
- |
bash .buildkite/scripts/hardware_ci/run-cpu-test.sh 10m "
bash .buildkite/scripts/hardware_ci/run-cpu-distributed-smoke-test.sh dp_tp"
- label: CPU-Multi-Modal Model Tests %N
key: cpu-multi-modal-model-tests-n
depends_on: []
device: intel_cpu
no_plugin: true
source_file_dependencies:
# - vllm/
- vllm/model_executor/layers/rotary_embedding
- tests/models/multimodal/generation/
commands:
- |
bash .buildkite/scripts/hardware_ci/run-cpu-test.sh 45m "
pytest -x -v -s tests/models/multimodal/generation --ignore=tests/models/multimodal/generation/test_pixtral.py --ignore=tests/models/multimodal/generation/test_qwen2_5_vl.py -m cpu_model --num-shards=$$BUILDKITE_PARALLEL_JOB_COUNT --shard-id=$$BUILDKITE_PARALLEL_JOB"
parallelism: 4
- label: CPU-Qwen2.5-VL Multimodal Tests
key: cpu-qwen2-5-vl-multimodal-tests
depends_on: []
device: intel_cpu
no_plugin: true
source_file_dependencies:
# - vllm/
- vllm/model_executor/layers/rotary_embedding
- tests/models/multimodal/generation/
commands:
- |
bash .buildkite/scripts/hardware_ci/run-cpu-test.sh 40m "
VLLM_CI_ENV=0 pytest -x -v -s tests/models/multimodal/generation/test_qwen2_5_vl.py"
- label: "Arm CPU Test Shard %N"
key: arm-cpu-test
depends_on:
- cpu-arm64-image-build
soft_fail: false
device: arm_cpu
no_plugin: true
commands:
- >-
bash .buildkite/scripts/hardware_ci/run-cpu-test-arm.sh
"$$BUILDKITE_PARALLEL_JOB"
"$REGISTRY/$REPO:$BUILDKITE_COMMIT-arm64-cpu"
parallelism: 3
-11
View File
@@ -1,11 +0,0 @@
group: Hardware
steps:
- label: "GH200 Test"
key: gh200-test
soft_fail: true
device: gh200
no_plugin: true
optional: true
commands:
- nvidia-smi
- bash .buildkite/scripts/hardware_ci/run-gh200-test.sh
-11
View File
@@ -1,11 +0,0 @@
group: Hardware
depends_on: ~
steps:
- label: "Intel HPU Test"
key: intel-hpu-test
soft_fail: true
device: intel_hpu
no_plugin: true
commands:
- bash .buildkite/scripts/hardware_ci/run-hpu-test.sh
@@ -1,175 +0,0 @@
group: Intel
steps:
- label: ":docker: Build XPU image"
soft_fail: true
optional: true
depends_on: []
key: image-build-xpu
commands:
- bash -lc '.buildkite/image_build/image_build_xpu.sh "public.ecr.aws/q9t5s3a7" "vllm-ci-test-repo" "$BUILDKITE_COMMIT"'
env:
DOCKER_BUILDKIT: "1"
retry:
automatic:
- exit_status: -1 # Agent was lost
limit: 2
- exit_status: -10 # Agent was lost
limit: 2
- label: "XPU example Test"
key: xpu-example-test
depends_on:
- image-build-xpu
timeout_in_minutes: 50
optional: true
device: intel_gpu
agent_tags:
label: production
gpu: 2+
mem: 24+
no_plugin: true
env:
REGISTRY: "public.ecr.aws/q9t5s3a7"
REPO: "vllm-ci-test-repo"
source_file_dependencies:
- .buildkite/hardware_tests/intel_xpu_ci/test-intel.yaml
- .buildkite/scripts/hardware_ci/run-intel-ci-test.sh
commands:
- >-
bash .buildkite/scripts/hardware_ci/run-intel-test.sh
'bash .buildkite/scripts/hardware_ci/run-intel-ci-test.sh example'
- label: "XPU V1 test"
key: xpu-v1-test
depends_on:
- image-build-xpu
timeout_in_minutes: 70
optional: true
device: intel_gpu
agent_tags:
label: production
gpu: 1+
mem: 24+
no_plugin: true
env:
REGISTRY: "public.ecr.aws/q9t5s3a7"
REPO: "vllm-ci-test-repo"
source_file_dependencies:
- .buildkite/hardware_tests/intel_xpu_ci/test-intel.yaml
- .buildkite/scripts/hardware_ci/run-intel-ci-test.sh
commands:
- >-
bash .buildkite/scripts/hardware_ci/run-intel-test.sh
'bash .buildkite/scripts/hardware_ci/run-intel-ci-test.sh v1'
parallelism: 2
- label: "XPU W8A8 FP8 Linear Examples"
key: xpu-w8a8-fp8-linear-examples
depends_on:
- image-build-xpu
timeout_in_minutes: 60
optional: true
device: intel_gpu
agent_tags:
label: production
gpu: 1+
mem: 24+
no_plugin: true
env:
REGISTRY: "public.ecr.aws/q9t5s3a7"
REPO: "vllm-ci-test-repo"
VLLM_TEST_DEVICE: "xpu"
source_file_dependencies:
- .buildkite/hardware_tests/intel_xpu_ci/test-intel.yaml
- .buildkite/scripts/hardware_ci/run-intel-ci-test.sh
commands:
- >-
bash .buildkite/scripts/hardware_ci/run-intel-test.sh
'bash .buildkite/scripts/hardware_ci/run-intel-ci-test.sh w8a8-fp8-linear'
- label: "XPU server test"
key: xpu-server-test
depends_on:
- image-build-xpu
timeout_in_minutes: 45
optional: true
device: intel_gpu
agent_tags:
label: production
gpu: 1+
mem: 16+
no_plugin: true
env:
REGISTRY: "public.ecr.aws/q9t5s3a7"
REPO: "vllm-ci-test-repo"
source_file_dependencies:
- .buildkite/hardware_tests/intel_xpu_ci/test-intel.yaml
- .buildkite/scripts/hardware_ci/run-intel-ci-test.sh
commands:
- >-
bash .buildkite/scripts/hardware_ci/run-intel-test.sh
'bash .buildkite/scripts/hardware_ci/run-intel-ci-test.sh server'
- label: "XPU quantization test"
key: xpu-quantization-test
depends_on:
- image-build-xpu
timeout_in_minutes: 60
optional: true
device: intel_gpu
agent_tags:
label: production
gpu: 1+
mem: 16+
no_plugin: true
env:
REGISTRY: "public.ecr.aws/q9t5s3a7"
REPO: "vllm-ci-test-repo"
source_file_dependencies:
- .buildkite/hardware_tests/intel_xpu_ci/test-intel.yaml
- .buildkite/scripts/hardware_ci/run-intel-ci-test.sh
commands:
- >-
bash .buildkite/scripts/hardware_ci/run-intel-test.sh
'bash .buildkite/scripts/hardware_ci/run-intel-ci-test.sh quantization'
- label: "XPU compressed tensors FP8 test"
key: xpu-compressed-tensors-fp8-test
depends_on:
- image-build-xpu
timeout_in_minutes: 60
optional: true
device: intel_gpu
agent_tags:
label: production
gpu: 1+
mem: 16+
no_plugin: true
env:
REGISTRY: "public.ecr.aws/q9t5s3a7"
REPO: "vllm-ci-test-repo"
VLLM_TEST_DEVICE: "xpu"
source_file_dependencies:
- .buildkite/hardware_tests/intel_xpu_ci/test-intel.yaml
- .buildkite/scripts/hardware_ci/run-intel-ci-test.sh
commands:
- >-
bash .buildkite/scripts/hardware_ci/run-intel-test.sh
'bash .buildkite/scripts/hardware_ci/run-intel-ci-test.sh compressed-tensors-fp8'
- label: "XPU Graph"
key: xpu-graph
depends_on:
- image-build-xpu
timeout_in_minutes: 60
optional: true
device: intel_gpu
agent_tags:
label: production
gpu: 1+
mem: 32+
no_plugin: true
env:
REGISTRY: "public.ecr.aws/q9t5s3a7"
REPO: "vllm-ci-test-repo"
VLLM_TEST_DEVICE: "xpu"
source_file_dependencies:
- .buildkite/hardware_tests/intel_xpu_ci/test-intel.yaml
- .buildkite/scripts/hardware_ci/run-intel-ci-test.sh
commands:
- >-
bash .buildkite/scripts/hardware_ci/run-intel-test.sh
'bash .buildkite/scripts/hardware_ci/run-intel-ci-test.sh graph'
-376
View File
@@ -1,376 +0,0 @@
#!/bin/bash
set -euo pipefail
# replace invalid characters in Docker image tags and truncate to 128 chars
clean_docker_tag() {
local input="$1"
echo "$input" | sed 's/[^a-zA-Z0-9._-]/_/g' | cut -c1-128
}
print_usage_and_exit() {
echo "Usage: $0 <registry> <repo> <commit> <branch> <image_tag> [<image_tag_latest>]"
exit 1
}
print_instance_info() {
echo ""
echo "=== Debug: Instance Information ==="
# Get IMDSv2 token
if TOKEN=$(curl -s -X PUT "http://169.254.169.254/latest/api/token" \
-H "X-aws-ec2-metadata-token-ttl-seconds: 21600" 2>/dev/null); then
AMI_ID=$(curl -s -H "X-aws-ec2-metadata-token: $TOKEN" \
http://169.254.169.254/latest/meta-data/ami-id 2>/dev/null || echo "unknown")
INSTANCE_TYPE=$(curl -s -H "X-aws-ec2-metadata-token: $TOKEN" \
http://169.254.169.254/latest/meta-data/instance-type 2>/dev/null || echo "unknown")
INSTANCE_ID=$(curl -s -H "X-aws-ec2-metadata-token: $TOKEN" \
http://169.254.169.254/latest/meta-data/instance-id 2>/dev/null || echo "unknown")
AZ=$(curl -s -H "X-aws-ec2-metadata-token: $TOKEN" \
http://169.254.169.254/latest/meta-data/placement/availability-zone 2>/dev/null || echo "unknown")
echo "AMI ID: ${AMI_ID}"
echo "Instance Type: ${INSTANCE_TYPE}"
echo "Instance ID: ${INSTANCE_ID}"
echo "AZ: ${AZ}"
else
echo "Not running on EC2 or IMDS not available"
fi
# Check for warm cache AMI (marker file baked into custom AMI)
if [[ -f /etc/vllm-ami-info ]]; then
echo "Cache: warm (custom vLLM AMI)"
cat /etc/vllm-ami-info
else
echo "Cache: cold (standard AMI)"
fi
echo "==================================="
echo ""
}
setup_buildx_builder() {
echo "--- :buildkite: Setting up buildx builder"
if [[ -S "${BUILDKIT_SOCKET}" ]]; then
# Custom AMI with standalone buildkitd - use remote driver for warm cache
echo "✅ Found local buildkitd socket at ${BUILDKIT_SOCKET}"
echo "Using remote driver to connect to buildkitd (warm cache available)"
if docker buildx inspect baked-vllm-builder >/dev/null 2>&1; then
echo "Using existing baked-vllm-builder"
docker buildx use baked-vllm-builder
else
echo "Creating baked-vllm-builder with remote driver"
docker buildx create \
--name baked-vllm-builder \
--driver remote \
--use \
"unix://${BUILDKIT_SOCKET}"
fi
docker buildx inspect --bootstrap
elif docker buildx inspect "${BUILDER_NAME}" >/dev/null 2>&1; then
# Existing builder available
echo "Using existing builder: ${BUILDER_NAME}"
docker buildx use "${BUILDER_NAME}"
docker buildx inspect --bootstrap
else
# No local buildkitd, no existing builder - create new docker-container builder
echo "No local buildkitd found, using docker-container driver"
docker buildx create --name "${BUILDER_NAME}" --driver docker-container --use
docker buildx inspect --bootstrap
fi
# builder info
echo "Active builder:"
docker buildx ls | grep -E '^\*|^NAME' || docker buildx ls
}
download_ci_hcl() {
# Returns non-zero instead of exiting: the optional export path must be
# able to give up without failing a build whose image already exists.
echo "--- :arrow_down: Downloading ci.hcl"
curl -sSfL -o "${CI_HCL_PATH}" "${CI_HCL_URL}" || return $?
echo "Downloaded to ${CI_HCL_PATH}"
if [[ ! -f "${CI_HCL_PATH}" ]]; then
echo "Error: ci.hcl not found at ${CI_HCL_PATH}"
return 1
fi
}
export_kernel_symbol_map_from_cache() {
# The existing-image path: everything here is best effort. Called as
# `... || echo`, so errexit is suspended inside and any failure just
# returns; the build keeps the image it already has.
local rc=0
download_ci_hcl \
&& setup_buildx_builder \
&& resolve_parent_commit \
&& export PARENT_COMMIT \
&& BUILD_TMP_DIR="$(mktemp -d)" \
&& export_kernel_symbol_map || rc=$?
rm -rf -- "${BUILD_TMP_DIR:-}"
return "${rc}"
}
export_kernel_symbol_map() {
# Only when the build asked for it (nightly/post-merge). The map was
# produced inside the csrc-build stage during the main bake; this second
# bake of the scratch stage is a cache hit that just writes the file out.
if [[ "${VLLM_KERNEL_SYMBOL_MAP:-0}" != "1" ]]; then
return 0
fi
echo "--- :world_map: Exporting kernel symbol map"
local out_dir="${BUILD_TMP_DIR}/kernel-symbol-map"
mkdir -p "${out_dir}"
if docker buildx bake -f "${VLLM_BAKE_FILE_PATH}" -f "${CI_HCL_PATH}" --progress plain \
--set "kernel-symbol-map.output=type=local,dest=${out_dir}" kernel-symbol-map \
&& [[ -s "${out_dir}/kernel_symbol_map.json.gz" ]]; then
ls -la "${out_dir}"
(cd "${out_dir}" && buildkite-agent artifact upload "kernel_symbol_map.json.gz")
else
echo "kernel symbol map export failed; continuing without it" >&2
fi
}
annotate_image_tags() {
.buildkite/scripts/annotate-image-build.sh \
"${IMAGE_TAG:-}" "${IMAGE_TAG_LATEST:-}"
}
check_and_skip_if_image_exists() {
if [[ -n "${IMAGE_TAG:-}" ]]; then
echo "--- :mag: Checking if image exists"
if docker manifest inspect "${IMAGE_TAG}" >/dev/null 2>&1; then
echo "Image already exists: ${IMAGE_TAG}"
echo "Skipping build"
if [[ "${VLLM_KERNEL_SYMBOL_MAP:-0}" == "1" ]]; then
# The image is reused, but the symbol map is an artifact of
# this build. Bake just the export target: its layers come
# from the registry cache, and the map step itself reruns
# if the cached image was built without the arg. Never
# fails the build: the image is already there.
export_kernel_symbol_map_from_cache \
|| echo "kernel symbol map: export from the cached image failed; continuing" >&2
fi
annotate_image_tags
exit 0
fi
echo "Image not found, proceeding with build"
fi
}
ecr_login() {
aws ecr-public get-login-password --region us-east-1 | docker login --username AWS --password-stdin "$REGISTRY" || true
aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin 936637512419.dkr.ecr.us-east-1.amazonaws.com || true
}
prepare_cache_tags() {
# resolve and set: CACHE_TO, CACHE_FROM, CACHE_FROM_BASE_BRANCH, CACHE_FROM_MAIN
TEST_CACHE_ECR="936637512419.dkr.ecr.us-east-1.amazonaws.com/vllm-ci-test-cache"
MAIN_CACHE_ECR="936637512419.dkr.ecr.us-east-1.amazonaws.com/vllm-ci-postmerge-cache"
if [[ "$BUILDKITE_PULL_REQUEST" == "false" ]]; then
if [[ "$BUILDKITE_BRANCH" == "main" ]]; then
cache="${MAIN_CACHE_ECR}:latest"
else
clean_branch=$(clean_docker_tag "$BUILDKITE_BRANCH")
cache="${TEST_CACHE_ECR}:${clean_branch}"
fi
CACHE_TO="$cache"
CACHE_FROM="$cache"
CACHE_FROM_BASE_BRANCH="$cache"
else
CACHE_TO="${TEST_CACHE_ECR}:pr-${BUILDKITE_PULL_REQUEST}"
CACHE_FROM="${TEST_CACHE_ECR}:pr-${BUILDKITE_PULL_REQUEST}"
if [[ "$BUILDKITE_PULL_REQUEST_BASE_BRANCH" == "main" ]]; then
CACHE_FROM_BASE_BRANCH="${MAIN_CACHE_ECR}:latest"
else
clean_base=$(clean_docker_tag "$BUILDKITE_PULL_REQUEST_BASE_BRANCH")
CACHE_FROM_BASE_BRANCH="${TEST_CACHE_ECR}:${clean_base}"
fi
fi
CACHE_FROM_MAIN="${MAIN_CACHE_ECR}:latest"
export CACHE_TO CACHE_FROM CACHE_FROM_BASE_BRANCH CACHE_FROM_MAIN
}
resolve_parent_commit() {
if [[ -z "${PARENT_COMMIT:-}" ]]; then
PARENT_COMMIT=$(git rev-parse HEAD~1 2>/dev/null || echo "")
if [[ -n "${PARENT_COMMIT}" ]]; then
echo "Computed parent commit for cache fallback: ${PARENT_COMMIT}"
export PARENT_COMMIT
else
echo "Could not determine parent commit (may be first commit in repo)"
fi
else
echo "Using provided PARENT_COMMIT: ${PARENT_COMMIT}"
fi
}
print_bake_config() {
echo "--- :page_facing_up: Resolved bake configuration"
# Write to a temp directory to avoid polluting the repo root (which is the
# Docker build context). Files left in the repo root get COPY'd into the
# image and can cause duplicate artifact uploads from downstream steps.
local bake_tmp
bake_tmp="$(mktemp -d)"
BAKE_CONFIG_FILE="${bake_tmp}/bake-config-build-${BUILDKITE_BUILD_NUMBER:-local}.json"
docker buildx bake "${BAKE_FILES[@]}" --print "${TARGET}" | tee "${BAKE_CONFIG_FILE}" || true
echo "Saved bake config to ${BAKE_CONFIG_FILE}"
echo "--- :arrow_down: Uploading bake config to Buildkite"
(cd "$(dirname "${BAKE_CONFIG_FILE}")" && buildkite-agent artifact upload "$(basename "${BAKE_CONFIG_FILE}")")
}
record_buildkit_trace() (
local metadata_file="$1"
local helper=".buildkite/scripts/ci-otel/ci_otel.py"
local trace_file="${BUILD_TMP_DIR}/buildkit-trace.json"
local build_ref record_id
if [[ "${BUILDKITE:-}" != "true" || ! -f "${helper}" ]] ||
! command -v timeout >/dev/null 2>&1 ||
! command -v python3 >/dev/null 2>&1; then
return 0
fi
build_ref="$(timeout 3s python3 "${helper}" build-ref "${metadata_file}" "${TARGET}" 2>/dev/null)" || {
echo "vLLM CI OTel: BuildKit metadata unavailable; trace skipped" >&2
return 0
}
record_id="${build_ref##*/}"
if [[ -z "${record_id}" ]]; then
return 0
fi
echo "--- :mag: Recording BuildKit trace"
if ! timeout 20s docker buildx history trace "${record_id}" >"${trace_file}"; then
echo "vLLM CI OTel: BuildKit history trace unavailable; upload skipped" >&2
return 0
fi
export CI_INFRA_BUILDX_REF="${build_ref}"
export CI_INFRA_OTEL_SPOOL_DIR="${BUILD_TMP_DIR}/otel-spans"
timeout 5s python3 "${helper}" record-buildkit "${trace_file}" || {
echo "vLLM CI OTel: BuildKit trace conversion skipped" >&2
return 0
}
timeout 5s python3 "${helper}" flush ||
echo "vLLM CI OTel: BuildKit trace upload skipped" >&2
)
#################################
# Main Script #
#################################
print_instance_info
if [[ $# -lt 5 ]]; then
print_usage_and_exit
fi
# input args
REGISTRY=$1
REPO=$2
BUILDKITE_COMMIT=$3
BRANCH=$4
IMAGE_TAG=$5
IMAGE_TAG_LATEST=${6:-} # only used for main branch, optional
# When TORCH_NIGHTLY=1, build the base CI image against PyTorch nightly so the
# entire existing pipeline runs on nightly torch (CUDA/GPU lane only). Delegate
# to the dedicated nightly build (PYTORCH_NIGHTLY=1, CUDA 13.0) and tag it at the
# normal IMAGE_TAG that every test step already pulls -- no separate image tag,
# no duplicate "vLLM Against PyTorch Nightly" pipeline section.
if [[ "${TORCH_NIGHTLY:-0}" == "1" ]]; then
echo "--- :warning: TORCH_NIGHTLY=1 -- building base image on PyTorch nightly"
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
exec "${SCRIPT_DIR}/image_build_torch_nightly.sh" \
"${REGISTRY}" "${REPO}" "${BUILDKITE_COMMIT}" "${BRANCH}" "${IMAGE_TAG}"
fi
# build config
TARGET="test-ci"
VLLM_BAKE_FILE_PATH="${VLLM_BAKE_FILE_PATH:-docker/docker-bake.hcl}"
BUILDER_NAME="${BUILDER_NAME:-vllm-builder}"
CI_HCL_URL="${CI_HCL_URL:-https://raw.githubusercontent.com/vllm-project/ci-infra/main/docker/ci.hcl}"
CI_HCL_PATH="${CI_HCL_PATH:-/tmp/ci.hcl}"
ZSTD_HCL_PATH="${ZSTD_HCL_PATH:-.buildkite/image_build/zstd.hcl}"
BUILDKIT_SOCKET="/run/buildkit/buildkitd.sock"
BAKE_FILES=(-f "${VLLM_BAKE_FILE_PATH}" -f "${CI_HCL_PATH}" -f "${ZSTD_HCL_PATH}")
prepare_cache_tags
ecr_login
# Environment info (for docs and human readers)
# VLLM_CI_BRANCH - ci-infra branch to use (default: main)
# VLLM_BAKE_FILE_PATH - Path to vLLM's bake file (default: docker/docker-bake.hcl)
# BUILDER_NAME - Name for buildx builder (default: vllm-builder)
#
# Build configuration (exported as environment variables for bake):
export BUILDKITE_COMMIT
export PARENT_COMMIT
export IMAGE_TAG
export IMAGE_TAG_LATEST
export COMMIT="${COMMIT:-${BUILDKITE_COMMIT}}"
export CACHE_FROM
export CACHE_FROM_BASE_BRANCH
export CACHE_FROM_MAIN
export CACHE_TO
# print args
echo "--- :mag: Arguments"
echo "REGISTRY: ${REGISTRY}"
echo "REPO: ${REPO}"
echo "BUILDKITE_COMMIT: ${BUILDKITE_COMMIT}"
echo "BRANCH: ${BRANCH}"
echo "IMAGE_TAG: ${IMAGE_TAG}"
echo "IMAGE_TAG_LATEST: ${IMAGE_TAG_LATEST}"
# print build configuration
echo "--- :mag: Build configuration"
echo "TARGET: ${TARGET}"
echo "vLLM bake file: ${VLLM_BAKE_FILE_PATH}"
echo "BUILDER_NAME: ${BUILDER_NAME}"
echo "CI_HCL_URL: ${CI_HCL_URL}"
echo "BUILDKIT_SOCKET: ${BUILDKIT_SOCKET}"
echo "--- :mag: Cache tags"
echo "CACHE_TO: ${CACHE_TO}"
echo "CACHE_FROM: ${CACHE_FROM}"
echo "CACHE_FROM_BASE_BRANCH: ${CACHE_FROM_BASE_BRANCH}"
echo "CACHE_FROM_MAIN: ${CACHE_FROM_MAIN}"
check_and_skip_if_image_exists
echo "--- :docker: Setting up Docker buildx bake"
echo "Target: ${TARGET}"
echo "vLLM bake file: ${VLLM_BAKE_FILE_PATH}"
echo "CI HCL path: ${CI_HCL_PATH}"
if [[ ! -f "${VLLM_BAKE_FILE_PATH}" ]]; then
echo "Error: vLLM bake file not found at ${VLLM_BAKE_FILE_PATH}"
echo "Make sure you're running from the vLLM repository root"
exit 1
fi
download_ci_hcl || exit 1
setup_buildx_builder
resolve_parent_commit
export PARENT_COMMIT
print_bake_config
echo "--- :docker: Building ${TARGET}"
BUILD_TMP_DIR="$(mktemp -d)"
trap 'rm -rf -- "${BUILD_TMP_DIR}"' EXIT
BUILD_METADATA_FILE="${BUILD_TMP_DIR}/build-metadata.json"
BUILD_STATUS=0
docker --debug buildx bake "${BAKE_FILES[@]}" \
--progress plain --metadata-file "${BUILD_METADATA_FILE}" "${TARGET}" || BUILD_STATUS=$?
record_buildkit_trace "${BUILD_METADATA_FILE}" || true
if [[ "${BUILD_STATUS}" -ne 0 ]]; then
exit "${BUILD_STATUS}"
fi
echo "--- :white_check_mark: Build complete"
export_kernel_symbol_map || true
annotate_image_tags
-130
View File
@@ -1,130 +0,0 @@
group: Abuild
steps:
- label: ":docker: Build image"
key: image-build
depends_on: []
timeout_in_minutes: 600
commands:
- if [[ "$BUILDKITE_BRANCH" == "main" ]]; then .buildkite/image_build/image_build.sh $REGISTRY $REPO $BUILDKITE_COMMIT $BRANCH $IMAGE_TAG $IMAGE_TAG_LATEST; else .buildkite/image_build/image_build.sh $REGISTRY $REPO $BUILDKITE_COMMIT $BRANCH $IMAGE_TAG; fi
retry:
automatic:
- exit_status: -1 # Agent was lost
limit: 2
- exit_status: -10 # Agent was lost
limit: 2
- label: ":docker: :smoking: Non-root smoke tests"
key: image-build-smoke-test
depends_on:
- image-build
commands:
# Smoke 1: the default (root) image must still be importable
# under a non-root UID via `--user 2000:0`. Validates the `vllm` passwd
# entry + group-0-writable /home/vllm + uv path cleanup from #31959.
# Uses `import vllm` rather than `vllm serve --help` because the latter
# instantiates `VllmConfig` which requires a GPU attached to the
# container.
- docker run --rm --user 2000:0 --entrypoint python3 "$IMAGE_TAG" -c "import vllm; print(vllm.__version__)"
# Smoke 2: assert the non-root enabling invariants are baked
# into the image. Runs as UID 2000:0 via a shell so we can verify
# filesystem perms + passwd/group file state + wrapper presence without
# triggering vLLM's GPU-requiring config-init path. The opt-in
# `vllm-openai-nonroot` target adds only `USER vllm`, `WORKDIR
# /home/vllm`, and an `ENTRYPOINT` override on top of these invariants;
# its build correctness is reviewed at the Dockerfile level. Wrapper
# logic is covered separately by the pre-commit hook
# `test-nonroot-entrypoint` (see .pre-commit-config.yaml).
- |
docker run --rm --user 2000:0 --entrypoint /bin/sh "$IMAGE_TAG" -ec '
if ! getent passwd 2000 | grep -q ^vllm:; then
echo FAIL: UID 2000 != vllm
exit 1
fi
if ! id -gn 2>/dev/null | grep -qx root; then
echo FAIL: GID 0 not root group
exit 1
fi
touch /home/vllm/.smoke && rm /home/vllm/.smoke
touch /opt/uv/cache/.smoke && rm /opt/uv/cache/.smoke
if ! test -x /usr/local/bin/vllm-nonroot-entrypoint.sh; then
echo FAIL: wrapper missing
exit 1
fi
if ! test -w /etc/passwd; then
echo FAIL: /etc/passwd not group-writable
exit 1
fi
if ! test -w /etc/group; then
echo FAIL: /etc/group not group-writable
exit 1
fi
echo non-root invariants OK
'
retry:
automatic:
- exit_status: -1 # Agent was lost
limit: 2
- exit_status: -10 # Agent was lost
limit: 2
- label: ":docker: Build CPU image"
key: image-build-cpu
depends_on: []
commands:
- .buildkite/image_build/image_build_cpu.sh $REGISTRY $REPO $BUILDKITE_COMMIT
env:
DOCKER_BUILDKIT: "1"
retry:
automatic:
- exit_status: -1 # Agent was lost
limit: 2
- exit_status: -10 # Agent was lost
limit: 2
- label: ":docker: Build HPU image"
soft_fail: true
depends_on: []
key: image-build-hpu
commands:
- .buildkite/image_build/image_build_hpu.sh $REGISTRY $REPO $BUILDKITE_COMMIT
env:
DOCKER_BUILDKIT: "1"
retry:
automatic:
- exit_status: -1 # Agent was lost
limit: 2
- exit_status: -10 # Agent was lost
limit: 2
- label: ":docker: Build CPU arm64 image"
key: cpu-arm64-image-build
depends_on: []
optional: true
commands:
- .buildkite/image_build/image_build_cpu_arm64.sh $REGISTRY $REPO $BUILDKITE_COMMIT
env:
DOCKER_BUILDKIT: "1"
retry:
automatic:
- exit_status: -1 # Agent was lost
limit: 2
- exit_status: -10 # Agent was lost
limit: 2
- label: ":docker: Build arm64 image"
key: arm64-image-build
depends_on: []
source_file_dependencies:
- ".buildkite/image_build/image_build.yaml"
- ".buildkite/image_build/image_build_arm64.sh"
- "docker/Dockerfile"
commands:
- .buildkite/image_build/image_build_arm64.sh $REGISTRY $REPO $BUILDKITE_COMMIT
env:
DOCKER_BUILDKIT: "1"
retry:
automatic:
- exit_status: -1 # Agent was lost
limit: 2
- exit_status: -10 # Agent was lost
limit: 2
@@ -1,51 +0,0 @@
#!/bin/bash
set -e
if [[ $# -lt 3 ]]; then
echo "Usage: $0 <registry> <repo> <commit>"
exit 1
fi
REGISTRY=$1
REPO=$2
BUILDKITE_COMMIT=$3
# When TORCH_NIGHTLY=1, build against torch nightly and tag it with the
# -torch-nightly-arm64 suffix the DGX Spark test steps pull on the nightly lane
# (ci-infra get_image(arm64=True) appends -torch-nightly before -arm64).
PYTORCH_NIGHTLY_ARGS=()
if [[ "${TORCH_NIGHTLY:-0}" == "1" ]]; then
IMAGE="$REGISTRY/$REPO:$BUILDKITE_COMMIT-torch-nightly-arm64"
PYTORCH_NIGHTLY_ARGS=(--build-arg PYTORCH_NIGHTLY=1)
else
IMAGE="$REGISTRY/$REPO:$BUILDKITE_COMMIT-arm64"
fi
# authenticate with AWS ECR
aws ecr-public get-login-password --region us-east-1 | docker login --username AWS --password-stdin "$REGISTRY" || true
# skip build if image already exists
if docker manifest inspect "$IMAGE" >/dev/null 2>&1; then
echo "Image found"
else
echo "Image not found, proceeding with build..."
# build for arm64 GPU targets: Grace/GH200 (sm_90),
# Blackwell/Thor (sm_100/sm_103/sm_110), and DGX Spark/GB10
# (sm_121, family-covered by 12.0 under CUDA 13)
docker build --file docker/Dockerfile \
--platform linux/arm64 \
--build-arg max_jobs=16 \
--build-arg nvcc_threads=4 \
--build-arg BUILD_BASE_IMAGE=pytorch/manylinuxaarch64-builder:cuda13.0-b8b5f17a7d9ccfc25bbc5cf17b3fcea12964a042 \
--build-arg torch_cuda_arch_list="9.0 10.0 11.0 12.0" \
--build-arg USE_SCCACHE=1 \
--build-arg buildkite_commit="$BUILDKITE_COMMIT" \
"${PYTORCH_NIGHTLY_ARGS[@]}" \
--tag "$IMAGE" \
--target test \
--progress plain .
# push
docker push "$IMAGE"
fi
.buildkite/scripts/annotate-image-build.sh "$IMAGE"
-118
View File
@@ -1,118 +0,0 @@
#!/bin/bash
set -e
if [[ $# -lt 3 ]]; then
echo "Usage: $0 <registry> <repo> <commit>"
exit 1
fi
REGISTRY=$1
REPO=$2
BUILDKITE_COMMIT=$3
# When TORCH_NIGHTLY=1, build the CPU image against torch nightly and tag it
# with the -torch-nightly-cpu suffix the test steps pull on the nightly lane.
PYTORCH_NIGHTLY_ARGS=()
if [[ "${TORCH_NIGHTLY:-0}" == "1" ]]; then
IMAGE="$REGISTRY/$REPO:$BUILDKITE_COMMIT-torch-nightly-cpu"
PYTORCH_NIGHTLY_ARGS=(--build-arg PYTORCH_NIGHTLY=1)
else
IMAGE="$REGISTRY/$REPO:$BUILDKITE_COMMIT-cpu"
fi
# replace invalid characters in Docker image tags and truncate to 128 chars
clean_docker_tag() {
local input="$1"
echo "$input" | sed 's/[^a-zA-Z0-9._-]/_/g' | cut -c1-128
}
# resolve and set: CACHE_TO, CACHE_FROM, CACHE_FROM_BASE_BRANCH, CACHE_FROM_MAIN
# Reuses the same ECR cache repos as the CUDA image build, with an
# "x86_cpu" suffix on every tag so CPU and CUDA cache blobs stay isolated.
# The nightly lane appends "-nightly": it resolves a different torch, so
# sharing a cache ref with the regular lane would have the two evict each
# other from the same ref on every run.
prepare_cache_tags() {
TEST_CACHE_ECR="936637512419.dkr.ecr.us-east-1.amazonaws.com/vllm-ci-test-cache"
MAIN_CACHE_ECR="936637512419.dkr.ecr.us-east-1.amazonaws.com/vllm-ci-postmerge-cache"
CACHE_SUFFIX="x86_cpu"
if [[ "${TORCH_NIGHTLY:-0}" == "1" ]]; then
CACHE_SUFFIX="x86_cpu-nightly"
fi
if [[ "${BUILDKITE_PULL_REQUEST:-false}" == "false" ]]; then
if [[ "${BUILDKITE_BRANCH:-}" == "main" ]]; then
cache="${MAIN_CACHE_ECR}:latest-${CACHE_SUFFIX}"
else
clean_branch=$(clean_docker_tag "${BUILDKITE_BRANCH:-unknown}")
cache="${TEST_CACHE_ECR}:${clean_branch}-${CACHE_SUFFIX}"
fi
CACHE_TO="$cache"
CACHE_FROM="$cache"
CACHE_FROM_BASE_BRANCH="$cache"
else
CACHE_TO="${TEST_CACHE_ECR}:pr-${BUILDKITE_PULL_REQUEST}-${CACHE_SUFFIX}"
CACHE_FROM="${TEST_CACHE_ECR}:pr-${BUILDKITE_PULL_REQUEST}-${CACHE_SUFFIX}"
if [[ "${BUILDKITE_PULL_REQUEST_BASE_BRANCH:-main}" == "main" ]]; then
CACHE_FROM_BASE_BRANCH="${MAIN_CACHE_ECR}:latest-${CACHE_SUFFIX}"
else
clean_base=$(clean_docker_tag "${BUILDKITE_PULL_REQUEST_BASE_BRANCH}")
CACHE_FROM_BASE_BRANCH="${TEST_CACHE_ECR}:${clean_base}-${CACHE_SUFFIX}"
fi
fi
CACHE_FROM_MAIN="${MAIN_CACHE_ECR}:latest-${CACHE_SUFFIX}"
}
# authenticate with AWS ECR (public, for the image; private, for the cache repos)
aws ecr-public get-login-password --region us-east-1 | docker login --username AWS --password-stdin "$REGISTRY" || true
aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin 936637512419.dkr.ecr.us-east-1.amazonaws.com || true
# skip build if image already exists
if docker manifest inspect "$IMAGE" >/dev/null 2>&1; then
echo "Image found"
else
echo "Image not found, proceeding with build..."
prepare_cache_tags
echo "--- :mag: Cache tags"
echo "CACHE_TO: ${CACHE_TO}"
echo "CACHE_FROM: ${CACHE_FROM}"
echo "CACHE_FROM_BASE_BRANCH: ${CACHE_FROM_BASE_BRANCH}"
echo "CACHE_FROM_MAIN: ${CACHE_FROM_MAIN}"
# dedupe cache-from refs so identical branch/main fallbacks aren't repeated
CACHE_FROM_ARGS=()
SEEN_REFS=""
for ref in "$CACHE_FROM" "$CACHE_FROM_BASE_BRANCH" "$CACHE_FROM_MAIN"; do
if [[ "$SEEN_REFS" != *"|${ref}|"* ]]; then
CACHE_FROM_ARGS+=(--cache-from "type=registry,ref=${ref}")
SEEN_REFS="${SEEN_REFS}|${ref}|"
fi
done
echo "--- :docker: Setting up buildx"
# network=host so the isolated docker-container builder can still reach the
# EC2 instance metadata service for sccache's S3 credentials; recreate on
# every run in case a stale builder without this driver-opt already exists
# on a warm CI host.
docker buildx rm vllm-cpu-builder >/dev/null 2>&1 || true
docker buildx create --name vllm-cpu-builder --driver docker-container --driver-opt network=host --use
docker buildx inspect --bootstrap
# build and push
docker buildx build --file docker/Dockerfile.cpu \
--build-arg max_jobs=16 \
--build-arg buildkite_commit="$BUILDKITE_COMMIT" \
--build-arg VLLM_CPU_X86=true \
--build-arg USE_SCCACHE=1 \
"${PYTORCH_NIGHTLY_ARGS[@]}" \
--tag "$IMAGE" \
--target vllm-test \
"${CACHE_FROM_ARGS[@]}" \
--cache-to "type=registry,ref=${CACHE_TO},mode=max" \
--push \
--progress plain .
fi
.buildkite/scripts/annotate-image-build.sh "$IMAGE"
@@ -1,33 +0,0 @@
#!/bin/bash
set -e
if [[ $# -lt 3 ]]; then
echo "Usage: $0 <registry> <repo> <commit>"
exit 1
fi
REGISTRY=$1
REPO=$2
BUILDKITE_COMMIT=$3
IMAGE="$REGISTRY/$REPO:$BUILDKITE_COMMIT-arm64-cpu"
# authenticate with AWS ECR
aws ecr-public get-login-password --region us-east-1 | docker login --username AWS --password-stdin "$REGISTRY" || true
# skip build if image already exists
if docker manifest inspect "$IMAGE" >/dev/null 2>&1; then
echo "Image found"
else
echo "Image not found, proceeding with build..."
# build
docker build --file docker/Dockerfile.cpu \
--build-arg max_jobs=16 \
--build-arg buildkite_commit="$BUILDKITE_COMMIT" \
--tag "$IMAGE" \
--target vllm-test \
--progress plain .
# push
docker push "$IMAGE"
fi
.buildkite/scripts/annotate-image-build.sh "$IMAGE"
-34
View File
@@ -1,34 +0,0 @@
#!/bin/bash
set -e
if [[ $# -lt 3 ]]; then
echo "Usage: $0 <registry> <repo> <commit>"
exit 1
fi
REGISTRY=$1
REPO=$2
BUILDKITE_COMMIT=$3
IMAGE="$REGISTRY/$REPO:$BUILDKITE_COMMIT-hpu"
# authenticate with AWS ECR
aws ecr-public get-login-password --region us-east-1 | docker login --username AWS --password-stdin "$REGISTRY" || true
# skip build if image already exists
if docker manifest inspect "$IMAGE" >/dev/null 2>&1; then
echo "Image found"
else
echo "Image not found, proceeding with build..."
# build
docker build \
--file tests/pytorch_ci_hud_benchmark/Dockerfile.hpu \
--build-arg max_jobs=16 \
--build-arg buildkite_commit="$BUILDKITE_COMMIT" \
--tag "$IMAGE" \
--progress plain \
https://github.com/vllm-project/vllm-gaudi.git
# push
docker push "$IMAGE"
fi
.buildkite/scripts/annotate-image-build.sh "$IMAGE"
@@ -1,80 +0,0 @@
#!/bin/bash
set -euo pipefail
# Build a vLLM test image with PyTorch nightly installed.
# Called by the pipeline generator's "vLLM Against PyTorch Nightly" group.
if [[ $# -lt 5 ]]; then
echo "Usage: $0 <registry> <repo> <commit> <branch> <image_tag>"
exit 1
fi
REGISTRY=$1
REPO=$2
BUILDKITE_COMMIT=$3
BRANCH=$4
IMAGE_TAG=$5
# --- Arguments ---
echo "--- :mag: Arguments"
echo "REGISTRY: ${REGISTRY}"
echo "REPO: ${REPO}"
echo "BUILDKITE_COMMIT: ${BUILDKITE_COMMIT}"
echo "BRANCH: ${BRANCH}"
echo "IMAGE_TAG: ${IMAGE_TAG}"
# --- ECR login ---
echo "--- :key: ECR login"
aws ecr-public get-login-password --region us-east-1 \
| docker login --username AWS --password-stdin "$REGISTRY"
aws ecr get-login-password --region us-east-1 \
| docker login --username AWS --password-stdin 936637512419.dkr.ecr.us-east-1.amazonaws.com
# --- Set up buildx ---
echo "--- :docker: Setting up buildx"
docker buildx create --name vllm-builder --driver docker-container --use || true
docker buildx inspect --bootstrap
docker buildx ls
# --- Skip if image already exists ---
echo "--- :mag: Checking if image already exists"
if docker manifest inspect "$IMAGE_TAG" >/dev/null 2>&1; then
echo "Image found: $IMAGE_TAG — skipping build"
.buildkite/scripts/annotate-image-build.sh "$IMAGE_TAG"
exit 0
fi
echo "Image not found, proceeding with build..."
# --- CUDA 13.0 for nightly builds ---
# Nightly CI uses CUDA 13.0 while regular CI stays on CUDA 12.9
NIGHTLY_CUDA_VERSION="13.0.2"
NIGHTLY_BUILD_BASE_IMAGE="pytorch/manylinux2_28-builder:cuda13.0-78e737ad29420ffc4800e677c51e2a852caf8359"
# Must track ARG UBUNTU_VERSION in docker/Dockerfile. The nightly lane exists to
# run the normal pipeline against nightly torch, so it has to differ from the
# regular image in torch only -- not in the OS underneath it. This was pinned to
# 22.04 while everything else moved to 24.04, and the runtime dependency list in
# tools/install_snapshot_runtime.sh is 24.04-only (libgnutls30t64 is the 64-bit
# time_t rename and does not exist on jammy), so the nightly build failed with
# "E: Unable to locate package libgnutls30t64" while regular CI stayed green on
# a cached layer.
NIGHTLY_UBUNTU_VERSION="24.04"
NIGHTLY_FINAL_BASE_IMAGE="nvidia/cuda:${NIGHTLY_CUDA_VERSION}-base-ubuntu${NIGHTLY_UBUNTU_VERSION}"
echo "--- :docker: Building torch nightly image (CUDA ${NIGHTLY_CUDA_VERSION})"
docker buildx build --file docker/Dockerfile \
--build-arg max_jobs=16 \
--build-arg buildkite_commit="$BUILDKITE_COMMIT" \
--build-arg USE_SCCACHE=1 \
--build-arg PYTORCH_NIGHTLY=1 \
--build-arg CUDA_VERSION="${NIGHTLY_CUDA_VERSION}" \
--build-arg BUILD_BASE_IMAGE="${NIGHTLY_BUILD_BASE_IMAGE}" \
--build-arg FINAL_BASE_IMAGE="${NIGHTLY_FINAL_BASE_IMAGE}" \
--build-arg torch_cuda_arch_list="8.0 8.9 9.0 10.0 12.0" \
--tag "$IMAGE_TAG" \
--push \
--target test \
--progress plain .
echo "--- :white_check_mark: Torch nightly image build complete: $IMAGE_TAG"
.buildkite/scripts/annotate-image-build.sh "$IMAGE_TAG"
-34
View File
@@ -1,34 +0,0 @@
#!/bin/bash
set -e
if [[ $# -lt 3 ]]; then
echo "Usage: $0 <registry> <repo> <commit>"
exit 1
fi
REGISTRY=$1
REPO=$2
BUILDKITE_COMMIT=$3
IMAGE="$REGISTRY/$REPO:$BUILDKITE_COMMIT-xpu"
# authenticate with AWS ECR
aws ecr-public get-login-password --region us-east-1 | docker login --username AWS --password-stdin "$REGISTRY" || true
aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin 936637512419.dkr.ecr.us-east-1.amazonaws.com || true
# skip build if image already exists
if docker manifest inspect "$IMAGE" &> /dev/null; then
echo "Image found"
else
echo "Image not found, proceeding with build..."
# build
docker build \
--file docker/Dockerfile.xpu \
--build-arg max_jobs=16 \
--build-arg buildkite_commit="$BUILDKITE_COMMIT" \
--tag "$IMAGE" \
--progress plain .
# push
docker push "$IMAGE"
fi
.buildkite/scripts/annotate-image-build.sh "$IMAGE"
-3
View File
@@ -1,3 +0,0 @@
target "test-ci" {
output = ["type=registry,compression=zstd,compression-level=3,force-compression=true,oci-mediatypes=true"]
}
@@ -1,29 +0,0 @@
group: Basic Correctness
depends_on:
- image-build-xpu
steps:
- label: XPU Sleep Mode
key: xpu-sleep-mode
timeout_in_minutes: 45
device: intel_gpu
agent_tags:
label: production
gpu: 2+
mem: 16+
no_plugin: true
working_dir: "."
env:
REGISTRY: "public.ecr.aws/q9t5s3a7"
REPO: "vllm-ci-test-repo"
VLLM_TEST_DEVICE: "xpu"
source_file_dependencies:
- vllm/
- tests/basic_correctness/memory/
- tests/basic_correctness/cpu_offload/
commands:
- >-
bash .buildkite/scripts/hardware_ci/run-intel-test.sh
'cd tests &&
export VLLM_WORKER_MULTIPROC_METHOD=spawn &&
pytest -v -s basic_correctness/cpu_offload &&
pytest -v -s basic_correctness/memory'
@@ -1,26 +0,0 @@
group: Benchmarks
depends_on:
- image-build-xpu
steps:
- label: Benchmarks CLI Test
key: benchmarks-cli-test
timeout_in_minutes: 40
device: intel_gpu
agent_tags:
label: production
gpu: 1+
mem: 16+
no_plugin: true
working_dir: "."
env:
REGISTRY: "public.ecr.aws/q9t5s3a7"
REPO: "vllm-ci-test-repo"
VLLM_TEST_DEVICE: "xpu"
source_file_dependencies:
- vllm/
- tests/benchmarks/
commands:
- >-
bash .buildkite/scripts/hardware_ci/run-intel-test.sh
'cd tests &&
pytest -v -s benchmarks/'
-107
View File
@@ -1,107 +0,0 @@
group: Engine Intel
depends_on:
- image-build-xpu
steps:
- label: Engine
key: engine
timeout_in_minutes: 40
device: intel_gpu
agent_tags:
label: production
gpu: 1+
mem: 16+
no_plugin: true
working_dir: "."
env:
REGISTRY: "public.ecr.aws/q9t5s3a7"
REPO: "vllm-ci-test-repo"
VLLM_TEST_DEVICE: "xpu"
source_file_dependencies:
- vllm/compilation/
- vllm/config/
- vllm/engine/
- vllm/entrypoints/logger.py
- vllm/envs.py
- vllm/logger.py
- vllm/logging_utils/
- vllm/platforms/
- vllm/sequence.py
- vllm/triton_utils/
- vllm/utils/
- tests/engine
- tests/test_sequence
- tests/test_config
- tests/test_logger
- tests/test_vllm_port
- tests/jit_monitor/test_hooks.py
commands:
- >-
bash .buildkite/scripts/hardware_ci/run-intel-test.sh
'cd tests &&
pytest -v -s engine/test_arg_utils.py test_sequence.py test_logger.py test_vllm_port.py jit_monitor/test_hooks.py'
- label: Engine (1 GPU)
key: engine-1-gpu
timeout_in_minutes: 30
device: intel_gpu
agent_tags:
label: production
gpu: 1+
mem: 16+
no_plugin: true
working_dir: "."
env:
REGISTRY: "public.ecr.aws/q9t5s3a7"
REPO: "vllm-ci-test-repo"
VLLM_TEST_DEVICE: "xpu"
source_file_dependencies:
- vllm/v1/engine/
- tests/v1/engine/
- tests/test_config/
commands:
- >-
bash .buildkite/scripts/hardware_ci/run-intel-test.sh
'cd tests &&
pytest -v -s v1/engine --ignore v1/engine/test_preprocess_error_handling.py --ignore v1/engine/test_core_engine_actor_manager.py &&
pytest -v -s test_config.py'
- label: V1 e2e (2 GPUs)
key: v1-e2e-2-gpus
timeout_in_minutes: 30
device: intel_gpu
agent_tags:
label: production
gpu: 2+
mem: 16+
no_plugin: true
working_dir: "."
env:
REGISTRY: "public.ecr.aws/q9t5s3a7"
REPO: "vllm-ci-test-repo"
VLLM_TEST_DEVICE: "xpu"
source_file_dependencies:
- vllm/compilation/
- vllm/config/
- vllm/distributed/
- vllm/engine/
- vllm/envs.py
- vllm/forward_context.py
- vllm/inputs/
- vllm/logger.py
- vllm/logging_utils/
- vllm/model_executor/
- vllm/multimodal/
- vllm/platforms/
- vllm/sampling_params.py
- vllm/transformers_utils/
- vllm/triton_utils/
- vllm/utils/
- vllm/v1/
- tests/v1/e2e/spec_decode/
commands:
- >-
bash .buildkite/scripts/hardware_ci/run-intel-test.sh
'cd tests &&
pytest -v -s
v1/e2e/spec_decode/draft_model/test_draft_model.py::test_draft_model_tensor_parallelism
v1/e2e/spec_decode/draft_model/test_draft_model.py::test_draft_model_engine_args_tensor_parallelism'
@@ -1,191 +0,0 @@
group: Entrypoints
depends_on:
- image-build-xpu
steps:
- label: ":intel: (BMG) Entrypoints Unit"
key: entrypoints-unit-tests-intel
timeout_in_minutes: 45
device: intel_gpu
agent_tags:
label: production
gpu: 1+
mem: 16+
no_plugin: true
working_dir: "."
env:
REGISTRY: "public.ecr.aws/q9t5s3a7"
REPO: "vllm-ci-test-repo"
VLLM_TEST_DEVICE: "xpu"
source_file_dependencies:
- vllm/entrypoints
- tests/entrypoints/unit_tests
- tests/entrypoints/weight_transfer
- tests/entrypoints/launchers
commands:
- >-
bash .buildkite/scripts/hardware_ci/run-intel-test.sh
'cd tests &&
pytest -v -s entrypoints/unit_tests &&
pytest -v -s entrypoints/weight_transfer'
- label: ":intel: (BMG) Entrypoints Integration (LLM)"
key: entrypoints-integration-llm-intel
timeout_in_minutes: 70
device: intel_gpu
agent_tags:
label: production
gpu: 1+
mem: 16+
no_plugin: true
working_dir: "."
env:
REGISTRY: "public.ecr.aws/q9t5s3a7"
REPO: "vllm-ci-test-repo"
VLLM_TEST_DEVICE: "xpu"
source_file_dependencies:
- vllm/
- tests/entrypoints/llm
commands:
- >-
bash .buildkite/scripts/hardware_ci/run-intel-test.sh
'export VLLM_WORKER_MULTIPROC_METHOD=spawn &&
cd tests &&
pytest -v -s entrypoints/llm/test_generate.py &&
pytest -v -s entrypoints/llm/offline_mode'
- label: ":intel: (BMG) Entrypoints Integration (API Server)"
key: entrypoints-integration-api-server-intel
timeout_in_minutes: 70
device: intel_gpu
agent_tags:
label: production
gpu: 1+
mem: 16+
no_plugin: true
working_dir: "."
env:
REGISTRY: "public.ecr.aws/q9t5s3a7"
REPO: "vllm-ci-test-repo"
VLLM_TEST_DEVICE: "xpu"
source_file_dependencies:
- vllm/
- "!vllm/distributed/kv_transfer/"
- tests/entrypoints/serve
commands:
- >-
bash .buildkite/scripts/hardware_ci/run-intel-test.sh
'export VLLM_WORKER_MULTIPROC_METHOD=spawn &&
cd tests &&
pytest -v -s entrypoints/serve --ignore=entrypoints/serve/dev/rpc
--ignore=entrypoints/serve/dev/rlhf/test_weight_operation_metrics.py
--deselect="tests/entrypoints/serve/dev/rlhf/state_transitions/test_pause_resume.py::TestPauseResume::test_clear_cache_preserves_output_and_controls_prefix_cache[MRV1]" &&
PYTHONPATH=/workspace/vllm pytest -v -s entrypoints/serve/dev/rpc'
- label: ":intel: (BMG) Entrypoints Integration (API Server Scale Out)"
key: entrypoints-integration-api-server-scale-out-intel
timeout_in_minutes: 40
device: intel_gpu
agent_tags:
label: production
gpu: 1+
mem: 16+
no_plugin: true
working_dir: "."
env:
REGISTRY: "public.ecr.aws/q9t5s3a7"
REPO: "vllm-ci-test-repo"
VLLM_TEST_DEVICE: "xpu"
source_file_dependencies:
- vllm/
- "!vllm/distributed/kv_transfer/"
- tests/entrypoints/scale_out
commands:
- >-
bash .buildkite/scripts/hardware_ci/run-intel-test.sh
'export VLLM_WORKER_MULTIPROC_METHOD=spawn &&
cd tests &&
pytest -v -s entrypoints/scale_out'
- label: ":intel: (BMG) Entrypoints Integration (API Server Generate)"
key: entrypoints-integration-api-server-generate-intel
timeout_in_minutes: 65
device: intel_gpu
agent_tags:
label: production
gpu: 1+
mem: 24+
no_plugin: true
working_dir: "."
env:
REGISTRY: "public.ecr.aws/q9t5s3a7"
REPO: "vllm-ci-test-repo"
VLLM_TEST_DEVICE: "xpu"
optional: true
source_file_dependencies:
- vllm/
- "!vllm/distributed/kv_transfer/"
- tests/tool_use
- tests/entrypoints/tool_parsers
- tests/entrypoints/generate
- tests/entrypoints/anthropic
- tests/entrypoints/cohere
commands:
- >-
bash .buildkite/scripts/hardware_ci/run-intel-test.sh
'cd tests &&
pytest -v -s tool_use
--deselect="tests/tool_use/mistral/test_mistral_tool_calls.py::test_tool_call_none_with_tools[ministral-3b]"
--deselect="tests/tool_use/test_chat_completions.py::test_chat_completion_with_tools[granite-3.0-8b]"
--deselect="tests/tool_use/test_chat_completions.py::test_chat_completion_with_tools[llama3.2]" &&
pytest -v -s entrypoints/tool_parsers &&
pytest -v -s entrypoints/generate &&
pytest -v -s entrypoints/anthropic &&
pytest -v -s entrypoints/cohere'
- label: ":intel: (BMG) Entrypoints Integration (Responses API)"
key: entrypoints-integration-responses-api-intel
timeout_in_minutes: 60
device: intel_gpu
agent_tags:
label: production
gpu: 1+
mem: 24+
no_plugin: true
working_dir: "."
env:
REGISTRY: "public.ecr.aws/q9t5s3a7"
REPO: "vllm-ci-test-repo"
VLLM_TEST_DEVICE: "xpu"
optional: true
source_file_dependencies:
- vllm/
- "!vllm/distributed/kv_transfer/"
- tests/entrypoints/openai/responses
commands:
- >-
bash .buildkite/scripts/hardware_ci/run-intel-test.sh
'cd tests &&
pytest -v -s entrypoints/openai/responses'
- label: ":intel: (BMG) OpenAI API Correctness"
key: openai-api-correctness-intel
timeout_in_minutes: 30
device: intel_gpu
agent_tags:
label: production
gpu: 1+
mem: 16+
no_plugin: true
working_dir: "."
env:
REGISTRY: "public.ecr.aws/q9t5s3a7"
REPO: "vllm-ci-test-repo"
VLLM_TEST_DEVICE: "xpu"
source_file_dependencies:
- csrc/
- vllm/entrypoints/openai/
commands:
- >-
bash .buildkite/scripts/hardware_ci/run-intel-test.sh
'cd tests &&
pytest -v -s entrypoints/openai/correctness/'
@@ -1,28 +0,0 @@
group: Expert Parallelism
depends_on:
- image-build-xpu
steps:
- label: EPLB Algorithm
key: eplb-algorithm
timeout_in_minutes: 45
device: intel_gpu
agent_tags:
label: production
gpu: 1+
mem: 16+
no_plugin: true
working_dir: "."
env:
REGISTRY: "public.ecr.aws/q9t5s3a7"
REPO: "vllm-ci-test-repo"
VLLM_TEST_DEVICE: "xpu"
source_file_dependencies:
- vllm/distributed/eplb
- tests/distributed/test_eplb_algo.py
- tests/distributed/test_eplb_utils.py
commands:
- >-
bash .buildkite/scripts/hardware_ci/run-intel-test.sh
'cd tests &&
pytest -v -s distributed/test_eplb_algo.py &&
pytest -v -s distributed/test_eplb_utils.py'
-50
View File
@@ -1,50 +0,0 @@
group: Kernels Intel
depends_on:
- image-build-xpu
steps:
- label: vLLM IR Tests
key: vllm-ir-tests
timeout_in_minutes: 30
device: intel_gpu
agent_tags:
label: production
gpu: 1+
mem: 16+
no_plugin: true
working_dir: "."
env:
REGISTRY: "public.ecr.aws/q9t5s3a7"
REPO: "vllm-ci-test-repo"
VLLM_TEST_DEVICE: "xpu"
source_file_dependencies:
- vllm/ir
- vllm/kernels
commands:
- >-
bash .buildkite/scripts/hardware_ci/run-intel-test.sh
'cd tests &&
pytest -v -s ir &&
pytest -v -s kernels/ir'
- label: UVA Kernel Tests
key: uva-kernel-tests
timeout_in_minutes: 20
device: intel_gpu
agent_tags:
label: production
gpu: 1+
mem: 16+
no_plugin: true
working_dir: "."
env:
REGISTRY: "public.ecr.aws/q9t5s3a7"
REPO: "vllm-ci-test-repo"
VLLM_TEST_DEVICE: "xpu"
source_file_dependencies:
- tests/kernels/core/test_uva.py
- vllm/utils/torch_utils.py
commands:
- >-
bash .buildkite/scripts/hardware_ci/run-intel-test.sh
'cd tests &&
pytest -v -s kernels/core/test_uva.py'
-51
View File
@@ -1,51 +0,0 @@
group: LM Eval
depends_on:
- image-build-xpu
steps:
- label: "XPU GPQA Eval (GPT-OSS)"
key: xpu-gpqa-eval-gpt-oss
timeout_in_minutes: 60
device: intel_gpu
agent_tags:
label: production
gpu: 1+
mem: 24+
no_plugin: true
working_dir: "."
env:
REGISTRY: "public.ecr.aws/q9t5s3a7"
REPO: "vllm-ci-test-repo"
VLLM_TEST_DEVICE: "xpu"
source_file_dependencies:
- csrc/
- vllm/model_executor/layers/quantization
- tests/evals/gpt_oss/
commands:
- >-
bash .buildkite/scripts/hardware_ci/run-intel-test.sh
'cd tests &&
pytest -s -v evals/gpt_oss/test_gpqa_correctness.py --config-list-file=configs/models-xpu.txt'
- label: LM Eval Large Models(2 GPUs)
key: lm-eval-large-models-2-gpus
timeout_in_minutes: 45
device: intel_gpu
agent_tags:
label: production
gpu: 2+
mem: 24+
no_plugin: true
optional: true
working_dir: "."
env:
REGISTRY: "public.ecr.aws/q9t5s3a7"
REPO: "vllm-ci-test-repo"
VLLM_TEST_DEVICE: "xpu"
source_file_dependencies:
- vllm/
- .buildkite/lm-eval-harness/
commands:
- >-
bash .buildkite/scripts/hardware_ci/run-intel-test.sh
'cd .buildkite/lm-eval-harness &&
export VLLM_USE_DEEP_GEMM=0 &&
pytest -s -v test_lm_eval_correctness.py --config-list-file=configs/models-large-xpu-fp8.txt --tp-size=2'
-167
View File
@@ -1,167 +0,0 @@
group: LoRA Intel
depends_on:
- image-build-xpu
steps:
- label: LoRA Runtime + Utils
key: lora-runtime-utils
timeout_in_minutes: 45
device: intel_gpu
agent_tags:
label: production
gpu: 1+
mem: 24+
no_plugin: true
working_dir: "."
env:
REGISTRY: "public.ecr.aws/q9t5s3a7"
REPO: "vllm-ci-test-repo"
VLLM_TEST_DEVICE: "xpu"
source_file_dependencies:
- vllm/lora
- tests/lora
commands:
- >-
bash .buildkite/scripts/hardware_ci/run-intel-test.sh
'cd tests &&
export VLLM_WORKER_MULTIPROC_METHOD=spawn &&
pytest -v -s lora/test_layers.py &&
pytest -v -s lora/test_lora_checkpoints.py &&
pytest -v -s lora/test_lora_functions.py &&
pytest -v -s lora/test_lora_huggingface.py &&
pytest -v -s lora/test_lora_manager.py &&
pytest -v -s lora/test_lora_utils.py &&
pytest -v -s lora/test_peft_helper.py &&
pytest -v -s lora/test_resolver.py &&
pytest -v -s lora/test_utils.py &&
pytest -v -s lora/test_add_lora.py &&
pytest -v -s lora/test_worker.py'
- label: LoRA Fused/MoE Kernels
key: lora-fused-moe-kernels
timeout_in_minutes: 45
device: intel_gpu
agent_tags:
label: production
gpu: 1+
mem: 16+
no_plugin: true
working_dir: "."
env:
REGISTRY: "public.ecr.aws/q9t5s3a7"
REPO: "vllm-ci-test-repo"
VLLM_TEST_DEVICE: "xpu"
source_file_dependencies:
- vllm/lora
- tests/lora
commands:
- >-
bash .buildkite/scripts/hardware_ci/run-intel-test.sh
'cd tests &&
export VLLM_WORKER_MULTIPROC_METHOD=spawn &&
pytest -v -s lora/test_fused_moe_lora_kernel.py &&
pytest -v -s lora/test_moe_lora_align_sum.py --deselect="tests/lora/test_moe_lora_align_sum.py::test_moe_lora_align_block_size_mixed_base_and_lora[1]"'
- label: LoRA Punica Kernels
key: lora-punica-kernels
timeout_in_minutes: 45
device: intel_gpu
agent_tags:
label: production
gpu: 1+
mem: 16+
no_plugin: true
working_dir: "."
env:
REGISTRY: "public.ecr.aws/q9t5s3a7"
REPO: "vllm-ci-test-repo"
VLLM_TEST_DEVICE: "xpu"
source_file_dependencies:
- vllm/lora
- tests/lora
commands:
- >-
bash .buildkite/scripts/hardware_ci/run-intel-test.sh
'cd tests &&
export VLLM_WORKER_MULTIPROC_METHOD=spawn &&
set -o pipefail &&
pytest -v -s lora/test_punica_ops.py::test_kernels &&
pytest -v -s lora/test_punica_ops.py::test_kernels_hidden_size &&
pytest -v -s lora/test_punica_ops.py::test_add_lora_fused_moe_early_exit'
- label: LoRA Punica FP8/XPU Ops
key: lora-punica-fp8-xpu-ops
timeout_in_minutes: 60
device: intel_gpu
agent_tags:
label: production
gpu: 1+
mem: 16+
no_plugin: true
working_dir: "."
env:
REGISTRY: "public.ecr.aws/q9t5s3a7"
REPO: "vllm-ci-test-repo"
VLLM_TEST_DEVICE: "xpu"
source_file_dependencies:
- vllm/lora
- tests/lora
commands:
- >-
bash .buildkite/scripts/hardware_ci/run-intel-test.sh
'cd tests &&
export VLLM_WORKER_MULTIPROC_METHOD=spawn &&
pytest -v -s lora/test_punica_ops_fp8.py &&
pytest -v -s lora/test_punica_xpu_ops.py'
- label: LoRA Models
key: lora-models
timeout_in_minutes: 45
device: intel_gpu
agent_tags:
label: production
gpu: 2+
mem: 24+
no_plugin: true
working_dir: "."
env:
REGISTRY: "public.ecr.aws/q9t5s3a7"
REPO: "vllm-ci-test-repo"
VLLM_TEST_DEVICE: "xpu"
source_file_dependencies:
- vllm/lora
- tests/lora
commands:
- >-
bash .buildkite/scripts/hardware_ci/run-intel-test.sh
'cd tests &&
export VLLM_WORKER_MULTIPROC_METHOD=spawn &&
pytest -v -s lora/test_quant_model.py --deselect="tests/lora/test_quant_model.py::test_quant_model_lora[model0]" --deselect="tests/lora/test_quant_model.py::test_quant_model_lora[model1]" --deselect="tests/lora/test_quant_model.py::test_quant_model_tp_equality[model0]" &&
pytest -v -s lora/test_transformers_model.py &&
pytest -v -s lora/test_chatglm3_tp.py &&
pytest -v -s lora/test_llama_tp.py::test_llama_lora &&
pytest -s -v lora/test_minicpmv_tp.py'
- label: LoRA Multimodal
key: lora-multimodal
timeout_in_minutes: 45
device: intel_gpu
agent_tags:
label: production
gpu: 1+
mem: 24+
no_plugin: true
working_dir: "."
env:
REGISTRY: "public.ecr.aws/q9t5s3a7"
REPO: "vllm-ci-test-repo"
VLLM_TEST_DEVICE: "xpu"
source_file_dependencies:
- vllm/lora
- tests/lora
commands:
- >-
bash .buildkite/scripts/hardware_ci/run-intel-test.sh
'cd tests &&
export VLLM_WORKER_MULTIPROC_METHOD=spawn &&
pytest -v -s lora/test_default_mm_loras.py &&
pytest -v -s lora/test_whisper.py'
-321
View File
@@ -1,321 +0,0 @@
group: Miscellaneous Intel
depends_on:
- image-build-xpu
steps:
- label: XPU Batch Invariance
key: xpu-batch-invariance
timeout_in_minutes: 60
num_devices: 2
device: intel_gpu
agent_tags:
label: production
gpu: 2+
mem: 24+
no_plugin: true
working_dir: "."
env:
REGISTRY: "public.ecr.aws/q9t5s3a7"
REPO: "vllm-ci-test-repo"
VLLM_TEST_DEVICE: "xpu"
source_file_dependencies:
- vllm/
- tests/v1/determinism/test_xpu_batch_invariant_ut.py
- .buildkite/scripts/hardware_ci/run-xpu-batch-invariance.sh
- .buildkite/intel_jobs/misc_intel.yaml
commands:
- >-
bash .buildkite/scripts/hardware_ci/run-intel-test.sh
'export VLLM_BATCH_INVARIANT=1 &&
cd tests &&
pytest -v -s v1/determinism/test_xpu_batch_invariant_ut.py &&
bash ../.buildkite/scripts/hardware_ci/run-xpu-batch-invariance.sh'
- label: V1 Core + KV + Metrics
key: v1-core-kv-metrics-intel
timeout_in_minutes: 45
device: intel_gpu
agent_tags:
label: production
gpu: 1+
mem: 24+
no_plugin: true
working_dir: "."
env:
REGISTRY: "public.ecr.aws/q9t5s3a7"
REPO: "vllm-ci-test-repo"
VLLM_TEST_DEVICE: "xpu"
source_file_dependencies:
- vllm/
- tests/v1/core
- tests/v1/executor
- tests/v1/kv_offload
- tests/v1/worker
- tests/v1/kv_connector/unit
- tests/v1/ec_connector/unit
- tests/v1/metrics
- tests/entrypoints/openai/correctness/test_lmeval.py
commands:
- >-
bash .buildkite/scripts/hardware_ci/run-intel-test.sh
'pip install -r requirements/kv_connectors.txt &&
export VLLM_WORKER_MULTIPROC_METHOD=spawn &&
cd tests &&
pytest -v -s v1/executor &&
VLLM_BATCH_INVARIANT=1 pytest -v -s -m "not cpu_test" v1/ec_connector/unit'
- label: V1 Sample + Logits
key: v1-sample-logits-intel
timeout_in_minutes: 90
device: intel_gpu
agent_tags:
label: production
gpu: 1+
mem: 24+
no_plugin: true
working_dir: "."
env:
REGISTRY: "public.ecr.aws/q9t5s3a7"
REPO: "vllm-ci-test-repo"
VLLM_TEST_DEVICE: "xpu"
source_file_dependencies:
- vllm/config/
- vllm/distributed/
- vllm/engine/
- vllm/inputs/
- vllm/logger.py
- vllm/model_executor/
- vllm/platforms/
- vllm/sampling_params.py
- vllm/transformers_utils/
- vllm/utils/
- vllm/v1/
- tests/v1/sample
- tests/v1/logits_processors
- tests/v1/test_oracle.py
- tests/v1/test_request.py
- tests/v1/test_outputs.py
commands:
- >-
bash .buildkite/scripts/hardware_ci/run-intel-test.sh
'export VLLM_WORKER_MULTIPROC_METHOD=spawn &&
cd tests &&
pytest -v -s v1/logits_processors --ignore=v1/logits_processors/test_custom_online.py --ignore=v1/logits_processors/test_custom_offline.py &&
pytest -v -s v1/test_oracle.py v1/test_request.py v1/test_outputs.py &&
pytest -v -s v1/sample'
retry:
automatic:
- exit_status: 1 # Test command failed
limit: 1
parallelism: 4
- label: Basic Models Tests (Initialization)
key: basic-models-tests-initialization
timeout_in_minutes: 60
device: intel_gpu
agent_tags:
label: production
gpu: 1+
mem: 16+
no_plugin: true
working_dir: "."
env:
REGISTRY: "public.ecr.aws/q9t5s3a7"
REPO: "vllm-ci-test-repo"
VLLM_TEST_DEVICE: "xpu"
source_file_dependencies:
- vllm/
- tests/models/test_initialization.py
- tests/models/registry.py
commands:
- >-
bash .buildkite/scripts/hardware_ci/run-intel-test.sh
'export VLLM_XPU_FUSED_MOE_USE_REF=1 &&
cd tests &&
pytest -v -s models/test_initialization.py::test_can_initialize_large_subset[Eagle3MiniMaxM2ForCausalLM]'
- label: XPU CPU Offload
key: xpu-cpu-offload
timeout_in_minutes: 60
device: intel_gpu
agent_tags:
label: production
gpu: 1+
mem: 16+
no_plugin: true
working_dir: "."
env:
REGISTRY: "public.ecr.aws/q9t5s3a7"
REPO: "vllm-ci-test-repo"
VLLM_TEST_DEVICE: "xpu"
source_file_dependencies:
- vllm/
- vllm/v1/kv_offload/
- vllm/v1/kv_connector/
- tests/v1/kv_offload/
- tests/v1/kv_connector/unit/test_offloading_connector.py
commands:
- >-
bash .buildkite/scripts/hardware_ci/run-intel-test.sh
'export VLLM_WORKER_MULTIPROC_METHOD=spawn &&
cd tests &&
pytest -v -s v1/kv_offload &&
pytest -v -s v1/kv_connector/unit/test_offloading_connector.py'
- label: NixlConnector PD accuracy (4 GPUs)
key: nixlconnector-pd-accuracy-4-gpus
timeout_in_minutes: 60
num_devices: 4
device: intel_gpu
agent_tags:
label: production
gpu: 4+
mem: 16+
no_plugin: true
working_dir: "."
env:
REGISTRY: "public.ecr.aws/q9t5s3a7"
REPO: "vllm-ci-test-repo"
VLLM_TEST_DEVICE: "xpu"
source_file_dependencies:
- vllm/distributed/kv_transfer/kv_connector/
- vllm/v1/worker/kv_connector_model_runner_mixin.py
- tests/v1/kv_connector/nixl_integration/
- vllm/platforms/xpu.py
commands:
- >-
bash .buildkite/scripts/hardware_ci/run-intel-test.sh
'cd tests &&
bash v1/kv_connector/nixl_integration/run_xpu_disagg_accuracy_test.sh &&
PREFILLER_TP_SIZE=2 DECODER_TP_SIZE=1 bash v1/kv_connector/nixl_integration/run_xpu_disagg_accuracy_test.sh &&
PREFILLER_TP_SIZE=1 DECODER_TP_SIZE=2 bash v1/kv_connector/nixl_integration/run_xpu_disagg_accuracy_test.sh &&
PREFILLER_TP_SIZE=2 DECODER_TP_SIZE=2 bash v1/kv_connector/nixl_integration/run_xpu_disagg_accuracy_test.sh'
- label: Regression
key: regression
timeout_in_minutes: 50
device: intel_gpu
agent_tags:
label: production
gpu: 1+
mem: 16+
no_plugin: true
working_dir: "."
env:
REGISTRY: "public.ecr.aws/q9t5s3a7"
REPO: "vllm-ci-test-repo"
VLLM_TEST_DEVICE: "xpu"
source_file_dependencies:
- vllm/config/
- vllm/distributed/
- vllm/engine/
- vllm/inputs/
- vllm/model_executor/
- vllm/multimodal/
- vllm/platforms/
- vllm/sampling_params.py
- vllm/transformers_utils/
- vllm/utils/
- vllm/v1/
- tests/test_regression
commands:
- >-
bash .buildkite/scripts/hardware_ci/run-intel-test.sh
'cd tests &&
pytest -v -s test_regression.py'
- label: Metrics, Tracing (2 GPUs)
key: metrics-tracing-2-gpus
timeout_in_minutes: 45
num_devices: 2
device: intel_gpu
agent_tags:
label: production
gpu: 2+
mem: 16+
no_plugin: true
working_dir: "."
env:
REGISTRY: "public.ecr.aws/q9t5s3a7"
REPO: "vllm-ci-test-repo"
VLLM_TEST_DEVICE: "xpu"
source_file_dependencies:
- vllm/config/
- vllm/distributed/
- vllm/engine/
- vllm/inputs/
- vllm/model_executor/
- vllm/multimodal/
- vllm/platforms/
- vllm/sampling_params.py
- vllm/tracing/
- vllm/transformers_utils/
- vllm/utils/
- vllm/v1/
- tests/v1/tracing
commands:
- >-
bash .buildkite/scripts/hardware_ci/run-intel-test.sh
'cd tests &&
pytest -v -s v1/tracing'
- label: Async Engine, Inputs, Utils, Worker
key: async-engine-inputs-utils-worker
timeout_in_minutes: 55
device: intel_gpu
agent_tags:
label: production
gpu: 1+
mem: 24+
no_plugin: true
working_dir: "."
env:
REGISTRY: "public.ecr.aws/q9t5s3a7"
REPO: "vllm-ci-test-repo"
VLLM_TEST_DEVICE: "xpu"
source_file_dependencies:
- vllm/assets/
- vllm/config/
- vllm/distributed/
- vllm/engine/
- vllm/inputs/
- vllm/model_executor/
- vllm/multimodal/
- vllm/platforms/
- vllm/sampling_params.py
- vllm/tokenizers/
- vllm/transformers_utils/
- vllm/utils/
- vllm/v1/
- tests/detokenizer
- tests/multimodal
- tests/utils_
commands:
- >-
bash .buildkite/scripts/hardware_ci/run-intel-test.sh
'cd tests &&
pytest -v -s detokenizer &&
pytest -v -s -m "not cpu_test" ./multimodal &&
pytest -v -s utils_ --ignore=utils_/test_mem_utils.py'
- label: Fusion Unit Tests
key: fusion-unit-tests
timeout_in_minutes: 30
device: intel_gpu
agent_tags:
label: production
gpu: 1+
mem: 16+
no_plugin: true
working_dir: "."
env:
REGISTRY: "public.ecr.aws/q9t5s3a7"
REPO: "vllm-ci-test-repo"
VLLM_TEST_DEVICE: "xpu"
source_file_dependencies:
- vllm/compilation/
- tests/compile/passes/test_qk_norm_rope_fusion.py
commands:
- >-
bash .buildkite/scripts/hardware_ci/run-intel-test.sh
'cd tests &&
pytest -v -s compile/passes/test_qk_norm_rope_fusion.py'
@@ -1,40 +0,0 @@
group: Model Executor Intel
depends_on:
- image-build-xpu
steps:
- label: Model Executor (Intel)
key: model-executor-intel
timeout_in_minutes: 45
device: intel_gpu
agent_tags:
label: production
gpu: 1+
mem: 24+
no_plugin: true
working_dir: "."
env:
REGISTRY: "public.ecr.aws/q9t5s3a7"
REPO: "vllm-ci-test-repo"
VLLM_TEST_DEVICE: "xpu"
source_file_dependencies:
- vllm/engine/arg_utils.py
- vllm/config/model.py
- vllm/model_executor
- tests/model_executor
commands:
- >-
bash .buildkite/scripts/hardware_ci/run-intel-test.sh
'apt-get update && apt-get install -y curl libsodium23 &&
export VLLM_WORKER_MULTIPROC_METHOD=spawn &&
export PYTHONFAULTHANDLER=1 &&
cd tests &&
pytest -v -s model_executor -m "not slow_test" --ignore="model_executor/layers/test_rocm_unquantized_gemm.py" --deselect="tests/model_executor/model_loader/test_reload.py::test_kv_scale_reload"
--deselect="tests/model_executor/layers/test_mm_input_norm.py::TestFusedMMInputNormModule::test_matches_reference_above_cudnn_grid_limit"
--deselect="tests/model_executor/layers/test_mm_input_norm.py::TestFusedMMInputNormDtypes::test_dtype_combinations[in_dtype0-out_dtype0]"
--deselect="tests/model_executor/layers/test_mm_input_norm.py::TestFusedMMInputNormDtypes::test_dtype_combinations[in_dtype1-out_dtype1]"
--deselect="tests/model_executor/layers/test_mm_input_norm.py::TestFusedMMInputNormDtypes::test_dtype_combinations[in_dtype2-out_dtype2]"
--deselect="tests/model_executor/layers/test_mm_input_norm.py::TestFusedMMInputNormDtypes::test_dtype_combinations[in_dtype3-out_dtype3]"
--deselect="tests/model_executor/layers/test_mm_input_norm.py::TestFusedMMInputNormDtypes::test_dtype_combinations[in_dtype4-out_dtype4]"
--deselect="tests/model_executor/layers/test_mm_input_norm.py::TestFusedMMInputNormDtypes::test_dtype_combinations[in_dtype5-out_dtype5]"
--deselect="tests/model_executor/layers/test_mm_input_norm.py::TestFusedMMInputNormShapes"
--deselect="tests/model_executor/layers/test_mm_input_norm.py::TestFusedMMInputNormInputHandling"'
@@ -1,27 +0,0 @@
group: Models - Distributed
depends_on:
- image-build-xpu
steps:
- label: Distributed Model Tests (2 GPUs)
key: distributed-model-tests-2-gpus
timeout_in_minutes: 65
device: intel_gpu
agent_tags:
label: production
gpu: 2+
mem: 16+
no_plugin: true
working_dir: "."
env:
REGISTRY: "public.ecr.aws/q9t5s3a7"
REPO: "vllm-ci-test-repo"
VLLM_TEST_DEVICE: "xpu"
source_file_dependencies:
- vllm/model_executor/model_loader/sharded_state_loader.py
- vllm/model_executor/models/
- tests/model_executor/model_loader/test_sharded_state_loader.py
commands:
- >-
bash .buildkite/scripts/hardware_ci/run-intel-test.sh
'cd tests &&
pytest -v -s model_executor/model_loader/test_sharded_state_loader.py -m "not slow_test"'
@@ -1,151 +0,0 @@
group: Models - Multimodal
depends_on:
- image-build-xpu
steps:
- label: "Multi-Modal Models (Standard) 1: qwen2"
key: multi-modal-models-standard-1-qwen2
timeout_in_minutes: 70
device: intel_gpu
agent_tags:
label: production
gpu: 1+
mem: 24+
no_plugin: true
working_dir: "."
env:
REGISTRY: "public.ecr.aws/q9t5s3a7"
REPO: "vllm-ci-test-repo"
VLLM_TEST_DEVICE: "xpu"
source_file_dependencies:
- vllm/
- tests/models/multimodal
commands:
- >-
bash .buildkite/scripts/hardware_ci/run-intel-test.sh
'cd tests &&
pytest -v -s models/multimodal/generation/test_common.py -m core_model -k "qwen2" &&
pytest -v -s models/multimodal/generation/test_ultravox.py -m core_model'
- label: "Multi-Modal Models (Standard) 2: qwen3 + gemma"
key: multi-modal-models-standard-2-qwen3-gemma
timeout_in_minutes: 90
device: intel_gpu
agent_tags:
label: production
gpu: 1+
mem: 24+
no_plugin: true
working_dir: "."
env:
REGISTRY: "public.ecr.aws/q9t5s3a7"
REPO: "vllm-ci-test-repo"
VLLM_TEST_DEVICE: "xpu"
source_file_dependencies:
- vllm/
- tests/models/multimodal
commands:
- >-
bash .buildkite/scripts/hardware_ci/run-intel-test.sh
'cd tests &&
pytest -v -s models/multimodal/generation/test_common.py -m core_model -k "qwen3 or gemma" &&
pytest -v -s models/multimodal/generation/test_mm_prefix_lm.py -m core_model &&
pytest -v -s models/multimodal/generation/test_qwen2_5_vl.py -m core_model'
- label: "Multi-Modal Models (Standard) 3: llava + qwen2_vl"
key: multi-modal-models-standard-3-llava-qwen2-vl
timeout_in_minutes: 65
device: intel_gpu
agent_tags:
label: production
gpu: 1+
mem: 24+
no_plugin: true
working_dir: "."
env:
REGISTRY: "public.ecr.aws/q9t5s3a7"
REPO: "vllm-ci-test-repo"
VLLM_TEST_DEVICE: "xpu"
source_file_dependencies:
- vllm/
- tests/models/multimodal
commands:
- >-
bash .buildkite/scripts/hardware_ci/run-intel-test.sh
'cd tests &&
pytest -v -s models/multimodal/generation/test_common.py -m core_model -k "not qwen2 and not qwen3 and not gemma"'
- label: "Multi-Modal Models (Standard) 4: other + whisper"
key: multi-modal-models-standard-4-other-whisper
timeout_in_minutes: 45
device: intel_gpu
agent_tags:
label: production
gpu: 1+
mem: 24+
no_plugin: true
working_dir: "."
env:
REGISTRY: "public.ecr.aws/q9t5s3a7"
REPO: "vllm-ci-test-repo"
VLLM_TEST_DEVICE: "xpu"
source_file_dependencies:
- vllm/
- tests/models/multimodal
commands:
- >-
bash .buildkite/scripts/hardware_ci/run-intel-test.sh
'cd tests &&
pytest -v -s models/multimodal -m core_model --ignore models/multimodal/generation/test_common.py --ignore models/multimodal/generation/test_ultravox.py --ignore models/multimodal/generation/test_qwen2_5_vl.py --ignore models/multimodal/generation/test_qwen2_vl.py --ignore models/multimodal/generation/test_whisper.py --ignore models/multimodal/generation/test_memory_leak.py --ignore models/multimodal/processing'
- label: Multi-Modal Processor # 44min
key: multi-modal-processor
timeout_in_minutes: 60
device: intel_gpu
agent_tags:
label: production
gpu: 1+
mem: 24+
no_plugin: true
working_dir: "."
env:
REGISTRY: "public.ecr.aws/q9t5s3a7"
REPO: "vllm-ci-test-repo"
VLLM_TEST_DEVICE: "xpu"
source_file_dependencies:
- vllm/
- tests/models/multimodal
- tests/models/registry.py
commands:
- >-
bash .buildkite/scripts/hardware_ci/run-intel-test.sh
'cd tests &&
pytest -v -s models/multimodal/processing/test_tensor_schema.py
--deselect="tests/models/multimodal/processing/test_tensor_schema.py::test_model_tensor_schema[deepseek-ai/DeepSeek-V4.1-Flash]"
--num-shards=$$BUILDKITE_PARALLEL_JOB_COUNT --shard-id=$$BUILDKITE_PARALLEL_JOB'
parallelism: 4
- label: Multi-Modal Models (Extended Generation 1)
key: multi-modal-models-extended-generation-1
timeout_in_minutes: 60
device: intel_gpu
agent_tags:
label: production
gpu: 1+
mem: 24+
no_plugin: true
working_dir: "."
env:
REGISTRY: "public.ecr.aws/q9t5s3a7"
REPO: "vllm-ci-test-repo"
VLLM_TEST_DEVICE: "xpu"
optional: true
source_file_dependencies:
- vllm/
- "!vllm/distributed/kv_transfer/"
- tests/models/multimodal/generation
- tests/models/multimodal/test_mapping.py
commands:
- >-
bash .buildkite/scripts/hardware_ci/run-intel-test.sh
'cd tests &&
pytest -v -s models/multimodal/test_mapping.py'
-29
View File
@@ -1,29 +0,0 @@
group: Quantization
depends_on:
- image-build-xpu
steps:
- label: Quantization
key: quantization
timeout_in_minutes: 30
env:
REGISTRY: "public.ecr.aws/q9t5s3a7"
REPO: "vllm-ci-test-repo"
VLLM_TEST_DEVICE: "xpu"
no_plugin: true
working_dir: "."
device: intel_gpu
agent_tags:
label: production
gpu: 1+
mem: 16+
source_file_dependencies:
- csrc/
- vllm/model_executor/layers/quantization
- tests/quantization
commands:
# - VLLM_TEST_FORCE_LOAD_FORMAT=auto pytest -v -s quantization/ --ignore quantization/test_blackwell_moe.py
- >-
bash .buildkite/scripts/hardware_ci/run-intel-test.sh
'VLLM_TEST_FORCE_LOAD_FORMAT=auto pytest -v -s tests/quantization/test_per_token_kv_cache.py --deselect="tests/quantization/test_per_token_kv_cache.py::test_triton_unified_attention_per_token_head_scale[int4-16-128-num_heads0-seq_lens1]"
&& VLLM_TEST_FORCE_LOAD_FORMAT=auto pytest -v -s tests/quantization/test_auto_awq.py'
-29
View File
@@ -1,29 +0,0 @@
group: Samplers Intel
depends_on:
- image-build-xpu
steps:
- label: Samplers Test (XPU Kernel)
key: samplers-test-xpu-kernel-intel
timeout_in_minutes: 40
device: intel_gpu
agent_tags:
label: production
gpu: 1+
mem: 24+
no_plugin: true
working_dir: "."
env:
REGISTRY: "public.ecr.aws/q9t5s3a7"
REPO: "vllm-ci-test-repo"
VLLM_TEST_DEVICE: "xpu"
source_file_dependencies:
- vllm/model_executor/layers
- vllm/sampling_metadata.py
- tests/samplers
- tests/conftest.py
- vllm/entrypoints/generate/beam_search
commands:
- >-
bash .buildkite/scripts/hardware_ci/run-intel-test.sh
'cd tests &&
VLLM_XPU_USE_SAMPLER_KERNEL=1 pytest -v -s samplers'
-215
View File
@@ -1,215 +0,0 @@
group: Intel
steps:
- label: ":docker: Build XPU image"
soft_fail: true
depends_on: []
key: image-build-xpu
commands:
- bash -lc '.buildkite/image_build/image_build_xpu.sh "public.ecr.aws/q9t5s3a7" "vllm-ci-test-repo" "$BUILDKITE_COMMIT"'
env:
DOCKER_BUILDKIT: "1"
retry:
automatic:
- exit_status: -1 # Agent was lost
limit: 2
- exit_status: -10 # Agent was lost
limit: 2
- label: "XPU example Test"
key: xpu-example-test
depends_on:
- image-build-xpu
timeout_in_minutes: 70
device: intel_gpu
agent_tags:
label: production
gpu: 2+
mem: 24+
no_plugin: true
env:
REGISTRY: "public.ecr.aws/q9t5s3a7"
REPO: "vllm-ci-test-repo"
source_file_dependencies:
- vllm/
- .buildkite/intel_jobs/test-intel.yaml
commands:
- >-
bash .buildkite/scripts/hardware_ci/run-intel-test.sh
'python3 examples/basic/offline_inference/generate.py --model facebook/opt-125m --block-size 64 --enforce-eager &&
python3 examples/basic/offline_inference/generate.py --model facebook/opt-125m --block-size 64 -O3 -cc.cudagraph_mode=NONE &&
python3 examples/basic/offline_inference/generate.py --model facebook/opt-125m --block-size 64 --enforce-eager -tp 2 --distributed-executor-backend mp &&
python3 examples/basic/offline_inference/generate.py --model facebook/opt-125m --block-size 64 --enforce-eager --attention-backend=TRITON_ATTN &&
python3 examples/basic/offline_inference/generate.py --model facebook/opt-125m --block-size 64 --enforce-eager --quantization fp8_per_tensor &&
python3 examples/basic/offline_inference/generate.py --model facebook/opt-125m --block-size 64 --enforce-eager --kv-cache-dtype fp8 &&
python3 examples/basic/offline_inference/generate.py --model nvidia/Llama-3.1-8B-Instruct-FP8 --block-size 64 --enforce-eager --quantization modelopt --kv-cache-dtype fp8 --attention-backend TRITON_ATTN --max-model-len 4096 &&
python3 examples/basic/offline_inference/generate.py --model superjob/Qwen3-4B-Instruct-2507-GPTQ-Int4 --block-size 64 --enforce-eager --max-model-len 8192 &&
python3 examples/basic/offline_inference/generate.py --model TheBloke/TinyLlama-1.1B-Chat-v0.3-AWQ --block-size 64 --enforce-eager &&
python3 examples/basic/offline_inference/generate.py --model ibm-research/PowerMoE-3b --block-size 64 --enforce-eager -tp 2 &&
python3 examples/basic/offline_inference/generate.py --model ibm-research/PowerMoE-3b --block-size 64 --enforce-eager -tp 2 --enable-expert-parallel &&
python3 examples/basic/offline_inference/generate.py --model superjob/Qwen3-4B-Instruct-2507-GPTQ-Int4 --max-model-len 8192 &&
VLLM_XPU_FUSED_MOE_USE_REF=1 python3 examples/basic/offline_inference/generate.py --model Qwen/Qwen3-30B-A3B-Instruct-2507-FP8 --enforce-eager -tp 2 --max-model-len 8192 &&
python3 examples/basic/offline_inference/generate.py --model INCModel/Qwen3-30B-A3B-Instruct-2507-MXFP4-LLMC -tp 2 --max-model-len 8192
'
- label: "XPU W8A8 FP8 Linear Examples"
key: xpu-w8a8-fp8-linear-examples
depends_on:
- image-build-xpu
timeout_in_minutes: 60
device: intel_gpu
agent_tags:
label: production
gpu: 1+
mem: 24+
no_plugin: true
env:
REGISTRY: "public.ecr.aws/q9t5s3a7"
REPO: "vllm-ci-test-repo"
VLLM_TEST_DEVICE: "xpu"
source_file_dependencies:
- vllm/
- .buildkite/intel_jobs/test-intel.yaml
commands:
- >-
bash .buildkite/scripts/hardware_ci/run-intel-test.sh
'python3 examples/basic/offline_inference/generate.py --linear-backend xpu --model RedHatAI/Meta-Llama-3.1-8B-Instruct-FP8 --enforce-eager --max-model-len 4096 &&
python3 examples/basic/offline_inference/generate.py --linear-backend torch --model RedHatAI/Meta-Llama-3.1-8B-Instruct-FP8 --enforce-eager --max-model-len 4096 &&
python3 examples/basic/offline_inference/generate.py --linear-backend xpu --model neuralmagic/Llama-3.2-1B-Instruct-FP8-dynamic --enforce-eager --max-model-len 4096 &&
python3 examples/basic/offline_inference/generate.py --linear-backend torch --model Qwen/Qwen3-4B-Instruct-2507-FP8 --enforce-eager --max-model-len 4096 &&
python3 examples/basic/offline_inference/generate.py --linear-backend xpu --model meta-llama/Llama-3.2-1B-Instruct --quantization fp8_per_tensor --enforce-eager --max-model-len 4096 &&
python3 examples/basic/offline_inference/generate.py --linear-backend torch --model meta-llama/Llama-3.2-1B-Instruct --quantization fp8_per_tensor --enforce-eager --max-model-len 4096
'
- label: "XPU V1 test"
key: xpu-v1-test
depends_on:
- image-build-xpu
timeout_in_minutes: 70
device: intel_gpu
agent_tags:
label: production
gpu: 1+
mem: 24+
no_plugin: true
env:
REGISTRY: "public.ecr.aws/q9t5s3a7"
REPO: "vllm-ci-test-repo"
source_file_dependencies:
- vllm/
- .buildkite/intel_jobs/test-intel.yaml
commands:
- >-
bash .buildkite/scripts/hardware_ci/run-intel-test.sh
'cd tests && pip install dpctl &&
pytest -v -s v1/core --ignore=v1/core/test_reset_prefix_cache_e2e.py --ignore=v1/core/test_scheduler_e2e.py &&
pytest -v -s v1/engine --ignore=v1/engine/test_output_processor.py --ignore=v1/engine/test_core_engine_actor_manager.py &&
pytest -v -s v1/sample --ignore=v1/sample/test_logprobs.py --ignore=v1/sample/test_logprobs_e2e.py -k "not test_topk_only and not test_topp_only and not test_topk_and_topp" &&
pytest -v -s v1/worker --ignore=v1/worker/test_gpu_model_runner.py --ignore=v1/worker/test_worker_memory_snapshot.py &&
pytest -v -s v1/structured_output &&
pytest -v -s v1/test_serial_utils.py &&
pytest -v -s v1/e2e/general/test_correctness_sliding_window.py &&
pytest -v -s v1/spec_decode --ignore=v1/spec_decode/test_max_len.py --ignore=v1/spec_decode/test_speculators_eagle3.py --ignore=v1/spec_decode/test_acceptance_length.py --ignore=v1/spec_decode/test_speculators_correctness.py --deselect=tests/v1/spec_decode/test_mtp.py::test_glm_mtp_defers_lm_head &&
pytest -v -s v1/kv_connector/unit --ignore=v1/kv_connector/unit/test_lmcache_integration.py --ignore=v1/kv_connector/unit/test_hf3fs_client.py --ignore=v1/kv_connector/unit/test_hf3fs_connector.py --ignore=v1/kv_connector/unit/test_hf3fs_metadata_server.py --ignore=v1/kv_connector/unit/test_offloading_connector.py --ignore=v1/kv_connector/unit/test_nixl_connector_hma.py --deselect=tests/v1/kv_connector/unit/test_nixl_connector.py::test_abort_timeout_on_prefiller[ray]'
parallelism: 2
- label: "XPU server test"
key: xpu-server-test
depends_on:
- image-build-xpu
timeout_in_minutes: 45
device: intel_gpu
agent_tags:
label: production
gpu: 1+
mem: 16+
no_plugin: true
env:
REGISTRY: "public.ecr.aws/q9t5s3a7"
REPO: "vllm-ci-test-repo"
source_file_dependencies:
- vllm/
- .buildkite/intel_jobs/test-intel.yaml
commands:
- >-
bash .buildkite/scripts/hardware_ci/run-intel-test.sh
'cd tests &&
pytest -v -s entrypoints/multimodal/openai/chat_completion/test_audio_in_video.py &&
pytest -v -s benchmarks/test_serve_cli.py'
- label: "XPU quantization test"
key: xpu-quantization-test
depends_on:
- image-build-xpu
timeout_in_minutes: 30
device: intel_gpu
agent_tags:
label: production
gpu: 1+
mem: 16+
no_plugin: true
env:
REGISTRY: "public.ecr.aws/q9t5s3a7"
REPO: "vllm-ci-test-repo"
source_file_dependencies:
- vllm/
- .buildkite/intel_jobs/test-intel.yaml
commands:
- >-
bash .buildkite/scripts/hardware_ci/run-intel-test.sh
'cd tests &&
pytest -v -s quantization/test_auto_round.py &&
pytest -v -s quantization/test_online.py --deselect=tests/quantization/test_online.py::test_online_quantization_loads_real_weights'
- label: "XPU compressed tensors FP8 test"
key: xpu-compressed-tensors-fp8-test
depends_on:
- image-build-xpu
timeout_in_minutes: 60
device: intel_gpu
agent_tags:
label: production
gpu: 1+
mem: 16+
no_plugin: true
env:
REGISTRY: "public.ecr.aws/q9t5s3a7"
REPO: "vllm-ci-test-repo"
VLLM_TEST_DEVICE: "xpu"
source_file_dependencies:
- vllm/
- tests/quantization/test_compressed_tensors.py
- .buildkite/intel_jobs/test-intel.yaml
commands:
- >-
bash .buildkite/scripts/hardware_ci/run-intel-test.sh
'cd tests &&
pytest -v -s quantization/test_compressed_tensors.py::test_compressed_tensors_fp8 &&
pytest -v -s quantization/test_compressed_tensors.py::test_compressed_tensors_fp8_block_enabled'
- label: "XPU Graph"
key: xpu-graph
depends_on:
- image-build-xpu
timeout_in_minutes: 60
device: intel_gpu
agent_tags:
label: production
gpu: 1+
mem: 32+
no_plugin: true
env:
REGISTRY: "public.ecr.aws/q9t5s3a7"
REPO: "vllm-ci-test-repo"
VLLM_TEST_DEVICE: "xpu"
source_file_dependencies:
- vllm/compilation/
- vllm/config/compilation.py
- vllm/platforms/xpu.py
- vllm/v1/cudagraph_dispatcher.py
- vllm/v1/worker/encoder_cudagraph.py
- vllm/v1/worker/encoder_cudagraph_defs.py
- vllm/v1/worker/xpu_model_runner.py
- vllm/v1/worker/xpu_worker.py
- tests/v1/cudagraph
- .buildkite/intel_jobs/test-intel.yaml
commands:
- >-
bash .buildkite/scripts/hardware_ci/run-intel-test.sh
'cd tests &&
pytest -v -s v1/cudagraph/test_cudagraph_dispatch.py &&
pytest -v -s v1/cudagraph/test_cudagraph_mode.py &&
pytest -v -s v1/cudagraph/test_breakable_cudagraph.py &&
pytest -v -s v1/cudagraph/test_encoder_cudagraph.py'
@@ -1,13 +0,0 @@
# For vllm script, with -t option (tensor parallel size).
# bash ./run-lm-eval-gsm-vllm-baseline.sh -m deepseek-ai/DeepSeek-V2-Lite-Chat -b "auto" -l 1000 -f 5 -t 2
model_name: "deepseek-ai/DeepSeek-V2-Lite-Chat"
tasks:
- name: "gsm8k"
metrics:
- name: "exact_match,strict-match"
value: 0.671
- name: "exact_match,flexible-extract"
value: 0.664
limit: 1000
num_fewshot: 5
trust_remote_code: True
@@ -1,25 +0,0 @@
# DeepSeek-V4-Flash MXFP4 (AMD Quark) accuracy test.
# Runs on ROCm (gfx950) with the AITER MoE + fused shared-experts path.
# Reference GSM8K (8-shot, lm-eval): strict 0.9522 / flexible 0.9515 measured
# on gfx950 TP=4 (AITER_MXFP4_BF16 backend); default rtol=0.08.
model_name: "amd/DeepSeek-V4-Flash-MXFP4"
tasks:
- name: "gsm8k"
metrics:
- name: "exact_match,strict-match"
value: 0.95
- name: "exact_match,flexible-extract"
value: 0.95
limit: 1319
num_fewshot: 8
max_model_len: 8192
kv_cache_dtype: fp8
moe_backend: aiter
enforce_eager: false
trust_remote_code: true
tokenizer_mode: deepseek_v4
required_gpu_arch:
- gfx950
env_vars:
VLLM_ROCM_USE_AITER: "1"
VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS: "1"
@@ -1,12 +0,0 @@
# For hf script, without -t option (tensor parallel size).
# bash .buildkite/lm-eval-harness/run-lm-eval-gsm-hf-baseline.sh -m nm-testing/Meta-Llama-3-70B-Instruct-FBGEMM-nonuniform -b auto -l 1000 -f 5
model_name: "nm-testing/Meta-Llama-3-70B-Instruct-FBGEMM-nonuniform"
tasks:
- name: "gsm8k"
metrics:
- name: "exact_match,strict-match"
value: 0.905
- name: "exact_match,flexible-extract"
value: 0.905
limit: 1000
num_fewshot: 5
@@ -1,12 +0,0 @@
# For hf script, without -t option (tensor parallel size).
# bash .buildkite/lm-eval-harness/run-lm-eval-gsm-hf-baseline.sh -m meta-llama/Meta-Llama-3-70B-Instruct -b 32 -l 250 -f 5
model_name: "meta-llama/Meta-Llama-3-70B-Instruct"
tasks:
- name: "gsm8k"
metrics:
- name: "exact_match,strict-match"
value: 0.892
- name: "exact_match,flexible-extract"
value: 0.892
limit: 250
num_fewshot: 5
@@ -1,12 +0,0 @@
# For vllm script, with -t option (tensor parallel size).
# bash .buildkite/lm-eval-harness/run-lm-eval-gsm-vllm-baseline.sh -m nm-testing/Meta-Llama-3-8B-Instruct-W8A8-FP8-Channelwise-compressed-tensors -b auto -l 1000 -f 5 -t 1
model_name: "nm-testing/Meta-Llama-3-8B-Instruct-W8A8-FP8-Channelwise-compressed-tensors"
tasks:
- name: "gsm8k"
metrics:
- name: "exact_match,strict-match"
value: 0.752
- name: "exact_match,flexible-extract"
value: 0.754
limit: 1000
num_fewshot: 5
@@ -1,12 +0,0 @@
# For vllm script, with -t option (tensor parallel size).
# bash .buildkite/lm-eval-harness/run-lm-eval-gsm-vllm-baseline.sh -m nm-testing/Meta-Llama-3-8B-Instruct-FBGEMM-nonuniform -b auto -l 1000 -f 5 -t 1
model_name: "nm-testing/Meta-Llama-3-8B-Instruct-FBGEMM-nonuniform"
tasks:
- name: "gsm8k"
metrics:
- name: "exact_match,strict-match"
value: 0.753
- name: "exact_match,flexible-extract"
value: 0.753
limit: 1000
num_fewshot: 5
@@ -1,12 +0,0 @@
# For vllm script, with -t option (tensor parallel size).
# bash .buildkite/lm-eval-harness/run-lm-eval-gsm-vllm-baseline.sh -m nm-testing/Meta-Llama-3-8B-FP8-compressed-tensors-test -b 32 -l 1000 -f 5 -t 1
model_name: "nm-testing/Meta-Llama-3-8B-FP8-compressed-tensors-test"
tasks:
- name: "gsm8k"
metrics:
- name: "exact_match,strict-match"
value: 0.755
- name: "exact_match,flexible-extract"
value: 0.755
limit: 1000
num_fewshot: 5
@@ -1,12 +0,0 @@
# For vllm script, with -t option (tensor parallel size).
# bash .buildkite/lm-eval-harness/run-lm-eval-gsm-vllm-baseline.sh -m neuralmagic/Meta-Llama-3-8B-Instruct-FP8 -b 32 -l 250 -f 5 -t 1
model_name: "neuralmagic/Meta-Llama-3-8B-Instruct-FP8"
tasks:
- name: "gsm8k"
metrics:
- name: "exact_match,strict-match"
value: 0.753
- name: "exact_match,flexible-extract"
value: 0.753
limit: 1000
num_fewshot: 5
@@ -1,12 +0,0 @@
# For vllm script, with -t option (tensor parallel size).
# bash .buildkite/lm-eval-harness/run-lm-eval-gsm-vllm-baseline.sh -m nm-testing/Meta-Llama-3-8B-Instruct-W8-Channel-A8-Dynamic-Asym-Per-Token-Test -b "auto" -l 250 -f 5 -t 1
model_name: "nm-testing/Meta-Llama-3-8B-Instruct-W8-Channel-A8-Dynamic-Asym-Per-Token-Test"
tasks:
- name: "gsm8k"
metrics:
- name: "exact_match,strict-match"
value: 0.764
- name: "exact_match,flexible-extract"
value: 0.764
limit: 250
num_fewshot: 5
@@ -1,12 +0,0 @@
# For vllm script, with -t option (tensor parallel size).
# bash .buildkite/lm-eval-harness/run-lm-eval-gsm-vllm-baseline.sh -m nm-testing/Meta-Llama-3-8B-Instruct-W8-Channel-A8-Dynamic-Per-Token-Test -b "auto" -l 250 -f 5 -t 1
model_name: "nm-testing/Meta-Llama-3-8B-Instruct-W8-Channel-A8-Dynamic-Per-Token-Test"
tasks:
- name: "gsm8k"
metrics:
- name: "exact_match,strict-match"
value: 0.728
- name: "exact_match,flexible-extract"
value: 0.728
limit: 250
num_fewshot: 5
@@ -1,12 +0,0 @@
# For vllm script, with -t option (tensor parallel size).
# bash .buildkite/lm-eval-harness/run-lm-eval-gsm-vllm-baseline.sh -m nm-testing/Meta-Llama-3-8B-Instruct-nonuniform-test -b auto -l 1000 -f 5 -t 1
model_name: "nm-testing/Meta-Llama-3-8B-Instruct-nonuniform-test"
tasks:
- name: "gsm8k"
metrics:
- name: "exact_match,strict-match"
value: 0.758
- name: "exact_match,flexible-extract"
value: 0.759
limit: 1000
num_fewshot: 5
@@ -1,12 +0,0 @@
# For hf script, without -t option (tensor parallel size).
# bash .buildkite/lm-eval-harness/run-lm-eval-gsm-hf-baseline.sh -m meta-llama/Meta-Llama-3-8B-Instruct -b 32 -l 250 -f 5
model_name: "meta-llama/Meta-Llama-3-8B-Instruct"
tasks:
- name: "gsm8k"
metrics:
- name: "exact_match,strict-match"
value: 0.756
- name: "exact_match,flexible-extract"
value: 0.752
limit: 250
num_fewshot: 5
@@ -1,12 +0,0 @@
# For vllm script, with -t option (tensor parallel size).
# bash .buildkite/lm-eval-harness/run-lm-eval-gsm-vllm-baseline.sh -m HandH1998/QQQ-Llama-3-8b-g128 -b 32 -l 1000 -f 5 -t 1
model_name: "HandH1998/QQQ-Llama-3-8b-g128"
tasks:
- name: "gsm8k"
metrics:
- name: "exact_match,strict-match"
value: 0.419
- name: "exact_match,flexible-extract"
value: 0.416
limit: 1000
num_fewshot: 5
@@ -1,11 +0,0 @@
# bash .buildkite/lm-eval-harness/run-lm-eval-gsm-vllm-baseline.sh -m RedHatAI/Llama-3.2-1B-Instruct-FP8 -b "auto" -l 1319 -f 5 -t 1
model_name: "RedHatAI/Llama-3.2-1B-Instruct-FP8"
tasks:
- name: "gsm8k"
metrics:
- name: "exact_match,strict-match"
value: 0.335
- name: "exact_match,flexible-extract"
value: 0.323
limit: 1319
num_fewshot: 5
@@ -1,12 +0,0 @@
# For vllm script, with -t option (tensor parallel size).
# bash .buildkite/lm-eval-harness/run-lm-eval-gsm-vllm-baseline.sh -m neuralmagic/Llama-3.2-1B-Instruct-quantized.w8a8 -b "auto" -l 1000 -f 5 -t 1
model_name: "neuralmagic/Llama-3.2-1B-Instruct-quantized.w8a8"
tasks:
- name: "gsm8k"
metrics:
- name: "exact_match,strict-match"
value: 0.356
- name: "exact_match,flexible-extract"
value: 0.358
limit: 1000
num_fewshot: 5
@@ -1,12 +0,0 @@
# For hf script, without -t option (tensor parallel size).
# bash .buildkite/lm-eval-harness/run-lm-eval-chartqa-vllm-vlm-baseline.sh -m meta-llama/Llama-4-Maverick-17B-128E-Instruct-FP8 -l 100 -t 8
model_name: "meta-llama/Llama-4-Maverick-17B-128E-Instruct-FP8"
backend: "vllm-vlm"
tasks:
- name: "chartqa"
metrics:
- name: "relaxed_accuracy,none"
# TODO(zhewenl): model card is 0.90, but the actual score is 0.80.
value: 0.80
limit: 100
num_fewshot: 0
@@ -1,15 +0,0 @@
# For hf script, without -t option (tensor parallel size).
# bash .buildkite/lm-eval-harness/run-lm-eval-mmlupro-vllm-baseline.sh -m meta-llama/Llama-4-Maverick-17B-128E-Instruct-FP8 -l 250 -t 8 -f 5
model_name: "meta-llama/Llama-4-Maverick-17B-128E-Instruct-FP8"
rocm_safetensors_load_strategy: prefetch
required_gpu_arch:
- gfx942
- gfx950
tasks:
- name: "mmlu_pro"
metrics:
- name: "exact_match,custom-extract"
value: 0.80
limit: 250 # will run on 250 * 14 subjects = 3500 samples
num_fewshot: 5
rtol: 0.05
@@ -1,12 +0,0 @@
# For vllm script, with -t option (tensor parallel size).
# bash .buildkite/lm-eval-harness/run-lm-eval-gsm-vllm-baseline.sh -m mgoin/Minitron-4B-Base-FP8 -b auto -l 1000 -f 5 -t 1
model_name: "mgoin/Minitron-4B-Base-FP8"
tasks:
- name: "gsm8k"
metrics:
- name: "exact_match,strict-match"
value: 0.231
- name: "exact_match,flexible-extract"
value: 0.22
limit: 1000
num_fewshot: 5
@@ -1,12 +0,0 @@
# For vllm script, with -t option (tensor parallel size).
# bash ./run-lm-eval-gsm-vllm-baseline.sh -m neuralmagic/Mixtral-8x22B-Instruct-v0.1-FP8-dynamic -b "auto" -l 250 -f 5 -t 8
model_name: "neuralmagic/Mixtral-8x22B-Instruct-v0.1-FP8-dynamic"
tasks:
- name: "gsm8k"
metrics:
- name: "exact_match,strict-match"
value: 0.86
- name: "exact_match,flexible-extract"
value: 0.86
limit: 250
num_fewshot: 5
@@ -1,12 +0,0 @@
# For vllm script, with -t option (tensor parallel size).
# bash ./run-lm-eval-gsm-vllm-baseline.sh -m neuralmagic/Mixtral-8x7B-Instruct-v0.1-FP8 -b "auto" -l 250 -f 5 -t 4
model_name: "neuralmagic/Mixtral-8x7B-Instruct-v0.1-FP8"
tasks:
- name: "gsm8k"
metrics:
- name: "exact_match,strict-match"
value: 0.624
- name: "exact_match,flexible-extract"
value: 0.624
limit: 250
num_fewshot: 5
@@ -1,12 +0,0 @@
# For hf script, without -t option (tensor parallel size).
# bash .buildkite/lm-eval-harness/run-lm-eval-gsm-hf-baseline.sh -m neuralmagic/Mixtral-8x7B-Instruct-v0.1 -b 32 -l 250 -f 5
model_name: "mistralai/Mixtral-8x7B-Instruct-v0.1"
tasks:
- name: "gsm8k"
metrics:
- name: "exact_match,strict-match"
value: 0.616
- name: "exact_match,flexible-extract"
value: 0.632
limit: 250
num_fewshot: 5
@@ -1,15 +0,0 @@
model_name: "nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16"
tasks:
- name: "gsm8k"
metrics:
- name: "exact_match,strict-match"
value: 0.695
- name: "exact_match,flexible-extract"
value: 0.447
limit: 1319
num_fewshot: 5
max_model_len: 262144
enforce_eager: false
apply_chat_template: true
fewshot_as_multiturn: true
trust_remote_code: true
@@ -1,16 +0,0 @@
model_name: "nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8"
tasks:
- name: "gsm8k"
metrics:
- name: "exact_match,strict-match"
value: 0.7142
- name: "exact_match,flexible-extract"
value: 0.4579
limit: 256
num_fewshot: 5
max_model_len: 262144
kv_cache_dtype: fp8
enforce_eager: true
apply_chat_template: true
fewshot_as_multiturn: true
trust_remote_code: true
@@ -1,17 +0,0 @@
model_name: "nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8"
tasks:
- name: "gsm8k"
metrics:
- name: "exact_match,strict-match"
value: 0.7142
- name: "exact_match,flexible-extract"
value: 0.4579
moe_backend: "flashinfer_cutlass"
limit: 1319
num_fewshot: 5
max_model_len: 262144
kv_cache_dtype: fp8
enforce_eager: false
apply_chat_template: true
fewshot_as_multiturn: true
trust_remote_code: true
@@ -1,12 +0,0 @@
# For vllm script, with -t option (tensor parallel size).
# bash .buildkite/lm-eval-harness/run-lm-eval-gsm-vllm-baseline.sh -m nm-testing/Qwen1.5-MoE-A2.7B-Chat-quantized.w4a16 -b auto -l 1319 -f 5 -t 1
model_name: "nm-testing/Qwen1.5-MoE-A2.7B-Chat-quantized.w4a16"
tasks:
- name: "gsm8k"
metrics:
- name: "exact_match,strict-match"
value: 0.30
- name: "exact_match,flexible-extract"
value: 0.465
limit: 1319
num_fewshot: 5
@@ -1,12 +0,0 @@
# For vllm script, with -t option (tensor parallel size).
# bash .buildkite/lm-eval-harness/run-lm-eval-gsm-vllm-baseline.sh -m nm-testing/Qwen2-1.5B-Instruct-FP8W8 -b auto -l 1000 -f 5 -t 1
model_name: "nm-testing/Qwen2-1.5B-Instruct-FP8W8"
tasks:
- name: "gsm8k"
metrics:
- name: "exact_match,strict-match"
value: 0.578
- name: "exact_match,flexible-extract"
value: 0.585
limit: 1000
num_fewshot: 5
@@ -1,12 +0,0 @@
# For vllm script, with -t option (tensor parallel size).
# bash .buildkite/lm-eval-harness/run-lm-eval-gsm-vllm-baseline.sh -m neuralmagic/Qwen2-1.5B-Instruct-quantized.w8a8 -b "auto" -l 1000 -f 5 -t 1
model_name: "neuralmagic/Qwen2-1.5B-Instruct-quantized.w8a8"
tasks:
- name: "gsm8k"
metrics:
- name: "exact_match,strict-match"
value: 0.593
- name: "exact_match,flexible-extract"
value: 0.588
limit: 1000
num_fewshot: 5
@@ -1,12 +0,0 @@
# For vllm script, with -t option (tensor parallel size).
# bash ./run-lm-eval-gsm-vllm-baseline.sh -m Qwen/Qwen2-57B-A14B-Instruct -b "auto" -l 250 -f 5 -t 4
model_name: "Qwen/Qwen2-57B-A14B-Instruct"
tasks:
- name: "gsm8k"
metrics:
- name: "exact_match,strict-match"
value: 0.792
- name: "exact_match,flexible-extract"
value: 0.824
limit: 250
num_fewshot: 5
@@ -1,11 +0,0 @@
# bash .buildkite/lm-eval-harness/run-lm-eval-gsm-vllm-baseline.sh -m Qwen/Qwen2.5-1.5B-Instruct -b auto -l 1319 -f 5 -t 1
model_name: "Qwen/Qwen2.5-1.5B-Instruct"
tasks:
- name: "gsm8k"
metrics:
- name: "exact_match,strict-match"
value: 0.54
- name: "exact_match,flexible-extract"
value: 0.59
limit: 1319
num_fewshot: 5
@@ -1,15 +0,0 @@
# For vllm script, with -t option (tensor parallel size)
# bash .buildkite/lm-eval-harness/run-lm-eval-gsm-vllm-baseline.sh -m RedHatAI/Qwen2.5-VL-3B-Instruct-FP8-Dynamic -l 1319 -t 1
model_name: "RedHatAI/Qwen2.5-VL-3B-Instruct-FP8-Dynamic"
required_gpu_arch:
- gfx942
- gfx950
tasks:
- name: "gsm8k"
metrics:
- name: "exact_match,strict-match"
value: 0.47
- name: "exact_match,flexible-extract"
value: 0.64
limit: 1319
num_fewshot: 5
@@ -1,12 +0,0 @@
# For vllm script, with -t option (tensor parallel size).
# bash .buildkite/lm-eval-harness/run-lm-eval-chartqa-vllm-vlm-baseline.sh -m Qwen/Qwen2.5-VL-7B-Instruct -l 2500 -t 1
model_name: "Qwen/Qwen2.5-VL-7B-Instruct"
backend: "vllm-vlm"
tasks:
- name: "chartqa"
metrics:
- name: "relaxed_accuracy,none"
value: 0.855
limit: 2500
num_fewshot: 0
@@ -1,17 +0,0 @@
model_name: "Qwen/Qwen3-235B-A22B-Instruct-2507-FP8"
required_gpu_arch:
- gfx942
- gfx950
tasks:
- name: "mmlu_pro"
metrics:
- name: "exact_match,custom-extract"
value: 0.82
limit: 250 # will run on 250 * 14 subjects = 3500 samples
num_fewshot: 5
enforce_eager: false # we use false to speed up the eval process
kv_cache_dtype: fp8 # we use fp8 to speed up the eval process
max_model_len: 40960
apply_chat_template: true
fewshot_as_multiturn: true
gen_kwargs: "temperature=0,top_p=1,top_k=0,max_gen_toks=5632,until=<|ENDANSWER|>"
@@ -1,2 +0,0 @@
Qwen3-235B-A22B-Instruct-2507-FP8.yaml
NVIDIA-Nemotron-3-Nano-30B-A3B-FP8.yaml
@@ -1,2 +0,0 @@
Qwen3-235B-A22B-Instruct-2507-FP8.yaml
DeepSeek-V4-Flash-MXFP4.yaml
@@ -1 +0,0 @@
Meta-Llama-4-Maverick-17B-128E-Instruct-FP8.yaml
@@ -1 +0,0 @@
NVIDIA-Nemotron-3-Nano-30B-A3B-FP8-xpu.yaml
@@ -1,6 +0,0 @@
Meta-Llama-3-70B-Instruct-FBGEMM-nonuniform.yaml
Meta-Llama-3-70B-Instruct.yaml
Mixtral-8x7B-Instruct-v0.1.yaml
Qwen2-57B-A14-Instruct.yaml
DeepSeek-V2-Lite-Chat.yaml
NVIDIA-Nemotron-3-Nano-30B-A3B-BF16.yaml
@@ -1 +0,0 @@
Meta-Llama-4-Maverick-17B-128E-Instruct-FP8-MM.yaml
@@ -1 +0,0 @@
Qwen2.5-VL-7B-Instruct.yaml
@@ -1,6 +0,0 @@
Qwen2.5-1.5B-Instruct.yaml
Meta-Llama-3.2-1B-Instruct-INT8-compressed-tensors.yaml
Meta-Llama-3-8B-Instruct-INT8-compressed-tensors-asym.yaml
Meta-Llama-3-8B-Instruct-nonuniform-compressed-tensors.yaml
Qwen2.5-VL-3B-Instruct-FP8-dynamic.yaml
Qwen1.5-MoE-W4A16-compressed-tensors.yaml
@@ -1,6 +0,0 @@
Qwen2.5-1.5B-Instruct.yaml
Meta-Llama-3.2-1B-Instruct-INT8-compressed-tensors.yaml
Meta-Llama-3-8B-Instruct-INT8-compressed-tensors-asym.yaml
Meta-Llama-3-8B-Instruct-nonuniform-compressed-tensors.yaml
Qwen2.5-VL-3B-Instruct-FP8-dynamic.yaml
Qwen1.5-MoE-W4A16-compressed-tensors.yaml
-44
View File
@@ -1,44 +0,0 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
from pathlib import Path
import pytest
def pytest_addoption(parser):
parser.addoption(
"--config-list-file",
action="store",
help="Path to the file listing model config YAMLs (one per line)",
)
parser.addoption(
"--tp-size",
action="store",
default="1",
help="Tensor parallel size to use for evaluation",
)
@pytest.fixture(scope="session")
def config_list_file(pytestconfig, config_dir):
rel_path = pytestconfig.getoption("--config-list-file")
return config_dir / rel_path
@pytest.fixture(scope="session")
def tp_size(pytestconfig):
return pytestconfig.getoption("--tp-size")
def pytest_generate_tests(metafunc):
if "config_filename" in metafunc.fixturenames:
rel_path = metafunc.config.getoption("--config-list-file")
config_list_file = Path(rel_path).resolve()
config_dir = config_list_file.parent
with open(config_list_file, encoding="utf-8") as f:
configs = [
config_dir / line.strip()
for line in f
if line.strip() and not line.startswith("#")
]
metafunc.parametrize("config_filename", configs)
@@ -1,44 +0,0 @@
#!/bin/bash
# We can use this script to compute baseline accuracy on chartqa for vllm.
#
# Make sure you have lm-eval-harness installed:
# pip install "lm-eval[api]>=0.4.12"
usage() {
echo``
echo "Runs lm eval harness on ChartQA using multimodal vllm."
echo "This pathway is intended to be used to create baselines for "
echo "our correctness tests in vllm's CI."
echo
echo "usage: ${0} <options>"
echo
echo " -m - huggingface stub or local directory of the model"
echo " -l - limit number of samples to run"
echo " -t - tensor parallel size to run at"
echo
}
while getopts "m:l:t:" OPT; do
case ${OPT} in
m )
MODEL="$OPTARG"
;;
l )
LIMIT="$OPTARG"
;;
t )
TP_SIZE="$OPTARG"
;;
\? )
usage
exit 1
;;
esac
done
lm_eval --model vllm-vlm \
--model_args "pretrained=$MODEL,tensor_parallel_size=$TP_SIZE" \
--tasks chartqa \
--batch_size auto \
--apply_chat_template \
--limit "$LIMIT"
@@ -1,46 +0,0 @@
#!/bin/bash
# We can use this script to compute baseline accuracy on GSM for transformers.
#
# Make sure you have lm-eval-harness installed:
# pip install "lm-eval[api]>=0.4.12"
usage() {
echo``
echo "Runs lm eval harness on GSM8k using huggingface transformers."
echo "This pathway is intended to be used to create baselines for "
echo "our automated nm-test-accuracy workflow"
echo
echo "usage: ${0} <options>"
echo
echo " -m - huggingface stub or local directory of the model"
echo " -b - batch size to run the evaluation at"
echo " -l - limit number of samples to run"
echo " -f - number of fewshot samples to use"
echo
}
while getopts "m:b:l:f:" OPT; do
case ${OPT} in
m )
MODEL="$OPTARG"
;;
b )
BATCH_SIZE="$OPTARG"
;;
l )
LIMIT="$OPTARG"
;;
f )
FEWSHOT="$OPTARG"
;;
\? )
usage
exit 1
;;
esac
done
lm_eval --model hf \
--model_args "pretrained=$MODEL,parallelize=True" \
--tasks gsm8k --num_fewshot "$FEWSHOT" --limit "$LIMIT" \
--batch_size "$BATCH_SIZE"
@@ -1,51 +0,0 @@
#!/bin/bash
# We can use this script to compute baseline accuracy on GSM for vllm.
# We use this for fp8, which HF does not support.
#
# Make sure you have lm-eval-harness installed:
# pip install "lm-eval[api]>=0.4.12"
usage() {
echo``
echo "Runs lm eval harness on GSM8k using huggingface transformers."
echo "This pathway is intended to be used to create baselines for "
echo "our automated nm-test-accuracy workflow"
echo
echo "usage: ${0} <options>"
echo
echo " -m - huggingface stub or local directory of the model"
echo " -b - batch size to run the evaluation at"
echo " -l - limit number of samples to run"
echo " -f - number of fewshot samples to use"
echo " -t - tensor parallel size to run at"
echo
}
while getopts "m:b:l:f:t:" OPT; do
case ${OPT} in
m )
MODEL="$OPTARG"
;;
b )
BATCH_SIZE="$OPTARG"
;;
l )
LIMIT="$OPTARG"
;;
f )
FEWSHOT="$OPTARG"
;;
t )
TP_SIZE="$OPTARG"
;;
\? )
usage
exit 1
;;
esac
done
lm_eval --model vllm \
--model_args "pretrained=$MODEL,tensor_parallel_size=$TP_SIZE,add_bos_token=true,trust_remote_code=true,max_model_len=4096" \
--tasks gsm8k --num_fewshot "$FEWSHOT" --limit "$LIMIT" \
--batch_size "$BATCH_SIZE"
@@ -1,47 +0,0 @@
#!/bin/bash
# We can use this script to compute baseline accuracy on MMLUPRO for vllm.
# We use this for fp8, which HF does not support.
#
# Make sure you have lm-eval-harness installed:
# pip install "lm-eval[api]>=0.4.12"
usage() {
echo``
echo "Runs lm eval harness on MMLU Pro using huggingface transformers."
echo "This pathway is intended to be used to create baselines for "
echo "our automated nm-test-accuracy workflow"
echo
echo "usage: ${0} <options>"
echo
echo " -m - huggingface stub or local directory of the model"
echo " -l - limit number of samples to run"
echo " -f - number of fewshot samples to use"
echo " -t - tensor parallel size to run at"
echo
}
while getopts "m:l:f:t:" OPT; do
case ${OPT} in
m )
MODEL="$OPTARG"
;;
l )
LIMIT="$OPTARG"
;;
f )
FEWSHOT="$OPTARG"
;;
t )
TP_SIZE="$OPTARG"
;;
\? )
usage
exit 1
;;
esac
done
lm_eval --model vllm \
--model_args "pretrained=$MODEL,tensor_parallel_size=$TP_SIZE,add_bos_token=true,trust_remote_code=true,max_model_len=4096" \
--tasks mmlu_pro --num_fewshot "$FEWSHOT" --limit "$LIMIT" \
--batch_size auto
@@ -1,157 +0,0 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
"""LM eval harness on model to compare vs HF baseline computed offline.
Configs are found in configs/$MODEL.yaml
pytest -s -v test_lm_eval_correctness.py \
--config-list-file=configs/models-small.txt \
--tp-size=1
"""
import os
from contextlib import contextmanager
import lm_eval
import pytest
import yaml
from vllm.platforms import current_platform
DEFAULT_RTOL = 0.08
@contextmanager
def scoped_env_vars(new_env: dict[str, str]):
if not new_env:
# Fast path: nothing to do
yield
return
old_values = {}
new_keys = []
try:
for key, value in new_env.items():
if key in os.environ:
old_values[key] = os.environ[key]
else:
new_keys.append(key)
os.environ[key] = str(value)
yield
finally:
# Restore / clean up
for key, value in old_values.items():
os.environ[key] = value
for key in new_keys:
os.environ.pop(key, None)
def launch_lm_eval(eval_config, tp_size):
trust_remote_code = eval_config.get("trust_remote_code", False)
max_model_len = eval_config.get("max_model_len", 4096)
batch_size = eval_config.get("batch_size", "auto")
backend = eval_config.get("backend", "vllm")
enforce_eager = eval_config.get("enforce_eager", "true")
kv_cache_dtype = eval_config.get("kv_cache_dtype", "auto")
model_args = (
f"pretrained={eval_config['model_name']},"
f"tensor_parallel_size={tp_size},"
f"enforce_eager={enforce_eager},"
f"kv_cache_dtype={kv_cache_dtype},"
f"add_bos_token=true,"
f"trust_remote_code={trust_remote_code},"
f"max_model_len={max_model_len},"
"allow_deprecated_quantization=True,"
)
if current_platform.is_rocm() and "Nemotron-3" in eval_config["model_name"]:
model_args += "attention_backend=TRITON_ATTN"
moe_backend = eval_config.get("moe_backend", None)
if moe_backend is not None:
model_args += f"moe_backend={moe_backend},"
if current_platform.is_rocm():
rocm_load_strategy = eval_config.get("rocm_safetensors_load_strategy")
if rocm_load_strategy is not None:
model_args += f"safetensors_load_strategy={rocm_load_strategy},"
tokenizer_mode = eval_config.get("tokenizer_mode", None)
if tokenizer_mode is not None:
model_args += f"tokenizer_mode={tokenizer_mode},"
env_vars = eval_config.get("env_vars", None)
with scoped_env_vars(env_vars):
results = lm_eval.simple_evaluate(
model=backend,
model_args=model_args,
tasks=[task["name"] for task in eval_config["tasks"]],
num_fewshot=eval_config["num_fewshot"],
limit=eval_config["limit"],
# TODO(yeq): using chat template w/ fewshot_as_multiturn is supposed help
# text models. however, this is regressing measured strict-match for
# existing text models in CI, so only apply it for mm, or explicitly set
apply_chat_template=eval_config.get(
"apply_chat_template", backend == "vllm-vlm"
),
fewshot_as_multiturn=eval_config.get("fewshot_as_multiturn", False),
# Forward decoding and early-stop controls (e.g., max_gen_toks, until=...)
gen_kwargs=eval_config.get("gen_kwargs"),
batch_size=batch_size,
)
return results
def _check_rocm_gpu_arch_requirement(eval_config):
"""Skip the test if the model requires a ROCm GPU arch not present.
Model YAML configs can specify::
required_gpu_arch:
- gfx942
- gfx950
The check only applies on ROCm. On other platforms (e.g. CUDA) the
field is ignored so that shared config files work for both NVIDIA and
AMD CI pipelines.
"""
required_archs = eval_config.get("required_gpu_arch")
if not required_archs:
return
if not current_platform.is_rocm():
return
from vllm.platforms.rocm import _GCN_ARCH # noqa: E402
if not any(arch in _GCN_ARCH for arch in required_archs):
pytest.skip(
f"Model requires GPU arch {required_archs}, "
f"but detected arch is '{_GCN_ARCH}'"
)
def test_lm_eval_correctness_param(config_filename, tp_size):
eval_config = yaml.safe_load(config_filename.read_text(encoding="utf-8"))
_check_rocm_gpu_arch_requirement(eval_config)
results = launch_lm_eval(eval_config, tp_size)
rtol = eval_config.get("rtol", DEFAULT_RTOL)
success = True
for task in eval_config["tasks"]:
for metric in task["metrics"]:
ground_truth = metric["value"]
measured_value = results["results"][task["name"]][metric["name"]]
print(
f"{task['name']} | {metric['name']}: "
f"ground_truth={ground_truth:.3f} | "
f"measured={measured_value:.3f} | rtol={rtol}"
)
min_acceptable = ground_truth * (1 - rtol)
success = success and measured_value >= min_acceptable
assert success

Some files were not shown because too many files have changed in this diff Show More