[Chore] Remove vendored code made redundant by Transformers 5.16.1 (#60623)

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

.agents/skills/transformers-upgrade/SKILL.md

@@ -0,0 +1,178 @@
+---
+name: transformers-upgrade
+description: "Playbook for changing vLLM's Transformers version bounds. Use when bumping the pinned or maximum Transformers version, raising the minimum Transformers version, or deleting vendored configs, processors, or compat shims that upstream Transformers now provides."
+---
+
+# Upgrading Transformers
+
+The supported range lives in `requirements/common.txt`. CI pins one exact
+version in `requirements/test/*.in` and the compiled `requirements/test/*.txt`.
+A version change is one of two jobs, and they usually come as a pair:
+
+- **Bump the pin**: a new Transformers release becomes the CI version and
+  the upper bound. vLLM grows compat code for it.
+- **Raise the floor**: the minimum moves up. That compat code, and every
+  vendored config or processor upstream now ships, becomes dead weight.
+  Every floor raise ends with an **unvendoring pass**.
+
+## Bump the pin
+
+1. Edit the pin line in each `requirements/test/*.in` and `*.txt` and the
+   upper bound in `requirements/common.txt`. Past bumps edit only the pin
+   line in the compiled files; see `git log -- requirements/common.txt`.
+2. Read the release notes for config validation changes, renamed auto-mapping
+   formats, and removed attributes; these are what break vLLM.
+3. Search `tests/` for `min_transformers_version` and skip markers naming the
+   new version, and un-gate them.
+4. Run the suites in [Testing](#testing) and fix what breaks. Mark every
+   shim with a `# Transformers >= X` comment so the next floor raise finds it.
+
+Done when the [Testing](#testing) suites pass on the new pin and every test
+waiting on this version is un-gated.
+
+## Raise the floor (unvendoring pass)
+
+### 1. Inventory
+
+Inventory candidates with the floor version installed (no vLLM install needed):
+
+```bash
+uv run --no-project --with transformers==<floor> \
+    python .agents/skills/transformers-upgrade/scripts/find_upstreamed.py
+```
+
+It lists:
+
+- vendored configs whose `model_type` upstream now owns;
+- `_CONFIG_REGISTRY` overrides of upstream types;
+- registered processors that collide with upstream class names or models;
+- version-gated code at or below the floor;
+- code that reads the installed Transformers version, whatever its threshold;
+- patches citing a Transformers issue or PR (with whether a referenced PR is
+  in the floor release) or a future minimum version.
+
+It finds candidates by name only. Every hit still needs a verdict.
+
+### 2. Port each candidate to upstream
+
+Every hit is deleted by default. A vendored copy is a liability: it drifts
+from upstream, hides upstream fixes, and shadows the native class. Port the
+vLLM model code to the upstream schema instead, renaming attribute reads and
+deriving values from upstream fields.
+
+Fan candidates out to parallel subagents. Each one compares the vendored and
+upstream classes, then greps every attribute the vLLM model code reads.
+[Upstreaming hazards](#upstreaming-hazards) lists the usual fixes.
+
+Keep a vendored class only when upstream cannot load a checkpoint vLLM must
+keep serving, and record why in the PR description. A different upstream
+schema, a model-code rewrite, or a needed eval is work for the PR, not a
+reason to keep.
+
+### 3. Prove it on real checkpoints
+
+For every checkpoint in `tests/models/registry.py` that uses a ported
+config, dump what vLLM builds before and after the change, then diff the two.
+Run both dumps; each catches what the other misses:
+
+- `dump_configs.py`: the resolved config, attribute by attribute.
+- `dump_model.py`: the model built on the meta device with no GPU or weights.
+  It records ModelConfig sizes, every parameter shape, and every module's
+  scalar attributes (`head_dim`, `num_kv_heads`, `scaling`, ...). Missing
+  fields that are read with defaults only show up here.
+
+```bash
+git worktree add --detach <scratch>/vllm-base upstream/main
+S=.agents/skills/transformers-upgrade/scripts
+for dump in dump_configs dump_model; do
+    PYTHONPATH=<scratch>/vllm-base .venv/bin/python $S/$dump.py before.jsonl <repos>
+    PYTHONPATH=$PWD .venv/bin/python $S/$dump.py after.jsonl <repos>
+    .venv/bin/python $S/diff_dumps.py before.jsonl after.jsonl
+done
+```
+
+Add `--trust-remote-code` for checkpoints whose `config.json` has an
+`auto_map`, and dump both ways.
+
+How to read the diffs:
+
+- **A `<MISSING>` config field the model reads:** this crashes or silently
+  falls back to a default. Port the model code to the upstream field.
+- **New upstream-only defaults** (`attention_bias=False`,
+  `pad_token_id=None`): usually benign. A model diff of 0 confirms it.
+- **Any model diff:** this is a numerics change, and the PR needs a model
+  eval. For example, A.X-K1 gained the checkpoint's YaRN rope scaling, which
+  the vendored class had been dropping; it shows up as
+  `layers.*.self_attn.scaling (x61)`.
+
+### 4. Delete
+
+For each deleted config:
+
+- Delete the file.
+- Remove its entries from `_CLASS_TO_MODULE` and `__all__` in
+  `vllm/transformers_utils/configs/__init__.py`.
+- Remove its entries from `_CONFIG_REGISTRY` in
+  `vllm/transformers_utils/config.py`.
+- Switch every importer, including tests, to `from transformers import ...`.
+- Remove the `source_file_dependencies` lines naming the file from
+  `.buildkite/`.
+
+Deleted processors follow the same steps in
+`vllm/transformers_utils/processors/__init__.py`.
+
+Done when every line of the inventory output is deleted, or kept with a
+reason that meets the bar in step 2. The PR also carries the config diff
+results and evals for any changed values.
+
+## Upstreaming hazards
+
+Upstream configs are `@strict` dataclasses. Checkpoint keys upstream does not
+declare survive as plain attributes. Defaults that only vLLM sets, and values
+vLLM derives, vanish.
+
+| Hazard | Signal | Fix |
+| --- | --- | --- |
+| Upstream consumes legacy keys | `__post_init__` pops keys the model reads (`deepseek_v4` `compress_ratios`) | Read the upstream replacement (`layer_types`, `mlp_layer_types`, per-layer `rope_parameters`) |
+| Renamed or dropped fields | Diff shows `<MISSING>` for a field the model reads | Read the upstream name; derive dropped flags from what implies them (`enable_moe_block` from `num_experts`) |
+| Type check on the config | `ctx.get_hf_config(SomeConfig)` relies on the vendored base class | Accept the upstream class too, e.g. a `Cosmos3ProcessingInfo` overriding `get_hf_config` |
+| Remote code takes over | Checkpoint `config.json` has `auto_map`; with `--trust-remote-code`, the hub class replaces the native one once vLLM stops registering its own | Check the remote class loads; if not, register the upstream class for that `model_type` |
+| Read-only upstream fields | Properties or derived fields break subclasses and `hf_overrides` (`nemotron_h` `hybrid_override_pattern`) | Override the source field (`layers_block_type`) instead |
+| Behavioural rewrite | Upstream derives fields differently (`diffusion_gemma` `sliding_window`) | Upstream matches the reference implementation; verify with an eval |
+| Original checkpoint format | Upstream targets `-hf` conversions; vLLM serves the originals (`qwen3_asr`, `qianfan_ocr`) | Keep only if upstream cannot load the original at all |
+| Processor output contract | Model code consumes keys upstream does not emit | Port the model to upstream's keys |
+| Name collision | A registered vendored processor shares an upstream class name and hijacks `get_processor` (`InternVLProcessor`) | Drop the registry entry |
+
+Version gates from the inventory need a decision:
+
+- **Code** only reachable below the floor is deleted. This covers:
+    - comparisons against `TRANSFORMERS_VERSION`;
+    - branches labelled `# Transformers < X`;
+    - fallbacks for fields that upstream configs now always set, e.g.
+    `mlp_layer_types`, `per_layer_config` or legacy `layer_types` names.
+
+  A `skipif` whose condition is now always true becomes an unconditional
+  `skip` with the same reason.
+- **`min_transformers_version`** at or below the floor is removed.
+- **`max_transformers_version`** below the floor means CI never tests that
+  model; report it rather than deleting it silently.
+- **Patches citing a Transformers issue or PR:**
+    - Delete the patch when its fix is in the floor release.
+    - For an issue, find the PR that closed it and check that PR. A closed
+      issue is not a fixed one: #43329 was closed while the bug was still in
+      5.16.1.
+    - Re-run the patched behaviour before deleting. A fix in the release does
+      not always cover vLLM's path: #47924 made Emu3's processor add BOS, but
+      vLLM tokenizes without the processor.
+    - Links that only explain a design choice (e.g. a float32 RoPE note) are
+      not patches; leave them.
+
+## Testing
+
+```bash
+.venv/bin/python -m pytest tests/transformers_utils tests/config tests/tokenizers_
+.venv/bin/python -m pytest tests/models/multimodal/processing/test_common.py -k "<models>"
+```
+
+Add `tests/models/test_initialization.py -k "<archs>"` where a GPU is
+available, and a model eval for any model whose resolved config changed.

.agents/skills/transformers-upgrade/agents/openai.yaml

@@ -0,0 +1,4 @@
+interface:
+  display_name: "Transformers Upgrade"
+  short_description: "Bump vLLM's Transformers version and delete vendored code upstream now provides."
+  default_prompt: "Use $transformers-upgrade to raise the minimum Transformers version and remove vendored configs, processors, and shims it makes redundant."

.agents/skills/transformers-upgrade/scripts/diff_dumps.py

@@ -0,0 +1,51 @@
+# SPDX-License-Identifier: Apache-2.0
+# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
+"""Diff two dump_configs.py or dump_model.py outputs, per repo.
+
+python diff_dumps.py before.jsonl after.jsonl
+"""
+
+import json
+import sys
+from collections import Counter
+
+import regex as re
+
+IGNORED = {"transformers_version", "_name_or_path", "architectures", "auto_map"}
+
+
+def load(path: str) -> dict[str, dict]:
+    with open(path) as f:
+        return {record["repo"]: record for record in map(json.loads, f)}
+
+
+def diff(before: dict, after: dict, prefix: str = ""):
+    for key in sorted(set(before) | set(after)):
+        if key in IGNORED:
+            continue
+        old, new = before.get(key, "<MISSING>"), after.get(key, "<MISSING>")
+        if isinstance(old, dict) and isinstance(new, dict) and key != "rope_parameters":
+            yield from diff(old, new, f"{prefix}{key}.")
+        elif old != new:
+            yield f"{prefix}{key}", json.dumps(old), json.dumps(new)
+
+
+def collapse(diffs) -> Counter:
+    """Merge diffs that differ only by layer index, e.g. `layers.*.scaling`."""
+    return Counter(
+        (re.sub(r"\.\d+\.", ".*.", path), old, new) for path, old, new in diffs
+    )
+
+
+before, after = load(sys.argv[1]), load(sys.argv[2])
+for repo, old in before.items():
+    new = after.get(repo, {"error": "missing from after"})
+    if "error" in old or "error" in new:
+        print(f"== {repo}\n   before: {old.get('error', 'ok')}")
+        print(f"   after:  {new.get('error', 'ok')}")
+        continue
+    diffs = collapse(diff(old["cfg"], new["cfg"]))
+    print(f"== {repo}: {old['cls']} -> {new['cls']} ({sum(diffs.values())} diffs)")
+    for (path, old_value, new_value), count in diffs.items():
+        times = f" (x{count})" if count > 1 else ""
+        print(f"   {path}{times}: before={old_value} after={new_value}")

.agents/skills/transformers-upgrade/scripts/dump_configs.py

@@ -0,0 +1,30 @@
+# SPDX-License-Identifier: Apache-2.0
+# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
+"""Dump the config vLLM resolves for each checkpoint, one JSON line per repo.
+
+Run once per vLLM tree (before and after a change), then compare the two
+outputs with diff_dumps.py:
+
+    PYTHONPATH=<vllm tree> .venv/bin/python dump_configs.py out.jsonl <repo>...
+
+Pass --trust-remote-code to resolve checkpoints the way users who set it do.
+"""
+
+import json
+import sys
+
+from vllm.transformers_utils.config import get_config
+
+out, *args = sys.argv[1:]
+trust_remote_code = "--trust-remote-code" in args
+with open(out, "w") as f:
+    for repo in (arg for arg in args if arg != "--trust-remote-code"):
+        try:
+            config = get_config(repo, trust_remote_code=trust_remote_code)
+            cls = f"{type(config).__module__}.{type(config).__name__}"
+            record = {"repo": repo, "cls": cls, "cfg": config.to_dict()}
+        except Exception as e:
+            error = f"{type(e).__name__}: {e}".splitlines()[0]
+            record = {"repo": repo, "error": error}
+        f.write(json.dumps(record, default=str, sort_keys=True) + "\n")
+        f.flush()

.agents/skills/transformers-upgrade/scripts/dump_model.py

@@ -0,0 +1,114 @@
+# SPDX-License-Identifier: Apache-2.0
+# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
+"""Build each model on the meta device and dump what vLLM constructs.
+
+Records ModelConfig-derived sizes, every parameter shape, and scalar
+attributes of every module (head_dim, num_kv_heads, sliding_window, ...).
+Needs no GPU and no weights. Run once per vLLM tree, then compare the two
+outputs with diff_dumps.py:
+
+    PYTHONPATH=<vllm tree> .venv/bin/python dump_model.py out.jsonl <repo>...
+
+Pass --trust-remote-code to resolve checkpoints the way users who set it do.
+"""
+
+import json
+import socket
+import sys
+
+import torch
+
+from vllm.config import set_current_vllm_config
+from vllm.config.vllm import VllmConfig
+from vllm.distributed import (
+    init_distributed_environment,
+    initialize_model_parallel,
+    model_parallel_is_initialized,
+)
+from vllm.engine.arg_utils import EngineArgs
+from vllm.model_executor.model_loader.utils import initialize_model
+from vllm.utils.torch_utils import set_default_torch_dtype
+
+# Construction only: skip model runner checks that need a GPU or Triton
+VllmConfig._get_v1_model_runner_unsupported_features = lambda self: []  # type: ignore[method-assign]
+
+# Model code that moves submodules to a device must leave them on meta
+_module_to = torch.nn.Module.to
+
+
+def _meta_safe_to(self, *args, **kwargs):
+    kwargs.pop("device", None)
+    args = tuple(a for a in args if not isinstance(a, (torch.device, str, int)))
+    return _module_to(self, *args, **kwargs)
+
+
+torch.nn.Module.to = _meta_safe_to  # type: ignore[method-assign]
+
+SCALARS = (bool, int, float, str)
+
+
+def module_attrs(model: torch.nn.Module) -> dict[str, dict]:
+    attrs = {}
+    for name, module in model.named_modules():
+        scalars = {
+            key: value
+            for key, value in vars(module).items()
+            if not key.startswith("_") and isinstance(value, SCALARS)
+        }
+        if scalars:
+            attrs[name or "<root>"] = scalars
+    return attrs
+
+
+def dump(repo: str, trust_remote_code: bool) -> dict:
+    args = EngineArgs(
+        model=repo,
+        load_format="dummy",
+        enforce_eager=True,
+        max_model_len=4096,
+        trust_remote_code=trust_remote_code,
+    )
+    vllm_config = args.create_engine_config()
+    model_config = vllm_config.model_config
+    with set_current_vllm_config(vllm_config):
+        if not model_parallel_is_initialized():
+            with socket.socket() as sock:
+                sock.bind(("", 0))
+                port = sock.getsockname()[1]
+            init_distributed_environment(1, 0, f"tcp://127.0.0.1:{port}", 0, "gloo")
+            initialize_model_parallel(1, 1)
+        with set_default_torch_dtype(model_config.dtype), torch.device("meta"):
+            model = initialize_model(vllm_config=vllm_config)
+    return {
+        "repo": repo,
+        "cls": type(model).__name__,
+        "cfg": {
+            "model_config": {
+                "head_size": model_config.get_head_size(),
+                "total_num_kv_heads": model_config.get_total_num_kv_heads(),
+                "sliding_window": model_config.get_sliding_window(),
+                "num_layers": model_config.get_num_layers(vllm_config.parallel_config),
+            },
+            "params": {n: list(p.shape) for n, p in model.named_parameters()},
+            "modules": module_attrs(model),
+        },
+    }
+
+
+def main() -> None:
+    out, *args = sys.argv[1:]
+    trust_remote_code = "--trust-remote-code" in args
+    repos = [arg for arg in args if arg != "--trust-remote-code"]
+    with open(out, "w") as f:
+        for repo in repos:
+            try:
+                record = dump(repo, trust_remote_code)
+            except Exception as e:
+                error = f"{type(e).__name__}: {e}".splitlines()[0]
+                record = {"repo": repo, "error": error}
+            f.write(json.dumps(record, default=str, sort_keys=True) + "\n")
+            f.flush()
+
+
+if __name__ == "__main__":
+    main()

.agents/skills/transformers-upgrade/scripts/find_upstreamed.py

@@ -0,0 +1,147 @@
+# SPDX-License-Identifier: Apache-2.0
+# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
+"""List vendored vLLM code that the installed Transformers may make redundant.
+
+Run with the Transformers version under test, from the vLLM repo root:
+
+    uv run --no-project --with transformers==<version> \
+        python .agents/skills/transformers-upgrade/scripts/find_upstreamed.py
+
+Imports nothing from vLLM, so no vLLM install is needed. Uses the `gh` CLI, if
+available, to check whether referenced Transformers PRs are in this release.
+"""
+
+import json
+import subprocess
+import sys
+from pathlib import Path
+
+import regex as re
+import transformers
+from packaging.version import Version
+from transformers.models.auto.configuration_auto import CONFIG_MAPPING_NAMES
+from transformers.models.auto.processing_auto import PROCESSOR_MAPPING_NAMES
+
+ROOT = Path(sys.argv[1] if len(sys.argv) > 1 else ".")
+FLOOR = Version(transformers.__version__)
+UPSTREAM_NAMES = set(dir(transformers))
+
+MODEL_TYPE = re.compile(r'model_type\s*(?::\s*str\s*)?=\s*"([^"]+)"')
+REGISTRY_KEY = re.compile(r'^\s+(?:\*\*\{)?"?([\w-]+)"?(?:=|": )"(\w+)"', re.M)
+CLASS_KEY = re.compile(r'^\s+"(\w+)": ', re.M)
+UPSTREAM_REF = re.compile(
+    r"huggingface/transformers(?:/(?:issues|pull)/|#)(\d+)"
+    r"|(?:once|until|when) [Tt]ransformers [\d.]+ is the minimum"
+)
+VERSIONED = re.compile(
+    r"(?:min_transformers_version|max_transformers_version|check_version|Version)"
+    r'\(?\s*=?\s*"([45]\.\d+(?:\.\d+)?)[^"]*"'
+    r"|[Tt]ransformers\s*(?:v|[<>]=?\s*)([45]\.\d+(?:\.\d+)?|4)\b"
+)
+READS_VERSION = re.compile(
+    r"transformers import __version__|transformers\.__version__"
+    r"|version\([\"']transformers[\"']\)|TRANSFORMERS_VERSION|check_version\("
+)
+
+
+def section(title: str) -> None:
+    print(f"\n## {title}")
+
+
+def vendored_configs() -> None:
+    section(f"Vendored configs whose model_type is upstream in {FLOOR}")
+    files = sorted(ROOT.glob("vllm/transformers_utils/configs/*.py"))
+    files += sorted(ROOT.glob("vllm/models/*/config*.py"))
+    files += sorted(ROOT.glob("vllm/models/*/configs/*.py"))
+    for path in files:
+        types = sorted(set(MODEL_TYPE.findall(path.read_text())))
+        hits = [t for t in types if t in CONFIG_MAPPING_NAMES]
+        if hits:
+            upstream = ", ".join(f"{t} -> {CONFIG_MAPPING_NAMES[t]}" for t in hits)
+            print(f"- {path.relative_to(ROOT)}: {upstream}")
+
+    registry = (ROOT / "vllm/transformers_utils/config.py").read_text()
+    start = registry.index("_CONFIG_REGISTRY")
+    body = registry[start : registry.index("\n)\n", start)]
+    section("_CONFIG_REGISTRY entries that override an upstream model_type")
+    for model_type, cls in REGISTRY_KEY.findall(body):
+        if model_type in CONFIG_MAPPING_NAMES:
+            print(f"- {model_type}: vLLM {cls} over {CONFIG_MAPPING_NAMES[model_type]}")
+
+
+def vendored_processors() -> None:
+    init = (ROOT / "vllm/transformers_utils/processors/__init__.py").read_text()
+    names = sorted(set(CLASS_KEY.findall(init)))
+    section("Registered processor names that collide with upstream classes")
+    for name in names:
+        if name in UPSTREAM_NAMES:
+            print(f"- {name}")
+    section("Processor files named after a model_type with an upstream processor")
+    for path in sorted(ROOT.glob("vllm/transformers_utils/processors/*.py")):
+        if path.stem in PROCESSOR_MAPPING_NAMES:
+            upstream = PROCESSOR_MAPPING_NAMES[path.stem]
+            print(f"- {path.relative_to(ROOT)} -> {upstream}")
+
+
+def version_gates() -> None:
+    files = sorted([*ROOT.glob("vllm/**/*.py"), *ROOT.glob("tests/**/*.py")])
+    section(f"Version-gated code at or below {FLOOR} (review each)")
+    for path in files:
+        for lineno, line in enumerate(path.read_text().splitlines(), 1):
+            for match in VERSIONED.finditer(line):
+                version = Version(match.group(1) or match.group(2))
+                if version <= FLOOR:
+                    print(f"- {path.relative_to(ROOT)}:{lineno}: {line.strip()}")
+                    break
+    section("Code that reads the installed Transformers version (review each)")
+    for path in files:
+        for lineno, line in enumerate(path.read_text().splitlines(), 1):
+            if READS_VERSION.search(line):
+                print(f"- {path.relative_to(ROOT)}:{lineno}: {line.strip()}")
+
+
+def upstream_status(number: str) -> str:
+    """Whether Transformers issue/PR `number` is resolved in this release."""
+
+    def gh(path: str) -> dict | None:
+        try:
+            out = subprocess.run(["gh", "api", path], capture_output=True, text=True)
+        except FileNotFoundError:
+            return None
+        return json.loads(out.stdout) if out.returncode == 0 else None
+
+    repo = "repos/huggingface/transformers"
+    if (issue := gh(f"{repo}/issues/{number}")) is None:
+        return "status unknown"
+    if "pull_request" not in issue:
+        return f"issue {issue['state']}, check which PR fixed it"
+    pr = gh(f"{repo}/pulls/{number}")
+    if pr is None or not pr["merged_at"]:
+        return "PR not merged"
+    compare = gh(f"{repo}/compare/v{FLOOR}...{pr['merge_commit_sha']}")
+    included = compare and compare["status"] in ("behind", "identical")
+    return f"PR merged, {'in' if included else 'not in'} v{FLOOR}"
+
+
+def upstream_refs() -> None:
+    section("Patches referencing Transformers issues/PRs or a future minimum")
+    files = sorted([*ROOT.glob("vllm/**/*.py"), *ROOT.glob("tests/**/*.py")])
+    statuses: dict[str, str] = {}
+    for path in files:
+        for lineno, line in enumerate(path.read_text().splitlines(), 1):
+            if (match := UPSTREAM_REF.search(line)) is None:
+                continue
+            note = ""
+            if number := match.group(1):
+                if number not in statuses:
+                    statuses[number] = upstream_status(number)
+                note = f" [#{number}: {statuses[number]}]"
+            print(f"- {path.relative_to(ROOT)}:{lineno}:{note} {line.strip()}")
+
+
+if __name__ == "__main__":
+    print(f"# Transformers {FLOOR}")
+    vendored_configs()
+    vendored_processors()
+    version_gates()
+    upstream_refs()