vLLM/recipes/models/Google/gemma-4-12B-it.yaml

meta:
  title: "Gemma 4 12B IT"
  slug: "gemma-4-12b-it"
  provider: "Google"
  description: "Google's encoder-free unified Gemma 4 dense model (12B) with native text, image, and audio, plus thinking mode and tool-use protocol."
  date_added: 2026-06-03
  date_updated: 2026-09-09
  tasks:
    - multimodal
    - text
  performance_headline: "Encoder-free unified multimodal model with audio, structured thinking, and function calling — runs on a single 40 GB+ GPU"
  related_recipes:
    - google/gemma-4-E4B-it
    - google/gemma-4-26B-A4B-it
    - google/gemma-4-31B-it
  hardware:
    h100: verified
    arc_pro_b60: verified
    arc_pro_b70: verified

model:
  model_id: "google/gemma-4-12B-it"
  min_vllm_version: "0.23.0"
  docker_image: "vllm/vllm-openai:gemma4-unified"
  install:
    docker: {}
    pip: {}
  architecture: dense
  parameter_count: "12B"
  active_parameters: "12B"
  context_length: 131072
  base_args: []
  base_env: {}

dependencies:
  - note: "Audio extras — only needed when serving the audio modality"
    command: 'uv pip install "vllm[audio]"'
    optional: true

features:
  tool_calling:
    description: "Enable automatic tool choice with Gemma 4 parser and chat template"
    args:
      - "--enable-auto-tool-choice"
      - "--tool-call-parser"
      - "gemma4"
      - "--chat-template"
      - "examples/tool_chat_template_gemma4.jinja"
  reasoning:
    description: "Enable structured thinking/reasoning output"
    args:
      - "--reasoning-parser"
      - "gemma4"
  spec_decoding:
    description: "MTP speculative decoding for accelerated inference"
    args:
      - "--speculative-config"
      - '{"model":"google/gemma-4-12B-it-assistant","num_speculative_tokens":4}'

opt_in_features:
  - spec_decoding

variants:
  default:
    precision: bf16
    vram_minimum_gb: 29
    description: "Full BF16 — single 40 GB+ NVIDIA GPU"
  nvfp4:
    model_id: "RedHatAI/gemma-4-12B-it-NVFP4"
    precision: nvfp4
    vram_minimum_gb: 8
    description: "RedHatAI compressed-tensors NVFP4 quantization for Blackwell GPUs"
    extra_args:
      - "--kv-cache-dtype"
      - "fp8"
  w4a16:
    model_id: "google/gemma-4-12B-it-qat-w4a16-ct"
    precision: int4
    vram_minimum_gb: 9
    description: "W4A16 QAT (4-bit weights, 16-bit activations, group_size=32) via compressed-tensors — runs on any GPU"

  fp8:
    model_id: "RedHatAI/gemma-4-12B-it-FP8-Dynamic"
    precision: fp8
    vram_minimum_gb: 15
    description: "RedHatAI FP8 (E4M3) with dynamic per-token activation quantization"
    extra_args:
      - "--kv-cache-dtype"
      - "fp8"

compatible_strategies:
  - single_node_tp
  - multi_node_tp

hardware_overrides:
  xpu:
    extra_args:
      - "--enforce-eager"
      - "--attention-backend"
      - "flash_attn"
      - "--trust-remote-code"
      - "--gpu-memory-utilization"
      - "0.9"
      - "--no-enable-prefix-caching"
      - "--max-num-batched-tokens"
      - "8192"
      - "--max-model-len"
      - "5120"
      - "--tensor-parallel-size"
      - "2"

strategy_overrides:
  single_node_tp:
    tp: 1

guide: |
  ## Overview

  [Gemma 4](https://ai.google.dev/gemma/docs) is Google's most capable open model family, featuring a unified multimodal architecture that natively processes text, images, and audio. The **12B Unified** model is *encoder-free*: instead of dedicated vision/audio encoders, it projects raw image patches and audio waveforms directly into the decoder's embedding space through lightweight linear layers, so every modality flows into a single decoder-only transformer. Gemma 4 models support structured thinking/reasoning, function calling with a custom tool-use protocol, and dynamic vision resolution — all available through vLLM's OpenAI-compatible API.

  ### Key Features
  - **Encoder-free multimodality**: Text, image, and audio inputs handled by a single decoder-only transformer — no separate encoders to load, reducing multimodal latency.
  - **Dual Attention**: Alternating sliding-window (local, 1024 tokens) and global attention; the final layer is always global, with unified Keys/Values and Proportional RoPE on global layers for long-context efficiency.
  - **Thinking Mode**: Structured reasoning via `<|channel>thought\n...<channel|>` delimiters.
  - **Function Calling**: Custom tool-call protocol with dedicated special tokens.
  - **Native System Prompt Support**: First-class `system` role.

  ### Supported Variants

  Dense:
  - `google/gemma-4-E2B-it` (effective 2B)
  - `google/gemma-4-E4B-it` (effective 4B)
  - `google/gemma-4-12B-it` (11.95B, encoder-free)
  - `google/gemma-4-31B-it` (31B)

  MoE:
  - `google/gemma-4-26B-A4B-it` (26B total / 4B active)

  > **Context length**: this recipe pins `context_length` to the value in `config.json` (`max_position_embeddings = 131072`, i.e. 128K). The Gemma 4 model card markets the 12B at up to 256K — raise `--max-model-len` only if your vLLM build accepts it.

  TPU support is provided through [vLLM TPU](https://github.com/vllm-project/tpu-inference) with recipes for [Trillium](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/trillium/vLLM/Gemma4) and [Ironwood](https://github.com/AI-Hypercomputer/tpu-recipes/blob/main/inference/ironwood/vLLM/Gemma4/).

  > **Nightly required**: support for the encoder-free 12B Unified model landed in [vllm-project/vllm#44429](https://github.com/vllm-project/vllm/pull/44429) and has not yet shipped in a stable release. Use the pinned Docker image (recommended) or a nightly pip wheel.

  ## Prerequisites

  ### Docker (recommended)
  ```bash
  docker pull vllm/vllm-openai:gemma4-unified            # NVIDIA (CUDA 13; append -cu129 for CUDA 12.9 hosts)
  docker pull vllm/vllm-openai-xpu:latest                # Intel XPU (B60 / B70)
  ```
  TPU images are published separately by [vllm-project/tpu-inference](https://github.com/vllm-project/tpu-inference); see the Trillium / Ironwood tpu-recipes below for the pinned tag.

  ### pip (nightly, NVIDIA CUDA)
  ```bash
  uv venv
  source .venv/bin/activate
  uv pip install -U vllm --pre \
    --extra-index-url https://wheels.vllm.ai/nightly/cu129 \
    --extra-index-url https://download.pytorch.org/whl/cu129 \
    --index-strategy unsafe-best-match
  ```

  ## Deployment Configurations

  ### Quick Start (Single GPU, BF16)
  The 12B fits comfortably on a single 40 GB+ GPU.
  ```bash
  vllm serve google/gemma-4-12B-it \
    --max-model-len 32768 \
    --gpu-memory-utilization 0.90
  ```

  ### Full-Featured Server Launch
  Enables text, image, audio, thinking, and tool calling:
  ```bash
  vllm serve google/gemma-4-12B-it \
    --max-model-len 16384 \
    --gpu-memory-utilization 0.90 \
    --enable-auto-tool-choice \
    --reasoning-parser gemma4 \
    --tool-call-parser gemma4 \
    --chat-template examples/tool_chat_template_gemma4.jinja \
    --limit-mm-per-prompt '{"image": 4, "audio": 1}' \
    --async-scheduling \
    --host 0.0.0.0 \
    --port 8000
  ```

  ### Docker (NVIDIA)
  ```bash
  docker run -itd --name gemma4 \
    --ipc=host --network host --shm-size 16G --gpus all \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    vllm/vllm-openai:gemma4-unified \
      --model google/gemma-4-12B-it \
      --max-model-len 32768 \
      --gpu-memory-utilization 0.90 \
      --host 0.0.0.0 --port 8000
  ```
  On CUDA 12.9 hosts, use the `vllm/vllm-openai:gemma4-unified-cu129` tag instead.

  ### Docker (Intel XPU B60 / B70, TP=2, Full-Featured)

  Validated on 2× Intel Arc Pro B60 / B70 with the official vLLM XPU image.

  ```bash
  docker run --device /dev/dri \
    -v /dev/dri/by-path:/dev/dri/by-path --shm-size=16g \
    --privileged --ipc=host -p 8000:8000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    vllm/vllm-openai-xpu:latest \
    google/gemma-4-12B-it \
    --enforce-eager \
    --attention-backend flash_attn \
    --host 0.0.0.0 \
    --port 8000 \
    --trust-remote-code \
    --enable-auto-tool-choice \
    --reasoning-parser gemma4 \
    --tool-call-parser gemma4 \
    --chat-template examples/tool_chat_template_gemma4.jinja \
    --limit-mm-per-prompt '{"image": 4, "audio": 1}' \
    --async-scheduling \
    --gpu-memory-utilization 0.9 \
    --no-enable-prefix-caching \
    --max-num-batched-tokens 8192 \
    --max-model-len 5120 \
    --tensor-parallel-size 2
  ```

  ### Docker (Cloud TPU — Trillium / Ironwood)
  TPU uses the separate `vllm/vllm-tpu` image (no pip wheel). Pull the tag specified by the upstream [Trillium](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/trillium/vLLM/Gemma4) or [Ironwood](https://github.com/AI-Hypercomputer/tpu-recipes/blob/main/inference/ironwood/vLLM/Gemma4/) recipe, then run:
  ```bash
  docker run -itd --name gemma4-tpu \
    --privileged --network host --shm-size 16G \
    -v /dev/shm:/dev/shm -e HF_TOKEN=$HF_TOKEN \
    vllm/vllm-tpu:latest \
      --model google/gemma-4-12B-it \
      --max-model-len 16384 \
      --disable_chunked_mm_input \
      --host 0.0.0.0 --port 8000
  ```

  ## Client Usage

  ### Text Generation
  ```python
  from openai import OpenAI
  client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
  response = client.chat.completions.create(
      model="google/gemma-4-12B-it",
      messages=[{"role": "user", "content": "Write a poem about the ocean."}],
      max_tokens=512, temperature=0.7,
  )
  print(response.choices[0].message.content)
  ```

  ### Image Understanding
  ```python
  response = client.chat.completions.create(
      model="google/gemma-4-12B-it",
      messages=[{"role": "user", "content": [
          {"type": "image_url", "image_url": {"url": "https://upload.wikimedia.org/wikipedia/commons/thumb/3/3a/Cat03.jpg/1200px-Cat03.jpg"}},
          {"type": "text", "text": "Describe this image in detail."},
      ]}],
      max_tokens=1024,
  )
  ```

  ### Audio
  Requires `uv pip install "vllm[audio]"`.
  ```bash
  vllm serve google/gemma-4-12B-it \
    --max-model-len 8192 \
    --limit-mm-per-prompt '{"image": 4, "audio": 1}'
  ```

  ### Thinking Mode
  ```bash
  vllm serve google/gemma-4-12B-it \
    --max-model-len 16384 \
    --enable-auto-tool-choice \
    --reasoning-parser gemma4 \
    --tool-call-parser gemma4 \
    --chat-template examples/tool_chat_template_gemma4.jinja
  ```
  Enable thinking per-request via `extra_body={"chat_template_kwargs": {"enable_thinking": True}}`, or default-on with `--default-chat-template-kwargs '{"enable_thinking": true}'`.

  ### Structured Outputs
  vLLM guided decoding constrains output to a JSON schema. Include semantic instructions in the system prompt — the model does not see schema descriptions.

  ## Configuration Tips

  - Set `--max-model-len` to match your workload.
  - `--gpu-memory-utilization 0.90–0.95` maximizes KV cache.
  - Image-only workloads: pass `--limit-mm-per-prompt.audio 0`.
  - Text-only workloads: pass `--limit-mm-per-prompt '{"image": 0, "audio": 0}'` to skip MM profiling.
  - `--async-scheduling` improves throughput.
  - FP8 KV cache (`--kv-cache-dtype fp8`) saves ~50% KV memory.

  ## Speculative Decoding (MTP)

  Enable the **Spec Decoding** feature toggle (above) or add `--speculative-config` manually to use MTP drafting with the [assistant model](https://huggingface.co/google/gemma-4-12B-it-assistant). Recommended `num_speculative_tokens`: 4–8 for this model.

  > **Note:** MTP speculative decoding for Gemma 4 requires the nightly build — the pinned `vllm/vllm-openai:gemma4-unified` image or a nightly pip wheel. The standard `:latest` stable tag does not include this feature.

  ## References

  - [Model card](https://huggingface.co/google/gemma-4-12B-it)
  - [vLLM support PR (#44429)](https://github.com/vllm-project/vllm/pull/44429)
  - [Assistant (MTP draft) model](https://huggingface.co/google/gemma-4-12B-it-assistant)
  - [Gemma docs](https://ai.google.dev/gemma/docs)
  - [vLLM Gemma 4 tool-call template](https://github.com/vllm-project/vllm/blob/main/examples/tool_chat_template_gemma4.jinja)
  - [TPU recipes: Trillium](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/trillium/vLLM/Gemma4)
  - [TPU recipes: Ironwood](https://github.com/AI-Hypercomputer/tpu-recipes/blob/main/inference/ironwood/vLLM/Gemma4/)