[Doc] Host model recipes in recipes/ (#61302)

Signed-off-by: esmeetu <jasonailu87@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Roy Wang
2026-10-17 09:14:36 +00:00
committed by GitHub
co-authored by Claude Opus 5.5
parent f98ae6a9fd
commit b515648641
193 changed files with 44859 additions and 6 deletions
+105
View File
@@ -0,0 +1,105 @@
meta:
title: "Ling-2.6-1T"
slug: "ling-2.6-1t"
provider: "inclusionAI"
description: "Ling-2.6-1T (BailingMoeV2_5) FP8 instruct model with 1T total / 50B active params, hybrid linear + MLA attention, 262K context"
date_added: 2026-04-29
date_updated: 2026-05-13
tasks:
- text
related_recipes:
- inclusionAI/Ling-2.6-flash
- inclusionAI/Ring-1T-FP8
hardware:
b300: verified
mi300x: verified
model:
model_id: "inclusionAI/Ling-2.6-1T"
min_vllm_version: "0.20.2"
architecture: moe
parameter_count: "1T"
active_parameters: "50B"
context_length: 262144
base_args:
- "--trust-remote-code"
- "--tensor-parallel-size"
- "8"
base_env: {}
features: {}
opt_in_features: []
variants:
default:
precision: fp8
vram_minimum_gb: 1200
description: "FP8 weights with TP=8 on B300 or 8x MI300X/MI355X-class nodes"
compatible_strategies:
- single_node_tp
- multi_node_tp
- multi_node_tp_pp
- multi_node_tep
- multi_node_dep
hardware_overrides:
amd:
extra_env:
VLLM_ROCM_USE_AITER: "1"
strategy_overrides:
single_node_tp:
tp: 8
guide: |
## Overview
[Ling-2.6-1T](https://huggingface.co/inclusionAI/Ling-2.6-1T) is inclusionAI's
BailingMoeV2_5 FP8 flagship model with 1T total / 50B active parameters, hybrid
linear + MLA attention, and a 262K context window.
## Deployment Configurations
### Docker (AMD MI300X / MI325X / MI355X, TP=8)
TP=8 has been verified on an MI300X-class node at the model-derived 262K
context. MI325X and MI355X have larger per-GPU HBM.
```bash
docker run --rm -it \
--cap-add=SYS_PTRACE \
--ipc=host \
--privileged=true \
--shm-size=128GB \
--network=host \
--device=/dev/kfd \
--device=/dev/dri \
--group-add video \
-e VLLM_ROCM_USE_AITER=1 \
vllm/vllm-openai-rocm:v0.28.0 \
inclusionAI/Ling-2.6-1T \
--tensor-parallel-size 8 \
--trust-remote-code
```
## Client Usage
### Text Generation
```python
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
response = client.chat.completions.create(
model="inclusionAI/Ling-2.6-1T",
messages=[{"role": "user", "content": "Write a poem about the ocean."}],
max_tokens=512, temperature=0.7,
)
print(response.choices[0].message.content)
```
## References
- [Ling-2.6-1T on Hugging Face](https://huggingface.co/inclusionAI/Ling-2.6-1T)
- [vLLM ROCm GPU install docs](https://docs.vllm.ai/en/latest/getting_started/installation/gpu/?device=rocm)
- [ROCm vLLM optimization guide](https://rocm.docs.amd.com/en/latest/how-to/rocm-for-ai/inference-optimization/vllm-optimization.html)
@@ -0,0 +1,104 @@
meta:
title: "Ling-2.6-flash"
slug: "ling-2.6-flash"
provider: "inclusionAI"
description: "Ling-2.6-flash (BailingMoeV2_5) instruct model with 104B total / 7.4B active params, hybrid linear + MLA attention, 128K context, optimized for agent workloads"
date_added: 2026-04-28
date_updated: 2026-05-13
tasks:
- text
related_recipes:
- inclusionAI/Ring-1T-FP8
hardware:
h200: verified
mi300x: verified
model:
model_id: "inclusionAI/Ling-2.6-flash"
min_vllm_version: "0.20.2"
architecture: moe
parameter_count: "104B"
active_parameters: "7.4B"
context_length: 131072
base_args:
- "--trust-remote-code"
- "--tensor-parallel-size"
- "4"
base_env: {}
features: {}
opt_in_features: []
variants:
default:
precision: bf16
vram_minimum_gb: 250
description: "BF16 weights with base TP=4; the guide shows a tested 2-GPU AMD command"
compatible_strategies:
- single_node_tp
- multi_node_tp
- multi_node_tp_pp
- multi_node_tep
- multi_node_dep
hardware_overrides:
amd:
extra_env:
VLLM_ROCM_USE_AITER: "1"
strategy_overrides:
single_node_tp:
tp: 4
guide: |
## Overview
[Ling-2.6-flash](https://huggingface.co/inclusionAI/Ling-2.6-flash) is a
BailingMoeV2_5 MoE instruct model with 104B total / 7.4B active parameters,
hybrid linear + MLA attention, and a 131K context window.
## Deployment Configurations
### Docker (AMD MI300X / MI325X / MI355X, TP=2)
MI300X / MI325X / MI355X GPUs have larger per-GPU HBM, so TP=2 fits the full
131K context.
```bash
docker run --rm -it \
--cap-add=SYS_PTRACE \
--ipc=host \
--privileged=true \
--shm-size=128GB \
--network=host \
--device=/dev/kfd \
--device=/dev/dri \
--group-add video \
-e VLLM_ROCM_USE_AITER=1 \
vllm/vllm-openai-rocm:v0.28.0 \
inclusionAI/Ling-2.6-flash \
--tensor-parallel-size 2 \
--trust-remote-code
```
## Client Usage
### Text Generation
```python
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
response = client.chat.completions.create(
model="inclusionAI/Ling-2.6-flash",
messages=[{"role": "user", "content": "Write a poem about the ocean."}],
max_tokens=512, temperature=0.7,
)
print(response.choices[0].message.content)
```
## References
- [Ling-2.6-flash on Hugging Face](https://huggingface.co/inclusionAI/Ling-2.6-flash)
- [vLLM ROCm GPU install docs](https://docs.vllm.ai/en/latest/getting_started/installation/gpu/?device=rocm)
- [ROCm vLLM optimization guide](https://rocm.docs.amd.com/en/latest/how-to/rocm-for-ai/inference-optimization/vllm-optimization.html)
@@ -0,0 +1,167 @@
meta:
title: "Ling-3.0-flash-VL"
slug: "ling-3-0-flash-vl"
provider: "inclusionAI"
description: "Ling-3.0-flash vision-language MoE model with BF16, FP8, FP4, and INT4 checkpoints"
date_added: 2026-09-08
date_updated: 2026-09-08
tasks:
- text
- multimodal
related_recipes:
- "inclusionAI/Ling-3.0-flash"
model:
model_id: "inclusionAI/Ling-3.0-flash-VL"
min_vllm_version: "nightly"
nightly_required: true
architecture: moe
parameter_count: "124.85B"
active_parameters: "5.51B"
context_length: 131072
base_args:
- "--trust-remote-code"
- "--dtype"
- "bfloat16"
- "--gpu-memory-utilization"
- "0.9"
- "--enable-chunked-prefill"
- "--compilation-config"
- '{"cudagraph_mode":"FULL_AND_PIECEWISE"}'
- "--enable-prefix-caching"
features:
tool_calling:
description: "Enable Ling 3 automatic tool calling"
args:
- "--enable-auto-tool-choice"
- "--tool-call-parser"
- "ling3"
reasoning:
description: "Split Ling 3 thinking traces into reasoning"
args:
- "--reasoning-parser"
- "ling3"
variants:
default:
precision: bf16
vram_minimum_gb: 300
description: "BF16 checkpoint; defaults to TP4"
fp8:
model_id: "inclusionAI/Ling-3.0-flash-VL-fp8"
precision: fp8
vram_minimum_gb: 150
tp: 2
description: "Serialized block-FP8 checkpoint; defaults to TP2"
fp4:
model_id: "inclusionAI/Ling-3.0-flash-VL-fp4"
precision: fp4
# FP8 language projections and MXFP4 routed experts; BF16 vision/projector.
# The Hopper fallback does not require native Blackwell FP4 tensor cores.
precision_hardware:
brand: NVIDIA
vram_minimum_gb: 85
tp: 1
description: "Mixed block-FP8 and MXFP4 checkpoint; defaults to TP1"
int4:
model_id: "inclusionAI/Ling-3.0-flash-VL-int4"
precision: int4
vram_minimum_gb: 95
tp: 1
description: "INT4 checkpoint; defaults to TP1"
compatible_strategies:
- single_node_tp
hardware_overrides: {}
strategy_overrides:
single_node_tp:
tp: 4
guide: |
## Overview
`inclusionAI/Ling-3.0-flash-VL` uses the
`BailingMoeV3VLForConditionalGeneration` architecture. It combines the
Ling-3.0-flash hybrid MLA/KDA language backbone with a vision encoder and
projector for image understanding. The language backbone has 512 routed
experts (8 active per token) and one shared expert.
BF16, FP8, mixed block-FP8/MXFP4, and INT4 checkpoints are available. The
vision tower and projector remain BF16 in the mixed FP4 checkpoint.
## Prerequisites
- **vLLM:** an official nightly build containing Ling-3.0-flash-VL support
- **Precision:** BF16, FP8, FP4, or INT4 weights with BF16 compute
- **Context length:** 131,072 tokens
## Launching the Server
```bash
NCCL_DEBUG=WARN vllm serve inclusionAI/Ling-3.0-flash-VL \
--trust-remote-code \
--dtype bfloat16 \
--tensor-parallel-size 4 \
--gpu-memory-utilization 0.9 \
--enable-chunked-prefill \
--compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE"}' \
--enable-prefix-caching \
--enable-auto-tool-choice \
--tool-call-parser ling3 \
--reasoning-parser ling3
```
For a quantized variant, use the same flags with the corresponding model
ID and tensor parallel size:
| Variant | Model ID | TP |
|---|---|---:|
| FP8 | `inclusionAI/Ling-3.0-flash-VL-fp8` | 2 |
| FP4 | `inclusionAI/Ling-3.0-flash-VL-fp4` | 1 |
| INT4 | `inclusionAI/Ling-3.0-flash-VL-int4` | 1 |
## Image Requests
```python
import base64
from pathlib import Path
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
image = base64.b64encode(Path("image.png").read_bytes()).decode("ascii")
response = client.chat.completions.create(
model="inclusionAI/Ling-3.0-flash-VL",
messages=[{
"role": "user",
"content": [
{"type": "image_url", "image_url": {"url": f"data:image/png;base64,{image}"}},
{"type": "text", "text": "Describe the image."},
],
}],
temperature=0.0,
max_tokens=1024,
extra_body={"chat_template_kwargs": {"enable_thinking": False}},
)
print(response.choices[0].message.content)
```
For multiple images, add more `image_url` entries to `content`. For
text-only requests, use ordinary string content. When serving a quantized
variant, use its checkpoint ID in the client request.
## Thinking Mode
As with Ling-3.0-flash, thinking is selected per request through
`chat_template_kwargs`. Set `enable_thinking` to `True` to enable it;
with `--reasoning-parser ling3`, the parsed trace is in `message.reasoning`
and the final answer is in `message.content`.
## References
- [Ling-3.0-flash-VL](https://huggingface.co/inclusionAI/Ling-3.0-flash-VL)
- [Ling-3.0-flash-VL-fp8](https://huggingface.co/inclusionAI/Ling-3.0-flash-VL-fp8)
- [Ling-3.0-flash-VL-fp4](https://huggingface.co/inclusionAI/Ling-3.0-flash-VL-fp4)
- [Ling-3.0-flash-VL-int4](https://huggingface.co/inclusionAI/Ling-3.0-flash-VL-int4)
@@ -0,0 +1,348 @@
meta:
title: "Ling-3.0-flash"
slug: "ling-3-0-flash"
provider: "inclusionAI"
description: "Ling-3.0-flash MoE model with BF16, FP8, FP4, and INT4 checkpoints, native MTP, and an external DSpark draft model"
date_updated: 2026-09-09
tasks:
- text
performance_headline: "BF16, FP8, FP4, and INT4 support NVIDIA H20/H200 and AMD MI300X/MI355X; DGX Spark verified TP1 FP4/INT4"
hardware:
h200: verified
dgx_spark_gb10: verified
mi300x: verified
mi355x: verified
model:
model_id: "inclusionAI/Ling-3.0-flash"
min_vllm_version: "0.28.0"
architecture: moe
parameter_count: "124B"
active_parameters: "5.5B"
context_length: 262144
base_args:
- "--trust-remote-code"
- "--dtype"
- "bfloat16"
- "--gpu-memory-utilization"
- "0.9"
- "--enable-chunked-prefill"
- "--compilation-config"
- '{"cudagraph_mode":"FULL_AND_PIECEWISE"}'
- "--enable-prefix-caching"
features:
tool_calling:
description: "Enable Ling 3 automatic tool calling"
args:
- "--enable-auto-tool-choice"
- "--tool-call-parser"
- "ling3"
reasoning:
description: "Split Ling 3 thinking traces into reasoning"
args:
- "--reasoning-parser"
- "ling3"
spec_decoding:
description: "Speculative decoding with the built-in MTP head or the external Ling DSpark draft model"
default_mode: mtp
modes:
mtp:
label: "MTP"
description: "Use the target checkpoint's native MTP head with three speculative tokens."
args:
- "--speculative-config"
- '{"method":"mtp","num_speculative_tokens":3}'
hardware_overrides:
mi355x:
args:
- "--speculative-config"
- '{"method":"mtp","num_speculative_tokens":3}'
- "--attention-backend"
- "TRITON_MLA"
dspark:
label: "DSpark"
description: "Use inclusionAI/Ling-3.0-flash-dspark as an external draft model with any Ling-3.0-flash target variant. K may be 1-8; this recipe recommends K7."
args:
- "--speculative-config"
- '{"method":"dspark","model":"inclusionAI/Ling-3.0-flash-dspark","num_speculative_tokens":7,"draft_sample_method":"probabilistic","attention_backend":"FLASH_ATTN","enable_adaptive_verification":false}'
opt_in_features:
- spec_decoding
variants:
default:
precision: bf16
vram_minimum_gb: 300
description: "BF16 weights, the reference precision."
fp8:
model_id: "inclusionAI/Ling-3.0-flash-fp8"
precision: fp8
vram_minimum_gb: 150
tp: 2
description: "Block-FP8 (E4M3) weights, 128x128 block scaling."
fp4:
model_id: "inclusionAI/Ling-3.0-flash-fp4"
precision: fp4
precision_hardware: {}
vram_minimum_gb: 85
tp: 1
description: "Block-FP8 dense with MXFP4 routed experts. Emulated on gfx942 — prefer INT4 there."
int4:
model_id: "inclusionAI/Ling-3.0-flash-int4"
precision: int4
vram_minimum_gb: 95
tp: 1
description: "compressed-tensors W4 (group 32) on the routed experts."
compatible_strategies:
- single_node_tp
hardware_overrides:
amd:
extra_env:
VLLM_ROCM_USE_AITER: "1"
strategy_overrides:
single_node_tp:
tp: 4
guide: |
## Overview
`inclusionAI/Ling-3.0-flash` uses the `BailingMoeV3ForCausalLM` architecture
with a hybrid MLA/KDA attention stack, 512 routed experts (8 active per token),
one shared expert, and a native multi-token prediction head. The 42-layer base
model has 124.4B total and 5.5B active parameters. The checkpoint also contains
a 3.1B MTP layer, bringing the complete checkpoint to 127.5B parameters. A
serialized block-FP8 checkpoint is available as
`inclusionAI/Ling-3.0-flash-fp8`. Lower-memory checkpoints are also available
as `inclusionAI/Ling-3.0-flash-fp4` and
`inclusionAI/Ling-3.0-flash-int4`. The external
`inclusionAI/Ling-3.0-flash-dspark` draft can accelerate any target variant
without changing which target checkpoint is served.
## Prerequisites
- **vLLM:** 0.28.0 or newer, which includes native Bailing V3 support;
- **Validated hardware:** NVIDIA H20, H20-3e, H200, DGX Spark (FP4/INT4), and AMD MI300X / MI355X
- **Precision:** BF16, serialized block FP8, mixed block-FP8/MXFP4, or compressed-tensors INT4 weights with BF16 compute
- **Context length:** 262,144 tokens
## Launching the Server
```bash
NCCL_DEBUG=WARN vllm serve inclusionAI/Ling-3.0-flash \
--trust-remote-code \
--dtype bfloat16 \
--tensor-parallel-size 4 \
--gpu-memory-utilization 0.9 \
--enable-chunked-prefill \
--compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE"}' \
--enable-prefix-caching \
--enable-auto-tool-choice \
--tool-call-parser ling3 \
--reasoning-parser ling3 \
--speculative-config '{"method":"mtp","num_speculative_tokens":3}'
```
For the serialized FP8 variant on 2x H200, use:
```bash
vllm serve inclusionAI/Ling-3.0-flash-fp8 \
--trust-remote-code \
--dtype bfloat16 \
--tensor-parallel-size 2 \
--gpu-memory-utilization 0.9 \
--enable-chunked-prefill \
--compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE"}' \
--enable-prefix-caching \
--enable-auto-tool-choice \
--tool-call-parser ling3 \
--reasoning-parser ling3
```
For the validated TP4+EP4 alternative on 4x H200, use:
```bash
vllm serve inclusionAI/Ling-3.0-flash-fp8 \
--trust-remote-code \
--dtype bfloat16 \
--tensor-parallel-size 4 \
--enable-expert-parallel \
--gpu-memory-utilization 0.9 \
--enable-chunked-prefill \
--compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE"}' \
--enable-prefix-caching \
--enable-auto-tool-choice \
--tool-call-parser ling3 \
--reasoning-parser ling3
```
For the FP4 variant on one GPU, use:
```bash
vllm serve inclusionAI/Ling-3.0-flash-fp4 \
--trust-remote-code \
--dtype bfloat16 \
--tensor-parallel-size 1 \
--gpu-memory-utilization 0.9 \
--enable-chunked-prefill \
--compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE"}' \
--enable-prefix-caching \
--enable-auto-tool-choice \
--tool-call-parser ling3 \
--reasoning-parser ling3
```
For the INT4 variant on one GPU, use:
```bash
vllm serve inclusionAI/Ling-3.0-flash-int4 \
--trust-remote-code \
--dtype bfloat16 \
--tensor-parallel-size 1 \
--gpu-memory-utilization 0.9 \
--enable-chunked-prefill \
--compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE"}' \
--enable-prefix-caching \
--enable-auto-tool-choice \
--tool-call-parser ling3 \
--reasoning-parser ling3
```
## Speculative Decoding
The **Spec decoding** control exposes MTP and DSpark in the same place. They
are mutually exclusive modes, so the generated command contains exactly one
`--speculative-config`.
### MTP
Use the target checkpoint's native MTP head with three draft tokens:
```bash
--speculative-config '{"method":"mtp","num_speculative_tokens":3}'
```
### DSpark
`inclusionAI/Ling-3.0-flash-dspark` is an external draft model. It does not
replace the served target checkpoint. It can be paired with any
Ling-3.0-flash target variant; pass the DSpark repository through
`--speculative-config.model` while keeping the selected target as the model
served by `vllm serve`.
The checkpoint was trained with query block size `Q=8`. Runtime
`num_speculative_tokens` (`K`) may be any integer from 1 through 8. The recipe
emits `K=7` as the recommended throughput/KV-capacity trade-off; change the
value to select another supported width. `K=8` remains valid.
```bash
--speculative-config '{"method":"dspark","model":"inclusionAI/Ling-3.0-flash-dspark","num_speculative_tokens":7,"draft_sample_method":"probabilistic","attention_backend":"FLASH_ATTN","enable_adaptive_verification":false}'
```
`enable_adaptive_verification` is disabled because the current Ling hybrid
MLA/KDA path is validated with fixed-width DSpark verification. The
`attention_backend` setting applies only to the external draft model; the
target model keeps its automatically selected attention backends.
## Single-GPU INT4
`inclusionAI/Ling-3.0-flash-int4` is a `compressed-tensors` / `pack-quantized`
checkpoint: symmetric W4, `group_size: 32`, applied to the **routed experts
only** — attention, `lm_head`, the shared expert and the dense projections are
all in the config's `ignore` list. It is not covered by the model-specific
quantization plumbing in `bailing_moe_v3.py` (which handles block FP8 and
MXFP4); it loads through vLLM's generic `compressed-tensors` path and runs at
TP1 on H200 or DGX Spark.
The engine names the mechanism at startup:
```
quantization=compressed-tensors ... trust_remote_code=True, dtype=torch.bfloat16
INFO [int_wna16.py:297] Using 'MARLIN' WNA16 MoE backend.
INFO [cuda.py:492] Using FLASH_ATTN_MLA attention backend
INFO [int_wna16.py:409] Using MarlinExperts
```
The vendor model card documents SGLang only, and there is no merged vLLM PR
specific to Ling INT4. This variant works through the generic quantization
path; tool calling, reasoning parsing, chunked prefill and CUDA graph capture
behave as they do on the other variants.
## Running on MI300X/MI355X
```bash
export VLLM_ROCM_USE_AITER=1
vllm serve inclusionAI/Ling-3.0-flash \
--trust-remote-code \
--dtype bfloat16 \
--tensor-parallel-size 4 \
--gpu-memory-utilization 0.9 \
--enable-chunked-prefill \
--compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE"}' \
--enable-prefix-caching \
--enable-auto-tool-choice \
--tool-call-parser ling3 \
--reasoning-parser ling3
```
On MI300X the BF16 run served the full 262,144-token context with a 105.15 GiB
KV pool and 52.38x max concurrency.
**MTP** needs a backend pin on gfx950: `ROCM_AITER_MLA` lets the speculator
capture a graph the backend cannot serve and the workers abort with a memory
access fault, so the mode pins `--attention-backend TRITON_MLA` on `mi355x`.
gfx942 is unaffected and keeps AITER MLA.
## Validation
The BF16, FP8, FP4, and INT4 checkpoints support NVIDIA H20 and H200. For FP8,
TP2 remains the recommended multi-GPU default; TP4+EP4 is a validated H200
alternative for deployments that prefer expert parallelism. DGX Spark is
limited to the TP1 FP4 and INT4 variants because the BF16 and FP8 checkpoints
are too large for its 128 GB unified memory. DSpark is external and works with
any target variant that fits the selected hardware.
The `int4` variant was verified on **1x H200 (SM90), TP=1**, on a `main` build
reporting `0.26.1rc1.dev1133+gf94666b60`, serving at `--max-model-len 32768`:
77.0 GB of weights on disk, 70.27 GiB resident after load (165 s), 5 min 56 s
from container start to ready, `FULL_AND_PIECEWISE` capture (68 PIECEWISE + 68
FULL, 2.15 GiB), and a 4,072,594-token KV pool. Single-stream decode measured
**159.5 tok/s** at TTFT p50 0.17 s; 973 output tok/s aggregate at concurrency
16, with 0 failed requests and 0 preemptions over a 1..24 concurrency sweep.
MI355X was verified on MI350X silicon, the same gfx950 class.
## Thinking Mode
Thinking is selected per request through the chat template rather than by a
server flag:
```python
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
response = client.chat.completions.create(
model="inclusionAI/Ling-3.0-flash",
messages=[{"role": "user", "content": "Solve the problem step by step."}],
temperature=0.0,
max_tokens=200000,
extra_body={"chat_template_kwargs": {"enable_thinking": True}},
)
msg = response.choices[0].message
print(msg.reasoning)
print(msg.content)
```
When serving a quantized variant, set `model` in the client request to the
corresponding FP8, FP4, or INT4 checkpoint ID.
## References
- [Ling-3.0-flash on Hugging Face](https://huggingface.co/inclusionAI/Ling-3.0-flash)
- [Ling-3.0-flash-fp8 on Hugging Face](https://huggingface.co/inclusionAI/Ling-3.0-flash-fp8)
- [Ling-3.0-flash-dspark on Hugging Face](https://huggingface.co/inclusionAI/Ling-3.0-flash-dspark)
- [Ling-3.0-flash-fp4 on Hugging Face](https://huggingface.co/inclusionAI/Ling-3.0-flash-fp4)
- [Ling-3.0-flash-int4 on Hugging Face](https://huggingface.co/inclusionAI/Ling-3.0-flash-int4)
@@ -0,0 +1,164 @@
meta:
title: "Ling-3.0-tiny"
slug: "ling-3-0-tiny"
provider: "inclusionAI"
description: "Ling-3.0-tiny lightweight hybrid-reasoning MoE with BF16, block-FP8, and compressed-tensors INT4 checkpoints"
date_added: 2026-08-31
date_updated: 2026-09-09
tasks:
- text
performance_headline: "7.9B total / 1.3B active parameters with a 3:1 KDA-MLA stack; all three official checkpoints run with TP1"
related_recipes:
- inclusionAI/Ling-3.0-flash
hardware:
h200: verified
model:
model_id: "inclusionAI/Ling-3.0-tiny"
min_vllm_version: "0.28.0"
architecture: moe
parameter_count: "7.9B"
active_parameters: "1.3B"
context_length: 131072
base_args:
- "--trust-remote-code"
- "--dtype"
- "bfloat16"
- "--gpu-memory-utilization"
- "0.85"
- "--enable-prefix-caching"
features:
tool_calling:
description: "Enable Ling 3 automatic tool calling"
args:
- "--enable-auto-tool-choice"
- "--tool-call-parser"
- "ling3"
reasoning:
description: "Split Ling 3 thinking traces into reasoning"
args:
- "--reasoning-parser"
- "ling3"
opt_in_features: []
variants:
default:
precision: bf16
vram_minimum_gb: 18
tp: 1
description: "Official BF16 checkpoint; approximately 15.8 GB of repository weights and served with TP1"
fp8:
model_id: "inclusionAI/Ling-3.0-tiny-fp8"
precision: fp8
vram_minimum_gb: 11
tp: 1
description: "Official serialized block-FP8 checkpoint with dynamic activations and 128x128 weight blocks; served with TP1"
int4:
model_id: "inclusionAI/Ling-3.0-tiny-int4"
precision: int4
vram_minimum_gb: 8
tp: 1
description: "Official compressed-tensors group-wise INT4 checkpoint; served with TP1"
compatible_strategies:
- single_node_tp
hardware_overrides: {}
strategy_overrides:
single_node_tp:
tp: 1
guide: |
## Overview
`inclusionAI/Ling-3.0-tiny` is the lightweight member of the Ling 3.0
family. It has 7.9B total parameters and activates 1.3B parameters per token.
Its 24-layer hybrid stack repeats three Kimi Delta Attention (KDA) layers and
one Multi-Head Latent Attention (MLA) layer, while each sparse MoE block
selects 8 of 128 routed experts plus one shared expert.
The official release provides three served checkpoints:
- `inclusionAI/Ling-3.0-tiny`: BF16;
- `inclusionAI/Ling-3.0-tiny-fp8`: serialized block FP8;
- `inclusionAI/Ling-3.0-tiny-int4`: compressed-tensors INT4.
All three use the same `BailingMoeV3ForCausalLM` architecture and a native
131,072-token context window. Select the corresponding Variant above; the
served model ID changes automatically.
## Prerequisites
- **vLLM:** 0.28.0 or newer, which includes Ling 3.0 / Bailing V3
hybrid-attention support;
- **Parallelism:** TP1 is recommended for all three checkpoints;
- **Thinking mode:** enabled per request by the chat template.
## Launching the Server
BF16 on one GPU:
```bash
vllm serve inclusionAI/Ling-3.0-tiny \
--trust-remote-code \
--dtype bfloat16 \
--tensor-parallel-size 1 \
--gpu-memory-utilization 0.85 \
--enable-prefix-caching \
--enable-auto-tool-choice \
--tool-call-parser ling3 \
--reasoning-parser ling3
```
For FP8 or INT4, keep the same arguments and replace the served checkpoint:
```bash
# Serialized block FP8
vllm serve inclusionAI/Ling-3.0-tiny-fp8 \
--trust-remote-code --dtype bfloat16 --tensor-parallel-size 1 \
--gpu-memory-utilization 0.85 --enable-prefix-caching --enable-auto-tool-choice \
--tool-call-parser ling3 --reasoning-parser ling3
# Compressed-tensors INT4
vllm serve inclusionAI/Ling-3.0-tiny-int4 \
--trust-remote-code --dtype bfloat16 --tensor-parallel-size 1 \
--gpu-memory-utilization 0.85 --enable-prefix-caching --enable-auto-tool-choice \
--tool-call-parser ling3 --reasoning-parser ling3
```
## Thinking Mode
Thinking is controlled per request. The model card recommends
`temperature=1.0`, `top_p=0.95`, and `top_k=20` when thinking is enabled:
```python
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
response = client.chat.completions.create(
model="inclusionAI/Ling-3.0-tiny",
messages=[{"role": "user", "content": "Solve the problem step by step."}],
temperature=1.0,
top_p=0.95,
extra_body={
"top_k": 20,
"chat_template_kwargs": {"enable_thinking": True},
},
)
msg = response.choices[0].message
print(msg.reasoning)
print(msg.content)
```
Set `enable_thinking` to `false` for direct answers. When using a quantized
variant, set the client `model` to its FP8 or INT4 checkpoint ID.
## References
- [Ling 3.0 collection](https://huggingface.co/collections/inclusionAI/ling-30)
- [Ling-3.0-tiny on Hugging Face](https://huggingface.co/inclusionAI/Ling-3.0-tiny)
- [Ling-3.0-tiny-fp8 on Hugging Face](https://huggingface.co/inclusionAI/Ling-3.0-tiny-fp8)
- [Ling-3.0-tiny-int4 on Hugging Face](https://huggingface.co/inclusionAI/Ling-3.0-tiny-int4)
@@ -0,0 +1,170 @@
meta:
title: "Ling-3.1-flash"
slug: "ling-3-1-flash"
provider: "inclusionAI"
description: "Ling-3.1-flash MoE model with block-FP8 weights, tool calling, and reasoning support"
date_added: 2026-10-14
date_updated: 2026-10-15
tasks:
- text
performance_headline: "Block-FP8 weights with BF16 compute and TP8 + EP8 deployment"
related_recipes:
- inclusionAI/Ling-3.0-flash
hardware:
mi355x: verified
model:
model_id: "inclusionAI/Ling-3.1-flash"
min_vllm_version: "nightly"
nightly_required: true
docker_image:
nvidia: "vllm/vllm-openai:nightly"
amd: "vllm/vllm-openai-rocm:nightly"
architecture: moe
parameter_count: "560B"
active_parameters: "25B"
context_length: 262144
base_args:
- "--trust-remote-code"
- "--enable-expert-parallel"
- "--gpu-memory-utilization"
- "0.9"
- "--enable-chunked-prefill"
- "--compilation-config"
- '{"cudagraph_mode":"FULL_AND_PIECEWISE"}'
- "--enable-prefix-caching"
features:
tool_calling:
description: "Enable Ling 3 automatic tool calling"
args:
- "--enable-auto-tool-choice"
- "--tool-call-parser"
- "ling3"
reasoning:
description: "Split Ling 3 thinking traces into reasoning"
args:
- "--reasoning-parser"
- "ling3"
spec_decoding:
description: "Speculative decoding with the checkpoint's native MTP head and three speculative tokens"
args:
- "--speculative-config"
- '{"method":"mtp","num_speculative_tokens":3}'
opt_in_features:
- spec_decoding
variants:
default:
precision: fp8
# Estimated weight footprint plus runtime allowance; KV capacity varies.
vram_minimum_gb: 689
tp: 8
description: "Block-FP8 (E4M3) weights, 128x128 block scaling, with BF16 compute."
compatible_strategies:
- single_node_tep
default_strategy: single_node_tep
hardware_overrides:
amd:
extra_env:
VLLM_ROCM_USE_AITER: "1"
guide: |
## Overview
`inclusionAI/Ling-3.1-flash` uses the `BailingMoeV3ForCausalLM`
architecture with a hybrid MLA/KDA attention stack, 72 layers, 512 routed
experts (8 active per token), and one shared expert. The checkpoint uses
serialized block-FP8 E4M3 weights with 128x128 block scaling and dynamic
activation quantization. Non-quantized modules use BF16.
## Prerequisites
- **vLLM:** nightly build (`vllm/vllm-openai-rocm:nightly` on AMD)
- **Deployment:** a single node with 8 GPUs, using TP8 + EP8
- **Precision:** serialized block FP8 weights with BF16 compute
- **Context length:** 262,144 tokens
- **Memory:** approximately 689 GB aggregate VRAM as a loading estimate;
allow additional capacity for KV cache and the intended workload
## Launching the Server
```bash
vllm serve inclusionAI/Ling-3.1-flash \
--trust-remote-code \
--tensor-parallel-size 8 \
--enable-expert-parallel \
--gpu-memory-utilization 0.9 \
--enable-chunked-prefill \
--compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE"}' \
--enable-prefix-caching \
--enable-auto-tool-choice \
--tool-call-parser ling3 \
--reasoning-parser ling3
```
### AMD MI355X
Use the ROCm nightly image `vllm/vllm-openai-rocm:nightly`,
enable AITER and launch using the command:
```bash
VLLM_ROCM_USE_AITER=1 vllm serve inclusionAI/Ling-3.1-flash \
--trust-remote-code \
--tensor-parallel-size 8 \
--enable-expert-parallel \
--gpu-memory-utilization 0.9 \
--enable-chunked-prefill \
--compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE"}' \
--enable-prefix-caching \
--enable-auto-tool-choice \
--tool-call-parser ling3 \
--reasoning-parser ling3
```
Each GPU has about 168 GiB of KV cache available.
## Speculative Decoding (MTP)
The checkpoint ships a native MTP head. Enable it with three speculative
tokens:
```bash
--speculative-config '{"method":"mtp","num_speculative_tokens":3}'
```
On MI355X this gave a mean acceptance length of 3.0 (per-position acceptance
0.96 / 0.67 / 0.38) on short reasoning prompts.
## Thinking Mode
Thinking is selected per request through the chat template rather than by a
server flag:
```python
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
response = client.chat.completions.create(
model="inclusionAI/Ling-3.1-flash",
messages=[{"role": "user", "content": "Solve the problem step by step."}],
temperature=0.0,
max_tokens=16384,
extra_body={"chat_template_kwargs": {"enable_thinking": True}},
)
msg = response.choices[0].message
print(msg.reasoning)
print(msg.content)
```
`--reasoning-parser ling3` separates thinking traces from the final answer.
Keep the prompt and requested output within the configured context limit.
## References
- [Ling-3.1-flash on Hugging Face](https://huggingface.co/inclusionAI/Ling-3.1-flash)
- [vLLM nightly installation](https://docs.vllm.ai/en/latest/getting_started/installation/gpu/#install-the-latest-code)
+133
View File
@@ -0,0 +1,133 @@
meta:
title: "Ring-1T-FP8"
slug: "ring-1t-fp8"
provider: "inclusionAI"
description: "Ring-1T (BailingMoeV2) FP8 model (~1T total params) for 8xH200 or 8xMI300X deployment"
date_added: 2025-10-15
date_updated: 2026-04-17
tasks:
- text
related_recipes: []
hardware:
h200: verified
mi300x: verified
mi325x: verified
mi355x: verified
model:
model_id: "inclusionAI/Ring-1T-FP8"
min_vllm_version: "0.11.0"
architecture: moe
parameter_count: "1T"
active_parameters: "50B"
context_length: 65536
base_args:
- "--trust-remote-code"
- "--tensor-parallel-size"
- "8"
- "--max-num-seqs"
- "32"
- "--kv-cache-dtype"
- "fp8"
- "--served-model-name"
- "Ring-1T-FP8"
base_env: {}
features: {}
opt_in_features: []
variants:
default:
precision: fp8
vram_minimum_gb: 1200
description: "FP8 weights on 8x H200 (80 GB) with FP8 KV cache"
extra_args:
- "--gpu-memory-utilization"
- "0.97"
- "--compilation-config"
- '{"use_inductor": false}'
compatible_strategies:
- single_node_tp
- multi_node_tp
- multi_node_tp_pp
- multi_node_tep
- multi_node_dep
- pd_cluster
hardware_overrides: {}
strategy_overrides: {}
guide: |
## Overview
Ring-1T-FP8 is inclusionAI's BailingMoeV2 FP8 model (~1T total parameters). This recipe
covers pure tensor-parallel deployment across 8 GPUs on NVIDIA H200 or AMD MI300X+.
## Prerequisites
- Hardware: 8x H200 or 8x MI300X/MI325X/MI355X
- vLLM >= 0.11.0
### Install vLLM (CUDA)
```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --torch-backend auto
```
### Install vLLM (AMD ROCm)
```bash
uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/0.14.1/rocm700
```
## Launch commands
8x H200 (FP8 KV cache):
```bash
vllm serve inclusionAI/Ring-1T-FP8 \
--trust-remote-code \
--tensor-parallel-size 8 \
--gpu-memory-utilization 0.97 \
--max-num-seqs 32 \
--kv-cache-dtype fp8 \
--compilation-config '{"use_inductor": false}' \
--served-model-name Ring-1T-FP8
```
8x MI300X/MI325X/MI355X:
```bash
export VLLM_ROCM_USE_AITER=1
vllm serve inclusionAI/Ring-1T-FP8 \
--trust-remote-code \
--tensor-parallel-size 8 \
--gpu-memory-utilization 0.9 \
--max-num-seqs 32 \
--kv-cache-dtype fp8 \
--served-model-name Ring-1T-FP8
```
Tuning flags:
- `--max-model-len=65536` works well for most scenarios.
- `--max-num-batched-tokens=32768` for prompt-heavy; 16384/8192 for lower latency.
- Reduce `--gpu-memory-utilization` below 0.97 if you hit OOM.
## Client Usage
```bash
curl http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "Ring-1T-FP8",
"messages": [{"role": "user", "content": "9.11 and 9.8, which is greater?"}]
}'
```
## References
- [Ring-1T-FP8 on Hugging Face](https://huggingface.co/inclusionAI/Ring-1T-FP8)
+127
View File
@@ -0,0 +1,127 @@
meta:
title: "Ring-2.6-1T"
slug: "ring-2.6-1t"
provider: "inclusionAI"
description: "Ring-2.6-1T (BailingMoeV2_5) FP8 thinking model with 1T total / 50B active params, hybrid linear + MLA attention, 128K context"
date_added: 2026-05-14
date_updated: 2026-05-15
tasks:
- text
related_recipes:
- inclusionAI/Ling-2.6-1T
- inclusionAI/Ring-1T-FP8
model:
model_id: "inclusionAI/Ring-2.6-1T"
min_vllm_version: "0.20.2"
architecture: moe
parameter_count: "1T"
active_parameters: "50B"
context_length: 131072
base_args:
- "--trust-remote-code"
- "--tensor-parallel-size"
- "8"
base_env: {}
features: {}
opt_in_features: []
variants:
default:
precision: fp8
vram_minimum_gb: 1200
description: "FP8 weights with TP=8 on B300 or 8x MI300X/MI355X-class nodes"
compatible_strategies:
- single_node_tp
- multi_node_tp
- multi_node_tp_pp
- multi_node_tep
- multi_node_dep
- pd_cluster
hardware_overrides:
amd:
extra_env:
VLLM_ROCM_USE_AITER: "1"
strategy_overrides:
single_node_tp:
tp: 8
guide: |
## Overview
[Ring-2.6-1T](https://huggingface.co/inclusionAI/Ring-2.6-1T) is inclusionAI's
BailingMoeV2_5 FP8 trillion-scale **thinking** model (1T total / 50B active
parameters) with hybrid linear + MLA attention and a 128K context window.
It is the reasoning-focused counterpart to [Ling-2.6-1T](https://huggingface.co/inclusionAI/Ling-2.6-1T)
in the Ring 2.6 series and emits explicit `<think>...</think>` traces; the
chat template accepts a `reasoning_effort` field (default `high`).
vLLM 0.20.1 shipped the BailingMoeV2.5 MLA RoPE fix (vllm-project/vllm#41185)
that is load-bearing for this architecture — pin v0.20.2 or newer.
## Deployment Configurations
### Docker (AMD MI300X / MI325X / MI355X, TP=8)
TP=8 fits the model-derived 128K context on an MI300X-class node; MI325X
and MI355X have larger per-GPU HBM and more headroom.
```bash
docker run --rm -it \
--cap-add=SYS_PTRACE \
--ipc=host \
--privileged=true \
--shm-size=128GB \
--network=host \
--device=/dev/kfd \
--device=/dev/dri \
--group-add video \
-e VLLM_ROCM_USE_AITER=1 \
vllm/vllm-openai-rocm:v0.28.0 \
inclusionAI/Ring-2.6-1T \
--tensor-parallel-size 8 \
--trust-remote-code
```
### Pip (NVIDIA B300, TP=8)
```bash
uv venv && source .venv/bin/activate
uv pip install -U vllm --torch-backend auto
vllm serve inclusionAI/Ring-2.6-1T \
--trust-remote-code \
--tensor-parallel-size 8
```
## Client Usage
### Text Generation
```python
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
response = client.chat.completions.create(
model="inclusionAI/Ring-2.6-1T",
messages=[{"role": "user", "content": "Prove there are infinitely many primes."}],
max_tokens=4096,
temperature=0.6,
extra_body={"chat_template_kwargs": {"reasoning_effort": "high"}},
)
print(response.choices[0].message.content)
```
Set `reasoning_effort` to `low` / `medium` / `high` to control the depth of
thinking. The model wraps its reasoning trace in `<think>...</think>` before
emitting the final answer.
## References
- [Ring-2.6-1T on Hugging Face](https://huggingface.co/inclusionAI/Ring-2.6-1T)
- [vLLM PR #41185 — BailingMoeV2.5 MLA RoPE fix](https://github.com/vllm-project/vllm/pull/41185)
- [vLLM ROCm GPU install docs](https://docs.vllm.ai/en/latest/getting_started/installation/gpu/?device=rocm)