[Doc] Host model recipes in recipes/ (#61302)
Signed-off-by: esmeetu <jasonailu87@gmail.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 5.5
parent
f98ae6a9fd
commit
b515648641
@@ -0,0 +1,105 @@
|
||||
meta:
|
||||
title: "Ling-2.6-1T"
|
||||
slug: "ling-2.6-1t"
|
||||
provider: "inclusionAI"
|
||||
description: "Ling-2.6-1T (BailingMoeV2_5) FP8 instruct model with 1T total / 50B active params, hybrid linear + MLA attention, 262K context"
|
||||
date_added: 2026-04-29
|
||||
date_updated: 2026-05-13
|
||||
tasks:
|
||||
- text
|
||||
related_recipes:
|
||||
- inclusionAI/Ling-2.6-flash
|
||||
- inclusionAI/Ring-1T-FP8
|
||||
hardware:
|
||||
b300: verified
|
||||
mi300x: verified
|
||||
|
||||
model:
|
||||
model_id: "inclusionAI/Ling-2.6-1T"
|
||||
min_vllm_version: "0.20.2"
|
||||
architecture: moe
|
||||
parameter_count: "1T"
|
||||
active_parameters: "50B"
|
||||
context_length: 262144
|
||||
base_args:
|
||||
- "--trust-remote-code"
|
||||
- "--tensor-parallel-size"
|
||||
- "8"
|
||||
base_env: {}
|
||||
|
||||
features: {}
|
||||
|
||||
opt_in_features: []
|
||||
|
||||
variants:
|
||||
default:
|
||||
precision: fp8
|
||||
vram_minimum_gb: 1200
|
||||
description: "FP8 weights with TP=8 on B300 or 8x MI300X/MI355X-class nodes"
|
||||
|
||||
compatible_strategies:
|
||||
- single_node_tp
|
||||
- multi_node_tp
|
||||
- multi_node_tp_pp
|
||||
- multi_node_tep
|
||||
- multi_node_dep
|
||||
|
||||
hardware_overrides:
|
||||
amd:
|
||||
extra_env:
|
||||
VLLM_ROCM_USE_AITER: "1"
|
||||
|
||||
strategy_overrides:
|
||||
single_node_tp:
|
||||
tp: 8
|
||||
|
||||
guide: |
|
||||
## Overview
|
||||
|
||||
[Ling-2.6-1T](https://huggingface.co/inclusionAI/Ling-2.6-1T) is inclusionAI's
|
||||
BailingMoeV2_5 FP8 flagship model with 1T total / 50B active parameters, hybrid
|
||||
linear + MLA attention, and a 262K context window.
|
||||
|
||||
## Deployment Configurations
|
||||
|
||||
### Docker (AMD MI300X / MI325X / MI355X, TP=8)
|
||||
|
||||
TP=8 has been verified on an MI300X-class node at the model-derived 262K
|
||||
context. MI325X and MI355X have larger per-GPU HBM.
|
||||
|
||||
```bash
|
||||
docker run --rm -it \
|
||||
--cap-add=SYS_PTRACE \
|
||||
--ipc=host \
|
||||
--privileged=true \
|
||||
--shm-size=128GB \
|
||||
--network=host \
|
||||
--device=/dev/kfd \
|
||||
--device=/dev/dri \
|
||||
--group-add video \
|
||||
-e VLLM_ROCM_USE_AITER=1 \
|
||||
vllm/vllm-openai-rocm:v0.28.0 \
|
||||
inclusionAI/Ling-2.6-1T \
|
||||
--tensor-parallel-size 8 \
|
||||
--trust-remote-code
|
||||
```
|
||||
|
||||
## Client Usage
|
||||
|
||||
### Text Generation
|
||||
```python
|
||||
from openai import OpenAI
|
||||
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
|
||||
response = client.chat.completions.create(
|
||||
model="inclusionAI/Ling-2.6-1T",
|
||||
messages=[{"role": "user", "content": "Write a poem about the ocean."}],
|
||||
max_tokens=512, temperature=0.7,
|
||||
)
|
||||
print(response.choices[0].message.content)
|
||||
```
|
||||
|
||||
## References
|
||||
|
||||
- [Ling-2.6-1T on Hugging Face](https://huggingface.co/inclusionAI/Ling-2.6-1T)
|
||||
- [vLLM ROCm GPU install docs](https://docs.vllm.ai/en/latest/getting_started/installation/gpu/?device=rocm)
|
||||
- [ROCm vLLM optimization guide](https://rocm.docs.amd.com/en/latest/how-to/rocm-for-ai/inference-optimization/vllm-optimization.html)
|
||||
@@ -0,0 +1,104 @@
|
||||
meta:
|
||||
title: "Ling-2.6-flash"
|
||||
slug: "ling-2.6-flash"
|
||||
provider: "inclusionAI"
|
||||
description: "Ling-2.6-flash (BailingMoeV2_5) instruct model with 104B total / 7.4B active params, hybrid linear + MLA attention, 128K context, optimized for agent workloads"
|
||||
date_added: 2026-04-28
|
||||
date_updated: 2026-05-13
|
||||
tasks:
|
||||
- text
|
||||
related_recipes:
|
||||
- inclusionAI/Ring-1T-FP8
|
||||
hardware:
|
||||
h200: verified
|
||||
mi300x: verified
|
||||
|
||||
model:
|
||||
model_id: "inclusionAI/Ling-2.6-flash"
|
||||
min_vllm_version: "0.20.2"
|
||||
architecture: moe
|
||||
parameter_count: "104B"
|
||||
active_parameters: "7.4B"
|
||||
context_length: 131072
|
||||
base_args:
|
||||
- "--trust-remote-code"
|
||||
- "--tensor-parallel-size"
|
||||
- "4"
|
||||
base_env: {}
|
||||
|
||||
features: {}
|
||||
|
||||
opt_in_features: []
|
||||
|
||||
variants:
|
||||
default:
|
||||
precision: bf16
|
||||
vram_minimum_gb: 250
|
||||
description: "BF16 weights with base TP=4; the guide shows a tested 2-GPU AMD command"
|
||||
|
||||
compatible_strategies:
|
||||
- single_node_tp
|
||||
- multi_node_tp
|
||||
- multi_node_tp_pp
|
||||
- multi_node_tep
|
||||
- multi_node_dep
|
||||
|
||||
hardware_overrides:
|
||||
amd:
|
||||
extra_env:
|
||||
VLLM_ROCM_USE_AITER: "1"
|
||||
|
||||
strategy_overrides:
|
||||
single_node_tp:
|
||||
tp: 4
|
||||
|
||||
guide: |
|
||||
## Overview
|
||||
|
||||
[Ling-2.6-flash](https://huggingface.co/inclusionAI/Ling-2.6-flash) is a
|
||||
BailingMoeV2_5 MoE instruct model with 104B total / 7.4B active parameters,
|
||||
hybrid linear + MLA attention, and a 131K context window.
|
||||
|
||||
## Deployment Configurations
|
||||
|
||||
### Docker (AMD MI300X / MI325X / MI355X, TP=2)
|
||||
|
||||
MI300X / MI325X / MI355X GPUs have larger per-GPU HBM, so TP=2 fits the full
|
||||
131K context.
|
||||
|
||||
```bash
|
||||
docker run --rm -it \
|
||||
--cap-add=SYS_PTRACE \
|
||||
--ipc=host \
|
||||
--privileged=true \
|
||||
--shm-size=128GB \
|
||||
--network=host \
|
||||
--device=/dev/kfd \
|
||||
--device=/dev/dri \
|
||||
--group-add video \
|
||||
-e VLLM_ROCM_USE_AITER=1 \
|
||||
vllm/vllm-openai-rocm:v0.28.0 \
|
||||
inclusionAI/Ling-2.6-flash \
|
||||
--tensor-parallel-size 2 \
|
||||
--trust-remote-code
|
||||
```
|
||||
|
||||
## Client Usage
|
||||
|
||||
### Text Generation
|
||||
```python
|
||||
from openai import OpenAI
|
||||
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
|
||||
response = client.chat.completions.create(
|
||||
model="inclusionAI/Ling-2.6-flash",
|
||||
messages=[{"role": "user", "content": "Write a poem about the ocean."}],
|
||||
max_tokens=512, temperature=0.7,
|
||||
)
|
||||
print(response.choices[0].message.content)
|
||||
```
|
||||
|
||||
## References
|
||||
|
||||
- [Ling-2.6-flash on Hugging Face](https://huggingface.co/inclusionAI/Ling-2.6-flash)
|
||||
- [vLLM ROCm GPU install docs](https://docs.vllm.ai/en/latest/getting_started/installation/gpu/?device=rocm)
|
||||
- [ROCm vLLM optimization guide](https://rocm.docs.amd.com/en/latest/how-to/rocm-for-ai/inference-optimization/vllm-optimization.html)
|
||||
@@ -0,0 +1,167 @@
|
||||
meta:
|
||||
title: "Ling-3.0-flash-VL"
|
||||
slug: "ling-3-0-flash-vl"
|
||||
provider: "inclusionAI"
|
||||
description: "Ling-3.0-flash vision-language MoE model with BF16, FP8, FP4, and INT4 checkpoints"
|
||||
date_added: 2026-09-08
|
||||
date_updated: 2026-09-08
|
||||
tasks:
|
||||
- text
|
||||
- multimodal
|
||||
related_recipes:
|
||||
- "inclusionAI/Ling-3.0-flash"
|
||||
|
||||
model:
|
||||
model_id: "inclusionAI/Ling-3.0-flash-VL"
|
||||
min_vllm_version: "nightly"
|
||||
nightly_required: true
|
||||
architecture: moe
|
||||
parameter_count: "124.85B"
|
||||
active_parameters: "5.51B"
|
||||
context_length: 131072
|
||||
base_args:
|
||||
- "--trust-remote-code"
|
||||
- "--dtype"
|
||||
- "bfloat16"
|
||||
- "--gpu-memory-utilization"
|
||||
- "0.9"
|
||||
- "--enable-chunked-prefill"
|
||||
- "--compilation-config"
|
||||
- '{"cudagraph_mode":"FULL_AND_PIECEWISE"}'
|
||||
- "--enable-prefix-caching"
|
||||
|
||||
features:
|
||||
tool_calling:
|
||||
description: "Enable Ling 3 automatic tool calling"
|
||||
args:
|
||||
- "--enable-auto-tool-choice"
|
||||
- "--tool-call-parser"
|
||||
- "ling3"
|
||||
reasoning:
|
||||
description: "Split Ling 3 thinking traces into reasoning"
|
||||
args:
|
||||
- "--reasoning-parser"
|
||||
- "ling3"
|
||||
|
||||
variants:
|
||||
default:
|
||||
precision: bf16
|
||||
vram_minimum_gb: 300
|
||||
description: "BF16 checkpoint; defaults to TP4"
|
||||
fp8:
|
||||
model_id: "inclusionAI/Ling-3.0-flash-VL-fp8"
|
||||
precision: fp8
|
||||
vram_minimum_gb: 150
|
||||
tp: 2
|
||||
description: "Serialized block-FP8 checkpoint; defaults to TP2"
|
||||
fp4:
|
||||
model_id: "inclusionAI/Ling-3.0-flash-VL-fp4"
|
||||
precision: fp4
|
||||
# FP8 language projections and MXFP4 routed experts; BF16 vision/projector.
|
||||
# The Hopper fallback does not require native Blackwell FP4 tensor cores.
|
||||
precision_hardware:
|
||||
brand: NVIDIA
|
||||
vram_minimum_gb: 85
|
||||
tp: 1
|
||||
description: "Mixed block-FP8 and MXFP4 checkpoint; defaults to TP1"
|
||||
int4:
|
||||
model_id: "inclusionAI/Ling-3.0-flash-VL-int4"
|
||||
precision: int4
|
||||
vram_minimum_gb: 95
|
||||
tp: 1
|
||||
description: "INT4 checkpoint; defaults to TP1"
|
||||
|
||||
compatible_strategies:
|
||||
- single_node_tp
|
||||
|
||||
hardware_overrides: {}
|
||||
|
||||
strategy_overrides:
|
||||
single_node_tp:
|
||||
tp: 4
|
||||
|
||||
guide: |
|
||||
## Overview
|
||||
|
||||
`inclusionAI/Ling-3.0-flash-VL` uses the
|
||||
`BailingMoeV3VLForConditionalGeneration` architecture. It combines the
|
||||
Ling-3.0-flash hybrid MLA/KDA language backbone with a vision encoder and
|
||||
projector for image understanding. The language backbone has 512 routed
|
||||
experts (8 active per token) and one shared expert.
|
||||
|
||||
BF16, FP8, mixed block-FP8/MXFP4, and INT4 checkpoints are available. The
|
||||
vision tower and projector remain BF16 in the mixed FP4 checkpoint.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- **vLLM:** an official nightly build containing Ling-3.0-flash-VL support
|
||||
- **Precision:** BF16, FP8, FP4, or INT4 weights with BF16 compute
|
||||
- **Context length:** 131,072 tokens
|
||||
|
||||
## Launching the Server
|
||||
|
||||
```bash
|
||||
NCCL_DEBUG=WARN vllm serve inclusionAI/Ling-3.0-flash-VL \
|
||||
--trust-remote-code \
|
||||
--dtype bfloat16 \
|
||||
--tensor-parallel-size 4 \
|
||||
--gpu-memory-utilization 0.9 \
|
||||
--enable-chunked-prefill \
|
||||
--compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE"}' \
|
||||
--enable-prefix-caching \
|
||||
--enable-auto-tool-choice \
|
||||
--tool-call-parser ling3 \
|
||||
--reasoning-parser ling3
|
||||
```
|
||||
|
||||
For a quantized variant, use the same flags with the corresponding model
|
||||
ID and tensor parallel size:
|
||||
|
||||
| Variant | Model ID | TP |
|
||||
|---|---|---:|
|
||||
| FP8 | `inclusionAI/Ling-3.0-flash-VL-fp8` | 2 |
|
||||
| FP4 | `inclusionAI/Ling-3.0-flash-VL-fp4` | 1 |
|
||||
| INT4 | `inclusionAI/Ling-3.0-flash-VL-int4` | 1 |
|
||||
|
||||
## Image Requests
|
||||
|
||||
```python
|
||||
import base64
|
||||
from pathlib import Path
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
|
||||
image = base64.b64encode(Path("image.png").read_bytes()).decode("ascii")
|
||||
response = client.chat.completions.create(
|
||||
model="inclusionAI/Ling-3.0-flash-VL",
|
||||
messages=[{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{"type": "image_url", "image_url": {"url": f"data:image/png;base64,{image}"}},
|
||||
{"type": "text", "text": "Describe the image."},
|
||||
],
|
||||
}],
|
||||
temperature=0.0,
|
||||
max_tokens=1024,
|
||||
extra_body={"chat_template_kwargs": {"enable_thinking": False}},
|
||||
)
|
||||
print(response.choices[0].message.content)
|
||||
```
|
||||
|
||||
For multiple images, add more `image_url` entries to `content`. For
|
||||
text-only requests, use ordinary string content. When serving a quantized
|
||||
variant, use its checkpoint ID in the client request.
|
||||
|
||||
## Thinking Mode
|
||||
|
||||
As with Ling-3.0-flash, thinking is selected per request through
|
||||
`chat_template_kwargs`. Set `enable_thinking` to `True` to enable it;
|
||||
with `--reasoning-parser ling3`, the parsed trace is in `message.reasoning`
|
||||
and the final answer is in `message.content`.
|
||||
|
||||
## References
|
||||
|
||||
- [Ling-3.0-flash-VL](https://huggingface.co/inclusionAI/Ling-3.0-flash-VL)
|
||||
- [Ling-3.0-flash-VL-fp8](https://huggingface.co/inclusionAI/Ling-3.0-flash-VL-fp8)
|
||||
- [Ling-3.0-flash-VL-fp4](https://huggingface.co/inclusionAI/Ling-3.0-flash-VL-fp4)
|
||||
- [Ling-3.0-flash-VL-int4](https://huggingface.co/inclusionAI/Ling-3.0-flash-VL-int4)
|
||||
@@ -0,0 +1,348 @@
|
||||
meta:
|
||||
title: "Ling-3.0-flash"
|
||||
slug: "ling-3-0-flash"
|
||||
provider: "inclusionAI"
|
||||
description: "Ling-3.0-flash MoE model with BF16, FP8, FP4, and INT4 checkpoints, native MTP, and an external DSpark draft model"
|
||||
date_updated: 2026-09-09
|
||||
tasks:
|
||||
- text
|
||||
performance_headline: "BF16, FP8, FP4, and INT4 support NVIDIA H20/H200 and AMD MI300X/MI355X; DGX Spark verified TP1 FP4/INT4"
|
||||
hardware:
|
||||
h200: verified
|
||||
dgx_spark_gb10: verified
|
||||
mi300x: verified
|
||||
mi355x: verified
|
||||
|
||||
model:
|
||||
model_id: "inclusionAI/Ling-3.0-flash"
|
||||
min_vllm_version: "0.28.0"
|
||||
architecture: moe
|
||||
parameter_count: "124B"
|
||||
active_parameters: "5.5B"
|
||||
context_length: 262144
|
||||
base_args:
|
||||
- "--trust-remote-code"
|
||||
- "--dtype"
|
||||
- "bfloat16"
|
||||
- "--gpu-memory-utilization"
|
||||
- "0.9"
|
||||
- "--enable-chunked-prefill"
|
||||
- "--compilation-config"
|
||||
- '{"cudagraph_mode":"FULL_AND_PIECEWISE"}'
|
||||
- "--enable-prefix-caching"
|
||||
|
||||
features:
|
||||
tool_calling:
|
||||
description: "Enable Ling 3 automatic tool calling"
|
||||
args:
|
||||
- "--enable-auto-tool-choice"
|
||||
- "--tool-call-parser"
|
||||
- "ling3"
|
||||
reasoning:
|
||||
description: "Split Ling 3 thinking traces into reasoning"
|
||||
args:
|
||||
- "--reasoning-parser"
|
||||
- "ling3"
|
||||
spec_decoding:
|
||||
description: "Speculative decoding with the built-in MTP head or the external Ling DSpark draft model"
|
||||
default_mode: mtp
|
||||
modes:
|
||||
mtp:
|
||||
label: "MTP"
|
||||
description: "Use the target checkpoint's native MTP head with three speculative tokens."
|
||||
args:
|
||||
- "--speculative-config"
|
||||
- '{"method":"mtp","num_speculative_tokens":3}'
|
||||
hardware_overrides:
|
||||
mi355x:
|
||||
args:
|
||||
- "--speculative-config"
|
||||
- '{"method":"mtp","num_speculative_tokens":3}'
|
||||
- "--attention-backend"
|
||||
- "TRITON_MLA"
|
||||
dspark:
|
||||
label: "DSpark"
|
||||
description: "Use inclusionAI/Ling-3.0-flash-dspark as an external draft model with any Ling-3.0-flash target variant. K may be 1-8; this recipe recommends K7."
|
||||
args:
|
||||
- "--speculative-config"
|
||||
- '{"method":"dspark","model":"inclusionAI/Ling-3.0-flash-dspark","num_speculative_tokens":7,"draft_sample_method":"probabilistic","attention_backend":"FLASH_ATTN","enable_adaptive_verification":false}'
|
||||
|
||||
opt_in_features:
|
||||
- spec_decoding
|
||||
|
||||
variants:
|
||||
default:
|
||||
precision: bf16
|
||||
vram_minimum_gb: 300
|
||||
description: "BF16 weights, the reference precision."
|
||||
fp8:
|
||||
model_id: "inclusionAI/Ling-3.0-flash-fp8"
|
||||
precision: fp8
|
||||
vram_minimum_gb: 150
|
||||
tp: 2
|
||||
description: "Block-FP8 (E4M3) weights, 128x128 block scaling."
|
||||
fp4:
|
||||
model_id: "inclusionAI/Ling-3.0-flash-fp4"
|
||||
precision: fp4
|
||||
precision_hardware: {}
|
||||
vram_minimum_gb: 85
|
||||
tp: 1
|
||||
description: "Block-FP8 dense with MXFP4 routed experts. Emulated on gfx942 — prefer INT4 there."
|
||||
int4:
|
||||
model_id: "inclusionAI/Ling-3.0-flash-int4"
|
||||
precision: int4
|
||||
vram_minimum_gb: 95
|
||||
tp: 1
|
||||
description: "compressed-tensors W4 (group 32) on the routed experts."
|
||||
|
||||
compatible_strategies:
|
||||
- single_node_tp
|
||||
|
||||
hardware_overrides:
|
||||
amd:
|
||||
extra_env:
|
||||
VLLM_ROCM_USE_AITER: "1"
|
||||
|
||||
strategy_overrides:
|
||||
single_node_tp:
|
||||
tp: 4
|
||||
|
||||
guide: |
|
||||
## Overview
|
||||
|
||||
`inclusionAI/Ling-3.0-flash` uses the `BailingMoeV3ForCausalLM` architecture
|
||||
with a hybrid MLA/KDA attention stack, 512 routed experts (8 active per token),
|
||||
one shared expert, and a native multi-token prediction head. The 42-layer base
|
||||
model has 124.4B total and 5.5B active parameters. The checkpoint also contains
|
||||
a 3.1B MTP layer, bringing the complete checkpoint to 127.5B parameters. A
|
||||
serialized block-FP8 checkpoint is available as
|
||||
`inclusionAI/Ling-3.0-flash-fp8`. Lower-memory checkpoints are also available
|
||||
as `inclusionAI/Ling-3.0-flash-fp4` and
|
||||
`inclusionAI/Ling-3.0-flash-int4`. The external
|
||||
`inclusionAI/Ling-3.0-flash-dspark` draft can accelerate any target variant
|
||||
without changing which target checkpoint is served.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- **vLLM:** 0.28.0 or newer, which includes native Bailing V3 support;
|
||||
- **Validated hardware:** NVIDIA H20, H20-3e, H200, DGX Spark (FP4/INT4), and AMD MI300X / MI355X
|
||||
- **Precision:** BF16, serialized block FP8, mixed block-FP8/MXFP4, or compressed-tensors INT4 weights with BF16 compute
|
||||
- **Context length:** 262,144 tokens
|
||||
|
||||
## Launching the Server
|
||||
|
||||
```bash
|
||||
NCCL_DEBUG=WARN vllm serve inclusionAI/Ling-3.0-flash \
|
||||
--trust-remote-code \
|
||||
--dtype bfloat16 \
|
||||
--tensor-parallel-size 4 \
|
||||
--gpu-memory-utilization 0.9 \
|
||||
--enable-chunked-prefill \
|
||||
--compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE"}' \
|
||||
--enable-prefix-caching \
|
||||
--enable-auto-tool-choice \
|
||||
--tool-call-parser ling3 \
|
||||
--reasoning-parser ling3 \
|
||||
--speculative-config '{"method":"mtp","num_speculative_tokens":3}'
|
||||
```
|
||||
|
||||
For the serialized FP8 variant on 2x H200, use:
|
||||
|
||||
```bash
|
||||
vllm serve inclusionAI/Ling-3.0-flash-fp8 \
|
||||
--trust-remote-code \
|
||||
--dtype bfloat16 \
|
||||
--tensor-parallel-size 2 \
|
||||
--gpu-memory-utilization 0.9 \
|
||||
--enable-chunked-prefill \
|
||||
--compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE"}' \
|
||||
--enable-prefix-caching \
|
||||
--enable-auto-tool-choice \
|
||||
--tool-call-parser ling3 \
|
||||
--reasoning-parser ling3
|
||||
```
|
||||
|
||||
For the validated TP4+EP4 alternative on 4x H200, use:
|
||||
|
||||
```bash
|
||||
vllm serve inclusionAI/Ling-3.0-flash-fp8 \
|
||||
--trust-remote-code \
|
||||
--dtype bfloat16 \
|
||||
--tensor-parallel-size 4 \
|
||||
--enable-expert-parallel \
|
||||
--gpu-memory-utilization 0.9 \
|
||||
--enable-chunked-prefill \
|
||||
--compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE"}' \
|
||||
--enable-prefix-caching \
|
||||
--enable-auto-tool-choice \
|
||||
--tool-call-parser ling3 \
|
||||
--reasoning-parser ling3
|
||||
```
|
||||
|
||||
For the FP4 variant on one GPU, use:
|
||||
|
||||
```bash
|
||||
vllm serve inclusionAI/Ling-3.0-flash-fp4 \
|
||||
--trust-remote-code \
|
||||
--dtype bfloat16 \
|
||||
--tensor-parallel-size 1 \
|
||||
--gpu-memory-utilization 0.9 \
|
||||
--enable-chunked-prefill \
|
||||
--compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE"}' \
|
||||
--enable-prefix-caching \
|
||||
--enable-auto-tool-choice \
|
||||
--tool-call-parser ling3 \
|
||||
--reasoning-parser ling3
|
||||
```
|
||||
|
||||
For the INT4 variant on one GPU, use:
|
||||
|
||||
```bash
|
||||
vllm serve inclusionAI/Ling-3.0-flash-int4 \
|
||||
--trust-remote-code \
|
||||
--dtype bfloat16 \
|
||||
--tensor-parallel-size 1 \
|
||||
--gpu-memory-utilization 0.9 \
|
||||
--enable-chunked-prefill \
|
||||
--compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE"}' \
|
||||
--enable-prefix-caching \
|
||||
--enable-auto-tool-choice \
|
||||
--tool-call-parser ling3 \
|
||||
--reasoning-parser ling3
|
||||
```
|
||||
|
||||
## Speculative Decoding
|
||||
|
||||
The **Spec decoding** control exposes MTP and DSpark in the same place. They
|
||||
are mutually exclusive modes, so the generated command contains exactly one
|
||||
`--speculative-config`.
|
||||
|
||||
### MTP
|
||||
|
||||
Use the target checkpoint's native MTP head with three draft tokens:
|
||||
|
||||
```bash
|
||||
--speculative-config '{"method":"mtp","num_speculative_tokens":3}'
|
||||
```
|
||||
|
||||
### DSpark
|
||||
|
||||
`inclusionAI/Ling-3.0-flash-dspark` is an external draft model. It does not
|
||||
replace the served target checkpoint. It can be paired with any
|
||||
Ling-3.0-flash target variant; pass the DSpark repository through
|
||||
`--speculative-config.model` while keeping the selected target as the model
|
||||
served by `vllm serve`.
|
||||
|
||||
The checkpoint was trained with query block size `Q=8`. Runtime
|
||||
`num_speculative_tokens` (`K`) may be any integer from 1 through 8. The recipe
|
||||
emits `K=7` as the recommended throughput/KV-capacity trade-off; change the
|
||||
value to select another supported width. `K=8` remains valid.
|
||||
|
||||
```bash
|
||||
--speculative-config '{"method":"dspark","model":"inclusionAI/Ling-3.0-flash-dspark","num_speculative_tokens":7,"draft_sample_method":"probabilistic","attention_backend":"FLASH_ATTN","enable_adaptive_verification":false}'
|
||||
```
|
||||
|
||||
`enable_adaptive_verification` is disabled because the current Ling hybrid
|
||||
MLA/KDA path is validated with fixed-width DSpark verification. The
|
||||
`attention_backend` setting applies only to the external draft model; the
|
||||
target model keeps its automatically selected attention backends.
|
||||
|
||||
## Single-GPU INT4
|
||||
|
||||
`inclusionAI/Ling-3.0-flash-int4` is a `compressed-tensors` / `pack-quantized`
|
||||
checkpoint: symmetric W4, `group_size: 32`, applied to the **routed experts
|
||||
only** — attention, `lm_head`, the shared expert and the dense projections are
|
||||
all in the config's `ignore` list. It is not covered by the model-specific
|
||||
quantization plumbing in `bailing_moe_v3.py` (which handles block FP8 and
|
||||
MXFP4); it loads through vLLM's generic `compressed-tensors` path and runs at
|
||||
TP1 on H200 or DGX Spark.
|
||||
|
||||
The engine names the mechanism at startup:
|
||||
|
||||
```
|
||||
quantization=compressed-tensors ... trust_remote_code=True, dtype=torch.bfloat16
|
||||
INFO [int_wna16.py:297] Using 'MARLIN' WNA16 MoE backend.
|
||||
INFO [cuda.py:492] Using FLASH_ATTN_MLA attention backend
|
||||
INFO [int_wna16.py:409] Using MarlinExperts
|
||||
```
|
||||
|
||||
The vendor model card documents SGLang only, and there is no merged vLLM PR
|
||||
specific to Ling INT4. This variant works through the generic quantization
|
||||
path; tool calling, reasoning parsing, chunked prefill and CUDA graph capture
|
||||
behave as they do on the other variants.
|
||||
|
||||
## Running on MI300X/MI355X
|
||||
|
||||
```bash
|
||||
export VLLM_ROCM_USE_AITER=1
|
||||
vllm serve inclusionAI/Ling-3.0-flash \
|
||||
--trust-remote-code \
|
||||
--dtype bfloat16 \
|
||||
--tensor-parallel-size 4 \
|
||||
--gpu-memory-utilization 0.9 \
|
||||
--enable-chunked-prefill \
|
||||
--compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE"}' \
|
||||
--enable-prefix-caching \
|
||||
--enable-auto-tool-choice \
|
||||
--tool-call-parser ling3 \
|
||||
--reasoning-parser ling3
|
||||
```
|
||||
|
||||
On MI300X the BF16 run served the full 262,144-token context with a 105.15 GiB
|
||||
KV pool and 52.38x max concurrency.
|
||||
|
||||
**MTP** needs a backend pin on gfx950: `ROCM_AITER_MLA` lets the speculator
|
||||
capture a graph the backend cannot serve and the workers abort with a memory
|
||||
access fault, so the mode pins `--attention-backend TRITON_MLA` on `mi355x`.
|
||||
gfx942 is unaffected and keeps AITER MLA.
|
||||
|
||||
## Validation
|
||||
|
||||
The BF16, FP8, FP4, and INT4 checkpoints support NVIDIA H20 and H200. For FP8,
|
||||
TP2 remains the recommended multi-GPU default; TP4+EP4 is a validated H200
|
||||
alternative for deployments that prefer expert parallelism. DGX Spark is
|
||||
limited to the TP1 FP4 and INT4 variants because the BF16 and FP8 checkpoints
|
||||
are too large for its 128 GB unified memory. DSpark is external and works with
|
||||
any target variant that fits the selected hardware.
|
||||
|
||||
The `int4` variant was verified on **1x H200 (SM90), TP=1**, on a `main` build
|
||||
reporting `0.26.1rc1.dev1133+gf94666b60`, serving at `--max-model-len 32768`:
|
||||
77.0 GB of weights on disk, 70.27 GiB resident after load (165 s), 5 min 56 s
|
||||
from container start to ready, `FULL_AND_PIECEWISE` capture (68 PIECEWISE + 68
|
||||
FULL, 2.15 GiB), and a 4,072,594-token KV pool. Single-stream decode measured
|
||||
**159.5 tok/s** at TTFT p50 0.17 s; 973 output tok/s aggregate at concurrency
|
||||
16, with 0 failed requests and 0 preemptions over a 1..24 concurrency sweep.
|
||||
|
||||
MI355X was verified on MI350X silicon, the same gfx950 class.
|
||||
|
||||
## Thinking Mode
|
||||
|
||||
Thinking is selected per request through the chat template rather than by a
|
||||
server flag:
|
||||
|
||||
```python
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
|
||||
response = client.chat.completions.create(
|
||||
model="inclusionAI/Ling-3.0-flash",
|
||||
messages=[{"role": "user", "content": "Solve the problem step by step."}],
|
||||
temperature=0.0,
|
||||
max_tokens=200000,
|
||||
extra_body={"chat_template_kwargs": {"enable_thinking": True}},
|
||||
)
|
||||
msg = response.choices[0].message
|
||||
print(msg.reasoning)
|
||||
print(msg.content)
|
||||
```
|
||||
|
||||
When serving a quantized variant, set `model` in the client request to the
|
||||
corresponding FP8, FP4, or INT4 checkpoint ID.
|
||||
|
||||
## References
|
||||
|
||||
- [Ling-3.0-flash on Hugging Face](https://huggingface.co/inclusionAI/Ling-3.0-flash)
|
||||
- [Ling-3.0-flash-fp8 on Hugging Face](https://huggingface.co/inclusionAI/Ling-3.0-flash-fp8)
|
||||
- [Ling-3.0-flash-dspark on Hugging Face](https://huggingface.co/inclusionAI/Ling-3.0-flash-dspark)
|
||||
- [Ling-3.0-flash-fp4 on Hugging Face](https://huggingface.co/inclusionAI/Ling-3.0-flash-fp4)
|
||||
- [Ling-3.0-flash-int4 on Hugging Face](https://huggingface.co/inclusionAI/Ling-3.0-flash-int4)
|
||||
@@ -0,0 +1,164 @@
|
||||
meta:
|
||||
title: "Ling-3.0-tiny"
|
||||
slug: "ling-3-0-tiny"
|
||||
provider: "inclusionAI"
|
||||
description: "Ling-3.0-tiny lightweight hybrid-reasoning MoE with BF16, block-FP8, and compressed-tensors INT4 checkpoints"
|
||||
date_added: 2026-08-31
|
||||
date_updated: 2026-09-09
|
||||
tasks:
|
||||
- text
|
||||
performance_headline: "7.9B total / 1.3B active parameters with a 3:1 KDA-MLA stack; all three official checkpoints run with TP1"
|
||||
related_recipes:
|
||||
- inclusionAI/Ling-3.0-flash
|
||||
hardware:
|
||||
h200: verified
|
||||
|
||||
model:
|
||||
model_id: "inclusionAI/Ling-3.0-tiny"
|
||||
min_vllm_version: "0.28.0"
|
||||
architecture: moe
|
||||
parameter_count: "7.9B"
|
||||
active_parameters: "1.3B"
|
||||
context_length: 131072
|
||||
base_args:
|
||||
- "--trust-remote-code"
|
||||
- "--dtype"
|
||||
- "bfloat16"
|
||||
- "--gpu-memory-utilization"
|
||||
- "0.85"
|
||||
- "--enable-prefix-caching"
|
||||
|
||||
features:
|
||||
tool_calling:
|
||||
description: "Enable Ling 3 automatic tool calling"
|
||||
args:
|
||||
- "--enable-auto-tool-choice"
|
||||
- "--tool-call-parser"
|
||||
- "ling3"
|
||||
reasoning:
|
||||
description: "Split Ling 3 thinking traces into reasoning"
|
||||
args:
|
||||
- "--reasoning-parser"
|
||||
- "ling3"
|
||||
|
||||
opt_in_features: []
|
||||
|
||||
variants:
|
||||
default:
|
||||
precision: bf16
|
||||
vram_minimum_gb: 18
|
||||
tp: 1
|
||||
description: "Official BF16 checkpoint; approximately 15.8 GB of repository weights and served with TP1"
|
||||
fp8:
|
||||
model_id: "inclusionAI/Ling-3.0-tiny-fp8"
|
||||
precision: fp8
|
||||
vram_minimum_gb: 11
|
||||
tp: 1
|
||||
description: "Official serialized block-FP8 checkpoint with dynamic activations and 128x128 weight blocks; served with TP1"
|
||||
int4:
|
||||
model_id: "inclusionAI/Ling-3.0-tiny-int4"
|
||||
precision: int4
|
||||
vram_minimum_gb: 8
|
||||
tp: 1
|
||||
description: "Official compressed-tensors group-wise INT4 checkpoint; served with TP1"
|
||||
|
||||
compatible_strategies:
|
||||
- single_node_tp
|
||||
|
||||
hardware_overrides: {}
|
||||
|
||||
strategy_overrides:
|
||||
single_node_tp:
|
||||
tp: 1
|
||||
|
||||
guide: |
|
||||
## Overview
|
||||
|
||||
`inclusionAI/Ling-3.0-tiny` is the lightweight member of the Ling 3.0
|
||||
family. It has 7.9B total parameters and activates 1.3B parameters per token.
|
||||
Its 24-layer hybrid stack repeats three Kimi Delta Attention (KDA) layers and
|
||||
one Multi-Head Latent Attention (MLA) layer, while each sparse MoE block
|
||||
selects 8 of 128 routed experts plus one shared expert.
|
||||
|
||||
The official release provides three served checkpoints:
|
||||
|
||||
- `inclusionAI/Ling-3.0-tiny`: BF16;
|
||||
- `inclusionAI/Ling-3.0-tiny-fp8`: serialized block FP8;
|
||||
- `inclusionAI/Ling-3.0-tiny-int4`: compressed-tensors INT4.
|
||||
|
||||
All three use the same `BailingMoeV3ForCausalLM` architecture and a native
|
||||
131,072-token context window. Select the corresponding Variant above; the
|
||||
served model ID changes automatically.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- **vLLM:** 0.28.0 or newer, which includes Ling 3.0 / Bailing V3
|
||||
hybrid-attention support;
|
||||
- **Parallelism:** TP1 is recommended for all three checkpoints;
|
||||
- **Thinking mode:** enabled per request by the chat template.
|
||||
|
||||
## Launching the Server
|
||||
|
||||
BF16 on one GPU:
|
||||
|
||||
```bash
|
||||
vllm serve inclusionAI/Ling-3.0-tiny \
|
||||
--trust-remote-code \
|
||||
--dtype bfloat16 \
|
||||
--tensor-parallel-size 1 \
|
||||
--gpu-memory-utilization 0.85 \
|
||||
--enable-prefix-caching \
|
||||
--enable-auto-tool-choice \
|
||||
--tool-call-parser ling3 \
|
||||
--reasoning-parser ling3
|
||||
```
|
||||
|
||||
For FP8 or INT4, keep the same arguments and replace the served checkpoint:
|
||||
|
||||
```bash
|
||||
# Serialized block FP8
|
||||
vllm serve inclusionAI/Ling-3.0-tiny-fp8 \
|
||||
--trust-remote-code --dtype bfloat16 --tensor-parallel-size 1 \
|
||||
--gpu-memory-utilization 0.85 --enable-prefix-caching --enable-auto-tool-choice \
|
||||
--tool-call-parser ling3 --reasoning-parser ling3
|
||||
|
||||
# Compressed-tensors INT4
|
||||
vllm serve inclusionAI/Ling-3.0-tiny-int4 \
|
||||
--trust-remote-code --dtype bfloat16 --tensor-parallel-size 1 \
|
||||
--gpu-memory-utilization 0.85 --enable-prefix-caching --enable-auto-tool-choice \
|
||||
--tool-call-parser ling3 --reasoning-parser ling3
|
||||
```
|
||||
|
||||
## Thinking Mode
|
||||
|
||||
Thinking is controlled per request. The model card recommends
|
||||
`temperature=1.0`, `top_p=0.95`, and `top_k=20` when thinking is enabled:
|
||||
|
||||
```python
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
|
||||
response = client.chat.completions.create(
|
||||
model="inclusionAI/Ling-3.0-tiny",
|
||||
messages=[{"role": "user", "content": "Solve the problem step by step."}],
|
||||
temperature=1.0,
|
||||
top_p=0.95,
|
||||
extra_body={
|
||||
"top_k": 20,
|
||||
"chat_template_kwargs": {"enable_thinking": True},
|
||||
},
|
||||
)
|
||||
msg = response.choices[0].message
|
||||
print(msg.reasoning)
|
||||
print(msg.content)
|
||||
```
|
||||
|
||||
Set `enable_thinking` to `false` for direct answers. When using a quantized
|
||||
variant, set the client `model` to its FP8 or INT4 checkpoint ID.
|
||||
|
||||
## References
|
||||
|
||||
- [Ling 3.0 collection](https://huggingface.co/collections/inclusionAI/ling-30)
|
||||
- [Ling-3.0-tiny on Hugging Face](https://huggingface.co/inclusionAI/Ling-3.0-tiny)
|
||||
- [Ling-3.0-tiny-fp8 on Hugging Face](https://huggingface.co/inclusionAI/Ling-3.0-tiny-fp8)
|
||||
- [Ling-3.0-tiny-int4 on Hugging Face](https://huggingface.co/inclusionAI/Ling-3.0-tiny-int4)
|
||||
@@ -0,0 +1,170 @@
|
||||
meta:
|
||||
title: "Ling-3.1-flash"
|
||||
slug: "ling-3-1-flash"
|
||||
provider: "inclusionAI"
|
||||
description: "Ling-3.1-flash MoE model with block-FP8 weights, tool calling, and reasoning support"
|
||||
date_added: 2026-10-14
|
||||
date_updated: 2026-10-15
|
||||
tasks:
|
||||
- text
|
||||
performance_headline: "Block-FP8 weights with BF16 compute and TP8 + EP8 deployment"
|
||||
related_recipes:
|
||||
- inclusionAI/Ling-3.0-flash
|
||||
hardware:
|
||||
mi355x: verified
|
||||
|
||||
model:
|
||||
model_id: "inclusionAI/Ling-3.1-flash"
|
||||
min_vllm_version: "nightly"
|
||||
nightly_required: true
|
||||
docker_image:
|
||||
nvidia: "vllm/vllm-openai:nightly"
|
||||
amd: "vllm/vllm-openai-rocm:nightly"
|
||||
architecture: moe
|
||||
parameter_count: "560B"
|
||||
active_parameters: "25B"
|
||||
context_length: 262144
|
||||
base_args:
|
||||
- "--trust-remote-code"
|
||||
- "--enable-expert-parallel"
|
||||
- "--gpu-memory-utilization"
|
||||
- "0.9"
|
||||
- "--enable-chunked-prefill"
|
||||
- "--compilation-config"
|
||||
- '{"cudagraph_mode":"FULL_AND_PIECEWISE"}'
|
||||
- "--enable-prefix-caching"
|
||||
|
||||
features:
|
||||
tool_calling:
|
||||
description: "Enable Ling 3 automatic tool calling"
|
||||
args:
|
||||
- "--enable-auto-tool-choice"
|
||||
- "--tool-call-parser"
|
||||
- "ling3"
|
||||
reasoning:
|
||||
description: "Split Ling 3 thinking traces into reasoning"
|
||||
args:
|
||||
- "--reasoning-parser"
|
||||
- "ling3"
|
||||
spec_decoding:
|
||||
description: "Speculative decoding with the checkpoint's native MTP head and three speculative tokens"
|
||||
args:
|
||||
- "--speculative-config"
|
||||
- '{"method":"mtp","num_speculative_tokens":3}'
|
||||
|
||||
opt_in_features:
|
||||
- spec_decoding
|
||||
|
||||
variants:
|
||||
default:
|
||||
precision: fp8
|
||||
# Estimated weight footprint plus runtime allowance; KV capacity varies.
|
||||
vram_minimum_gb: 689
|
||||
tp: 8
|
||||
description: "Block-FP8 (E4M3) weights, 128x128 block scaling, with BF16 compute."
|
||||
|
||||
compatible_strategies:
|
||||
- single_node_tep
|
||||
|
||||
default_strategy: single_node_tep
|
||||
|
||||
hardware_overrides:
|
||||
amd:
|
||||
extra_env:
|
||||
VLLM_ROCM_USE_AITER: "1"
|
||||
|
||||
guide: |
|
||||
## Overview
|
||||
|
||||
`inclusionAI/Ling-3.1-flash` uses the `BailingMoeV3ForCausalLM`
|
||||
architecture with a hybrid MLA/KDA attention stack, 72 layers, 512 routed
|
||||
experts (8 active per token), and one shared expert. The checkpoint uses
|
||||
serialized block-FP8 E4M3 weights with 128x128 block scaling and dynamic
|
||||
activation quantization. Non-quantized modules use BF16.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- **vLLM:** nightly build (`vllm/vllm-openai-rocm:nightly` on AMD)
|
||||
- **Deployment:** a single node with 8 GPUs, using TP8 + EP8
|
||||
- **Precision:** serialized block FP8 weights with BF16 compute
|
||||
- **Context length:** 262,144 tokens
|
||||
- **Memory:** approximately 689 GB aggregate VRAM as a loading estimate;
|
||||
allow additional capacity for KV cache and the intended workload
|
||||
|
||||
## Launching the Server
|
||||
|
||||
```bash
|
||||
vllm serve inclusionAI/Ling-3.1-flash \
|
||||
--trust-remote-code \
|
||||
--tensor-parallel-size 8 \
|
||||
--enable-expert-parallel \
|
||||
--gpu-memory-utilization 0.9 \
|
||||
--enable-chunked-prefill \
|
||||
--compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE"}' \
|
||||
--enable-prefix-caching \
|
||||
--enable-auto-tool-choice \
|
||||
--tool-call-parser ling3 \
|
||||
--reasoning-parser ling3
|
||||
```
|
||||
|
||||
### AMD MI355X
|
||||
|
||||
Use the ROCm nightly image `vllm/vllm-openai-rocm:nightly`,
|
||||
enable AITER and launch using the command:
|
||||
|
||||
```bash
|
||||
VLLM_ROCM_USE_AITER=1 vllm serve inclusionAI/Ling-3.1-flash \
|
||||
--trust-remote-code \
|
||||
--tensor-parallel-size 8 \
|
||||
--enable-expert-parallel \
|
||||
--gpu-memory-utilization 0.9 \
|
||||
--enable-chunked-prefill \
|
||||
--compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE"}' \
|
||||
--enable-prefix-caching \
|
||||
--enable-auto-tool-choice \
|
||||
--tool-call-parser ling3 \
|
||||
--reasoning-parser ling3
|
||||
```
|
||||
|
||||
Each GPU has about 168 GiB of KV cache available.
|
||||
|
||||
## Speculative Decoding (MTP)
|
||||
|
||||
The checkpoint ships a native MTP head. Enable it with three speculative
|
||||
tokens:
|
||||
|
||||
```bash
|
||||
--speculative-config '{"method":"mtp","num_speculative_tokens":3}'
|
||||
```
|
||||
|
||||
On MI355X this gave a mean acceptance length of 3.0 (per-position acceptance
|
||||
0.96 / 0.67 / 0.38) on short reasoning prompts.
|
||||
|
||||
## Thinking Mode
|
||||
|
||||
Thinking is selected per request through the chat template rather than by a
|
||||
server flag:
|
||||
|
||||
```python
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
|
||||
response = client.chat.completions.create(
|
||||
model="inclusionAI/Ling-3.1-flash",
|
||||
messages=[{"role": "user", "content": "Solve the problem step by step."}],
|
||||
temperature=0.0,
|
||||
max_tokens=16384,
|
||||
extra_body={"chat_template_kwargs": {"enable_thinking": True}},
|
||||
)
|
||||
msg = response.choices[0].message
|
||||
print(msg.reasoning)
|
||||
print(msg.content)
|
||||
```
|
||||
|
||||
`--reasoning-parser ling3` separates thinking traces from the final answer.
|
||||
Keep the prompt and requested output within the configured context limit.
|
||||
|
||||
## References
|
||||
|
||||
- [Ling-3.1-flash on Hugging Face](https://huggingface.co/inclusionAI/Ling-3.1-flash)
|
||||
- [vLLM nightly installation](https://docs.vllm.ai/en/latest/getting_started/installation/gpu/#install-the-latest-code)
|
||||
@@ -0,0 +1,133 @@
|
||||
meta:
|
||||
title: "Ring-1T-FP8"
|
||||
slug: "ring-1t-fp8"
|
||||
provider: "inclusionAI"
|
||||
description: "Ring-1T (BailingMoeV2) FP8 model (~1T total params) for 8xH200 or 8xMI300X deployment"
|
||||
date_added: 2025-10-15
|
||||
date_updated: 2026-04-17
|
||||
tasks:
|
||||
- text
|
||||
related_recipes: []
|
||||
hardware:
|
||||
h200: verified
|
||||
mi300x: verified
|
||||
mi325x: verified
|
||||
mi355x: verified
|
||||
|
||||
model:
|
||||
model_id: "inclusionAI/Ring-1T-FP8"
|
||||
min_vllm_version: "0.11.0"
|
||||
architecture: moe
|
||||
parameter_count: "1T"
|
||||
active_parameters: "50B"
|
||||
context_length: 65536
|
||||
base_args:
|
||||
- "--trust-remote-code"
|
||||
- "--tensor-parallel-size"
|
||||
- "8"
|
||||
- "--max-num-seqs"
|
||||
- "32"
|
||||
- "--kv-cache-dtype"
|
||||
- "fp8"
|
||||
- "--served-model-name"
|
||||
- "Ring-1T-FP8"
|
||||
base_env: {}
|
||||
|
||||
features: {}
|
||||
|
||||
opt_in_features: []
|
||||
|
||||
variants:
|
||||
default:
|
||||
precision: fp8
|
||||
vram_minimum_gb: 1200
|
||||
description: "FP8 weights on 8x H200 (80 GB) with FP8 KV cache"
|
||||
extra_args:
|
||||
- "--gpu-memory-utilization"
|
||||
- "0.97"
|
||||
- "--compilation-config"
|
||||
- '{"use_inductor": false}'
|
||||
|
||||
compatible_strategies:
|
||||
- single_node_tp
|
||||
- multi_node_tp
|
||||
- multi_node_tp_pp
|
||||
- multi_node_tep
|
||||
- multi_node_dep
|
||||
- pd_cluster
|
||||
|
||||
hardware_overrides: {}
|
||||
strategy_overrides: {}
|
||||
|
||||
guide: |
|
||||
## Overview
|
||||
|
||||
Ring-1T-FP8 is inclusionAI's BailingMoeV2 FP8 model (~1T total parameters). This recipe
|
||||
covers pure tensor-parallel deployment across 8 GPUs on NVIDIA H200 or AMD MI300X+.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Hardware: 8x H200 or 8x MI300X/MI325X/MI355X
|
||||
- vLLM >= 0.11.0
|
||||
|
||||
### Install vLLM (CUDA)
|
||||
|
||||
```bash
|
||||
uv venv
|
||||
source .venv/bin/activate
|
||||
uv pip install -U vllm --torch-backend auto
|
||||
```
|
||||
|
||||
### Install vLLM (AMD ROCm)
|
||||
|
||||
```bash
|
||||
uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/0.14.1/rocm700
|
||||
```
|
||||
|
||||
## Launch commands
|
||||
|
||||
8x H200 (FP8 KV cache):
|
||||
|
||||
```bash
|
||||
vllm serve inclusionAI/Ring-1T-FP8 \
|
||||
--trust-remote-code \
|
||||
--tensor-parallel-size 8 \
|
||||
--gpu-memory-utilization 0.97 \
|
||||
--max-num-seqs 32 \
|
||||
--kv-cache-dtype fp8 \
|
||||
--compilation-config '{"use_inductor": false}' \
|
||||
--served-model-name Ring-1T-FP8
|
||||
```
|
||||
|
||||
8x MI300X/MI325X/MI355X:
|
||||
|
||||
```bash
|
||||
export VLLM_ROCM_USE_AITER=1
|
||||
vllm serve inclusionAI/Ring-1T-FP8 \
|
||||
--trust-remote-code \
|
||||
--tensor-parallel-size 8 \
|
||||
--gpu-memory-utilization 0.9 \
|
||||
--max-num-seqs 32 \
|
||||
--kv-cache-dtype fp8 \
|
||||
--served-model-name Ring-1T-FP8
|
||||
```
|
||||
|
||||
Tuning flags:
|
||||
- `--max-model-len=65536` works well for most scenarios.
|
||||
- `--max-num-batched-tokens=32768` for prompt-heavy; 16384/8192 for lower latency.
|
||||
- Reduce `--gpu-memory-utilization` below 0.97 if you hit OOM.
|
||||
|
||||
## Client Usage
|
||||
|
||||
```bash
|
||||
curl http://localhost:8000/v1/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"model": "Ring-1T-FP8",
|
||||
"messages": [{"role": "user", "content": "9.11 and 9.8, which is greater?"}]
|
||||
}'
|
||||
```
|
||||
|
||||
## References
|
||||
|
||||
- [Ring-1T-FP8 on Hugging Face](https://huggingface.co/inclusionAI/Ring-1T-FP8)
|
||||
@@ -0,0 +1,127 @@
|
||||
meta:
|
||||
title: "Ring-2.6-1T"
|
||||
slug: "ring-2.6-1t"
|
||||
provider: "inclusionAI"
|
||||
description: "Ring-2.6-1T (BailingMoeV2_5) FP8 thinking model with 1T total / 50B active params, hybrid linear + MLA attention, 128K context"
|
||||
date_added: 2026-05-14
|
||||
date_updated: 2026-05-15
|
||||
tasks:
|
||||
- text
|
||||
related_recipes:
|
||||
- inclusionAI/Ling-2.6-1T
|
||||
- inclusionAI/Ring-1T-FP8
|
||||
|
||||
model:
|
||||
model_id: "inclusionAI/Ring-2.6-1T"
|
||||
min_vllm_version: "0.20.2"
|
||||
architecture: moe
|
||||
parameter_count: "1T"
|
||||
active_parameters: "50B"
|
||||
context_length: 131072
|
||||
base_args:
|
||||
- "--trust-remote-code"
|
||||
- "--tensor-parallel-size"
|
||||
- "8"
|
||||
base_env: {}
|
||||
|
||||
features: {}
|
||||
|
||||
opt_in_features: []
|
||||
|
||||
variants:
|
||||
default:
|
||||
precision: fp8
|
||||
vram_minimum_gb: 1200
|
||||
description: "FP8 weights with TP=8 on B300 or 8x MI300X/MI355X-class nodes"
|
||||
|
||||
compatible_strategies:
|
||||
- single_node_tp
|
||||
- multi_node_tp
|
||||
- multi_node_tp_pp
|
||||
- multi_node_tep
|
||||
- multi_node_dep
|
||||
- pd_cluster
|
||||
|
||||
hardware_overrides:
|
||||
amd:
|
||||
extra_env:
|
||||
VLLM_ROCM_USE_AITER: "1"
|
||||
|
||||
strategy_overrides:
|
||||
single_node_tp:
|
||||
tp: 8
|
||||
|
||||
guide: |
|
||||
## Overview
|
||||
|
||||
[Ring-2.6-1T](https://huggingface.co/inclusionAI/Ring-2.6-1T) is inclusionAI's
|
||||
BailingMoeV2_5 FP8 trillion-scale **thinking** model (1T total / 50B active
|
||||
parameters) with hybrid linear + MLA attention and a 128K context window.
|
||||
It is the reasoning-focused counterpart to [Ling-2.6-1T](https://huggingface.co/inclusionAI/Ling-2.6-1T)
|
||||
in the Ring 2.6 series and emits explicit `<think>...</think>` traces; the
|
||||
chat template accepts a `reasoning_effort` field (default `high`).
|
||||
|
||||
vLLM 0.20.1 shipped the BailingMoeV2.5 MLA RoPE fix (vllm-project/vllm#41185)
|
||||
that is load-bearing for this architecture — pin v0.20.2 or newer.
|
||||
|
||||
## Deployment Configurations
|
||||
|
||||
### Docker (AMD MI300X / MI325X / MI355X, TP=8)
|
||||
|
||||
TP=8 fits the model-derived 128K context on an MI300X-class node; MI325X
|
||||
and MI355X have larger per-GPU HBM and more headroom.
|
||||
|
||||
```bash
|
||||
docker run --rm -it \
|
||||
--cap-add=SYS_PTRACE \
|
||||
--ipc=host \
|
||||
--privileged=true \
|
||||
--shm-size=128GB \
|
||||
--network=host \
|
||||
--device=/dev/kfd \
|
||||
--device=/dev/dri \
|
||||
--group-add video \
|
||||
-e VLLM_ROCM_USE_AITER=1 \
|
||||
vllm/vllm-openai-rocm:v0.28.0 \
|
||||
inclusionAI/Ring-2.6-1T \
|
||||
--tensor-parallel-size 8 \
|
||||
--trust-remote-code
|
||||
```
|
||||
|
||||
### Pip (NVIDIA B300, TP=8)
|
||||
|
||||
```bash
|
||||
uv venv && source .venv/bin/activate
|
||||
uv pip install -U vllm --torch-backend auto
|
||||
|
||||
vllm serve inclusionAI/Ring-2.6-1T \
|
||||
--trust-remote-code \
|
||||
--tensor-parallel-size 8
|
||||
```
|
||||
|
||||
## Client Usage
|
||||
|
||||
### Text Generation
|
||||
|
||||
```python
|
||||
from openai import OpenAI
|
||||
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
|
||||
response = client.chat.completions.create(
|
||||
model="inclusionAI/Ring-2.6-1T",
|
||||
messages=[{"role": "user", "content": "Prove there are infinitely many primes."}],
|
||||
max_tokens=4096,
|
||||
temperature=0.6,
|
||||
extra_body={"chat_template_kwargs": {"reasoning_effort": "high"}},
|
||||
)
|
||||
print(response.choices[0].message.content)
|
||||
```
|
||||
|
||||
Set `reasoning_effort` to `low` / `medium` / `high` to control the depth of
|
||||
thinking. The model wraps its reasoning trace in `<think>...</think>` before
|
||||
emitting the final answer.
|
||||
|
||||
## References
|
||||
|
||||
- [Ring-2.6-1T on Hugging Face](https://huggingface.co/inclusionAI/Ring-2.6-1T)
|
||||
- [vLLM PR #41185 — BailingMoeV2.5 MLA RoPE fix](https://github.com/vllm-project/vllm/pull/41185)
|
||||
- [vLLM ROCm GPU install docs](https://docs.vllm.ai/en/latest/getting_started/installation/gpu/?device=rocm)
|
||||
Reference in New Issue
Block a user