[Doc] Host model recipes in recipes/ (#61302)
Signed-off-by: esmeetu <jasonailu87@gmail.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 5.5
parent
f98ae6a9fd
commit
b515648641
@@ -0,0 +1,109 @@
|
||||
meta:
|
||||
title: "Jais-3-0.9B"
|
||||
slug: "jais-3-0.9b"
|
||||
provider: "IFM"
|
||||
description: "Compact dense model for mathematics, code, instruction following, and STEM tasks"
|
||||
date_added: 2026-10-05
|
||||
date_updated: 2026-10-05
|
||||
tasks:
|
||||
- text
|
||||
performance_headline: "1.08B stored parameters including embeddings, with 131K context and native K2-Horizon parsing"
|
||||
related_recipes:
|
||||
- "IFM/Jais-3-3.7B"
|
||||
hardware:
|
||||
h200: verified
|
||||
model:
|
||||
model_id: "IFM/Jais-3-0.9B"
|
||||
architecture: dense
|
||||
parameter_count: "1.08B"
|
||||
active_parameters: "1.08B"
|
||||
context_length: 131072
|
||||
base_args:
|
||||
- "--trust-remote-code"
|
||||
- "--chat-template-content-format"
|
||||
- "string"
|
||||
- "--dtype"
|
||||
- "bfloat16"
|
||||
- "--max-model-len"
|
||||
- "131072"
|
||||
- "--gpu-memory-utilization"
|
||||
- "0.85"
|
||||
base_env: {}
|
||||
features:
|
||||
reasoning:
|
||||
description: "Extract Jais-3 reasoning into message.reasoning"
|
||||
args: ["--reasoning-parser", "k2_horizon"]
|
||||
tool_calling:
|
||||
description: "Enable automatic tool choice with the K2-Horizon parser"
|
||||
args: ["--enable-auto-tool-choice", "--tool-call-parser", "k2_horizon"]
|
||||
variants:
|
||||
default:
|
||||
precision: bf16
|
||||
vram_minimum_gb: 3
|
||||
description: "Released BF16 checkpoint; the command starts at an 8K runtime context"
|
||||
compatible_strategies:
|
||||
- single_node_tp
|
||||
- multi_node_tp
|
||||
- multi_node_tp_pp
|
||||
hardware_overrides: {}
|
||||
guide: |
|
||||
## Overview
|
||||
|
||||
Jais-3-0.9B is a compact dense model in the IFM Jais-3
|
||||
family, with 1.08B stored parameters including embeddings and a 131,072-token
|
||||
context window.
|
||||
|
||||
Jais-3 is an open-weight IFM model family built for transparent
|
||||
foundation-model research, staged checkpoint analysis, and practical
|
||||
deployment. The series spans compact dense models such as Jais-3-0.9B
|
||||
for local experimentation, dense mid-size and large models for high-quality
|
||||
research workloads, and mixture-of-expert (MoE) models for higher-capacity
|
||||
serving.
|
||||
|
||||
## Launch Commands
|
||||
|
||||
Jais-3-0.9B's `config.json` already enables YaRN, so the full 131,072-token context needs no RoPE override.
|
||||
|
||||
```bash
|
||||
vllm serve IFM/Jais-3-0.9B \
|
||||
--trust-remote-code \
|
||||
--chat-template-content-format string \
|
||||
--dtype bfloat16 \
|
||||
--max-model-len 131072 \
|
||||
--gpu-memory-utilization 0.85 \
|
||||
--reasoning-parser k2_horizon \
|
||||
--enable-auto-tool-choice \
|
||||
--tool-call-parser k2_horizon
|
||||
```
|
||||
|
||||
Lower `--max-model-len` (e.g. 8192) to reduce KV-cache memory.
|
||||
|
||||
## Client usage
|
||||
|
||||
```python
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
api_key="EMPTY", base_url="http://localhost:8000/v1", timeout=3600
|
||||
)
|
||||
|
||||
resp = client.chat.completions.create(
|
||||
model="IFM/Jais-3-0.9B",
|
||||
messages=[{"role": "user", "content": "لدينا عددان مجموعهما 50 وحاصل ضربهما 600. ما هما العددان؟ اشرح خطوات الحل."}],
|
||||
temperature=1.0, top_p=0.95, max_tokens=2048,
|
||||
)
|
||||
print(resp.choices[0].message.content)
|
||||
```
|
||||
|
||||
### Thinking modes
|
||||
|
||||
The model supports selectable thinking through `chat_template_kwargs`, per
|
||||
request or server-wide via `--default-chat-template-kwargs`:
|
||||
|
||||
- `{"reasoning_effort": "high"}` — full thinking (default).
|
||||
- `{"reasoning_effort": "low"}` — no thinking.
|
||||
|
||||
## References
|
||||
|
||||
- [IFM/Jais-3-0.9B on Hugging Face](https://huggingface.co/IFM/Jais-3-0.9B)
|
||||
- [Jais 3 Collection](https://huggingface.co/collections/IFM/jais-3)
|
||||
@@ -0,0 +1,88 @@
|
||||
meta:
|
||||
title: "Jais-3-3.7B"
|
||||
slug: "jais-3-3.7b"
|
||||
provider: "IFM"
|
||||
description: "Dense model for long-context research, serving, and downstream adaptation"
|
||||
date_added: 2026-10-05
|
||||
date_updated: 2026-10-05
|
||||
tasks: [text]
|
||||
performance_headline: "5.06B stored parameters including embeddings, with up to 512K context"
|
||||
related_recipes: ["IFM/Jais-3-0.9B", "IFM/Jais-3-7B"]
|
||||
hardware:
|
||||
h200: verified
|
||||
model:
|
||||
model_id: "IFM/Jais-3-3.7B"
|
||||
architecture: dense
|
||||
parameter_count: "5.06B"
|
||||
active_parameters: "5.06B"
|
||||
context_length: 524288
|
||||
base_args: ["--trust-remote-code", "--chat-template-content-format", "string", "--dtype", "bfloat16"]
|
||||
base_env: {}
|
||||
features:
|
||||
reasoning:
|
||||
description: "Extract Jais-3 reasoning into message.reasoning"
|
||||
args: ["--reasoning-parser", "k2_horizon"]
|
||||
tool_calling:
|
||||
description: "Enable automatic tool choice with the K2-Horizon parser"
|
||||
args: ["--enable-auto-tool-choice", "--tool-call-parser", "k2_horizon"]
|
||||
variants:
|
||||
default:
|
||||
precision: bf16
|
||||
vram_minimum_gb: 13
|
||||
description: "BF16 serving runtime with a 131K context baseline"
|
||||
compatible_strategies: [single_node_tp, multi_node_tp, multi_node_tp_pp]
|
||||
hardware_overrides: {}
|
||||
guide: |
|
||||
## Overview
|
||||
|
||||
Jais-3-3.7B is a small dense model in the IFM Jais-3 family,
|
||||
with 5.06B stored parameters including embeddings and support for up to
|
||||
524,288 tokens of context.
|
||||
|
||||
Jais-3 is an open-weight IFM model family built for transparent
|
||||
foundation-model research, staged checkpoint analysis, and practical
|
||||
deployment. The series spans compact dense models for local experimentation,
|
||||
dense mid-size and large models for high-quality research workloads, and
|
||||
mixture-of-expert (MoE) models for higher-capacity serving.
|
||||
|
||||
## Launch Commands
|
||||
|
||||
```bash
|
||||
vllm serve IFM/Jais-3-3.7B \
|
||||
--trust-remote-code \
|
||||
--chat-template-content-format string \
|
||||
--dtype bfloat16 \
|
||||
--reasoning-parser k2_horizon \
|
||||
--enable-auto-tool-choice \
|
||||
--tool-call-parser k2_horizon
|
||||
```
|
||||
|
||||
## Client usage
|
||||
|
||||
```python
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
api_key="EMPTY", base_url="http://localhost:8000/v1", timeout=3600
|
||||
)
|
||||
|
||||
resp = client.chat.completions.create(
|
||||
model="IFM/Jais-3-3.7B",
|
||||
messages=[{"role": "user", "content": "لدينا عددان مجموعهما 50 وحاصل ضربهما 600. ما هما العددان؟ اشرح خطوات الحل."}],
|
||||
temperature=1.0, top_p=0.95, max_tokens=2048,
|
||||
)
|
||||
print(resp.choices[0].message.content)
|
||||
```
|
||||
|
||||
### Thinking modes
|
||||
|
||||
The model supports selectable thinking through `chat_template_kwargs`, per
|
||||
request or server-wide via `--default-chat-template-kwargs`:
|
||||
|
||||
- `{"reasoning_effort": "high"}` — full thinking (default).
|
||||
- `{"reasoning_effort": "low"}` — no thinking.
|
||||
|
||||
## References
|
||||
|
||||
- [IFM/Jais-3-3.7B on Hugging Face](https://huggingface.co/IFM/Jais-3-3.7B)
|
||||
- [Jais 3 Collection](https://huggingface.co/collections/IFM/jais-3)
|
||||
@@ -0,0 +1,93 @@
|
||||
meta:
|
||||
title: "Jais-3-32B"
|
||||
slug: "jais-3-32b"
|
||||
provider: "IFM"
|
||||
description: "Dense long-context model for reasoning research and high-capacity serving"
|
||||
date_added: 2026-10-05
|
||||
date_updated: 2026-10-05
|
||||
tasks: [text]
|
||||
performance_headline: "34.78B stored parameters including embeddings, with native vLLM execution and up to 512K context"
|
||||
related_recipes: ["IFM/Jais-3-7B", "IFM/Jais-3-MoVA-36B-A4B"]
|
||||
hardware:
|
||||
h200: verified
|
||||
model:
|
||||
model_id: "IFM/Jais-3-32B"
|
||||
architecture: dense
|
||||
parameter_count: "34.78B"
|
||||
active_parameters: "34.78B"
|
||||
context_length: 524288
|
||||
base_args: ["--trust-remote-code", "--chat-template-content-format", "string", "--dtype", "bfloat16"]
|
||||
base_env: {}
|
||||
features:
|
||||
reasoning:
|
||||
description: "Extract Jais-3 reasoning into message.reasoning"
|
||||
args: ["--reasoning-parser", "k2_horizon"]
|
||||
tool_calling:
|
||||
description: "Enable automatic tool choice with the K2-Horizon parser"
|
||||
args: ["--enable-auto-tool-choice", "--tool-call-parser", "k2_horizon"]
|
||||
variants:
|
||||
default:
|
||||
precision: bf16
|
||||
vram_minimum_gb: 84
|
||||
description: "BF16 serving runtime with native vLLM execution"
|
||||
compatible_strategies: [single_node_tp, multi_node_tp, multi_node_tp_pp]
|
||||
hardware_overrides: {}
|
||||
strategy_overrides:
|
||||
single_node_tp:
|
||||
tp: 2
|
||||
guide: |
|
||||
## Overview
|
||||
|
||||
Jais-3-32B is a large dense model in the IFM Jais-3 family,
|
||||
with 34.78B stored parameters including embeddings and support for up to
|
||||
524,288 tokens of context.
|
||||
|
||||
Jais-3 is an open-weight IFM model family built for transparent
|
||||
foundation-model research, staged checkpoint analysis, and practical
|
||||
deployment. The series spans compact dense models for local experimentation,
|
||||
dense mid-size and large models such as Jais-3-32B for high-quality
|
||||
research workloads, and mixture-of-expert (MoE) models for higher-capacity
|
||||
serving.
|
||||
|
||||
## Launch Commands
|
||||
|
||||
```bash
|
||||
vllm serve IFM/Jais-3-32B \
|
||||
--tensor-parallel-size 2 \
|
||||
--trust-remote-code \
|
||||
--chat-template-content-format string \
|
||||
--dtype bfloat16 \
|
||||
--reasoning-parser k2_horizon \
|
||||
--enable-auto-tool-choice \
|
||||
--tool-call-parser k2_horizon
|
||||
```
|
||||
|
||||
## Client usage
|
||||
|
||||
```python
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
api_key="EMPTY", base_url="http://localhost:8000/v1", timeout=3600
|
||||
)
|
||||
|
||||
resp = client.chat.completions.create(
|
||||
model="IFM/Jais-3-32B",
|
||||
messages=[{"role": "user", "content": "لدينا عددان مجموعهما 50 وحاصل ضربهما 600. ما هما العددان؟ اشرح خطوات الحل."}],
|
||||
temperature=1.0, top_p=0.95, max_tokens=2048,
|
||||
)
|
||||
print(resp.choices[0].message.content)
|
||||
```
|
||||
|
||||
### Thinking modes
|
||||
|
||||
The model supports selectable thinking through `chat_template_kwargs`, per
|
||||
request or server-wide via `--default-chat-template-kwargs`:
|
||||
|
||||
- `{"reasoning_effort": "high"}` — full thinking (default).
|
||||
- `{"reasoning_effort": "low"}` — no thinking.
|
||||
|
||||
## References
|
||||
|
||||
- [IFM/Jais-3-32B on Hugging Face](https://huggingface.co/IFM/Jais-3-32B)
|
||||
- [Jais 3 Collection](https://huggingface.co/collections/IFM/jais-3)
|
||||
@@ -0,0 +1,148 @@
|
||||
meta:
|
||||
title: "Jais-3-375B-A23B"
|
||||
slug: "jais-3-375b-a23b"
|
||||
provider: "IFM"
|
||||
description: "Frontier-scale sparse MoE model for long-context research and high-capacity serving"
|
||||
date_added: 2026-10-05
|
||||
date_updated: 2026-10-05
|
||||
tasks: [text]
|
||||
performance_headline: "379.17B stored parameters including embeddings / 26.67B active parameters per token"
|
||||
related_recipes: ["IFM/Jais-3-MoVA-36B-A4B"]
|
||||
hardware:
|
||||
h200: verified
|
||||
mi355x: verified
|
||||
model:
|
||||
model_id: "IFM/Jais-3-375B-A23B"
|
||||
architecture: moe
|
||||
parameter_count: "379.17B"
|
||||
active_parameters: "26.67B"
|
||||
context_length: 524288
|
||||
# --enable-expert-parallel is not here: in base_args it would reach the AMD
|
||||
# path too, which deliberately runs pure TP (see strategy_hardware). The TEP
|
||||
# strategies emit it themselves; multi_node_tp_pp does not, so it carries its
|
||||
# own copy in strategy_overrides below.
|
||||
base_args: ["--trust-remote-code", "--chat-template-content-format", "string", "--dtype", "bfloat16"]
|
||||
base_env: {}
|
||||
features:
|
||||
reasoning:
|
||||
description: "Extract Jais-3 reasoning into message.reasoning"
|
||||
args: ["--reasoning-parser", "k2_horizon"]
|
||||
tool_calling:
|
||||
description: "Enable automatic tool choice with the K2-Horizon parser"
|
||||
args: ["--enable-auto-tool-choice", "--tool-call-parser", "k2_horizon"]
|
||||
variants:
|
||||
default:
|
||||
precision: bf16
|
||||
vram_minimum_gb: 911
|
||||
description: "BF16 serving runtime with an eight-GPU tensor-parallel baseline"
|
||||
compatible_strategies: [single_node_tep, single_node_tp, multi_node_tep, multi_node_tp_pp]
|
||||
default_strategy: single_node_tep
|
||||
|
||||
# On AMD the expert-parallel dispatch is slower than replicating the experts
|
||||
# across a pure tensor-parallel shard, so the TEP strategies are opted out and
|
||||
# single-node TP is the offered path. multi_node_tp_pp is left available: EP
|
||||
# works there, it is just not the fast default at single-node scale.
|
||||
strategy_hardware:
|
||||
single_node_tep:
|
||||
amd: unsupported
|
||||
multi_node_tep:
|
||||
amd: unsupported
|
||||
hardware_overrides:
|
||||
amd:
|
||||
extra_args:
|
||||
- "--attention-backend"
|
||||
- "ROCM_AITER_UNIFIED_ATTN"
|
||||
- "--gpu-memory-utilization"
|
||||
- "0.9"
|
||||
extra_env:
|
||||
VLLM_ROCM_USE_AITER: "1"
|
||||
NCCL_MIN_NCHANNELS: "112"
|
||||
strategy_overrides:
|
||||
single_node_tep:
|
||||
tp: 8
|
||||
# multi_node_tp_pp emits no EP flag of its own, and this recipe has always
|
||||
# run it with expert parallel on. Restored here rather than in base_args so
|
||||
# it stays off the AMD single-node TP path.
|
||||
multi_node_tp_pp:
|
||||
extra_args: ["--enable-expert-parallel"]
|
||||
guide: |
|
||||
## Overview
|
||||
|
||||
Jais-3-375B-A23B is a frontier-scale mixture-of-expert (MoE) model in the IFM
|
||||
Jais-3 family, with 379.17B stored parameters including embeddings,
|
||||
26.67B active parameters per token, and support for up to 524,288 tokens of
|
||||
context.
|
||||
|
||||
Jais-3 is an open-weight IFM model family built for transparent
|
||||
foundation-model research, staged checkpoint analysis, and practical
|
||||
deployment. The series spans compact dense models for local experimentation,
|
||||
dense mid-size and large models for high-quality research workloads, and
|
||||
mixture-of-expert (MoE) models such as Jais-3-375B-A23B for frontier-scale
|
||||
research and higher-capacity serving.
|
||||
|
||||
## Launch Commands
|
||||
|
||||
```bash
|
||||
vllm serve IFM/Jais-3-375B-A23B \
|
||||
--tensor-parallel-size 8 \
|
||||
--trust-remote-code \
|
||||
--chat-template-content-format string \
|
||||
--dtype bfloat16 \
|
||||
--enable-expert-parallel \
|
||||
--reasoning-parser k2_horizon \
|
||||
--enable-auto-tool-choice \
|
||||
--tool-call-parser k2_horizon
|
||||
```
|
||||
|
||||
### AMD MI355X
|
||||
|
||||
Verified on an 8-GPU MI355X (gfx950) node. AITER supplies the fused MoE and
|
||||
the unified attention backend:
|
||||
|
||||
```bash
|
||||
export VLLM_ROCM_USE_AITER=1
|
||||
export NCCL_MIN_NCHANNELS=112
|
||||
|
||||
vllm serve IFM/Jais-3-375B-A23B \
|
||||
--tensor-parallel-size 8 \
|
||||
--trust-remote-code \
|
||||
--chat-template-content-format string \
|
||||
--dtype bfloat16 \
|
||||
--max-model-len 131072 \
|
||||
--gpu-memory-utilization 0.9 \
|
||||
--attention-backend ROCM_AITER_UNIFIED_ATTN \
|
||||
--generation-config auto \
|
||||
--reasoning-parser k2_horizon \
|
||||
--enable-auto-tool-choice \
|
||||
--tool-call-parser k2_horizon
|
||||
```
|
||||
|
||||
## Client usage
|
||||
|
||||
```python
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
api_key="EMPTY", base_url="http://localhost:8000/v1", timeout=3600
|
||||
)
|
||||
|
||||
resp = client.chat.completions.create(
|
||||
model="IFM/Jais-3-375B-A23B",
|
||||
messages=[{"role": "user", "content": "لدينا عددان مجموعهما 50 وحاصل ضربهما 600. ما هما العددان؟ اشرح خطوات الحل."}],
|
||||
temperature=1.0, top_p=0.95, max_tokens=2048,
|
||||
)
|
||||
print(resp.choices[0].message.content)
|
||||
```
|
||||
|
||||
### Thinking modes
|
||||
|
||||
The model supports selectable thinking through `chat_template_kwargs`, per
|
||||
request or server-wide via `--default-chat-template-kwargs`:
|
||||
|
||||
- `{"reasoning_effort": "high"}` — full thinking (default).
|
||||
- `{"reasoning_effort": "low"}` — no thinking.
|
||||
|
||||
## References
|
||||
|
||||
- [IFM/Jais-3-375B-A23B on Hugging Face](https://huggingface.co/IFM/Jais-3-375B-A23B)
|
||||
- [Jais 3 Collection](https://huggingface.co/collections/IFM/jais-3)
|
||||
@@ -0,0 +1,89 @@
|
||||
meta:
|
||||
title: "Jais-3-7B"
|
||||
slug: "jais-3-7b"
|
||||
provider: "IFM"
|
||||
description: "Dense model for long-context research, fine-tuning, and cost-conscious deployment"
|
||||
date_added: 2026-10-05
|
||||
date_updated: 2026-10-05
|
||||
tasks: [text]
|
||||
performance_headline: "9B stored parameters including embeddings, with up to 512K context"
|
||||
related_recipes: ["IFM/Jais-3-3.7B", "IFM/Jais-3-32B"]
|
||||
hardware:
|
||||
h200: verified
|
||||
model:
|
||||
model_id: "IFM/Jais-3-7B"
|
||||
architecture: dense
|
||||
parameter_count: "9B"
|
||||
active_parameters: "9B"
|
||||
context_length: 524288
|
||||
base_args: ["--trust-remote-code", "--chat-template-content-format", "string", "--dtype", "bfloat16"]
|
||||
base_env: {}
|
||||
features:
|
||||
reasoning:
|
||||
description: "Extract Jais-3 reasoning into message.reasoning"
|
||||
args: ["--reasoning-parser", "k2_horizon"]
|
||||
tool_calling:
|
||||
description: "Enable automatic tool choice with the K2-Horizon parser"
|
||||
args: ["--enable-auto-tool-choice", "--tool-call-parser", "k2_horizon"]
|
||||
variants:
|
||||
default:
|
||||
precision: bf16
|
||||
vram_minimum_gb: 22
|
||||
description: "BF16 serving runtime with a 131K context baseline"
|
||||
compatible_strategies: [single_node_tp, multi_node_tp, multi_node_tp_pp]
|
||||
hardware_overrides: {}
|
||||
guide: |
|
||||
## Overview
|
||||
|
||||
Jais-3-7B is a compact dense model in the IFM Jais-3 family,
|
||||
with 9B stored parameters including embeddings and support for up to 524,288
|
||||
tokens of context.
|
||||
|
||||
Jais-3 is an open-weight IFM model family built for transparent
|
||||
foundation-model research, staged checkpoint analysis, and practical
|
||||
deployment. The series spans compact dense models such as Jais-3-7B for
|
||||
local experimentation, dense mid-size and large models for high-quality
|
||||
research workloads, and mixture-of-expert (MoE) models for higher-capacity
|
||||
serving.
|
||||
|
||||
## Launch Commands
|
||||
|
||||
```bash
|
||||
vllm serve IFM/Jais-3-7B \
|
||||
--trust-remote-code \
|
||||
--chat-template-content-format string \
|
||||
--dtype bfloat16 \
|
||||
--reasoning-parser k2_horizon \
|
||||
--enable-auto-tool-choice \
|
||||
--tool-call-parser k2_horizon
|
||||
```
|
||||
|
||||
## Client usage
|
||||
|
||||
```python
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
api_key="EMPTY", base_url="http://localhost:8000/v1", timeout=3600
|
||||
)
|
||||
|
||||
resp = client.chat.completions.create(
|
||||
model="IFM/Jais-3-7B",
|
||||
messages=[{"role": "user", "content": "لدينا عددان مجموعهما 50 وحاصل ضربهما 600. ما هما العددان؟ اشرح خطوات الحل."}],
|
||||
temperature=1.0, top_p=0.95, max_tokens=2048,
|
||||
)
|
||||
print(resp.choices[0].message.content)
|
||||
```
|
||||
|
||||
### Thinking modes
|
||||
|
||||
The model supports selectable thinking through `chat_template_kwargs`, per
|
||||
request or server-wide via `--default-chat-template-kwargs`:
|
||||
|
||||
- `{"reasoning_effort": "high"}` — full thinking (default).
|
||||
- `{"reasoning_effort": "low"}` — no thinking.
|
||||
|
||||
## References
|
||||
|
||||
- [IFM/Jais-3-7B on Hugging Face](https://huggingface.co/IFM/Jais-3-7B)
|
||||
- [Jais 3 Collection](https://huggingface.co/collections/IFM/jais-3)
|
||||
@@ -0,0 +1,96 @@
|
||||
meta:
|
||||
title: "Jais-3-MoVA-36B-A4B"
|
||||
slug: "jais-3-mova-36b-a4b"
|
||||
provider: "IFM"
|
||||
description: "Sparse MoVA and MoE long-context model for research and production-style serving experiments"
|
||||
date_added: 2026-10-05
|
||||
date_updated: 2026-10-05
|
||||
tasks: [text]
|
||||
performance_headline: "37.44B stored parameters including embeddings / 5.95B active parameters per token"
|
||||
related_recipes: ["IFM/Jais-3-32B", "IFM/Jais-3-375B-A23B"]
|
||||
hardware:
|
||||
h200: verified
|
||||
model:
|
||||
model_id: "IFM/Jais-3-MoVA-36B-A4B"
|
||||
architecture: moe
|
||||
parameter_count: "37.44B"
|
||||
active_parameters: "5.95B"
|
||||
context_length: 524288
|
||||
base_args: ["--trust-remote-code", "--chat-template-content-format", "string", "--dtype", "bfloat16", "--enable-expert-parallel"]
|
||||
base_env: {}
|
||||
features:
|
||||
reasoning:
|
||||
description: "Extract Jais-3 reasoning into message.reasoning"
|
||||
args: ["--reasoning-parser", "k2_horizon"]
|
||||
tool_calling:
|
||||
description: "Enable automatic tool choice with the K2-Horizon parser"
|
||||
args: ["--enable-auto-tool-choice", "--tool-call-parser", "k2_horizon"]
|
||||
variants:
|
||||
default:
|
||||
precision: bf16
|
||||
vram_minimum_gb: 90
|
||||
description: "BF16 serving runtime with native vLLM execution"
|
||||
compatible_strategies: [single_node_tep, multi_node_tep, multi_node_tp_pp]
|
||||
default_strategy: single_node_tep
|
||||
hardware_overrides: {}
|
||||
strategy_overrides:
|
||||
single_node_tep:
|
||||
tp: 2
|
||||
guide: |
|
||||
## Overview
|
||||
|
||||
Jais-3-MoVA-36B-A4B is a mixture-of-expert (MoE) model in the IFM Jais-3
|
||||
family. It combines MoVA attention and MoE layers, with 37.44B stored parameters
|
||||
including embeddings, 5.95B active parameters per token, and support for up
|
||||
to 524,288 tokens of context.
|
||||
|
||||
Jais-3 is an open-weight IFM model family built for transparent
|
||||
foundation-model research, staged checkpoint analysis, and practical
|
||||
deployment. The series spans compact dense models for local experimentation,
|
||||
dense mid-size and large models for high-quality research workloads, and
|
||||
mixture-of-expert (MoE) models such as Jais-3-MoVA-36B-A4B for higher-capacity
|
||||
serving.
|
||||
|
||||
## Launch Commands
|
||||
|
||||
```bash
|
||||
vllm serve IFM/Jais-3-MoVA-36B-A4B \
|
||||
--tensor-parallel-size 2 \
|
||||
--trust-remote-code \
|
||||
--chat-template-content-format string \
|
||||
--dtype bfloat16 \
|
||||
--enable-expert-parallel \
|
||||
--reasoning-parser k2_horizon \
|
||||
--enable-auto-tool-choice \
|
||||
--tool-call-parser k2_horizon
|
||||
```
|
||||
|
||||
## Client usage
|
||||
|
||||
```python
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
api_key="EMPTY", base_url="http://localhost:8000/v1", timeout=3600
|
||||
)
|
||||
|
||||
resp = client.chat.completions.create(
|
||||
model="IFM/Jais-3-MoVA-36B-A4B",
|
||||
messages=[{"role": "user", "content": "لدينا عددان مجموعهما 50 وحاصل ضربهما 600. ما هما العددان؟ اشرح خطوات الحل."}],
|
||||
temperature=1.0, top_p=0.95, max_tokens=2048,
|
||||
)
|
||||
print(resp.choices[0].message.content)
|
||||
```
|
||||
|
||||
### Thinking modes
|
||||
|
||||
The model supports selectable thinking through `chat_template_kwargs`, per
|
||||
request or server-wide via `--default-chat-template-kwargs`:
|
||||
|
||||
- `{"reasoning_effort": "high"}` — full thinking (default).
|
||||
- `{"reasoning_effort": "low"}` — no thinking.
|
||||
|
||||
## References
|
||||
|
||||
- [IFM/Jais-3-MoVA-36B-A4B on Hugging Face](https://huggingface.co/IFM/Jais-3-MoVA-36B-A4B)
|
||||
- [Jais 3 Collection](https://huggingface.co/collections/IFM/jais-3)
|
||||
@@ -0,0 +1,110 @@
|
||||
meta:
|
||||
title: "K2-Horizon-0.9B"
|
||||
slug: "k2-horizon-0.9b"
|
||||
provider: "IFM"
|
||||
description: "Compact dense model for mathematics, code, instruction following, and STEM tasks"
|
||||
date_added: 2026-08-31
|
||||
date_updated: 2026-08-31
|
||||
tasks:
|
||||
- text
|
||||
performance_headline: "1.08B stored parameters including embeddings, with 131K context and native K2-Horizon parsing"
|
||||
related_recipes:
|
||||
- "IFM/K2-Horizon-3.7B"
|
||||
hardware:
|
||||
h200: verified
|
||||
model:
|
||||
model_id: "IFM/K2-Horizon-0.9B"
|
||||
architecture: dense
|
||||
parameter_count: "1.08B"
|
||||
active_parameters: "1.08B"
|
||||
context_length: 131072
|
||||
base_args:
|
||||
- "--trust-remote-code"
|
||||
- "--chat-template-content-format"
|
||||
- "string"
|
||||
- "--dtype"
|
||||
- "bfloat16"
|
||||
- "--max-model-len"
|
||||
- "131072"
|
||||
- "--gpu-memory-utilization"
|
||||
- "0.85"
|
||||
base_env: {}
|
||||
features:
|
||||
reasoning:
|
||||
description: "Extract K2-Horizon reasoning into message.reasoning"
|
||||
args: ["--reasoning-parser", "k2_horizon"]
|
||||
tool_calling:
|
||||
description: "Enable automatic tool choice with the K2-Horizon parser"
|
||||
args: ["--enable-auto-tool-choice", "--tool-call-parser", "k2_horizon"]
|
||||
variants:
|
||||
default:
|
||||
precision: bf16
|
||||
vram_minimum_gb: 3
|
||||
description: "Released BF16 checkpoint; the command starts at an 8K runtime context"
|
||||
compatible_strategies:
|
||||
- single_node_tp
|
||||
- multi_node_tp
|
||||
- multi_node_tp_pp
|
||||
hardware_overrides: {}
|
||||
guide: |
|
||||
## Overview
|
||||
|
||||
K2-Horizon-0.9B is a compact dense model in the IFM K2-Horizon
|
||||
family, with 1.08B stored parameters including embeddings and a 131,072-token
|
||||
context window.
|
||||
|
||||
K2-Horizon is an open-weight IFM model family built for transparent
|
||||
foundation-model research, staged checkpoint analysis, and practical
|
||||
deployment. The series spans compact dense models such as K2-Horizon-0.9B
|
||||
for local experimentation, dense mid-size and large models for high-quality
|
||||
research workloads, and mixture-of-expert (MoE) models for higher-capacity
|
||||
serving.
|
||||
|
||||
## Launch Commands
|
||||
|
||||
K2-Horizon-0.9B's `config.json` already enables YaRN, so the full 131,072-token context needs no RoPE override.
|
||||
|
||||
```bash
|
||||
vllm serve IFM/K2-Horizon-0.9B \
|
||||
--trust-remote-code \
|
||||
--chat-template-content-format string \
|
||||
--dtype bfloat16 \
|
||||
--max-model-len 131072 \
|
||||
--gpu-memory-utilization 0.85 \
|
||||
--reasoning-parser k2_horizon \
|
||||
--enable-auto-tool-choice \
|
||||
--tool-call-parser k2_horizon
|
||||
```
|
||||
|
||||
Lower `--max-model-len` (e.g. 8192) to reduce KV-cache memory.
|
||||
|
||||
## Client usage
|
||||
|
||||
```python
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
api_key="EMPTY", base_url="http://localhost:8000/v1", timeout=3600
|
||||
)
|
||||
|
||||
resp = client.chat.completions.create(
|
||||
model="IFM/K2-Horizon-0.9B",
|
||||
messages=[{"role": "user", "content": "Give me three primes above 100."}],
|
||||
temperature=1.0, top_p=0.95, max_tokens=2048,
|
||||
)
|
||||
print(resp.choices[0].message.content)
|
||||
```
|
||||
|
||||
### Thinking modes
|
||||
|
||||
The model supports selectable thinking through `chat_template_kwargs`, per
|
||||
request or server-wide via `--default-chat-template-kwargs`:
|
||||
|
||||
- `{"reasoning_effort": "high"}` — full thinking (default).
|
||||
- `{"reasoning_effort": "medium"}` — faster thinking.
|
||||
- `{"reasoning_effort": "low"}` — fastest thinking.
|
||||
|
||||
## References
|
||||
|
||||
- [IFM/K2-Horizon-0.9B on Hugging Face](https://huggingface.co/IFM/K2-Horizon-0.9B)
|
||||
- [K2 Horizon Collection](https://huggingface.co/collections/IFM/k2-horizon)
|
||||
@@ -0,0 +1,89 @@
|
||||
meta:
|
||||
title: "K2-Horizon-3.7B"
|
||||
slug: "k2-horizon-3.7b"
|
||||
provider: "IFM"
|
||||
description: "Dense model for long-context research, serving, and downstream adaptation"
|
||||
date_added: 2026-08-31
|
||||
date_updated: 2026-09-02
|
||||
tasks: [text]
|
||||
performance_headline: "5.06B stored parameters including embeddings, with up to 512K context"
|
||||
related_recipes: ["IFM/K2-Horizon-0.9B", "IFM/K2-Horizon-7B"]
|
||||
hardware:
|
||||
h200: verified
|
||||
model:
|
||||
model_id: "IFM/K2-Horizon-3.7B"
|
||||
architecture: dense
|
||||
parameter_count: "5.06B"
|
||||
active_parameters: "5.06B"
|
||||
context_length: 524288
|
||||
base_args: ["--trust-remote-code", "--chat-template-content-format", "string", "--dtype", "bfloat16"]
|
||||
base_env: {}
|
||||
features:
|
||||
reasoning:
|
||||
description: "Extract K2-Horizon reasoning into message.reasoning"
|
||||
args: ["--reasoning-parser", "k2_horizon"]
|
||||
tool_calling:
|
||||
description: "Enable automatic tool choice with the K2-Horizon parser"
|
||||
args: ["--enable-auto-tool-choice", "--tool-call-parser", "k2_horizon"]
|
||||
variants:
|
||||
default:
|
||||
precision: bf16
|
||||
vram_minimum_gb: 13
|
||||
description: "BF16 serving runtime with a 131K context baseline"
|
||||
compatible_strategies: [single_node_tp, multi_node_tp, multi_node_tp_pp]
|
||||
hardware_overrides: {}
|
||||
guide: |
|
||||
## Overview
|
||||
|
||||
K2-Horizon-3.7B is a small dense model in the IFM K2-Horizon family,
|
||||
with 5.06B stored parameters including embeddings and support for up to
|
||||
524,288 tokens of context.
|
||||
|
||||
K2-Horizon is an open-weight IFM model family built for transparent
|
||||
foundation-model research, staged checkpoint analysis, and practical
|
||||
deployment. The series spans compact dense models for local experimentation,
|
||||
dense mid-size and large models for high-quality research workloads, and
|
||||
mixture-of-expert (MoE) models for higher-capacity serving.
|
||||
|
||||
## Launch Commands
|
||||
|
||||
```bash
|
||||
vllm serve IFM/K2-Horizon-3.7B \
|
||||
--trust-remote-code \
|
||||
--chat-template-content-format string \
|
||||
--dtype bfloat16 \
|
||||
--reasoning-parser k2_horizon \
|
||||
--enable-auto-tool-choice \
|
||||
--tool-call-parser k2_horizon
|
||||
```
|
||||
|
||||
## Client usage
|
||||
|
||||
```python
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
api_key="EMPTY", base_url="http://localhost:8000/v1", timeout=3600
|
||||
)
|
||||
|
||||
resp = client.chat.completions.create(
|
||||
model="IFM/K2-Horizon-3.7B",
|
||||
messages=[{"role": "user", "content": "Give me three primes above 100."}],
|
||||
temperature=1.0, top_p=0.95, max_tokens=2048,
|
||||
)
|
||||
print(resp.choices[0].message.content)
|
||||
```
|
||||
|
||||
### Thinking modes
|
||||
|
||||
The model supports selectable thinking through `chat_template_kwargs`, per
|
||||
request or server-wide via `--default-chat-template-kwargs`:
|
||||
|
||||
- `{"reasoning_effort": "high"}` — full thinking (default).
|
||||
- `{"reasoning_effort": "medium"}` — faster thinking.
|
||||
- `{"reasoning_effort": "low"}` — fastest thinking.
|
||||
|
||||
## References
|
||||
|
||||
- [IFM/K2-Horizon-3.7B on Hugging Face](https://huggingface.co/IFM/K2-Horizon-3.7B)
|
||||
- [K2 Horizon Collection](https://huggingface.co/collections/IFM/k2-horizon)
|
||||
@@ -0,0 +1,94 @@
|
||||
meta:
|
||||
title: "K2-Horizon-32B"
|
||||
slug: "k2-horizon-32b"
|
||||
provider: "IFM"
|
||||
description: "Dense long-context model for reasoning research and high-capacity serving"
|
||||
date_added: 2026-08-31
|
||||
date_updated: 2026-09-02
|
||||
tasks: [text]
|
||||
performance_headline: "34.78B stored parameters including embeddings, with native vLLM execution and up to 512K context"
|
||||
related_recipes: ["IFM/K2-Horizon-7B", "IFM/K2-Horizon-MoVA-36B-A4B"]
|
||||
hardware:
|
||||
h200: verified
|
||||
model:
|
||||
model_id: "IFM/K2-Horizon-32B"
|
||||
architecture: dense
|
||||
parameter_count: "34.78B"
|
||||
active_parameters: "34.78B"
|
||||
context_length: 524288
|
||||
base_args: ["--trust-remote-code", "--chat-template-content-format", "string", "--dtype", "bfloat16"]
|
||||
base_env: {}
|
||||
features:
|
||||
reasoning:
|
||||
description: "Extract K2-Horizon reasoning into message.reasoning"
|
||||
args: ["--reasoning-parser", "k2_horizon"]
|
||||
tool_calling:
|
||||
description: "Enable automatic tool choice with the K2-Horizon parser"
|
||||
args: ["--enable-auto-tool-choice", "--tool-call-parser", "k2_horizon"]
|
||||
variants:
|
||||
default:
|
||||
precision: bf16
|
||||
vram_minimum_gb: 84
|
||||
description: "BF16 serving runtime with native vLLM execution"
|
||||
compatible_strategies: [single_node_tp, multi_node_tp, multi_node_tp_pp]
|
||||
hardware_overrides: {}
|
||||
strategy_overrides:
|
||||
single_node_tp:
|
||||
tp: 2
|
||||
guide: |
|
||||
## Overview
|
||||
|
||||
K2-Horizon-32B is a large dense model in the IFM K2-Horizon family,
|
||||
with 34.78B stored parameters including embeddings and support for up to
|
||||
524,288 tokens of context.
|
||||
|
||||
K2-Horizon is an open-weight IFM model family built for transparent
|
||||
foundation-model research, staged checkpoint analysis, and practical
|
||||
deployment. The series spans compact dense models for local experimentation,
|
||||
dense mid-size and large models such as K2-Horizon-32B for high-quality
|
||||
research workloads, and mixture-of-expert (MoE) models for higher-capacity
|
||||
serving.
|
||||
|
||||
## Launch Commands
|
||||
|
||||
```bash
|
||||
vllm serve IFM/K2-Horizon-32B \
|
||||
--tensor-parallel-size 2 \
|
||||
--trust-remote-code \
|
||||
--chat-template-content-format string \
|
||||
--dtype bfloat16 \
|
||||
--reasoning-parser k2_horizon \
|
||||
--enable-auto-tool-choice \
|
||||
--tool-call-parser k2_horizon
|
||||
```
|
||||
|
||||
## Client usage
|
||||
|
||||
```python
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
api_key="EMPTY", base_url="http://localhost:8000/v1", timeout=3600
|
||||
)
|
||||
|
||||
resp = client.chat.completions.create(
|
||||
model="IFM/K2-Horizon-32B",
|
||||
messages=[{"role": "user", "content": "Give me three primes above 100."}],
|
||||
temperature=1.0, top_p=0.95, max_tokens=2048,
|
||||
)
|
||||
print(resp.choices[0].message.content)
|
||||
```
|
||||
|
||||
### Thinking modes
|
||||
|
||||
The model supports selectable thinking through `chat_template_kwargs`, per
|
||||
request or server-wide via `--default-chat-template-kwargs`:
|
||||
|
||||
- `{"reasoning_effort": "high"}` — full thinking (default).
|
||||
- `{"reasoning_effort": "medium"}` — faster thinking.
|
||||
- `{"reasoning_effort": "low"}` — fastest thinking.
|
||||
|
||||
## References
|
||||
|
||||
- [IFM/K2-Horizon-32B on Hugging Face](https://huggingface.co/IFM/K2-Horizon-32B)
|
||||
- [K2 Horizon Collection](https://huggingface.co/collections/IFM/k2-horizon)
|
||||
@@ -0,0 +1,149 @@
|
||||
meta:
|
||||
title: "K2-Horizon-375B-A23B"
|
||||
slug: "k2-horizon-375b-a23b"
|
||||
provider: "IFM"
|
||||
description: "Frontier-scale sparse MoE model for long-context research and high-capacity serving"
|
||||
date_added: 2026-08-31
|
||||
date_updated: 2026-09-08
|
||||
tasks: [text]
|
||||
performance_headline: "379.17B stored parameters including embeddings / 26.67B active parameters per token"
|
||||
related_recipes: ["IFM/K2-Horizon-MoVA-36B-A4B"]
|
||||
hardware:
|
||||
h200: verified
|
||||
mi355x: verified
|
||||
model:
|
||||
model_id: "IFM/K2-Horizon-375B-A23B"
|
||||
architecture: moe
|
||||
parameter_count: "379.17B"
|
||||
active_parameters: "26.67B"
|
||||
context_length: 524288
|
||||
# --enable-expert-parallel is not here: in base_args it would reach the AMD
|
||||
# path too, which deliberately runs pure TP (see strategy_hardware). The TEP
|
||||
# strategies emit it themselves; multi_node_tp_pp does not, so it carries its
|
||||
# own copy in strategy_overrides below.
|
||||
base_args: ["--trust-remote-code", "--chat-template-content-format", "string", "--dtype", "bfloat16"]
|
||||
base_env: {}
|
||||
features:
|
||||
reasoning:
|
||||
description: "Extract K2-Horizon reasoning into message.reasoning"
|
||||
args: ["--reasoning-parser", "k2_horizon"]
|
||||
tool_calling:
|
||||
description: "Enable automatic tool choice with the K2-Horizon parser"
|
||||
args: ["--enable-auto-tool-choice", "--tool-call-parser", "k2_horizon"]
|
||||
variants:
|
||||
default:
|
||||
precision: bf16
|
||||
vram_minimum_gb: 911
|
||||
description: "BF16 serving runtime with an eight-GPU tensor-parallel baseline"
|
||||
compatible_strategies: [single_node_tep, single_node_tp, multi_node_tep, multi_node_tp_pp]
|
||||
default_strategy: single_node_tep
|
||||
|
||||
# On AMD the expert-parallel dispatch is slower than replicating the experts
|
||||
# across a pure tensor-parallel shard, so the TEP strategies are opted out and
|
||||
# single-node TP is the offered path. multi_node_tp_pp is left available: EP
|
||||
# works there, it is just not the fast default at single-node scale.
|
||||
strategy_hardware:
|
||||
single_node_tep:
|
||||
amd: unsupported
|
||||
multi_node_tep:
|
||||
amd: unsupported
|
||||
hardware_overrides:
|
||||
amd:
|
||||
extra_args:
|
||||
- "--attention-backend"
|
||||
- "ROCM_AITER_UNIFIED_ATTN"
|
||||
- "--gpu-memory-utilization"
|
||||
- "0.9"
|
||||
extra_env:
|
||||
VLLM_ROCM_USE_AITER: "1"
|
||||
NCCL_MIN_NCHANNELS: "112"
|
||||
strategy_overrides:
|
||||
single_node_tep:
|
||||
tp: 8
|
||||
# multi_node_tp_pp emits no EP flag of its own, and this recipe has always
|
||||
# run it with expert parallel on. Restored here rather than in base_args so
|
||||
# it stays off the AMD single-node TP path.
|
||||
multi_node_tp_pp:
|
||||
extra_args: ["--enable-expert-parallel"]
|
||||
guide: |
|
||||
## Overview
|
||||
|
||||
K2-Horizon-375B-A23B is a frontier-scale mixture-of-expert (MoE) model in the IFM
|
||||
K2-Horizon family, with 379.17B stored parameters including embeddings,
|
||||
26.67B active parameters per token, and support for up to 524,288 tokens of
|
||||
context.
|
||||
|
||||
K2-Horizon is an open-weight IFM model family built for transparent
|
||||
foundation-model research, staged checkpoint analysis, and practical
|
||||
deployment. The series spans compact dense models for local experimentation,
|
||||
dense mid-size and large models for high-quality research workloads, and
|
||||
mixture-of-expert (MoE) models such as K2-Horizon-375B-A23B for frontier-scale
|
||||
research and higher-capacity serving.
|
||||
|
||||
## Launch Commands
|
||||
|
||||
```bash
|
||||
vllm serve IFM/K2-Horizon-375B-A23B \
|
||||
--tensor-parallel-size 8 \
|
||||
--trust-remote-code \
|
||||
--chat-template-content-format string \
|
||||
--dtype bfloat16 \
|
||||
--enable-expert-parallel \
|
||||
--reasoning-parser k2_horizon \
|
||||
--enable-auto-tool-choice \
|
||||
--tool-call-parser k2_horizon
|
||||
```
|
||||
|
||||
### AMD MI355X
|
||||
|
||||
Verified on an 8-GPU MI355X (gfx950) node. AITER supplies the fused MoE and
|
||||
the unified attention backend:
|
||||
|
||||
```bash
|
||||
export VLLM_ROCM_USE_AITER=1
|
||||
export NCCL_MIN_NCHANNELS=112
|
||||
|
||||
vllm serve IFM/K2-Horizon-375B-A23B \
|
||||
--tensor-parallel-size 8 \
|
||||
--trust-remote-code \
|
||||
--chat-template-content-format string \
|
||||
--dtype bfloat16 \
|
||||
--max-model-len 131072 \
|
||||
--gpu-memory-utilization 0.9 \
|
||||
--attention-backend ROCM_AITER_UNIFIED_ATTN \
|
||||
--generation-config auto \
|
||||
--reasoning-parser k2_horizon \
|
||||
--enable-auto-tool-choice \
|
||||
--tool-call-parser k2_horizon
|
||||
```
|
||||
|
||||
## Client usage
|
||||
|
||||
```python
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
api_key="EMPTY", base_url="http://localhost:8000/v1", timeout=3600
|
||||
)
|
||||
|
||||
resp = client.chat.completions.create(
|
||||
model="IFM/K2-Horizon-375B-A23B",
|
||||
messages=[{"role": "user", "content": "Give me three primes above 100."}],
|
||||
temperature=1.0, top_p=0.95, max_tokens=2048,
|
||||
)
|
||||
print(resp.choices[0].message.content)
|
||||
```
|
||||
|
||||
### Thinking modes
|
||||
|
||||
The model supports selectable thinking through `chat_template_kwargs`, per
|
||||
request or server-wide via `--default-chat-template-kwargs`:
|
||||
|
||||
- `{"reasoning_effort": "high"}` — full thinking (default).
|
||||
- `{"reasoning_effort": "medium"}` — faster thinking.
|
||||
- `{"reasoning_effort": "low"}` — fastest thinking.
|
||||
|
||||
## References
|
||||
|
||||
- [IFM/K2-Horizon-375B-A23B on Hugging Face](https://huggingface.co/IFM/K2-Horizon-375B-A23B)
|
||||
- [K2 Horizon Collection](https://huggingface.co/collections/IFM/k2-horizon)
|
||||
@@ -0,0 +1,90 @@
|
||||
meta:
|
||||
title: "K2-Horizon-7B"
|
||||
slug: "k2-horizon-7b"
|
||||
provider: "IFM"
|
||||
description: "Dense model for long-context research, fine-tuning, and cost-conscious deployment"
|
||||
date_added: 2026-08-31
|
||||
date_updated: 2026-09-02
|
||||
tasks: [text]
|
||||
performance_headline: "9B stored parameters including embeddings, with up to 512K context"
|
||||
related_recipes: ["IFM/K2-Horizon-3.7B", "IFM/K2-Horizon-32B"]
|
||||
hardware:
|
||||
h200: verified
|
||||
model:
|
||||
model_id: "IFM/K2-Horizon-7B"
|
||||
architecture: dense
|
||||
parameter_count: "9B"
|
||||
active_parameters: "9B"
|
||||
context_length: 524288
|
||||
base_args: ["--trust-remote-code", "--chat-template-content-format", "string", "--dtype", "bfloat16"]
|
||||
base_env: {}
|
||||
features:
|
||||
reasoning:
|
||||
description: "Extract K2-Horizon reasoning into message.reasoning"
|
||||
args: ["--reasoning-parser", "k2_horizon"]
|
||||
tool_calling:
|
||||
description: "Enable automatic tool choice with the K2-Horizon parser"
|
||||
args: ["--enable-auto-tool-choice", "--tool-call-parser", "k2_horizon"]
|
||||
variants:
|
||||
default:
|
||||
precision: bf16
|
||||
vram_minimum_gb: 22
|
||||
description: "BF16 serving runtime with a 131K context baseline"
|
||||
compatible_strategies: [single_node_tp, multi_node_tp, multi_node_tp_pp]
|
||||
hardware_overrides: {}
|
||||
guide: |
|
||||
## Overview
|
||||
|
||||
K2-Horizon-7B is a compact dense model in the IFM K2-Horizon family,
|
||||
with 9B stored parameters including embeddings and support for up to 524,288
|
||||
tokens of context.
|
||||
|
||||
K2-Horizon is an open-weight IFM model family built for transparent
|
||||
foundation-model research, staged checkpoint analysis, and practical
|
||||
deployment. The series spans compact dense models such as K2-Horizon-7B for
|
||||
local experimentation, dense mid-size and large models for high-quality
|
||||
research workloads, and mixture-of-expert (MoE) models for higher-capacity
|
||||
serving.
|
||||
|
||||
## Launch Commands
|
||||
|
||||
```bash
|
||||
vllm serve IFM/K2-Horizon-7B \
|
||||
--trust-remote-code \
|
||||
--chat-template-content-format string \
|
||||
--dtype bfloat16 \
|
||||
--reasoning-parser k2_horizon \
|
||||
--enable-auto-tool-choice \
|
||||
--tool-call-parser k2_horizon
|
||||
```
|
||||
|
||||
## Client usage
|
||||
|
||||
```python
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
api_key="EMPTY", base_url="http://localhost:8000/v1", timeout=3600
|
||||
)
|
||||
|
||||
resp = client.chat.completions.create(
|
||||
model="IFM/K2-Horizon-7B",
|
||||
messages=[{"role": "user", "content": "Give me three primes above 100."}],
|
||||
temperature=1.0, top_p=0.95, max_tokens=2048,
|
||||
)
|
||||
print(resp.choices[0].message.content)
|
||||
```
|
||||
|
||||
### Thinking modes
|
||||
|
||||
The model supports selectable thinking through `chat_template_kwargs`, per
|
||||
request or server-wide via `--default-chat-template-kwargs`:
|
||||
|
||||
- `{"reasoning_effort": "high"}` — full thinking (default).
|
||||
- `{"reasoning_effort": "medium"}` — faster thinking.
|
||||
- `{"reasoning_effort": "low"}` — fastest thinking.
|
||||
|
||||
## References
|
||||
|
||||
- [IFM/K2-Horizon-7B on Hugging Face](https://huggingface.co/IFM/K2-Horizon-7B)
|
||||
- [K2 Horizon Collection](https://huggingface.co/collections/IFM/k2-horizon)
|
||||
@@ -0,0 +1,97 @@
|
||||
meta:
|
||||
title: "K2-Horizon-MoVA-36B-A4B"
|
||||
slug: "k2-horizon-mova-36b-a4b"
|
||||
provider: "IFM"
|
||||
description: "Sparse MoVA and MoE long-context model for research and production-style serving experiments"
|
||||
date_added: 2026-08-31
|
||||
date_updated: 2026-09-03
|
||||
tasks: [text]
|
||||
performance_headline: "37.44B stored parameters including embeddings / 5.95B active parameters per token"
|
||||
related_recipes: ["IFM/K2-Horizon-32B", "IFM/K2-Horizon-375B-A23B"]
|
||||
hardware:
|
||||
h200: verified
|
||||
model:
|
||||
model_id: "IFM/K2-Horizon-MoVA-36B-A4B"
|
||||
architecture: moe
|
||||
parameter_count: "37.44B"
|
||||
active_parameters: "5.95B"
|
||||
context_length: 524288
|
||||
base_args: ["--trust-remote-code", "--chat-template-content-format", "string", "--dtype", "bfloat16", "--enable-expert-parallel"]
|
||||
base_env: {}
|
||||
features:
|
||||
reasoning:
|
||||
description: "Extract K2-Horizon reasoning into message.reasoning"
|
||||
args: ["--reasoning-parser", "k2_horizon"]
|
||||
tool_calling:
|
||||
description: "Enable automatic tool choice with the K2-Horizon parser"
|
||||
args: ["--enable-auto-tool-choice", "--tool-call-parser", "k2_horizon"]
|
||||
variants:
|
||||
default:
|
||||
precision: bf16
|
||||
vram_minimum_gb: 90
|
||||
description: "BF16 serving runtime with native vLLM execution"
|
||||
compatible_strategies: [single_node_tep, multi_node_tep, multi_node_tp_pp]
|
||||
default_strategy: single_node_tep
|
||||
hardware_overrides: {}
|
||||
strategy_overrides:
|
||||
single_node_tep:
|
||||
tp: 2
|
||||
guide: |
|
||||
## Overview
|
||||
|
||||
K2-Horizon-MoVA-36B-A4B is a mixture-of-expert (MoE) model in the IFM K2-Horizon
|
||||
family. It combines MoVA attention and MoE layers, with 37.44B stored parameters
|
||||
including embeddings, 5.95B active parameters per token, and support for up
|
||||
to 524,288 tokens of context.
|
||||
|
||||
K2-Horizon is an open-weight IFM model family built for transparent
|
||||
foundation-model research, staged checkpoint analysis, and practical
|
||||
deployment. The series spans compact dense models for local experimentation,
|
||||
dense mid-size and large models for high-quality research workloads, and
|
||||
mixture-of-expert (MoE) models such as K2-Horizon-MoVA-36B-A4B for higher-capacity
|
||||
serving.
|
||||
|
||||
## Launch Commands
|
||||
|
||||
```bash
|
||||
vllm serve IFM/K2-Horizon-MoVA-36B-A4B \
|
||||
--tensor-parallel-size 2 \
|
||||
--trust-remote-code \
|
||||
--chat-template-content-format string \
|
||||
--dtype bfloat16 \
|
||||
--enable-expert-parallel \
|
||||
--reasoning-parser k2_horizon \
|
||||
--enable-auto-tool-choice \
|
||||
--tool-call-parser k2_horizon
|
||||
```
|
||||
|
||||
## Client usage
|
||||
|
||||
```python
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
api_key="EMPTY", base_url="http://localhost:8000/v1", timeout=3600
|
||||
)
|
||||
|
||||
resp = client.chat.completions.create(
|
||||
model="IFM/K2-Horizon-MoVA-36B-A4B",
|
||||
messages=[{"role": "user", "content": "Give me three primes above 100."}],
|
||||
temperature=1.0, top_p=0.95, max_tokens=2048,
|
||||
)
|
||||
print(resp.choices[0].message.content)
|
||||
```
|
||||
|
||||
### Thinking modes
|
||||
|
||||
The model supports selectable thinking through `chat_template_kwargs`, per
|
||||
request or server-wide via `--default-chat-template-kwargs`:
|
||||
|
||||
- `{"reasoning_effort": "high"}` — full thinking (default).
|
||||
- `{"reasoning_effort": "medium"}` — faster thinking.
|
||||
- `{"reasoning_effort": "low"}` — fastest thinking.
|
||||
|
||||
## References
|
||||
|
||||
- [IFM/K2-Horizon-MoVA-36B-A4B on Hugging Face](https://huggingface.co/IFM/K2-Horizon-MoVA-36B-A4B)
|
||||
- [K2 Horizon Collection](https://huggingface.co/collections/IFM/k2-horizon)
|
||||
Reference in New Issue
Block a user