[Doc] Host model recipes in recipes/ (#61302)
Signed-off-by: esmeetu <jasonailu87@gmail.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 5.5
parent
f98ae6a9fd
commit
b515648641
@@ -0,0 +1,139 @@
|
||||
meta:
|
||||
title: "Laguna M.1"
|
||||
slug: "laguna-m.1"
|
||||
provider: "Poolside"
|
||||
description: "Poolside's 225B total / 23B activated MoE coding model with global attention, native interleaved reasoning, and 256K context — designed for agentic coding and long-horizon work."
|
||||
date_added: 2026-06-18
|
||||
date_updated: 2026-06-18
|
||||
tasks:
|
||||
- text
|
||||
performance_headline: "225B/23B-A MoE for agentic coding — 74.6% SWE-bench Verified"
|
||||
related_recipes:
|
||||
- "poolside/Laguna-XS.2"
|
||||
|
||||
model:
|
||||
model_id: "poolside/Laguna-M.1"
|
||||
min_vllm_version: "0.21.0"
|
||||
architecture: moe
|
||||
parameter_count: "225B"
|
||||
active_parameters: "23B"
|
||||
context_length: 262144
|
||||
base_args: []
|
||||
base_env: {}
|
||||
|
||||
features:
|
||||
tool_calling:
|
||||
description: "Enable automatic tool choice with Poolside's tool-call parser"
|
||||
args:
|
||||
- "--enable-auto-tool-choice"
|
||||
- "--tool-call-parser"
|
||||
- "poolside_v1"
|
||||
reasoning:
|
||||
description: "Enable interleaved thinking (on by default) with Poolside's reasoning parser"
|
||||
args:
|
||||
- "--reasoning-parser"
|
||||
- "poolside_v1"
|
||||
- "--default-chat-template-kwargs"
|
||||
- '{"enable_thinking": true}'
|
||||
|
||||
opt_in_features: []
|
||||
|
||||
variants:
|
||||
default:
|
||||
precision: bf16
|
||||
vram_minimum_gb: 540
|
||||
description: "BF16 weights (225B) — fits on a single 8×H200/B200 node"
|
||||
fp8:
|
||||
model_id: "poolside/Laguna-M.1-FP8"
|
||||
precision: fp8
|
||||
vram_minimum_gb: 270
|
||||
description: "Block-FP8 weights + FP8 KV cache — halves VRAM, runs on a single 4×H200/B200 node"
|
||||
nvfp4:
|
||||
model_id: "poolside/Laguna-M.1-NVFP4"
|
||||
precision: nvfp4
|
||||
vram_minimum_gb: 135
|
||||
description: "NVFP4 weights (Blackwell only) — fits within ~135GB; pairs with FP8 KV cache"
|
||||
|
||||
compatible_strategies:
|
||||
- single_node_tp
|
||||
- single_node_tep
|
||||
- single_node_dep
|
||||
- multi_node_tp
|
||||
- multi_node_dep
|
||||
- multi_node_tep
|
||||
|
||||
hardware_overrides: {}
|
||||
|
||||
strategy_overrides: {}
|
||||
|
||||
guide: |
|
||||
## Overview
|
||||
|
||||
[Laguna M.1](https://huggingface.co/poolside/Laguna-M.1) is Poolside's 225B-total / 23B-activated Mixture-of-Experts model purpose-built for agentic coding and long-horizon work. It is competitive with state-of-the-art open-weight and frontier models on SWE-bench Verified (74.6%), SWE-bench Multilingual, SWE-Bench Pro, and Terminal-Bench 2.0.
|
||||
|
||||
### Key features
|
||||
- **Large sparse MoE**: 70-layer transformer — the first 3 layers are dense SwiGLU, the remaining 67 are sparse MoE with **256 experts + 1 shared expert** and top-k=16 routing with auxiliary-loss-free load balancing.
|
||||
- **Global attention**: global attention across all layers with 64 Q-heads, 8 KV-heads, head dim 128, and softplus attention output gating.
|
||||
- **256K context**: RoPE with YaRN extends the context window to 262,144 tokens.
|
||||
- **Native interleaved reasoning**: thinking blocks emitted before and between tool calls, with *preserved thinking* (returning prior `reasoning` in history) recommended for best agentic performance. Toggled per-request via `enable_thinking`.
|
||||
- **Tool calling**: Poolside-specific tool-call protocol, parsed via `poolside_v1`.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
Laguna M.1 support landed in vLLM via [PR #41129](https://github.com/vllm-project/vllm/pull/41129) (shared implementation with [Laguna XS.2](https://huggingface.co/poolside/Laguna-XS.2)) and ships in **vLLM 0.21.0 and later**.
|
||||
|
||||
```bash
|
||||
uv pip install -U 'vllm>=0.21.0'
|
||||
```
|
||||
|
||||
FP8 and NVFP4 quantized checkpoints are available at [Laguna-M.1-FP8](https://huggingface.co/poolside/Laguna-M.1-FP8) and [Laguna-M.1-NVFP4](https://huggingface.co/poolside/Laguna-M.1-NVFP4). Quantization is detected automatically from the checkpoint's `quantization_config`, so the launch command is identical — just swap the model ID (select the variant above).
|
||||
|
||||
## Launch command
|
||||
|
||||
### Single node (8×H200, BF16)
|
||||
```bash
|
||||
vllm serve poolside/Laguna-M.1 \
|
||||
--tensor-parallel-size 8 \
|
||||
--served-model-name laguna \
|
||||
--enable-auto-tool-choice \
|
||||
--tool-call-parser poolside_v1 \
|
||||
--reasoning-parser poolside_v1 \
|
||||
--default-chat-template-kwargs '{"enable_thinking": true}'
|
||||
```
|
||||
|
||||
The BF16 checkpoint (~450 GB of weights) fits on a single 8×H200 node. Use the **FP8** variant to run on 4 GPUs, or the **NVFP4** variant (Blackwell only) to fit within ~135 GB.
|
||||
|
||||
## Controlling reasoning
|
||||
|
||||
Reasoning is enabled at the server when you pass `--default-chat-template-kwargs '{"enable_thinking": true}'` (included above). Poolside recommends **preserved thinking** for agentic coding — return the previous assistant turn's `reasoning` in the message history.
|
||||
|
||||
```python
|
||||
from openai import OpenAI
|
||||
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
|
||||
|
||||
resp = client.chat.completions.create(
|
||||
model="laguna",
|
||||
messages=[{"role": "user", "content": "Write a Python retry wrapper with exponential backoff."}],
|
||||
temperature=1.0,
|
||||
extra_query={"top_k": 20},
|
||||
stream=True,
|
||||
)
|
||||
for chunk in resp:
|
||||
delta = chunk.choices[0].delta
|
||||
rc = delta.reasoning
|
||||
if rc:
|
||||
print(rc, end="")
|
||||
if delta.content:
|
||||
print(delta.content, end="")
|
||||
```
|
||||
|
||||
Recommended sampling (matching Poolside's benchmark runs): **temperature=1.0, top_k=20**, thinking enabled.
|
||||
|
||||
To disable thinking, omit `--default-chat-template-kwargs` at launch, or pass `extra_body={"chat_template_kwargs": {"enable_thinking": False}}` per request.
|
||||
|
||||
## References
|
||||
|
||||
- [Model card](https://huggingface.co/poolside/Laguna-M.1)
|
||||
- [Release blog post](https://poolside.ai/blog/laguna-a-deeper-dive)
|
||||
- [Technical report](https://poolside.ai/assets/laguna/laguna-m1-xs2-technical-report.pdf)
|
||||
- [vLLM support PR #41129](https://github.com/vllm-project/vllm/pull/41129)
|
||||
@@ -0,0 +1,258 @@
|
||||
meta:
|
||||
title: "Laguna S-2.1"
|
||||
slug: "laguna-s-2.1"
|
||||
provider: "Poolside"
|
||||
description: "Poolside's 118B total / 8B activated MoE coding model with mixed sliding-window + global attention, native interleaved reasoning, and 256K context — the larger sibling of Laguna XS-2.1, tuned for agentic coding."
|
||||
date_added: 2026-07-21
|
||||
date_updated: 2026-07-22
|
||||
tasks:
|
||||
- text
|
||||
performance_headline: "118B/8B-A MoE for agentic coding with interleaved thinking and tool use"
|
||||
related_recipes:
|
||||
- "poolside/Laguna-XS-2.1"
|
||||
- "poolside/Laguna-M.1"
|
||||
hardware:
|
||||
h200: verified
|
||||
mi325x: verified
|
||||
|
||||
model:
|
||||
model_id: "poolside/Laguna-S-2.1"
|
||||
min_vllm_version: "0.25.0"
|
||||
architecture: moe
|
||||
parameter_count: "118B"
|
||||
active_parameters: "8B"
|
||||
context_length: 262144
|
||||
base_args:
|
||||
- "--trust-remote-code"
|
||||
base_env: {}
|
||||
|
||||
features:
|
||||
tool_calling:
|
||||
description: "Enable automatic tool choice with Poolside's tool-call parser"
|
||||
args:
|
||||
- "--enable-auto-tool-choice"
|
||||
- "--tool-call-parser"
|
||||
- "poolside_v1"
|
||||
reasoning:
|
||||
description: "Enable interleaved thinking with Poolside's reasoning parser"
|
||||
args:
|
||||
- "--reasoning-parser"
|
||||
- "poolside_v1"
|
||||
spec_decoding:
|
||||
description: "DFlash speculative decoding with the Laguna-S-2.1 draft model (15 tokens, greedy)"
|
||||
args:
|
||||
- "--speculative-config"
|
||||
- '{"model":"poolside/Laguna-S-2.1-DFlash","num_speculative_tokens":15,"method":"dflash"}'
|
||||
- "--moe-backend"
|
||||
- "triton"
|
||||
|
||||
opt_in_features:
|
||||
- spec_decoding
|
||||
|
||||
variants:
|
||||
default:
|
||||
precision: bf16
|
||||
vram_minimum_gb: 282
|
||||
description: "BF16 weights (~235GB) — needs a multi-GPU node (≥2× H200/B200, or a full HGX node)"
|
||||
hardware_overrides:
|
||||
mi325x:
|
||||
extra_env:
|
||||
VLLM_ROCM_USE_AITER: "1"
|
||||
VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS: "1"
|
||||
VLLM_ROCM_QUICK_REDUCE_QUANTIZATION: "INT4"
|
||||
fp8:
|
||||
model_id: "poolside/Laguna-S-2.1-FP8"
|
||||
precision: fp8
|
||||
vram_minimum_gb: 145
|
||||
description: "Block-FP8 weights + FP8 KV cache (~121GB) — a hair over one 141GB H200, so TP2 (or a single 268GB B300)"
|
||||
extra_env:
|
||||
VLLM_BLOCKSCALE_FP8_GEMM_FLASHINFER: "0"
|
||||
hardware_overrides:
|
||||
mi325x:
|
||||
extra_env:
|
||||
VLLM_ROCM_USE_AITER: "1"
|
||||
VLLM_ROCM_USE_AITER_MOE: "0"
|
||||
VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS: "1"
|
||||
VLLM_ROCM_QUICK_REDUCE_QUANTIZATION: "INT4"
|
||||
nvfp4:
|
||||
model_id: "poolside/Laguna-S-2.1-NVFP4"
|
||||
precision: nvfp4
|
||||
vram_minimum_gb: 86
|
||||
description: "NVFP4 weights + FP8 KV cache (~72GB, Blackwell only) — fits a single B200/B300"
|
||||
int4:
|
||||
model_id: "poolside/Laguna-S-2.1-INT4"
|
||||
precision: int4
|
||||
vram_minimum_gb: 86
|
||||
description: "INT4 (W4A16) weights + FP8 KV cache (~72GB) — fits a single 96GB+ GPU (H200/B200)"
|
||||
|
||||
compatible_strategies:
|
||||
- single_node_tp
|
||||
- single_node_tep
|
||||
- single_node_dep
|
||||
|
||||
hardware_overrides: {}
|
||||
|
||||
strategy_overrides: {}
|
||||
|
||||
guide: |
|
||||
## Overview
|
||||
|
||||
[Laguna S-2.1](https://huggingface.co/poolside/Laguna-S-2.1) is Poolside's
|
||||
118B-total / 8B-activated Mixture-of-Experts model — the larger sibling of
|
||||
[Laguna XS-2.1](https://huggingface.co/poolside/Laguna-XS-2.1), built for
|
||||
agentic coding and long-horizon work. It shares the `laguna` architecture:
|
||||
mixed sliding-window + global attention (3:1 across 48 layers) with softplus
|
||||
per-head gating and an FP8 KV cache, plus native interleaved reasoning and
|
||||
tool calling.
|
||||
|
||||
### Key features
|
||||
- **Mixed SWA + global attention**: 36 sliding-window layers (window=512)
|
||||
interleaved with 12 global-attention layers (3:1), each with per-layer
|
||||
rotary scaling.
|
||||
- **256 experts + 1 shared expert** with top-10 routing.
|
||||
- **Grouped-query attention**: 8 KV heads, head dim 128, per-head softplus
|
||||
output gating.
|
||||
- **Native FP8 KV cache** to reduce memory per token.
|
||||
- **Interleaved reasoning**: thinking blocks emitted between tool calls;
|
||||
toggled per-request via `enable_thinking`.
|
||||
- **Tool calling**: Poolside's XML-style tool-call protocol, parsed via
|
||||
`poolside_v1`.
|
||||
|
||||
Same architecture as XS-2.1, so vLLM needs no S-specific code — support
|
||||
lands with the shared `laguna` implementation.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
### pip
|
||||
```bash
|
||||
uv pip install -U 'vllm>=0.25.0'
|
||||
```
|
||||
|
||||
### Docker
|
||||
```bash
|
||||
docker pull vllm/vllm-openai:latest
|
||||
```
|
||||
|
||||
## Variants and VRAM
|
||||
|
||||
Laguna S-2.1 is large: the BF16 weights are ~235GB, so unlike XS-2.1 the
|
||||
default **does not fit a single GPU** — plan for a multi-GPU node or pick a
|
||||
quantized checkpoint. Select one with the **Variant** control above:
|
||||
|
||||
| Variant | Checkpoint | Weights | ~Size | Fits on |
|
||||
|---------|-----------|---------|-------|---------|
|
||||
| BF16 | [Laguna-S-2.1](https://huggingface.co/poolside/Laguna-S-2.1) | BF16 | 235 GB | ≥2× H200/B200, or a full HGX node |
|
||||
| FP8 | [Laguna-S-2.1-FP8](https://huggingface.co/poolside/Laguna-S-2.1-FP8) | Block-FP8 | 121 GB | TP2 on H200, or one B300; set `VLLM_BLOCKSCALE_FP8_GEMM_FLASHINFER=0` |
|
||||
| NVFP4 | [Laguna-S-2.1-NVFP4](https://huggingface.co/poolside/Laguna-S-2.1-NVFP4) | NVFP4 | 72 GB | single B200/B300 (Blackwell only) |
|
||||
| INT4 | [Laguna-S-2.1-INT4](https://huggingface.co/poolside/Laguna-S-2.1-INT4) | INT4 (W4A16) | 72 GB | single H200/B200 |
|
||||
|
||||
Quantization is detected automatically from each checkpoint's
|
||||
`quantization_config`, so the launch command is identical — just swap the
|
||||
model ID. All three quantized checkpoints ship an **FP8-quantized KV cache**.
|
||||
|
||||
Precision-matched DFlash speculators ship for each quantization —
|
||||
[DFlash-FP8](https://huggingface.co/poolside/Laguna-S-2.1-DFlash-FP8),
|
||||
[DFlash-NVFP4](https://huggingface.co/poolside/Laguna-S-2.1-DFlash-NVFP4),
|
||||
[DFlash-INT4](https://huggingface.co/poolside/Laguna-S-2.1-DFlash-INT4) — to
|
||||
pair speculative decoding with a quantized base at matching precision.
|
||||
|
||||
> **Context length.** The BF16 checkpoint supports up to 1,048,576 tokens;
|
||||
> the quantized checkpoints are configured for 256K. The commands below use
|
||||
> `--max-model-len 262144`; raise or lower it to trade context for KV memory.
|
||||
|
||||
## Launch command
|
||||
|
||||
### Multi-GPU node (H200/B200, BF16)
|
||||
```bash
|
||||
vllm serve poolside/Laguna-S-2.1 \
|
||||
--trust-remote-code \
|
||||
--tensor-parallel-size 8 \
|
||||
--max-model-len 262144 \
|
||||
--enable-auto-tool-choice \
|
||||
--tool-call-parser poolside_v1 \
|
||||
--reasoning-parser poolside_v1
|
||||
```
|
||||
|
||||
### AMD ROCm — 8× MI325X
|
||||
|
||||
For FP8, the MI325X override disables AITER MoE because its blockscale kernel
|
||||
currently corrupts logits; other AITER operators remain enabled, while vLLM
|
||||
uses Triton MoE.
|
||||
|
||||
### Single GPU (INT4 on H200, or NVFP4 on B200)
|
||||
```bash
|
||||
vllm serve poolside/Laguna-S-2.1-INT4 \
|
||||
--trust-remote-code \
|
||||
--max-model-len 262144 \
|
||||
--enable-auto-tool-choice \
|
||||
--tool-call-parser poolside_v1 \
|
||||
--reasoning-parser poolside_v1
|
||||
```
|
||||
|
||||
## Controlling reasoning
|
||||
|
||||
Reasoning is **off by default** in the chat template. Enable it per-request:
|
||||
|
||||
```python
|
||||
from openai import OpenAI
|
||||
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
|
||||
|
||||
resp = client.chat.completions.create(
|
||||
model="poolside/Laguna-S-2.1",
|
||||
messages=[{"role": "user", "content": "Write a Python retry wrapper with exponential backoff."}],
|
||||
extra_body={"chat_template_kwargs": {"enable_thinking": True}},
|
||||
temperature=0.7,
|
||||
top_p=1.0,
|
||||
extra_query={"top_k": 20},
|
||||
)
|
||||
print(resp.choices[0].message.reasoning)
|
||||
print(resp.choices[0].message.content)
|
||||
```
|
||||
|
||||
Or default-on with `--default-chat-template-kwargs '{"enable_thinking": true}'`.
|
||||
|
||||
The model's `generation_config.json` sets `top_k 20` (eval-certified
|
||||
truncation) so clients that send no sampling params still get sane output.
|
||||
Do **not** add `min_p` when speculative decoding is on — vLLM rejects
|
||||
`min_p`/`logit_bias` with speculation; use `top_k`/`top_p`.
|
||||
|
||||
## Speculative decoding (DFlash)
|
||||
|
||||
Enable the **Spec Decoding** toggle above to attach Poolside's
|
||||
[DFlash draft model](https://huggingface.co/poolside/Laguna-S-2.1-DFlash) — a
|
||||
parallel drafter that proposes up to 15 draft tokens per step. Measured mean
|
||||
accepted length is ~3.1 tokens/step on natural text (higher on code),
|
||||
matching Poolside's production deployment.
|
||||
|
||||
Requires:
|
||||
- DFlash drafter support for Laguna, merged in
|
||||
[PR #46853](https://github.com/vllm-project/vllm/pull/46853) (2026-07-03) —
|
||||
use a vLLM build that includes it (validated on 0.25.1).
|
||||
- `--moe-backend triton` — the default DeepGEMM MoE backend is incompatible
|
||||
with the DFlash draft path. Enabling the toggle adds this flag.
|
||||
|
||||
```bash
|
||||
vllm serve poolside/Laguna-S-2.1 \
|
||||
--trust-remote-code \
|
||||
--tensor-parallel-size 8 \
|
||||
--max-model-len 16384 \
|
||||
--enable-auto-tool-choice \
|
||||
--tool-call-parser poolside_v1 \
|
||||
--reasoning-parser poolside_v1 \
|
||||
--speculative-config '{"model":"poolside/Laguna-S-2.1-DFlash","num_speculative_tokens":15,"method":"dflash"}' \
|
||||
--moe-backend triton
|
||||
```
|
||||
|
||||
### DGX Spark (GB10)
|
||||
|
||||
The NVFP4 checkpoint is validated on a single DGX Spark (GB10, `sm_121a`) with
|
||||
vLLM 0.25.1 + a FlashInfer nightly and DFlash — see the
|
||||
[Laguna-S-2.1-NVFP4 model card](https://huggingface.co/poolside/Laguna-S-2.1-NVFP4)
|
||||
for the full one-box recipe.
|
||||
|
||||
## References
|
||||
|
||||
- [Model card](https://huggingface.co/poolside/Laguna-S-2.1)
|
||||
- [DFlash draft model](https://huggingface.co/poolside/Laguna-S-2.1-DFlash)
|
||||
- [vLLM DFlash spec-decoding PR #46853](https://github.com/vllm-project/vllm/pull/46853)
|
||||
- [Laguna XS-2.1 recipe](https://huggingface.co/poolside/Laguna-XS-2.1)
|
||||
@@ -0,0 +1,207 @@
|
||||
meta:
|
||||
title: "Laguna XS-2.1"
|
||||
slug: "laguna-xs-2.1"
|
||||
provider: "Poolside"
|
||||
description: "Poolside's 33B total / 3B activated MoE coding model with mixed sliding-window + global attention, native interleaved reasoning, and 256K context — designed for agentic coding."
|
||||
date_added: 2026-07-02
|
||||
date_updated: 2026-07-02
|
||||
tasks:
|
||||
- text
|
||||
performance_headline: "33B/3B-A MoE for agentic coding with interleaved thinking and tool use"
|
||||
related_recipes:
|
||||
- "poolside/Laguna-XS.2"
|
||||
- "poolside/Laguna-M.1"
|
||||
hardware:
|
||||
h200: verified
|
||||
|
||||
model:
|
||||
model_id: "poolside/Laguna-XS-2.1"
|
||||
min_vllm_version: "0.21.0"
|
||||
architecture: moe
|
||||
parameter_count: "33B"
|
||||
active_parameters: "3B"
|
||||
context_length: 262144
|
||||
base_args:
|
||||
- "--trust-remote-code"
|
||||
base_env: {}
|
||||
|
||||
features:
|
||||
tool_calling:
|
||||
description: "Enable automatic tool choice with Poolside's tool-call parser"
|
||||
args:
|
||||
- "--enable-auto-tool-choice"
|
||||
- "--tool-call-parser"
|
||||
- "poolside_v1"
|
||||
reasoning:
|
||||
description: "Enable interleaved thinking with Poolside's reasoning parser"
|
||||
args:
|
||||
- "--reasoning-parser"
|
||||
- "poolside_v1"
|
||||
spec_decoding:
|
||||
description: "DFlash speculative decoding with the Laguna-XS-2.1 draft model (15 tokens, greedy)"
|
||||
args:
|
||||
- "--speculative-config"
|
||||
- '{"model":"poolside/Laguna-XS-2.1-DFlash","num_speculative_tokens":15,"method":"dflash"}'
|
||||
- "--moe-backend"
|
||||
- "triton"
|
||||
|
||||
opt_in_features:
|
||||
- spec_decoding
|
||||
|
||||
variants:
|
||||
default:
|
||||
precision: bf16
|
||||
vram_minimum_gb: 80
|
||||
description: "BF16 weights — fits on a single 80GB+ GPU (H100/H200/B200)"
|
||||
fp8:
|
||||
model_id: "poolside/Laguna-XS-2.1-FP8"
|
||||
precision: fp8
|
||||
vram_minimum_gb: 40
|
||||
min_vllm_version: "0.22.0"
|
||||
description: "Block-FP8 weights + FP8 KV cache — halves VRAM, fits on a single 40GB+ GPU"
|
||||
extra_env:
|
||||
VLLM_BLOCKSCALE_FP8_GEMM_FLASHINFER: "0"
|
||||
nvfp4:
|
||||
model_id: "poolside/Laguna-XS-2.1-NVFP4"
|
||||
precision: nvfp4
|
||||
vram_minimum_gb: 26
|
||||
min_vllm_version: "0.22.0"
|
||||
description: "NVFP4 weights + FP8 KV cache (Blackwell only) — the most compact variant at ~26GB"
|
||||
int4:
|
||||
model_id: "poolside/Laguna-XS-2.1-INT4"
|
||||
precision: int4
|
||||
vram_minimum_gb: 29
|
||||
min_vllm_version: "0.22.0"
|
||||
description: "INT4 (W4A16) weights + FP8 KV cache — fits on a single 32GB+ GPU"
|
||||
|
||||
compatible_strategies:
|
||||
- single_node_tp
|
||||
- single_node_tep
|
||||
- single_node_dep
|
||||
|
||||
hardware_overrides: {}
|
||||
|
||||
strategy_overrides:
|
||||
single_node_tp:
|
||||
tp: 1
|
||||
|
||||
guide: |
|
||||
## Overview
|
||||
|
||||
[Laguna XS-2.1](https://huggingface.co/poolside/Laguna-XS-2.1) is Poolside's 33B-total / 3B-activated Mixture-of-Experts model purpose-built for agentic coding and long-horizon work. It combines mixed sliding-window + global attention (3:1 across 40 layers) with sigmoid per-head gating and FP8 KV cache, so it stays compact enough to run locally while supporting a 256K-token context. The 2.1 point release improves coding quality over XS.2 (SWE-bench Multilingual 57.7% → 63.1%, SWE-bench Verified 69.9% → 70.9%).
|
||||
|
||||
### Key features
|
||||
- **Mixed SWA + global attention**: 30 sliding-window layers (window=512) interleaved with 10 global-attention layers, each with per-layer rotary scaling.
|
||||
- **Native FP8 KV cache**: KV cache is quantized to FP8 to reduce memory per token.
|
||||
- **Interleaved reasoning**: thinking blocks emitted between tool calls; toggled per-request via `enable_thinking`.
|
||||
- **Tool calling**: Poolside-specific XML-style tool-call protocol, parsed via `poolside_v1`.
|
||||
- **256 experts + 1 shared expert** with top-8 routing.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
Laguna XS-2.1 support is available in vLLM 0.21.0 and later (via [PR #41129](https://github.com/vllm-project/vllm/pull/41129)).
|
||||
|
||||
### pip
|
||||
```bash
|
||||
uv pip install -U 'vllm>=0.21.0'
|
||||
```
|
||||
|
||||
### Docker
|
||||
```bash
|
||||
docker pull vllm/vllm-openai:latest
|
||||
```
|
||||
|
||||
## Quantized variants
|
||||
|
||||
Pre-quantized checkpoints are published alongside the BF16 weights — select one with the **Variant** control above:
|
||||
|
||||
| Variant | Checkpoint | Weights | ~VRAM | Notes |
|
||||
|---------|-----------|---------|-------|-------|
|
||||
| BF16 | [Laguna-XS-2.1](https://huggingface.co/poolside/Laguna-XS-2.1) | BF16 | 80 GB | Default |
|
||||
| FP8 | [Laguna-XS-2.1-FP8](https://huggingface.co/poolside/Laguna-XS-2.1-FP8) | Block-FP8 | 40 GB | Set `VLLM_BLOCKSCALE_FP8_GEMM_FLASHINFER=0` |
|
||||
| NVFP4 | [Laguna-XS-2.1-NVFP4](https://huggingface.co/poolside/Laguna-XS-2.1-NVFP4) | NVFP4 | 26 GB | Blackwell only |
|
||||
| INT4 | [Laguna-XS-2.1-INT4](https://huggingface.co/poolside/Laguna-XS-2.1-INT4) | INT4 (W4A16) | 29 GB | Hopper or Blackwell |
|
||||
|
||||
Quantization is detected automatically from each checkpoint's `quantization_config`, so the launch command is identical — just swap the model ID (select the variant above). All three quantized checkpoints ship an **FP8-quantized KV cache**.
|
||||
|
||||
> **vLLM ≥ 0.22.0 required for the quantized variants.** The FP8 KV cache needs the per-layer attention-head fix in [vllm#42650](https://github.com/vllm-project/vllm/pull/42650); earlier versions produce scrambled output on non-Hopper GPUs. On older vLLM you can disable the FP8 KV cache with `--kv-cache-dtype-skip-layers $(seq 0 39)`.
|
||||
|
||||
Precision-matched DFlash speculators are also available for each quantization — [DFlash-FP8](https://huggingface.co/poolside/Laguna-XS-2.1-DFlash-FP8), [DFlash-NVFP4](https://huggingface.co/poolside/Laguna-XS-2.1-DFlash-NVFP4), and [DFlash-INT4](https://huggingface.co/poolside/Laguna-XS-2.1-DFlash-INT4) — if you want to pair speculative decoding with a quantized base at matching precision.
|
||||
|
||||
## Launch command
|
||||
|
||||
### Single GPU (H100/H200/B200, BF16)
|
||||
```bash
|
||||
vllm serve poolside/Laguna-XS-2.1 \
|
||||
--trust-remote-code \
|
||||
--max-model-len 262144 \
|
||||
--enable-auto-tool-choice \
|
||||
--tool-call-parser poolside_v1 \
|
||||
--reasoning-parser poolside_v1
|
||||
```
|
||||
|
||||
### Docker
|
||||
```bash
|
||||
docker run -itd --name laguna-xs21 \
|
||||
--ipc=host --network host --shm-size 16G --gpus all \
|
||||
-v ~/.cache/huggingface:/root/.cache/huggingface \
|
||||
vllm/vllm-openai:latest \
|
||||
--model poolside/Laguna-XS-2.1 \
|
||||
--trust-remote-code \
|
||||
--max-model-len 262144 \
|
||||
--enable-auto-tool-choice \
|
||||
--tool-call-parser poolside_v1 \
|
||||
--reasoning-parser poolside_v1 \
|
||||
--host 0.0.0.0 --port 8000
|
||||
```
|
||||
|
||||
## Controlling reasoning
|
||||
|
||||
Reasoning is **off by default** in the chat template. Enable it per-request:
|
||||
|
||||
```python
|
||||
from openai import OpenAI
|
||||
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
|
||||
|
||||
resp = client.chat.completions.create(
|
||||
model="poolside/Laguna-XS-2.1",
|
||||
messages=[{"role": "user", "content": "Write a Python retry wrapper with exponential backoff."}],
|
||||
extra_body={"chat_template_kwargs": {"enable_thinking": True}},
|
||||
temperature=0.7,
|
||||
top_p=1.0,
|
||||
extra_query={"top_k": 20},
|
||||
)
|
||||
print(resp.choices[0].message.reasoning)
|
||||
print(resp.choices[0].message.content)
|
||||
```
|
||||
|
||||
Or default-on with `--default-chat-template-kwargs '{"enable_thinking": true}'`.
|
||||
|
||||
## Speculative decoding (DFlash)
|
||||
|
||||
Enable the **Spec Decoding** toggle above to attach Poolside's [DFlash draft model](https://huggingface.co/poolside/Laguna-XS-2.1-DFlash) — a 5-layer Llama-style speculator that proposes up to 15 tokens per step. Reported end-to-end speedup is 1.67x–2.64x across evaluation datasets, with a mean accepted length of 3.55–4.57 tokens per step.
|
||||
|
||||
Requires:
|
||||
- vLLM built from [PR #46853](https://github.com/vllm-project/vllm/pull/46853) (adds DFlash drafter support for Laguna-XS-2.1) — not yet in a stable release.
|
||||
- `--moe-backend triton` — the default DeepGEMM MoE backend is currently incompatible with the DFlash draft path, so force the Triton MoE backend. Enabling the **Spec Decoding** toggle above adds this flag automatically.
|
||||
|
||||
Example:
|
||||
|
||||
```bash
|
||||
vllm serve poolside/Laguna-XS-2.1 \
|
||||
--trust-remote-code \
|
||||
--max-model-len 16384 \
|
||||
--enable-auto-tool-choice \
|
||||
--tool-call-parser poolside_v1 \
|
||||
--reasoning-parser poolside_v1 \
|
||||
--speculative-config '{"model":"poolside/Laguna-XS-2.1-DFlash","num_speculative_tokens":15,"method":"dflash"}' \
|
||||
--moe-backend triton
|
||||
```
|
||||
|
||||
## References
|
||||
|
||||
- [Model card](https://huggingface.co/poolside/Laguna-XS-2.1)
|
||||
- [Release blog post](https://poolside.ai/blog/laguna-a-deeper-dive)
|
||||
- [vLLM support PR #41129](https://github.com/vllm-project/vllm/pull/41129)
|
||||
- [vLLM DFlash spec-decoding PR #46853](https://github.com/vllm-project/vllm/pull/46853)
|
||||
- [DFlash draft model](https://huggingface.co/poolside/Laguna-XS-2.1-DFlash)
|
||||
@@ -0,0 +1,257 @@
|
||||
meta:
|
||||
title: "Laguna XS.2"
|
||||
slug: "laguna-xs.2"
|
||||
provider: "Poolside"
|
||||
description: "Poolside's 33B total / 3B activated MoE coding model with mixed sliding-window + global attention, native interleaved reasoning, and 256K context — designed for agentic coding."
|
||||
date_added: 2026-04-28
|
||||
date_updated: 2026-04-29
|
||||
tasks:
|
||||
- text
|
||||
performance_headline: "33B/3B-A MoE for agentic coding with interleaved thinking and tool use"
|
||||
related_recipes: []
|
||||
hardware:
|
||||
h200: verified
|
||||
mi300x: verified
|
||||
mi325x: verified
|
||||
mi355x: verified
|
||||
|
||||
model:
|
||||
model_id: "poolside/Laguna-XS.2"
|
||||
min_vllm_version: "0.21.0"
|
||||
architecture: moe
|
||||
parameter_count: "33B"
|
||||
active_parameters: "3B"
|
||||
context_length: 262144
|
||||
base_args:
|
||||
- "--trust-remote-code"
|
||||
base_env: {}
|
||||
|
||||
features:
|
||||
tool_calling:
|
||||
description: "Enable automatic tool choice with Poolside's tool-call parser"
|
||||
args:
|
||||
- "--enable-auto-tool-choice"
|
||||
- "--tool-call-parser"
|
||||
- "poolside_v1"
|
||||
reasoning:
|
||||
description: "Enable interleaved thinking with Poolside's reasoning parser"
|
||||
args:
|
||||
- "--reasoning-parser"
|
||||
- "poolside_v1"
|
||||
spec_decoding:
|
||||
description: "DFlash speculative decoding with the Laguna-XS.2 draft model (7 tokens, greedy)"
|
||||
args:
|
||||
- "--speculative-config"
|
||||
- '{"model":"poolside/Laguna-XS.2-speculator.dflash","num_speculative_tokens":7,"method":"dflash"}'
|
||||
|
||||
opt_in_features:
|
||||
- spec_decoding
|
||||
|
||||
variants:
|
||||
default:
|
||||
precision: bf16
|
||||
vram_minimum_gb: 80
|
||||
description: "BF16 weights — fits on a single 80GB+ GPU (H100/H200/B200)"
|
||||
fp8:
|
||||
model_id: "poolside/Laguna-XS.2-FP8"
|
||||
precision: fp8
|
||||
vram_minimum_gb: 40
|
||||
min_vllm_version: "0.22.0"
|
||||
description: "Block-FP8 weights + FP8 KV cache — halves VRAM, fits on a single 40GB+ GPU"
|
||||
extra_env:
|
||||
VLLM_BLOCKSCALE_FP8_GEMM_FLASHINFER: "0"
|
||||
nvfp4:
|
||||
model_id: "poolside/Laguna-XS.2-NVFP4"
|
||||
precision: nvfp4
|
||||
vram_minimum_gb: 26
|
||||
min_vllm_version: "0.22.0"
|
||||
description: "NVFP4 weights + FP8 KV cache (Blackwell only) — the most compact variant at ~26GB"
|
||||
int4:
|
||||
model_id: "poolside/Laguna-XS.2-INT4"
|
||||
precision: int4
|
||||
vram_minimum_gb: 29
|
||||
min_vllm_version: "0.22.0"
|
||||
description: "INT4 (W4A16) weights + FP8 KV cache — fits on a single 32GB+ GPU"
|
||||
|
||||
redhat_fp8:
|
||||
model_id: "RedHatAI/Laguna-XS.2-FP8"
|
||||
precision: fp8
|
||||
label: "FP8 (RedHat)"
|
||||
vram_minimum_gb: 40
|
||||
description: "RedHatAI FP8 quantization"
|
||||
extra_args:
|
||||
- "--kv-cache-dtype"
|
||||
- "fp8"
|
||||
|
||||
redhat_nvfp4:
|
||||
model_id: "RedHatAI/Laguna-XS.2-NVFP4"
|
||||
precision: nvfp4
|
||||
label: "NVFP4 (RedHat)"
|
||||
vram_minimum_gb: 22
|
||||
description: "RedHatAI NVFP4 quantization"
|
||||
|
||||
compatible_strategies:
|
||||
- single_node_tp
|
||||
- single_node_tep
|
||||
- single_node_dep
|
||||
|
||||
hardware_overrides: {}
|
||||
|
||||
strategy_overrides:
|
||||
single_node_tp:
|
||||
tp: 1
|
||||
|
||||
guide: |
|
||||
## Overview
|
||||
|
||||
[Laguna XS.2](https://huggingface.co/poolside/Laguna-XS.2) is Poolside's 33B-total / 3B-activated Mixture-of-Experts model purpose-built for agentic coding and long-horizon work. It combines mixed sliding-window + global attention (3:1 across 40 layers) with sigmoid per-head gating and FP8 KV cache, so it stays compact enough to run locally while supporting a 256K-token context.
|
||||
|
||||
### Key features
|
||||
- **Mixed SWA + global attention**: 30 sliding-window layers (window=512) interleaved with 10 global-attention layers, each with per-layer rotary scaling.
|
||||
- **Native FP8 KV cache**: KV cache is quantized to FP8 to reduce memory per token.
|
||||
- **Interleaved reasoning**: thinking blocks emitted between tool calls; toggled per-request via `enable_thinking`.
|
||||
- **Tool calling**: Poolside-specific XML-style tool-call protocol, parsed via `poolside_v1`.
|
||||
- **256 experts + 1 shared expert** with top-8 routing.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
Laguna XS.2 support is on the open vLLM PR ([vllm-project/vllm#41129](https://github.com/vllm-project/vllm/pull/41129)) — install from a nightly wheel or the pinned Docker image below until the PR lands in a stable release.
|
||||
|
||||
### Docker (recommended)
|
||||
```bash
|
||||
docker pull vllm/vllm-openai:laguna
|
||||
```
|
||||
|
||||
### pip (nightly)
|
||||
```bash
|
||||
uv venv
|
||||
source .venv/bin/activate
|
||||
uv pip install -U vllm --pre \
|
||||
--extra-index-url https://wheels.vllm.ai/nightly/cu130 \
|
||||
--extra-index-url https://download.pytorch.org/whl/cu130 \
|
||||
--index-strategy unsafe-best-match
|
||||
```
|
||||
|
||||
## Quantized variants
|
||||
|
||||
Pre-quantized checkpoints are published alongside the BF16 weights — select one with the **Variant** control above:
|
||||
|
||||
| Variant | Checkpoint | Weights | ~VRAM | Notes |
|
||||
|---------|-----------|---------|-------|-------|
|
||||
| BF16 | [Laguna-XS.2](https://huggingface.co/poolside/Laguna-XS.2) | BF16 | 80 GB | Default |
|
||||
| FP8 | [Laguna-XS.2-FP8](https://huggingface.co/poolside/Laguna-XS.2-FP8) | Block-FP8 | 40 GB | Set `VLLM_BLOCKSCALE_FP8_GEMM_FLASHINFER=0` |
|
||||
| NVFP4 | [Laguna-XS.2-NVFP4](https://huggingface.co/poolside/Laguna-XS.2-NVFP4) | NVFP4 | 26 GB | Blackwell only |
|
||||
| INT4 | [Laguna-XS.2-INT4](https://huggingface.co/poolside/Laguna-XS.2-INT4) | INT4 (W4A16) | 29 GB | Hopper or Blackwell |
|
||||
|
||||
Quantization is detected automatically from each checkpoint's `quantization_config`, so the launch command is identical — just swap the model ID (select the variant above). All three quantized checkpoints ship an **FP8-quantized KV cache**.
|
||||
|
||||
> **vLLM ≥ 0.22.0 required for the quantized variants.** The FP8 KV cache needs the per-layer attention-head fix in [vllm#42650](https://github.com/vllm-project/vllm/pull/42650); earlier versions produce scrambled output on non-Hopper GPUs. On older vLLM you can disable the FP8 KV cache with `--kv-cache-dtype-skip-layers $(seq 0 39)`.
|
||||
|
||||
## Deployment Configurations
|
||||
|
||||
### Single GPU (H100/H200/B200, BF16)
|
||||
```bash
|
||||
vllm serve poolside/Laguna-XS.2 \
|
||||
--trust-remote-code \
|
||||
--max-model-len 262144 \
|
||||
--enable-auto-tool-choice \
|
||||
--tool-call-parser poolside_v1 \
|
||||
--reasoning-parser poolside_v1
|
||||
```
|
||||
|
||||
### Docker
|
||||
```bash
|
||||
docker run -itd --name laguna-xs2 \
|
||||
--ipc=host --network host --shm-size 16G --gpus all \
|
||||
-v ~/.cache/huggingface:/root/.cache/huggingface \
|
||||
vllm/vllm-openai:laguna \
|
||||
--model poolside/Laguna-XS.2 \
|
||||
--trust-remote-code \
|
||||
--max-model-len 262144 \
|
||||
--enable-auto-tool-choice \
|
||||
--tool-call-parser poolside_v1 \
|
||||
--reasoning-parser poolside_v1 \
|
||||
--host 0.0.0.0 --port 8000
|
||||
```
|
||||
|
||||
### AMD MI300X / MI325X / MI355X
|
||||
|
||||
```bash
|
||||
vllm serve poolside/Laguna-XS.2 \
|
||||
--trust-remote-code \
|
||||
--max-model-len 262144 \
|
||||
--enable-auto-tool-choice \
|
||||
--tool-call-parser poolside_v1 \
|
||||
--reasoning-parser poolside_v1
|
||||
```
|
||||
|
||||
### Docker (AMD MI300X / MI325X / MI355X)
|
||||
|
||||
```bash
|
||||
docker run --rm -it \
|
||||
--cap-add=SYS_PTRACE \
|
||||
--ipc=host \
|
||||
--privileged=true \
|
||||
--shm-size=128GB \
|
||||
--network=host \
|
||||
--device=/dev/kfd \
|
||||
--device=/dev/dri \
|
||||
--group-add video \
|
||||
vllm/vllm-openai-rocm:latest \
|
||||
--model poolside/Laguna-XS.2 \
|
||||
--trust-remote-code \
|
||||
--max-model-len 262144 \
|
||||
--enable-auto-tool-choice \
|
||||
--tool-call-parser poolside_v1 \
|
||||
--reasoning-parser poolside_v1
|
||||
```
|
||||
|
||||
## Controlling reasoning
|
||||
|
||||
Reasoning is **off by default** in the chat template. Enable it per-request:
|
||||
|
||||
```python
|
||||
from openai import OpenAI
|
||||
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
|
||||
|
||||
resp = client.chat.completions.create(
|
||||
model="poolside/Laguna-XS.2",
|
||||
messages=[{"role": "user", "content": "Write a Python retry wrapper with exponential backoff."}],
|
||||
extra_body={"chat_template_kwargs": {"enable_thinking": True}},
|
||||
temperature=0.7,
|
||||
top_p=1.0,
|
||||
extra_query={"top_k": 20},
|
||||
)
|
||||
print(resp.choices[0].message.reasoning)
|
||||
print(resp.choices[0].message.content)
|
||||
```
|
||||
|
||||
Or default-on with `--default-chat-template-kwargs '{"enable_thinking": true}'`.
|
||||
|
||||
## Speculative decoding (DFlash)
|
||||
|
||||
Enable the **Spec Decoding** toggle above to attach Poolside's [DFlash draft model](https://huggingface.co/poolside/Laguna-XS.2-speculator.dflash) — a 5-layer Llama-style speculator that proposes up to 7 tokens per step. Reported per-position acceptance with reasoning enabled is ~70% at position 1 across coding, math, QA, and writing workloads.
|
||||
|
||||
Requires:
|
||||
- vLLM built from [PR #41880](https://github.com/vllm-project/vllm/pull/41880) (extends the base Laguna PR with DFlash support).
|
||||
- `VLLM_USE_DEEP_GEMM=0` in the launch environment — DeepGEMM is currently incompatible with the DFlash draft path.
|
||||
|
||||
Example:
|
||||
|
||||
```bash
|
||||
VLLM_USE_DEEP_GEMM=0 vllm serve poolside/Laguna-XS.2 \
|
||||
--trust-remote-code \
|
||||
--max-model-len 16384 \
|
||||
--enable-auto-tool-choice \
|
||||
--tool-call-parser poolside_v1 \
|
||||
--reasoning-parser poolside_v1 \
|
||||
--speculative-config '{"model":"poolside/Laguna-XS.2-speculator.dflash","num_speculative_tokens":7,"method":"dflash"}'
|
||||
```
|
||||
|
||||
## References
|
||||
|
||||
- [Model card](https://huggingface.co/poolside/Laguna-XS.2)
|
||||
- [Release blog post](https://poolside.ai/blog/laguna-a-deeper-dive)
|
||||
- [vLLM support PR #41129](https://github.com/vllm-project/vllm/pull/41129)
|
||||
- [vLLM DFlash spec-decoding PR #41880](https://github.com/vllm-project/vllm/pull/41880)
|
||||
- [DFlash draft model](https://huggingface.co/poolside/Laguna-XS.2-speculator.dflash)
|
||||
Reference in New Issue
Block a user