Roy WangandClaude Opus 5.5
b515648641
Signed-off-by: esmeetu <jasonailu87@gmail.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
134 lines
3.2 KiB
YAML
134 lines
3.2 KiB
YAML
meta:
|
||
title: "Qwen3.5-0.8B"
|
||
slug: "qwen3.5-0.8b"
|
||
provider: "Qwen"
|
||
description: "Qwen3.5 tiny dense multimodal model (0.8B) — ultra-low-VRAM / edge serving with 262K context"
|
||
date_added: 2026-04-22
|
||
date_updated: 2026-09-14
|
||
tasks:
|
||
- multimodal
|
||
- text
|
||
performance_headline: "Tiny Qwen3.5 dense for edge / draft-model use"
|
||
related_recipes:
|
||
- "Qwen/Qwen3.5-2B"
|
||
hardware:
|
||
arc_pro_b60: verified
|
||
arc_pro_b70: verified
|
||
|
||
model:
|
||
model_id: "Qwen/Qwen3.5-0.8B"
|
||
min_vllm_version: "0.17.0"
|
||
default_frontend: rust
|
||
architecture: dense
|
||
parameter_count: "0.8B"
|
||
active_parameters: "0.8B"
|
||
context_length: 262144
|
||
base_args:
|
||
- "--trust-remote-code"
|
||
base_env: {}
|
||
|
||
features:
|
||
tool_calling:
|
||
description: "Enable automatic tool choice with Qwen3 Coder parser"
|
||
args:
|
||
- "--enable-auto-tool-choice"
|
||
- "--tool-call-parser"
|
||
- "qwen3_coder"
|
||
reasoning:
|
||
description: "Enable chain-of-thought reasoning with Qwen3 parser"
|
||
args:
|
||
- "--reasoning-parser"
|
||
- "qwen3"
|
||
text_only:
|
||
description: "Skip loading the vision encoder for text-only workloads."
|
||
args:
|
||
- "--language-model-only"
|
||
|
||
opt_in_features:
|
||
- text_only
|
||
|
||
variants:
|
||
default:
|
||
precision: bf16
|
||
vram_minimum_gb: 2
|
||
description: "Full precision BF16 — runs on any modern GPU"
|
||
|
||
compatible_strategies:
|
||
- single_node_tp
|
||
|
||
hardware_overrides:
|
||
xpu:
|
||
extra_args:
|
||
- "--enforce-eager"
|
||
|
||
strategy_overrides: {}
|
||
|
||
guide: |
|
||
## Overview
|
||
|
||
[Qwen3.5-0.8B](https://huggingface.co/Qwen/Qwen3.5-0.8B) is the smallest
|
||
member of the Qwen3.5 family — same hybrid gated delta networks architecture
|
||
and 262K context, at a size suited to edge devices or as a draft model for
|
||
speculative decoding with larger Qwen3.5 checkpoints.
|
||
|
||
## Prerequisites
|
||
|
||
- **vLLM version:** >= 0.17.0
|
||
- **Hardware:** any modern GPU (>=4 GB VRAM) or Intel Arc Pro B60/B70
|
||
|
||
### Install vLLM
|
||
|
||
```bash
|
||
uv venv
|
||
source .venv/bin/activate
|
||
uv pip install -U vllm --torch-backend=auto
|
||
```
|
||
|
||
### Docker
|
||
```bash
|
||
docker pull vllm/vllm-openai-xpu:latest # Intel XPU (B60 / B70)
|
||
```
|
||
|
||
## Launching the Server
|
||
|
||
```bash
|
||
vllm serve Qwen/Qwen3.5-0.8B \
|
||
--max-model-len 262144 \
|
||
--reasoning-parser qwen3
|
||
```
|
||
|
||
### Docker (Intel XPU B60 / B70)
|
||
|
||
Validated on 1× Intel Arc Pro B60 / B70 (B60 24 GB, B70 32 GB per card) with the
|
||
official vLLM XPU image `vllm/vllm-openai-xpu:latest`.
|
||
|
||
```bash
|
||
docker run --device /dev/dri \
|
||
-v /dev/dri/by-path:/dev/dri/by-path --shm-size=16g \
|
||
--privileged --ipc=host -p 8000:8000 \
|
||
-v ~/.cache/huggingface:/root/.cache/huggingface \
|
||
vllm/vllm-openai-xpu:latest \
|
||
Qwen/Qwen3.5-0.8B \
|
||
--reasoning-parser qwen3 \
|
||
--enforce-eager
|
||
```
|
||
|
||
## Client Usage
|
||
|
||
```python
|
||
from openai import OpenAI
|
||
|
||
client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1")
|
||
resp = client.chat.completions.create(
|
||
model="Qwen/Qwen3.5-0.8B",
|
||
messages=[{"role": "user", "content": "Hi!"}],
|
||
max_tokens=64,
|
||
)
|
||
print(resp.choices[0].message.content)
|
||
```
|
||
|
||
## References
|
||
|
||
- [Model card](https://huggingface.co/Qwen/Qwen3.5-0.8B)
|
||
- [Base checkpoint](https://huggingface.co/Qwen/Qwen3.5-0.8B-Base)
|