Roy WangandClaude Opus 5.5
b515648641
Signed-off-by: esmeetu <jasonailu87@gmail.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
123 lines
4.5 KiB
YAML
123 lines
4.5 KiB
YAML
meta:
|
|
title: "TranslateGemma 27B IT"
|
|
slug: "translategemma-27b-it"
|
|
provider: "Google"
|
|
description: "Lightweight open translation model from Google (based on Gemma 3) supporting 55 languages. Served via the vLLM-optimized Infomaniak-AI checkpoint."
|
|
date_added: 2026-03-06
|
|
date_updated: 2026-04-17
|
|
tasks:
|
|
- text
|
|
performance_headline: "Deployable on laptops/desktops and cloud GPUs; vLLM-optimized checkpoint removes custom JSON inputs"
|
|
related_recipes: []
|
|
|
|
model:
|
|
model_id: "google/translategemma-27b-it"
|
|
min_vllm_version: "0.14.1"
|
|
architecture: dense
|
|
parameter_count: "27B"
|
|
active_parameters: "27B"
|
|
context_length: 131072
|
|
base_args: []
|
|
base_env: {}
|
|
|
|
features: {}
|
|
|
|
opt_in_features: []
|
|
|
|
variants:
|
|
default:
|
|
precision: bf16
|
|
vram_minimum_gb: 65
|
|
description: "Original Google checkpoint in BF16 (has vLLM compatibility issues — prefer vllm-optimized variant)"
|
|
vllm_optimized:
|
|
model_id: "Infomaniak-AI/vllm-translategemma-27b-it"
|
|
precision: bf16
|
|
vram_minimum_gb: 65
|
|
description: "Infomaniak-AI vLLM-optimized checkpoint — recommended. Fixes RoPE config, EOS token, and replaces custom JSON inputs with string delimiters."
|
|
small_4b:
|
|
model_id: "google/translategemma-4b-it"
|
|
precision: bf16
|
|
vram_minimum_gb: 10
|
|
description: "4B variant for lower-resource deployments (prefer Infomaniak-AI/vllm-translategemma-4b-it)"
|
|
|
|
compatible_strategies:
|
|
- single_node_tp
|
|
- multi_node_tp
|
|
|
|
hardware_overrides: {}
|
|
|
|
strategy_overrides: {}
|
|
|
|
guide: |
|
|
## Overview
|
|
|
|
[TranslateGemma](https://huggingface.co/collections/google/translategemma) is a family of lightweight, state-of-the-art open translation models from Google, based on the Gemma 3 family. TranslateGemma models handle translation across 55 languages and are small enough to deploy on laptops, desktops, and modest cloud GPU environments.
|
|
|
|
### Original Models
|
|
- [google/translategemma-27b-it](https://huggingface.co/google/translategemma-27b-it)
|
|
- [google/translategemma-4b-it](https://huggingface.co/google/translategemma-4b-it)
|
|
|
|
### Optimized vLLM Models
|
|
- [Infomaniak-AI/vllm-translategemma-27b-it](https://huggingface.co/Infomaniak-AI/vllm-translategemma-27b-it)
|
|
- [Infomaniak-AI/vllm-translategemma-4b-it](https://huggingface.co/Infomaniak-AI/vllm-translategemma-4b-it)
|
|
|
|
### Why use the vLLM-optimized models?
|
|
|
|
The original Google models have compatibility issues with standard inference engines like vLLM. The optimized versions from Infomaniak-AI ([detailed changes](https://huggingface.co/Infomaniak-AI/vllm-translategemma-27b-it#changes-from-original-model)):
|
|
|
|
- **vLLM Compatibility**: Originals require custom JSON parameters (`source_lang_code`/`target_lang_code`). The optimized version uses string delimiters.
|
|
- **RoPE Simplification**: Originals use a complex RoPE configuration for sliding attention. Optimized uses a standard linear RoPE format (`factor: 8.0`).
|
|
- **EOS Token Fix**: Corrects the EOS token from `<end_of_turn>` to `<eos>`.
|
|
|
|
## Prerequisites
|
|
|
|
### Docker
|
|
```bash
|
|
docker pull vllm/vllm-openai:latest
|
|
```
|
|
|
|
## Deployment Configurations
|
|
|
|
Verified for both 4B and 27B:
|
|
```bash
|
|
docker run -itd --name google-translategemma-27b-it \
|
|
--ipc=host \
|
|
--network host \
|
|
--shm-size 16G \
|
|
--gpus all \
|
|
-v ~/.cache/huggingface:/root/.cache/huggingface \
|
|
vllm/vllm-openai:latest \
|
|
Infomaniak-AI/vllm-translategemma-27b-it \
|
|
--served-model-name translategemma-27b-it \
|
|
--gpu-memory-utilization 0.8 \
|
|
--host 0.0.0.0 \
|
|
--port 8000
|
|
```
|
|
|
|
## Client Usage
|
|
|
|
Tips:
|
|
- **Prompt Delimiters**: Encode language metadata directly in the content string:
|
|
`<<<source>>>{src_lang}<<<target>>>{tgt_lang}<<<text>>>{text}`
|
|
- **Language Codes**: ISO 639-1 Alpha-2 (e.g. `en`, `zh`) and regional variants (e.g. `en_US`, `zh_CN`).
|
|
- **Context Limit**: ~2K tokens.
|
|
|
|
### cURL
|
|
```bash
|
|
curl -X POST http://localhost:8000/v1/chat/completions \
|
|
-H "Content-Type: application/json" \
|
|
-d '{
|
|
"model": "translategemma-27b-it",
|
|
"messages": [{
|
|
"role": "user",
|
|
"content": "<<<source>>>en<<<target>>>zh<<<text>>>We distribute two models for language identification, which can recognize 176 languages."
|
|
}]
|
|
}'
|
|
```
|
|
|
|
## References
|
|
|
|
- [TranslateGemma collection](https://huggingface.co/collections/google/translategemma)
|
|
- [Infomaniak-AI optimized 27B](https://huggingface.co/Infomaniak-AI/vllm-translategemma-27b-it)
|
|
- [Infomaniak-AI optimized 4B](https://huggingface.co/Infomaniak-AI/vllm-translategemma-4b-it)
|