[Model] Remove ten deprecated model architectures (#53608)
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com> Co-authored-by: Xianbao QIAN <xianbao.qian@gmail.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Xianbao QIAN
Claude Opus 5
parent
a0f1b9ad05
commit
7ca336929c
@@ -719,7 +719,6 @@ Examples:
|
||||
|
||||
Examples:
|
||||
|
||||
- Chameleon (appends `sep_token`): [vllm/model_executor/models/chameleon.py](../../../vllm/model_executor/models/chameleon.py)
|
||||
- Molmo2 (prepends `bos_token`): [vllm/model_executor/models/molmo2.py](../../../vllm/model_executor/models/molmo2.py)
|
||||
- Molmo (applies chat template which is not defined elsewhere): [vllm/model_executor/models/molmo.py](../../../vllm/model_executor/models/molmo.py)
|
||||
|
||||
|
||||
@@ -42,7 +42,6 @@ You can compute pairwise similarity scores to build a similarity matrix using th
|
||||
| `BgeM3EmbeddingModel` | BGE-M3 | `BAAI/bge-m3` | | |
|
||||
| `Gemma2Model`<sup>C</sup> | Gemma 2-based | `BAAI/bge-multilingual-gemma2`, etc. | ✅︎ | ✅︎ |
|
||||
| `Gemma3TextModel`<sup>C</sup> | Gemma 3-based | `google/embeddinggemma-300m`, etc. | ✅︎ | ✅︎ |
|
||||
| `GritLM` | GritLM | `parasail-ai/GritLM-7B-vllm`. | ✅︎ | ✅︎ |
|
||||
| `GteModel` | Arctic-Embed-2.0-M | `Snowflake/snowflake-arctic-embed-m-v2.0`. | | |
|
||||
| `GteNewModel` | mGTE-TRM (see note) | `Alibaba-NLP/gte-multilingual-base`, etc. | | |
|
||||
| `JinaEmbeddingsV5Model`<sup>C</sup> | Qwen3-decoder or EuroBERT-encoder backbone with task-specific LoRA adapters | `jinaai/jina-embeddings-v5-text-small`, `jinaai/jina-embeddings-v5-text-nano` (see note) | ✅︎ | ✅︎ |
|
||||
|
||||
@@ -358,7 +358,6 @@ th {
|
||||
| `AfmoeForCausalLM` | Afmoe | TBA | ✅︎ | ✅︎ |
|
||||
| `ApertusForCausalLM` | Apertus | `swiss-ai/Apertus-8B-2509`, `swiss-ai/Apertus-70B-Instruct-2509`, etc. | ✅︎ | ✅︎ |
|
||||
| `ArceeForCausalLM` | Arcee (AFM) | `arcee-ai/AFM-4.5B-Base`, etc. | ✅︎ | ✅︎ |
|
||||
| `ArcticForCausalLM` | Arctic | `Snowflake/snowflake-arctic-base`, `Snowflake/snowflake-arctic-instruct`, etc. | | ✅︎ |
|
||||
| `AXK1ForCausalLM` | A.X-K1 | `skt/A.X-K1`, etc. | | ✅︎ |
|
||||
| `BailingMoeForCausalLM` | Ling | `inclusionAI/Ling-lite-1.5`, `inclusionAI/Ling-plus`, etc. | ✅︎ | ✅︎ |
|
||||
| `BailingMoeV2ForCausalLM` | Ling | `inclusionAI/Ling-mini-2.0`, etc. | ✅︎ | ✅︎ |
|
||||
@@ -382,7 +381,6 @@ th {
|
||||
| `ExaoneForCausalLM` | EXAONE-3 | `LGAI-EXAONE/EXAONE-3.0-7.8B-Instruct`, etc. | ✅︎ | ✅︎ |
|
||||
| `ExaoneMoeForCausalLM` | K-EXAONE, K-EXAONE-2 | `LGAI-EXAONE/K-EXAONE-236B-A23B`, `LGAI-EXAONE/K-EXAONE-2.0-750B-A37B`, etc. | | |
|
||||
| `Exaone4ForCausalLM` | EXAONE-4 | `LGAI-EXAONE/EXAONE-4.0-32B`, etc. | ✅︎ | ✅︎ |
|
||||
| `Fairseq2LlamaForCausalLM` | Llama (fairseq2 format) | `mgleize/fairseq2-dummy-Llama-3.2-1B`, etc. | ✅︎ | ✅︎ |
|
||||
| `FalconForCausalLM` | Falcon | `tiiuae/falcon-7b`, `tiiuae/falcon-40b`, `tiiuae/falcon-rw-7b`, etc. | | ✅︎ |
|
||||
| `FalconMambaForCausalLM` | FalconMamba | `tiiuae/falcon-mamba-7b`, `tiiuae/falcon-mamba-7b-instruct`, etc. | | ✅︎ |
|
||||
| `FalconH1ForCausalLM` | Falcon-H1 | `tiiuae/Falcon-H1-34B-Base`, `tiiuae/Falcon-H1-34B-Instruct`, etc. | ✅︎ | ✅︎ |
|
||||
@@ -407,7 +405,6 @@ th {
|
||||
| `GraniteMoeSharedForCausalLM` | Granite MoE Shared | `ibm-research/moe-7b-1b-active-shared-experts` (test model) | ✅︎ | ✅︎ |
|
||||
| `GraniteMoeSWAForCausalLM` | Granite MoE SWA | `ibm-granite/granite-swash-3b-a600m` | ✅︎ | ✅︎ |
|
||||
| `GraniteSWAForCausalLM` | Granite SWA | `ibm-granite/granite-swash-2b` | ✅︎ | ✅︎ |
|
||||
| `GritLM` | GritLM | `parasail-ai/GritLM-7B-vllm`. | ✅︎ | ✅︎ |
|
||||
| `HrmTextForCausalLM` | HRM-Text | `sapientinc/HRM-Text-1B`, etc. | | |
|
||||
| `HunYuanDenseV1ForCausalLM` | Hunyuan Dense | `tencent/Hunyuan-7B-Instruct` | ✅︎ | ✅︎ |
|
||||
| `HunYuanMoEV1ForCausalLM` | Hunyuan-A13B | `tencent/Hunyuan-A13B-Instruct`, `tencent/Hunyuan-A13B-Pretrain`, `tencent/Hunyuan-A13B-Instruct-FP8`, etc. | ✅︎ | ✅︎ |
|
||||
@@ -437,7 +434,6 @@ th {
|
||||
| `MistralForCausalLM` | Ministral-3, Mistral, Mistral-Instruct | `mistralai/Ministral-3-3B-Instruct-2512`, `mistralai/Mistral-7B-v0.1`, `mistralai/Mistral-7B-Instruct-v0.1`, etc. | ✅︎ | ✅︎ |
|
||||
| `MistralLarge3ForCausalLM` | Mistral-Large-3-675B-Base-2512, Mistral-Large-3-675B-Instruct-2512 | `mistralai/Mistral-Large-3-675B-Base-2512`, `mistralai/Mistral-Large-3-675B-Instruct-2512`, etc. | ✅︎ | ✅︎ |
|
||||
| `MixtralForCausalLM` | Mixtral-8x7B, Mixtral-8x7B-Instruct | `mistralai/Mixtral-8x7B-v0.1`, `mistralai/Mixtral-8x7B-Instruct-v0.1`, `mistral-community/Mixtral-8x22B-v0.1`, etc. | ✅︎ | ✅︎ |
|
||||
| `MPTForCausalLM` | MPT, MPT-Instruct, MPT-Chat, MPT-StoryWriter | `mosaicml/mpt-7b`, `mosaicml/mpt-7b-storywriter`, `mosaicml/mpt-30b`, etc. | | ✅︎ |
|
||||
| `NemotronForCausalLM` | Nemotron-3, Nemotron-4, Minitron | `nvidia/Minitron-8B-Base`, `mgoin/Nemotron-4-340B-Base-hf-FP8`, etc. | ✅︎ | ✅︎ |
|
||||
| `NemotronHForCausalLM` | Nemotron-H | `nvidia/Nemotron-H-8B-Base-8K`, `nvidia/Nemotron-H-47B-Base-8K`, `nvidia/Nemotron-H-56B-Base-8K`, etc. | ✅︎ | ✅︎ |
|
||||
| `Olmo3ForCausalLM` | OLMo3 | `allenai/Olmo-3-7B-Instruct`, `allenai/Olmo-3-32B-Think`, etc. | ✅︎ | ✅︎ |
|
||||
@@ -458,14 +454,12 @@ th {
|
||||
| `Qwen3ForCausalLM` | Qwen3 | `Qwen/Qwen3-8B`, etc. | ✅︎ | ✅︎ |
|
||||
| `Qwen3MoeForCausalLM` | Qwen3MoE | `Qwen/Qwen3-30B-A3B`, etc. | ✅︎ | ✅︎ |
|
||||
| `Qwen3NextForCausalLM` | Qwen3NextMoE | `Qwen/Qwen3-Next-80B-A3B-Instruct`, etc. | ✅︎ | ✅︎ |
|
||||
| `RWForCausalLM` | Falcon RW | `tiiuae/falcon-40b`, etc. | | ✅︎ |
|
||||
| `Rnj1ForCausalLM` | Rnj1 | `EssentialAI/rnj-1-instruct`, etc. | | |
|
||||
| `SarvamMoEForCausalLM` | Sarvam 2 | `sarvamai/sarvam2-30b-a3b`, etc. | ✅︎ | ✅︎ |
|
||||
| `SarvamMLAForCausalLM` | Sarvam 2 | `sarvamai/sarvam2-105b-a9b`, etc. | | ✅︎ |
|
||||
| `SeedOssForCausalLM` | SeedOss | `ByteDance-Seed/Seed-OSS-36B-Instruct`, etc. | ✅︎ | ✅︎ |
|
||||
| `SolarForCausalLM` | Solar Pro | `upstage/solar-pro-preview-instruct`, etc. | ✅︎ | ✅︎ |
|
||||
| `StableLmForCausalLM` | StableLM | `stabilityai/stablelm-3b-4e1t`, `stabilityai/stablelm-base-alpha-7b-v2`, etc. | | |
|
||||
| `StableLMEpochForCausalLM` | StableLM Epoch | `stabilityai/stablelm-zephyr-3b`, etc. | | ✅︎ |
|
||||
| `Step1ForCausalLM` | Step-Audio | `stepfun-ai/Step-Audio-EditX`, etc. | ✅︎ | ✅︎ |
|
||||
| `Step3p5ForCausalLM` | Step-3.5-flash | `stepfun-ai/Step-3.5-Flash`, etc. | | ✅︎ |
|
||||
| `TeleChat2ForCausalLM` | TeleChat2 | `Tele-AI/TeleChat2-3B`, `Tele-AI/TeleChat2-7B`, `Tele-AI/TeleChat2-35B`, etc. | ✅︎ | ✅︎ |
|
||||
@@ -527,8 +521,6 @@ These models primarily accept the [`LLM.generate`](./generative_models.md#llmgen
|
||||
| `BagelForConditionalGeneration` | BAGEL | T + I<sup>+</sup> | `ByteDance-Seed/BAGEL-7B-MoT` | ✅︎ | ✅︎ |
|
||||
| `BeeForConditionalGeneration` | Bee-8B | T + I<sup>E+</sup> | `Open-Bee/Bee-8B-RL`, `Open-Bee/Bee-8B-SFT` | | ✅︎ |
|
||||
| `Blip2ForConditionalGeneration` | BLIP-2 | T + I<sup>E</sup> | `Salesforce/blip2-opt-2.7b`, `Salesforce/blip2-opt-6.7b`, etc. | ✅︎ | ✅︎ |
|
||||
| `ChameleonForConditionalGeneration` | Chameleon | T + I | `facebook/chameleon-7b`, etc. | | ✅︎ |
|
||||
| `CheersForConditionalGeneration` | Cheers | T + I | `ai9stars/Cheers` | | ✅︎ |
|
||||
| `Cohere2VisionForConditionalGeneration` | Command A Vision, Command-A+ | T + I<sup>+</sup> | `CohereLabs/command-a-vision-07-2025`, `CohereLabs/command-a-plus-05-2026`, etc. | | ✅︎ |
|
||||
| `Cosmos3ForConditionalGeneration` | Cosmos3 (understanding tower) | T + I<sup>E+</sup> + V<sup>E+</sup> | `nvidia/Cosmos3-Nano`, `nvidia/Cosmos3-Super` | | ✅︎ |
|
||||
| `Cosmos3EdgeForConditionalGeneration` | Cosmos3-Edge (understanding tower) | T + I<sup>E+</sup> + V<sup>E+</sup> | `nvidia/Cosmos3-Edge` | | ✅︎ |
|
||||
@@ -549,7 +541,6 @@ These models primarily accept the [`LLM.generate`](./generative_models.md#llmgen
|
||||
| `Granite4VisionForConditionalGeneration` | Granite 4 Vision | T + I<sup>E+</sup> | `ibm-granite/granite-4.1-3b-vision`, etc. | ✅︎ | ✅︎ |
|
||||
| `GraniteSpeechForConditionalGeneration` | Granite Speech | T + A | `ibm-granite/granite-speech-3.3-8b` | ✅︎ | ✅︎ |
|
||||
| `GraniteSpeechPlusForConditionalGeneration` | Granite Speech Plus | T + A | `ibm-granite/granite-speech-4.1-2b-plus` | ✅︎ | ✅︎ |
|
||||
| `HCXVisionForCausalLM` | HyperCLOVAX-SEED-Vision-Instruct-3B | T + I<sup>+</sup> + V<sup>+</sup> | `naver-hyperclovax/HyperCLOVAX-SEED-Vision-Instruct-3B` | | |
|
||||
| `HCXVisionV2ForCausalLM` | HyperCLOVAX-SEED-Think-32B | T + I<sup>+</sup> + V<sup>+</sup> | `naver-hyperclovax/HyperCLOVAX-SEED-Think-32B` | | |
|
||||
| `H2OVLChatModel` | H2OVL | T + I<sup>E+</sup> | `h2oai/h2ovl-mississippi-800m`, `h2oai/h2ovl-mississippi-2b`, etc. | ✅︎ | ✅︎ |
|
||||
| `HunYuanVLForConditionalGeneration` | HunyuanOCR | T + I<sup>E+</sup> | `tencent/HunyuanOCR`, etc. | ✅︎ | ✅︎ |
|
||||
@@ -700,7 +691,6 @@ Speech2Text models trained specifically for Automatic Speech Recognition.
|
||||
| ------------ | ------ | ----------------- | -------------------- | ------------------------- |
|
||||
| `CohereAsrForConditionalGeneration` | Cohere-Transcribe | `CohereLabs/cohere-transcribe-03-2026` | | |
|
||||
| `FireRedASR2ForConditionalGeneration` | FireRedASR2 | `allendou/FireRedASR2-LLM-vllm`, etc. | | |
|
||||
| `FireRedLIDForConditionalGeneration` | FireRedLID | `PatchyTisa/FireRedLID-vllm`, etc. | | |
|
||||
| `FunASRForConditionalGeneration` | FunASR | `allendou/Fun-ASR-Nano-2512-vllm`, etc. | | |
|
||||
| `Gemma3nForConditionalGeneration` | Gemma3n | `google/gemma-3n-E2B-it`, `google/gemma-3n-E4B-it`, etc. | | |
|
||||
| `GlmAsrForConditionalGeneration` | GLM-ASR | `zai-org/GLM-ASR-Nano-2512` | ✅︎ | ✅︎ |
|
||||
|
||||
@@ -496,30 +496,9 @@ def run_whisper(question: str, audio_count: int) -> ModelRequestData:
|
||||
)
|
||||
|
||||
|
||||
# FireRedLID
|
||||
def run_fireredlid(question: str, audio_count: int) -> ModelRequestData:
|
||||
assert audio_count == 1, "FireRedLID only supports single audio input per prompt"
|
||||
model_name = "PatchyTisa/FireRedLID-vllm"
|
||||
|
||||
prompt = "<sos>"
|
||||
|
||||
engine_args = EngineArgs(
|
||||
model=model_name,
|
||||
max_model_len=8,
|
||||
max_num_seqs=5,
|
||||
limit_mm_per_prompt={"audio": audio_count},
|
||||
)
|
||||
|
||||
return ModelRequestData(
|
||||
engine_args=engine_args,
|
||||
prompt=prompt,
|
||||
)
|
||||
|
||||
|
||||
model_example_map = {
|
||||
"audioflamingo3": run_audioflamingo3,
|
||||
"cohere_asr": run_cohere_asr,
|
||||
"fireredlid": run_fireredlid,
|
||||
"funaudiochat": run_funaudiochat,
|
||||
"gemma3n": run_gemma3n,
|
||||
"glmasr": run_glmasr,
|
||||
|
||||
@@ -95,51 +95,8 @@ def run_fireredasr2():
|
||||
)
|
||||
|
||||
|
||||
def run_fireredlid():
|
||||
"""
|
||||
FireRedLID – Language Identification model.
|
||||
|
||||
This encoder-decoder model identifies the spoken language of an audio
|
||||
clip. It outputs at most 2 tokens representing the detected language
|
||||
(e.g. "en", "zh mandarin").
|
||||
"""
|
||||
engine_args = EngineArgs(
|
||||
model="PatchyTisa/FireRedLID-vllm",
|
||||
max_model_len=8,
|
||||
max_num_seqs=16,
|
||||
limit_mm_per_prompt={"audio": 1},
|
||||
)
|
||||
|
||||
prompts = [
|
||||
{ # Test explicit encoder/decoder prompt
|
||||
"encoder_prompt": {
|
||||
"prompt": "",
|
||||
"multi_modal_data": {
|
||||
"audio": AudioAsset("mary_had_lamb").audio_and_sample_rate,
|
||||
},
|
||||
},
|
||||
"decoder_prompt": "<sos>",
|
||||
},
|
||||
{ # Another audio sample
|
||||
"encoder_prompt": {
|
||||
"prompt": "",
|
||||
"multi_modal_data": {
|
||||
"audio": AudioAsset("winning_call").audio_and_sample_rate,
|
||||
},
|
||||
},
|
||||
"decoder_prompt": "<sos>",
|
||||
},
|
||||
]
|
||||
|
||||
return ModelRequestData(
|
||||
engine_args=engine_args,
|
||||
prompts=prompts,
|
||||
)
|
||||
|
||||
|
||||
model_example_map = {
|
||||
"fireredasr2": run_fireredasr2,
|
||||
"fireredlid": run_fireredlid,
|
||||
"whisper": run_whisper,
|
||||
}
|
||||
|
||||
|
||||
@@ -365,53 +365,6 @@ def load_hunyuan_vl(question: str, image_urls: list[str]) -> ModelRequestData:
|
||||
)
|
||||
|
||||
|
||||
def load_hyperclovax_seed_vision(
|
||||
question: str, image_urls: list[str]
|
||||
) -> ModelRequestData:
|
||||
model_name = "naver-hyperclovax/HyperCLOVAX-SEED-Vision-Instruct-3B"
|
||||
tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
|
||||
|
||||
engine_args = EngineArgs(
|
||||
model=model_name,
|
||||
trust_remote_code=True,
|
||||
max_model_len=16384,
|
||||
limit_mm_per_prompt={"image": len(image_urls)},
|
||||
)
|
||||
|
||||
message = {"role": "user", "content": list()}
|
||||
for _image_url in image_urls:
|
||||
message["content"].append(
|
||||
{
|
||||
"type": "image",
|
||||
"image": _image_url,
|
||||
"ocr": "",
|
||||
"lens_keywords": "",
|
||||
"lens_local_keywords": "",
|
||||
}
|
||||
)
|
||||
message["content"].append(
|
||||
{
|
||||
"type": "text",
|
||||
"text": question,
|
||||
}
|
||||
)
|
||||
|
||||
prompt = tokenizer.apply_chat_template(
|
||||
[
|
||||
message,
|
||||
],
|
||||
tokenize=False,
|
||||
add_generation_prompt=True,
|
||||
)
|
||||
|
||||
return ModelRequestData(
|
||||
engine_args=engine_args,
|
||||
prompt=prompt,
|
||||
stop_token_ids=None,
|
||||
image_data=[fetch_image(url) for url in image_urls],
|
||||
)
|
||||
|
||||
|
||||
def load_idefics3(question: str, image_urls: list[str]) -> ModelRequestData:
|
||||
model_name = "HuggingFaceM4/Idefics3-8B-Llama3"
|
||||
|
||||
@@ -1397,7 +1350,6 @@ model_example_map = {
|
||||
"granite4_vision": load_granite4_vision,
|
||||
"h2ovl_chat": load_h2ovl,
|
||||
"hunyuan_vl": load_hunyuan_vl,
|
||||
"hyperclovax_seed_vision": load_hyperclovax_seed_vision,
|
||||
"idefics3": load_idefics3,
|
||||
"interns1": load_interns1,
|
||||
"internvl_chat": load_internvl,
|
||||
|
||||
@@ -140,51 +140,6 @@ def run_blip2(questions: list[str], modality: str) -> ModelRequestData:
|
||||
)
|
||||
|
||||
|
||||
# Chameleon
|
||||
def run_chameleon(questions: list[str], modality: str) -> ModelRequestData:
|
||||
assert modality == "image"
|
||||
|
||||
prompts = [f"{question}<image>" for question in questions]
|
||||
engine_args = EngineArgs(
|
||||
model="facebook/chameleon-7b",
|
||||
max_model_len=4096,
|
||||
max_num_seqs=2,
|
||||
limit_mm_per_prompt={modality: 1},
|
||||
)
|
||||
|
||||
return ModelRequestData(
|
||||
engine_args=engine_args,
|
||||
prompts=prompts,
|
||||
)
|
||||
|
||||
|
||||
# Cheers
|
||||
def run_cheers(questions: list[str], modality: str) -> ModelRequestData:
|
||||
assert modality == "image"
|
||||
model_name = "ai9stars/Cheers"
|
||||
|
||||
engine_args = EngineArgs(
|
||||
model=model_name,
|
||||
trust_remote_code=True,
|
||||
max_model_len=4096,
|
||||
limit_mm_per_prompt={modality: 1},
|
||||
)
|
||||
|
||||
prompts = [
|
||||
(
|
||||
f"<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n"
|
||||
f"<|im_start|>user\n<|image_pad|>{question}<|im_end|>\n"
|
||||
f"<|im_start|>assistant\n"
|
||||
)
|
||||
for question in questions
|
||||
]
|
||||
|
||||
return ModelRequestData(
|
||||
engine_args=engine_args,
|
||||
prompts=prompts,
|
||||
)
|
||||
|
||||
|
||||
def run_command_a_vision(questions: list[str], modality: str) -> ModelRequestData:
|
||||
assert modality == "image"
|
||||
|
||||
@@ -801,109 +756,6 @@ def run_hunyuan_vl(questions: list[str], modality: str) -> ModelRequestData:
|
||||
)
|
||||
|
||||
|
||||
# naver-hyperclovax/HyperCLOVAX-SEED-Vision-Instruct-3B
|
||||
def run_hyperclovax_seed_vision(
|
||||
questions: list[str], modality: str
|
||||
) -> ModelRequestData:
|
||||
model_name = "naver-hyperclovax/HyperCLOVAX-SEED-Vision-Instruct-3B"
|
||||
tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
|
||||
|
||||
mm_limit = {"image": 1, "video": 1} if modality == "image+video" else {modality: 1}
|
||||
engine_args = EngineArgs(
|
||||
model=model_name,
|
||||
trust_remote_code=True,
|
||||
max_model_len=16384 if modality in ("video", "image+video") else 8192,
|
||||
limit_mm_per_prompt=mm_limit,
|
||||
)
|
||||
|
||||
messages = list()
|
||||
for question in questions:
|
||||
if modality == "image":
|
||||
"""
|
||||
ocr: List the words in the image in raster order.
|
||||
Even if the word order feels unnatural for reading,
|
||||
the model will handle it as long as it follows raster order.
|
||||
e.g. "Naver, CLOVA, bigshane"
|
||||
lens_keywords: List the entity names in the image.
|
||||
e.g. "iPhone"
|
||||
lens_local_keywords: List the entity names with quads in the image.
|
||||
e.g. "[0.07, 0.21, 0.92, 0.90] iPhone"
|
||||
"""
|
||||
messages.append(
|
||||
[
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{
|
||||
"type": "image",
|
||||
"ocr": "",
|
||||
"lens_keywords": "",
|
||||
"lens_local_keywords": "",
|
||||
},
|
||||
{
|
||||
"type": "text",
|
||||
"text": question,
|
||||
},
|
||||
],
|
||||
}
|
||||
]
|
||||
)
|
||||
elif modality == "video":
|
||||
messages.append(
|
||||
[
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{
|
||||
"type": "video",
|
||||
},
|
||||
{
|
||||
"type": "text",
|
||||
"text": question,
|
||||
},
|
||||
],
|
||||
}
|
||||
]
|
||||
)
|
||||
elif modality == "image+video":
|
||||
messages.append(
|
||||
[
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{
|
||||
"type": "image",
|
||||
"ocr": "",
|
||||
"lens_keywords": "",
|
||||
"lens_local_keywords": "",
|
||||
},
|
||||
{
|
||||
"type": "video",
|
||||
},
|
||||
{
|
||||
"type": "text",
|
||||
"text": question,
|
||||
},
|
||||
],
|
||||
}
|
||||
]
|
||||
)
|
||||
else:
|
||||
raise ValueError(f"Unsupported modality: {modality}")
|
||||
|
||||
prompts = tokenizer.apply_chat_template(
|
||||
messages,
|
||||
tokenize=False,
|
||||
add_generation_prompt=True,
|
||||
)
|
||||
|
||||
return ModelRequestData(
|
||||
engine_args=engine_args,
|
||||
prompts=prompts,
|
||||
stop_token_ids=None,
|
||||
)
|
||||
|
||||
|
||||
# Idefics3-8B-Llama3
|
||||
def run_idefics3(questions: list[str], modality: str) -> ModelRequestData:
|
||||
assert modality == "image"
|
||||
@@ -2328,10 +2180,8 @@ def run_step_vl(questions: list[str], modality: str) -> ModelRequestData:
|
||||
model_example_map = {
|
||||
"aria": run_aria,
|
||||
"bagel": run_bagel,
|
||||
"cheers": run_cheers,
|
||||
"bee": run_bee,
|
||||
"blip-2": run_blip2,
|
||||
"chameleon": run_chameleon,
|
||||
"command_a_vision": run_command_a_vision,
|
||||
"deepseek_vl_v2": run_deepseek_vl2,
|
||||
"deepseek_ocr": run_deepseek_ocr,
|
||||
@@ -2350,7 +2200,6 @@ model_example_map = {
|
||||
"glm_ocr": run_glm_ocr,
|
||||
"h2ovl_chat": run_h2ovl,
|
||||
"hunyuan_vl": run_hunyuan_vl,
|
||||
"hyperclovax_seed_vision": run_hyperclovax_seed_vision,
|
||||
"idefics3": run_idefics3,
|
||||
"interns1": run_interns1,
|
||||
"interns1_pro": run_interns1_pro,
|
||||
|
||||
@@ -1,193 +0,0 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
|
||||
"""
|
||||
Language Identification (LID) demo using the FireRedLID model on vLLM.
|
||||
|
||||
FireRedLID is an audio encoder-decoder model that identifies the spoken
|
||||
language of an audio clip. Unlike ASR models that output full transcriptions,
|
||||
FireRedLID outputs at most 2 tokens representing the detected language
|
||||
(e.g. "en", "zh mandarin").
|
||||
|
||||
Start the vLLM server:
|
||||
|
||||
vllm serve PatchyTisa/FireRedLID-vllm
|
||||
|
||||
Then run this script:
|
||||
|
||||
# Use the built-in sample audio
|
||||
python examples/speech_to_text/lid/openai_lid_client.py
|
||||
|
||||
# Use your own audio file(s)
|
||||
python examples/speech_to_text/lid/openai_lid_client.py \
|
||||
--audio_paths audio_en.wav audio_zh.wav audio_fr.wav
|
||||
|
||||
# Batch-identify multiple files in one run
|
||||
python examples/speech_to_text/lid/openai_lid_client.py \
|
||||
--audio_paths /path/to/dir/*.wav
|
||||
|
||||
Requirements:
|
||||
- vLLM with audio support
|
||||
- openai Python SDK
|
||||
- kaldi_native_fbank (pulled in by the model)
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
|
||||
from openai import OpenAI
|
||||
|
||||
from vllm.assets.audio import AudioAsset
|
||||
|
||||
# ──────────────────────────────────────────────────────────────────────
|
||||
# Helpers
|
||||
# ──────────────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def identify_language(
|
||||
audio_path: str,
|
||||
client: OpenAI,
|
||||
model: str,
|
||||
) -> str:
|
||||
"""
|
||||
Send a single audio file to the vLLM transcription endpoint and return
|
||||
the detected language tag.
|
||||
|
||||
FireRedLID re-uses the OpenAI-compatible ``/v1/audio/transcriptions``
|
||||
endpoint. The "transcription" it returns is actually the language label
|
||||
(e.g. ``"en"`` or ``"zh mandarin"``).
|
||||
"""
|
||||
with open(audio_path, "rb") as f:
|
||||
result = client.audio.transcriptions.create(
|
||||
file=f,
|
||||
model=model,
|
||||
response_format="json",
|
||||
temperature=0.0,
|
||||
)
|
||||
return result.text.strip()
|
||||
|
||||
|
||||
def identify_language_raw(
|
||||
audio_path: str,
|
||||
model: str,
|
||||
api_base: str,
|
||||
) -> str:
|
||||
"""
|
||||
Same as :func:`identify_language` but uses raw HTTP so that the demo
|
||||
works without the ``openai`` SDK (useful for quick debugging).
|
||||
"""
|
||||
import requests
|
||||
|
||||
url = f"{api_base}/audio/transcriptions"
|
||||
with open(audio_path, "rb") as f:
|
||||
files = {"file": (os.path.basename(audio_path), f)}
|
||||
data = {
|
||||
"model": model,
|
||||
"response_format": "json",
|
||||
}
|
||||
resp = requests.post(url, files=files, data=data)
|
||||
resp.raise_for_status()
|
||||
return resp.json()["text"].strip()
|
||||
|
||||
|
||||
def identify_language_streaming(
|
||||
audio_path: str,
|
||||
model: str,
|
||||
api_base: str,
|
||||
) -> str:
|
||||
"""
|
||||
Streaming variant – demonstrates the streaming transcription endpoint.
|
||||
For a 1-2 token output the stream finishes almost instantly, but this
|
||||
shows that the API path works end-to-end.
|
||||
"""
|
||||
import requests
|
||||
|
||||
url = f"{api_base}/audio/transcriptions"
|
||||
with open(audio_path, "rb") as f:
|
||||
files = {"file": (os.path.basename(audio_path), f)}
|
||||
data = {
|
||||
"stream": "true",
|
||||
"model": model,
|
||||
"response_format": "json",
|
||||
}
|
||||
response = requests.post(url, files=files, data=data, stream=True)
|
||||
response.raise_for_status()
|
||||
|
||||
tokens: list[str] = []
|
||||
for chunk in response.iter_lines(
|
||||
chunk_size=8192, decode_unicode=False, delimiter=b"\n"
|
||||
):
|
||||
if not chunk:
|
||||
continue
|
||||
payload = json.loads(chunk[len("data: ") :].decode("utf-8"))
|
||||
choice = payload["choices"][0]
|
||||
delta = choice.get("delta", {}).get("content", "")
|
||||
if delta:
|
||||
tokens.append(delta)
|
||||
if choice.get("finish_reason") is not None:
|
||||
break
|
||||
|
||||
return "".join(tokens).strip()
|
||||
|
||||
|
||||
# ──────────────────────────────────────────────────────────────────────
|
||||
# Main
|
||||
# ──────────────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def main(args: argparse.Namespace) -> None:
|
||||
api_base = args.api_base.rstrip("/")
|
||||
client = OpenAI(api_key="EMPTY", base_url=api_base)
|
||||
model = client.models.list().data[0].id
|
||||
print(f"Model : {model}")
|
||||
print(f"Server: {api_base}\n")
|
||||
|
||||
# Resolve audio paths ------------------------------------------------
|
||||
if args.audio_paths:
|
||||
audio_paths = args.audio_paths
|
||||
else:
|
||||
# Fall back to the built-in vLLM sample audios (both are English).
|
||||
audio_paths = [
|
||||
str(AudioAsset("mary_had_lamb").get_local_path()),
|
||||
str(AudioAsset("winning_call").get_local_path()),
|
||||
]
|
||||
|
||||
# Run LID for each file ----------------------------------------------
|
||||
print(f"{'Audio File':<50} {'Language (sync)':<20} {'Language (stream)'}")
|
||||
print("-" * 90)
|
||||
|
||||
for path in audio_paths:
|
||||
basename = os.path.basename(path)
|
||||
|
||||
# 1) Synchronous via OpenAI SDK
|
||||
lang_sync = identify_language(path, client, model)
|
||||
|
||||
# 2) Streaming via raw HTTP
|
||||
lang_stream = identify_language_streaming(path, model, api_base)
|
||||
|
||||
print(f"{basename:<50} {lang_sync:<20} {lang_stream}")
|
||||
|
||||
print()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
parser = argparse.ArgumentParser(
|
||||
description="FireRedLID – Language Identification demo via vLLM",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--audio_paths",
|
||||
nargs="+",
|
||||
default=None,
|
||||
help=(
|
||||
"One or more audio files to identify. "
|
||||
"If omitted, uses vLLM's built-in sample audios."
|
||||
),
|
||||
)
|
||||
parser.add_argument(
|
||||
"--api_base",
|
||||
type=str,
|
||||
default="http://localhost:8000/v1",
|
||||
help="vLLM API base URL (default: http://localhost:8000/v1)",
|
||||
)
|
||||
args = parser.parse_args()
|
||||
main(args)
|
||||
@@ -10,7 +10,7 @@ pytest-timeout
|
||||
# testing utils
|
||||
backoff # required for phi4mm test
|
||||
blobfile # required for kimi-vl test
|
||||
einops # required for MPT, qwen-vl and Mamba
|
||||
einops # required for qwen-vl and Mamba
|
||||
httpx
|
||||
librosa # required for audio tests
|
||||
vocos # required for minicpmo_26 test
|
||||
|
||||
@@ -84,23 +84,6 @@
|
||||
"is_multimodal_model": false,
|
||||
"dtype": "torch.bfloat16"
|
||||
},
|
||||
"mosaicml/mpt-7b": {
|
||||
"architectures": [
|
||||
"MPTForCausalLM"
|
||||
],
|
||||
"model_type": "mpt",
|
||||
"text_model_type": "mpt",
|
||||
"hidden_size": 4096,
|
||||
"total_num_hidden_layers": 32,
|
||||
"total_num_attention_heads": 32,
|
||||
"head_size": 128,
|
||||
"vocab_size": 50432,
|
||||
"total_num_kv_heads": 32,
|
||||
"num_experts": 0,
|
||||
"is_deepseek_mla": false,
|
||||
"is_multimodal_model": false,
|
||||
"dtype": "torch.bfloat16"
|
||||
},
|
||||
"databricks/dbrx-instruct": {
|
||||
"architectures": [
|
||||
"DbrxForCausalLM"
|
||||
|
||||
@@ -38,8 +38,6 @@ BASE_MODELS_TO_TEST = [
|
||||
# (NonGeoDataset import error). Tested in model initialization tests.
|
||||
# "ibm-nasa-geospatial/Prithvi-EO-2.0-300M-TL-Sen1Floods11",
|
||||
"Zyphra/Zamba2-7B-instruct",
|
||||
# FIXME: mosaicml/mpt-7b has been deleted
|
||||
# "mosaicml/mpt-7b",
|
||||
# FIXME: databricks/dbrx-instruct has been deleted
|
||||
# "databricks/dbrx-instruct",
|
||||
"tiiuae/falcon-7b",
|
||||
|
||||
@@ -102,7 +102,6 @@ class PPTestSettings:
|
||||
|
||||
TEXT_GENERATION_MODELS = {
|
||||
# [Decoder-only]
|
||||
"Snowflake/snowflake-arctic-instruct": PPTestSettings.fast(load_format="dummy"),
|
||||
"bigscience/bloomz-1b1": PPTestSettings.fast(),
|
||||
"zai-org/chatglm3-6b": PPTestSettings.fast(),
|
||||
"CohereLabs/c4ai-command-r-v01": PPTestSettings.fast(load_format="dummy"),
|
||||
@@ -131,7 +130,6 @@ TEXT_GENERATION_MODELS = {
|
||||
# "mistralai/Mistral-7B-Instruct-v0.1": PPTestSettings.fast(),
|
||||
"state-spaces/mamba-130m-hf": PPTestSettings.fast(),
|
||||
"mistralai/Mixtral-8x7B-Instruct-v0.1": PPTestSettings.fast(load_format="dummy"),
|
||||
"mosaicml/mpt-7b": PPTestSettings.fast(),
|
||||
"nvidia/Minitron-8B-Base": PPTestSettings.fast(),
|
||||
"allenai/OLMo-1B-hf": PPTestSettings.fast(),
|
||||
"allenai/OLMo-2-0425-1B": PPTestSettings.fast(),
|
||||
@@ -165,7 +163,6 @@ EMBEDDING_MODELS = { # type: ignore[var-annotated]
|
||||
MULTIMODAL_MODELS = {
|
||||
# [Decoder-only]
|
||||
"Salesforce/blip2-opt-6.7b": PPTestSettings.fast(),
|
||||
"facebook/chameleon-7b": PPTestSettings.fast(),
|
||||
"zai-org/glm-4v-9b": PPTestSettings.fast(),
|
||||
"OpenGVLab/InternVL3-1B": PPTestSettings.fast(),
|
||||
"llava-hf/llava-1.5-7b-hf": PPTestSettings.fast(),
|
||||
|
||||
@@ -1,188 +0,0 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
|
||||
import numpy as np
|
||||
import pytest
|
||||
from scipy.spatial.distance import cosine
|
||||
|
||||
from vllm import LLM, SamplingParams
|
||||
from vllm.config import ModelConfig
|
||||
|
||||
from ....utils import ROCM_EXTRA_ARGS, RemoteOpenAIServer
|
||||
from .embed_utils import run_client_embeddings
|
||||
|
||||
MODEL_NAME = "parasail-ai/GritLM-7B-vllm"
|
||||
MAX_MODEL_LEN = 4000
|
||||
ATOL = 2.3e-3
|
||||
|
||||
|
||||
def _arr(arr):
|
||||
"""
|
||||
Convert a list of integers to an array of integers.
|
||||
"""
|
||||
return np.array(arr)
|
||||
|
||||
|
||||
def test_find_array():
|
||||
from vllm.model_executor.models.gritlm import GritLMMeanPool
|
||||
|
||||
model_config = ModelConfig(
|
||||
MODEL_NAME,
|
||||
runner="pooling",
|
||||
dtype="bfloat16",
|
||||
seed=0,
|
||||
)
|
||||
pooling = GritLMMeanPool(model_config=model_config)
|
||||
|
||||
arr = _arr([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])
|
||||
|
||||
assert pooling._find_array(arr, _arr([3, 4, 5]), start_idx=0) == 3
|
||||
assert pooling._find_array(arr, _arr([3, 4, 5]), start_idx=1) == 3
|
||||
assert pooling._find_array(arr, _arr([3, 4, 5]), start_idx=5) == -1
|
||||
assert pooling._find_array(arr, _arr([3, 4, 5]), end_idx=3) == -1
|
||||
assert pooling._find_array(arr, _arr([3, 4, 5]), end_idx=4) == 3
|
||||
assert pooling._find_array(arr, _arr([3, 5]), start_idx=0) == -1
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
pooling._find_array(arr, _arr([3, 4, 5]), start_idx=-1)
|
||||
|
||||
|
||||
def run_llm_encode(
|
||||
llm: LLM,
|
||||
queries: list[str],
|
||||
instruction: str,
|
||||
) -> list[list[float]]:
|
||||
outputs = llm.embed([instruction + q for q in queries])
|
||||
return [output.outputs.embedding for output in outputs]
|
||||
|
||||
|
||||
def gritlm_instruction(instruction):
|
||||
return (
|
||||
"<|user|>\n" + instruction + "\n<|embed|>\n" if instruction else "<|embed|>\n"
|
||||
)
|
||||
|
||||
|
||||
def get_test_data():
|
||||
"""
|
||||
Grabbed this test data and the expected values from
|
||||
README.md in https://github.com/ContextualAI/gritlm
|
||||
"""
|
||||
q_instruction = gritlm_instruction(
|
||||
"Given a scientific paper title, retrieve the paper's abstract",
|
||||
)
|
||||
queries = [
|
||||
"Bitcoin: A Peer-to-Peer Electronic Cash System",
|
||||
"Generative Representational Instruction Tuning",
|
||||
]
|
||||
|
||||
d_instruction = gritlm_instruction("")
|
||||
documents = [
|
||||
# ruff: noqa: E501
|
||||
"A purely peer-to-peer version of electronic cash would allow online payments to be sent directly from one party to another without going through a financial institution. Digital signatures provide part of the solution, but the main benefits are lost if a trusted third party is still required to prevent double-spending. We propose a solution to the double-spending problem using a peer-to-peer network. The network timestamps transactions by hashing them into an ongoing chain of hash-based proof-of-work, forming a record that cannot be changed without redoing the proof-of-work. The longest chain not only serves as proof of the sequence of events witnessed, but proof that it came from the largest pool of CPU power. As long as a majority of CPU power is controlled by nodes that are not cooperating to attack the network, they'll generate the longest chain and outpace attackers. The network itself requires minimal structure. Messages are broadcast on a best effort basis, and nodes can leave and rejoin the network at will, accepting the longest proof-of-work chain as proof of what happened while they were gone.",
|
||||
"All text-based language problems can be reduced to either generation or embedding. Current models only perform well at one or the other. We introduce generative representational instruction tuning (GRIT) whereby a large language model is trained to handle both generative and embedding tasks by distinguishing between them through instructions. Compared to other open models, our resulting GritLM 7B sets a new state of the art on the Massive Text Embedding Benchmark (MTEB) and outperforms all models up to its size on a range of generative tasks. By scaling up further, GritLM 8X7B outperforms all open generative language models that we tried while still being among the best embedding models. Notably, we find that GRIT matches training on only generative or embedding data, thus we can unify both at no performance loss. Among other benefits, the unification via GRIT speeds up Retrieval-Augmented Generation (RAG) by > 60% for long documents, by no longer requiring separate retrieval and generation models. Models, code, etc. are freely available at https://github.com/ContextualAI/gritlm.",
|
||||
]
|
||||
|
||||
return queries, q_instruction, documents, d_instruction
|
||||
|
||||
|
||||
def validate_embed_output(q_rep: list[list[float]], d_rep: list[list[float]]):
|
||||
cosine_sim_q0_d0 = 1 - cosine(q_rep[0], d_rep[0])
|
||||
assert cosine_sim_q0_d0 == pytest.approx(0.609, abs=ATOL)
|
||||
|
||||
cosine_sim_q0_d1 = 1 - cosine(q_rep[0], d_rep[1])
|
||||
assert cosine_sim_q0_d1 == pytest.approx(0.101, abs=ATOL)
|
||||
|
||||
cosine_sim_q1_d0 = 1 - cosine(q_rep[1], d_rep[0])
|
||||
assert cosine_sim_q1_d0 == pytest.approx(0.120, abs=ATOL)
|
||||
|
||||
cosine_sim_q1_d1 = 1 - cosine(q_rep[1], d_rep[1])
|
||||
assert cosine_sim_q1_d1 == pytest.approx(0.534, abs=ATOL)
|
||||
|
||||
|
||||
def test_gritlm_offline_embedding(vllm_runner):
|
||||
queries, q_instruction, documents, d_instruction = get_test_data()
|
||||
|
||||
with vllm_runner(
|
||||
MODEL_NAME,
|
||||
runner="pooling",
|
||||
max_model_len=MAX_MODEL_LEN,
|
||||
) as vllm_model:
|
||||
llm = vllm_model.llm
|
||||
|
||||
d_rep = run_llm_encode(
|
||||
llm,
|
||||
documents,
|
||||
d_instruction,
|
||||
)
|
||||
q_rep = run_llm_encode(
|
||||
llm,
|
||||
queries,
|
||||
q_instruction,
|
||||
)
|
||||
|
||||
validate_embed_output(q_rep, d_rep)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_gritlm_api_server_embedding():
|
||||
queries, q_instruction, documents, d_instruction = get_test_data()
|
||||
|
||||
args = [
|
||||
"--runner",
|
||||
"pooling",
|
||||
"--max_model_len",
|
||||
str(MAX_MODEL_LEN),
|
||||
*ROCM_EXTRA_ARGS,
|
||||
]
|
||||
|
||||
with RemoteOpenAIServer(MODEL_NAME, args) as server:
|
||||
client_embedding = server.get_async_client()
|
||||
|
||||
d_rep = await run_client_embeddings(
|
||||
client_embedding,
|
||||
MODEL_NAME,
|
||||
documents,
|
||||
d_instruction,
|
||||
)
|
||||
q_rep = await run_client_embeddings(
|
||||
client_embedding,
|
||||
MODEL_NAME,
|
||||
queries,
|
||||
q_instruction,
|
||||
)
|
||||
|
||||
validate_embed_output(q_rep, d_rep)
|
||||
|
||||
|
||||
def test_gritlm_offline_generate(monkeypatch: pytest.MonkeyPatch, vllm_runner):
|
||||
input = "<|user|>\nWhat is the capital of France?\n<|assistant|>\n"
|
||||
|
||||
with vllm_runner(
|
||||
MODEL_NAME,
|
||||
runner="generate",
|
||||
max_model_len=MAX_MODEL_LEN,
|
||||
) as vllm_model:
|
||||
llm = vllm_model.llm
|
||||
|
||||
sampling_params = SamplingParams(temperature=0.0, max_tokens=256)
|
||||
outputs = llm.generate(input, sampling_params=sampling_params)
|
||||
|
||||
assert outputs[0].outputs[0].text == "The capital of France is Paris."
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_gritlm_api_server_generate():
|
||||
input = "<|user|>\nWhat is the capital of France?\n<|assistant|>\n"
|
||||
|
||||
args = ["--runner", "generate", "--max_model_len", str(MAX_MODEL_LEN)]
|
||||
|
||||
with RemoteOpenAIServer(MODEL_NAME, args) as server:
|
||||
client_generate = server.get_async_client()
|
||||
|
||||
outputs = await client_generate.completions.create(
|
||||
model=MODEL_NAME,
|
||||
prompt=input,
|
||||
max_tokens=256,
|
||||
temperature=0.0,
|
||||
)
|
||||
|
||||
assert outputs.choices[0].text == "The capital of France is Paris."
|
||||
@@ -30,7 +30,6 @@ from ....conftest import (
|
||||
VllmRunner,
|
||||
)
|
||||
from ....utils import create_new_process_for_each_test, large_gpu_mark, multi_gpu_marks
|
||||
from ...utils import check_outputs_equal
|
||||
from .vlm_utils import custom_inputs, model_utils, runners
|
||||
from .vlm_utils.case_filtering import get_parametrized_options
|
||||
from .vlm_utils.types import (
|
||||
@@ -321,20 +320,6 @@ VLM_TEST_SETTINGS = {
|
||||
# FIXME: https://github.com/huggingface/transformers/pull/38510
|
||||
marks=[pytest.mark.skip("Model is broken")],
|
||||
),
|
||||
"chameleon": VLMTestInfo(
|
||||
models=["facebook/chameleon-7b"],
|
||||
test_type=VLMTestType.IMAGE,
|
||||
prompt_formatter=lambda img_prompt: f"USER: {img_prompt}\nASSISTANT:",
|
||||
max_model_len=4096,
|
||||
max_num_seqs=2,
|
||||
auto_cls=AutoModelForImageTextToText,
|
||||
# For chameleon, we only compare the sequences
|
||||
vllm_output_post_proc=lambda vllm_output, model: vllm_output[:2],
|
||||
hf_output_post_proc=lambda hf_output, model: hf_output[:2],
|
||||
comparator=check_outputs_equal,
|
||||
max_tokens=8,
|
||||
dtype="bfloat16",
|
||||
),
|
||||
"cosmos3": VLMTestInfo(
|
||||
models=["nvidia/Cosmos3-Nano"],
|
||||
test_type=(
|
||||
@@ -935,17 +920,6 @@ VLM_TEST_SETTINGS = {
|
||||
num_logprobs=10,
|
||||
),
|
||||
### Tensor parallel / multi-gpu broadcast tests
|
||||
"chameleon-broadcast": VLMTestInfo(
|
||||
models=["facebook/chameleon-7b"],
|
||||
prompt_formatter=lambda img_prompt: f"USER: {img_prompt}\nASSISTANT:",
|
||||
max_model_len=4096,
|
||||
auto_cls=AutoModelForImageTextToText,
|
||||
vllm_output_post_proc=lambda vllm_output, model: vllm_output[:2],
|
||||
hf_output_post_proc=lambda hf_output, model: hf_output[:2],
|
||||
comparator=check_outputs_equal,
|
||||
marks=multi_gpu_marks(num_gpus=2),
|
||||
**COMMON_BROADCAST_SETTINGS, # type: ignore
|
||||
),
|
||||
"llava-broadcast": VLMTestInfo(
|
||||
models=["llava-hf/llava-1.5-7b-hf"],
|
||||
prompt_formatter=lambda img_prompt: f"USER: {img_prompt}\nASSISTANT:",
|
||||
|
||||
@@ -202,7 +202,6 @@ _TEXT_GENERATION_EXAMPLE_MODELS = {
|
||||
"AfmoeForCausalLM": _HfExamplesInfo("arcee-ai/Trinity-Nano-Preview"),
|
||||
"ApertusForCausalLM": _HfExamplesInfo("swiss-ai/Apertus-8B-Instruct-2509"),
|
||||
"ArceeForCausalLM": _HfExamplesInfo("arcee-ai/AFM-4.5B-Base"),
|
||||
"ArcticForCausalLM": _HfExamplesInfo("Snowflake/snowflake-arctic-instruct"),
|
||||
"AXK1ForCausalLM": _HfExamplesInfo("skt/A.X-K1", trust_remote_code=True),
|
||||
"BailingMoeForCausalLM": _HfExamplesInfo(
|
||||
"inclusionAI/Ling-lite-1.5", trust_remote_code=True
|
||||
@@ -273,7 +272,6 @@ _TEXT_GENERATION_EXAMPLE_MODELS = {
|
||||
"ExaoneMoeForCausalLM": _HfExamplesInfo(
|
||||
"LGAI-EXAONE/K-EXAONE-236B-A23B", min_transformers_version="5.1.0"
|
||||
),
|
||||
"Fairseq2LlamaForCausalLM": _HfExamplesInfo("mgleize/fairseq2-dummy-Llama-3.2-1B"),
|
||||
"FalconForCausalLM": _HfExamplesInfo("tiiuae/falcon-7b"),
|
||||
"FalconH1ForCausalLM": _HfExamplesInfo("tiiuae/Falcon-H1-0.5B-Base"),
|
||||
"FlexOlmoForCausalLM": _HfExamplesInfo("allenai/Flex-reddit-2x7B-1T"),
|
||||
@@ -448,9 +446,6 @@ _TEXT_GENERATION_EXAMPLE_MODELS = {
|
||||
"mistralai/Mixtral-8x7B-Instruct-v0.1",
|
||||
{"tiny": "TitanML/tiny-mixtral"},
|
||||
),
|
||||
"MptForCausalLM": _HfExamplesInfo("mpt", is_available_online=False),
|
||||
# FIXME: mosaicml/mpt-7b has been deleted
|
||||
"MPTForCausalLM": _HfExamplesInfo("mosaicml/mpt-7b", is_available_online=False),
|
||||
"NemotronForCausalLM": _HfExamplesInfo("nvidia/Minitron-8B-Base"),
|
||||
"NemotronHForCausalLM": _HfExamplesInfo(
|
||||
"nvidia/Nemotron-H-8B-Base-8K", trust_remote_code=True
|
||||
@@ -523,7 +518,6 @@ _TEXT_GENERATION_EXAMPLE_MODELS = {
|
||||
"EssentialAI/rnj-1-instruct",
|
||||
is_available_online=False,
|
||||
),
|
||||
"RWForCausalLM": _HfExamplesInfo("tiiuae/falcon-40b"),
|
||||
"SarvamMoEForCausalLM": _HfExamplesInfo(
|
||||
"sarvamai/sarvam-30b",
|
||||
trust_remote_code=True,
|
||||
@@ -548,7 +542,6 @@ _TEXT_GENERATION_EXAMPLE_MODELS = {
|
||||
trust_remote_code=True,
|
||||
),
|
||||
"SmolLM3ForCausalLM": _HfExamplesInfo("HuggingFaceTB/SmolLM3-3B"),
|
||||
"StableLMEpochForCausalLM": _HfExamplesInfo("stabilityai/stablelm-zephyr-3b"),
|
||||
"StableLmForCausalLM": _HfExamplesInfo("stabilityai/stablelm-3b-4e1t"),
|
||||
"Starcoder2ForCausalLM": _HfExamplesInfo("bigcode/starcoder2-3b"),
|
||||
"Step1ForCausalLM": _HfExamplesInfo(
|
||||
@@ -600,7 +593,6 @@ _EMBEDDING_EXAMPLE_MODELS = {
|
||||
),
|
||||
"Gemma2Model": _HfExamplesInfo("BAAI/bge-multilingual-gemma2"),
|
||||
"Gemma3TextModel": _HfExamplesInfo("google/embeddinggemma-300m"),
|
||||
"GritLM": _HfExamplesInfo("parasail-ai/GritLM-7B-vllm"),
|
||||
"GteModel": _HfExamplesInfo(
|
||||
"Snowflake/snowflake-arctic-embed-m-v2.0", trust_remote_code=True
|
||||
),
|
||||
@@ -642,14 +634,6 @@ _EMBEDDING_EXAMPLE_MODELS = {
|
||||
),
|
||||
"Qwen2VLForConditionalGeneration": _HfExamplesInfo("MrLight/dse-qwen2-2b-mrl-v1"),
|
||||
"SiglipModel": _HfExamplesInfo("google/siglip-base-patch16-224"),
|
||||
"PrithviGeoSpatialMAE": _HfExamplesInfo(
|
||||
"ibm-nasa-geospatial/Prithvi-EO-2.0-300M-TL-Sen1Floods11",
|
||||
dtype="float16",
|
||||
enforce_eager=True,
|
||||
require_embed_inputs=True,
|
||||
# This is to avoid the model going OOM in CI
|
||||
max_num_seqs=32,
|
||||
),
|
||||
"Terratorch": _HfExamplesInfo(
|
||||
"ibm-nasa-geospatial/Prithvi-EO-2.0-300M-TL-Sen1Floods11",
|
||||
dtype="float16",
|
||||
@@ -813,15 +797,6 @@ _MULTIMODAL_EXAMPLE_MODELS = {
|
||||
"Salesforce/blip2-opt-2.7b",
|
||||
extras={"6b": "Salesforce/blip2-opt-6.7b"},
|
||||
),
|
||||
"ChameleonForConditionalGeneration": _HfExamplesInfo("facebook/chameleon-7b"),
|
||||
"Cheers": _HfExamplesInfo(
|
||||
"ai9stars/Cheers",
|
||||
trust_remote_code=True,
|
||||
),
|
||||
"CheersForConditionalGeneration": _HfExamplesInfo(
|
||||
"ai9stars/Cheers",
|
||||
trust_remote_code=True,
|
||||
),
|
||||
"Cohere2VisionForConditionalGeneration": _HfExamplesInfo(
|
||||
"CohereLabs/command-a-vision-07-2025",
|
||||
extras={"command-a-plus": "CohereLabs/command-a-plus-05-2026-bf16"},
|
||||
@@ -890,15 +865,6 @@ _MULTIMODAL_EXAMPLE_MODELS = {
|
||||
"(dict object has no attribute '__name__').",
|
||||
},
|
||||
),
|
||||
"FireRedLIDForConditionalGeneration": _HfExamplesInfo(
|
||||
"PatchyTisa/FireRedLID-vllm",
|
||||
trust_remote_code=True,
|
||||
max_transformers_version="5.1",
|
||||
transformers_version_reason={
|
||||
"vllm": "Incompatible with transformers v5.2+ "
|
||||
"(dict object has no attribute '__name__').",
|
||||
},
|
||||
),
|
||||
"FunASRForConditionalGeneration": _HfExamplesInfo(
|
||||
"allendou/Fun-ASR-Nano-2512-vllm",
|
||||
trust_remote_code=True,
|
||||
@@ -966,10 +932,6 @@ _MULTIMODAL_EXAMPLE_MODELS = {
|
||||
max_transformers_version="4.48",
|
||||
transformers_version_reason={"hf": "HF model is not compatible."},
|
||||
),
|
||||
"HCXVisionForCausalLM": _HfExamplesInfo(
|
||||
"naver-hyperclovax/HyperCLOVAX-SEED-Vision-Instruct-3B",
|
||||
trust_remote_code=True,
|
||||
),
|
||||
"HCXVisionV2ForCausalLM": _HfExamplesInfo(
|
||||
"naver-hyperclovax/HyperCLOVAX-SEED-Think-32B",
|
||||
trust_remote_code=True,
|
||||
|
||||
@@ -39,7 +39,7 @@ MINIMAL_MODEL_ARCH_LIST = [
|
||||
"InternVLChatModel",
|
||||
"InternLM2ForRewardModel",
|
||||
"TransformersMultiModalForCausalLM",
|
||||
"PrithviGeoSpatialMAE",
|
||||
"Terratorch",
|
||||
"UltravoxModel",
|
||||
"DeepSeekMTPModel",
|
||||
"XLMRobertaModel",
|
||||
@@ -111,7 +111,7 @@ def can_initialize(
|
||||
"which is not configured in test environment"
|
||||
)
|
||||
|
||||
if model_arch in ("PrithviGeoSpatialMAE", "Terratorch"):
|
||||
if model_arch == "Terratorch":
|
||||
import importlib.util
|
||||
|
||||
if importlib.util.find_spec("terratorch") is None:
|
||||
|
||||
@@ -38,7 +38,7 @@ def test_registry_imports(model_arch):
|
||||
check_version_reason="vllm",
|
||||
)
|
||||
|
||||
if model_arch in ("PrithviGeoSpatialMAE", "Terratorch"):
|
||||
if model_arch == "Terratorch":
|
||||
import importlib.util
|
||||
|
||||
if importlib.util.find_spec("terratorch") is None:
|
||||
|
||||
@@ -424,7 +424,6 @@ def test_resolve_content_format_hf_defined(model, expected_format):
|
||||
("model", "expected_format"),
|
||||
[
|
||||
("Salesforce/blip2-opt-2.7b", "string"),
|
||||
("facebook/chameleon-7b", "string"),
|
||||
("deepseek-ai/deepseek-vl2-tiny", "string"),
|
||||
("google/paligemma-3b-mix-224", "string"),
|
||||
],
|
||||
|
||||
@@ -39,8 +39,6 @@ TOKENIZERS = [
|
||||
"EleutherAI/gpt-j-6b",
|
||||
"EleutherAI/pythia-70m",
|
||||
"bigscience/bloom-560m",
|
||||
# FIXME: mosaicml/mpt-7b has been deleted
|
||||
# "mosaicml/mpt-7b",
|
||||
"tiiuae/falcon-7b",
|
||||
"meta-llama/Llama-3.2-1B-Instruct",
|
||||
"codellama/CodeLlama-7b-hf",
|
||||
|
||||
@@ -1,13 +1,34 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
|
||||
|
||||
from typing import Annotated, Literal
|
||||
|
||||
import pytest
|
||||
import torch
|
||||
|
||||
from vllm.model_executor.models.glm4_1v import Glm4vImageEmbeddingInputs
|
||||
from vllm.model_executor.models.granite_speech import GraniteSpeechAudioInputs
|
||||
from vllm.model_executor.models.hyperclovax_vision import HCXVisionVideoPixelInputs
|
||||
from vllm.model_executor.models.phi3v import Phi3VImagePixelInputs
|
||||
from vllm.utils.tensor_schema import TensorSchema, TensorShape
|
||||
|
||||
|
||||
class _DoubleNestedVideoPixelInputs(TensorSchema):
|
||||
"""Minimal schema exercising doubly-nested list[list[torch.Tensor]] inputs.
|
||||
|
||||
Dimensions:
|
||||
- n: Number of videos
|
||||
- f: Number of frames
|
||||
- g: Number of grids
|
||||
- c: Number of channels (3)
|
||||
- h: Height
|
||||
- w: Width
|
||||
"""
|
||||
|
||||
type: Literal["pixel_values_videos"] = "pixel_values_videos"
|
||||
pixel_values_videos: Annotated[
|
||||
list[list[torch.Tensor]],
|
||||
TensorShape("n", "f", "g", 3, "h", "w", dynamic_dims={"f", "g"}),
|
||||
]
|
||||
|
||||
|
||||
def test_tensor_schema_valid_tensor():
|
||||
@@ -100,7 +121,7 @@ def test_tensor_schema_double_nested_tensors():
|
||||
x = torch.rand(4, 3, 32, 32)
|
||||
y = torch.rand(2, 3, 32, 32)
|
||||
|
||||
HCXVisionVideoPixelInputs(pixel_values_videos=([x, y, x], [y], [x, y]))
|
||||
_DoubleNestedVideoPixelInputs(pixel_values_videos=([x, y, x], [y], [x, y]))
|
||||
|
||||
|
||||
def test_tensor_schema_inconsistent_shapes_in_list():
|
||||
|
||||
@@ -16,5 +16,4 @@ compressed-tensors, neuralmagic/Phi-3-medium-128k-instruct-quantized.w4a16, main
|
||||
#compressed-tensors, mgoin/DeepSeek-Coder-V2-Lite-Instruct-FP8, main
|
||||
awq, casperhansen/mixtral-instruct-awq, main
|
||||
awq_marlin, casperhansen/mixtral-instruct-awq, main
|
||||
fp8, neuralmagic/Meta-Llama-3-8B-Instruct-FP8-KV, main
|
||||
None, mgleize/fairseq2-dummy-Llama-3.2-1B, main
|
||||
fp8, neuralmagic/Meta-Llama-3-8B-Instruct-FP8-KV, main
|
||||
@@ -255,8 +255,7 @@ class ApertusDecoderLayer(nn.Module):
|
||||
|
||||
# Apertus defaults to causal attention as it is a decoder-only model.
|
||||
# You can override the HF config with `is_causal=False` to enable
|
||||
# bidirectional attention, which is used in some embedding models
|
||||
# (e.g. parasail-ai/GritLM-7B-vllm)
|
||||
# bidirectional attention, which is used in some embedding models.
|
||||
if getattr(config, "is_causal", True):
|
||||
attn_type = AttentionType.DECODER
|
||||
else:
|
||||
|
||||
@@ -1,593 +0,0 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
|
||||
"""Inference-only Snowflake Arctic model."""
|
||||
|
||||
from collections.abc import Iterable
|
||||
from itertools import islice
|
||||
|
||||
import torch
|
||||
from torch import nn
|
||||
|
||||
from vllm.compilation.decorators import support_torch_compile
|
||||
from vllm.config import CacheConfig, VllmConfig
|
||||
from vllm.distributed import (
|
||||
get_pp_group,
|
||||
get_tensor_model_parallel_rank,
|
||||
get_tensor_model_parallel_world_size,
|
||||
tensor_model_parallel_all_reduce,
|
||||
)
|
||||
from vllm.model_executor.layers.activation import SiluAndMul
|
||||
from vllm.model_executor.layers.attention import Attention
|
||||
from vllm.model_executor.layers.fused_moe import (
|
||||
fused_experts,
|
||||
fused_topk,
|
||||
)
|
||||
from vllm.model_executor.layers.layernorm import RMSNorm
|
||||
from vllm.model_executor.layers.linear import (
|
||||
MergedColumnParallelLinear,
|
||||
QKVParallelLinear,
|
||||
ReplicatedLinear,
|
||||
RowParallelLinear,
|
||||
)
|
||||
from vllm.model_executor.layers.logits_processor import LogitsProcessor
|
||||
from vllm.model_executor.layers.quantization import QuantizationConfig
|
||||
from vllm.model_executor.layers.rotary_embedding import get_rope
|
||||
from vllm.model_executor.layers.vocab_parallel_embedding import (
|
||||
ParallelLMHead,
|
||||
VocabParallelEmbedding,
|
||||
)
|
||||
from vllm.model_executor.model_loader.weight_utils import default_weight_loader
|
||||
from vllm.model_executor.utils import set_weight_attrs
|
||||
from vllm.platforms import current_platform
|
||||
from vllm.sequence import IntermediateTensors
|
||||
from vllm.transformers_utils.configs.arctic import ArcticConfig
|
||||
|
||||
from .interfaces import SupportsPP, SupportsQuant
|
||||
from .utils import (
|
||||
AutoWeightsLoader,
|
||||
extract_layer_index,
|
||||
is_pp_missing_parameter,
|
||||
make_empty_intermediate_tensors_factory,
|
||||
make_layers,
|
||||
maybe_prefix,
|
||||
)
|
||||
|
||||
|
||||
class ArcticMLP(nn.Module):
|
||||
def __init__(
|
||||
self,
|
||||
config: ArcticConfig,
|
||||
expert_id: int = -1,
|
||||
is_residual_mlp: bool = False,
|
||||
quant_config: QuantizationConfig | None = None,
|
||||
reduce_results: bool = True,
|
||||
prefix: str = "",
|
||||
):
|
||||
super().__init__()
|
||||
self.hidden_size = config.hidden_size
|
||||
self.expert_id = expert_id
|
||||
|
||||
self.ffn_dim = (
|
||||
config.intermediate_size if not is_residual_mlp else self.hidden_size
|
||||
)
|
||||
|
||||
self.w13 = MergedColumnParallelLinear(
|
||||
self.hidden_size,
|
||||
[self.ffn_dim] * 2,
|
||||
bias=False,
|
||||
quant_config=quant_config,
|
||||
prefix=f"{prefix}.w13",
|
||||
)
|
||||
self.w2 = RowParallelLinear(
|
||||
self.ffn_dim,
|
||||
self.hidden_size,
|
||||
bias=False,
|
||||
reduce_results=reduce_results,
|
||||
quant_config=quant_config,
|
||||
prefix=f"{prefix}.w2",
|
||||
)
|
||||
if config.hidden_act != "silu":
|
||||
raise ValueError(
|
||||
f"Unsupported activation: {config.hidden_act}. "
|
||||
"Only silu is supported for now."
|
||||
)
|
||||
self.act_fn = SiluAndMul()
|
||||
|
||||
def forward(self, hidden_states):
|
||||
gate_up, _ = self.w13(hidden_states)
|
||||
hidden_states = self.act_fn(gate_up)
|
||||
hidden_states, _ = self.w2(hidden_states)
|
||||
return hidden_states
|
||||
|
||||
|
||||
class ArcticMoE(nn.Module):
|
||||
"""
|
||||
Model-parallel implementation of Arctic MoE Layer.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
config: ArcticConfig,
|
||||
tp_size: int | None = None,
|
||||
params_dtype: torch.dtype | None = None,
|
||||
quant_config: QuantizationConfig | None = None,
|
||||
reduce_results: bool = True,
|
||||
prefix: str = "",
|
||||
):
|
||||
super().__init__()
|
||||
|
||||
layer_id = extract_layer_index(prefix)
|
||||
self.tp_size = tp_size or get_tensor_model_parallel_world_size()
|
||||
self.hidden_size = config.hidden_size
|
||||
self.num_experts = config.num_local_experts
|
||||
self.layer_id = layer_id
|
||||
self.top_k = config.num_experts_per_tok
|
||||
self.intermediate_size = config.intermediate_size // self.tp_size
|
||||
|
||||
self.is_moe_layer = (layer_id + 1) % config.moe_layer_frequency == 0
|
||||
self.reduce_results = reduce_results
|
||||
# Some other parameters
|
||||
if params_dtype is None:
|
||||
params_dtype = torch.get_default_dtype()
|
||||
self.params_dtype = params_dtype
|
||||
|
||||
if not self.is_moe_layer:
|
||||
self.mlp = ArcticMLP(
|
||||
config,
|
||||
quant_config=quant_config,
|
||||
reduce_results=reduce_results,
|
||||
prefix=f"{prefix}.mlp",
|
||||
)
|
||||
else:
|
||||
self.gate = ReplicatedLinear(
|
||||
self.hidden_size,
|
||||
self.num_experts,
|
||||
bias=False,
|
||||
params_dtype=self.params_dtype,
|
||||
quant_config=quant_config,
|
||||
prefix=f"{prefix}.gate",
|
||||
)
|
||||
self.ws = nn.Parameter(
|
||||
torch.empty(
|
||||
self.num_experts,
|
||||
2 * self.intermediate_size,
|
||||
self.hidden_size,
|
||||
device=current_platform.device_type,
|
||||
dtype=self.params_dtype,
|
||||
)
|
||||
)
|
||||
self.w2s = nn.Parameter(
|
||||
torch.empty(
|
||||
self.num_experts,
|
||||
self.hidden_size,
|
||||
self.intermediate_size,
|
||||
device=current_platform.device_type,
|
||||
dtype=self.params_dtype,
|
||||
)
|
||||
)
|
||||
set_weight_attrs(
|
||||
self.ws,
|
||||
{
|
||||
"weight_loader": self.weight_loader,
|
||||
},
|
||||
)
|
||||
set_weight_attrs(
|
||||
self.w2s,
|
||||
{
|
||||
"weight_loader": self.weight_loader,
|
||||
},
|
||||
)
|
||||
|
||||
def weight_loader(
|
||||
self,
|
||||
param: nn.Parameter,
|
||||
loaded_weight: torch.Tensor,
|
||||
weight_name: str,
|
||||
expert_id: int,
|
||||
):
|
||||
tp_rank = get_tensor_model_parallel_rank()
|
||||
param_data = param.data
|
||||
shard_size = self.intermediate_size
|
||||
shard = slice(tp_rank * shard_size, (tp_rank + 1) * shard_size)
|
||||
if weight_name.endswith("w1.weight"):
|
||||
param_data[expert_id, 0:shard_size, :] = loaded_weight[shard, :]
|
||||
if weight_name.endswith("w3.weight"):
|
||||
param_data[expert_id, shard_size : 2 * shard_size, :] = loaded_weight[
|
||||
shard, :
|
||||
]
|
||||
if weight_name.endswith("w2.weight"):
|
||||
param_data[expert_id, :, :] = loaded_weight[:, shard]
|
||||
|
||||
def local_moe_fused(self, hidden_states: torch.Tensor) -> torch.Tensor:
|
||||
num_tokens, hidden_size = hidden_states.shape
|
||||
hidden_states = hidden_states.view(-1, self.hidden_size)
|
||||
# router_logits: (num_tokens, n_experts)
|
||||
router_logits, _ = self.gate(hidden_states)
|
||||
do_normalize = self.top_k > 1
|
||||
topk_weights, topk_ids, token_expert_indices = fused_topk(
|
||||
hidden_states, router_logits, self.top_k, renormalize=do_normalize
|
||||
)
|
||||
final_hidden_states = fused_experts(
|
||||
hidden_states,
|
||||
self.ws,
|
||||
self.w2s,
|
||||
topk_weights,
|
||||
topk_ids,
|
||||
)
|
||||
if self.reduce_results and self.tp_size > 1:
|
||||
final_hidden_states = tensor_model_parallel_all_reduce(final_hidden_states)
|
||||
return final_hidden_states.view(num_tokens, hidden_size)
|
||||
|
||||
def forward(self, hidden_states: torch.Tensor):
|
||||
if self.is_moe_layer:
|
||||
final_hidden_states = self.local_moe_fused(hidden_states)
|
||||
else:
|
||||
final_hidden_states = self.mlp(hidden_states)
|
||||
return final_hidden_states
|
||||
|
||||
|
||||
class ArcticAttention(nn.Module):
|
||||
def __init__(
|
||||
self,
|
||||
config: ArcticConfig,
|
||||
cache_config: CacheConfig | None = None,
|
||||
quant_config: QuantizationConfig | None = None,
|
||||
prefix: str = "",
|
||||
):
|
||||
super().__init__()
|
||||
self.config = config
|
||||
self.hidden_size = config.hidden_size
|
||||
|
||||
tp_size = get_tensor_model_parallel_world_size()
|
||||
self.total_num_heads = config.num_attention_heads
|
||||
assert self.total_num_heads % tp_size == 0
|
||||
self.num_heads = self.total_num_heads // tp_size
|
||||
self.total_num_kv_heads = config.num_key_value_heads
|
||||
if self.total_num_kv_heads >= tp_size:
|
||||
assert self.total_num_kv_heads % tp_size == 0
|
||||
else:
|
||||
assert tp_size % self.total_num_kv_heads == 0
|
||||
self.num_kv_heads = max(1, self.total_num_kv_heads // tp_size)
|
||||
self.head_dim = self.hidden_size // self.total_num_heads
|
||||
self.q_size = self.num_heads * self.head_dim
|
||||
self.kv_size = self.num_kv_heads * self.head_dim
|
||||
|
||||
self.max_position_embeddings = config.max_position_embeddings
|
||||
self.scaling = self.head_dim**-0.5
|
||||
|
||||
self.qkv_proj = QKVParallelLinear(
|
||||
self.hidden_size,
|
||||
self.head_dim,
|
||||
self.total_num_heads,
|
||||
self.total_num_kv_heads,
|
||||
bias=False,
|
||||
quant_config=quant_config,
|
||||
prefix=f"{prefix}.qkv_proj",
|
||||
)
|
||||
self.o_proj = RowParallelLinear(
|
||||
self.total_num_heads * self.head_dim,
|
||||
self.hidden_size,
|
||||
bias=False,
|
||||
reduce_results=True,
|
||||
quant_config=quant_config,
|
||||
prefix=f"{prefix}.o_proj",
|
||||
)
|
||||
|
||||
self.rotary_emb = get_rope(
|
||||
self.head_dim,
|
||||
max_position=self.max_position_embeddings,
|
||||
rope_parameters=config.rope_parameters,
|
||||
is_neox_style=True,
|
||||
)
|
||||
|
||||
self.attn = Attention(
|
||||
self.num_heads,
|
||||
self.head_dim,
|
||||
self.scaling,
|
||||
num_kv_heads=self.num_kv_heads,
|
||||
cache_config=cache_config,
|
||||
quant_config=quant_config,
|
||||
prefix=f"{prefix}.attn",
|
||||
)
|
||||
|
||||
def forward(
|
||||
self,
|
||||
positions: torch.Tensor,
|
||||
hidden_states: torch.Tensor,
|
||||
) -> torch.Tensor:
|
||||
qkv, _ = self.qkv_proj(hidden_states)
|
||||
q, k, v = qkv.split([self.q_size, self.kv_size, self.kv_size], dim=-1)
|
||||
q, k = self.rotary_emb(positions, q, k)
|
||||
attn_output = self.attn(q, k, v)
|
||||
output, _ = self.o_proj(attn_output)
|
||||
return output
|
||||
|
||||
|
||||
class ArcticDecoderLayer(nn.Module):
|
||||
def __init__(
|
||||
self,
|
||||
config: ArcticConfig,
|
||||
cache_config: CacheConfig | None = None,
|
||||
quant_config: QuantizationConfig | None = None,
|
||||
prefix: str = "",
|
||||
) -> None:
|
||||
super().__init__()
|
||||
self.hidden_size = config.hidden_size
|
||||
layer_idx = extract_layer_index(prefix)
|
||||
is_moe_layer = (layer_idx + 1) % config.moe_layer_frequency == 0
|
||||
self.use_residual = config.use_residual and is_moe_layer
|
||||
self.self_attn = ArcticAttention(
|
||||
config,
|
||||
cache_config,
|
||||
quant_config=quant_config,
|
||||
prefix=f"{prefix}.self_attn",
|
||||
)
|
||||
self.block_sparse_moe = ArcticMoE(
|
||||
config,
|
||||
quant_config=quant_config,
|
||||
reduce_results=(not self.use_residual),
|
||||
prefix=f"{prefix}.block_sparse_moe",
|
||||
)
|
||||
|
||||
self.input_layernorm = RMSNorm(config.hidden_size, eps=config.rms_norm_eps)
|
||||
self.post_attention_layernorm = RMSNorm(
|
||||
config.hidden_size, eps=config.rms_norm_eps
|
||||
)
|
||||
|
||||
if self.use_residual:
|
||||
self.residual_layernorm = RMSNorm(
|
||||
config.hidden_size, eps=config.rms_norm_eps
|
||||
)
|
||||
self.residual_mlp = ArcticMLP(
|
||||
config,
|
||||
is_residual_mlp=True,
|
||||
reduce_results=False,
|
||||
prefix=f"{prefix}.residual_mlp",
|
||||
)
|
||||
|
||||
def forward(
|
||||
self,
|
||||
positions: torch.Tensor,
|
||||
hidden_states: torch.Tensor,
|
||||
) -> torch.Tensor:
|
||||
residual_input = hidden_states
|
||||
hidden_states = self.input_layernorm(hidden_states)
|
||||
hidden_states = self.self_attn(
|
||||
positions=positions,
|
||||
hidden_states=hidden_states,
|
||||
)
|
||||
hidden_states = residual_input + hidden_states
|
||||
|
||||
residual_attn = hidden_states
|
||||
if self.use_residual:
|
||||
hidden_states = self.residual_layernorm(hidden_states)
|
||||
hidden_states = self.residual_mlp(hidden_states)
|
||||
residual_mlp = hidden_states
|
||||
hidden_states = self.post_attention_layernorm(residual_input)
|
||||
hidden_states = self.block_sparse_moe(hidden_states)
|
||||
hidden_states = residual_mlp + hidden_states
|
||||
hidden_states = tensor_model_parallel_all_reduce(hidden_states)
|
||||
hidden_states = residual_attn + hidden_states
|
||||
else:
|
||||
hidden_states = self.post_attention_layernorm(hidden_states)
|
||||
hidden_states = self.block_sparse_moe(hidden_states)
|
||||
hidden_states = residual_attn + hidden_states
|
||||
return hidden_states
|
||||
|
||||
|
||||
@support_torch_compile
|
||||
class ArcticModel(nn.Module):
|
||||
def __init__(self, *, vllm_config: VllmConfig, prefix: str = ""):
|
||||
super().__init__()
|
||||
|
||||
config = vllm_config.model_config.hf_config
|
||||
cache_config = vllm_config.cache_config
|
||||
quant_config = vllm_config.quant_config
|
||||
|
||||
self.config = config
|
||||
self.vocab_size = config.vocab_size
|
||||
self.embed_tokens = VocabParallelEmbedding(
|
||||
self.vocab_size, config.hidden_size, org_num_embeddings=self.vocab_size
|
||||
)
|
||||
self.start_layer, self.end_layer, self.layers = make_layers(
|
||||
config.num_hidden_layers,
|
||||
lambda prefix: ArcticDecoderLayer(
|
||||
config, cache_config, quant_config, prefix=prefix
|
||||
),
|
||||
prefix=f"{prefix}.layers",
|
||||
)
|
||||
self._attn_implementation = config._attn_implementation
|
||||
self.norm = RMSNorm(config.hidden_size, eps=config.rms_norm_eps)
|
||||
self.make_empty_intermediate_tensors = make_empty_intermediate_tensors_factory(
|
||||
["hidden_states"], config.hidden_size
|
||||
)
|
||||
|
||||
def embed_input_ids(self, input_ids: torch.Tensor) -> torch.Tensor:
|
||||
return self.embed_tokens(input_ids)
|
||||
|
||||
def forward(
|
||||
self,
|
||||
input_ids: torch.Tensor | None,
|
||||
positions: torch.Tensor,
|
||||
intermediate_tensors: IntermediateTensors | None,
|
||||
inputs_embeds: torch.Tensor | None = None,
|
||||
) -> torch.Tensor | IntermediateTensors:
|
||||
if get_pp_group().is_first_rank:
|
||||
if inputs_embeds is not None:
|
||||
hidden_states = inputs_embeds
|
||||
else:
|
||||
hidden_states = self.embed_input_ids(input_ids)
|
||||
else:
|
||||
assert intermediate_tensors is not None
|
||||
hidden_states = intermediate_tensors["hidden_states"]
|
||||
for layer in islice(self.layers, self.start_layer, self.end_layer):
|
||||
hidden_states = layer(positions, hidden_states)
|
||||
if not get_pp_group().is_last_rank:
|
||||
return IntermediateTensors({"hidden_states": hidden_states})
|
||||
hidden_states = self.norm(hidden_states)
|
||||
return hidden_states
|
||||
|
||||
def load_weights(self, weights: Iterable[tuple[str, torch.Tensor]]) -> set[str]:
|
||||
stacked_params_mapping = [
|
||||
# (param_name, shard_name, shard_id)
|
||||
("qkv_proj", "q_proj", "q"),
|
||||
("qkv_proj", "k_proj", "k"),
|
||||
("qkv_proj", "v_proj", "v"),
|
||||
]
|
||||
|
||||
mlp_params_mapping: list[tuple[str, str, int]] = []
|
||||
expert_params_mapping: list[tuple[str, str, int]] = []
|
||||
|
||||
for layer in range(self.config.num_hidden_layers):
|
||||
is_moe_layer = (layer + 1) % self.config.moe_layer_frequency == 0
|
||||
if is_moe_layer and self.config.use_residual:
|
||||
mlp_params_mapping.append(
|
||||
(
|
||||
f"layers.{layer}.residual_mlp.w13.weight",
|
||||
f"layers.{layer}.residual_mlp.w1.weight",
|
||||
0,
|
||||
)
|
||||
)
|
||||
mlp_params_mapping.append(
|
||||
(
|
||||
f"layers.{layer}.residual_mlp.w13.weight",
|
||||
f"layers.{layer}.residual_mlp.w3.weight",
|
||||
1,
|
||||
)
|
||||
)
|
||||
|
||||
if is_moe_layer:
|
||||
for expert_id in range(self.config.num_local_experts):
|
||||
expert_params_mapping.append(
|
||||
("ws", f"experts.{expert_id}.w1.weight", expert_id)
|
||||
)
|
||||
expert_params_mapping.append(
|
||||
("w2s", f"experts.{expert_id}.w2.weight", expert_id)
|
||||
)
|
||||
expert_params_mapping.append(
|
||||
("ws", f"experts.{expert_id}.w3.weight", expert_id)
|
||||
)
|
||||
else:
|
||||
mlp_params_mapping.append(
|
||||
(
|
||||
f"layers.{layer}.block_sparse_moe.mlp.w13.weight",
|
||||
f"layers.{layer}.block_sparse_moe.mlp.w1.weight",
|
||||
0,
|
||||
)
|
||||
)
|
||||
mlp_params_mapping.append(
|
||||
(
|
||||
f"layers.{layer}.block_sparse_moe.mlp.w13.weight",
|
||||
f"layers.{layer}.block_sparse_moe.mlp.w3.weight",
|
||||
1,
|
||||
)
|
||||
)
|
||||
|
||||
params_dict = dict(self.named_parameters())
|
||||
loaded_params: set[str] = set()
|
||||
|
||||
for name, loaded_weight in weights:
|
||||
for param_name, weight_name, shard_id in stacked_params_mapping:
|
||||
if weight_name not in name:
|
||||
continue
|
||||
name = name.replace(weight_name, param_name)
|
||||
# Skip loading extra bias for GPTQ models.
|
||||
if name.endswith(".bias") and name not in params_dict:
|
||||
continue
|
||||
if is_pp_missing_parameter(name, self):
|
||||
continue
|
||||
param = params_dict[name]
|
||||
weight_loader = param.weight_loader
|
||||
weight_loader(param, loaded_weight, shard_id)
|
||||
break
|
||||
else:
|
||||
for param_name, weight_name, mlp_shard_id in mlp_params_mapping:
|
||||
if weight_name not in name:
|
||||
continue
|
||||
name = name.replace(weight_name, param_name)
|
||||
if is_pp_missing_parameter(name, self):
|
||||
continue
|
||||
param = params_dict[name]
|
||||
weight_loader = param.weight_loader
|
||||
weight_loader(param, loaded_weight, mlp_shard_id)
|
||||
break
|
||||
else:
|
||||
for param_name, weight_name, expert_id in expert_params_mapping:
|
||||
if weight_name not in name:
|
||||
continue
|
||||
name = name.replace(weight_name, param_name)
|
||||
if is_pp_missing_parameter(name, self):
|
||||
continue
|
||||
param = params_dict[name]
|
||||
weight_loader = param.weight_loader
|
||||
weight_loader(
|
||||
param, loaded_weight, weight_name, expert_id=expert_id
|
||||
)
|
||||
break
|
||||
else:
|
||||
if name.endswith(".bias") and name not in params_dict:
|
||||
continue
|
||||
if is_pp_missing_parameter(name, self):
|
||||
continue
|
||||
param = params_dict[name]
|
||||
weight_loader = getattr(
|
||||
param, "weight_loader", default_weight_loader
|
||||
)
|
||||
weight_loader(param, loaded_weight)
|
||||
loaded_params.add(name)
|
||||
return loaded_params
|
||||
|
||||
|
||||
class ArcticForCausalLM(nn.Module, SupportsPP, SupportsQuant):
|
||||
packed_modules_mapping = {"qkv_proj": ["q_proj", "k_proj", "v_proj"]}
|
||||
|
||||
def __init__(self, *, vllm_config: VllmConfig, prefix: str = ""):
|
||||
super().__init__()
|
||||
config = vllm_config.model_config.hf_config
|
||||
quant_config = vllm_config.quant_config
|
||||
self.config = config
|
||||
self.model = ArcticModel(
|
||||
vllm_config=vllm_config, prefix=maybe_prefix(prefix, "model")
|
||||
)
|
||||
self.vocab_size = config.vocab_size
|
||||
self.lm_head = ParallelLMHead(
|
||||
self.vocab_size,
|
||||
config.hidden_size,
|
||||
quant_config=quant_config,
|
||||
prefix=maybe_prefix(prefix, "lm_head"),
|
||||
)
|
||||
if self.config.tie_word_embeddings:
|
||||
self.lm_head = self.lm_head.tie_weights(self.model.embed_tokens)
|
||||
self.num_experts = config.num_local_experts
|
||||
self.num_experts_per_tok = config.num_experts_per_tok
|
||||
|
||||
self.logits_processor = LogitsProcessor(config.vocab_size)
|
||||
self.make_empty_intermediate_tensors = (
|
||||
self.model.make_empty_intermediate_tensors
|
||||
)
|
||||
|
||||
def embed_input_ids(self, input_ids: torch.Tensor) -> torch.Tensor:
|
||||
return self.model.embed_input_ids(input_ids)
|
||||
|
||||
def forward(
|
||||
self,
|
||||
input_ids: torch.Tensor | None,
|
||||
positions: torch.Tensor,
|
||||
intermediate_tensors: IntermediateTensors | None = None,
|
||||
inputs_embeds: torch.Tensor | None = None,
|
||||
) -> torch.Tensor | IntermediateTensors:
|
||||
hidden_states = self.model(
|
||||
input_ids, positions, intermediate_tensors, inputs_embeds
|
||||
)
|
||||
return hidden_states
|
||||
|
||||
def compute_logits(
|
||||
self,
|
||||
hidden_states: torch.Tensor,
|
||||
) -> torch.Tensor | None:
|
||||
logits = self.logits_processor(self.lm_head, hidden_states)
|
||||
return logits
|
||||
|
||||
def load_weights(self, weights: Iterable[tuple[str, torch.Tensor]]) -> set[str]:
|
||||
loader = AutoWeightsLoader(self)
|
||||
return loader.load_weights(weights)
|
||||
File diff suppressed because it is too large
Load Diff
@@ -1,734 +0,0 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
|
||||
"""Inference-only Cheers (UMM) model compatible with HuggingFace weights.
|
||||
|
||||
Cheers is a unified multimodal model for image understanding and generation.
|
||||
For vLLM, we focus on the image understanding (vision-to-text) capabilities.
|
||||
The image generation part (gen_projector, hi_gate, etc.) is not supported,
|
||||
but the VAE encoder + decoder projector are required for image understanding.
|
||||
"""
|
||||
|
||||
import math
|
||||
from collections.abc import Iterable, Mapping, Sequence
|
||||
from typing import Any, Literal, TypeAlias
|
||||
|
||||
import torch
|
||||
import torch.nn as nn
|
||||
import torch.nn.functional as F
|
||||
from einops import rearrange
|
||||
|
||||
from vllm.config import VllmConfig
|
||||
from vllm.config.multimodal import BaseDummyOptions
|
||||
from vllm.inputs import MultiModalDataDict
|
||||
from vllm.logger import init_logger
|
||||
from vllm.model_executor.layers.quantization import QuantizationConfig
|
||||
from vllm.multimodal import MULTIMODAL_REGISTRY
|
||||
from vllm.multimodal.inputs import (
|
||||
MultiModalFieldConfig,
|
||||
MultiModalKwargsItems,
|
||||
)
|
||||
from vllm.multimodal.parse import MultiModalDataItems
|
||||
from vllm.multimodal.processing import (
|
||||
BaseDummyInputsBuilder,
|
||||
BaseMultiModalProcessor,
|
||||
BaseProcessingInfo,
|
||||
PromptReplacement,
|
||||
)
|
||||
from vllm.sequence import IntermediateTensors
|
||||
from vllm.transformers_utils.processors.cheers import CheersProcessor
|
||||
from vllm.utils.tensor_schema import TensorSchema
|
||||
|
||||
from .interfaces import (
|
||||
MultiModalEmbeddings,
|
||||
SupportsLoRA,
|
||||
SupportsMultiModal,
|
||||
SupportsPP,
|
||||
)
|
||||
from .siglip import SiglipVisionModel
|
||||
from .utils import (
|
||||
AutoWeightsLoader,
|
||||
WeightsMapper,
|
||||
init_vllm_registered_model,
|
||||
maybe_prefix,
|
||||
)
|
||||
|
||||
logger = init_logger(__name__)
|
||||
|
||||
|
||||
# ── VAE components (needed for image understanding pipeline) ────────
|
||||
|
||||
|
||||
def _swish(x: torch.Tensor) -> torch.Tensor:
|
||||
return x * torch.sigmoid(x)
|
||||
|
||||
|
||||
class _AttnBlock(nn.Module):
|
||||
def __init__(self, in_channels: int):
|
||||
super().__init__()
|
||||
self.norm = nn.GroupNorm(32, in_channels, eps=1e-6, affine=True)
|
||||
self.q = nn.Conv2d(in_channels, in_channels, 1)
|
||||
self.k = nn.Conv2d(in_channels, in_channels, 1)
|
||||
self.v = nn.Conv2d(in_channels, in_channels, 1)
|
||||
self.proj_out = nn.Conv2d(in_channels, in_channels, 1)
|
||||
|
||||
def forward(self, x: torch.Tensor) -> torch.Tensor:
|
||||
h_ = self.norm(x)
|
||||
q = self.q(h_)
|
||||
k = self.k(h_)
|
||||
v = self.v(h_)
|
||||
b, c, h, w = q.shape
|
||||
q = rearrange(q, "b c h w -> b 1 (h w) c").contiguous()
|
||||
k = rearrange(k, "b c h w -> b 1 (h w) c").contiguous()
|
||||
v = rearrange(v, "b c h w -> b 1 (h w) c").contiguous()
|
||||
h_ = F.scaled_dot_product_attention(q, k, v)
|
||||
h_ = rearrange(h_, "b 1 (h w) c -> b c h w", h=h, w=w, c=c, b=b)
|
||||
return x + self.proj_out(h_)
|
||||
|
||||
|
||||
class _ResnetBlock(nn.Module):
|
||||
def __init__(self, in_channels: int, out_channels: int):
|
||||
super().__init__()
|
||||
self.in_channels = in_channels
|
||||
self.out_channels = out_channels
|
||||
self.norm1 = nn.GroupNorm(32, in_channels, eps=1e-6, affine=True)
|
||||
self.conv1 = nn.Conv2d(in_channels, out_channels, 3, 1, 1)
|
||||
self.norm2 = nn.GroupNorm(32, out_channels, eps=1e-6, affine=True)
|
||||
self.conv2 = nn.Conv2d(out_channels, out_channels, 3, 1, 1)
|
||||
if in_channels != out_channels:
|
||||
self.nin_shortcut = nn.Conv2d(in_channels, out_channels, 1)
|
||||
|
||||
def forward(self, x: torch.Tensor) -> torch.Tensor:
|
||||
h = _swish(self.norm1(x))
|
||||
h = self.conv1(h)
|
||||
h = _swish(self.norm2(h))
|
||||
h = self.conv2(h)
|
||||
if self.in_channels != self.out_channels:
|
||||
x = self.nin_shortcut(x)
|
||||
return x + h
|
||||
|
||||
|
||||
class _Downsample(nn.Module):
|
||||
def __init__(self, in_channels: int):
|
||||
super().__init__()
|
||||
self.conv = nn.Conv2d(in_channels, in_channels, 3, stride=2, padding=0)
|
||||
|
||||
def forward(self, x: torch.Tensor) -> torch.Tensor:
|
||||
x = F.pad(x, (0, 1, 0, 1), mode="constant", value=0)
|
||||
return self.conv(x)
|
||||
|
||||
|
||||
class _Upsample(nn.Module):
|
||||
def __init__(self, in_channels: int):
|
||||
super().__init__()
|
||||
self.conv = nn.Conv2d(in_channels, in_channels, 3, 1, 1)
|
||||
|
||||
def forward(self, x: torch.Tensor) -> torch.Tensor:
|
||||
x = F.interpolate(x, scale_factor=2.0, mode="nearest")
|
||||
return self.conv(x)
|
||||
|
||||
|
||||
_VAE_ENCODER_DEFAULTS = {
|
||||
"in_channels": 3,
|
||||
"ch": 128,
|
||||
"ch_mult": [1, 2, 4, 4],
|
||||
"num_res_blocks": 2,
|
||||
"z_channels": 32,
|
||||
}
|
||||
_VAE_DECODER_DEFAULTS = {
|
||||
"in_channels": 3,
|
||||
"out_ch": 3,
|
||||
"ch": 128,
|
||||
"ch_mult": [1, 2, 4, 4],
|
||||
"num_res_blocks": 2,
|
||||
"z_channels": 32,
|
||||
}
|
||||
|
||||
|
||||
def _cfg(config, key, defaults=None):
|
||||
"""Access config attribute whether it's a dict or namespace object."""
|
||||
if isinstance(config, dict):
|
||||
if key in config:
|
||||
return config[key]
|
||||
if defaults and key in defaults:
|
||||
return defaults[key]
|
||||
raise KeyError(f"Key '{key}' not found in config dict: {list(config.keys())}")
|
||||
return getattr(config, key)
|
||||
|
||||
|
||||
class CheersVAEEncoder(nn.Module):
|
||||
"""VAE encoder from the Cheers/UMM model."""
|
||||
|
||||
def __init__(self, config):
|
||||
super().__init__()
|
||||
d = _VAE_ENCODER_DEFAULTS
|
||||
ch = _cfg(config, "ch", d)
|
||||
ch_mult = _cfg(config, "ch_mult", d)
|
||||
num_res_blocks = _cfg(config, "num_res_blocks", d)
|
||||
z_channels = _cfg(config, "z_channels", d)
|
||||
in_channels = _cfg(config, "in_channels", d)
|
||||
num_resolutions = len(ch_mult)
|
||||
|
||||
self.quant_conv = nn.Conv2d(2 * z_channels, 2 * z_channels, 1)
|
||||
self.conv_in = nn.Conv2d(in_channels, ch, 3, 1, 1)
|
||||
|
||||
in_ch_mult = (1,) + tuple(ch_mult)
|
||||
self.down = nn.ModuleList()
|
||||
block_in = ch
|
||||
for i_level in range(num_resolutions):
|
||||
block = nn.ModuleList()
|
||||
attn = nn.ModuleList()
|
||||
block_in = ch * in_ch_mult[i_level]
|
||||
block_out = ch * ch_mult[i_level]
|
||||
for _ in range(num_res_blocks):
|
||||
block.append(_ResnetBlock(block_in, block_out))
|
||||
block_in = block_out
|
||||
down = nn.Module()
|
||||
down.block = block
|
||||
down.attn = attn
|
||||
if i_level != num_resolutions - 1:
|
||||
down.downsample = _Downsample(block_in)
|
||||
self.down.append(down)
|
||||
|
||||
self.mid = nn.Module()
|
||||
self.mid.block_1 = _ResnetBlock(block_in, block_in)
|
||||
self.mid.attn_1 = _AttnBlock(block_in)
|
||||
self.mid.block_2 = _ResnetBlock(block_in, block_in)
|
||||
|
||||
self.norm_out = nn.GroupNorm(32, block_in, eps=1e-6, affine=True)
|
||||
self.conv_out = nn.Conv2d(block_in, 2 * z_channels, 3, 1, 1)
|
||||
self._num_resolutions = num_resolutions
|
||||
self._num_res_blocks = num_res_blocks
|
||||
|
||||
def forward(self, x: torch.Tensor) -> torch.Tensor:
|
||||
hs = [self.conv_in(x)]
|
||||
for i_level in range(self._num_resolutions):
|
||||
for i_block in range(self._num_res_blocks):
|
||||
h = self.down[i_level].block[i_block](hs[-1])
|
||||
if len(self.down[i_level].attn) > 0:
|
||||
h = self.down[i_level].attn[i_block](h)
|
||||
hs.append(h)
|
||||
if hasattr(self.down[i_level], "downsample"):
|
||||
hs.append(self.down[i_level].downsample(hs[-1]))
|
||||
h = hs[-1]
|
||||
h = self.mid.block_1(h)
|
||||
h = self.mid.attn_1(h)
|
||||
h = self.mid.block_2(h)
|
||||
h = _swish(self.norm_out(h))
|
||||
h = self.conv_out(h)
|
||||
h = self.quant_conv(h)
|
||||
return h
|
||||
|
||||
|
||||
class CheersVAEDecoder(nn.Module):
|
||||
"""VAE decoder (used inside VAEDecoderProjector)."""
|
||||
|
||||
def __init__(self, config):
|
||||
super().__init__()
|
||||
d = _VAE_DECODER_DEFAULTS
|
||||
ch = _cfg(config, "ch", d)
|
||||
ch_mult = _cfg(config, "ch_mult", d)
|
||||
num_res_blocks = _cfg(config, "num_res_blocks", d)
|
||||
z_channels = _cfg(config, "z_channels", d)
|
||||
out_ch = _cfg(config, "out_ch", d)
|
||||
num_resolutions = len(ch_mult)
|
||||
|
||||
self.post_quant_conv = nn.Conv2d(z_channels, z_channels, 1)
|
||||
block_in = ch * ch_mult[num_resolutions - 1]
|
||||
self.conv_in = nn.Conv2d(z_channels, block_in, 3, 1, 1)
|
||||
|
||||
self.mid = nn.Module()
|
||||
self.mid.block_1 = _ResnetBlock(block_in, block_in)
|
||||
self.mid.attn_1 = _AttnBlock(block_in)
|
||||
self.mid.block_2 = _ResnetBlock(block_in, block_in)
|
||||
|
||||
self.up = nn.ModuleList()
|
||||
for i_level in reversed(range(num_resolutions)):
|
||||
block = nn.ModuleList()
|
||||
attn = nn.ModuleList()
|
||||
block_out = ch * ch_mult[i_level]
|
||||
for _ in range(num_res_blocks + 1):
|
||||
block.append(_ResnetBlock(block_in, block_out))
|
||||
block_in = block_out
|
||||
up = nn.Module()
|
||||
up.block = block
|
||||
up.attn = attn
|
||||
if i_level != 0:
|
||||
up.upsample = _Upsample(block_in)
|
||||
self.up.insert(0, up)
|
||||
|
||||
self.norm_out = nn.GroupNorm(32, block_in, eps=1e-6, affine=True)
|
||||
self.conv_out = nn.Conv2d(block_in, out_ch, 3, 1, 1)
|
||||
self._num_resolutions = num_resolutions
|
||||
self._num_res_blocks = num_res_blocks
|
||||
|
||||
def forward(self, z: torch.Tensor) -> torch.Tensor:
|
||||
z = self.post_quant_conv(z)
|
||||
upscale_dtype = next(self.up.parameters()).dtype
|
||||
h = self.conv_in(z)
|
||||
h = self.mid.block_1(h)
|
||||
h = self.mid.attn_1(h)
|
||||
h = self.mid.block_2(h)
|
||||
h = h.to(upscale_dtype)
|
||||
for i_level in reversed(range(self._num_resolutions)):
|
||||
for i_block in range(self._num_res_blocks + 1):
|
||||
h = self.up[i_level].block[i_block](h)
|
||||
if len(self.up[i_level].attn) > 0:
|
||||
h = self.up[i_level].attn[i_block](h)
|
||||
if i_level != 0:
|
||||
h = self.up[i_level].upsample(h)
|
||||
h = _swish(self.norm_out(h))
|
||||
return self.conv_out(h)
|
||||
|
||||
|
||||
class CheersVAEModel(nn.Module):
|
||||
"""VAE model with encoder only (for image understanding)."""
|
||||
|
||||
def __init__(self, config):
|
||||
super().__init__()
|
||||
enc_cfg = _cfg(config, "vae_encoder_config")
|
||||
self.encoder = CheersVAEEncoder(enc_cfg)
|
||||
self.ps = [2, 2]
|
||||
z_ch = _cfg(enc_cfg, "z_channels", _VAE_ENCODER_DEFAULTS)
|
||||
self.bn = nn.BatchNorm2d(
|
||||
math.prod(self.ps) * z_ch,
|
||||
eps=1e-4,
|
||||
momentum=0.1,
|
||||
affine=False,
|
||||
track_running_stats=True,
|
||||
)
|
||||
|
||||
def encode(self, x: torch.Tensor) -> torch.Tensor:
|
||||
self.bn.eval()
|
||||
moments = self.encoder(x)
|
||||
mean = torch.chunk(moments, 2, dim=1)[0]
|
||||
z = rearrange(
|
||||
mean,
|
||||
"... c (i pi) (j pj) -> ... (c pi pj) i j",
|
||||
pi=self.ps[0],
|
||||
pj=self.ps[1],
|
||||
)
|
||||
return self.bn(z)
|
||||
|
||||
|
||||
class CheersVAEDecoderProjector(nn.Module):
|
||||
"""VAE decoder projector that converts latent back to pixel-like space."""
|
||||
|
||||
def __init__(self, config):
|
||||
super().__init__()
|
||||
dec_cfg = _cfg(config, "vae_decoder_config")
|
||||
enc_cfg = _cfg(config, "vae_encoder_config")
|
||||
self.decoder = CheersVAEDecoder(dec_cfg)
|
||||
self.ps = [2, 2]
|
||||
z_ch = _cfg(enc_cfg, "z_channels", _VAE_ENCODER_DEFAULTS)
|
||||
self.bn = nn.BatchNorm2d(
|
||||
math.prod(self.ps) * z_ch,
|
||||
eps=1e-4,
|
||||
momentum=0.1,
|
||||
affine=False,
|
||||
track_running_stats=True,
|
||||
)
|
||||
|
||||
def forward(self, z: torch.Tensor) -> torch.Tensor:
|
||||
self.bn.eval()
|
||||
s = torch.sqrt(self.bn.running_var.view(1, -1, 1, 1) + 1e-4)
|
||||
m = self.bn.running_mean.view(1, -1, 1, 1)
|
||||
z = z * s + m
|
||||
z = rearrange(
|
||||
z,
|
||||
"... (c pi pj) i j -> ... c (i pi) (j pj)",
|
||||
pi=self.ps[0],
|
||||
pj=self.ps[1],
|
||||
)
|
||||
return self.decoder(z)
|
||||
|
||||
|
||||
class CheersImagePixelInputs(TensorSchema):
|
||||
"""
|
||||
Dimensions:
|
||||
- bn: Batch size * number of images
|
||||
- c: Number of channels (3)
|
||||
- h: Height of each image
|
||||
- w: Width of each image
|
||||
"""
|
||||
|
||||
type: Literal["pixel_values"]
|
||||
pixel_values: torch.Tensor # Shape: (bn, 3, h, w)
|
||||
|
||||
|
||||
CheersImageInputs: TypeAlias = CheersImagePixelInputs
|
||||
|
||||
|
||||
class CheersUndProjector(nn.Module):
|
||||
"""Understanding projector that maps vision features to LLM dimension
|
||||
with 2x2 spatial compression (4x token reduction)."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
image_embed_dim: int,
|
||||
text_embed_dim: int,
|
||||
compression_factor: tuple[int, int] = (2, 2),
|
||||
quant_config: QuantizationConfig | None = None,
|
||||
prefix: str = "",
|
||||
):
|
||||
super().__init__()
|
||||
self.image_embed_dim = image_embed_dim
|
||||
self.text_embed_dim = text_embed_dim
|
||||
self.compression_factor = compression_factor
|
||||
self.layernorm = nn.LayerNorm(image_embed_dim)
|
||||
hidden_size = image_embed_dim * (compression_factor[0] * compression_factor[1])
|
||||
self.mlp = nn.Sequential(
|
||||
nn.Linear(hidden_size, hidden_size),
|
||||
nn.GELU(),
|
||||
nn.Linear(hidden_size, text_embed_dim),
|
||||
)
|
||||
|
||||
def forward(self, x: torch.Tensor) -> torch.Tensor:
|
||||
x = self.layernorm(x)
|
||||
height = width = int(x.size(1) ** 0.5)
|
||||
x = x.permute(0, 2, 1).unflatten(-1, (height, width))
|
||||
batch_size, dim, height, width = x.shape
|
||||
unfolded = x.unfold(
|
||||
2, self.compression_factor[0], self.compression_factor[0]
|
||||
).unfold(3, self.compression_factor[1], self.compression_factor[1])
|
||||
unfolded = unfolded.contiguous().view(
|
||||
batch_size,
|
||||
dim,
|
||||
-1,
|
||||
self.compression_factor[0] * self.compression_factor[1],
|
||||
)
|
||||
unfolded = (
|
||||
unfolded.permute(0, 2, 3, 1)
|
||||
.contiguous()
|
||||
.view(
|
||||
batch_size,
|
||||
-1,
|
||||
dim * self.compression_factor[0] * self.compression_factor[1],
|
||||
)
|
||||
)
|
||||
return self.mlp(unfolded)
|
||||
|
||||
|
||||
class CheersProcessingInfo(BaseProcessingInfo):
|
||||
"""Processing information for Cheers model."""
|
||||
|
||||
def get_hf_processor(self, **kwargs: object) -> CheersProcessor:
|
||||
from vllm.transformers_utils.processor import cached_get_image_processor
|
||||
|
||||
image_processor = cached_get_image_processor(
|
||||
self.ctx.model_config.model,
|
||||
revision=self.ctx.model_config.revision,
|
||||
trust_remote_code=self.ctx.model_config.trust_remote_code,
|
||||
)
|
||||
|
||||
tokenizer = self.get_tokenizer()
|
||||
|
||||
return CheersProcessor(
|
||||
image_processor=image_processor,
|
||||
tokenizer=tokenizer,
|
||||
**kwargs,
|
||||
)
|
||||
|
||||
def get_supported_mm_limits(self) -> Mapping[str, int | None]:
|
||||
return {"image": None}
|
||||
|
||||
def get_mm_max_tokens_per_item(
|
||||
self,
|
||||
seq_len: int,
|
||||
mm_counts: Mapping[str, int],
|
||||
) -> Mapping[str, int]:
|
||||
hf_config = self.get_hf_config()
|
||||
vit_config = hf_config.vision_representation_config
|
||||
patch_size = vit_config.patch_size
|
||||
image_size = vit_config.image_size
|
||||
num_patches = (image_size // patch_size) ** 2
|
||||
# After 2x2 compression, tokens reduce by 4x
|
||||
num_tokens = num_patches // 4
|
||||
return {"image": num_tokens}
|
||||
|
||||
def get_num_image_tokens(
|
||||
self,
|
||||
*,
|
||||
image_width: int,
|
||||
image_height: int,
|
||||
) -> int:
|
||||
hf_config = self.get_hf_config()
|
||||
vit_config = hf_config.vision_representation_config
|
||||
patch_size = vit_config.patch_size
|
||||
image_size = vit_config.image_size
|
||||
num_patches = (image_size // patch_size) ** 2
|
||||
return num_patches // 4
|
||||
|
||||
|
||||
class CheersDummyInputsBuilder(BaseDummyInputsBuilder[CheersProcessingInfo]):
|
||||
"""Build dummy inputs for Cheers model profiling."""
|
||||
|
||||
def get_dummy_text(self, mm_counts: Mapping[str, int]) -> str:
|
||||
num_images = mm_counts.get("image", 0)
|
||||
return "<|image_pad|>" * num_images
|
||||
|
||||
def get_dummy_mm_data(
|
||||
self,
|
||||
seq_len: int,
|
||||
mm_counts: Mapping[str, int],
|
||||
mm_options: Mapping[str, BaseDummyOptions] | None = None,
|
||||
) -> MultiModalDataDict:
|
||||
num_images = mm_counts.get("image", 0)
|
||||
hf_config = self.info.get_hf_config()
|
||||
vit_config = hf_config.vision_representation_config
|
||||
image_size = vit_config.image_size
|
||||
image_overrides = mm_options.get("image") if mm_options else None
|
||||
|
||||
return {
|
||||
"image": self._get_dummy_images(
|
||||
width=image_size,
|
||||
height=image_size,
|
||||
num_images=num_images,
|
||||
overrides=image_overrides,
|
||||
),
|
||||
}
|
||||
|
||||
|
||||
class CheersMultiModalProcessor(BaseMultiModalProcessor[CheersProcessingInfo]):
|
||||
"""Multimodal processor for Cheers model."""
|
||||
|
||||
def _get_prompt_updates(
|
||||
self,
|
||||
mm_items: MultiModalDataItems,
|
||||
hf_processor_mm_kwargs: Mapping[str, Any],
|
||||
out_mm_kwargs: MultiModalKwargsItems,
|
||||
) -> Sequence[PromptReplacement]:
|
||||
hf_config = self.info.get_hf_config()
|
||||
vit_config = hf_config.vision_representation_config
|
||||
patch_size = vit_config.patch_size
|
||||
image_size = vit_config.image_size
|
||||
|
||||
tokenizer = self.info.get_tokenizer()
|
||||
image_token_id = tokenizer.get_vocab().get("<|image_pad|>")
|
||||
if image_token_id is None:
|
||||
raise ValueError(
|
||||
"Image token '<|image_pad|>' not found in tokenizer vocabulary"
|
||||
)
|
||||
|
||||
def get_replacement_cheers(item_idx: int):
|
||||
num_patches = (image_size // patch_size) ** 2
|
||||
num_tokens = num_patches // 4
|
||||
return [image_token_id] * num_tokens
|
||||
|
||||
return [
|
||||
PromptReplacement(
|
||||
modality="image",
|
||||
target=[image_token_id],
|
||||
replacement=get_replacement_cheers,
|
||||
)
|
||||
]
|
||||
|
||||
def _get_mm_fields_config(
|
||||
self,
|
||||
hf_inputs: Any,
|
||||
hf_processor_mm_kwargs: Mapping[str, object],
|
||||
) -> Mapping[str, MultiModalFieldConfig]:
|
||||
return {
|
||||
"pixel_values": MultiModalFieldConfig.batched("image"),
|
||||
}
|
||||
|
||||
|
||||
@MULTIMODAL_REGISTRY.register_processor(
|
||||
CheersMultiModalProcessor,
|
||||
info=CheersProcessingInfo,
|
||||
dummy_inputs=CheersDummyInputsBuilder,
|
||||
)
|
||||
class CheersForConditionalGeneration(
|
||||
nn.Module, SupportsMultiModal, SupportsLoRA, SupportsPP
|
||||
):
|
||||
"""
|
||||
Cheers: A unified multimodal model for image understanding and generation.
|
||||
|
||||
For vLLM, we focus on the image understanding (vision-to-text) capabilities.
|
||||
The image generation part is not supported in vLLM.
|
||||
"""
|
||||
|
||||
requires_raw_input_tokens = True
|
||||
|
||||
hf_to_vllm_mapper = WeightsMapper(
|
||||
orig_to_new_prefix={
|
||||
"model.language_model.": "language_model.model.",
|
||||
"model.vision_representation.": "vision_representation.vision_model.",
|
||||
"model.und_projector.": "und_projector.",
|
||||
"model.vae_model.": "vae_model.",
|
||||
"model.vae_decoder_projector.": "vae_decoder_projector.",
|
||||
"lm_head.": "language_model.lm_head.",
|
||||
}
|
||||
)
|
||||
|
||||
@classmethod
|
||||
def get_placeholder_str(cls, modality: str, i: int) -> str | None:
|
||||
if modality.startswith("image"):
|
||||
return "<|image_pad|>"
|
||||
raise ValueError("Only image modality is supported")
|
||||
|
||||
def __init__(self, *, vllm_config: VllmConfig, prefix: str = ""):
|
||||
super().__init__()
|
||||
|
||||
config = vllm_config.model_config.hf_config
|
||||
quant_config = vllm_config.quant_config
|
||||
multimodal_config = vllm_config.model_config.multimodal_config
|
||||
|
||||
if type(config).__name__ not in ("CheersConfig", "UMMConfig"):
|
||||
raise ValueError(
|
||||
f"Expected CheersConfig or UMMConfig, got {type(config).__name__}."
|
||||
)
|
||||
|
||||
self.config = config
|
||||
self.multimodal_config = multimodal_config
|
||||
|
||||
# The Cheers model's custom Qwen2Config defaults rope_theta to
|
||||
# 1_000_000, but this isn't stored in the JSON. vLLM's standard
|
||||
# Qwen2Config defaults to 10_000, causing a 100× mismatch.
|
||||
# We must patch BOTH the attribute AND rope_parameters (which
|
||||
# patch_rope_parameters may have already populated from the wrong
|
||||
# default before __init__ runs).
|
||||
_CHEERS_ROPE_THETA = 1_000_000.0
|
||||
tc = config.text_config
|
||||
old_theta = getattr(tc, "rope_theta", None)
|
||||
if old_theta != _CHEERS_ROPE_THETA:
|
||||
logger.info(
|
||||
"Overriding text_config.rope_theta from %s to %s",
|
||||
old_theta,
|
||||
_CHEERS_ROPE_THETA,
|
||||
)
|
||||
tc.rope_theta = _CHEERS_ROPE_THETA
|
||||
rp = getattr(tc, "rope_parameters", None)
|
||||
if rp is not None and rp.get("rope_theta") != _CHEERS_ROPE_THETA:
|
||||
logger.info(
|
||||
"Overriding rope_parameters.rope_theta from %s to %s",
|
||||
rp.get("rope_theta"),
|
||||
_CHEERS_ROPE_THETA,
|
||||
)
|
||||
rp["rope_theta"] = _CHEERS_ROPE_THETA
|
||||
|
||||
with self._mark_language_model(vllm_config):
|
||||
self.language_model = init_vllm_registered_model(
|
||||
vllm_config=vllm_config,
|
||||
hf_config=config.text_config,
|
||||
prefix=maybe_prefix(prefix, "language_model"),
|
||||
architectures=["Qwen2ForCausalLM"],
|
||||
)
|
||||
|
||||
vit_config = config.vision_representation_config
|
||||
|
||||
with self._mark_tower_model(vllm_config, "image"):
|
||||
self.vae_model = CheersVAEModel(config)
|
||||
self.vae_decoder_projector = CheersVAEDecoderProjector(config)
|
||||
|
||||
self.vision_representation = SiglipVisionModel(
|
||||
config=vit_config,
|
||||
quant_config=quant_config,
|
||||
prefix=maybe_prefix(prefix, "vision_representation"),
|
||||
)
|
||||
|
||||
vit_hidden_size = vit_config.hidden_size
|
||||
llm_hidden_size = config.text_config.hidden_size
|
||||
|
||||
self.und_projector = CheersUndProjector(
|
||||
image_embed_dim=vit_hidden_size,
|
||||
text_embed_dim=llm_hidden_size,
|
||||
compression_factor=(2, 2),
|
||||
quant_config=quant_config,
|
||||
prefix=maybe_prefix(prefix, "und_projector"),
|
||||
)
|
||||
|
||||
self.make_empty_intermediate_tensors = (
|
||||
self.language_model.make_empty_intermediate_tensors
|
||||
)
|
||||
|
||||
def _parse_and_validate_image_input(
|
||||
self, **kwargs: object
|
||||
) -> CheersImageInputs | None:
|
||||
pixel_values = kwargs.pop("pixel_values", None)
|
||||
if pixel_values is None:
|
||||
return None
|
||||
return CheersImagePixelInputs(
|
||||
type="pixel_values",
|
||||
pixel_values=pixel_values,
|
||||
)
|
||||
|
||||
def _process_image_input(
|
||||
self, image_input: CheersImageInputs
|
||||
) -> tuple[torch.Tensor, ...]:
|
||||
"""Process image inputs through VAE → SigLIP → projector pipeline.
|
||||
|
||||
HF native path: pixel_values → VAE.encode(t=1.0) → vae_decoder_projector
|
||||
→ SigLIP → und_projector → text-space embeddings
|
||||
"""
|
||||
pixel_values = image_input["pixel_values"]
|
||||
|
||||
if pixel_values.ndim == 5:
|
||||
batch_size, num_images, channels, height, width = pixel_values.shape
|
||||
pixel_values = pixel_values.reshape(
|
||||
batch_size * num_images, channels, height, width
|
||||
)
|
||||
|
||||
with torch.no_grad():
|
||||
vae_dtype = next(self.vae_model.parameters()).dtype
|
||||
image_latent = self.vae_model.encode(pixel_values.to(dtype=vae_dtype))
|
||||
image_pixel_hat = self.vae_decoder_projector(image_latent)
|
||||
|
||||
vision_features = self.vision_representation(image_pixel_hat)
|
||||
vision_embeds = self.und_projector(vision_features)
|
||||
|
||||
return tuple(vision_embeds)
|
||||
|
||||
def embed_multimodal(self, **kwargs: object) -> MultiModalEmbeddings:
|
||||
image_input = self._parse_and_validate_image_input(**kwargs)
|
||||
if image_input is None:
|
||||
return []
|
||||
return self._process_image_input(image_input)
|
||||
|
||||
def forward(
|
||||
self,
|
||||
input_ids: torch.Tensor | None,
|
||||
positions: torch.Tensor,
|
||||
intermediate_tensors: IntermediateTensors | None = None,
|
||||
inputs_embeds: torch.Tensor | None = None,
|
||||
**kwargs: object,
|
||||
) -> torch.Tensor | IntermediateTensors:
|
||||
if intermediate_tensors is not None:
|
||||
inputs_embeds = None
|
||||
|
||||
hidden_states = self.language_model.model(
|
||||
input_ids=input_ids,
|
||||
positions=positions,
|
||||
intermediate_tensors=intermediate_tensors,
|
||||
inputs_embeds=inputs_embeds,
|
||||
)
|
||||
return hidden_states
|
||||
|
||||
def compute_logits(
|
||||
self,
|
||||
hidden_states: torch.Tensor,
|
||||
) -> torch.Tensor | None:
|
||||
return self.language_model.compute_logits(hidden_states)
|
||||
|
||||
def load_weights(self, weights: Iterable[tuple[str, torch.Tensor]]) -> set[str]:
|
||||
"""Load weights, keeping VAE encoder/decoder projector for understanding."""
|
||||
skip_prefixes = [
|
||||
"model.time_embed.",
|
||||
"model.gen_projector.",
|
||||
"model.hi_gate.",
|
||||
"model.hi_projector.",
|
||||
"model.vae_model.decoder.",
|
||||
]
|
||||
skip_keywords = [
|
||||
"text_loss_fc",
|
||||
]
|
||||
|
||||
filtered_weights = []
|
||||
for name, tensor in weights:
|
||||
if any(name.startswith(p) for p in skip_prefixes):
|
||||
continue
|
||||
if any(kw in name for kw in skip_keywords):
|
||||
continue
|
||||
filtered_weights.append((name, tensor))
|
||||
|
||||
loader = AutoWeightsLoader(self)
|
||||
return loader.load_weights(filtered_weights, mapper=self.hf_to_vllm_mapper)
|
||||
@@ -1,11 +1,10 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
|
||||
"""
|
||||
Shared Conformer encoder components for FireRedASR2 and FireRedLID.
|
||||
Conformer encoder components for FireRedASR2.
|
||||
|
||||
Both models use the same Conformer-based audio encoder architecture
|
||||
The audio encoder architecture is
|
||||
(Conv2dSubsampling → RelPositionalEncoding → N × RelPosEmbConformerBlock).
|
||||
This module factors out the common building blocks to avoid duplication.
|
||||
"""
|
||||
|
||||
import torch
|
||||
@@ -288,7 +287,7 @@ class RelPosEmbConformerBlock(nn.Module):
|
||||
|
||||
class ConformerEncoder(nn.Module):
|
||||
"""
|
||||
Conformer encoder shared by FireRedASR2 and FireRedLID.
|
||||
Conformer encoder used by FireRedASR2.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
|
||||
@@ -1,149 +0,0 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
|
||||
|
||||
# Copyright 2024 The vLLM team.
|
||||
# Copyright 2024 Meta Platforms, Inc. and affiliates. All rights reserved.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
"""Llama model for fairseq2 weights."""
|
||||
|
||||
from collections.abc import Iterable
|
||||
|
||||
import torch
|
||||
from torch.nn import Parameter
|
||||
|
||||
from vllm.config import VllmConfig
|
||||
from vllm.distributed import (
|
||||
get_tensor_model_parallel_rank,
|
||||
get_tensor_model_parallel_world_size,
|
||||
)
|
||||
from vllm.model_executor.layers.linear import set_weight_attrs
|
||||
from vllm.model_executor.models.llama import LlamaForCausalLM
|
||||
|
||||
from .utils import AutoWeightsLoader, WeightsMapper
|
||||
|
||||
|
||||
class Fairseq2LlamaForCausalLM(LlamaForCausalLM):
|
||||
def __init__(self, *, vllm_config: VllmConfig, prefix: str = ""):
|
||||
super().__init__(vllm_config=vllm_config, prefix=prefix)
|
||||
self.tp_rank = get_tensor_model_parallel_rank()
|
||||
self.tp_size = get_tensor_model_parallel_world_size()
|
||||
# For the model loader to read only the relevant checkpoint files
|
||||
self.allow_patterns_overrides = [
|
||||
# either the full checkpoint
|
||||
"model.pt",
|
||||
# or the tp-sharded checkpoint of the current rank
|
||||
f"model.{self.tp_rank}.pt",
|
||||
]
|
||||
|
||||
def load_weights(self, weights: Iterable[tuple[str, torch.Tensor]]) -> set[str]:
|
||||
# fairseq2's serialization adds a wrapper to usual .pt state_dict's:
|
||||
# { "model_key": my_model_name, "my_model_name": state_dict }
|
||||
# which we first need to unpack
|
||||
weights_wrapped = dict(weights)
|
||||
weights = weights_wrapped[weights_wrapped["model_key"]].items() # type: ignore
|
||||
|
||||
# remap keys
|
||||
fs2_to_vllm_mapper = WeightsMapper(
|
||||
orig_to_new_prefix={
|
||||
"decoder_frontend.embed.": "model.embed_tokens.",
|
||||
"decoder.": "model.",
|
||||
"final_proj.": "lm_head.",
|
||||
},
|
||||
orig_to_new_substr={
|
||||
".self_attn_layer_norm.": ".input_layernorm.",
|
||||
".ffn_layer_norm.": ".post_attention_layernorm.",
|
||||
".self_attn.output_proj.": ".self_attn.o_proj.",
|
||||
".ffn.gate_proj.": ".mlp.gate_proj.",
|
||||
".ffn.inner_proj.": ".mlp.up_proj.",
|
||||
".ffn.output_proj.": ".mlp.down_proj.",
|
||||
".layer_norm.": ".norm.",
|
||||
},
|
||||
)
|
||||
weights = fs2_to_vllm_mapper.apply(weights)
|
||||
|
||||
params = dict(self.named_parameters())
|
||||
|
||||
loader = AutoWeightsLoader(self)
|
||||
return loader.load_weights(
|
||||
self.reshape_fairseq2_weights(name, loaded_weight, params)
|
||||
for name, loaded_weight in weights
|
||||
)
|
||||
|
||||
def flag_sharded_weights(self, params: dict[str, Parameter]):
|
||||
"""Sets the `is_sharded_weight` flag to True for all sharded weights"""
|
||||
for name, param in params.items():
|
||||
modules = name.split(".")
|
||||
if "norm" in name and len(param.size()) < 2:
|
||||
# layer norms are not sharded
|
||||
continue
|
||||
elif any(emb in modules for emb in ["embed_tokens", "lm_head"]):
|
||||
# for now we repeat embedding layers for compatibility
|
||||
continue
|
||||
else:
|
||||
# all other layers are sharded
|
||||
set_weight_attrs(param, {"is_sharded_weight": True})
|
||||
|
||||
def reshape_fairseq2_weights(
|
||||
self,
|
||||
name: str,
|
||||
loaded_weight: torch.Tensor,
|
||||
params: dict[str, Parameter],
|
||||
) -> tuple[str, torch.Tensor]:
|
||||
"""Reshape fairseq2's weights."""
|
||||
|
||||
def permute(w: torch.Tensor, n_heads: int) -> torch.Tensor:
|
||||
attn_in = self.config.head_dim * n_heads
|
||||
# check for a sharded weight on dim 0
|
||||
if attn_in // self.tp_size == w.size()[0]:
|
||||
attn_in //= self.tp_size
|
||||
n_heads //= self.tp_size
|
||||
attn_out = self.config.hidden_size
|
||||
return (
|
||||
w.view(n_heads, attn_in // n_heads // 2, 2, attn_out)
|
||||
.transpose(1, 2)
|
||||
.reshape(attn_in, attn_out)
|
||||
)
|
||||
|
||||
modules = name.split(".")
|
||||
|
||||
# rotary embeds should be sliced
|
||||
if "k_proj" in modules:
|
||||
loaded_weight = permute(loaded_weight, self.config.num_key_value_heads)
|
||||
|
||||
elif "q_proj" in modules:
|
||||
loaded_weight = permute(loaded_weight, self.config.num_attention_heads)
|
||||
|
||||
# We make the loaded weights compatible with both
|
||||
# full checkpoints and tp sharded checkpoints.
|
||||
# Embeddings are repeated to fit the vocab size.
|
||||
# Other weights are flagged for the weight_loader calls.
|
||||
if any(emb in modules for emb in ["embed_tokens", "lm_head"]):
|
||||
# Embeddings are sharded on dim 0
|
||||
dim = 0
|
||||
# In fairseq2, vocab size has to be divisible by tp_size
|
||||
# so we don't worry about padding
|
||||
if self.tp_size > 1 and loaded_weight.shape[dim] < self.config.vocab_size:
|
||||
assert (
|
||||
loaded_weight.shape[dim] * self.tp_size == self.config.vocab_size
|
||||
), "vocab_size should be divisible by tp_size."
|
||||
repeats = [1] * len(loaded_weight.size())
|
||||
repeats[dim] = self.tp_size
|
||||
# repeat to match vocab size and to be easily 'narrow'able
|
||||
loaded_weight = loaded_weight.repeat(repeats)
|
||||
set_weight_attrs(params[name], {"is_sharded_weight": False})
|
||||
# if embeddings are sharded, the rest is too
|
||||
if "embed_tokens" in modules:
|
||||
self.flag_sharded_weights(params)
|
||||
|
||||
return name, loaded_weight
|
||||
@@ -1,794 +0,0 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
|
||||
"""
|
||||
FireRedLID – Language Identification model adapted for vLLM.
|
||||
|
||||
Architecture: ConformerEncoder + TransformerDecoder (6-layer cross-attn)
|
||||
Vocabulary: 120 LID tokens (dict.txt)
|
||||
Output: Up to 2 tokens (e.g. "en", "zh mandarin")
|
||||
|
||||
This implementation follows the Whisper-style encoder-decoder pattern:
|
||||
• Encoder processes audio features (Fbank + CMVN via FeatureExtractor)
|
||||
• Decoder performs single-step autoregressive forward
|
||||
• vLLM's generation loop handles beam search / sampling
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Iterable, Mapping, Sequence
|
||||
from typing import Annotated, Literal
|
||||
|
||||
import torch
|
||||
from torch import nn
|
||||
from transformers import BatchFeature
|
||||
|
||||
from vllm.config import ModelConfig, VllmConfig
|
||||
from vllm.config.multimodal import AudioDummyOptions, BaseDummyOptions
|
||||
from vllm.config.speech_to_text import SpeechToTextConfig, SpeechToTextParams
|
||||
from vllm.distributed import get_tensor_model_parallel_world_size
|
||||
from vllm.inputs import MultiModalDataDict, PromptType
|
||||
from vllm.logger import init_logger
|
||||
from vllm.model_executor.layers.attention import Attention, CrossAttention
|
||||
from vllm.model_executor.layers.linear import (
|
||||
ColumnParallelLinear,
|
||||
ReplicatedLinear,
|
||||
RowParallelLinear,
|
||||
)
|
||||
from vllm.model_executor.layers.logits_processor import LogitsProcessor
|
||||
from vllm.model_executor.layers.vocab_parallel_embedding import ParallelLMHead
|
||||
from vllm.multimodal import MULTIMODAL_REGISTRY
|
||||
from vllm.multimodal.inputs import (
|
||||
MultiModalFieldConfig,
|
||||
MultiModalKwargsItems,
|
||||
)
|
||||
from vllm.multimodal.parse import MultiModalDataItems, MultiModalDataParser
|
||||
from vllm.multimodal.processing import (
|
||||
BaseDummyInputsBuilder,
|
||||
BaseProcessingInfo,
|
||||
EncDecMultiModalProcessor,
|
||||
PromptReplacement,
|
||||
PromptUpdate,
|
||||
)
|
||||
from vllm.transformers_utils.processor import cached_processor_from_config
|
||||
from vllm.utils.tensor_schema import TensorSchema, TensorShape
|
||||
|
||||
from .conformer_encoder import ConformerEncoder
|
||||
from .interfaces import (
|
||||
MultiModalEmbeddings,
|
||||
SupportsMultiModal,
|
||||
SupportsTranscription,
|
||||
)
|
||||
from .utils import (
|
||||
AutoWeightsLoader,
|
||||
WeightsMapper,
|
||||
maybe_prefix,
|
||||
)
|
||||
from .whisper_utils import ISO639_1_SUPPORTED_LANGS
|
||||
|
||||
logger = init_logger(__name__)
|
||||
|
||||
|
||||
class FireRedLIDAudioInputs(TensorSchema):
|
||||
"""
|
||||
Dimensions:
|
||||
- b: Batch size
|
||||
- t: Time frames (variable across utterances)
|
||||
- nmb: Number of mel bins (80)
|
||||
"""
|
||||
|
||||
input_features: Annotated[
|
||||
list[torch.Tensor] | None,
|
||||
TensorShape("b", "t", "nmb", dynamic_dims={"t"}),
|
||||
]
|
||||
speech_lengths: Annotated[
|
||||
list[torch.Tensor] | None,
|
||||
TensorShape("b"),
|
||||
]
|
||||
fake_token_lengths: Annotated[
|
||||
list[torch.Tensor] | None,
|
||||
TensorShape("b"),
|
||||
]
|
||||
|
||||
|
||||
FireRedLIDEncoder = ConformerEncoder
|
||||
|
||||
|
||||
class FireRedLIDPositionalEmbedding(nn.Module):
|
||||
"""Absolute sinusoidal positional embedding indexed by `positions`."""
|
||||
|
||||
def __init__(self, d_model: int, max_len: int = 5000):
|
||||
super().__init__()
|
||||
assert d_model % 2 == 0
|
||||
pe = torch.zeros(max_len, d_model, requires_grad=False)
|
||||
position = torch.arange(0, max_len).unsqueeze(1).float()
|
||||
div_term = torch.exp(
|
||||
torch.arange(0, d_model, 2).float()
|
||||
* -(torch.log(torch.tensor(10000.0)).item() / d_model)
|
||||
)
|
||||
pe[:, 0::2] = torch.sin(position * div_term)
|
||||
pe[:, 1::2] = torch.cos(position * div_term)
|
||||
self.register_buffer("pe", pe, persistent=False)
|
||||
|
||||
def forward(self, position_ids: torch.Tensor) -> torch.Tensor:
|
||||
return self.pe[position_ids]
|
||||
|
||||
|
||||
class FireRedLIDAttention(nn.Module):
|
||||
"""Base attention with shared QKV/FC projections for the LID decoder."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
d_model: int,
|
||||
n_head: int,
|
||||
*,
|
||||
vllm_config: VllmConfig,
|
||||
prefix: str = "",
|
||||
):
|
||||
super().__init__()
|
||||
tp_size = get_tensor_model_parallel_world_size()
|
||||
assert n_head % tp_size == 0
|
||||
self.total_num_heads = n_head
|
||||
self.num_heads = n_head // tp_size
|
||||
self.num_kv_heads = max(1, n_head // tp_size)
|
||||
self.head_dim = d_model // n_head
|
||||
self.scaling = self.head_dim**-0.5
|
||||
|
||||
cache_config = vllm_config.cache_config
|
||||
quant_config = vllm_config.quant_config
|
||||
|
||||
self.w_qs = ColumnParallelLinear(
|
||||
d_model,
|
||||
d_model,
|
||||
bias=True,
|
||||
quant_config=quant_config,
|
||||
prefix=f"{prefix}.w_qs",
|
||||
)
|
||||
self.w_ks = ColumnParallelLinear(
|
||||
d_model,
|
||||
d_model,
|
||||
bias=False,
|
||||
quant_config=quant_config,
|
||||
prefix=f"{prefix}.w_ks",
|
||||
)
|
||||
self.w_vs = ColumnParallelLinear(
|
||||
d_model,
|
||||
d_model,
|
||||
bias=True,
|
||||
quant_config=quant_config,
|
||||
prefix=f"{prefix}.w_vs",
|
||||
)
|
||||
self.fc = RowParallelLinear(
|
||||
d_model,
|
||||
d_model,
|
||||
bias=True,
|
||||
quant_config=quant_config,
|
||||
prefix=f"{prefix}.fc",
|
||||
)
|
||||
self._init_attn(cache_config, quant_config, prefix)
|
||||
|
||||
def _init_attn(self, cache_config, quant_config, prefix: str) -> None:
|
||||
raise NotImplementedError
|
||||
|
||||
|
||||
class FireRedLIDSelfAttention(FireRedLIDAttention):
|
||||
def _init_attn(self, cache_config, quant_config, prefix: str) -> None:
|
||||
self.attn = Attention(
|
||||
self.num_heads,
|
||||
self.head_dim,
|
||||
self.scaling,
|
||||
num_kv_heads=self.num_kv_heads,
|
||||
cache_config=cache_config,
|
||||
quant_config=quant_config,
|
||||
prefix=f"{prefix}.attn",
|
||||
)
|
||||
|
||||
def forward(self, hidden_states: torch.Tensor) -> torch.Tensor:
|
||||
q, _ = self.w_qs(hidden_states)
|
||||
k, _ = self.w_ks(hidden_states)
|
||||
v, _ = self.w_vs(hidden_states)
|
||||
attn_output = self.attn(q, k, v)
|
||||
output, _ = self.fc(attn_output)
|
||||
return output
|
||||
|
||||
|
||||
class FireRedLIDCrossAttention(FireRedLIDAttention):
|
||||
def _init_attn(self, cache_config, quant_config, prefix: str) -> None:
|
||||
self.attn = CrossAttention(
|
||||
self.num_heads,
|
||||
self.head_dim,
|
||||
self.scaling,
|
||||
num_kv_heads=self.num_kv_heads,
|
||||
cache_config=cache_config,
|
||||
quant_config=quant_config,
|
||||
prefix=f"{prefix}.attn",
|
||||
)
|
||||
|
||||
def forward(
|
||||
self,
|
||||
hidden_states: torch.Tensor,
|
||||
encoder_hidden_states: torch.Tensor | None,
|
||||
) -> torch.Tensor:
|
||||
q, _ = self.w_qs(hidden_states)
|
||||
if encoder_hidden_states is not None:
|
||||
k, _ = self.w_ks(encoder_hidden_states)
|
||||
v, _ = self.w_vs(encoder_hidden_states)
|
||||
else:
|
||||
k = v = None
|
||||
|
||||
attn_output = self.attn(q, k, v)
|
||||
output, _ = self.fc(attn_output)
|
||||
return output
|
||||
|
||||
|
||||
class FireRedLIDFFN(nn.Module):
|
||||
def __init__(self, d_model: int, d_ff: int):
|
||||
super().__init__()
|
||||
self.w_1 = ReplicatedLinear(d_model, d_ff, bias=True)
|
||||
self.act = nn.GELU()
|
||||
self.w_2 = ReplicatedLinear(d_ff, d_model, bias=True)
|
||||
|
||||
def forward(self, x: torch.Tensor) -> torch.Tensor:
|
||||
x, _ = self.w_1(x)
|
||||
x = self.act(x)
|
||||
x, _ = self.w_2(x)
|
||||
return x
|
||||
|
||||
|
||||
class FireRedLIDDecoderLayer(nn.Module):
|
||||
"""vLLM-native decoder layer while preserving FireRedLID parameter names."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
d_model: int,
|
||||
n_head: int,
|
||||
*,
|
||||
vllm_config: VllmConfig,
|
||||
prefix: str = "",
|
||||
):
|
||||
super().__init__()
|
||||
self.self_attn_norm = nn.LayerNorm(d_model)
|
||||
self.self_attn = FireRedLIDSelfAttention(
|
||||
d_model,
|
||||
n_head,
|
||||
vllm_config=vllm_config,
|
||||
prefix=f"{prefix}.self_attn",
|
||||
)
|
||||
|
||||
self.cross_attn_norm = nn.LayerNorm(d_model)
|
||||
self.cross_attn = FireRedLIDCrossAttention(
|
||||
d_model,
|
||||
n_head,
|
||||
vllm_config=vllm_config,
|
||||
prefix=f"{prefix}.cross_attn",
|
||||
)
|
||||
|
||||
self.mlp_norm = nn.LayerNorm(d_model)
|
||||
self.mlp = FireRedLIDFFN(d_model, d_model * 4)
|
||||
|
||||
def forward(
|
||||
self,
|
||||
hidden_states: torch.Tensor,
|
||||
encoder_hidden_states: torch.Tensor | None,
|
||||
) -> torch.Tensor:
|
||||
residual = hidden_states
|
||||
hidden_states = self.self_attn_norm(hidden_states)
|
||||
hidden_states = self.self_attn(hidden_states)
|
||||
hidden_states = residual + hidden_states
|
||||
|
||||
residual = hidden_states
|
||||
hidden_states = self.cross_attn_norm(hidden_states)
|
||||
hidden_states = self.cross_attn(hidden_states, encoder_hidden_states)
|
||||
hidden_states = residual + hidden_states
|
||||
|
||||
residual = hidden_states
|
||||
hidden_states = self.mlp_norm(hidden_states)
|
||||
hidden_states = residual + self.mlp(hidden_states)
|
||||
|
||||
return hidden_states
|
||||
|
||||
|
||||
class FireRedLIDDecoder(nn.Module):
|
||||
def __init__(self, *, vllm_config: VllmConfig, prefix: str = ""):
|
||||
super().__init__()
|
||||
config = vllm_config.model_config.hf_config
|
||||
self.pad_id = getattr(config, "pad_token_id", 2)
|
||||
self.n_layers = getattr(config, "n_layers_lid_dec", 6)
|
||||
self.d_model = getattr(config, "d_model", 1280)
|
||||
self.scale = self.d_model**0.5
|
||||
|
||||
self.tgt_word_emb = nn.Embedding(
|
||||
getattr(config, "vocab_size", 120),
|
||||
self.d_model,
|
||||
padding_idx=self.pad_id,
|
||||
)
|
||||
self.positional_encoding = FireRedLIDPositionalEmbedding(
|
||||
self.d_model,
|
||||
max_len=getattr(config, "pe_maxlen", 5000),
|
||||
)
|
||||
|
||||
self.layer_stack = nn.ModuleList(
|
||||
[
|
||||
FireRedLIDDecoderLayer(
|
||||
self.d_model,
|
||||
getattr(config, "n_head", 20),
|
||||
vllm_config=vllm_config,
|
||||
prefix=f"{prefix}.layer_stack.{idx}",
|
||||
)
|
||||
for idx in range(self.n_layers)
|
||||
]
|
||||
)
|
||||
self.layer_norm_out = nn.LayerNorm(self.d_model)
|
||||
|
||||
def forward(
|
||||
self,
|
||||
input_ids: torch.Tensor,
|
||||
positions: torch.Tensor,
|
||||
encoder_hidden_states: torch.Tensor | None,
|
||||
) -> torch.Tensor:
|
||||
hidden_states = self.tgt_word_emb(input_ids) * self.scale
|
||||
hidden_states = hidden_states + self.positional_encoding(positions)
|
||||
|
||||
for layer in self.layer_stack:
|
||||
hidden_states = layer(hidden_states, encoder_hidden_states)
|
||||
|
||||
hidden_states = self.layer_norm_out(hidden_states)
|
||||
return hidden_states
|
||||
|
||||
def embed_input_ids(self, input_ids: torch.Tensor) -> torch.Tensor:
|
||||
return self.tgt_word_emb(input_ids)
|
||||
|
||||
|
||||
class FireRedLIDModel(nn.Module):
|
||||
def __init__(self, *, vllm_config: VllmConfig, prefix: str = ""):
|
||||
super().__init__()
|
||||
config = vllm_config.model_config.hf_config
|
||||
|
||||
self.encoder = FireRedLIDEncoder(
|
||||
idim=getattr(config, "idim", 80),
|
||||
n_layers_enc=getattr(config, "n_layers_enc", 16),
|
||||
n_head=getattr(config, "n_head", 20),
|
||||
d_model=getattr(config, "d_model", 1280),
|
||||
kernel_size=getattr(config, "kernel_size", 33),
|
||||
pe_maxlen=getattr(config, "pe_maxlen", 5000),
|
||||
)
|
||||
|
||||
self.decoder = FireRedLIDDecoder(
|
||||
vllm_config=vllm_config,
|
||||
prefix=maybe_prefix(prefix, "decoder"),
|
||||
)
|
||||
|
||||
def forward(
|
||||
self,
|
||||
input_ids: torch.Tensor,
|
||||
positions: torch.Tensor,
|
||||
encoder_outputs: list[torch.Tensor] | None = None,
|
||||
) -> torch.Tensor:
|
||||
enc_states = (
|
||||
torch.cat(encoder_outputs, dim=0)
|
||||
if encoder_outputs and len(encoder_outputs) > 0
|
||||
else None
|
||||
)
|
||||
decoder_outputs = self.decoder(
|
||||
input_ids=input_ids,
|
||||
positions=positions,
|
||||
encoder_hidden_states=enc_states,
|
||||
)
|
||||
return decoder_outputs
|
||||
|
||||
def get_encoder_outputs(
|
||||
self,
|
||||
speech: torch.Tensor | list[torch.Tensor],
|
||||
speech_lengths: torch.Tensor | list[torch.Tensor],
|
||||
) -> tuple[torch.Tensor, torch.Tensor]:
|
||||
"""Run the encoder and return padded outputs plus true sequence lengths."""
|
||||
enc_output, enc_lengths, _ = self.encoder(speech, speech_lengths)
|
||||
return enc_output, enc_lengths
|
||||
|
||||
|
||||
class FireRedLIDProcessingInfo(BaseProcessingInfo):
|
||||
def get_hf_config(self):
|
||||
return self.ctx.get_hf_config()
|
||||
|
||||
def get_supported_mm_limits(self) -> Mapping[str, int | None]:
|
||||
return {"audio": 1}
|
||||
|
||||
def get_feature_extractor(self, **kwargs):
|
||||
hf_processor = self.get_hf_processor(**kwargs)
|
||||
feature_extractor = hf_processor.feature_extractor
|
||||
return feature_extractor
|
||||
|
||||
def get_data_parser(self) -> MultiModalDataParser:
|
||||
feature_extractor = self.get_feature_extractor()
|
||||
return MultiModalDataParser(
|
||||
target_sr=feature_extractor.sampling_rate,
|
||||
target_channels=1,
|
||||
)
|
||||
|
||||
@property
|
||||
def skip_prompt_length_check(self) -> bool:
|
||||
return True
|
||||
|
||||
def get_num_audio_tokens(self) -> int:
|
||||
# For encoder profiling – return a reasonable dummy length.
|
||||
# This doesn't affect actual inference since encoder processes
|
||||
# variable-length features.
|
||||
return 1
|
||||
|
||||
|
||||
class FireRedLIDDummyInputsBuilder(BaseDummyInputsBuilder[FireRedLIDProcessingInfo]):
|
||||
def get_dummy_text(self, mm_counts: Mapping[str, int]) -> str:
|
||||
return "<sos>"
|
||||
|
||||
def get_dummy_mm_data(
|
||||
self,
|
||||
seq_len: int,
|
||||
mm_counts: Mapping[str, int],
|
||||
mm_options: Mapping[str, BaseDummyOptions],
|
||||
) -> MultiModalDataDict:
|
||||
feature_extractor = self.info.get_feature_extractor()
|
||||
sampling_rate = feature_extractor.sampling_rate
|
||||
audio_len = feature_extractor.chunk_length * sampling_rate
|
||||
num_audios = mm_counts.get("audio", 0)
|
||||
audio_overrides = mm_options.get("audio")
|
||||
assert audio_overrides is None or isinstance(audio_overrides, AudioDummyOptions)
|
||||
return {
|
||||
"audio": self._get_dummy_audios(
|
||||
length=audio_len,
|
||||
num_audios=num_audios,
|
||||
overrides=audio_overrides,
|
||||
)
|
||||
}
|
||||
|
||||
|
||||
class FireRedLIDMultiModalProcessor(
|
||||
EncDecMultiModalProcessor[FireRedLIDProcessingInfo]
|
||||
):
|
||||
def create_encoder_prompt(
|
||||
self,
|
||||
prompt: list[int],
|
||||
mm_items: MultiModalDataItems,
|
||||
) -> list[int]:
|
||||
# Dummy encoder prompt for profiling (encoder only processes audio).
|
||||
return [0]
|
||||
|
||||
def _get_hf_processor_text(self, mm_counts: Mapping[str, int]) -> str:
|
||||
return self.dummy_inputs.get_dummy_text(mm_counts)
|
||||
|
||||
def _preprocess_hf_mm_data(
|
||||
self,
|
||||
mm_data: Mapping[str, object],
|
||||
hf_processor_mm_kwargs: Mapping[str, object],
|
||||
) -> tuple[Mapping[str, object], Mapping[str, object]]:
|
||||
feature_extractor = self.info.get_feature_extractor(**hf_processor_mm_kwargs)
|
||||
|
||||
mm_data = dict(mm_data)
|
||||
mm_data["audio"] = mm_data.pop("audios")
|
||||
|
||||
hf_processor_mm_kwargs = dict(
|
||||
**hf_processor_mm_kwargs,
|
||||
sampling_rate=feature_extractor.sampling_rate,
|
||||
)
|
||||
|
||||
return mm_data, hf_processor_mm_kwargs
|
||||
|
||||
def _postprocess_hf_mm_data(
|
||||
self,
|
||||
mm_data: Mapping[str, object],
|
||||
hf_processor_mm_kwargs: Mapping[str, object],
|
||||
processed_data: BatchFeature,
|
||||
) -> BatchFeature:
|
||||
if "labels" in processed_data:
|
||||
processed_data["input_ids"] = processed_data.pop("labels")
|
||||
|
||||
return processed_data
|
||||
|
||||
def _get_mm_fields_config(
|
||||
self,
|
||||
hf_inputs: BatchFeature,
|
||||
hf_processor_mm_kwargs: Mapping[str, object],
|
||||
) -> Mapping[str, MultiModalFieldConfig]:
|
||||
return dict(
|
||||
input_features=MultiModalFieldConfig.batched("audio"),
|
||||
speech_lengths=MultiModalFieldConfig.batched("audio"),
|
||||
fake_token_lengths=MultiModalFieldConfig.batched("audio", keep_on_cpu=True),
|
||||
)
|
||||
|
||||
def _get_prompt_updates(
|
||||
self,
|
||||
mm_items: MultiModalDataItems,
|
||||
hf_processor_mm_kwargs: Mapping[str, object],
|
||||
out_mm_kwargs: MultiModalKwargsItems,
|
||||
) -> Sequence[PromptUpdate]:
|
||||
out_mm_data = out_mm_kwargs.get_data()
|
||||
fake_token_lengths = out_mm_data.get("fake_token_lengths")
|
||||
|
||||
if fake_token_lengths is None:
|
||||
# Fallback to max encoder output length if not available
|
||||
audio_output_lengths = []
|
||||
else:
|
||||
assert isinstance(fake_token_lengths, torch.Tensor)
|
||||
audio_output_lengths = fake_token_lengths.tolist()
|
||||
|
||||
def get_replacement(item_idx: int):
|
||||
if audio_output_lengths:
|
||||
num_tokens = int(audio_output_lengths[item_idx])
|
||||
else:
|
||||
num_tokens = self.info.get_num_audio_tokens()
|
||||
return [0] * num_tokens
|
||||
|
||||
return [
|
||||
PromptReplacement(
|
||||
modality="audio",
|
||||
target=[0],
|
||||
replacement=get_replacement,
|
||||
)
|
||||
]
|
||||
|
||||
|
||||
# FireRedLID supports a wider set of languages than Whisper's shared list.
|
||||
# Only ISO 639-1 codes are listed; FireRedLID's dialect tokens (mandarin,
|
||||
# xinan, wu, …) are output tokens but not valid language *request* codes.
|
||||
_FIREREDLID_SUPPORTED_LANGUAGES: Mapping[str, str] = {
|
||||
**ISO639_1_SUPPORTED_LANGS,
|
||||
"am": "Amharic",
|
||||
"as": "Assamese",
|
||||
"ba": "Bashkir",
|
||||
"bn": "Bengali",
|
||||
"bo": "Tibetan",
|
||||
"br": "Breton",
|
||||
"eu": "Basque",
|
||||
"fo": "Faroese",
|
||||
"gu": "Gujarati",
|
||||
"ha": "Hausa",
|
||||
"haw": "Hawaiian",
|
||||
"ht": "Haitian Creole",
|
||||
"jw": "Javanese",
|
||||
"ka": "Georgian",
|
||||
"km": "Khmer",
|
||||
"la": "Latin",
|
||||
"lb": "Luxembourgish",
|
||||
"ln": "Lingala",
|
||||
"lo": "Lao",
|
||||
"mg": "Malagasy",
|
||||
"ml": "Malayalam",
|
||||
"mn": "Mongolian",
|
||||
"mt": "Maltese",
|
||||
"my": "Myanmar",
|
||||
"nn": "Nynorsk",
|
||||
"oc": "Occitan",
|
||||
"pa": "Panjabi",
|
||||
"ps": "Pashto",
|
||||
"sa": "Sanskrit",
|
||||
"sd": "Sindhi",
|
||||
"si": "Sinhala",
|
||||
"sn": "Shona",
|
||||
"so": "Somali",
|
||||
"sq": "Albanian",
|
||||
"su": "Sundanese",
|
||||
"te": "Telugu",
|
||||
"tg": "Tajik",
|
||||
"tk": "Turkmen",
|
||||
"tt": "Tatar",
|
||||
"uz": "Uzbek",
|
||||
"yi": "Yiddish",
|
||||
"yo": "Yoruba",
|
||||
"yue": "Cantonese",
|
||||
}
|
||||
|
||||
|
||||
@MULTIMODAL_REGISTRY.register_processor(
|
||||
FireRedLIDMultiModalProcessor,
|
||||
info=FireRedLIDProcessingInfo,
|
||||
dummy_inputs=FireRedLIDDummyInputsBuilder,
|
||||
)
|
||||
class FireRedLIDForConditionalGeneration(
|
||||
nn.Module, SupportsTranscription, SupportsMultiModal
|
||||
):
|
||||
# -- SupportsTranscription protocol attributes --
|
||||
supports_transcription_only = True
|
||||
supported_languages = _FIREREDLID_SUPPORTED_LANGUAGES
|
||||
|
||||
hf_to_vllm_mapper = WeightsMapper(
|
||||
orig_to_new_substr={
|
||||
"encoder.": "model.encoder.",
|
||||
"lid_decoder.": "model.decoder.",
|
||||
# Encoder FFN: nn.Sequential indices → named children
|
||||
"net.0": "pre_layer_norm",
|
||||
"net.1": "linear_expand",
|
||||
"net.4": "linear_project",
|
||||
},
|
||||
orig_to_new_prefix={
|
||||
# Position encoding buffers are rebuilt at init, and the output
|
||||
# projection is tied to the embedding.
|
||||
"model.encoder.positional_encoding.pe": None,
|
||||
"model.decoder.positional_encoding.pe": None,
|
||||
"model.decoder.tgt_word_prj.weight": None,
|
||||
"proj_out.": None,
|
||||
},
|
||||
)
|
||||
|
||||
def __init__(self, *, vllm_config: VllmConfig, prefix: str = ""):
|
||||
super().__init__()
|
||||
config = vllm_config.model_config.hf_config
|
||||
self.config = config
|
||||
self.dtype = vllm_config.model_config.dtype
|
||||
|
||||
with self._mark_composite_model(
|
||||
vllm_config,
|
||||
language_targets=FireRedLIDDecoder,
|
||||
tower_targets={"audio": FireRedLIDEncoder},
|
||||
):
|
||||
self.model = FireRedLIDModel(
|
||||
vllm_config=vllm_config,
|
||||
prefix=maybe_prefix(prefix, "model"),
|
||||
)
|
||||
|
||||
self.proj_out = ParallelLMHead(
|
||||
getattr(config, "vocab_size", 120),
|
||||
getattr(config, "d_model", 1280),
|
||||
quant_config=vllm_config.quant_config,
|
||||
prefix=maybe_prefix(prefix, "proj_out"),
|
||||
)
|
||||
self.proj_out = self.proj_out.tie_weights(self.model.decoder.tgt_word_emb)
|
||||
|
||||
logit_scale = getattr(config, "logit_scale", 1.0)
|
||||
self.logits_processor = LogitsProcessor(
|
||||
getattr(config, "vocab_size", 120),
|
||||
scale=logit_scale,
|
||||
)
|
||||
|
||||
def forward(
|
||||
self,
|
||||
input_ids: torch.Tensor,
|
||||
positions: torch.Tensor,
|
||||
encoder_outputs: list[torch.Tensor] | None = None,
|
||||
**kwargs,
|
||||
) -> torch.Tensor:
|
||||
if encoder_outputs is None:
|
||||
encoder_outputs = []
|
||||
decoder_outputs = self.model(
|
||||
input_ids=input_ids,
|
||||
positions=positions,
|
||||
encoder_outputs=encoder_outputs,
|
||||
)
|
||||
return decoder_outputs
|
||||
|
||||
def embed_multimodal(self, **kwargs: object) -> MultiModalEmbeddings:
|
||||
"""Run encoder on audio features and return per-item embeddings."""
|
||||
audio_input = self._parse_and_validate_audio_input(**kwargs)
|
||||
|
||||
speech = audio_input["input_features"]
|
||||
speech_lengths = audio_input["speech_lengths"]
|
||||
if speech is None or speech_lengths is None:
|
||||
return []
|
||||
|
||||
# When audio items have different time lengths, vLLM's
|
||||
# MultiModalBatchedField._reduce_data returns a plain
|
||||
# list[Tensor] instead of a stacked Tensor. The encoder
|
||||
# expects a padded [B, Tmax, feat_dim] Tensor, so we
|
||||
# normalise both speech and speech_lengths here.
|
||||
if isinstance(speech, (list, tuple)):
|
||||
# Each element: [Ti, feat_dim] (or [1, Ti, feat_dim])
|
||||
tensors = [
|
||||
s.squeeze(0) if s.dim() == 3 and s.size(0) == 1 else s for s in speech
|
||||
]
|
||||
device = tensors[0].device
|
||||
dtype = tensors[0].dtype
|
||||
feat_dim = tensors[0].shape[-1]
|
||||
lengths = torch.tensor(
|
||||
[t.size(0) for t in tensors],
|
||||
device=device,
|
||||
dtype=torch.int32,
|
||||
)
|
||||
t_max = int(lengths.max().item())
|
||||
# Pre-allocate zero-padded batch tensor
|
||||
speech = torch.zeros(
|
||||
(len(tensors), t_max, feat_dim),
|
||||
device=device,
|
||||
dtype=dtype,
|
||||
)
|
||||
for i, t in enumerate(tensors):
|
||||
speech[i, : t.size(0)] = t
|
||||
speech_lengths = lengths
|
||||
else:
|
||||
# Already a batched Tensor [B, T, feat_dim]
|
||||
if speech.dim() == 2:
|
||||
speech = speech.unsqueeze(0)
|
||||
|
||||
speech_lengths = torch.as_tensor(
|
||||
speech_lengths, dtype=torch.int32, device=speech.device
|
||||
)
|
||||
|
||||
enc_output, enc_lengths = self.model.get_encoder_outputs(
|
||||
speech=speech,
|
||||
speech_lengths=speech_lengths,
|
||||
)
|
||||
|
||||
# vLLM expects one 2D tensor per multimodal item. Slice each batch entry
|
||||
# by the true encoder length so cross-attention never sees padded frames.
|
||||
return tuple(
|
||||
enc_output[i, : max(0, int(enc_lengths[i].item()))]
|
||||
for i in range(enc_output.size(0))
|
||||
)
|
||||
|
||||
def embed_input_ids(
|
||||
self,
|
||||
input_ids: torch.Tensor,
|
||||
multimodal_embeddings: MultiModalEmbeddings | None = None,
|
||||
*,
|
||||
is_multimodal: torch.Tensor | None = None,
|
||||
) -> torch.Tensor:
|
||||
return self.model.decoder.embed_input_ids(input_ids)
|
||||
|
||||
def _parse_and_validate_audio_input(
|
||||
self, **kwargs: object
|
||||
) -> FireRedLIDAudioInputs:
|
||||
input_features = kwargs.pop("input_features", None)
|
||||
speech_lengths = kwargs.pop("speech_lengths", None)
|
||||
fake_token_lengths = kwargs.pop("fake_token_lengths", None)
|
||||
return FireRedLIDAudioInputs(
|
||||
input_features=input_features,
|
||||
speech_lengths=speech_lengths,
|
||||
fake_token_lengths=fake_token_lengths,
|
||||
)
|
||||
|
||||
def compute_logits(self, hidden_states: torch.Tensor) -> torch.Tensor:
|
||||
logits = self.logits_processor(self.proj_out, hidden_states)
|
||||
return logits
|
||||
|
||||
@classmethod
|
||||
def validate_language(cls, language: str | None) -> str | None:
|
||||
# FireRedLID is a language *identification* model – the caller does
|
||||
# not need to specify a language up-front. Accept None silently.
|
||||
if language is None:
|
||||
return None
|
||||
return super().validate_language(language)
|
||||
|
||||
@classmethod
|
||||
def get_generation_prompt(
|
||||
cls,
|
||||
stt_params: SpeechToTextParams,
|
||||
) -> PromptType:
|
||||
"""Build the prompt for the FireRedLID encoder-decoder model.
|
||||
|
||||
The decoder receives a single <sos> token; the encoder processes
|
||||
the raw audio waveform via the multimodal pipeline.
|
||||
"""
|
||||
prompt: PromptType = {
|
||||
"encoder_prompt": {
|
||||
"prompt": "",
|
||||
"multi_modal_data": {
|
||||
"audio": (
|
||||
stt_params.audio,
|
||||
int(stt_params.stt_config.sample_rate),
|
||||
),
|
||||
},
|
||||
},
|
||||
"decoder_prompt": {
|
||||
"prompt": "<sos>",
|
||||
},
|
||||
}
|
||||
return prompt
|
||||
|
||||
@classmethod
|
||||
def get_speech_to_text_config(
|
||||
cls,
|
||||
model_config: ModelConfig,
|
||||
task_type: Literal["transcribe", "translate"],
|
||||
) -> SpeechToTextConfig:
|
||||
processor = cached_processor_from_config(model_config)
|
||||
return SpeechToTextConfig(
|
||||
max_audio_clip_s=processor.feature_extractor.chunk_length,
|
||||
sample_rate=processor.feature_extractor.sampling_rate,
|
||||
# LID output is at most 2 tokens – no chunking needed.
|
||||
min_energy_split_window_size=None,
|
||||
)
|
||||
|
||||
@classmethod
|
||||
def post_process_output(cls, text: str) -> str:
|
||||
# Strip any leading/trailing whitespace from the raw LID output.
|
||||
return text.strip()
|
||||
|
||||
def load_weights(self, weights: Iterable[tuple[str, torch.Tensor]]) -> set[str]:
|
||||
loader = AutoWeightsLoader(self)
|
||||
return loader.load_weights(weights, mapper=self.hf_to_vllm_mapper)
|
||||
@@ -1,243 +0,0 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
|
||||
from collections.abc import Set
|
||||
|
||||
import numpy as np
|
||||
import torch
|
||||
|
||||
from vllm.config import ModelConfig, VllmConfig
|
||||
from vllm.logger import init_logger
|
||||
from vllm.model_executor.layers.pooler import (
|
||||
DispatchPooler,
|
||||
PoolingParamsUpdate,
|
||||
)
|
||||
from vllm.model_executor.layers.pooler.activations import PoolerNormalize
|
||||
from vllm.model_executor.layers.pooler.seqwise import (
|
||||
EmbeddingPoolerHead,
|
||||
SequencePooler,
|
||||
SequencePoolingMethod,
|
||||
SequencePoolingMethodOutput,
|
||||
get_seq_pooling_method,
|
||||
)
|
||||
from vllm.model_executor.layers.pooler.tokwise import pooler_for_token_embed
|
||||
from vllm.model_executor.models.llama import LlamaForCausalLM
|
||||
from vllm.tasks import PoolingTask
|
||||
from vllm.tokenizers import cached_tokenizer_from_config
|
||||
from vllm.v1.pool.metadata import PoolingMetadata
|
||||
|
||||
from .interfaces_base import default_pooling_type
|
||||
|
||||
logger = init_logger(__name__)
|
||||
|
||||
|
||||
class GritLMMeanPool(SequencePoolingMethod):
|
||||
"""As `MeanPool`, but only includes non-instruction tokens."""
|
||||
|
||||
def __init__(self, model_config: ModelConfig):
|
||||
super().__init__()
|
||||
|
||||
self.model_config = model_config
|
||||
|
||||
tokenizer = cached_tokenizer_from_config(self.model_config)
|
||||
|
||||
# Collect the tokens needed for pattern matching.
|
||||
# "▁<" is different from "_<". The former uses "▁" to indicate that
|
||||
# the next token is the start of a word.
|
||||
# "<0x0A>" is the newline token (i.e. "\n")."
|
||||
self.token_ids = {
|
||||
tok: tokenizer.convert_tokens_to_ids([tok])[0]
|
||||
for tok in ["<s>", "▁<", "<", "|", "embed", ">", "<0x0A>", "user"]
|
||||
}
|
||||
|
||||
def tokens_to_ids(tokens: list[str]) -> np.ndarray:
|
||||
return np.array([self.token_ids[token] for token in tokens])
|
||||
|
||||
self.user_pattern_ids = tokens_to_ids(["▁<", "|", "user", "|", ">", "<0x0A>"])
|
||||
self.embed_newline_pattern_ids = tokens_to_ids(
|
||||
["<0x0A>", "<", "|", "embed", "|", ">", "<0x0A>"]
|
||||
)
|
||||
self.embed_pattern_ids = tokens_to_ids(["▁<", "|", "embed", "|", ">", "<0x0A>"])
|
||||
|
||||
def _find_array(
|
||||
self,
|
||||
arr: np.ndarray,
|
||||
target: np.ndarray,
|
||||
start_idx: int = 0,
|
||||
end_idx: int | None = None,
|
||||
) -> int:
|
||||
"""
|
||||
Find the first occurrence of `target` in `arr` starting from
|
||||
`start_idx`.
|
||||
|
||||
Args:
|
||||
arr: The array to search within.
|
||||
target: The consecutive subsequence to find.
|
||||
start_idx: The starting index to search from (inclusive).
|
||||
end_idx: The ending index to search from (exclusive).
|
||||
|
||||
Returns:
|
||||
The index of the first occurrence of `target` in `arr`.
|
||||
"""
|
||||
if start_idx < 0:
|
||||
raise ValueError("`start_idx` must be non-negative")
|
||||
if len(arr) == 0 or len(target) == 0:
|
||||
raise ValueError("Empty `arr` or `target` not allowed")
|
||||
|
||||
arr_len = len(arr)
|
||||
target_len = len(target)
|
||||
|
||||
if end_idx is None:
|
||||
end_idx = arr_len
|
||||
|
||||
for i in range(start_idx, min(end_idx, arr_len - target_len + 1)):
|
||||
if (arr[i : i + target_len] == target).all():
|
||||
return i
|
||||
|
||||
return -1
|
||||
|
||||
def _get_instruction_len(self, prompt_token_ids: np.ndarray) -> int:
|
||||
"""
|
||||
Get the length of the instruction in the prompt.
|
||||
|
||||
We do a pattern matching to find the instruction in the prompt,
|
||||
and then return the length of the instruction.
|
||||
|
||||
The pattern matching is done using integers instead of strings
|
||||
because the prompt is given as a list of token IDs.
|
||||
"""
|
||||
instruction_len = 0
|
||||
|
||||
# Return no instruction in case of missing BOS token.
|
||||
if prompt_token_ids[0] != self.token_ids["<s>"]:
|
||||
logger.warning(
|
||||
"BOS token not found in prompt, "
|
||||
"thus using empty string for instruction. "
|
||||
"GritLM requires BOS token in prompt."
|
||||
)
|
||||
return instruction_len
|
||||
|
||||
# If user pattern is found in the prompt, that means there should be
|
||||
# a newline token before the embed pattern.
|
||||
embed_pattern_ids = self.embed_pattern_ids
|
||||
if (
|
||||
self._find_array(
|
||||
prompt_token_ids, self.user_pattern_ids, start_idx=1, end_idx=2
|
||||
)
|
||||
== 1
|
||||
):
|
||||
embed_pattern_ids = self.embed_newline_pattern_ids
|
||||
|
||||
# Find the embed pattern in the prompt.
|
||||
found_embed_pattern_idx = self._find_array(
|
||||
prompt_token_ids, embed_pattern_ids, start_idx=1
|
||||
)
|
||||
|
||||
if found_embed_pattern_idx != -1:
|
||||
instruction_len = found_embed_pattern_idx + len(embed_pattern_ids)
|
||||
else:
|
||||
logger.warning(
|
||||
"Query instruction not found in prompt, "
|
||||
"thus using BOS token as instruction instead. "
|
||||
"GritLM requires query instruction in prompt."
|
||||
)
|
||||
instruction_len = 1
|
||||
|
||||
return instruction_len
|
||||
|
||||
def get_supported_tasks(self) -> Set[PoolingTask]:
|
||||
return {"embed"}
|
||||
|
||||
def get_pooling_updates(self, task: PoolingTask) -> PoolingParamsUpdate:
|
||||
return PoolingParamsUpdate(requires_token_ids=True)
|
||||
|
||||
def forward(
|
||||
self,
|
||||
hidden_states: torch.Tensor,
|
||||
pooling_metadata: PoolingMetadata,
|
||||
) -> SequencePoolingMethodOutput:
|
||||
prompt_lens = pooling_metadata.prompt_lens
|
||||
prompt_token_ids = pooling_metadata.get_prompt_token_ids_cpu()
|
||||
instr_lens = torch.tensor(
|
||||
[
|
||||
self._get_instruction_len(token_ids.numpy())
|
||||
for token_ids in prompt_token_ids
|
||||
],
|
||||
device="cpu",
|
||||
)
|
||||
|
||||
offset = 0
|
||||
pooled_data = list[torch.Tensor]()
|
||||
for prompt_len, instr_len in zip(prompt_lens, instr_lens):
|
||||
pooled_data.append(
|
||||
hidden_states[offset + instr_len : offset + prompt_len].mean(
|
||||
dim=0, dtype=torch.float32
|
||||
)
|
||||
)
|
||||
offset += prompt_len
|
||||
|
||||
return pooled_data
|
||||
|
||||
|
||||
class GritLMPooler(SequencePooler):
|
||||
def __init__(self, model_config: ModelConfig):
|
||||
pooler_config = model_config.pooler_config
|
||||
assert pooler_config is not None
|
||||
|
||||
super().__init__(
|
||||
pooling=(
|
||||
GritLMMeanPool(model_config)
|
||||
if pooler_config.seq_pooling_type == "MEAN"
|
||||
else get_seq_pooling_method(pooler_config.seq_pooling_type)
|
||||
),
|
||||
head=EmbeddingPoolerHead(
|
||||
head_dtype=model_config.head_dtype,
|
||||
activation=PoolerNormalize(),
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
@default_pooling_type(seq_pooling_type="MEAN")
|
||||
class GritLM(LlamaForCausalLM):
|
||||
"""This class implements the embedding model for parasail-ai/GritLM-7B-vllm.
|
||||
|
||||
The class inherits from LlamaForCausalLM and provides a custom pooling
|
||||
layer.
|
||||
|
||||
The main difference between the pooling layer in GritLM and the one in
|
||||
LlamaForCausalLM is that GritLM ignores the query instruction in the prompt
|
||||
when pooling the hidden states.
|
||||
|
||||
Embedding prompts should be in the following format:
|
||||
- With instruction: "<|user|>\nINSTRUCTION\n<|embed|>\nPROMPT".
|
||||
- Without instruction: "<|embed|>\nPROMPT".
|
||||
|
||||
Generation prompts should be in the following format:
|
||||
- "<|user|>\nPROMPT\n<|assistant|>\n"
|
||||
"""
|
||||
|
||||
is_pooling_model = True
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
vllm_config: VllmConfig,
|
||||
prefix: str = "",
|
||||
**kwargs,
|
||||
) -> None:
|
||||
if vllm_config.model_config.runner_type == "pooling":
|
||||
hf_config = vllm_config.model_config.hf_config
|
||||
hf_config.is_causal = False
|
||||
|
||||
vllm_config.cache_config.sliding_window = None
|
||||
|
||||
hf_config.sliding_window = None
|
||||
|
||||
super().__init__(vllm_config=vllm_config, prefix=prefix, **kwargs)
|
||||
|
||||
pooler_config = vllm_config.model_config.pooler_config
|
||||
if pooler_config is not None:
|
||||
self.pooler = DispatchPooler(
|
||||
{
|
||||
"token_embed": pooler_for_token_embed(pooler_config),
|
||||
"embed": GritLMPooler(vllm_config.model_config),
|
||||
}
|
||||
)
|
||||
File diff suppressed because it is too large
Load Diff
@@ -273,7 +273,7 @@ class LlamaDecoderLayer(nn.Module):
|
||||
# By default, Llama uses causal attention as it is a decoder-only model.
|
||||
# You can override the HF config with `is_causal=False` to enable
|
||||
# bidirectional attention, which is used in some embedding models
|
||||
# (e.g. parasail-ai/GritLM-7B-vllm)
|
||||
# (e.g. nvidia/llama-nemotron-embed-1b-v2)
|
||||
if getattr(config, "is_causal", True):
|
||||
attn_type = AttentionType.DECODER
|
||||
else:
|
||||
|
||||
@@ -1,317 +0,0 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
|
||||
|
||||
# Adapted from https://huggingface.co/mosaicml/mpt-7b/tree/main
|
||||
import math
|
||||
from collections.abc import Iterable
|
||||
from itertools import islice
|
||||
|
||||
import torch
|
||||
import torch.nn as nn
|
||||
from transformers import MptConfig
|
||||
|
||||
from vllm.compilation.decorators import support_torch_compile
|
||||
from vllm.config import CacheConfig, VllmConfig
|
||||
from vllm.distributed import (
|
||||
get_pp_group,
|
||||
get_tensor_model_parallel_rank,
|
||||
get_tensor_model_parallel_world_size,
|
||||
)
|
||||
from vllm.model_executor.layers.activation import get_act_fn
|
||||
from vllm.model_executor.layers.attention import Attention
|
||||
from vllm.model_executor.layers.linear import (
|
||||
ColumnParallelLinear,
|
||||
QKVParallelLinear,
|
||||
RowParallelLinear,
|
||||
)
|
||||
from vllm.model_executor.layers.logits_processor import LogitsProcessor
|
||||
from vllm.model_executor.layers.quantization import QuantizationConfig
|
||||
from vllm.model_executor.layers.vocab_parallel_embedding import VocabParallelEmbedding
|
||||
from vllm.sequence import IntermediateTensors
|
||||
|
||||
from .interfaces import SupportsPP
|
||||
from .utils import (
|
||||
AutoWeightsLoader,
|
||||
make_empty_intermediate_tensors_factory,
|
||||
make_layers,
|
||||
maybe_prefix,
|
||||
)
|
||||
|
||||
|
||||
def _get_alibi_slopes(
|
||||
total_num_heads: int,
|
||||
alibi_bias_max: int,
|
||||
) -> torch.Tensor:
|
||||
next_power_of_2 = 2 ** math.ceil(math.log2(total_num_heads))
|
||||
m = torch.arange(1, next_power_of_2 + 1, dtype=torch.float32)
|
||||
m = m.mul(alibi_bias_max / next_power_of_2)
|
||||
slopes = 1.0 / torch.pow(2, m)
|
||||
if next_power_of_2 != total_num_heads:
|
||||
slopes = torch.concat([slopes[1::2], slopes[::2]])[:total_num_heads]
|
||||
return slopes
|
||||
|
||||
|
||||
class MPTAttention(nn.Module):
|
||||
def __init__(
|
||||
self,
|
||||
config: MptConfig,
|
||||
cache_config: CacheConfig | None = None,
|
||||
quant_config: QuantizationConfig | None = None,
|
||||
prefix: str = "",
|
||||
):
|
||||
super().__init__()
|
||||
self.d_model = config.d_model
|
||||
self.total_num_heads = config.n_heads
|
||||
self.head_dim = self.d_model // self.total_num_heads
|
||||
self.clip_qkv = config.attn_config.clip_qkv
|
||||
self.qk_ln = config.attn_config.qk_ln
|
||||
self.alibi_bias_max = config.attn_config.alibi_bias_max
|
||||
if "kv_n_heads" in config.attn_config:
|
||||
self.total_num_kv_heads = config.attn_config.kv_n_heads
|
||||
else:
|
||||
self.total_num_kv_heads = self.total_num_heads
|
||||
assert not config.attn_config.prefix_lm
|
||||
assert config.attn_config.alibi
|
||||
|
||||
# pylint: disable=invalid-name
|
||||
self.Wqkv = QKVParallelLinear(
|
||||
self.d_model,
|
||||
self.d_model // self.total_num_heads,
|
||||
self.total_num_heads,
|
||||
self.total_num_kv_heads,
|
||||
bias=not config.no_bias,
|
||||
quant_config=quant_config,
|
||||
prefix=f"{prefix}.Wqkv",
|
||||
)
|
||||
if self.qk_ln:
|
||||
self.q_ln = nn.LayerNorm(self.d_model)
|
||||
self.k_ln = nn.LayerNorm(self.d_model)
|
||||
self.out_proj = RowParallelLinear(
|
||||
self.d_model,
|
||||
self.d_model,
|
||||
bias=not config.no_bias,
|
||||
quant_config=quant_config,
|
||||
prefix=f"{prefix}.out_proj",
|
||||
)
|
||||
|
||||
tp_world_size = get_tensor_model_parallel_world_size()
|
||||
assert self.total_num_heads % tp_world_size == 0
|
||||
self.num_heads = self.total_num_heads // tp_world_size
|
||||
|
||||
if self.total_num_kv_heads >= tp_world_size:
|
||||
# Number of KV heads is greater than TP size, so we partition
|
||||
# the KV heads across multiple tensor parallel GPUs.
|
||||
assert self.total_num_kv_heads % tp_world_size == 0
|
||||
else:
|
||||
# Number of KV heads is less than TP size, so we replicate
|
||||
# the KV heads across multiple tensor parallel GPUs.
|
||||
assert tp_world_size % self.total_num_kv_heads == 0
|
||||
self.num_kv_heads = max(1, self.total_num_kv_heads // tp_world_size)
|
||||
self.q_size = self.num_heads * self.head_dim
|
||||
self.kv_size = self.num_kv_heads * self.head_dim
|
||||
# Create the alibi slopes and slice them.
|
||||
tp_rank = get_tensor_model_parallel_rank()
|
||||
head_start = tp_rank * self.num_heads
|
||||
head_end = (tp_rank + 1) * self.num_heads
|
||||
alibi_slopes = _get_alibi_slopes(self.total_num_heads, self.alibi_bias_max)
|
||||
alibi_slopes = alibi_slopes[head_start:head_end].tolist()
|
||||
|
||||
self.head_dim = self.d_model // self.total_num_heads
|
||||
scaling = self.head_dim**-0.5
|
||||
self.attn = Attention(
|
||||
self.num_heads,
|
||||
self.head_dim,
|
||||
scaling,
|
||||
alibi_slopes=alibi_slopes,
|
||||
num_kv_heads=self.num_kv_heads,
|
||||
cache_config=cache_config,
|
||||
quant_config=quant_config,
|
||||
prefix=f"{prefix}.attn",
|
||||
)
|
||||
|
||||
def forward(
|
||||
self,
|
||||
position_ids: torch.Tensor,
|
||||
hidden_states: torch.Tensor,
|
||||
) -> torch.Tensor:
|
||||
del position_ids # unused.
|
||||
qkv, _ = self.Wqkv(hidden_states)
|
||||
if self.clip_qkv is not None:
|
||||
qkv.clamp_(min=-self.clip_qkv, max=self.clip_qkv)
|
||||
q, k, v = qkv.split([self.q_size, self.kv_size, self.kv_size], dim=-1)
|
||||
if self.qk_ln:
|
||||
q = self.q_ln(q)
|
||||
k = self.k_ln(k)
|
||||
attn_output = self.attn(q, k, v)
|
||||
output, _ = self.out_proj(attn_output)
|
||||
return output
|
||||
|
||||
|
||||
class MPTMLP(nn.Module):
|
||||
def __init__(
|
||||
self,
|
||||
config: MptConfig,
|
||||
quant_config: QuantizationConfig | None = None,
|
||||
prefix: str = "",
|
||||
):
|
||||
super().__init__()
|
||||
hidden_size = config.d_model
|
||||
expansion_ratio = config.expansion_ratio
|
||||
intermediate_size = expansion_ratio * hidden_size
|
||||
self.up_proj = ColumnParallelLinear(
|
||||
hidden_size,
|
||||
intermediate_size,
|
||||
bias=not config.no_bias,
|
||||
quant_config=quant_config,
|
||||
prefix=f"{prefix}.up_proj",
|
||||
)
|
||||
self.act = get_act_fn("gelu")
|
||||
self.down_proj = RowParallelLinear(
|
||||
intermediate_size,
|
||||
hidden_size,
|
||||
bias=not config.no_bias,
|
||||
quant_config=quant_config,
|
||||
prefix=f"{prefix}.down_proj",
|
||||
)
|
||||
|
||||
def forward(self, x: torch.Tensor) -> torch.Tensor:
|
||||
x, _ = self.up_proj(x)
|
||||
x = self.act(x)
|
||||
x, _ = self.down_proj(x)
|
||||
return x
|
||||
|
||||
|
||||
class MPTBlock(nn.Module):
|
||||
def __init__(
|
||||
self,
|
||||
config: MptConfig,
|
||||
cache_config: CacheConfig | None = None,
|
||||
quant_config: QuantizationConfig | None = None,
|
||||
prefix: str = "",
|
||||
):
|
||||
super().__init__()
|
||||
hidden_size = config.d_model
|
||||
self.norm_1 = nn.LayerNorm(hidden_size)
|
||||
self.attn = MPTAttention(
|
||||
config, cache_config, quant_config, prefix=f"{prefix}.attn"
|
||||
)
|
||||
self.norm_2 = nn.LayerNorm(hidden_size)
|
||||
self.ffn = MPTMLP(config, quant_config, prefix=f"{prefix}.ffn")
|
||||
|
||||
def forward(
|
||||
self,
|
||||
position_ids: torch.Tensor,
|
||||
hidden_states: torch.Tensor,
|
||||
) -> torch.Tensor:
|
||||
x = self.norm_1(hidden_states)
|
||||
x = self.attn(
|
||||
position_ids=position_ids,
|
||||
hidden_states=x,
|
||||
)
|
||||
hidden_states = hidden_states + x
|
||||
x = self.norm_2(hidden_states)
|
||||
x = self.ffn(x)
|
||||
hidden_states = hidden_states + x
|
||||
return hidden_states
|
||||
|
||||
|
||||
@support_torch_compile
|
||||
class MPTModel(nn.Module):
|
||||
def __init__(self, *, vllm_config: VllmConfig, prefix: str = ""):
|
||||
super().__init__()
|
||||
|
||||
config = vllm_config.model_config.hf_config
|
||||
cache_config = vllm_config.cache_config
|
||||
quant_config = vllm_config.quant_config
|
||||
|
||||
assert config.embedding_fraction == 1.0
|
||||
assert config.norm_type == "low_precision_layernorm"
|
||||
|
||||
self.wte = VocabParallelEmbedding(
|
||||
config.vocab_size,
|
||||
config.d_model,
|
||||
)
|
||||
self.start_layer, self.end_layer, self.blocks = make_layers(
|
||||
config.n_layers,
|
||||
lambda prefix: MPTBlock(config, cache_config, quant_config, prefix=prefix),
|
||||
prefix=f"{prefix}.blocks",
|
||||
)
|
||||
self.norm_f = nn.LayerNorm(config.d_model)
|
||||
if config.no_bias:
|
||||
for module in self.modules():
|
||||
if hasattr(module, "bias") and isinstance(module.bias, nn.Parameter):
|
||||
# Remove the bias term in Linear and LayerNorm.
|
||||
module.register_parameter("bias", None)
|
||||
self.make_empty_intermediate_tensors = make_empty_intermediate_tensors_factory(
|
||||
["hidden_states"], config.d_model
|
||||
)
|
||||
|
||||
def embed_input_ids(self, input_ids: torch.Tensor) -> torch.Tensor:
|
||||
return self.wte(input_ids)
|
||||
|
||||
def forward(
|
||||
self,
|
||||
input_ids: torch.Tensor | None,
|
||||
position_ids: torch.Tensor,
|
||||
intermediate_tensors: IntermediateTensors | None,
|
||||
inputs_embeds: torch.Tensor | None = None,
|
||||
) -> torch.Tensor | IntermediateTensors:
|
||||
if get_pp_group().is_first_rank:
|
||||
if inputs_embeds is not None:
|
||||
hidden_states = inputs_embeds
|
||||
else:
|
||||
hidden_states = self.embed_input_ids(input_ids)
|
||||
else:
|
||||
assert intermediate_tensors is not None
|
||||
hidden_states = intermediate_tensors["hidden_states"]
|
||||
|
||||
for block in islice(self.blocks, self.start_layer, self.end_layer):
|
||||
hidden_states = block(position_ids, hidden_states)
|
||||
if not get_pp_group().is_last_rank:
|
||||
return IntermediateTensors({"hidden_states": hidden_states})
|
||||
hidden_states = self.norm_f(hidden_states)
|
||||
return hidden_states
|
||||
|
||||
|
||||
class MPTForCausalLM(nn.Module, SupportsPP):
|
||||
def __init__(self, *, vllm_config: VllmConfig, prefix: str = ""):
|
||||
super().__init__()
|
||||
config = vllm_config.model_config.hf_config
|
||||
quant_config = vllm_config.quant_config
|
||||
self.config = config
|
||||
self.quant_config = quant_config
|
||||
|
||||
self.transformer = MPTModel(
|
||||
vllm_config=vllm_config, prefix=maybe_prefix(prefix, "transformer")
|
||||
)
|
||||
self.lm_head = self.transformer.wte
|
||||
self.logits_processor = LogitsProcessor(config.vocab_size)
|
||||
self.make_empty_intermediate_tensors = (
|
||||
self.transformer.make_empty_intermediate_tensors
|
||||
)
|
||||
|
||||
def embed_input_ids(self, input_ids: torch.Tensor) -> torch.Tensor:
|
||||
return self.transformer.embed_input_ids(input_ids)
|
||||
|
||||
def forward(
|
||||
self,
|
||||
input_ids: torch.Tensor | None,
|
||||
positions: torch.Tensor,
|
||||
intermediate_tensors: IntermediateTensors | None = None,
|
||||
inputs_embeds: torch.Tensor | None = None,
|
||||
) -> torch.Tensor | IntermediateTensors:
|
||||
hidden_states = self.transformer(
|
||||
input_ids, positions, intermediate_tensors, inputs_embeds
|
||||
)
|
||||
return hidden_states
|
||||
|
||||
def compute_logits(
|
||||
self,
|
||||
hidden_states: torch.Tensor,
|
||||
) -> torch.Tensor | None:
|
||||
logits = self.logits_processor(self.lm_head, hidden_states)
|
||||
return logits
|
||||
|
||||
def load_weights(self, weights: Iterable[tuple[str, torch.Tensor]]) -> set[str]:
|
||||
loader = AutoWeightsLoader(self)
|
||||
return loader.load_weights(weights)
|
||||
@@ -74,7 +74,6 @@ _TEXT_GENERATION_MODELS = {
|
||||
"AfmoeForCausalLM": ("afmoe", "AfmoeForCausalLM"),
|
||||
"ApertusForCausalLM": ("apertus", "ApertusForCausalLM"),
|
||||
"ArceeForCausalLM": ("arcee", "ArceeForCausalLM"),
|
||||
"ArcticForCausalLM": ("arctic", "ArcticForCausalLM"),
|
||||
"AXK1ForCausalLM": ("AXK1", "AXK1ForCausalLM"),
|
||||
"BailingMoeForCausalLM": ("bailing_moe", "BailingMoeForCausalLM"),
|
||||
"BailingMoeV2ForCausalLM": ("bailing_moe", "BailingMoeV2ForCausalLM"),
|
||||
@@ -102,7 +101,6 @@ _TEXT_GENERATION_MODELS = {
|
||||
"ExaoneForCausalLM": ("exaone", "ExaoneForCausalLM"),
|
||||
"Exaone4ForCausalLM": ("exaone4", "Exaone4ForCausalLM"),
|
||||
"ExaoneMoeForCausalLM": ("exaone_moe", "ExaoneMoeForCausalLM"),
|
||||
"Fairseq2LlamaForCausalLM": ("fairseq2_llama", "Fairseq2LlamaForCausalLM"),
|
||||
"FalconForCausalLM": ("falcon", "FalconForCausalLM"),
|
||||
"FalconMambaForCausalLM": ("mamba", "MambaForCausalLM"),
|
||||
"FalconH1ForCausalLM": ("falcon_h1", "FalconH1ForCausalLM"),
|
||||
@@ -129,12 +127,10 @@ _TEXT_GENERATION_MODELS = {
|
||||
"GraniteMoeSharedForCausalLM": ("granitemoeshared", "GraniteMoeSharedForCausalLM"),
|
||||
"GraniteMoeSWAForCausalLM": ("granitemoeshared", "GraniteMoeSharedForCausalLM"),
|
||||
"GraniteSWAForCausalLM": ("granite", "GraniteForCausalLM"),
|
||||
"GritLM": ("gritlm", "GritLM"),
|
||||
"HrmTextForCausalLM": ("hrm_text", "HrmTextForCausalLM"),
|
||||
"HunYuanMoEV1ForCausalLM": ("hunyuan_v1", "HunYuanMoEV1ForCausalLM"),
|
||||
"HunYuanDenseV1ForCausalLM": ("hunyuan_v1", "HunYuanDenseV1ForCausalLM"),
|
||||
"HYV3ForCausalLM": ("hy_v3", "HYV3ForCausalLM"),
|
||||
"HCXVisionForCausalLM": ("hyperclovax_vision", "HCXVisionForCausalLM"),
|
||||
"HCXVisionV2ForCausalLM": ("hyperclovax_vision_v2", "HCXVisionV2ForCausalLM"),
|
||||
"HyperCLOVAXForCausalLM": ("hyperclovax", "HyperCLOVAXForCausalLM"),
|
||||
"InternLM2ForCausalLM": ("internlm2", "InternLM2ForCausalLM"),
|
||||
@@ -178,9 +174,6 @@ _TEXT_GENERATION_MODELS = {
|
||||
"MistralForCausalLM": ("mistral", "MistralForCausalLM"),
|
||||
"MistralLarge3ForCausalLM": ("mistral_large_3", "MistralLarge3ForCausalLM"),
|
||||
"MixtralForCausalLM": ("mixtral", "MixtralForCausalLM"),
|
||||
# transformers's mpt class has lower case
|
||||
"MptForCausalLM": ("mpt", "MPTForCausalLM"),
|
||||
"MPTForCausalLM": ("mpt", "MPTForCausalLM"),
|
||||
"MiMoForCausalLM": ("mimo", "MiMoForCausalLM"),
|
||||
"MiMoV2FlashForCausalLM": ("mimo_v2", "MiMoV2FlashForCausalLM"),
|
||||
"MiMoV2ForCausalLM": ("mimo_v2", "MiMoV2ForCausalLM"),
|
||||
@@ -207,14 +200,12 @@ _TEXT_GENERATION_MODELS = {
|
||||
"Qwen3MoeForCausalLM": ("qwen3_moe", "Qwen3MoeForCausalLM"),
|
||||
"Qwen3_5ForCausalLM": ("qwen3_5", "Qwen3_5ForCausalLM"),
|
||||
"Qwen3_5MoeForCausalLM": ("qwen3_5", "Qwen3_5MoeForCausalLM"),
|
||||
"RWForCausalLM": ("falcon", "FalconForCausalLM"),
|
||||
"SarvamMoEForCausalLM": ("sarvam", "SarvamMoEForCausalLM"),
|
||||
"SarvamMLAForCausalLM": ("sarvam", "SarvamMLAForCausalLM"),
|
||||
"SeedOssForCausalLM": ("seed_oss", "SeedOssForCausalLM"),
|
||||
"Step1ForCausalLM": ("step1", "Step1ForCausalLM"),
|
||||
"Step3TextForCausalLM": ("step3_text", "Step3TextForCausalLM"),
|
||||
"Step3p5ForCausalLM": ("step3p5", "Step3p5ForCausalLM"),
|
||||
"StableLMEpochForCausalLM": ("stablelm", "StablelmForCausalLM"),
|
||||
"StableLmForCausalLM": ("stablelm", "StablelmForCausalLM"),
|
||||
"SolarForCausalLM": ("solar", "SolarForCausalLM"),
|
||||
"TeleChat2ForCausalLM": ("telechat2", "TeleChat2ForCausalLM"),
|
||||
@@ -234,7 +225,6 @@ _EMBEDDING_MODELS = {
|
||||
"Gemma2Model": ("gemma2", "Gemma2ForCausalLM"),
|
||||
"Gemma3TextModel": ("gemma3", "Gemma3Model"),
|
||||
"GlmForCausalLM": ("glm", "GlmForCausalLM"),
|
||||
"GritLM": ("gritlm", "GritLM"),
|
||||
"GteModel": ("bert_with_rope", "SnowflakeGteNewModel"),
|
||||
"GteNewModel": ("bert_with_rope", "GteNewModel"),
|
||||
"JinaEmbeddingsV5Model": ("jina", "JinaEmbeddingsV5Model"),
|
||||
@@ -274,7 +264,6 @@ _EMBEDDING_MODELS = {
|
||||
# Technically Terratorch models work on images, both in
|
||||
# input and output. I am adding it here because it piggy-backs on embedding
|
||||
# models for the time being.
|
||||
"PrithviGeoSpatialMAE": ("terratorch", "Terratorch"),
|
||||
"Terratorch": ("terratorch", "Terratorch"),
|
||||
}
|
||||
|
||||
@@ -360,12 +349,6 @@ _MULTIMODAL_MODELS = {
|
||||
"BagelForConditionalGeneration": ("bagel", "BagelForConditionalGeneration"),
|
||||
"BeeForConditionalGeneration": ("bee", "BeeForConditionalGeneration"),
|
||||
"Blip2ForConditionalGeneration": ("blip2", "Blip2ForConditionalGeneration"),
|
||||
"ChameleonForConditionalGeneration": (
|
||||
"chameleon",
|
||||
"ChameleonForConditionalGeneration",
|
||||
),
|
||||
"Cheers": ("cheers", "CheersForConditionalGeneration"),
|
||||
"CheersForConditionalGeneration": ("cheers", "CheersForConditionalGeneration"),
|
||||
"Cohere2VisionForConditionalGeneration": (
|
||||
"cohere2_vision",
|
||||
"Cohere2VisionForConditionalGeneration",
|
||||
@@ -401,10 +384,6 @@ _MULTIMODAL_MODELS = {
|
||||
"FireRedASR2ForConditionalGeneration",
|
||||
),
|
||||
"FunASRForConditionalGeneration": ("funasr", "FunASRForConditionalGeneration"),
|
||||
"FireRedLIDForConditionalGeneration": (
|
||||
"fireredlid",
|
||||
"FireRedLIDForConditionalGeneration",
|
||||
),
|
||||
"FunAudioChatForConditionalGeneration": (
|
||||
"funaudiochat",
|
||||
"FunAudioChatForConditionalGeneration",
|
||||
@@ -811,6 +790,20 @@ _PREVIOUSLY_SUPPORTED_MODELS = {
|
||||
"FuyuForCausalLM": "0.25.0",
|
||||
"Plamo2ForCausalLM": "0.26.0",
|
||||
"OuroForCausalLM": "0.26.0",
|
||||
"ArcticForCausalLM": "0.28.0",
|
||||
"ChameleonForConditionalGeneration": "0.28.0",
|
||||
"Cheers": "0.28.0",
|
||||
"CheersForConditionalGeneration": "0.28.0",
|
||||
"Fairseq2LlamaForCausalLM": "0.28.0",
|
||||
"FireRedLIDForConditionalGeneration": "0.28.0",
|
||||
"GritLM": "0.28.0",
|
||||
"HCXVisionForCausalLM": "0.28.0",
|
||||
"MPTForCausalLM": "0.28.0",
|
||||
"MptForCausalLM": "0.28.0",
|
||||
"RWForCausalLM": "0.28.0",
|
||||
"StableLMEpochForCausalLM": "0.28.0",
|
||||
# Superseded by the generic "Terratorch" architecture
|
||||
"PrithviGeoSpatialMAE": "0.28.0",
|
||||
}
|
||||
|
||||
_OOT_SUPPORTED_MODELS = {
|
||||
|
||||
@@ -24,7 +24,6 @@ def _get_minicpmv_chat_template_fallback(tokenizer_name_or_path: str) -> Path |
|
||||
|
||||
_MODEL_TYPE_TO_CHAT_TEMPLATE_FALLBACK: dict[str, ChatTemplatePath] = {
|
||||
"blip-2": CHAT_TEMPLATES_DIR / "template_blip2.jinja",
|
||||
"chameleon": CHAT_TEMPLATES_DIR / "template_basic.jinja",
|
||||
"clip": CHAT_TEMPLATES_DIR / "template_basic.jinja",
|
||||
"colpali": CHAT_TEMPLATES_DIR / "template_basic.jinja",
|
||||
"deepseek_ocr": CHAT_TEMPLATES_DIR / "template_deepseek_ocr.jinja",
|
||||
|
||||
@@ -71,10 +71,8 @@ class LazyConfigDict(dict):
|
||||
|
||||
_CONFIG_REGISTRY: dict[str, type[PretrainedConfig]] = LazyConfigDict(
|
||||
afmoe="AfmoeConfig",
|
||||
arctic="ArcticConfig",
|
||||
axk1="AXK1Config",
|
||||
bagel="BagelConfig",
|
||||
umm="CheersConfig",
|
||||
chatglm="ChatGLMConfig",
|
||||
modernvbert="ColModernVBertConfig",
|
||||
colpali="ColPaliConfig",
|
||||
@@ -90,11 +88,9 @@ _CONFIG_REGISTRY: dict[str, type[PretrainedConfig]] = LazyConfigDict(
|
||||
dots3_note="Dots3NoteConfig",
|
||||
k3_dspark="K3DSparkConfig",
|
||||
flex_olmo="FlexOlmoConfig",
|
||||
fireredlid="FireRedLIDConfig",
|
||||
funaudiochat="FunAudioChatConfig",
|
||||
granite4_vision="Granite4VisionConfig",
|
||||
hyperclovax="HyperCLOVAXConfig",
|
||||
hyperclovax_vlm="HCXVisionConfig",
|
||||
hunyuan_vl="HunYuanVLConfig",
|
||||
hy_v3="HYV3Config",
|
||||
isaac="IsaacConfig",
|
||||
|
||||
@@ -16,10 +16,8 @@ import importlib
|
||||
|
||||
_CLASS_TO_MODULE: dict[str, str] = {
|
||||
"AfmoeConfig": "vllm.transformers_utils.configs.afmoe",
|
||||
"ArcticConfig": "vllm.transformers_utils.configs.arctic",
|
||||
"AXK1Config": "vllm.transformers_utils.configs.AXK1",
|
||||
"BagelConfig": "vllm.transformers_utils.configs.bagel",
|
||||
"CheersConfig": "vllm.transformers_utils.configs.cheers",
|
||||
"ChatGLMConfig": "vllm.transformers_utils.configs.chatglm",
|
||||
"ColModernVBertConfig": "vllm.transformers_utils.configs.colmodernvbert",
|
||||
"ColPaliConfig": "vllm.transformers_utils.configs.colpali",
|
||||
@@ -39,7 +37,6 @@ _CLASS_TO_MODULE: dict[str, str] = {
|
||||
"K3DSparkConfig": "vllm.transformers_utils.configs.k3_dspark",
|
||||
"DotsOCRConfig": "vllm.transformers_utils.configs.dotsocr",
|
||||
"EAGLEConfig": "vllm.transformers_utils.configs.eagle",
|
||||
"FireRedLIDConfig": "vllm.transformers_utils.configs.fireredlid",
|
||||
"FlexOlmoConfig": "vllm.transformers_utils.configs.flex_olmo",
|
||||
"FunAudioChatConfig": "vllm.transformers_utils.configs.funaudiochat",
|
||||
"FunAudioChatAudioEncoderConfig": "vllm.transformers_utils.configs.funaudiochat",
|
||||
@@ -47,7 +44,6 @@ _CLASS_TO_MODULE: dict[str, str] = {
|
||||
"HunYuanVLConfig": "vllm.transformers_utils.configs.hunyuan_vl",
|
||||
"HunYuanVLTextConfig": "vllm.transformers_utils.configs.hunyuan_vl",
|
||||
"HunYuanVLVisionConfig": "vllm.transformers_utils.configs.hunyuan_vl",
|
||||
"HCXVisionConfig": "vllm.transformers_utils.configs.hyperclovax",
|
||||
"HYV3Config": "vllm.transformers_utils.configs.hy_v3",
|
||||
"HyperCLOVAXConfig": "vllm.transformers_utils.configs.hyperclovax",
|
||||
"IsaacConfig": "vllm.transformers_utils.configs.isaac",
|
||||
@@ -112,10 +108,8 @@ _CLASS_TO_MODULE: dict[str, str] = {
|
||||
|
||||
__all__ = [
|
||||
"AfmoeConfig",
|
||||
"ArcticConfig",
|
||||
"AXK1Config",
|
||||
"BagelConfig",
|
||||
"CheersConfig",
|
||||
"ChatGLMConfig",
|
||||
"ColModernVBertConfig",
|
||||
"ColPaliConfig",
|
||||
@@ -137,14 +131,12 @@ __all__ = [
|
||||
"DotsOCRConfig",
|
||||
"EAGLEConfig",
|
||||
"FlexOlmoConfig",
|
||||
"FireRedLIDConfig",
|
||||
"FunAudioChatConfig",
|
||||
"FunAudioChatAudioEncoderConfig",
|
||||
"Granite4VisionConfig",
|
||||
"HunYuanVLConfig",
|
||||
"HunYuanVLTextConfig",
|
||||
"HunYuanVLVisionConfig",
|
||||
"HCXVisionConfig",
|
||||
"HYV3Config",
|
||||
"HyperCLOVAXConfig",
|
||||
"IsaacConfig",
|
||||
|
||||
@@ -1,216 +0,0 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
|
||||
|
||||
# ruff: noqa: E501
|
||||
# coding=utf-8
|
||||
# Copied from
|
||||
# https://huggingface.co/Snowflake/snowflake-arctic-instruct/blob/main/configuration_arctic.py
|
||||
"""Arctic model configuration"""
|
||||
|
||||
from dataclasses import asdict, dataclass
|
||||
from typing import Any
|
||||
|
||||
from transformers.configuration_utils import PretrainedConfig
|
||||
from transformers.utils import logging
|
||||
|
||||
logger = logging.get_logger(__name__)
|
||||
|
||||
ARCTIC_PRETRAINED_CONFIG_ARCHIVE_MAP = {
|
||||
"arctic": "https://huggingface.co/Snowflake/snowflake-arctic-instruct/tree/main/config.json",
|
||||
}
|
||||
|
||||
|
||||
@dataclass
|
||||
class ArcticLoRAConfig:
|
||||
lora_r: int = 64
|
||||
lora_alpha: float = 16
|
||||
shard_base_weights: bool = False
|
||||
|
||||
|
||||
@dataclass
|
||||
class ArcticQuantizationConfig:
|
||||
q_bits: int = 8
|
||||
rounding: str = "nearest"
|
||||
mantissa_bits: int = 3
|
||||
group_size: int = 128
|
||||
|
||||
|
||||
class ArcticConfig(PretrainedConfig):
|
||||
r"""
|
||||
This is the configuration class to store the configuration of a [`ArcticModel`]. It is used to instantiate an
|
||||
Arctic model according to the specified arguments, defining the model architecture. Instantiating a configuration
|
||||
with the defaults will yield a similar configuration to that of the #TODO(rsamdani): add what model has the default config..
|
||||
|
||||
|
||||
Configuration objects inherit from [`PretrainedConfig`] and can be used to control the model outputs. Read the
|
||||
documentation from [`PretrainedConfig`] for more information.
|
||||
|
||||
|
||||
Args:
|
||||
vocab_size (`int`, *optional*, defaults to 32000):
|
||||
Vocabulary size of the Arctic model. Defines the number of different tokens that can be represented by the
|
||||
`inputs_ids` passed when calling [`ArcticModel`]
|
||||
hidden_size (`int`, *optional*, defaults to 4096):
|
||||
Dimension of the hidden representations.
|
||||
intermediate_size (`int`, *optional*, defaults to 14336):
|
||||
Dimension of the MLP representations.
|
||||
num_hidden_layers (`int`, *optional*, defaults to 32):
|
||||
Number of hidden layers in the Transformer encoder.
|
||||
num_attention_heads (`int`, *optional*, defaults to 32):
|
||||
Number of attention heads for each attention layer in the Transformer encoder.
|
||||
num_key_value_heads (`int`, *optional*, defaults to 8):
|
||||
This is the number of key_value heads that should be used to implement Grouped Query Attention. If
|
||||
`num_key_value_heads=num_attention_heads`, the model will use Multi Head Attention (MHA), if
|
||||
`num_key_value_heads=1 the model will use Multi Query Attention (MQA) otherwise GQA is used. When
|
||||
converting a multi-head checkpoint to a GQA checkpoint, each group key and value head should be constructed
|
||||
by meanpooling all the original heads within that group. For more details checkout [this
|
||||
paper](https://arxiv.org/pdf/2305.13245.pdf). If it is not specified, will default to `8`.
|
||||
hidden_act (`str` or `function`, *optional*, defaults to `"silu"`):
|
||||
The non-linear activation function (function or string) in the decoder.
|
||||
max_position_embeddings (`int`, *optional*, defaults to `4096*32`):
|
||||
The maximum sequence length that this model might ever be used with. Arctic's sliding window attention
|
||||
allows sequence of up to 4096*32 tokens.
|
||||
initializer_range (`float`, *optional*, defaults to 0.02):
|
||||
The standard deviation of the truncated_normal_initializer for initializing all weight matrices.
|
||||
rms_norm_eps (`float`, *optional*, defaults to 1e-05):
|
||||
The epsilon used by the rms normalization layers.
|
||||
use_cache (`bool`, *optional*, defaults to `True`):
|
||||
Whether or not the model should return the last key/values attentions (not used by all models). Only
|
||||
relevant if `config.is_decoder=True`.
|
||||
pad_token_id (`int`, *optional*):
|
||||
The id of the padding token.
|
||||
bos_token_id (`int`, *optional*, defaults to 1):
|
||||
The id of the "beginning-of-sequence" token.
|
||||
eos_token_id (`int`, *optional*, defaults to 2):
|
||||
The id of the "end-of-sequence" token.
|
||||
tie_word_embeddings (`bool`, *optional*, defaults to `False`):
|
||||
Whether the model's input and output word embeddings should be tied.
|
||||
rope_parameters (`dict`, *optional*):
|
||||
Dictionary containing the scaling configuration for the RoPE embeddings. NOTE: if you apply new rope type
|
||||
and you expect the model to work on longer `max_position_embeddings`, we recommend you to update this value
|
||||
accordingly.
|
||||
Expected contents:
|
||||
`rope_theta` (`float`): The base period of the RoPE embeddings.
|
||||
`rope_type` (`str`):
|
||||
The sub-variant of RoPE to use. Can be one of ['default', 'linear', 'dynamic', 'yarn', 'longrope',
|
||||
'llama3'], with 'default' being the original RoPE implementation.
|
||||
sliding_window (`int`, *optional*):
|
||||
Sliding window attention window size. If not specified, will default to `4096`.
|
||||
attention_dropout (`float`, *optional*, defaults to 0.0):
|
||||
The dropout ratio for the attention probabilities.
|
||||
num_experts_per_tok (`int`, *optional*, defaults to 2):
|
||||
The number of experts to root per-token, can be also interpreted as the `top-p` routing
|
||||
parameter
|
||||
num_local_experts (`int`, *optional*, defaults to 8):
|
||||
Number of experts per Sparse MLP layer.
|
||||
router_aux_loss_coef (`float`, *optional*, defaults to 0.001):
|
||||
The aux loss factor for the total loss.
|
||||
|
||||
```python
|
||||
>>> from transformers import ArcticModel, ArcticConfig
|
||||
|
||||
>>> # Initializing a Arctic 7B style configuration TODO(rsamdani): verify which model does the default configuration correspond to.
|
||||
>>> configuration = ArcticConfig()
|
||||
|
||||
>>> # Initializing a model from the Arctic 7B style configuration
|
||||
>>> model = ArcticModel(configuration)
|
||||
|
||||
>>> # Accessing the model configuration
|
||||
>>> configuration = model.config
|
||||
```"""
|
||||
|
||||
model_type = "arctic"
|
||||
keys_to_ignore_at_inference = ["past_key_values"]
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
vocab_size=32000,
|
||||
hidden_size=4096,
|
||||
intermediate_size=14336,
|
||||
num_hidden_layers=32,
|
||||
num_attention_heads=32,
|
||||
num_key_value_heads=None,
|
||||
hidden_act="silu",
|
||||
max_position_embeddings=4096,
|
||||
initializer_range=0.02,
|
||||
rms_norm_eps=1e-5,
|
||||
use_cache=True,
|
||||
pad_token_id=None,
|
||||
bos_token_id=1,
|
||||
eos_token_id=2,
|
||||
tie_word_embeddings=False,
|
||||
rope_parameters: dict[str, Any] | None = None,
|
||||
sliding_window=None,
|
||||
attention_dropout=0.0,
|
||||
num_experts_per_tok=1,
|
||||
num_local_experts=8,
|
||||
router_aux_loss_coef=0.001,
|
||||
moe_layer_frequency=2,
|
||||
parallel_attn_mlp_res=False,
|
||||
moe_train_capacity_factor=1,
|
||||
moe_eval_capacity_factor=1,
|
||||
enable_expert_tensor_parallelism=False,
|
||||
moe_min_capacity=0,
|
||||
moe_token_dropping=True,
|
||||
quantization=None,
|
||||
**kwargs,
|
||||
):
|
||||
self.vocab_size = vocab_size
|
||||
self.max_position_embeddings = max_position_embeddings
|
||||
self.hidden_size = hidden_size
|
||||
self.intermediate_size = intermediate_size
|
||||
self.num_hidden_layers = num_hidden_layers
|
||||
self.num_attention_heads = num_attention_heads
|
||||
self.sliding_window = sliding_window
|
||||
|
||||
# for backward compatibility
|
||||
if num_key_value_heads is None:
|
||||
num_key_value_heads = num_attention_heads
|
||||
|
||||
self.num_key_value_heads = num_key_value_heads
|
||||
self.hidden_act = hidden_act
|
||||
self.initializer_range = initializer_range
|
||||
self.rms_norm_eps = rms_norm_eps
|
||||
self.use_cache = use_cache
|
||||
rope_theta = kwargs.pop("rope_theta", 1e6)
|
||||
if rope_parameters is None:
|
||||
rope_parameters = {"rope_type": "default", "rope_theta": rope_theta}
|
||||
self.rope_parameters = rope_parameters
|
||||
self.attention_dropout = attention_dropout
|
||||
|
||||
self.num_experts_per_tok = num_experts_per_tok
|
||||
self.num_local_experts = num_local_experts
|
||||
self.router_aux_loss_coef = router_aux_loss_coef
|
||||
self.moe_layer_frequency = moe_layer_frequency
|
||||
self.moe_train_capacity_factor = moe_train_capacity_factor
|
||||
self.moe_eval_capacity_factor = moe_eval_capacity_factor
|
||||
self.enable_expert_tensor_parallelism = enable_expert_tensor_parallelism
|
||||
self.moe_min_capacity = moe_min_capacity
|
||||
self.moe_token_dropping = moe_token_dropping
|
||||
self.parallel_attn_mlp_res = parallel_attn_mlp_res
|
||||
if isinstance(quantization, dict):
|
||||
self.quantization = ArcticQuantizationConfig(**quantization)
|
||||
else:
|
||||
self.quantization = quantization
|
||||
|
||||
super().__init__(
|
||||
pad_token_id=pad_token_id,
|
||||
bos_token_id=bos_token_id,
|
||||
eos_token_id=eos_token_id,
|
||||
tie_word_embeddings=tie_word_embeddings,
|
||||
**kwargs,
|
||||
)
|
||||
|
||||
@classmethod
|
||||
def from_dict(cls, config_dict: dict[str, Any], **kwargs) -> "ArcticConfig":
|
||||
result = super().from_dict(config_dict, **kwargs)
|
||||
config = result[0] if isinstance(result, tuple) else result
|
||||
if isinstance(config.quantization, dict):
|
||||
config.quantization = ArcticQuantizationConfig(**config.quantization)
|
||||
return result
|
||||
|
||||
def to_dict(self) -> dict[str, Any]:
|
||||
ret = super().to_dict()
|
||||
if isinstance(ret["quantization"], ArcticQuantizationConfig):
|
||||
ret["quantization"] = asdict(ret["quantization"])
|
||||
return ret
|
||||
@@ -1,109 +0,0 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
|
||||
from transformers import PretrainedConfig, SiglipVisionConfig
|
||||
from transformers.modeling_rope_utils import rope_config_validation
|
||||
|
||||
|
||||
class CheersTextConfig(PretrainedConfig):
|
||||
"""Qwen2-based text config with Cheers-specific defaults."""
|
||||
|
||||
model_type = "umm"
|
||||
base_config_key = "text_config"
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
vocab_size=152064,
|
||||
hidden_size=3584,
|
||||
intermediate_size=18944,
|
||||
num_hidden_layers=28,
|
||||
num_attention_heads=28,
|
||||
num_key_value_heads=4,
|
||||
hidden_act="silu",
|
||||
max_position_embeddings=131072,
|
||||
initializer_range=0.02,
|
||||
rms_norm_eps=1e-6,
|
||||
use_cache=True,
|
||||
tie_word_embeddings=False,
|
||||
rope_theta=1000000.0,
|
||||
rope_scaling=None,
|
||||
use_sliding_window=False,
|
||||
sliding_window=131072,
|
||||
max_window_layers=28,
|
||||
layer_types=None,
|
||||
attention_dropout=0.0,
|
||||
**kwargs,
|
||||
):
|
||||
self.vocab_size = vocab_size
|
||||
self.max_position_embeddings = max_position_embeddings
|
||||
self.hidden_size = hidden_size
|
||||
self.intermediate_size = intermediate_size
|
||||
self.num_hidden_layers = num_hidden_layers
|
||||
self.num_attention_heads = num_attention_heads
|
||||
self.use_sliding_window = use_sliding_window
|
||||
self.sliding_window = sliding_window if self.use_sliding_window else None
|
||||
self.max_window_layers = max_window_layers
|
||||
if num_key_value_heads is None:
|
||||
num_key_value_heads = num_attention_heads
|
||||
self.num_key_value_heads = num_key_value_heads
|
||||
self.hidden_act = hidden_act
|
||||
self.initializer_range = initializer_range
|
||||
self.rms_norm_eps = rms_norm_eps
|
||||
self.use_cache = use_cache
|
||||
self.rope_theta = rope_theta
|
||||
self.rope_scaling = rope_scaling
|
||||
self.attention_dropout = attention_dropout
|
||||
if self.rope_scaling is not None and "type" in self.rope_scaling:
|
||||
self.rope_scaling["rope_type"] = self.rope_scaling["type"]
|
||||
rope_config_validation(self)
|
||||
|
||||
self.layer_types = layer_types
|
||||
if self.layer_types is None:
|
||||
self.layer_types = [
|
||||
"sliding_attention"
|
||||
if self.sliding_window is not None and i >= self.max_window_layers
|
||||
else "full_attention"
|
||||
for i in range(self.num_hidden_layers)
|
||||
]
|
||||
|
||||
super().__init__(
|
||||
tie_word_embeddings=tie_word_embeddings,
|
||||
**kwargs,
|
||||
)
|
||||
|
||||
|
||||
class CheersConfig(PretrainedConfig):
|
||||
"""Configuration class for Cheers (UMM) model."""
|
||||
|
||||
model_type = "umm"
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
text_config: dict | CheersTextConfig | None = None,
|
||||
vision_representation_config: dict | SiglipVisionConfig | None = None,
|
||||
vae_encoder_config: dict | None = None,
|
||||
vae_decoder_config: dict | None = None,
|
||||
**kwargs,
|
||||
):
|
||||
super().__init__(**kwargs)
|
||||
|
||||
if isinstance(text_config, dict):
|
||||
self.text_config = CheersTextConfig(**text_config)
|
||||
else:
|
||||
self.text_config = text_config or CheersTextConfig()
|
||||
|
||||
if isinstance(vision_representation_config, dict):
|
||||
self.vision_representation_config = SiglipVisionConfig(
|
||||
**vision_representation_config
|
||||
)
|
||||
else:
|
||||
self.vision_representation_config = (
|
||||
vision_representation_config or SiglipVisionConfig()
|
||||
)
|
||||
|
||||
self.vae_encoder_config = vae_encoder_config or {"resolution": 512}
|
||||
self.vae_decoder_config = vae_decoder_config or {"resolution": 512}
|
||||
|
||||
@property
|
||||
def hidden_size(self) -> int:
|
||||
"""Return the hidden size of the language model."""
|
||||
return self.text_config.hidden_size
|
||||
@@ -1,73 +0,0 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import contextlib
|
||||
|
||||
from transformers import AutoConfig
|
||||
from transformers.configuration_utils import PretrainedConfig
|
||||
|
||||
|
||||
class FireRedLIDConfig(PretrainedConfig):
|
||||
"""Minimal config class for native vLLM FireRedLID support."""
|
||||
|
||||
model_type = "fireredlid"
|
||||
keys_to_ignore_at_inference = ["past_key_values"]
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
vocab_size: int = 120,
|
||||
lid_odim: int = 120,
|
||||
idim: int = 80,
|
||||
d_model: int = 1280,
|
||||
n_head: int = 20,
|
||||
n_layers_enc: int = 16,
|
||||
n_layers_lid_dec: int = 6,
|
||||
kernel_size: int = 33,
|
||||
residual_dropout: float = 0.05,
|
||||
dropout_rate: float = 0.05,
|
||||
pe_maxlen: int = 5000,
|
||||
pad_token_id: int = 2,
|
||||
bos_token_id: int = 3,
|
||||
eos_token_id: int = 4,
|
||||
decoder_start_token_id: int = 3,
|
||||
tie_word_embeddings: bool = True,
|
||||
is_encoder_decoder: bool = True,
|
||||
architectures: list[str] | None = None,
|
||||
**kwargs,
|
||||
):
|
||||
self.vocab_size = vocab_size
|
||||
self.lid_odim = lid_odim
|
||||
self.idim = idim
|
||||
self.d_model = d_model
|
||||
self.hidden_size = d_model
|
||||
self.n_head = n_head
|
||||
self.num_attention_heads = n_head
|
||||
self.n_layers_enc = n_layers_enc
|
||||
self.encoder_layers = n_layers_enc
|
||||
self.n_layers_lid_dec = n_layers_lid_dec
|
||||
self.decoder_layers = n_layers_lid_dec
|
||||
self.num_hidden_layers = n_layers_lid_dec
|
||||
self.kernel_size = kernel_size
|
||||
self.residual_dropout = residual_dropout
|
||||
self.dropout_rate = dropout_rate
|
||||
self.pe_maxlen = pe_maxlen
|
||||
self.tie_word_embeddings = tie_word_embeddings
|
||||
self.is_encoder_decoder = is_encoder_decoder
|
||||
self.architectures = architectures or ["FireRedLIDForConditionalGeneration"]
|
||||
|
||||
super().__init__(
|
||||
pad_token_id=pad_token_id,
|
||||
bos_token_id=bos_token_id,
|
||||
eos_token_id=eos_token_id,
|
||||
decoder_start_token_id=decoder_start_token_id,
|
||||
tie_word_embeddings=tie_word_embeddings,
|
||||
is_encoder_decoder=is_encoder_decoder,
|
||||
architectures=self.architectures,
|
||||
**kwargs,
|
||||
)
|
||||
|
||||
|
||||
with contextlib.suppress(ValueError):
|
||||
AutoConfig.register(FireRedLIDConfig.model_type, FireRedLIDConfig)
|
||||
@@ -17,7 +17,6 @@
|
||||
# limitations under the License.
|
||||
"""HyperCLOVA X model configuration."""
|
||||
|
||||
from transformers import AutoConfig
|
||||
from transformers.configuration_utils import PretrainedConfig
|
||||
|
||||
|
||||
@@ -276,74 +275,3 @@ class HyperCLOVAXConfig(PretrainedConfig):
|
||||
auto_map=auto_map,
|
||||
**kwargs,
|
||||
)
|
||||
|
||||
|
||||
class HCXVisionConfig(PretrainedConfig):
|
||||
"""Vendored HyperCLOVAX Vision config with transformers v5 fix.
|
||||
|
||||
The original remote code config does not handle empty initialization
|
||||
(text_config=None), which breaks transformers v5's @strict validation.
|
||||
|
||||
TODO: Remove this class once HyperCLOVAX is upstreamed to transformers.
|
||||
Tracking PR: https://github.com/huggingface/transformers/pull/44956
|
||||
"""
|
||||
|
||||
model_type = "hyperclovax_vlm"
|
||||
keys_to_ignore_at_inference = ["past_key_values"]
|
||||
|
||||
text_config_attribute_map = {
|
||||
"n_embd": "hidden_size",
|
||||
"n_positions": "max_position_embeddings",
|
||||
"n_head": "num_attention_heads",
|
||||
"n_layer": "num_hidden_layers",
|
||||
}
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
text_config=None,
|
||||
vision_config=None,
|
||||
use_nth_layer=-2,
|
||||
img_start_id=100009,
|
||||
decoder_max_length=4096,
|
||||
anyres=False,
|
||||
unpad=False,
|
||||
max_num_grids=-1,
|
||||
num_queries_vis_abstractor=-1,
|
||||
ignore_index=-100,
|
||||
proj_pos_emb=True,
|
||||
proj_prenorm=False,
|
||||
use_1x1_grid=False,
|
||||
**kwargs,
|
||||
):
|
||||
for key, val in self.text_config_attribute_map.items():
|
||||
if text_config is not None and key in text_config:
|
||||
text_config[val] = text_config.pop(key)
|
||||
|
||||
self.text_config = None
|
||||
if text_config is not None:
|
||||
_text_config = AutoConfig.for_model(text_config["model_type"])
|
||||
self.text_config = _text_config.from_dict(text_config)
|
||||
self.hidden_size = self.text_config.hidden_size
|
||||
|
||||
self.vision_config = None
|
||||
if vision_config is not None:
|
||||
_vision_config = AutoConfig.for_model(vision_config["model_type"])
|
||||
self.vision_config = _vision_config.from_dict(vision_config)
|
||||
|
||||
self.use_nth_layer = use_nth_layer
|
||||
self.decoder_max_length = decoder_max_length
|
||||
self.anyres = anyres
|
||||
self.unpad = unpad
|
||||
self.max_num_grids = max_num_grids
|
||||
self.num_queries_vis_abstractor = num_queries_vis_abstractor
|
||||
self.img_start_id = img_start_id
|
||||
self.ignore_index = ignore_index
|
||||
self.proj_pos_emb = proj_pos_emb
|
||||
self.proj_prenorm = proj_prenorm
|
||||
self.use_1x1_grid = use_1x1_grid
|
||||
super().__init__(**kwargs)
|
||||
|
||||
def get_text_config(self, decoder=False):
|
||||
if self.text_config is not None:
|
||||
return self.text_config
|
||||
return self
|
||||
|
||||
@@ -373,7 +373,6 @@ class ModelArchConfigConvertorBase:
|
||||
"molmo2",
|
||||
"moondream3",
|
||||
"paligemma",
|
||||
"umm",
|
||||
)
|
||||
if not hasattr(self.hf_config, "model_type"):
|
||||
return False
|
||||
@@ -545,13 +544,6 @@ class FalconModelArchConfigConvertor(ModelArchConfigConvertorBase):
|
||||
return super().get_total_num_kv_heads()
|
||||
|
||||
|
||||
class MPTModelArchConfigConvertor(ModelArchConfigConvertorBase):
|
||||
def get_total_num_kv_heads(self) -> int:
|
||||
if "kv_n_heads" in self.hf_text_config.attn_config:
|
||||
return self.hf_text_config.attn_config["kv_n_heads"]
|
||||
return self.hf_text_config.num_attention_heads
|
||||
|
||||
|
||||
class DbrxModelArchConfigConvertor(ModelArchConfigConvertorBase):
|
||||
def get_total_num_kv_heads(self) -> int:
|
||||
return getattr(
|
||||
@@ -792,7 +784,6 @@ MODEL_ARCH_CONFIG_CONVERTORS = {
|
||||
"mimo_v2_mtp": MimoV2MTPModelArchConfigConvertor,
|
||||
"mimo_v2_omni_mtp": MimoV2MTPModelArchConfigConvertor,
|
||||
"moss_audio": MossAudioModelArchConfigConvertor,
|
||||
"mpt": MPTModelArchConfigConvertor,
|
||||
"nemotron-nas": NemotronNasModelArchConfigConvertor,
|
||||
"bailing_hybrid_v3_mtp": BailingHybridV3MTPModelArchConfigConvertor,
|
||||
"pangu_ultra_moe_mtp": PanguUltraMoeMTPModelArchConfigConvertor,
|
||||
|
||||
@@ -37,9 +37,7 @@ def _transformers_v4_compatibility_import():
|
||||
subset of `ProcessorChatTemplateKwargs` as defined in Transformers v4.
|
||||
In Transformers v5 these were merged into `ProcessorChatTemplateKwargs` and
|
||||
`ChatTemplateLoadKwargs` was removed. For backward compatibility, we add an alias
|
||||
for `ChatTemplateLoadKwargs` if it doesn't exist.
|
||||
|
||||
This can be removed if `HCXVisionForCausalLM` is upstreamed to Transformers."""
|
||||
for `ChatTemplateLoadKwargs` if it doesn't exist."""
|
||||
old_import = getattr(processing_utils, "ChatTemplateLoadKwargs", None)
|
||||
new_import = getattr(processing_utils, "ProcessorChatTemplateKwargs", None)
|
||||
if old_import is None and new_import is not None:
|
||||
|
||||
@@ -12,12 +12,10 @@ import importlib
|
||||
|
||||
__all__ = [
|
||||
"BagelProcessor",
|
||||
"CheersProcessor",
|
||||
"CohereASRProcessor",
|
||||
"Cosmos3EdgeProcessor",
|
||||
"DeepseekVLV2Processor",
|
||||
"FireRedASR2Processor",
|
||||
"FireRedLIDProcessor",
|
||||
"FunASRProcessor",
|
||||
"GLM4VProcessor",
|
||||
"Granite4VisionProcessor",
|
||||
@@ -52,12 +50,10 @@ __all__ = [
|
||||
|
||||
_CLASS_TO_MODULE: dict[str, str] = {
|
||||
"BagelProcessor": "vllm.transformers_utils.processors.bagel",
|
||||
"CheersProcessor": "vllm.transformers_utils.processors.cheers",
|
||||
"CohereASRProcessor": "vllm.transformers_utils.processors.cohere_asr",
|
||||
"Cosmos3EdgeProcessor": "vllm.transformers_utils.processors.cosmos3_edge",
|
||||
"DeepseekVLV2Processor": "vllm.transformers_utils.processors.deepseek_vl2",
|
||||
"FireRedASR2Processor": "vllm.transformers_utils.processors.fireredasr2",
|
||||
"FireRedLIDProcessor": "vllm.transformers_utils.processors.fireredlid",
|
||||
"FunASRProcessor": "vllm.transformers_utils.processors.funasr",
|
||||
"GLM4VProcessor": "vllm.transformers_utils.processors.glm4v",
|
||||
"Granite4VisionProcessor": "vllm.transformers_utils.processors.granite4_vision",
|
||||
|
||||
@@ -1,89 +0,0 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
|
||||
"""Cheers (UMM) processor for image and text inputs."""
|
||||
|
||||
from transformers import AutoProcessor
|
||||
from transformers.feature_extraction_utils import BatchFeature
|
||||
from transformers.image_utils import ImageInput
|
||||
from transformers.processing_utils import ProcessingKwargs, ProcessorMixin, Unpack
|
||||
from transformers.tokenization_utils_base import PreTokenizedInput, TextInput
|
||||
|
||||
|
||||
class CheersProcessorKwargs(ProcessingKwargs, total=False): # type: ignore[call-arg]
|
||||
_defaults = {
|
||||
"images_kwargs": {
|
||||
"return_tensors": "pt",
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
class CheersProcessor(ProcessorMixin):
|
||||
"""
|
||||
Constructs a Cheers processor which wraps a
|
||||
SigLIP image processor and a Qwen2 tokenizer.
|
||||
"""
|
||||
|
||||
attributes = ["image_processor", "tokenizer"]
|
||||
image_processor_class = "AutoImageProcessor"
|
||||
tokenizer_class = "AutoTokenizer"
|
||||
|
||||
def __call__(
|
||||
self,
|
||||
text: TextInput
|
||||
| PreTokenizedInput
|
||||
| list[TextInput]
|
||||
| list[PreTokenizedInput] = None,
|
||||
images: ImageInput = None,
|
||||
**kwargs: Unpack[CheersProcessorKwargs],
|
||||
):
|
||||
output_kwargs = self._merge_kwargs(
|
||||
CheersProcessorKwargs,
|
||||
tokenizer_init_kwargs=self.tokenizer.init_kwargs,
|
||||
**kwargs,
|
||||
)
|
||||
|
||||
if images is not None:
|
||||
import torch
|
||||
|
||||
if isinstance(images, (list, tuple)):
|
||||
all_pv = []
|
||||
all_ghw = []
|
||||
for img in images:
|
||||
result = self.image_processor(img, **output_kwargs["images_kwargs"])
|
||||
all_pv.append(result["pixel_values"])
|
||||
if "grid_hws" in result:
|
||||
all_ghw.append(result["grid_hws"])
|
||||
pixel_values = {
|
||||
"pixel_values": torch.cat(all_pv, dim=0),
|
||||
}
|
||||
if all_ghw:
|
||||
pixel_values["grid_hws"] = torch.cat(all_ghw, dim=0)
|
||||
else:
|
||||
pixel_values = self.image_processor(
|
||||
images, **output_kwargs["images_kwargs"]
|
||||
)
|
||||
else:
|
||||
pixel_values = {}
|
||||
|
||||
text_inputs = (
|
||||
self.tokenizer(text, **output_kwargs["text_kwargs"])
|
||||
if text is not None
|
||||
else {}
|
||||
)
|
||||
|
||||
return BatchFeature(data={**pixel_values, **text_inputs})
|
||||
|
||||
def batch_decode(self, *args, **kwargs):
|
||||
return self.tokenizer.batch_decode(*args, **kwargs)
|
||||
|
||||
def decode(self, *args, **kwargs):
|
||||
return self.tokenizer.decode(*args, **kwargs)
|
||||
|
||||
@property
|
||||
def model_input_names(self):
|
||||
tokenizer_input_names = self.tokenizer.model_input_names
|
||||
image_processor_input_names = self.image_processor.model_input_names
|
||||
return list(dict.fromkeys(tokenizer_input_names + image_processor_input_names))
|
||||
|
||||
|
||||
AutoProcessor.register("CheersProcessor", CheersProcessor)
|
||||
@@ -1,276 +0,0 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
|
||||
"""
|
||||
FireRedLID feature extractor and processor.
|
||||
|
||||
The FeatureExtractor handles:
|
||||
- Raw waveform → 80-dim log-mel filterbank (via kaldi_native_fbank)
|
||||
- CMVN normalization (means / inverse_std_variences from preprocessor_config)
|
||||
- Padding + length tracking
|
||||
|
||||
The Processor wraps the FeatureExtractor and a tokenizer.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
import numpy as np
|
||||
import torch
|
||||
import torch.nn.functional as F
|
||||
from transformers import (
|
||||
AutoFeatureExtractor,
|
||||
BatchFeature,
|
||||
)
|
||||
from transformers.feature_extraction_sequence_utils import SequenceFeatureExtractor
|
||||
from transformers.processing_utils import ProcessorMixin
|
||||
from transformers.utils import TensorType
|
||||
|
||||
from vllm.logger import init_logger
|
||||
from vllm.utils.import_utils import LazyLoader
|
||||
|
||||
if TYPE_CHECKING:
|
||||
import kaldi_native_fbank as knf
|
||||
else:
|
||||
knf = LazyLoader("knf", globals(), "kaldi_native_fbank")
|
||||
|
||||
|
||||
logger = init_logger(__name__)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Helpers (shared with FireRedASR2 processor)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class CMVN:
|
||||
def __init__(self, dim, means, inverse_std_variences):
|
||||
self.dim = dim
|
||||
self.means = np.array(means)
|
||||
self.inverse_std_variences = np.array(inverse_std_variences)
|
||||
|
||||
def __call__(self, x):
|
||||
assert x.shape[-1] == self.dim, "CMVN dim mismatch"
|
||||
out = x - self.means
|
||||
out = out * self.inverse_std_variences
|
||||
return out
|
||||
|
||||
|
||||
class KaldifeatFbank:
|
||||
def __init__(
|
||||
self,
|
||||
num_mel_bins: int = 80,
|
||||
frame_length: int = 25,
|
||||
frame_shift: int = 10,
|
||||
dither: float = 0.0,
|
||||
):
|
||||
self.dither = dither
|
||||
opts = knf.FbankOptions()
|
||||
opts.frame_opts.dither = dither
|
||||
opts.mel_opts.num_bins = num_mel_bins
|
||||
opts.frame_opts.snip_edges = True
|
||||
opts.mel_opts.debug_mel = False
|
||||
self.opts = opts
|
||||
|
||||
def __call__(self, sample_rate, wav_np, is_train=False):
|
||||
dither = self.dither if is_train else 0.0
|
||||
self.opts.frame_opts.dither = dither
|
||||
fbank = knf.OnlineFbank(self.opts)
|
||||
fbank.accept_waveform(sample_rate, wav_np.tolist())
|
||||
feat = []
|
||||
for i in range(fbank.num_frames_ready):
|
||||
feat.append(fbank.get_frame(i))
|
||||
if len(feat) == 0:
|
||||
return np.zeros((0, self.opts.mel_opts.num_bins))
|
||||
return np.vstack(feat)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Feature Extractor
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class FireRedLIDFeatureExtractor(SequenceFeatureExtractor):
|
||||
"""
|
||||
Extracts 80-dim log-mel filterbank features from raw waveforms,
|
||||
applies CMVN, and returns padded feature tensors with lengths.
|
||||
|
||||
Also computes ``fake_token_lengths`` — the actual encoder output
|
||||
length for each audio — so that vLLM can allocate the correct
|
||||
number of cross-attention KV cache slots.
|
||||
"""
|
||||
|
||||
model_input_names = ["input_features"]
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
feature_size=80,
|
||||
sampling_rate=16000,
|
||||
chunk_length=30,
|
||||
padding_value=0.0,
|
||||
return_attention_mask=False,
|
||||
dim=80,
|
||||
means=None,
|
||||
inverse_std_variences=None,
|
||||
num_mel_bins=80,
|
||||
frame_length=25,
|
||||
frame_shift=10,
|
||||
dither=0.0,
|
||||
left_context=3,
|
||||
right_context=3,
|
||||
**kwargs,
|
||||
):
|
||||
super().__init__(
|
||||
feature_size=feature_size,
|
||||
sampling_rate=sampling_rate,
|
||||
padding_value=padding_value,
|
||||
return_attention_mask=return_attention_mask,
|
||||
**kwargs,
|
||||
)
|
||||
self.chunk_length = chunk_length
|
||||
self.dim = dim
|
||||
self.means = means
|
||||
self.inverse_std_variences = inverse_std_variences
|
||||
self.num_mel_bins = num_mel_bins
|
||||
self.frame_length = frame_length
|
||||
self.frame_shift = frame_shift
|
||||
self.dither = dither
|
||||
self.sampling_rate = sampling_rate
|
||||
self.context = left_context + 1 + right_context
|
||||
|
||||
def __call__(
|
||||
self,
|
||||
raw_speech: np.ndarray | list[float] | list[np.ndarray] | list[list[float]],
|
||||
truncation: bool = True,
|
||||
pad_to_multiple_of: int | None = None,
|
||||
return_tensors: str | TensorType | None = None,
|
||||
return_attention_mask: bool | None = None,
|
||||
padding: str | None = "max_length",
|
||||
max_length: int | None = None,
|
||||
sampling_rate: int | None = None,
|
||||
do_normalize: bool | None = None,
|
||||
**kwargs,
|
||||
) -> BatchFeature:
|
||||
if sampling_rate is not None and sampling_rate != self.sampling_rate:
|
||||
raise ValueError(
|
||||
f"FireRedLIDFeatureExtractor expects sampling_rate="
|
||||
f"{self.sampling_rate}, got {sampling_rate}."
|
||||
)
|
||||
|
||||
# Initialize helpers
|
||||
cmvn = CMVN(self.dim, self.means, self.inverse_std_variences)
|
||||
fbank = KaldifeatFbank(
|
||||
num_mel_bins=self.num_mel_bins,
|
||||
frame_length=self.frame_length,
|
||||
frame_shift=self.frame_shift,
|
||||
dither=self.dither,
|
||||
)
|
||||
|
||||
def padding_position_is_0(padded_input, input_lengths):
|
||||
N, T = padded_input.size()[:2]
|
||||
mask = torch.ones((N, T)).to(padded_input.device)
|
||||
for i in range(N):
|
||||
mask[i, input_lengths[i] :] = 0
|
||||
mask = mask.unsqueeze(dim=1)
|
||||
return mask.to(torch.uint8)
|
||||
|
||||
feats = []
|
||||
speech_lengths = []
|
||||
fake_token_lengths = []
|
||||
|
||||
for speech in raw_speech:
|
||||
# vLLM loads audio via librosa (float32 in [-1,1]),
|
||||
# but kaldi_native_fbank expects int16-scale values.
|
||||
speech_scaled = speech * 32768
|
||||
feat = fbank(self.sampling_rate, speech_scaled)
|
||||
feat = cmvn(feat)
|
||||
feat = torch.from_numpy(feat).float()
|
||||
length = feat.size(0)
|
||||
feats.append(feat)
|
||||
speech_lengths.append(length)
|
||||
|
||||
# Compute the actual Conv2dSubsampling output length.
|
||||
# This mirrors the mask logic in Conv2dSubsampling.forward:
|
||||
# pad context frames, then mask[:, :, :-2:2][:, :, :-2:2].sum()
|
||||
padded_input = F.pad(feat, (0, 0, 0, self.context - 1), "constant", 0.0)
|
||||
src_mask = padding_position_is_0(
|
||||
padded_input[None, :, :],
|
||||
torch.tensor([length], dtype=torch.int32),
|
||||
)
|
||||
mask = src_mask[:, :, :-2:2][:, :, :-2:2]
|
||||
enc_len = mask[:, -1, :].sum(dim=-1)
|
||||
fake_token_len = torch.clamp(enc_len, min=1)
|
||||
fake_token_lengths.append(fake_token_len)
|
||||
|
||||
if len(feats) == 0:
|
||||
return BatchFeature()
|
||||
|
||||
# Pad to uniform length
|
||||
max_feat_len = max(f.size(0) for f in feats)
|
||||
padded = feats[0].new_zeros(len(feats), max_feat_len, feats[0].size(1))
|
||||
for i, feat in enumerate(feats):
|
||||
padded[i, : feat.size(0)] = feat
|
||||
|
||||
result = BatchFeature({"input_features": padded})
|
||||
|
||||
if return_tensors is not None:
|
||||
result = result.convert_to_tensors(return_tensors)
|
||||
|
||||
result["speech_lengths"] = torch.tensor(speech_lengths, dtype=torch.long)
|
||||
result["fake_token_lengths"] = torch.concat(fake_token_lengths)
|
||||
return result
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Processor
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class FireRedLIDProcessor(ProcessorMixin):
|
||||
"""
|
||||
Wraps FireRedLIDFeatureExtractor + a tokenizer.
|
||||
"""
|
||||
|
||||
feature_extractor_class = "FireRedLIDFeatureExtractor"
|
||||
tokenizer_class = ("PythonBackend", "TokenizersBackend")
|
||||
|
||||
def __init__(self, feature_extractor, tokenizer):
|
||||
super().__init__(feature_extractor, tokenizer)
|
||||
self.current_processor = self.feature_extractor
|
||||
self._in_target_context_manager = False
|
||||
|
||||
def __call__(self, *args, **kwargs):
|
||||
if self._in_target_context_manager:
|
||||
return self.current_processor(*args, **kwargs)
|
||||
|
||||
audio = kwargs.pop("audio", None)
|
||||
sampling_rate = kwargs.pop("sampling_rate", None)
|
||||
text = kwargs.pop("text", None)
|
||||
if len(args) > 0:
|
||||
audio = args[0]
|
||||
args = args[1:]
|
||||
|
||||
if audio is not None:
|
||||
inputs = self.feature_extractor(
|
||||
audio, *args, sampling_rate=sampling_rate, **kwargs
|
||||
)
|
||||
else:
|
||||
inputs = BatchFeature()
|
||||
|
||||
if text is not None:
|
||||
if isinstance(text, str):
|
||||
text = [text]
|
||||
encodings = self.tokenizer(text, **kwargs)
|
||||
if audio is not None:
|
||||
inputs["labels"] = encodings["input_ids"]
|
||||
else:
|
||||
return encodings
|
||||
|
||||
return inputs
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Registration
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
AutoFeatureExtractor.register("FireRedLIDFeatureExtractor", FireRedLIDFeatureExtractor)
|
||||
@@ -43,7 +43,7 @@ class EncoderDecoderAttnMetadata(ModelSpecificAttnMetadata):
|
||||
|
||||
class EncoderDecoderModelState(ModelState):
|
||||
"""ModelState for cross-attention encoder-decoder models
|
||||
(Whisper, CohereASR, NemotronParse, FireRedLID, ...)
|
||||
(Whisper, CohereASR, NemotronParse, ...)
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
|
||||
Reference in New Issue
Block a user