[LoRA] Support modules_to_save for sequence classification (#53555)
Signed-off-by: linitra24 <Joy25810@foxmail.com> Signed-off-by: linitra24 <renshuang.zhou@daocloud.io> Co-authored-by: Jee Jee Li <pandaleefree@gmail.com>
This commit is contained in:
@@ -422,6 +422,18 @@ vllm serve ibm-granite/granite-speech-3.3-2b \
|
||||
|
||||
Note: Default multimodal LoRAs are currently only available for `.generate` and chat completions.
|
||||
|
||||
## Sequence-Classification LoRA Adapters
|
||||
|
||||
vLLM supports PEFT sequence-classification adapters that save a complete, single-layer linear classification head through `modules_to_save`. The saved module must be named `score` or `classifier`.
|
||||
|
||||
See [classification_with_lora_offline.py](../../examples/pooling/classify/classification_with_lora_offline.py) for an offline classification example using a LoRA adapter.
|
||||
|
||||
This support has the following limitations:
|
||||
|
||||
- All adapters in one engine must have the same `num_labels` and hidden size as the base classification head. An incompatible adapter is rejected when it is loaded.
|
||||
- A classification head stored as float32 is converted to the runtime head dtype when it is loaded.
|
||||
- Token-classification adapters are not supported by this feature.
|
||||
|
||||
## Using Tips
|
||||
|
||||
### Configuring `max_lora_rank`
|
||||
|
||||
Reference in New Issue
Block a user