Merge remote-tracking branch 'upstream/main' into patches
Signed-off-by: MOVZX <movzx@yahoo.com>
This commit is contained in:
@@ -77,6 +77,8 @@ decode = client.chat.completions.create(
|
||||
)
|
||||
```
|
||||
|
||||
If `messages` has non-text content or `echo` is set, the ids are ignored and `messages` is rendered instead, so it must match the prefill request. Otherwise `kv_transfer_params["prompt_token_ids"]` must be a non-empty list of non-negative integers, or the request fails with HTTP 400, as it always does on `/v1/chat/completions/batch`.
|
||||
|
||||
## Development
|
||||
|
||||
We implement disaggregated prefilling by running 2 vLLM instances. One for prefill (we call it prefill instance) and one for decode (we call it decode instance), and then use a connector to transfer the prefill KV caches and results from prefill instance to decode instance.
|
||||
|
||||
@@ -428,9 +428,16 @@ vLLM supports PEFT sequence-classification adapters that save a complete, single
|
||||
|
||||
See [classification_with_lora_offline.py](../../examples/pooling/classify/classification_with_lora_offline.py) for an offline classification example using a LoRA adapter.
|
||||
|
||||
To batch adapters with different `num_labels`, set the maximum number of labels:
|
||||
|
||||
```bash
|
||||
vllm serve model --enable-lora --max-lora-cls-labels 8
|
||||
```
|
||||
|
||||
The equivalent `LLM` argument is `max_lora_cls_labels`. It defaults to the base model's `num_labels`, and each request returns its adapter's number of labels.
|
||||
|
||||
This support has the following limitations:
|
||||
|
||||
- All adapters in one engine must have the same `num_labels` and hidden size as the base classification head. An incompatible adapter is rejected when it is loaded.
|
||||
- A classification head stored as float32 is converted to the runtime head dtype when it is loaded.
|
||||
- Token-classification adapters are not supported by this feature.
|
||||
|
||||
|
||||
@@ -120,7 +120,8 @@ Current push PP + HMA limitations:
|
||||
- Only the prefiller (producer) may be PP-sharded; decode-side PP is not supported.
|
||||
- Hybrid SSM/Mamba layouts are not supported under PP.
|
||||
- HMA requires the same block size on P and D.
|
||||
- Attention-HMA member routing requires decode TP to be no greater than prefill TP.
|
||||
- Non-MLA attention-HMA layer routing requires decode TP to be no greater than prefill TP.
|
||||
- Packed layouts under PP require MLA caches. Local and remote packed block strides may differ; layer pages must have equal sizes. MLA supports decode TP greater than prefill TP because its KV is replicated.
|
||||
|
||||
### Quantized KV cache
|
||||
|
||||
|
||||
Reference in New Issue
Block a user