[LoRA] Remove tensorizer (#60024)
Signed-off-by: Jee Jee Li <jeejeelee@inferact.ai>
This commit is contained in:
@@ -32,25 +32,15 @@ python examples/features/tensorize_vllm_model.py \
|
||||
--suffix v1
|
||||
```
|
||||
|
||||
This saves the model tensors at `s3://my-bucket/vllm/facebook/opt-125m/v1`. If you intend on applying a LoRA adapter to your tensorized model, you can pass the HF id of the LoRA adapter in the above command, and the artifacts will be saved there too:
|
||||
|
||||
```bash
|
||||
python examples/features/tensorize_vllm_model.py \
|
||||
--model facebook/opt-125m \
|
||||
--lora-path <lora_id> \
|
||||
serialize \
|
||||
--serialized-directory s3://my-bucket \
|
||||
--suffix v1
|
||||
```
|
||||
This saves the model tensors at `s3://my-bucket/vllm/facebook/opt-125m/v1`.
|
||||
|
||||
## Serving the model using Tensorizer
|
||||
|
||||
Once the model is serialized where you want it, you can load the model using `vllm serve` or the `LLM` entrypoint. You can pass the directory where you saved the model to the `model` argument for `LLM()` and `vllm serve`. For example, to serve the tensorized model saved previously with the LoRA adapter, you'd do:
|
||||
Once the model is serialized where you want it, you can load the model using `vllm serve` or the `LLM` entrypoint. You can pass the directory where you saved the model to the `model` argument for `LLM()` and `vllm serve`. For example, to serve the tensorized model saved previously, you'd do:
|
||||
|
||||
```bash
|
||||
vllm serve s3://my-bucket/vllm/facebook/opt-125m/v1 \
|
||||
--load-format tensorizer \
|
||||
--enable-lora
|
||||
--load-format tensorizer
|
||||
```
|
||||
|
||||
Or, with `LLM()`:
|
||||
@@ -60,7 +50,6 @@ from vllm import LLM
|
||||
llm = LLM(
|
||||
"s3://my-bucket/vllm/facebook/opt-125m/v1",
|
||||
load_format="tensorizer",
|
||||
enable_lora=True,
|
||||
)
|
||||
```
|
||||
|
||||
@@ -73,7 +62,6 @@ As an example, CPU concurrency can be limited when serializing with `tensorizer`
|
||||
```bash
|
||||
python examples/features/tensorize_vllm_model.py \
|
||||
--model facebook/opt-125m \
|
||||
--lora-path <lora_id> \
|
||||
serialize \
|
||||
--serialized-directory s3://my-bucket \
|
||||
--serialization-kwargs '{"limit_cpu_concurrency": 2}' \
|
||||
@@ -85,7 +73,6 @@ As an example when customizing the loading process via `TensorDeserializer`, you
|
||||
```bash
|
||||
vllm serve s3://my-bucket/vllm/facebook/opt-125m/v1 \
|
||||
--load-format tensorizer \
|
||||
--enable-lora \
|
||||
--model-loader-extra-config '{"deserialization_kwargs": {"num_readers": 2}}'
|
||||
```
|
||||
|
||||
@@ -96,7 +83,6 @@ from vllm import LLM
|
||||
llm = LLM(
|
||||
"s3://my-bucket/vllm/facebook/opt-125m/v1",
|
||||
load_format="tensorizer",
|
||||
enable_lora=True,
|
||||
model_loader_extra_config={"deserialization_kwargs": {"num_readers": 2}},
|
||||
)
|
||||
```
|
||||
|
||||
Reference in New Issue
Block a user