Support Mistral Model Inference with transformers-neuronx (#3153)
This commit is contained in:
Regular → Executable
+6
-4
@@ -14,14 +14,16 @@ sampling_params = SamplingParams(temperature=0.8, top_p=0.95)
|
||||
llm = LLM(
|
||||
model="openlm-research/open_llama_3b",
|
||||
max_num_seqs=8,
|
||||
# The max_model_len and block_size arguments are required to be same as max sequence length,
|
||||
# when targeting neuron device. Currently, this is a known limitation in continuous batching
|
||||
# support in transformers-neuronx.
|
||||
# The max_model_len and block_size arguments are required to be same as
|
||||
# max sequence length when targeting neuron device.
|
||||
# Currently, this is a known limitation in continuous batching support
|
||||
# in transformers-neuronx.
|
||||
# TODO(liangfu): Support paged-attention in transformers-neuronx.
|
||||
max_model_len=128,
|
||||
block_size=128,
|
||||
# The device can be automatically detected when AWS Neuron SDK is installed.
|
||||
# The device argument can be either unspecified for automated detection, or explicitly assigned.
|
||||
# The device argument can be either unspecified for automated detection,
|
||||
# or explicitly assigned.
|
||||
device="neuron")
|
||||
# Generate texts from the prompts. The output is a list of RequestOutput objects
|
||||
# that contain the prompt, generated text, and other information.
|
||||
|
||||
Reference in New Issue
Block a user