[ModelRunnerV2] Support prompt embeds (#42963)
Signed-off-by: gcanlin <canlinguosdu@gmail.com> Signed-off-by: Canlin Guo <961750412@qq.com> Signed-off-by: Nick Hill <nickhill123@gmail.com> Co-authored-by: Nick Hill <nickhill123@gmail.com>
This commit is contained in:
@@ -273,6 +273,9 @@ class LongcatFlashNgramForCausalLM(nn.Module, SupportsLoRA, SupportsPP):
|
||||
|
||||
|
||||
class LongcatNgramModelState(DefaultModelState):
|
||||
# prepare_inputs builds its own inputs_embeds from n-gram token embeddings.
|
||||
supports_prompt_embeds = False
|
||||
|
||||
"""Per-request n-gram token history for LongCat-Flash-Lite.
|
||||
|
||||
Maintains a small CPU-side per-slot context (last ``n-1`` processed tokens)
|
||||
|
||||
Reference in New Issue
Block a user