[ModelRunnerV2] Support prompt embeds (#42963)

Signed-off-by: gcanlin <canlinguosdu@gmail.com>
Signed-off-by: Canlin Guo <961750412@qq.com>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
This commit is contained in:
Canlin Guo
2026-08-17 21:58:50 +00:00
committed by GitHub
co-authored by Nick Hill
parent e68fb75b25
commit 455edc022b
17 changed files with 551 additions and 54 deletions
@@ -273,6 +273,9 @@ class LongcatFlashNgramForCausalLM(nn.Module, SupportsLoRA, SupportsPP):
class LongcatNgramModelState(DefaultModelState):
# prepare_inputs builds its own inputs_embeds from n-gram token embeddings.
supports_prompt_embeds = False
"""Per-request n-gram token history for LongCat-Flash-Lite.
Maintains a small CPU-side per-slot context (last ``n-1`` processed tokens)