[Bugfix] Preserve sampling masks in DELTA and TITO streaming outputs (#55935)

Signed-off-by: aoshen02 <aoshen@inferact.ai>
Signed-off-by: fh3 <a@b>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: fh3 <a@b>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
aoshen02
2026-10-01 14:43:38 +08:00
committed by GitHub
co-authored by Codex fh3 Claude Fable 5.1
parent c9578f10ab
commit 4a9d6f9572
8 changed files with 135 additions and 6 deletions
+2 -2
View File
@@ -53,6 +53,8 @@ The mask is also available via the `/inference/v1/generate` HTTP endpoint:
}
```
With `stream=true`, each chunk carries the masks of the tokens it emits.
## Requirements
| Requirement | Reason |
@@ -145,5 +147,3 @@ Each position then carries the sampled token's logprob and the top-128
- **Engine-level flag:** `--return-sampling-mask` globally disables the
FlashInfer fused sampler. All requests pay the cost of the PyTorch sampling
path, even if they don't need the mask.
- **No streaming support:** The mask is returned only in the final response,
not in intermediate streaming chunks.