diff --git a/docs/serving/online_serving/token_in_token_out.md b/docs/serving/online_serving/token_in_token_out.md index eade608bc4..6279fbd78b 100644 --- a/docs/serving/online_serving/token_in_token_out.md +++ b/docs/serving/online_serving/token_in_token_out.md @@ -1,8 +1,50 @@ # Tokens In <> Tokens Out API -`/inference/v1/generate` takes prompt token IDs and returns generated token IDs. It is the generation step of the `render → generate → derender` pipeline used in disaggregated serving (see the [Renderer APIs](renderer.md) and [Derenderer APIs](derenderer.md)). +`POST /inference/v1/generate` is the token-in/token-out (TITO) HTTP endpoint: it takes prompt token IDs and returns generated token IDs, logprobs keyed by integer token ID, and `finish_reason`/`stop_reason`, with no detokenization. Use it when the caller does its own tokenization and chat templating, such as an RL rollout worker. It is also the generation step of the `render → generate → derender` pipeline used in disaggregated serving (see the [Renderer APIs](renderer.md) and [Derenderer APIs](derenderer.md)). -The endpoint is registered on `vllm serve` with `--enable-scale-out`, and always with `--tokens-only`. `--tokens-only` also skips loading the tokenizer, so the server can't turn token IDs back into text. +## Enabling the endpoint (`--enable-scale-out`, `--tokens-only`) + +- `--enable-scale-out` registers it, with `/render`, `/derender` and `/inference/v1/abort_requests`. +- `--tokens-only` always registers it, skips loading the tokenizer and forces `detokenize=False`. Stop strings (`sampling_params.stop`) are then rejected with a 400; use `stop_token_ids`. +- With neither flag, it returns 403 `Scale-out endpoints are disabled. Restart vllm serve with --enable-scale-out to enable them.` + +```bash +vllm serve Qwen/Qwen3-0.6B --enable-scale-out + +curl -X POST http://localhost:8000/inference/v1/generate \ + -H "Content-Type: application/json" \ + -d '{"token_ids": [151644, 872, 198], "sampling_params": {"max_tokens": 32, "logprobs": 1}}' +``` + +A full client is in [examples/scale_out/token_generation_client.py](../../../examples/scale_out/token_generation_client.py). + +## Request fields + +| Field | Notes | +| --- | --- | +| `token_ids` | Required `list[int]`: the already-templated prompt. Non-empty, no negative IDs. | +| `sampling_params` | Required. Any [SamplingParams](../../api/README.md#inference-parameters) field, e.g. `max_tokens`, `temperature`, `logprobs`, `stop_token_ids`. Without `max_tokens`, the server applies its own default (as on the OpenAI endpoints), not 16. | +| `output_mode` | `"tokens"` (default) or `"text"`, see [Output modes](#output-modes). | +| `stream`, `stream_options` | SSE chunks ending with `data: [DONE]`. | +| `return_token_ids` | Echo the final prompt IDs as `prompt_token_ids`. | +| `features` / `content_parts` | Multimodal input, mutually exclusive. | +| `request_id`, `model`, `cache_salt`, `priority`, `kv_transfer_params`, `ec_transfer_params` | Optional. | + +## Response fields + +| `choices[i]` field | Notes | +| --- | --- | +| `token_ids` | Generated token IDs. | +| `finish_reason` | `stop`, `length` or `abort`. | +| `stop_reason` | Stop token ID or string; `null` on EOS, length and abort. | +| `logprobs` | `{"content": [{"token_id", "logprob", "rank", "top_logprobs": [...]}]}`, one entry per token, sampled token first. `token_id` is an integer, not the older `"token_id:"` string. Needs `sampling_params.logprobs`. | +| `weight_versions`, `routed_experts`, `sampling_mask` | Optional, see [Weight versions](#weight-versions); the last two need `--enable-return-routed-experts` / `--return-sampling-mask`. | + +The top level adds `request_id`, `usage`, `prompt_token_ids` and `prompt_logprobs`. Stream chunks carry the same per-choice fields for the new tokens. + +## Targeting a data-parallel rank (`X-data-parallel-rank`) + +Send the header `X-data-parallel-rank: ` to pin a request to one data-parallel (DP) engine rank; there is no body field. Without the header, or with a non-integer value, the request is load balanced. A rank outside `[0, data_parallel_size)` is rejected. ## Output modes diff --git a/docs/training/sampling_mask.md b/docs/training/sampling_mask.md index 7de2366800..25dfb414d0 100644 --- a/docs/training/sampling_mask.md +++ b/docs/training/sampling_mask.md @@ -139,8 +139,9 @@ Each position then carries the sampled token's logprob and the top-128 drop them. The finite entries are the sampling mask when it has at most `k` tokens. - The sampled token is always the first entry and may repeat inside the top-k. -- `/inference/v1/generate` encodes IDs as `token_id:`; the OpenAI - endpoints need `return_tokens_as_token_ids`. +- The `token_id:` strings come from the OpenAI endpoints with + `return_tokens_as_token_ids`; `/inference/v1/generate` returns an integer + `token_id` instead. ## Limitations diff --git a/examples/scale_out/token_generation_client.py b/examples/scale_out/token_generation_client.py index a4792a68f9..063e8ae39b 100644 --- a/examples/scale_out/token_generation_client.py +++ b/examples/scale_out/token_generation_client.py @@ -3,6 +3,8 @@ """Call a token-in/token-out server started with: vllm serve Qwen/Qwen3-0.6B --enable-scale-out + +Fields and DP rank routing: docs/serving/online_serving/token_in_token_out.md """ import httpx