[Doc] Document /inference/v1/generate request/response fields and DP rank header (#62305)

Signed-off-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

docs/serving/online_serving/token_in_token_out.md

@@ -1,8 +1,50 @@
 # Tokens In <> Tokens Out API
 
-`/inference/v1/generate` takes prompt token IDs and returns generated token IDs. It is the generation step of the `render → generate → derender` pipeline used in disaggregated serving (see the [Renderer APIs](renderer.md) and [Derenderer APIs](derenderer.md)).
+`POST /inference/v1/generate` is the token-in/token-out (TITO) HTTP endpoint: it takes prompt token IDs and returns generated token IDs, logprobs keyed by integer token ID, and `finish_reason`/`stop_reason`, with no detokenization. Use it when the caller does its own tokenization and chat templating, such as an RL rollout worker. It is also the generation step of the `render → generate → derender` pipeline used in disaggregated serving (see the [Renderer APIs](renderer.md) and [Derenderer APIs](derenderer.md)).
 
-The endpoint is registered on `vllm serve` with `--enable-scale-out`, and always with `--tokens-only`. `--tokens-only` also skips loading the tokenizer, so the server can't turn token IDs back into text.
+## Enabling the endpoint (`--enable-scale-out`, `--tokens-only`)
+
+- `--enable-scale-out` registers it, with `/render`, `/derender` and `/inference/v1/abort_requests`.
+- `--tokens-only` always registers it, skips loading the tokenizer and forces `detokenize=False`. Stop strings (`sampling_params.stop`) are then rejected with a 400; use `stop_token_ids`.
+- With neither flag, it returns 403 `Scale-out endpoints are disabled. Restart vllm serve with --enable-scale-out to enable them.`
+
+```bash
+vllm serve Qwen/Qwen3-0.6B --enable-scale-out
+
+curl -X POST http://localhost:8000/inference/v1/generate \
+  -H "Content-Type: application/json" \
+  -d '{"token_ids": [151644, 872, 198], "sampling_params": {"max_tokens": 32, "logprobs": 1}}'
+```
+
+A full client is in [examples/scale_out/token_generation_client.py](../../../examples/scale_out/token_generation_client.py).
+
+## Request fields
+
+| Field | Notes |
+| --- | --- |
+| `token_ids` | Required `list[int]`: the already-templated prompt. Non-empty, no negative IDs. |
+| `sampling_params` | Required. Any [SamplingParams](../../api/README.md#inference-parameters) field, e.g. `max_tokens`, `temperature`, `logprobs`, `stop_token_ids`. Without `max_tokens`, the server applies its own default (as on the OpenAI endpoints), not 16. |
+| `output_mode` | `"tokens"` (default) or `"text"`, see [Output modes](#output-modes). |
+| `stream`, `stream_options` | SSE chunks ending with `data: [DONE]`. |
+| `return_token_ids` | Echo the final prompt IDs as `prompt_token_ids`. |
+| `features` / `content_parts` | Multimodal input, mutually exclusive. |
+| `request_id`, `model`, `cache_salt`, `priority`, `kv_transfer_params`, `ec_transfer_params` | Optional. |
+
+## Response fields
+
+| `choices[i]` field | Notes |
+| --- | --- |
+| `token_ids` | Generated token IDs. |
+| `finish_reason` | `stop`, `length` or `abort`. |
+| `stop_reason` | Stop token ID or string; `null` on EOS, length and abort. |
+| `logprobs` | `{"content": [{"token_id", "logprob", "rank", "top_logprobs": [...]}]}`, one entry per token, sampled token first. `token_id` is an integer, not the older `"token_id:<id>"` string. Needs `sampling_params.logprobs`. |
+| `weight_versions`, `routed_experts`, `sampling_mask` | Optional, see [Weight versions](#weight-versions); the last two need `--enable-return-routed-experts` / `--return-sampling-mask`. |
+
+The top level adds `request_id`, `usage`, `prompt_token_ids` and `prompt_logprobs`. Stream chunks carry the same per-choice fields for the new tokens.
+
+## Targeting a data-parallel rank (`X-data-parallel-rank`)
+
+Send the header `X-data-parallel-rank: <int>` to pin a request to one data-parallel (DP) engine rank; there is no body field. Without the header, or with a non-integer value, the request is load balanced. A rank outside `[0, data_parallel_size)` is rejected.
 
 ## Output modes
 

docs/training/sampling_mask.md

@@ -139,8 +139,9 @@ Each position then carries the sampled token's logprob and the top-128
   drop them. The finite entries are the sampling mask when it has at most `k`
   tokens.
 - The sampled token is always the first entry and may repeat inside the top-k.
-- `/inference/v1/generate` encodes IDs as `token_id:<id>`; the OpenAI
-  endpoints need `return_tokens_as_token_ids`.
+- The `token_id:<id>` strings come from the OpenAI endpoints with
+  `return_tokens_as_token_ids`; `/inference/v1/generate` returns an integer
+  `token_id` instead.
 
 ## Limitations
 

examples/scale_out/token_generation_client.py

@@ -3,6 +3,8 @@
 """Call a token-in/token-out server started with:
 
 vllm serve Qwen/Qwen3-0.6B --enable-scale-out
+
+Fields and DP rank routing: docs/serving/online_serving/token_in_token_out.md
 """
 
 import httpx