[Frontend] Consolidate scale out entrypoints (#44512)
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io>
This commit is contained in:
@@ -13,5 +13,6 @@ vLLM's examples are organized into the following categories:
|
||||
- **[`rl/`](../../examples/rl)** – Reinforcement learning examples.
|
||||
- **[`deployment/`](../../examples/deployment)** – Examples for deploying vLLM in production.
|
||||
- **[`ray_serving/`](../../examples/ray_serving)** – Scalable serving using Ray.
|
||||
- **[`disaggregated/`](../../examples/disaggregated)** – Examples for disaggregated serving (separate prefill and decode), including various kv cache connectors (LMCache, Mooncake, FlexKV, P2P NCCL) and failure recovery.
|
||||
- **[`disaggregated/`](../../examples/disaggregated)** – Examples for Disaggregated P/D (Prefill/Decoding) inference, including various kv cache connectors (LMCache, Mooncake, FlexKV, P2P NCCL) and failure recovery.
|
||||
- **[`scale_out/`](../../examples/scale_out)** – Examples for Token In <> Token Out API Server.
|
||||
- **[`observability/`](../../examples/observability)** – Metrics, logging, tracing (OpenTelemetry), and dashboards (Grafana, Perses).
|
||||
|
||||
@@ -119,9 +119,9 @@ For further details on profiling vLLM, please refer to [this page](../../contrib
|
||||
- `/ping` - SageMaker health check
|
||||
- `/invocations` - SageMaker-compatible endpoint (routes to the same inference functions as `/v1` endpoints)
|
||||
|
||||
## Disaggregated Everything
|
||||
## Scale-Out APIs
|
||||
|
||||
### Tokens IN <> Tokens OUT
|
||||
### Tokens IN <> Tokens OUT APIs
|
||||
|
||||
- `/inference/v1/generate` - Generate completions
|
||||
- `/abort_requests` - Abort in-flight requests (only when `--tokens-only` is also set)
|
||||
|
||||
Reference in New Issue
Block a user