[RL][Frontend] Return token-level weight-version spans on final outputs (#59582)
Signed-off-by: Shuolei Wang <shuoleiwang123@gmail.com> Signed-off-by: Shuolei Wang <948904026@qq.com> Co-authored-by: aoshen02 <aoshen@inferact.ai>
This commit is contained in:
@@ -55,6 +55,21 @@ The server returns a 400 for:
|
||||
|
||||
For using `output_mode` with separate prefill and decode pools, see [Disaggregated Prefilling](../../features/disagg_prefill.md#generate-api-output-modes).
|
||||
|
||||
## Weight versions
|
||||
|
||||
The final output of every choice carries `weight_versions`, the label of the weights that sampled each range of its output tokens. Ranges are half-open over output-token indexes (prompt excluded), contiguous and in generation order, so a request that crossed one weight update gets two:
|
||||
|
||||
```json
|
||||
"weight_versions": [
|
||||
{"version": "v7", "start": 0, "end": 57},
|
||||
{"version": "v8", "start": 57, "end": 128}
|
||||
]
|
||||
```
|
||||
|
||||
- With `stream: true`, only the last chunk of each choice carries the list, including the last chunk after an abort. Its offsets cover the whole output. Earlier chunks carry `null`.
|
||||
- An output with no tokens, such as a request aborted before its first token, gets `[]`.
|
||||
- The labels are the ones you set with `/update_weight_version` or `/finish_weight_update`, see [Async Reinforcement Learning](../../training/async_rl.md#weight-versions-in-outputs). vLLM doesn't order or check them, and the label is `"default"` until you set one.
|
||||
|
||||
## Aborting requests
|
||||
|
||||
`POST /inference/v1/abort_requests` aborts in-flight requests. It is registered wherever `/inference/v1/generate` is and requires the API key when `--api-key` is set, like `/inference/v1/generate`.
|
||||
|
||||
@@ -43,10 +43,11 @@ With `VLLM_SERVER_DEV_MODE=1`, the vLLM HTTP server exposes the same functionali
|
||||
- `POST /pause?mode=keep` - Pause generation
|
||||
- `POST /resume` - Resume generation
|
||||
- `POST /abort_requests` - Abort in-flight requests without pausing the scheduler (send `{}` to abort all, or `{"request_ids": [...]}`)
|
||||
- `POST /update_weight_version` - Set the `weight_version` label without changing weights (send `{"new_version": "v8"}`)
|
||||
- `GET /weight_info` - Return the latest committed `weight_version`
|
||||
|
||||
!!! note "Data Parallelism"
|
||||
When using data parallelism with vLLM's **internal load balancer** (i.e. `data_parallel_backend="ray"`), pause and resume are handled automatically across all DP ranks -- a single call is sufficient. When using an **external load balancer** (i.e. multiple independent vLLM instances behind a proxy), you must send pause and resume requests to **every** engine instance individually before and after the weight update.
|
||||
When using data parallelism with vLLM's **internal load balancer** (i.e. `data_parallel_backend="ray"`), pause and resume are handled automatically across all DP ranks -- a single call is sufficient. When using an **external load balancer** (i.e. multiple independent vLLM instances behind a proxy), you must send pause and resume requests to **every** engine instance individually before and after the weight update, and set the weight version on every instance too.
|
||||
|
||||
## Typical Async RL Flow
|
||||
|
||||
@@ -54,12 +55,27 @@ A typical async RL loop with weight syncing looks like this:
|
||||
|
||||
1. Start generating rollouts from the current policy
|
||||
2. Once trainer has new weights to update to, pause generation with `mode="keep"`
|
||||
3. Sync the updated weights from the trainer to the inference engine (see [Weight Transfer](weight_transfer/README.md))
|
||||
3. Sync the updated weights from the trainer to the inference engine (see [Weight Transfer](weight_transfer/README.md)) and set their weight version
|
||||
4. Resume generation -- in-flight requests continue with the new weights
|
||||
5. Repeat
|
||||
|
||||
The key insight is that requests paused with `mode="keep"` will produce tokens from the **old** weights before the pause and tokens from the **new** weights after resume. The `clear_cache` parameter controls whether the KV cache is invalidated during the pause. When `clear_cache=True`, previously cached key-value entries are discarded, so all tokens generated after resume will be computed entirely with the new weights. When `clear_cache=False`, existing KV cache entries are retained, meaning some tokens in context may still reflect the old weights (stale KV cache).
|
||||
|
||||
## Weight Versions in Outputs
|
||||
|
||||
To tell which tokens came from which weights, set a weight-version label while generation is paused, after the new weights are loaded: pass `weight_version` to `finish_weight_update`, or call `update_weight_version` (`POST /update_weight_version` over HTTP). Labels are strings you choose, such as the trainer step; vLLM doesn't order or check them.
|
||||
|
||||
The final output of every sequence then carries `weight_versions`, one span per label over its output-token indexes. A request paused with `mode="keep"` across one update gets two spans:
|
||||
|
||||
```python
|
||||
completion = final_output.outputs[0]
|
||||
for span in completion.weight_versions:
|
||||
# The tokens in [span.start, span.end) were sampled by weights `span.version`.
|
||||
print(span.version, completion.token_ids[span.start : span.end])
|
||||
```
|
||||
|
||||
The offsets cover the whole output, so with streamed delta outputs, collect the tokens before slicing. `/inference/v1/generate` returns the same list on each choice; see [Tokens In <> Tokens Out API](../serving/online_serving/token_in_token_out.md#weight-versions).
|
||||
|
||||
## Example
|
||||
|
||||
The [async RLHF example](../../examples/rl/rlhf_async_new_apis.py) demonstrates this pattern with `vllm.AsyncLLMEngine`, NCCL weight transfer, and mid-flight pause/resume with validation.
|
||||
|
||||
@@ -26,8 +26,8 @@ The example performs the following steps:
|
||||
* Pause generation once any request reaches a token threshold.
|
||||
* Broadcast the training model's weights to the inference engine
|
||||
via the NCCL weight transfer engine, replacing the base weights.
|
||||
* Resume generation and collect results, noting which tokens were
|
||||
generated before vs. after the weight swap.
|
||||
* Label the new weights, resume generation, and use `weight_versions`
|
||||
to locate the weight swap in each output.
|
||||
* Validate correctness by launching a fresh vLLM instance loaded
|
||||
directly with the training model and comparing its output to the
|
||||
post-swap tokens from the weight-synced engine.
|
||||
@@ -61,6 +61,7 @@ from vllm.v1.executor import Executor
|
||||
MODEL_NAME_V1 = "Qwen/Qwen3-1.7B-Base"
|
||||
MODEL_NAME_V2 = "Qwen/Qwen3-1.7B"
|
||||
PAUSE_TOKEN_THRESHOLD = 10
|
||||
WEIGHT_VERSION_V2 = "v2"
|
||||
ATTN_BACKEND = "TRITON_ATTN" if current_platform.is_rocm() else "FLASH_ATTN"
|
||||
|
||||
|
||||
@@ -77,44 +78,27 @@ class MyLLM(vllm.AsyncLLMEngine):
|
||||
log_requests=engine_args.enable_log_requests,
|
||||
log_stats=not engine_args.disable_log_stats,
|
||||
)
|
||||
self._generation_paused = False
|
||||
self._request_pause_flag = False
|
||||
|
||||
async def do_generate(
|
||||
self, prompt_token_ids: list[int], sampling_params: vllm.SamplingParams
|
||||
) -> tuple[vllm.RequestOutput, int]:
|
||||
"""Generate a single request, setting the request pause flag once the
|
||||
token count reaches the threshold.
|
||||
|
||||
Returns (output, pause_token_index). pause_token_index is the number
|
||||
of tokens generated before the weight change, or -1 if no pause.
|
||||
"""
|
||||
pause_token_index = -1
|
||||
prev_token_count = 0
|
||||
) -> vllm.RequestOutput:
|
||||
"""Generate a request and set the pause flag at the token threshold."""
|
||||
async for request_output in self.generate(
|
||||
{"prompt_token_ids": prompt_token_ids},
|
||||
sampling_params,
|
||||
request_id=str(uuid.uuid4()),
|
||||
):
|
||||
output = request_output
|
||||
cur_token_count = len(output.outputs[0].token_ids)
|
||||
if (
|
||||
cur_token_count >= PAUSE_TOKEN_THRESHOLD
|
||||
and not self._request_pause_flag
|
||||
):
|
||||
if len(output.outputs[0].token_ids) >= PAUSE_TOKEN_THRESHOLD:
|
||||
self._request_pause_flag = True
|
||||
if self._generation_paused and pause_token_index == -1:
|
||||
pause_token_index = prev_token_count
|
||||
prev_token_count = cur_token_count
|
||||
return output, pause_token_index
|
||||
return output
|
||||
|
||||
async def pause_after_n_tokens(self):
|
||||
"""Wait for any request to set the pause flag, then pause."""
|
||||
while not self._request_pause_flag:
|
||||
await asyncio.sleep(0)
|
||||
await super().pause_generation(mode="keep")
|
||||
await asyncio.sleep(5)
|
||||
self._generation_paused = True
|
||||
|
||||
|
||||
@ray.remote(num_gpus=1)
|
||||
@@ -262,9 +246,18 @@ gen_futures = [
|
||||
ray.get(llm.pause_after_n_tokens.remote())
|
||||
|
||||
ray.get(train_model.broadcast_weights.remote())
|
||||
# Label the new weights before resuming generation.
|
||||
ray.get(llm.update_weight_version.remote(WEIGHT_VERSION_V2))
|
||||
|
||||
ray.get(llm.resume_generation.remote())
|
||||
results = ray.get(gen_futures)
|
||||
results = []
|
||||
for output in ray.get(gen_futures):
|
||||
completion = output.outputs[0]
|
||||
pause_idx = next(
|
||||
(s.start for s in completion.weight_versions if s.version == WEIGHT_VERSION_V2),
|
||||
len(completion.token_ids),
|
||||
)
|
||||
results.append((output, pause_idx))
|
||||
|
||||
for i, (output, pause_idx) in enumerate(results):
|
||||
all_token_ids = list(output.outputs[0].token_ids)
|
||||
@@ -326,7 +319,7 @@ val_results = ray.get(val_futures)
|
||||
|
||||
num_pass = 0
|
||||
num_total = len(results)
|
||||
for i, ((output, pause_idx), (val_output, _)) in enumerate(zip(results, val_results)):
|
||||
for i, ((output, pause_idx), val_output) in enumerate(zip(results, val_results)):
|
||||
expected = list(output.outputs[0].token_ids)[pause_idx:]
|
||||
actual = list(val_output.outputs[0].token_ids)
|
||||
match = actual == expected
|
||||
|
||||
@@ -228,6 +228,9 @@ struct WireEngineCoreOutputs {
|
||||
/// wave needs to start in other engines.
|
||||
#[serde(default)]
|
||||
start_wave: Option<u32>,
|
||||
/// Decoded for Python wire compatibility; not exposed by the Rust frontend.
|
||||
#[serde(default)]
|
||||
weight_version: Option<String>,
|
||||
}
|
||||
|
||||
/// Data-parallel control notifications multiplexed through `EngineCoreOutputs`.
|
||||
|
||||
@@ -139,6 +139,7 @@ class EngineCoreOutputs(
|
||||
finished_requests: set[str] | None = None
|
||||
wave_complete: int | None = None
|
||||
start_wave: int | None = None
|
||||
weight_version: str | None = None
|
||||
|
||||
|
||||
request = EngineCoreRequest(
|
||||
@@ -240,6 +241,7 @@ outputs = EngineCoreOutputs(
|
||||
)
|
||||
],
|
||||
finished_requests={"req-1"},
|
||||
weight_version="v7",
|
||||
)
|
||||
|
||||
extended_outputs = EngineCoreOutputs(
|
||||
@@ -252,6 +254,7 @@ extended_outputs = EngineCoreOutputs(
|
||||
)
|
||||
],
|
||||
finished_requests=outputs.finished_requests,
|
||||
weight_version=outputs.weight_version,
|
||||
)
|
||||
extended_outputs_bytes = msgspec.msgpack.encode(extended_outputs)
|
||||
# The ordinary frontend's schema ignores even non-default extension values.
|
||||
|
||||
@@ -24,6 +24,7 @@ from vllm.outputs import (
|
||||
PoolingRequestOutput,
|
||||
RequestOutput,
|
||||
SamplingMask,
|
||||
WeightVersionSpan,
|
||||
)
|
||||
from vllm.sampling_params import RequestOutputKind, SamplingParams
|
||||
from vllm.tokenizers import TokenizerLike
|
||||
@@ -87,6 +88,7 @@ def test_completion_output_preserves_each_sampling_mask_position() -> None:
|
||||
]
|
||||
state.routed_experts_chunks = []
|
||||
state.spec_decode_metrics = None
|
||||
state.weight_version_starts = []
|
||||
|
||||
output = state._new_completion_output([1, 2, 3], FinishReason.LENGTH, None)
|
||||
|
||||
@@ -1596,6 +1598,7 @@ def test_sampling_masks_follow_output_kind(output_kind):
|
||||
state.sampling_mask_chunks = []
|
||||
state.routed_experts_chunks = []
|
||||
state.spec_decode_metrics = None
|
||||
state.weight_version_starts = []
|
||||
|
||||
supports = [[10, 11], [20], [30, 31]]
|
||||
masks = []
|
||||
@@ -1725,3 +1728,53 @@ def test_async_pooling_cache_miss_does_not_fail_output_processor():
|
||||
assert output.error is not None
|
||||
assert output.error.retryable
|
||||
assert not output_processor.has_unfinished_requests()
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"output_kind", [RequestOutputKind.DELTA, RequestOutputKind.FINAL_ONLY]
|
||||
)
|
||||
@pytest.mark.parametrize(
|
||||
"steps,expected",
|
||||
[
|
||||
# Merge adjacent equal labels; A -> B -> A remains three spans.
|
||||
(
|
||||
[("v7", [10, 11]), ("v7", [12]), ("v8", [13, 14]), ("v7", [15])],
|
||||
[("v7", 0, 3), ("v8", 3, 5), ("v7", 5, 6)],
|
||||
),
|
||||
# An output without tokens has no spans.
|
||||
([], []),
|
||||
],
|
||||
)
|
||||
def test_weight_version_spans(output_kind: RequestOutputKind, steps, expected):
|
||||
output_processor = OutputProcessor(tokenizer=None, log_stats=False)
|
||||
queue = RequestOutputCollector(output_kind, request_id="request-0-int")
|
||||
request = EngineCoreRequest(
|
||||
request_id="request-0-int",
|
||||
external_req_id="request-0",
|
||||
prompt_token_ids=[1, 2, 3],
|
||||
mm_features=None,
|
||||
arrival_time=0,
|
||||
lora_request=None,
|
||||
cache_salt=None,
|
||||
data_parallel_rank=None,
|
||||
sampling_params=SamplingParams(detokenize=False, output_kind=output_kind),
|
||||
pooling_params=None,
|
||||
)
|
||||
output_processor.add_request(request, prompt=None, queue=queue)
|
||||
|
||||
for i, (weight_version, token_ids) in enumerate(steps):
|
||||
# Copy: merging deltas extends token lists in place.
|
||||
output = EngineCoreOutput(
|
||||
request_id="request-0-int", new_token_ids=list(token_ids)
|
||||
)
|
||||
output_processor.process_outputs([output], weight_version=weight_version)
|
||||
if i == 0 and output_kind == RequestOutputKind.DELTA:
|
||||
intermediate = queue.get_nowait()
|
||||
assert intermediate is not None
|
||||
assert intermediate.outputs[0].weight_versions is None
|
||||
|
||||
# Unread deltas merge into the abort output without losing spans.
|
||||
output_processor.abort_requests(["request-0"], internal=False)
|
||||
final = queue.get_nowait()
|
||||
assert final is not None and final.finished
|
||||
assert final.outputs[0].weight_versions == [WeightVersionSpan(*s) for s in expected]
|
||||
|
||||
@@ -31,6 +31,7 @@ from vllm.entrypoints.openai.completion.protocol import (
|
||||
)
|
||||
from vllm.entrypoints.serve.engine.protocol import UsageInfo
|
||||
from vllm.logprobs import Logprob
|
||||
from vllm.outputs import WeightVersionSpan
|
||||
from vllm.renderers import TokenizeParams
|
||||
from vllm.sampling_params import SamplingParams
|
||||
from vllm.utils import random_uuid
|
||||
@@ -411,6 +412,8 @@ class GenerateChoiceBase(BaseModel):
|
||||
# or (b) `enable_return_routed_experts` is off server-side.
|
||||
routed_experts: str | None = None
|
||||
sampling_mask: list[list[int]] | None = None
|
||||
# Contiguous weight version spans over output-token indexes.
|
||||
weight_versions: list[WeightVersionSpan] | None = None
|
||||
|
||||
@field_validator("token_ids")
|
||||
@classmethod
|
||||
@@ -443,6 +446,8 @@ class GenerateStreamChoiceBase(BaseModel):
|
||||
token_ids: list[int] | None = None
|
||||
routed_experts: str | None = None
|
||||
sampling_mask: list[list[int]] | None = None
|
||||
# Set on the final chunk only; covers the whole output.
|
||||
weight_versions: list[WeightVersionSpan] | None = None
|
||||
|
||||
|
||||
class GenerateTokensStreamChoice(GenerateStreamChoiceBase):
|
||||
|
||||
@@ -453,6 +453,7 @@ class ServingTokens(GenerateBaseServing):
|
||||
token_ids=as_list(output.token_ids),
|
||||
routed_experts=routed_experts_b64,
|
||||
sampling_mask=sampling_mask,
|
||||
weight_versions=output.weight_versions,
|
||||
)
|
||||
if text_mode:
|
||||
text_choices.append(
|
||||
@@ -625,6 +626,7 @@ class ServingTokens(GenerateBaseServing):
|
||||
token_ids=as_list(delta_token_ids),
|
||||
routed_experts=routed_experts_b64,
|
||||
sampling_mask=sampling_mask,
|
||||
weight_versions=output.weight_versions,
|
||||
)
|
||||
chunk: GenerateTokensStreamResponse | GenerateTextStreamResponse
|
||||
if text_mode:
|
||||
|
||||
@@ -38,6 +38,15 @@ class SamplingMask:
|
||||
token_ids: list[list[int]]
|
||||
|
||||
|
||||
@dataclass
|
||||
class WeightVersionSpan:
|
||||
"""Output tokens `[start, end)` were sampled by weights labelled `version`."""
|
||||
|
||||
version: str
|
||||
start: int
|
||||
end: int
|
||||
|
||||
|
||||
@dataclass
|
||||
class CompletionOutput:
|
||||
"""The output data of one completion output of a request.
|
||||
@@ -62,6 +71,8 @@ class CompletionOutput:
|
||||
``--per-request-spec-decode-metrics`` is enabled; None otherwise.
|
||||
Surfaced in the response as ``metrics.speculative_decoding`` for
|
||||
single-sequence (``n == 1``) requests.
|
||||
weight_versions: Contiguous weight version spans over output-token indexes.
|
||||
Set on the final output only.
|
||||
|
||||
"""
|
||||
|
||||
@@ -76,6 +87,7 @@ class CompletionOutput:
|
||||
lora_request: LoRARequest | None = None
|
||||
sampling_mask: SamplingMask | None = None
|
||||
spec_decode_metrics: RequestSpecDecodeMetrics | None = None
|
||||
weight_versions: list[WeightVersionSpan] | None = None
|
||||
|
||||
def finished(self) -> bool:
|
||||
return self.finish_reason is not None
|
||||
@@ -223,6 +235,8 @@ class RequestOutput:
|
||||
# aggregation with earlier chunks that have no R3.
|
||||
if next_completion.routed_experts is not None:
|
||||
completion.routed_experts = next_completion.routed_experts
|
||||
# Like finish_reason, spans come only with the terminal output.
|
||||
completion.weight_versions = next_completion.weight_versions
|
||||
completion.finish_reason = next_completion.finish_reason
|
||||
completion.stop_reason = next_completion.stop_reason
|
||||
else:
|
||||
|
||||
@@ -282,6 +282,9 @@ class EngineCoreOutputs(
|
||||
# "old" wave, so the next wave needs to be started in other engines.
|
||||
start_wave: int | None = None
|
||||
|
||||
# Weight version used for this step.
|
||||
weight_version: str | None = None
|
||||
|
||||
def __post_init__(self):
|
||||
if self.timestamp == 0.0:
|
||||
self.timestamp = time.monotonic()
|
||||
|
||||
@@ -855,7 +855,10 @@ class AsyncLLM(EngineClient):
|
||||
outputs_slice = engine_core_outputs[start:end]
|
||||
# 2) Process EngineCoreOutputs.
|
||||
processed_outputs = output_processor.process_outputs(
|
||||
outputs_slice, outputs.timestamp, iteration_stats
|
||||
outputs_slice,
|
||||
outputs.timestamp,
|
||||
iteration_stats,
|
||||
weight_version=outputs.weight_version,
|
||||
)
|
||||
# NOTE: RequestOutputs are pushed to their queues.
|
||||
assert not processed_outputs.request_outputs
|
||||
|
||||
@@ -639,6 +639,11 @@ class EngineCore:
|
||||
else:
|
||||
eco.scheduler_stats.iteration_details = iteration_details
|
||||
|
||||
def _attach_weight_version(self, outputs: dict[int, EngineCoreOutputs]) -> None:
|
||||
# Weight versions change only while generation is paused.
|
||||
for eco in outputs.values():
|
||||
eco.weight_version = self._weight_version
|
||||
|
||||
def _should_throttle_prefills(self) -> bool:
|
||||
"""Whether to defer new prefills this step (DP prefill balancing).
|
||||
Overridden by the DP engine core; never throttles otherwise."""
|
||||
@@ -672,6 +677,7 @@ class EngineCore:
|
||||
scheduler_output, model_output
|
||||
)
|
||||
self._attach_iteration_details(engine_core_outputs, iteration_details)
|
||||
self._attach_weight_version(engine_core_outputs)
|
||||
|
||||
return engine_core_outputs, scheduler_output.total_num_scheduled_tokens > 0
|
||||
|
||||
@@ -773,6 +779,7 @@ class EngineCore:
|
||||
scheduler_output, model_output
|
||||
)
|
||||
self._attach_iteration_details(engine_core_outputs, iteration_details)
|
||||
self._attach_weight_version(engine_core_outputs)
|
||||
|
||||
# NOTE(nick): We can either handle the deferred tasks here or save
|
||||
# in a field and do it immediately once step_with_batch_queue is
|
||||
|
||||
@@ -335,6 +335,7 @@ class LLMEngine:
|
||||
outputs.outputs,
|
||||
engine_core_timestamp=outputs.timestamp,
|
||||
iteration_stats=iteration_stats,
|
||||
weight_version=outputs.weight_version,
|
||||
)
|
||||
|
||||
mm_processor_cache = self.renderer.mm_processor_cache
|
||||
|
||||
@@ -19,6 +19,7 @@ from vllm.outputs import (
|
||||
RequestError,
|
||||
RequestOutput,
|
||||
SamplingMask,
|
||||
WeightVersionSpan,
|
||||
)
|
||||
from vllm.sampling_params import RequestOutputKind
|
||||
from vllm.tokenizers import TokenizerLike
|
||||
@@ -190,6 +191,8 @@ class RequestState:
|
||||
# Routed experts accumulation (prompt + sample chunks)
|
||||
self.routed_experts_chunks: list[np.ndarray] = []
|
||||
self.sampling_mask_chunks: list[SamplingMaskLists] = []
|
||||
# (weight version, first output-token index) for each span.
|
||||
self.weight_version_starts: list[tuple[str, int]] = []
|
||||
|
||||
# Stream Interval
|
||||
self.stream_interval = stream_interval
|
||||
@@ -471,8 +474,25 @@ class RequestState:
|
||||
finish_reason=str(finish_reason) if finished else None,
|
||||
stop_reason=stop_reason if finished else None,
|
||||
spec_decode_metrics=self.spec_decode_metrics if finished else None,
|
||||
weight_versions=self._weight_version_spans() if finished else None,
|
||||
)
|
||||
|
||||
def track_weight_version(self, weight_version: str) -> None:
|
||||
"""Start a span when the weight version changes."""
|
||||
assert self.detokenizer is not None
|
||||
starts = self.weight_version_starts
|
||||
if not starts or starts[-1][0] != weight_version:
|
||||
starts.append((weight_version, self.detokenizer.num_output_tokens()))
|
||||
|
||||
def _weight_version_spans(self) -> list[WeightVersionSpan] | None:
|
||||
assert self.detokenizer is not None
|
||||
num_tokens = self.detokenizer.num_output_tokens()
|
||||
starts = self.weight_version_starts
|
||||
if not starts and num_tokens:
|
||||
return None # No label was reported for these tokens.
|
||||
ends = [start for _, start in starts[1:]] + [num_tokens]
|
||||
return [WeightVersionSpan(v, s, e) for (v, s), e in zip(starts, ends)]
|
||||
|
||||
def _new_pooling_output(self, pooling_output: torch.Tensor) -> PoolingOutput:
|
||||
return PoolingOutput(data=pooling_output)
|
||||
|
||||
@@ -659,6 +679,7 @@ class OutputProcessor:
|
||||
engine_core_outputs: list[EngineCoreOutput],
|
||||
engine_core_timestamp: float | None = None,
|
||||
iteration_stats: IterationStats | None = None,
|
||||
weight_version: str | None = None,
|
||||
) -> OutputProcessorOutput:
|
||||
"""Process the EngineCoreOutputs:
|
||||
1) Compute stats for logging
|
||||
@@ -736,6 +757,8 @@ class OutputProcessor:
|
||||
if pooling_output is None:
|
||||
assert req_state.detokenizer is not None
|
||||
assert req_state.logprobs_processor is not None
|
||||
if new_token_ids and weight_version is not None:
|
||||
req_state.track_weight_version(weight_version)
|
||||
# 2) Detokenize the token ids into text and perform stop checks.
|
||||
num_prev_tokens = req_state.detokenizer.num_output_tokens()
|
||||
stop_string = req_state.detokenizer.update(
|
||||
|
||||
Reference in New Issue
Block a user