[feat] Add explicit /start_weight_update and /finish_weight_update APIs for weight transfer (#39212)

This commit is contained in:
Sumanth R Hegde
2026-05-08 18:03:33 -07:00
committed by GitHub
parent 30f519e947
commit e3b65a5ba0
19 changed files with 287 additions and 53 deletions
@@ -211,6 +211,11 @@ class IPCWeightTransferEngine(
- 'ray': Sends weights via Ray RPC to a Ray-based LLM handle
- 'http': Sends weights via HTTP POST to a vLLM HTTP server
.. note::
This method calls ``update_weights`` internally. The caller must
call ``start_weight_update`` before and ``finish_weight_update``
after this method.
Args:
iterator: Iterator of model parameters. Returns (name, tensor) tuples.
Tensors should be on the same GPU as the inference workers.