[Guards] The bounds-guard telemetry code default is warn
VLLM_E2_GUARD_MODE now defaults to warn in code, matching what the shipped launchers set; v2 through v2.2.0 defaulted to off. The comment above it, the MODES docstring, the vllm/envs.py default and its annotation follow. tests/v1/worker/gpu/test_guard_default_mode.py checks that code, comment and envs.py agree, from the sources, so it runs without CUDA, and fails on the earlier source. Co-Authored-By: Hinata <hinata@idihore.id> Signed-off-by: MOVZX <movzx@yahoo.com>
This commit is contained in:
+2
-2
@@ -405,7 +405,7 @@ if TYPE_CHECKING:
|
|||||||
# Guard telemetry: mode raise|warn|off and the reader poll interval in
|
# Guard telemetry: mode raise|warn|off and the reader poll interval in
|
||||||
# seconds. counter() returns a real tensor in every mode, so the Triton
|
# seconds. counter() returns a real tensor in every mode, so the Triton
|
||||||
# arguments and the captured graph do not depend on either.
|
# arguments and the captured graph do not depend on either.
|
||||||
VLLM_E2_GUARD_MODE: str = "off"
|
VLLM_E2_GUARD_MODE: str = "warn"
|
||||||
VLLM_E2_GUARD_INTERVAL: str = "0.05"
|
VLLM_E2_GUARD_INTERVAL: str = "0.05"
|
||||||
# Debug allocator telemetry (poll seconds + output directory) and the
|
# Debug allocator telemetry (poll seconds + output directory) and the
|
||||||
# scheduler allocation log path.
|
# scheduler allocation log path.
|
||||||
@@ -2507,7 +2507,7 @@ environment_variables: dict[str, Callable[[], Any]] = {
|
|||||||
).strip(),
|
).strip(),
|
||||||
"VLLM_QSA_KV_SCALES": lambda: os.getenv("VLLM_QSA_KV_SCALES", "").strip(),
|
"VLLM_QSA_KV_SCALES": lambda: os.getenv("VLLM_QSA_KV_SCALES", "").strip(),
|
||||||
"VLLM_E2_GUARD_MODE": lambda: (
|
"VLLM_E2_GUARD_MODE": lambda: (
|
||||||
os.getenv("VLLM_E2_GUARD_MODE", "off").strip().lower()
|
os.getenv("VLLM_E2_GUARD_MODE", "warn").strip().lower()
|
||||||
),
|
),
|
||||||
"VLLM_E2_GUARD_INTERVAL": lambda: os.getenv("VLLM_E2_GUARD_INTERVAL", "0.05"),
|
"VLLM_E2_GUARD_INTERVAL": lambda: os.getenv("VLLM_E2_GUARD_INTERVAL", "0.05"),
|
||||||
"VLLM_E2_MEMSTATS": lambda: os.getenv("VLLM_E2_MEMSTATS", "0"),
|
"VLLM_E2_MEMSTATS": lambda: os.getenv("VLLM_E2_MEMSTATS", "0"),
|
||||||
|
|||||||
@@ -40,13 +40,13 @@ MODES -- VLLM_E2_GUARD_MODE:
|
|||||||
serving product than a loud stop. Read check_pending below before
|
serving product than a loud stop. Read check_pending below before
|
||||||
using it -- the raise is late and it is local to one rank.
|
using it -- the raise is late and it is local to one rank.
|
||||||
Targeted repro boots.
|
Targeted repro boots.
|
||||||
warn log only -- EXCEPT for the sites in _ALWAYS_FATAL_SITES, which arm
|
warn (DEFAULT since v2.5) log only -- EXCEPT for the sites in
|
||||||
the raise whatever the mode says, because their masked value has
|
_ALWAYS_FATAL_SITES, which arm the raise whatever the mode says,
|
||||||
already been written somewhere durable. USE THIS FOR LONG STRESS
|
because their masked value has already been written somewhere durable.
|
||||||
BOOTS.
|
USE THIS FOR LONG STRESS BOOTS.
|
||||||
off (DEFAULT since v2: the reader thread
|
off (the default in v2 through v2.2.0: the reader thread and the QSA clip
|
||||||
and the QSA clip reader cost ~0.3 ms per decode step on TP2xPP2; opt in
|
reader cost ~0.3 ms per decode step on TP2xPP2; set
|
||||||
with VLLM_E2_GUARD_MODE=warn for diagnosis)
|
VLLM_E2_GUARD_MODE=off to skip the reader)
|
||||||
no reader thread. The kernels still mask and still count; nothing is
|
no reader thread. The kernels still mask and still count; nothing is
|
||||||
read -- INCLUDING the sites in _ALWAYS_FATAL_SITES. Use only when you
|
read -- INCLUDING the sites in _ALWAYS_FATAL_SITES. Use only when you
|
||||||
have decided you do not want the evidence.
|
have decided you do not want the evidence.
|
||||||
@@ -218,10 +218,12 @@ SITE_NAMES = {
|
|||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
# Release defaults: off, 0.05 s.
|
# Release defaults: warn, 0.05 s.
|
||||||
# Earlier revisions shipped raise / 0.5 while every qualification boot set both
|
# Earlier revisions shipped raise / 0.5 while every qualification boot set both
|
||||||
# variables explicitly, so the defaults were never the thing being tested.
|
# variables explicitly, so the defaults were never the thing being tested.
|
||||||
_MODE = os.environ.get("VLLM_E2_GUARD_MODE", "off").strip().lower()
|
# v2 through v2.2.0 shipped the code default off; v2.5 puts
|
||||||
|
# it back to warn so the code default, this comment and the shipped launchers agree.
|
||||||
|
_MODE = os.environ.get("VLLM_E2_GUARD_MODE", "warn").strip().lower()
|
||||||
if _MODE not in ("raise", "warn", "off"):
|
if _MODE not in ("raise", "warn", "off"):
|
||||||
# fail closed: a set-but-broken gate must not silently disarm the instrument
|
# fail closed: a set-but-broken gate must not silently disarm the instrument
|
||||||
raise ValueError(f"VLLM_E2_GUARD_MODE must be raise|warn|off, got {_MODE!r}")
|
raise ValueError(f"VLLM_E2_GUARD_MODE must be raise|warn|off, got {_MODE!r}")
|
||||||
@@ -491,7 +493,7 @@ def start_reader() -> None:
|
|||||||
host, and a device-to-host copy from a second thread while a capture is
|
host, and a device-to-host copy from a second thread while a capture is
|
||||||
active is exactly the call class cudaStreamCaptureModeGlobal -- the default
|
active is exactly the call class cudaStreamCaptureModeGlobal -- the default
|
||||||
for torch.cuda.graph -- rejects. The QSA clip-counter reader survives it in
|
for torch.cuda.graph -- rejects. The QSA clip-counter reader survives it in
|
||||||
practice, but that one is maintainer-gated and this one can be enabled by any user, so
|
practice, but that one is opt-in and this one is on by default, so
|
||||||
the precedent is evidence, not proof. Starting after capture deletes the
|
the precedent is evidence, not proof. Starting after capture deletes the
|
||||||
question rather than arguing it.
|
question rather than arguing it.
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user