[Guards] The bounds-guard telemetry code default is warn
VLLM_E2_GUARD_MODE now defaults to warn in code, matching what the shipped launchers set; v2 through v2.2.0 defaulted to off. The comment above it, the MODES docstring, the vllm/envs.py default and its annotation follow. tests/v1/worker/gpu/test_guard_default_mode.py checks that code, comment and envs.py agree, from the sources, so it runs without CUDA, and fails on the earlier source. Co-Authored-By: Hinata <hinata@idihore.id> Signed-off-by: MOVZX <movzx@yahoo.com>
This commit is contained in:
+2
-2
@@ -405,7 +405,7 @@ if TYPE_CHECKING:
|
||||
# Guard telemetry: mode raise|warn|off and the reader poll interval in
|
||||
# seconds. counter() returns a real tensor in every mode, so the Triton
|
||||
# arguments and the captured graph do not depend on either.
|
||||
VLLM_E2_GUARD_MODE: str = "off"
|
||||
VLLM_E2_GUARD_MODE: str = "warn"
|
||||
VLLM_E2_GUARD_INTERVAL: str = "0.05"
|
||||
# Debug allocator telemetry (poll seconds + output directory) and the
|
||||
# scheduler allocation log path.
|
||||
@@ -2507,7 +2507,7 @@ environment_variables: dict[str, Callable[[], Any]] = {
|
||||
).strip(),
|
||||
"VLLM_QSA_KV_SCALES": lambda: os.getenv("VLLM_QSA_KV_SCALES", "").strip(),
|
||||
"VLLM_E2_GUARD_MODE": lambda: (
|
||||
os.getenv("VLLM_E2_GUARD_MODE", "off").strip().lower()
|
||||
os.getenv("VLLM_E2_GUARD_MODE", "warn").strip().lower()
|
||||
),
|
||||
"VLLM_E2_GUARD_INTERVAL": lambda: os.getenv("VLLM_E2_GUARD_INTERVAL", "0.05"),
|
||||
"VLLM_E2_MEMSTATS": lambda: os.getenv("VLLM_E2_MEMSTATS", "0"),
|
||||
|
||||
@@ -40,13 +40,13 @@ MODES -- VLLM_E2_GUARD_MODE:
|
||||
serving product than a loud stop. Read check_pending below before
|
||||
using it -- the raise is late and it is local to one rank.
|
||||
Targeted repro boots.
|
||||
warn log only -- EXCEPT for the sites in _ALWAYS_FATAL_SITES, which arm
|
||||
the raise whatever the mode says, because their masked value has
|
||||
already been written somewhere durable. USE THIS FOR LONG STRESS
|
||||
BOOTS.
|
||||
off (DEFAULT since v2: the reader thread
|
||||
and the QSA clip reader cost ~0.3 ms per decode step on TP2xPP2; opt in
|
||||
with VLLM_E2_GUARD_MODE=warn for diagnosis)
|
||||
warn (DEFAULT since v2.5) log only -- EXCEPT for the sites in
|
||||
_ALWAYS_FATAL_SITES, which arm the raise whatever the mode says,
|
||||
because their masked value has already been written somewhere durable.
|
||||
USE THIS FOR LONG STRESS BOOTS.
|
||||
off (the default in v2 through v2.2.0: the reader thread and the QSA clip
|
||||
reader cost ~0.3 ms per decode step on TP2xPP2; set
|
||||
VLLM_E2_GUARD_MODE=off to skip the reader)
|
||||
no reader thread. The kernels still mask and still count; nothing is
|
||||
read -- INCLUDING the sites in _ALWAYS_FATAL_SITES. Use only when you
|
||||
have decided you do not want the evidence.
|
||||
@@ -218,10 +218,12 @@ SITE_NAMES = {
|
||||
}
|
||||
|
||||
|
||||
# Release defaults: off, 0.05 s.
|
||||
# Release defaults: warn, 0.05 s.
|
||||
# Earlier revisions shipped raise / 0.5 while every qualification boot set both
|
||||
# variables explicitly, so the defaults were never the thing being tested.
|
||||
_MODE = os.environ.get("VLLM_E2_GUARD_MODE", "off").strip().lower()
|
||||
# v2 through v2.2.0 shipped the code default off; v2.5 puts
|
||||
# it back to warn so the code default, this comment and the shipped launchers agree.
|
||||
_MODE = os.environ.get("VLLM_E2_GUARD_MODE", "warn").strip().lower()
|
||||
if _MODE not in ("raise", "warn", "off"):
|
||||
# fail closed: a set-but-broken gate must not silently disarm the instrument
|
||||
raise ValueError(f"VLLM_E2_GUARD_MODE must be raise|warn|off, got {_MODE!r}")
|
||||
@@ -491,7 +493,7 @@ def start_reader() -> None:
|
||||
host, and a device-to-host copy from a second thread while a capture is
|
||||
active is exactly the call class cudaStreamCaptureModeGlobal -- the default
|
||||
for torch.cuda.graph -- rejects. The QSA clip-counter reader survives it in
|
||||
practice, but that one is maintainer-gated and this one can be enabled by any user, so
|
||||
practice, but that one is opt-in and this one is on by default, so
|
||||
the precedent is evidence, not proof. Starting after capture deletes the
|
||||
question rather than arguing it.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user