 MOVZX
|
eb8c302bb3
|
fix: Adjust start script to avoid crashing on high pressure KV cache
Stress-tested using 4 concurrent requests with 240K tokens each:
```
(APIServer pid=1) INFO 10-15 06:21:25 [loggers.py:315] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 3.0 tokens/s, Running: 4 reqs, Waiting: 1 reqs, GPU KV cache usage: 98.8%, Prefix cache hit rate: 0.0%
(APIServer pid=1) INFO 10-15 06:21:26 [loggers.py:315] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 17.0 tokens/s, Running: 4 reqs, Waiting: 1 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0%
(APIServer pid=1) INFO 10-15 06:21:27 [loggers.py:315] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 8.0 tokens/s, Running: 4 reqs, Waiting: 1 reqs, GPU KV cache usage: 99.6%, Prefix cache hit rate: 0.0%
(APIServer pid=1) INFO 10-15 06:21:28 [loggers.py:315] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.9 tokens/s, Running: 4 reqs, Waiting: 1 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0%
```
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
|
2026-10-15 06:24:25 +00:00 |
|
