vLLM-4x-RTX-3090

3 Commits 1 Branch 0 Tags

3 Commits

Author	SHA1	Message	Date
MOVZXandHinata	7e80cc67ee	feat: Add pretty vLLM logging scripts ``` [10-15 06:31:36] PP: 0.0 t/s \| TG: 4.0 t/s \| K/V Cache: 61.5% \| Pfx Cache: 0.0% \| Draft Acc: 11.1% [10-15 06:31:37] PP: 0.0 t/s \| TG: 5.0 t/s \| K/V Cache: 61.5% \| Pfx Cache: 0.0% \| Draft Acc: 50.0% [10-15 06:31:38] PP: 0.0 t/s \| TG: 6.0 t/s \| K/V Cache: 60.7% \| Pfx Cache: 0.0% \| Draft Acc: 33.3% [10-15 06:31:39] PP: 0.0 t/s \| TG: 10.0 t/s \| K/V Cache: 61.5% \| Pfx Cache: 0.0% \| Draft Acc: 77.8% [10-15 06:31:40] PP: 0.0 t/s \| TG: 6.0 t/s \| K/V Cache: 63.2% \| Pfx Cache: 0.0% \| Draft Acc: 66.7% [10-15 06:31:41] PP: 0.0 t/s \| TG: 5.0 t/s \| K/V Cache: 62.3% \| Pfx Cache: 0.0% \| Draft Acc: 22.2% [10-15 06:31:42] PP: 0.0 t/s \| TG: 8.0 t/s \| K/V Cache: 63.2% \| Pfx Cache: 0.0% \| Draft Acc: 100.0% [10-15 06:31:43] PP: 0.0 t/s \| TG: 11.0 t/s \| K/V Cache: 64.8% \| Pfx Cache: 0.0% \| Draft Acc: 88.9% ``` Co-Authored-By: Hinata <hinata@idihore.id> Signed-off-by: MOVZX <movzx@yahoo.com>	2026-10-15 06:36:24 +00:00
MOVZX	eb8c302bb3	fix: Adjust start script to avoid crashing on high pressure KV cache Stress-tested using 4 concurrent requests with 240K tokens each: ``` (APIServer pid=1) INFO 10-15 06:21:25 [loggers.py:315] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 3.0 tokens/s, Running: 4 reqs, Waiting: 1 reqs, GPU KV cache usage: 98.8%, Prefix cache hit rate: 0.0% (APIServer pid=1) INFO 10-15 06:21:26 [loggers.py:315] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 17.0 tokens/s, Running: 4 reqs, Waiting: 1 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1) INFO 10-15 06:21:27 [loggers.py:315] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 8.0 tokens/s, Running: 4 reqs, Waiting: 1 reqs, GPU KV cache usage: 99.6%, Prefix cache hit rate: 0.0% (APIServer pid=1) INFO 10-15 06:21:28 [loggers.py:315] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.9 tokens/s, Running: 4 reqs, Waiting: 1 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% ``` Co-Authored-By: Hinata <hinata@idihore.id> Signed-off-by: MOVZX <movzx@yahoo.com>	2026-10-15 06:24:25 +00:00
MOVZX	777e910dd0	Initial commit Signed-off-by: MOVZX <movzx@yahoo.com>	2026-10-15 01:01:52 +07:00

Author

SHA1

Message

Date

MOVZXandHinata

7e80cc67ee

feat: Add pretty vLLM logging scripts

```
[10-15 06:31:36] PP:    0.0 t/s | TG:    4.0 t/s | K/V Cache:  61.5% | Pfx Cache:   0.0% | Draft Acc:  11.1%
[10-15 06:31:37] PP:    0.0 t/s | TG:    5.0 t/s | K/V Cache:  61.5% | Pfx Cache:   0.0% | Draft Acc:  50.0%
[10-15 06:31:38] PP:    0.0 t/s | TG:    6.0 t/s | K/V Cache:  60.7% | Pfx Cache:   0.0% | Draft Acc:  33.3%
[10-15 06:31:39] PP:    0.0 t/s | TG:   10.0 t/s | K/V Cache:  61.5% | Pfx Cache:   0.0% | Draft Acc:  77.8%
[10-15 06:31:40] PP:    0.0 t/s | TG:    6.0 t/s | K/V Cache:  63.2% | Pfx Cache:   0.0% | Draft Acc:  66.7%
[10-15 06:31:41] PP:    0.0 t/s | TG:    5.0 t/s | K/V Cache:  62.3% | Pfx Cache:   0.0% | Draft Acc:  22.2%
[10-15 06:31:42] PP:    0.0 t/s | TG:    8.0 t/s | K/V Cache:  63.2% | Pfx Cache:   0.0% | Draft Acc: 100.0%
[10-15 06:31:43] PP:    0.0 t/s | TG:   11.0 t/s | K/V Cache:  64.8% | Pfx Cache:   0.0% | Draft Acc:  88.9%
```

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>

2026-10-15 06:36:24 +00:00

MOVZX

eb8c302bb3

fix: Adjust start script to avoid crashing on high pressure KV cache

Stress-tested using 4 concurrent requests with 240K tokens each:
```
(APIServer pid=1) INFO 10-15 06:21:25 [loggers.py:315] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 3.0 tokens/s, Running: 4 reqs, Waiting: 1 reqs, GPU KV cache usage: 98.8%, Prefix cache hit rate: 0.0%
(APIServer pid=1) INFO 10-15 06:21:26 [loggers.py:315] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 17.0 tokens/s, Running: 4 reqs, Waiting: 1 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0%
(APIServer pid=1) INFO 10-15 06:21:27 [loggers.py:315] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 8.0 tokens/s, Running: 4 reqs, Waiting: 1 reqs, GPU KV cache usage: 99.6%, Prefix cache hit rate: 0.0%
(APIServer pid=1) INFO 10-15 06:21:28 [loggers.py:315] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.9 tokens/s, Running: 4 reqs, Waiting: 1 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0%
```
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>

2026-10-15 06:24:25 +00:00

MOVZX

777e910dd0

Initial commit

Signed-off-by: MOVZX <movzx@yahoo.com>

2026-10-15 01:01:52 +07:00