iHore/vLLM-4x-RTX-3090
8
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
10 Commits 1 Branch 0 Tags
af239c4229b3fe5c5102c0d5602c77dda83556d0
Commit Graph
4 Commits
Author SHA1 Message Date
MOVZXandHinata 1281bf9c03 fix: Load the FP8 KV scales from the shipped file name 
The default sidecar pointed at qsa_kv_scales_swift.json, a name no
checkpoint on this box carries. The guard in start.sh then stopped every
start with "scales sidecar missing", and a start without the guard would
run FP8 KV at scale 1.0.

Point the default at qsa_kv_scales.json, the file both checkpoints ship,
and rename the sidecar in the start.sh header, both READMEs, and the
Dockerfile comment. Name SCALES in the guard message.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
 2026-10-15 18:52:06 +00:00
MOVZXandHinata 5cc9ee2505 chore: Reset vLLM logging interval back to default 
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
 2026-10-15 06:37:23 +00:00
MOVZX eb8c302bb3 fix: Adjust start script to avoid crashing on high pressure KV cache 
Stress-tested using 4 concurrent requests with 240K tokens each:
```
(APIServer pid=1) INFO 10-15 06:21:25 [loggers.py:315] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 3.0 tokens/s, Running: 4 reqs, Waiting: 1 reqs, GPU KV cache usage: 98.8%, Prefix cache hit rate: 0.0%
(APIServer pid=1) INFO 10-15 06:21:26 [loggers.py:315] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 17.0 tokens/s, Running: 4 reqs, Waiting: 1 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0%
(APIServer pid=1) INFO 10-15 06:21:27 [loggers.py:315] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 8.0 tokens/s, Running: 4 reqs, Waiting: 1 reqs, GPU KV cache usage: 99.6%, Prefix cache hit rate: 0.0%
(APIServer pid=1) INFO 10-15 06:21:28 [loggers.py:315] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.9 tokens/s, Running: 4 reqs, Waiting: 1 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0%
```
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
 2026-10-15 06:24:25 +00:00
MOVZX 777e910dd0 Initial commit 
Signed-off-by: MOVZX <movzx@yahoo.com>
 2026-10-15 01:01:52 +07:00