[Misc][Tools][Benchmark] Add benchmark_serving supports for llama.cpp. (#18692)

Signed-off-by: Duyi-Wang <duyi.wang@intel.com>
This commit is contained in:
Duyi-Wang
2025-05-29 20:02:08 +08:00
committed by GitHub
parent f8977c233f
commit b169d5f7b6
2 changed files with 6 additions and 1 deletions
+4
View File
@@ -762,6 +762,10 @@ def main(args: argparse.Namespace):
if "temperature" not in sampling_params:
sampling_params["temperature"] = 0.0 # Default to greedy decoding.
if args.backend == "llama.cpp":
# Disable prompt caching in llama.cpp backend
sampling_params["cache_prompt"] = False
# Avoid GC processing "static" data - reduce pause times.
gc.collect()
gc.freeze()