iHore/vLLM
8
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity
14,386 Commits 3 Branches 0 Tags
fa6a6be51978bd4b49ba0da17039e60f96dc5b13
Commit Graph
74 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Ning XieChaunceyRussell Bryant
 		13397841ab [structured output] validate unsupported json features first (#33233) 
Signed-off-by: Andy Xie <andy.xning@gmail.com>
Co-authored-by: Chauncey <chaunceyjiang@gmail.com>
Co-authored-by: Russell Bryant <rbryant@redhat.com>
 2026-02-09 23:49:09 +00:00
Ning Xie b781eeaa15 [code clean] remove duplicate code (#33135) 
Signed-off-by: Andy Xie <andy.xning@gmail.com>
 2026-01-27 04:57:16 +00:00
Ning Xie 9d7ae3fcdb [code clean] remove duplicate check (#32376) 
Signed-off-by: Andy Xie <andy.xning@gmail.com>
 2026-01-15 05:29:34 +00:00
Ning Xie 552b262936 rename tokenize serving api request id prefix to tokenize (#32328) 
Signed-off-by: Andy Xie <andy.xning@gmail.com>
 2026-01-14 14:52:20 +00:00
Ning Xie d74132ca3b fix offline inference chat response prompt (#32088) 
Signed-off-by: Andy Xie <andy.xning@gmail.com>
 2026-01-11 14:01:18 +00:00
Ning Xie 14fc7a68c7 [Bugfix] fix offline chat output prompt (#32076) 
Signed-off-by: Andy Xie <andy.xning@gmail.com>
 2026-01-10 07:50:57 +00:00
Ning Xie abd9224280 resolve pydantic error in startup benchmark (#31348) 
Signed-off-by: Andy Xie <andy.xning@gmail.com>
 2026-01-10 02:41:27 +00:00
Ning Xie c907d22158 [refactor] refactor memory constants usage (#31865) 
Signed-off-by: Andy Xie <andy.xning@gmail.com>
 2026-01-07 18:37:31 +00:00
Ning Xie 6f5e653383 [Log] add log about gpu worker init snapshot and requested memory (#29493) 
Signed-off-by: Andy Xie <andy.xning@gmail.com>
 2026-01-06 17:32:55 +00:00
Ning Xie 98b8b3abaa [log] enable max_log_len trim only when needed (#31482) 
Signed-off-by: Andy Xie <andy.xning@gmail.com>
 2026-01-05 03:55:43 +00:00
Ning Xie 6522721d17 [misc] Sort uvicorn log level description according to verbosity (#31137) 
Signed-off-by: Andy Xie <andy.xning@gmail.com>
 2026-01-04 18:45:37 +00:00
Ning Xie 3b8f31b362 [benchmark] use model card root instead of id (#31329) 
Signed-off-by: Andy Xie <andy.xning@gmail.com>
 2025-12-26 10:55:56 +08:00
Ning Xie 5d93089686 [cli] complete vllm cli help message (#31226) 
Signed-off-by: Andy Xie <andy.xning@gmail.com>
 2025-12-24 15:45:47 +00:00
Ning Xie d02d1043de fix: enhance human_readable_int function (#30337) 
Signed-off-by: Andy Xie <andy.xning@gmail.com>
 2025-12-10 23:30:33 -08:00
Ning Xie 7ae13c66ba [typing] fix type (#29964) 
Signed-off-by: Andy Xie <andy.xning@gmail.com>
 2025-12-05 10:46:08 +00:00
Ning Xie 53a1ba6ec5 [log] add weights loading time log to sharded_state loader (#28628) 
Signed-off-by: Andy Xie <andy.xning@gmail.com>
 2025-11-21 21:06:09 +00:00
Ning Xie 0168f69e50 [Misc] Remove unnecessary parentheses from log statements (#28897) 
Signed-off-by: Andy Xie <andy.xning@gmail.com>
 2025-11-17 20:33:46 -08:00
Ning Xie ac1daf3233 fix comment typo (#28802) 
Signed-off-by: Andy Xie <andy.xning@gmail.com>
 2025-11-16 17:03:21 +00:00
Ning Xie e5e9067e61 [Misc] fix typo and add detailed log (#28178) 
Signed-off-by: Andy Xie <andy.xning@gmail.com>
 2025-11-09 05:33:46 +00:00
Ning Xie 786030721e [Docs] add runai_streamer_sharded to LoadConfig (#27937) 
Signed-off-by: Andy Xie <andy.xning@gmail.com>
 2025-11-03 20:35:16 +00:00
Ning Xie bc0f6059a2 [UT] enhance free kv cache block queue popleft_n (#24220) 
Signed-off-by: Andy Xie <andy.xning@gmail.com>
 2025-09-15 10:04:37 +00:00
Ning Xie 59e17dd4a0 [Misc] rename interval to max_recent_requests (#24229) 
Signed-off-by: Andy Xie <andy.xning@gmail.com>
 2025-09-15 09:18:42 +00:00
Ning Xie 3f3313981c [kv cache] update num_free_blocks in the end (#24228) 
Signed-off-by: Andy Xie <andy.xning@gmail.com>
 2025-09-15 05:15:12 +00:00
Ning Xie 5438967fbc [Misc] add hash_function doc string (#24014) 
Signed-off-by: Andy Xie <andy.xning@gmail.com>
 2025-08-31 23:11:20 -07:00
Ning Xie 499b074bfd [Misc] refactor code by import as for torch._inductor.config (#23677) 
Signed-off-by: Andy Xie <andy.xning@gmail.com>
 2025-09-01 14:05:42 +08:00
Ning Xie 68a349114f [Misc] enhance type hint for rearrange return value (#23519) 
Signed-off-by: Andy Xie <andy.xning@gmail.com>
 2025-08-30 06:43:33 -07:00
Ning Xie e80bca309e [Refactor] refactor freezing_value/cuda_event initialize outside try finally (#23758) 
Signed-off-by: Andy Xie <andy.xning@gmail.com>
 2025-08-30 06:42:25 -07:00
Ning Xie fb4983e112 [Misc] add reorder_batch AttentionMetadataBuilder (#23798) 
Signed-off-by: Andy Xie <andy.xning@gmail.com>
 2025-08-30 06:41:45 -07:00
Ning Xie 5490d633ce [UT] fix unify_kv_cache_configs when kv cache config needs sort (#23843) 2025-08-30 11:22:14 +00:00
Ning Xie 325aa3dee9 [Misc] local import code clean (#23420) 
Signed-off-by: Andy Xie <andy.xning@gmail.com>
 2025-08-22 14:01:35 +00:00
Ning Xie 3496274663 [Misc] Convert VLLM_TORCH_PROFILER_DIR path to absolute (#23191) 
Signed-off-by: Andy Xie <andy.xning@gmail.com>
 2025-08-21 15:49:09 -04:00
Ning Xie 5a30bd10d8 [Bugfix] fix IntermediateTensors equal method (#23027) 
Signed-off-by: Andy Xie <andy.xning@gmail.com>
 2025-08-18 02:58:11 -07:00
Ning Xie 08d5f7113a [Misc] refactor function name (#23029) 
Signed-off-by: Andy Xie <andy.xning@gmail.com>
 2025-08-17 22:16:21 -07:00
Ning Xie 7be3a59d8e [Misc] enhance static type hint (#23059) 
Signed-off-by: Andy Xie <andy.xning@gmail.com>
 2025-08-17 22:09:08 -07:00
Ning Xie 87f48623a5 [Misc] method name typo fix (#23042) 
Signed-off-by: Andy Xie <andy.xning@gmail.com>
 2025-08-16 21:49:14 -07:00
Ning Xie 326976291b [Misc] code clean duplicate set_current_vllm_config in _set_vllm_config (#22566) 
Signed-off-by: Andy Xie <andy.xning@gmail.com>
 2025-08-10 00:08:48 -07:00
Ning Xie acf8aeb79e [Misc] normalize multiprocessing Queue usage (#22371) 
Signed-off-by: Andy Xie <andy.xning@gmail.com>
 2025-08-08 01:57:27 +00:00
Ning Xie 74333ae2f6 [Misc] correct static type check for GroupCoordinator (#21946) 
Signed-off-by: Andy Xie <andy.xning@gmail.com>
 2025-08-05 03:17:46 -07:00
Ning Xie bd3db7f469 [Misc] log more detailed message for ensure_model_parallel_initialized (#22144) 
Signed-off-by: Andy Xie <andy.xning@gmail.com>
 2025-08-04 19:36:55 -07:00
Ning Xie 29b97c0995 [Doc] add backend to doc string of initialize_model_parallel (#22142) 
Signed-off-by: Andy Xie <andy.xning@gmail.com>
 2025-08-04 19:36:20 -07:00
Ning Xie c2e75b3c11 remove duplicate code within cleanup_dist_env_and_memory (#22147) 
Signed-off-by: Andy Xie <andy.xning@gmail.com>
 2025-08-03 20:03:58 -07:00
Ning Xie 7de45db9a5 [Misc] update doc comment for send (#22026) 
Signed-off-by: Andy Xie <andy.xning@gmail.com>
 2025-08-03 00:55:20 -07:00
Ning Xie 3e36fcbee6 [Bugfix]: fix metadata file copy in test_sharded_state_loader (#21830) 
Signed-off-by: Andy Xie <andy.xning@gmail.com>
 2025-07-31 06:22:11 +00:00
Ning Xie 150d9e6337 [Bugfix] fix max-file-size type from str to int (#21675) 
Signed-off-by: Andy Xie <andy.xning@gmail.com>
 2025-07-28 00:06:52 -07:00
Ning Xie a9b2a1d704 [Misc] Refactor vllm config str (#21666) 2025-07-27 09:51:44 -07:00
Ning Xie 1cbf951ba2 [Misc] add default value for file pattern arg (#21659) 
Signed-off-by: Andy Xie <andy.xning@gmail.com>
 2025-07-27 05:14:51 +00:00
Ning Xie adaf2c6d4f [Bugfix] fix modelscope snapshot_download serialization (#21536) 
Signed-off-by: Andy Xie <andy.xning@gmail.com>
 2025-07-24 22:44:38 -07:00
Ning Xie 0df4d9b06b [Misc] unify variable for LLM instance v2 (#21356) 
Signed-off-by: Andy Xie <andy.xning@gmail.com>
 2025-07-22 06:32:36 -07:00
Ning Xie d97841078b [Misc] unify variable for LLM instance (#20996) 
Signed-off-by: Andy Xie <andy.xning@gmail.com>
 2025-07-21 12:18:33 +01:00
Ning Xie 1dba2c4ebe [Misc] adjust for ipv6 for mookcacke url parse (#20107) 
Signed-off-by: Andy Xie <andy.xning@gmail.com>
 2025-07-03 20:27:17 +00:00
Ning Xie fb14d53cf6 [Kernel] refactor cpu worker v0 cache dtype (#20080) 
Signed-off-by: Andy Xie <andy.xning@gmail.com>
 2025-07-03 08:39:14 +00:00
Ning Xie 2bb246b8f7 [MISC] add cpu_kvcache_space_bytes to CacheConfig (#19812) 
Signed-off-by: Andy Xie <andy.xning@gmail.com>
 2025-06-22 13:39:09 +08:00
Ning Xie 4c409cabc2 [Misc] add vllm_config in __init__ (#19866) 
Signed-off-by: Andy Xie <andy.xning@gmail.com>
 2025-06-21 23:10:46 -04:00
Ning Xie 71baf85ae1 [Kernel] mark TorchSDPABackend swap_blocks NotImplementedError (#19749) 2025-06-20 18:18:11 +00:00
Ning Xie 71d1219545 [Kernel] correct cpu worker function parameter type (#19745) 
Signed-off-by: Andy Xie <andy.xning@gmail.com>
 2025-06-20 10:50:13 +00:00
Ning Xie c7b370c603 raise exception for pin_lora (#19809) 
Signed-off-by: Andy Xie <andy.xning@gmail.com>
 2025-06-18 22:57:35 -07:00
Ning Xie 6e9cc73f67 [MISC] correct DeviceConfig device field static type analysis (#19699) 
Signed-off-by: Andy Xie <andy.xning@gmail.com>
 2025-06-17 17:21:50 -07:00
Ning Xie c53711bd63 [MISC] correct copy_blocks src_to_dists param type (#19696) 
Signed-off-by: Andy Xie <andy.xning@gmail.com>
 2025-06-17 17:21:06 -07:00
Ning Xie c3fec47bb7 [MISC] bump huggingface_hub pkg to 0.33.0 (#19547) 
Signed-off-by: Andy Xie <andy.xning@gmail.com>
 2025-06-16 05:22:28 -07:00
Ning Xie 26bc46ef89 [MISC] typo fix (#19672) 
Signed-off-by: Andy Xie <andy.xning@gmail.com>
 2025-06-16 07:18:49 +00:00
Ning Xie 2f1c19b245 [CI] change spell checker from codespell to typos (#18711) 
Signed-off-by: Andy Xie <andy.xning@gmail.com>
 2025-06-11 19:57:10 -07:00
Ning Xie 5a2c76cbe1 [CI] fix dump_input for str type (#18697) 
Signed-off-by: Andy Xie <andy.xning@gmail.com>
 2025-05-26 18:23:35 +08:00
Ning Xie 4ea62c0ea0 [CI] add missing argument (#18694) 
Signed-off-by: Andy Xie <andy.xning@gmail.com>
 2025-05-26 00:22:04 -07:00
Ning Xie 44073a7ac3 [BUGFIX] catch subclass first for try...except (#18672) 
Signed-off-by: Andy Xie <andy.xning@gmail.com>
 2025-05-25 05:34:24 +00:00
Ning Xie 6c6dcd8611 [MISC] correct signature for LoaderFunction (#18670) 
Signed-off-by: Andy Xie <andy.xning@gmail.com>
 2025-05-24 20:17:47 -07:00
Ning Xie 4ceafb6299 [MISC] typo fix and clean import (#18664) 
Signed-off-by: Andy Xie <andy.xning@gmail.com>
 2025-05-24 07:52:09 -07:00
Ning Xie cd821ea5d2 [CI] fix kv_cache_type argument (#18594) 
Signed-off-by: Andy Xie <andy.xning@gmail.com>
 2025-05-23 04:49:18 -07:00
Ning Xie 60cad94b86 [Hardware] correct method signatures for HPU,ROCm,XPU (#18551) 
Signed-off-by: Andy Xie <andy.xning@gmail.com>
 2025-05-22 22:31:59 -07:00
Ning Xie 176d62e4ea [MISC] update project urls in pyproject.toml (#18519) 
Signed-off-by: Andy Xie <andy.xning@gmail.com>
 2025-05-21 20:17:34 -07:00
Ning Xie 66e63e86ec [MISC] fix typo (#18305) 
Signed-off-by: Andy Xie <andy.xning@gmail.com>
 2025-05-17 10:52:09 -07:00
Ning Xie 420caf7557 [UT] Add ut for none hash (#17892) 
Signed-off-by: Andy Xie <andy.xning@gmail.com>
 2025-05-15 13:28:11 +08:00
Ning Xie d310e6de98 [BUGFIX]: return fast when request requires prompt logprobs (#17251) 2025-05-08 21:25:41 -07:00
Ning Xie fd11a325b8 [MISC] rename interval to max_recent_requests (#14285) 2025-04-26 16:59:18 +00:00
Ning Xie dc2ceca5c5 [BUGFIX] use random for NONE_HASH only when PYTHONHASHSEED not set (#17088) 
Signed-off-by: Andy Xie <andy.xning@gmail.com>
 2025-04-26 14:34:24 +00:00