iHore/vLLM
8
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity
23,341 Commits 3 Branches 0 Tags
49d708e158f7b7cfbb00c26f088cd9139c0cf180
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Michael Goin aa3b3d76e0 Enforce valid max_num_batched_tokens when disable_chunked_mm_input=True (#16447) 
Signed-off-by: mgoin <mgoin64@gmail.com>
 2025-04-11 08:09:52 +00:00
Michael Goin ed37599544 Update supported_hardware.md for TPU INT8 (#16437) 2025-04-11 12:28:07 +08:00
Michael Goin c70cf0fe06 [Kernel] Use moe_wna16 kernel for compressed tensors wna16 moe models (#16038) 
Signed-off-by: mgoin <mgoin64@gmail.com>
 2025-04-10 15:08:47 +08:00
Michael Goin baada0e737 [Bugfix][TPU] Fix TPU validate_request (#16369) 
Signed-off-by: Michael Goin <mgoin64@gmail.com>
 2025-04-10 12:55:12 +08:00
Michael Goin b2ce859bd2 Fix benchmark_throughput.py --backend=hf (#16352) 
Signed-off-by: mgoin <mgoin64@gmail.com>
 2025-04-09 19:09:28 +00:00
Michael Goin 098900d7c2 Revert "Update label-tpu mergify and remove removal bot" (#16350) 2025-04-09 07:59:36 -07:00
Michael Goin 477d2a8aa2 Update label-tpu mergify and remove removal bot (#16298) 2025-04-09 07:56:25 +00:00
Michael Goin 87b4ac56c2 [CI][Bugfix] Fix bad tolerance for test_batch_base64_embedding (#16221) 
Signed-off-by: mgoin <mgoin64@gmail.com>
 2025-04-09 04:14:46 +00:00
Michael Goin 8e5314a468 [V1] Add disable_chunked_mm_input arg to disable partial mm input prefill (#15837) 
Signed-off-by: mgoin <mgoin64@gmail.com>
 2025-04-07 23:24:07 -07:00
Michael Goin b99733d092 [Bugfix] Do not skip "empty" parts of chats that are parsable (#16219) 
Signed-off-by: mgoin <mgoin64@gmail.com>
 2025-04-08 05:14:15 +00:00
Michael Goin 21802c4b6d [ROCm][Bugfix][FP8] Make fp8 quant respect fused modules mapping (#16031) 
Signed-off-by: mgoin <michael@neuralmagic.com>
 2025-04-07 21:28:14 -04:00
Michael Goin 70ad3f9e98 [Bugfix][TPU] Fix V1 TPU worker for sliding window (#16059) 
Signed-off-by: Michael Goin <mgoin64@gmail.com>
 2025-04-04 23:31:19 +00:00
Michael Goin 4708f13a9c [Bugfix] Fix default behavior/fallback for pp in v1 (#16057) 
Signed-off-by: mgoin <mgoin64@gmail.com>
 2025-04-04 17:58:08 +00:00
Michael Goin f021b97993 [V1] Support Mistral3 in V1 (#15950) 
Signed-off-by: mgoin <mgoin64@gmail.com>
 2025-04-02 15:36:24 -07:00
Michael GoinDarkLight1337Cyrus Leung
 		51d7c6a2b2 [Model] Support Mistral3 in the HF Transformers format (#15505) 
Signed-off-by: mgoin <mgoin64@gmail.com>
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Co-authored-by: DarkLight1337 <tlleungac@connect.ust.hk>
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>
 2025-04-01 06:10:05 -07:00
Michael Goin 47e9038d23 Fix cpu offload testing for gptq/awq/ct (#15648) 
Signed-off-by: mgoin <mgoin64@gmail.com>
 2025-03-29 00:29:32 +08:00
Michael Goin ce78f9af4e Add automatic tpu label to mergify.yml (#15560) 2025-03-26 21:39:58 -04:00
Michael GoinLuka GovedičDarkLight1337Tyler Michael SmithHarry Mellor
 		14f301b541 Update to torch==2.6.0 (#12721) 
Signed-off-by: mgoin <michael@neuralmagic.com>
Signed-off-by: mgoin <mgoin64@gmail.com>
Signed-off-by: luka <luka@neuralmagic.com>
Signed-off-by: Tyler Michael Smith <tyler@neuralmagic.com>
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Luka Govedič <ProExpertProg@users.noreply.github.com>
Co-authored-by: DarkLight1337 <tlleungac@connect.ust.hk>
Co-authored-by: Tyler Michael Smith <tyler@neuralmagic.com>
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2025-03-14 16:58:30 -04:00
Michael Goin 04222984f8 [Docs] Add nsight guide to profiling docs (#14298) 
Signed-off-by: mgoin <mgoin64@gmail.com>
 2025-03-06 14:19:58 -08:00
Michael Goin 6832707e90 [V1][Bugfix] Standardize quantized kv cache rejection for attention backends (#14221) 
Signed-off-by: mgoin <mgoin64@gmail.com>
 2025-03-06 14:18:29 -08:00
Michael Goin 6b2ef5cd17 [Bug] Fix Attention when ignored in by quant_method (#14313) 
Signed-off-by: mgoin <mgoin64@gmail.com>
 2025-03-06 14:18:06 -08:00
Michael Goin d9292786e1 [CI/Build] Use uv python for docker rather than ppa:deadsnakes/ppa (#13569) 
Signed-off-by: mgoin <mgoin64@gmail.com>
 2025-03-06 16:08:36 -05:00
Michael Goin ca100c90fe Add benchmark for DeepGEMM and vLLM Block FP8 Dense GEMM (#13917) 
Signed-off-by: mgoin <mgoin64@gmail.com>
 2025-03-05 17:08:51 -08:00
Michael Goin dae9ec464c Temporarily disable test_awq_gemm_opcheck (#14251) 
Signed-off-by: mgoin <mgoin64@gmail.com>
 2025-03-05 06:10:35 +00:00
Michael Goin e123aafdf0 Disable GPTQ AllSpark kernels for CUDA Compiler < 12.0 (#14157) 
Signed-off-by: mgoin <mgoin64@gmail.com>
 2025-03-05 12:25:24 +08:00
Michael Goin fbfc3ee37e [V1][TPU] TPU multimodal model support for ragged attention (#14158) 
Signed-off-by: Michael Goin <mgoin64@gmail.com>
 2025-03-04 19:58:48 -05:00
Michael Goin 550c7ba3dc [Docs] Update Dockerfile dependency image (#14215) 
Signed-off-by: mgoin <mgoin64@gmail.com>
 2025-03-04 20:22:11 +00:00
Michael Goin 6247bae6c6 [Bugfix] Restrict MacOS CPU detection (#14210) 
Signed-off-by: mgoin <mgoin64@gmail.com>
 2025-03-04 22:25:27 +08:00
Michael Goin f78c0be80a Fix benchmark_moe.py tuning for CUDA devices (#14164) 2025-03-03 21:11:03 -08:00
Michael Goin 19d98e0c7d [Kernel] Optimize moe intermediate_cache usage (#13625) 
Signed-off-by: mgoin <mgoin64@gmail.com>
 2025-03-03 16:29:53 -05:00
Michael Goin 2b04c209ee [Bugfix] Allow shared_experts skip quantization for DeepSeekV2/V3 (#14100) 
Signed-off-by: mgoin <mgoin64@gmail.com>
 2025-03-03 14:20:24 -07:00
Michael Goin 788f284b53 Fix test_block_fp8.py test for MoE (#13915) 
Signed-off-by: mgoin <mgoin64@gmail.com>
 2025-02-27 18:00:00 +08:00
Michael Goin ca377cf1b9 Use CUDA 12.4 as default for release and nightly wheels (#12098) 2025-02-26 19:06:37 -08:00
Michael Goin 07c4353057 [Model] Support Grok1 (#13795) 
Signed-off-by: mgoin <mgoin64@gmail.com>
 2025-02-26 01:07:12 +00:00
Michael Goin 4d251ad00e Fix CompressedTensorsWNA16MoE with grouped scales (#13769) 2025-02-25 00:17:14 -08:00
Michael Goin 18e505930d [Bugfix] Support MLA for CompressedTensorsWNA16 (#13725) 
Signed-off-by: mgoin <mgoin64@gmail.com>
 2025-02-25 06:10:31 +00:00
Michael Goin db986c19ea Fix precommit fail in fused_moe intermediate_cache2 chunking (#13772) 
Signed-off-by: mgoin <mgoin64@gmail.com>
 2025-02-24 09:25:47 -08:00
Michael Goin 71face8540 [Bugfix] Fix max_num_batched_tokens for MLA (#13620) 
Signed-off-by: mgoin <mgoin64@gmail.com>
 2025-02-20 17:45:20 -08:00
Michael Goin 497bc83124 [CI/Build] Use uv in the Dockerfile (#13566) 2025-02-19 23:05:44 -08:00
Michael Goin b53d79983c Add outlines fallback when JSON schema has enum (#13449) 
Signed-off-by: mgoin <mgoin64@gmail.com>
 2025-02-18 06:49:41 +00:00
Michael Goin d3d547e057 [Bugfix] Pin xgrammar to 0.1.11 (#13338) 2025-02-15 19:42:25 -08:00
Michael Goin 5e5c8e091e [Quant][Perf] Use moe_wna16 kernel by default for MoEs with many experts (#13236) 
Signed-off-by: mgoin <mgoin64@gmail.com>
 2025-02-14 12:53:42 -08:00
Michael Goin f0b2da72a8 Expand MLA to support most types of quantization (#13181) 2025-02-13 22:19:22 -08:00
Michael Goin 2344192a55 Optimize moe_align_block_size for deepseek_v3 (#12850) 
Signed-off-by: mgoin <mgoin64@gmail.com>
 2025-02-13 18:43:37 -05:00
Michael Goin d88c8666a1 [Bugfix][Example] Fix GCed profiling server for TPU (#12792) 
Signed-off-by: mgoin <michael@neuralmagic.com>
 2025-02-13 11:52:11 +08:00
Michael Goin 14b7899d10 [CI] Fix failing FP8 cpu offload test (#13170) 
Signed-off-by: mgoin <mgoin64@gmail.com>
 2025-02-12 19:16:06 +00:00
Michael Goin 09972e716c [Bugfix] Allow fallback to AWQ from AWQMarlin at per-layer granularity (#13119) 2025-02-12 09:19:53 -08:00
Michael Goin 449d1bce02 [Misc] Remove duplicated DeepSeek V2/V3 model definition (#12793) 2025-02-05 23:16:20 -08:00
Michael Goin c53dc466b1 [Doc] Remove performance warning for auto_awq.md (#12743) 2025-02-04 22:43:11 -08:00
Michael Goin 58b218d7ae [Doc] Update PR Reminder with link to Developer Slack (#12748) 2025-02-04 22:42:09 -08:00
Michael Goin 5d98d56089 Support Pixtral-Large HF by using llava multimodal_projector_bias config (#12710) 
Signed-off-by: mgoin <michael@neuralmagic.com>
 2025-02-04 11:55:46 +08:00
Michael Goin 3194039c0e Apply torch.compile to fused_moe/grouped_topk (#12637) 2025-02-01 16:16:19 +00:00
Michael Goin 4078052f09 [V1][Log] Add max request concurrency log to V1 (#12569) 
Signed-off-by: mgoin <michael@neuralmagic.com>
 2025-01-30 23:07:19 +00:00
Michael Goin bd02164cf9 Bugfix for whisper quantization due to fake k_proj bias (#12524) 
Signed-off-by: mgoin <michael@neuralmagic.com>
 2025-01-29 04:49:03 +00:00
Michael Goin fbb5bd4cef [TPU] Add example for profiling TPU inference (#12531) 
Signed-off-by: mgoin <mgoin@redhat.com>
 2025-01-29 03:16:47 +00:00
Michael Goin 0f657bdc52 Replace missed warning_once for rerank API (#12472) 
Signed-off-by: mgoin <michael@neuralmagic.com>
 2025-01-28 19:06:32 +00:00
Michael Goin 426a5c3625 Fix bad path in prometheus example (#12481) 
Signed-off-by: mgoin <michael@neuralmagic.com>
 2025-01-27 18:56:31 -07:00
Michael GoinandCyrus Leung 01a55941f5 [Docs] Update FP8 KV Cache documentation (#12238) 
Signed-off-by: mgoin <michael@neuralmagic.com>
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>
 2025-01-23 11:18:09 +08:00
Michael Goin ecf67814f1 Add quantization and guided decoding CODEOWNERS (#12228) 
Signed-off-by: mgoin <michael@neuralmagic.com>
 2025-01-20 18:23:40 -07:00
Michael Goin 9aa1519f08 Various cosmetic/comment fixes (#12089) 
Signed-off-by: mgoin <michael@neuralmagic.com>
 2025-01-16 09:59:06 +00:00
Michael Goin 730e9592e9 [Doc] Recommend uv and python 3.12 for quickstart guide (#11849) 
Signed-off-by: mgoin <michael@neuralmagic.com>
 2025-01-09 11:37:48 +08:00
Michael Goin bf0d97d786 Update requirements-tpu.txt to support python 3.9 and 3.11 (#11695) 
Signed-off-by: mgoin <michael@neuralmagic.com>
 2025-01-03 22:36:46 +00:00
Michael Goin 74fa1d123c [Bugfix] Fix OpenAI parallel sampling when using xgrammar (#11637) 
Signed-off-by: mgoin <michael@neuralmagic.com>
 2024-12-31 03:43:54 +00:00
Michael Goin 0aa38d16f5 Remove print statement in DeepseekScalingRotaryEmbedding (#11604) 2024-12-29 20:16:46 +00:00
Michael Goinsimon-mosimon-moHandH1998
 		2072924d14 [Model] [Quantization] Support deepseek_v3 w8a8 fp8 block-wise quantization (#11523) 
Signed-off-by: mgoin <michael@neuralmagic.com>
Signed-off-by: simon-mo <simon.mo@hey.com>
Signed-off-by: simon-mo <xmo@berkeley.edu>
Co-authored-by: simon-mo <simon.mo@hey.com>
Co-authored-by: simon-mo <xmo@berkeley.edu>
Co-authored-by: HandH1998 <1335248067@qq.com>
 2024-12-26 15:33:30 -08:00
Michael Goin 60fb4f3bcf [Bugfix] Add kv cache scales to gemma2.py (#11269) 2024-12-23 19:30:45 +00:00
Michael Goin 63afbe9215 [CI] Expand OpenAI test_chat.py guided decoding tests (#11048) 
Signed-off-by: mgoin <michael@neuralmagic.com>
 2024-12-23 18:35:38 +00:00
Michael Goin 5bfb30a529 [Bugfix] Fix CFGGuide and use outlines for grammars that can't convert to GBNF (#11389) 
Signed-off-by: mgoin <michael@neuralmagic.com>
 2024-12-23 23:06:20 +08:00
Michael Goin d573aeadcc [Bugfix] Don't log OpenAI field aliases as ignored (#11378) 
Signed-off-by: mgoin <michael@neuralmagic.com>
 2024-12-20 19:03:50 +00:00
Michael Goin b880ffb87e [Misc] Add tqdm progress bar during graph capture (#11349) 
Signed-off-by: mgoin <michael@neuralmagic.com>
 2024-12-20 04:35:18 +00:00
Michael Goin a30482f054 [CI] Expand test_guided_generate to test all backends (#11313) 
Signed-off-by: mgoin <michael@neuralmagic.com>
 2024-12-19 04:00:38 +00:00
Michael Goin c77eb8a33c [Bugfix] Set temperature=0.7 in test_guided_choice_chat (#11264) 2024-12-17 16:34:06 -08:00
Michael Goin 0064f697d3 [CI] Add test case with JSON schema using references + use xgrammar by default with OpenAI parse (#10935) 
Signed-off-by: mgoin <michael@neuralmagic.com>
 2024-12-17 11:39:58 +08:00
Michael GoinandTyler Michael Smith 6d525288c1 [Docs] Add dedicated tool calling page to docs (#10554) 
Signed-off-by: mgoin <michael@neuralmagic.com>
Co-authored-by: Tyler Michael Smith <tyler@neuralmagic.com>
 2024-12-09 20:15:34 -05:00
Michael Goin 8b59631855 [Core] Support Lark grammars for XGrammar (#10870) 
Signed-off-by: mgoin <michael@neuralmagic.com>
 2024-12-06 08:34:29 -07:00
Michael Goin 8d370e91cb [Bugfix] Fallback to outlines for complex json schemas (#10899) 
Signed-off-by: mgoin <michael@neuralmagic.com>
 2024-12-05 11:14:06 +08:00
Michael Goin 7090c27bb2 [Bugfix] Only require XGrammar on x86 (#10865) 
Signed-off-by: mgoin <michael@neuralmagic.com>
 2024-12-03 10:32:21 -08:00
Michael Goin 4433195ab7 [Bugfix] Prevent benchmark_throughput.py from using duplicated random prompts (#10753) 2024-12-03 02:26:15 +00:00
Michael Goin 7576cd38df [Bugfix] Check bnb_4bit_quant_storage for bitsandbytes (#10642) 2024-11-26 12:29:00 -08:00
Michael Goin 9a99273b48 [Bugfix] Fix using -O[0,3] with LLM entrypoint (#10677) 
Signed-off-by: mgoin <michael@neuralmagic.com>
 2024-11-26 10:44:01 -08:00
Michael Goin 02a43f82a9 Update default max_num_batch_tokens for chunked prefill to 2048 (#10544) 2024-11-22 21:14:19 -08:00
Michael Goin 9afa014552 Add small example to metrics.rst (#10550) 2024-11-21 23:43:43 +00:00
Michael Goin 74f8c2cf5f Add openai.beta.chat.completions.parse example to structured_outputs.rst (#10433) 2024-11-19 04:37:46 +00:00
Michael Goin 281cc4b3cd [Model][Bugfix] Support TP for PixtralHF ViT (#10405) 
Signed-off-by: mgoin <michael@neuralmagic.com>
 2024-11-18 10:04:14 -08:00
Michael Goin f83feccd7f [Bugfix] Ignore GPTQ quantization of Qwen2-VL visual module (#10169) 
Signed-off-by: mgoin <michael@neuralmagic.com>
 2024-11-09 03:36:46 +00:00
Michael Goin aea6ad629f Add hf_transfer to testing image (#10096) 2024-11-08 08:35:25 +00:00
Michael Goin 4ab3256644 [Bugfix] Fix FP8 torch._scaled_mm fallback for torch>2.5 with CUDA<12.4 (#10095) 
Signed-off-by: mgoin <michael@neuralmagic.com>
 2024-11-07 00:54:13 +00:00
Michael Goin 399c798608 Remove ScaledActivation for AWQ (#10057) 
Signed-off-by: mgoin <michael@neuralmagic.com>
 2024-11-06 14:27:06 +00:00
Michael Goin 235366fe2e [CI] Prune back the number of tests in tests/kernels/* (#9932) 
Signed-off-by: mgoin <michael@neuralmagic.com>
 2024-11-05 16:02:32 -05:00
Michael Goin 02462465ea [CI] Prune tests/models/decoder_only/language/* tests (#9940) 
Signed-off-by: mgoin <michael@neuralmagic.com>
 2024-11-05 16:02:23 -05:00
Michael Goin a53046b16f [Model] Support quantization of PixtralHFTransformer for PixtralHF (#9921) 
Signed-off-by: mgoin <michael@neuralmagic.com>
 2024-11-05 10:42:20 -08:00
Michael Goin 8f0a9ca890 [Bugfix] Respect modules_to_not_convert within awq_marlin (#9895) 
Signed-off-by: mgoin <michael@neuralmagic.com>
 2024-11-04 16:57:44 -07:00
Michael Goin 30a2e80742 [CI/Build] Add Model Tests for PixtralHF (#9813) 2024-11-01 07:55:29 -06:00
Michael Goin 93a76dd21d [Model] Support bitsandbytes for MiniCPMV (#9891) 
Signed-off-by: mgoin <michael@neuralmagic.com>
 2024-11-01 13:31:56 +08:00
Michael Goin 37a4947dcd [Bugfix] Fix layer skip logic with bitsandbytes (#9887) 
Signed-off-by: mgoin <michael@neuralmagic.com>
 2024-11-01 13:12:44 +08:00
Michael Goin d087bf863e [Model] Support quantization of Qwen2VisionTransformer (#9817) 
Signed-off-by: mgoin <michael@neuralmagic.com>
 2024-10-30 22:41:20 -07:00
Michael Goin 226688bd61 [Bugfix][VLM] Make apply_fp8_linear work with >2D input (#9812) 2024-10-29 19:49:44 -07:00
Michael Goin bc73e9821c [Bugfix] Fix prefix strings for quantized VLMs (#9772) 2024-10-29 16:02:59 -07:00
Michael Goin ab6f981671 [CI][Bugfix] Skip chameleon for transformers 4.46.1 (#9808) 2024-10-29 11:12:43 -07:00
Michael Goin ca0d92227e [Bugfix] Fix compressed_tensors_moe bad config.strategy (#9677) 2024-10-25 12:40:33 -07:00