Commit Graph

Select branches

Hide Pull Requests

main

patches

sm86

Mono Color

• 98356735ac [misc] benchmark_throughput : Add LoRA (#11267) Varun Sundar RabindranathandVarun Sundar Rabindranath 2024-12-19 02:43:16 -05:00
• f26c4aeecb [Misc] Optimize ray worker initialization time (#11275) Rui QiaoandCody Yu 2024-12-18 23:38:02 -08:00
• 8936316d58 [Kernel] Refactor Cutlass c3x (#10049) Varun Sundar RabindranathandVarun Sundar Rabindranath 2024-12-19 02:00:18 -05:00
• 6142ef0ada [VLM] Merged multimodal processor for Qwen2-Audio (#11303) Cyrus Leung 2024-12-19 14:14:17 +08:00
• c6b0a7d3ba [V1] Simplify prefix caching logic by removing num_evictable_computed_blocks (#11310) Chen Zhang 2024-12-18 20:17:12 -08:00
• a30482f054 [CI] Expand test_guided_generate to test all backends (#11313) Michael Goin 2024-12-18 23:00:38 -05:00
• 17ca964273 [Model] IBM Granite 3.1 (#11307) Travis Johnson 2024-12-18 20:27:24 -07:00
• 5a9da2e6e9 [Bugfix][Build/CI] Fix sparse CUTLASS compilation on CUDA [12.0, 12.2) (#11311) Tyler Michael Smith 2024-12-18 21:43:30 -05:00
• fdea8ec167 [V1] VLM - enable processor cache by default (#11305) Alexander Matveev 2024-12-18 18:54:46 -05:00
• ca5f54a9b9 [Bugfix] fix minicpmv test (#11304) Joe Runde 2024-12-18 10:34:26 -08:00
• f954fe0e65 [FIX] update openai version (#11287) Kunshang Ji 2024-12-19 02:17:05 +08:00
• 362cff1eb3 [CI][Misc] Remove Github Action Release Workflow (#11274) Simon Mo 2024-12-18 10:16:53 -08:00
• 996aa70f00 [Bugfix] Fix broken phi3-v mm_processor_kwargs tests (#11263) Isotr0py 2024-12-19 02:16:40 +08:00
• 60508ffda9 [Kernel]: Cutlass 2:4 Sparsity + FP8/Int8 Quant Support (#10995) 5 people 2024-12-18 09:57:16 -05:00
• f04e407e6b [MISC][XPU]update ipex link for CI fix (#11278) Yan Ma 2024-12-18 14:34:23 +08:00
• 8b79f9e107 [Bugfix] Fix guided decoding with tokenizer mode mistral (#11046) Wallas Henrique 2024-12-18 03:34:08 -03:00
• 866fa4550d [Bugfix] Restore support for larger block sizes (#11259) Konrad Zawora 2024-12-18 01:39:07 +01:00
• bf8717ebae [V1] Prefix caching for vision language models (#11187) Cody Yu 2024-12-17 16:37:59 -08:00
• c77eb8a33c [Bugfix] Set temperature=0.7 in test_guided_choice_chat (#11264) Michael Goin 2024-12-17 19:34:06 -05:00
• 2d1b9baa8f [Bugfix] Fix request cancellation without polling (#11190) Joe Runde 2024-12-17 13:26:32 -07:00
• f9ecbb18bf [Misc] Allow passing logits_soft_cap for xformers backend (#11252) Isotr0py 2024-12-17 16:37:04 +08:00
• 02222a0256 [Misc] Kernel Benchmark for RMSNorm (#11241) Roger WangandXiaoyu Zhang 2024-12-16 22:57:02 -08:00
• 2bfdbf2a36 [V1][Core] Use weakref.finalize instead of atexit (#11242) Tyler Michael Smith 2024-12-17 01:11:33 -05:00
• e88db68cf5 [Platform] platform agnostic for EngineArgs initialization (#11225) wangxiyuan 2024-12-17 14:11:06 +08:00
• 59c9b6ebeb [V1][VLM] Proper memory profiling for image language models (#11210) Roger Wangandywang96 2024-12-16 22:10:57 -08:00
• 66d4b16724 [Frontend] Add OpenAI API support for input_audio (#11027) kYLeandDarkLight1337 2024-12-17 00:09:58 -06:00
• 0064f697d3 [CI] Add test case with JSON schema using references + use xgrammar by default with OpenAI parse (#10935) Michael Goin 2024-12-16 22:39:58 -05:00
• 35bae114a8 fix gh200 tests on main (#11246) youkaichao 2024-12-16 17:22:38 -08:00
• 88a412ed3d [torch.compile] fast inductor (#11108) youkaichaoandTyler Michael Smith 2024-12-16 16:15:22 -08:00
• c301616ed2 [ci][tests] add gh200 tests (#11244) youkaichao 2024-12-16 15:53:18 -08:00
• 35ffa682b1 [Docs] hint to enable use of GPU performance counters in profiling tools for multi-node distributed serving (#11235) bk-TurbaAIandMichael Goin 2024-12-16 23:20:39 +01:00
• 551603feff [core] overhaul memory profiling and fix backward compatibility (#10511) youkaichao 2024-12-16 13:32:25 -08:00
• efbce85f4d [misc] Layerwise profile updates (#10242) Varun Sundar RabindranathandVarun Sundar Rabindranath 2024-12-16 13:14:57 -05:00
• 2ca830dbaa [Doc] Reorder vision language examples in alphabet order (#11228) Isotr0py 2024-12-16 19:23:33 +08:00
• d927dbcd88 [Model] Refactor Ultravox to use merged input processor (#11198) Isotr0pyandCyrus Leung 2024-12-16 18:09:53 +08:00
• bddbbcb132 [Model] Support Cohere2ForCausalLM (Cohere R7B) (#11203) Jani Monoses 2024-12-16 11:56:19 +02:00
• b3b1526f03 WIP: [CI/Build] simplify Dockerfile build for ARM64 / GH200 (#11212) cennnanddrikster80 2024-12-16 17:20:49 +08:00
• 17138af7c4 [Bugfix] Fix the default value for temperature in ChatCompletionRequest (#11219) yansh97 2024-12-16 16:15:40 +08:00
• 69ba344de8 [Bugfix] Fix block size validation (#10938) chenqianfzh 2024-12-15 16:38:40 -08:00
• da6f409246 Update deploying_with_k8s.rst (#10922) AlexHe99 2024-12-16 08:33:58 +08:00
• 25ebed2f8c [V1][Minor] Cache np arange to reduce input preparation overhead (#11214) Woosuk Kwon 2024-12-15 13:33:00 -08:00
• d263bd9df7 [Core] Support disaggregated prefill with Mooncake Transfer Engine (#10884) shangmingc 2024-12-16 05:28:18 +08:00
• 38e599d6a8 [Doc] add documentation for disaggregated prefilling (#11197) Kuntai Du 2024-12-15 13:31:16 -06:00
• 96d673e0f8 [Bugfix] Fix error handling of unsupported sliding window (#11213) Cyrus Leung 2024-12-16 01:59:42 +08:00
• b10609e6a1 [Misc] Clean up multi-modal processor (#11207) Cyrus Leung 2024-12-15 14:30:28 +08:00
• a1c02058ba [torch.compile] allow tracking forward time (#11081) youkaichao 2024-12-14 19:45:00 -08:00
• 15859f2357 [[Misc]Upgrade bitsandbytes to the latest version 0.45.0 (#11201) Jee Jee Li 2024-12-15 11:03:06 +08:00
• 886936837c [Performance][Core] Optimize the performance of evictor v1 and v2 by applying a priority queue and lazy deletion (#7209) Sungjae Lee 2024-12-15 04:38:10 +09:00
• 6d917d0eeb Enable mypy checking on V1 code (#11105) Mark McLoughlin 2024-12-14 17:54:04 +00:00
• 93abf23a64 [VLM] Fully dynamic prompt replacement in merged input processor (#11199) Cyrus Leung 2024-12-15 01:52:18 +08:00
• 9c3dadd1c9 [Frontend] Add logits_processors as an extra completion argument (#11150) Brad Hilton 2024-12-14 09:46:42 -07:00
• 3cb5769883 [Misc] Minor improvements to the readability of PunicaWrapperBase (#11200) Jee Jee Li 2024-12-15 00:38:27 +08:00
• ea7bd68d10 [V1][Bugfix] Fix V1 TP trust-remote-code (#11182) Tyler Michael Smith 2024-12-14 03:21:23 -05:00
• 48259264a4 [Core] Update outlines and increase its threadpool size (#11140) Russell Bryant 2024-12-14 02:46:18 -05:00
• 24a3d12b82 update compressed-tensors to latest version (#11183) dhuangnmanddhuangnm 2024-12-13 22:22:44 -05:00
• 9855aea21b [Bugfix][V1] Re-compute an entire block when fully cache hit (#11186) Cody Yu 2024-12-13 17:08:23 -08:00
• 4b5b8a6a3b [V1][Bugfix] Fix EngineCoreProc profile (#11185) Tyler Michael Smith 2024-12-13 20:02:35 -05:00
• 4863e5fba5 [Core] V1: Use multiprocessing by default (#11074) Russell Bryant 2024-12-13 19:27:32 -05:00
• 0d8451c3a4 [Distributed] Allow the placement group more time to wait for resources to be ready (#11138) Jiaxin Shan 2024-12-13 12:17:37 -08:00
• 0a56bcc03d [Bugfix][Hardware][CPU] Enable Gemma2 with SDPA on CPU backend (#11169) Jani Monoses 2024-12-13 20:00:40 +02:00
• 0920ab9131 [Doc] Reorganize online pooling APIs (#11172) Cyrus Leung 2024-12-14 00:22:22 +08:00
• 238c0d93b4 [Misc] Add tokenizer_mode param to benchmark_serving.py (#11174) Alexander Matveev 2024-12-13 11:19:10 -05:00
• 5b0ed8391d [Bugfix] using len(tokenizer) instead of tokenizer.vocab_size in AllowedTokenIdsLogitsProcessor (#11156) zhangjf 2024-12-13 23:56:19 +08:00
• c31d4a57a6 [Core] support LoRA and prompt adapter in content-based hashing for Block Manager v2 prefix caching (#8240) Sungjae Lee 2024-12-14 00:51:25 +09:00
• d1fa714cb1 [Refactor]A simple device-related refactor (#11163) Chenguang Liandnoemotiovon 2024-12-13 21:39:00 +08:00
• 969da7d70b [V1][VLM] Fix edge case bug for InternVL2 (#11165) Roger Wang 2024-12-13 03:09:30 -08:00
• eeec9e3390 [Frontend] Separate pooling APIs in offline inference (#11129) Cyrus Leung 2024-12-13 18:40:07 +08:00
• f93bf2b189 [Bugfix][CI][CPU] add missing datasets package to requirements-cpu.txt (#11159) Li, Jiang 2024-12-13 16:50:35 +08:00
• 7cd7409142 PaliGemma 2 support (#11142) Jani Monoses 2024-12-13 09:40:07 +02:00
• be39e3cd18 [core] clean up cudagraph batchsize padding logic (#10996) youkaichao 2024-12-12 22:57:50 -08:00
• 34f1a806d5 [Bugfix][V1] Fix 'NoneType' object has no attribute 'hash_value' (#11157) Cody Yu 2024-12-12 22:30:06 -08:00
• 00c1bde5d8 [ROCm][AMD] Disable auto enabling chunked prefill on ROCm (#11146) Gregory Shtrasberg 2024-12-13 00:31:26 -05:00
• 3989a79824 [Bugfix] Update starcoder2 to remap k/v scale names for kv_cache quantization (#11148) Dipika Sikka 2024-12-13 00:07:20 -05:00
• 1efce68605 [Bugfix] Use runner_type instead of task in GritLM (#11144) Pooya Davoodi 2024-12-12 20:09:53 -08:00
• 30870b4f66 [torch.compile] Dynamic fp8 + rms_norm fusion (#10906) Luka GovedičandVarun Sundar Rabindranath 2024-12-12 22:19:23 -05:00
• 78ed8f57d8 [Misc][V1] Fix type in v1 prefix caching (#11151) Cody Yu 2024-12-12 16:57:40 -08:00
• db6c264a1e [Bugfix] Fix value unpack error of simple connector for KVCache transfer. (#11058) shangmingc 2024-12-13 05:19:17 +08:00
• 9f3974a319 Fix logging of the vLLM Config (#11143) Jeremy Arnold 2024-12-12 14:05:57 -06:00
• 2c97eca1ff [Misc] Validate grammar and fail early (#11119) Cody Yu 2024-12-12 10:34:26 -08:00
• 5d712571af [Bugfix] Quick fix to make Pixtral-HF load correctly again after 39e227c7ae. (#11024) Jeff Cook 2024-12-12 11:09:20 -07:00
• d4d5291cc2 fix(docs): typo in helm install instructions (#11141) Ramon Ziai 2024-12-12 18:36:32 +01:00
• 4816d20aa4 [V1] Fix torch profiling for offline inference (#11125) Roger Wang 2024-12-12 07:51:53 -08:00
• 85362f028c [Misc][LoRA] Ensure Lora Adapter requests return adapter name (#11094) Jiaxin ShanandJee Jee Li 2024-12-12 01:25:16 -08:00
• 62de37a38e [core][distributed] initialization from StatelessProcessGroup (#10986) youkaichao 2024-12-12 01:04:19 -08:00
• 8195824206 [Hardware][Intel-Gaudi] Enable LoRA support for Intel Gaudi (HPU) (#10565) Sanju C Sudhakaran 2024-12-12 13:39:28 +05:30
• f092153fbe [V1] Use more persistent buffers to optimize input preparation overheads (#11111) Woosuk Kwon 2024-12-11 23:14:20 -08:00
• 1da8f0e1dd [Model] Add support for embedding model GritLM (#10816) Pooya Davoodi 2024-12-11 22:39:16 -08:00
• ccede2b264 [Core] cleanup zmq ipc sockets on exit (#11115) Russell Bryant 2024-12-11 22:12:24 -05:00
• 24a36d6d5f Update link to LlamaStack remote vLLM guide in serving_with_llamastack.rst (#11112) Yuan Tang 2024-12-11 21:39:21 -05:00
• 8fb26dac61 [Docs] Add media kit (#11121) Simon Mo 2024-12-11 17:33:11 -08:00
• 7439a8b5fc [Bugfix] Multiple fixes to tool streaming with hermes and mistral (#10979) Clayton 2024-12-11 17:10:12 -08:00
• 4e11683368 [V1] VLM preprocessor hashing (#11020) 3 people 2024-12-11 19:55:30 -05:00
• 452a723bf2 [V1][Core] Remove should_shutdown to simplify core process termination (#11113) Tyler Michael Smith 2024-12-11 18:34:54 -05:00
• d1e21a979b [CI/Build] Split up VLM tests (#11083) Cyrus Leung 2024-12-12 06:18:16 +08:00
• 72ff3a9686 [core] Bump ray to use _overlap_gpu_communication in compiled graph tests (#10410) Rui QiaoandRui Qiao 2024-12-11 11:36:35 -08:00
• 66aaa7722d [torch.compile] remove graph logging in ci (#11110) youkaichao 2024-12-11 10:59:50 -08:00
• d643c2aba1 [V1] Use input_ids as input for text-only models (#11032) Woosuk Kwon 2024-12-11 10:49:23 -08:00
• 91642db952 [torch.compile] use depyf to dump torch.compile internals (#10972) youkaichao 2024-12-11 10:43:05 -08:00
• fd22220687 [Doc] Installed version of llmcompressor for int8/fp8 quantization (#11103) bingpsandGuangda Liu 2024-12-11 23:43:24 +08:00
• b2f775456e [CI/Build] Enable prefix caching test for AMD (#11098) hissu-hyvarinen 2024-12-11 17:23:37 +02:00