5 Commits

Include renames

Author	SHA1	Message	Date
SangBin Cho	2e9a2227ec	[Lora] Support long context lora (#4787) ... Currently we need to call rotary embedding kernel for each LoRA, which makes it hard to serve multiple long context length LoRA. Add batched rotary embedding kernel and pipe it through. It replaces the rotary embedding layer to the one that is aware of multiple cos-sin-cache per scaling factors. Follow up of https://github.com/vllm-project/vllm/pull/3095/files	2024-05-18 16:05:23 +09:00
Cade Daniel	ab50275111	[Speculative decoding] Support target-model logprobs (#4378)	2024-05-03 15:52:01 -07:00
SangBin Cho	603ad84815	[Core] Refactoring sampler and support prompt logprob for chunked prefill (#4309)	2024-04-26 13:02:02 +00:00
SangBin Cho	0ae11f78ab	[Mypy] Part 3 fix typing for nested directories for most of directory (#4161)	2024-04-22 21:32:44 -07:00
Cade Daniel	e95cd87959	[Speculative decoding 6/9] Integrate speculative decoding with LLMEngine (#3894)	2024-04-16 13:09:21 -07:00