6 Commits

Include renames

Author	SHA1	Message	Date
SangBin Cho	65bf2ac165	[Core][2/N] Model runner refactoring part 2. Combine prepare prefill / decode to a single API (#4681) ... This PR combines prepare_prompt and prepare_decode into a single API. This PR also coelsce the attn metadata for prefill/decode to a single class and allow to slice them when running attn backend. It also refactors subquery_start_loc which was not refactored in the previous PR	2024-05-15 14:00:10 +09:00
Cody YuandCade Daniel	f942efb5a3	[Dynamic Spec Decoding] Auto-disable by the running queue size (#4592) ... Co-authored-by: Cade Daniel <edacih@gmail.com>	2024-05-08 21:44:00 +00:00
Cade Daniel	62b8aebc6f	[Speculative decoding 7/9] Speculative decoding end-to-end correctness tests. (#3951)	2024-04-23 08:02:36 +00:00
SangBin Cho	01bfb22b41	[CI] Try introducing isort. (#3495)	2024-03-25 07:59:47 -07:00
Cade Daniel	8437bae6ef	[Speculative decoding 3/9] Worker which speculates, scores, and applies rejection sampling (#3103)	2024-03-08 23:32:46 -08:00
Cade Daniel	79d64c4954	[Speculative decoding 1/9] Optimized rejection sampler (#2336)	2024-01-09 15:38:41 -08:00