This website requires JavaScript.
c2a9671510
[Misc] Improve model redirect to accept json dictionary (#16119 )
Isotr0py
2025-04-06 20:51:45 +08:00
d5ae4f7f42
[Doc][Bugfix] Add missing EOF in k8s deploy doc (#16025 )
Paul Schweigert
2025-04-06 08:10:57 -04:00
b6c502a150
[Misc] refactor example eagle (#16100 )
Reid and reidliu41
2025-04-06 17:42:48 +08:00
9ca710e525
[CI][V1] Fix passing tokenizer as kwarg to validate_guidance_grammar (#16117 )
Roger Wang
2025-04-06 01:18:00 -07:00
eb07c8cb5b
[Frontend] Fix typo in tool chat templates for llama3.2 and toolace (#14501 )
Ben Jackson
2025-04-06 00:44:36 -07:00
ba10801961
[Benchmark] Add sampling parameters to benchmark_serving. (#16022 )
Hyesoo Yang
2025-04-05 21:30:35 -07:00
620fc2d09e
[Model] fix model testing for TeleChat2ForCausalLM and V0 llama4 (#16112 )
Lucia Fang
2025-04-05 21:23:40 -07:00
29283eaa7e
[Model] use AutoWeightsLoader for phi, gemma, deepseek (#16088 )
Jonghyun Choe
2025-04-06 12:34:38 +09:00
2fa66ef713
[Bugfix] fix use_atomic_add support of marlin kernel when using v1 engine (#15946 )
Jinzhen Lin
2025-04-06 11:04:22 +08:00
13affc432d
[Misc] Remove redundant code (#16098 )
Chauncey
2025-04-06 11:03:50 +08:00
d8f094a92a
[Misc] format output for encoder_decoder.py (#16095 )
Reid and reidliu41
2025-04-06 10:57:18 +08:00
97ae6d777f
Fix some capitalisations in generated examples doc titles (#16094 )
Harry Mellor
2025-04-05 14:44:03 +01:00
6baeee70d1
Revert "doc: add info for macos clang errors (#16049 )" (#16091 )
yihong
2025-04-05 19:51:51 +08:00
d2517a4939
[doc] fix 404 (#16082 )
Reid and reidliu41
2025-04-05 19:39:18 +08:00
6342adc438
fix: support clang17 for macos and fix the real libomp (#16086 )
yihong
2025-04-05 19:00:12 +08:00
0adba91547
[CI] Fix benchmark script level (#16089 )
Kevin H. Luu
2025-04-05 03:36:01 -07:00
4285e423a6
[Misc] Auto detect bitsandbytes pre-quantized models (#16027 )
Tristan Leclercq
2025-04-05 08:30:45 +02:00
63375f0cdb
[V1][Spec Decode] Update N-gram Proposer Interface (#15750 )
Woosuk Kwon
2025-04-04 16:32:54 -07:00
70ad3f9e98
[Bugfix][TPU] Fix V1 TPU worker for sliding window (#16059 )
Michael Goin
2025-04-04 17:31:19 -06:00
d6fc629f4d
[Kernel][Minor] Re-fuse triton moe weight application (#16071 )
bnellnm
2025-04-04 19:27:34 -04:00
af51d80fa1
Revert "[V1] Scatter and gather placeholders in the model runner" (#16075 )
Roger Wang
2025-04-04 14:50:57 -07:00
f5722a5052
[V1] Scatter and gather placeholders in the model runner (#15712 )
2025-04-05 05:26:44 +08:00
651cf0fec1
[V1] DP scale-out (1/N): Use zmq ROUTER/DEALER sockets for input queue (#15906 )
Nick Hill
2025-04-04 12:56:43 -07:00
4dc52e1c53
[CI] Reorganize .buildkite directory (#16001 )
Kevin H. Luu
2025-04-04 12:16:20 -07:00
4708f13a9c
[Bugfix] Fix default behavior/fallback for pp in v1 (#16057 )
Michael Goin
2025-04-04 11:58:08 -06:00
a6d042df0a
[ROCm][Bugfix] Bring back fallback to eager mode removed in #14917 , but for ROCm only (#15413 )
Gregory Shtrasberg
2025-04-04 12:40:37 -04:00
40a36ccfeb
[ROCm][Bugfix] Use platform specific FP8 dtype (#15717 )
Gregory Shtrasberg
2025-04-04 12:40:20 -04:00
ef608c37a7
[Distributed] [ROCM] Fix custom allreduce enable checks (#16010 )
Ilya Markov and ilmarkov
2025-04-04 18:39:08 +02:00
2386803f2a
[CPU] Change default block_size for CPU backend (#16002 )
Li, Jiang
2025-04-05 00:39:05 +08:00
95862f7b4d
[Benchmark][Doc] Update throughput benchmark and README (#15998 )
Ziji Shi (Steven) and Roger Wang
2025-04-04 09:39:02 -07:00
230b131b54
[Bugfix][kernels] Fix half2float conversion in gguf kernels (#15995 )
Isotr0py
2025-04-05 00:38:58 +08:00
0812d8dd41
[Hardware][Gaudi][BugFix] fix arguments of hpu fused moe (#15945 )
liuzhenwei
2025-04-05 00:38:55 +08:00
bf7e3c51ae
[Model] use AutoWeightsLoader for baichuan, gpt-neox, mpt (#15939 )
Jonghyun Choe
2025-04-05 01:38:52 +09:00
a35a8a8392
[V1][Spec Decode] Avoid logging useless nan metrics (#16023 )
Mark McLoughlin
2025-04-04 16:52:41 +01:00
4ef0bb1fcf
doc: add info for macos clang errors (#16049 )
yihong
2025-04-04 22:58:16 +08:00
fadc59c0e6
[TPU][V1] Remove ragged attention kernel parameter hard coding (#16041 )
Chengji Yao
2025-04-04 04:48:50 -07:00
86cbd2eee9
[Misc] improve gguf check (#15974 )
Reid and reidliu41
2025-04-04 09:33:36 +08:00
092475f738
[ROCm] Tweak the benchmark script to run on ROCm (#14252 )
Huy Do
2025-04-03 17:12:48 -07:00
dcc56d62da
[Bugfix] Fix function names in test_block_fp8.py (#16033 )
bnellnm
2025-04-03 19:01:34 -04:00
f15e70d906
[TPU] Switch Test to Non-Sliding Window (#15981 )
Robert Shaw and Robert Shaw
2025-04-03 14:28:45 -07:00
b6be6f8d1e
[TPU] Support sliding window and logit soft capping in the paged attention kernel for TPU. (#15732 )
iefgnoix
2025-04-03 14:23:28 -07:00
03a70eacaf
Re-enable the AMD Testing for the passing tests. (#15586 )
Alexei-V-Ivanov-AMD
2025-04-03 13:05:17 -05:00
45b1ff7a25
[Misc][Performance] Advance tpu.txt to the most recent nightly torch … (#16024 )
yarongmu-google
2025-04-03 10:32:54 -07:00
15ba07ef25
[Minor] Fused experts refactor (#15914 )
bnellnm
2025-04-03 13:19:38 -04:00
d2b58ca203
[Neuron][kernel] Fuse kv cache into a single tensor (#15911 )
Liangfu Chen
2025-04-03 09:51:32 -07:00
82e7e19a6e
[SupportsQuant] Chameleon, Chatglm, Commandr (#15952 )
Kyle Sayers
2025-04-03 11:25:22 -04:00
421c462948
[SupportsQuant] Bert, Blip, Blip2, Bloom (#15573 )
Kyle Sayers
2025-04-03 11:23:19 -04:00
84884cd9ac
fix: tiny fix make format.sh excutable (#16015 )
yihong
2025-04-03 23:18:05 +08:00
a43aa183dc
[doc] update contribution link (#15922 )
Reid and reidliu41
2025-04-03 18:47:31 +08:00
463bbb1835
[Bugfix][V1] Fix bug from putting llm_engine.model_executor in a background process (#15367 )
wwl2755
2025-04-03 02:32:10 -05:00
5e125e74d1
[misc] improve error message for "Failed to infer device type" (#15994 )
youkaichao
2025-04-03 14:45:03 +08:00
06f21ce7a5
[Benchmark] Add AIMO Dataset to Benchmark (#15955 )
Ziji Shi (Steven)
2025-04-02 23:09:18 -07:00
57a810db9c
[ROCM][V0] PA kennel selection when no sliding window provided (#15982 )
Aleksandr Malyshev and Aleksandr Malyshev
2025-04-02 22:28:44 -07:00
8b664706aa
[bugfix] add seed in torchrun_example.py (#15980 )
youkaichao
2025-04-03 12:25:01 +08:00
37bfee92bf
fix: better error message for get_config close #13889 (#15943 )
yihong
2025-04-03 11:53:19 +08:00
e73ff24e31
[ROCM][KERNEL] Paged attention for V1 (#15720 )
2025-04-02 19:48:00 -07:00
bd7599d34a
[V1][TPU] Do not compile sampling more than needed (#15883 )
Nicolò Lucchesi
2025-04-03 03:36:01 +02:00
01b6113659
[TPU] optimize the all-reduce performance (#15903 )
Chengji Yao
2025-04-02 17:25:14 -07:00
1b84eff03a
[V1][TPU] TPU-optimized top-p implementation (avoids scattering). (#15736 )
Hyesoo Yang and root
2025-04-02 17:18:08 -07:00
55acf86bf8
Fix huggingface-cli[hf-xet] -> huggingface-cli[hf_xet] (#15969 )
Harry Mellor
2025-04-03 00:37:30 +01:00
f021b97993
[V1] Support Mistral3 in V1 (#15950 )
Michael Goin
2025-04-02 16:36:24 -06:00
1cab43c2d2
[misc] instruct pytorch to use nvml-based cuda check (#15951 )
youkaichao
2025-04-03 01:02:58 +08:00
8bd651b318
Restricted cmake to be less than version 4 as 4.x breaks the build of… (#15859 )
Nishidha
2025-04-02 21:49:39 +05:30
58e234a754
[Misc] V1 LoRA support CPU offload (#15843 )
Jee Jee Li
2025-04-02 23:04:43 +08:00
e86c414d6a
[Model] use AutoWeightsLoader in model load_weights (#15770 )
rongfu.leng
2025-04-02 22:47:31 +08:00
550b2801ad
[CPU][Bugfix] Using custom allreduce for CPU backend (#15934 )
Li, Jiang
2025-04-02 22:46:47 +08:00
cefb9e5a28
[Frontend] Implement Tool Calling with tool_choice='required' (#13483 )
2025-04-02 16:45:45 +02:00
98d7367b61
[Metrics] Hide deprecated metrics (#15458 )
Mark McLoughlin
2025-04-02 15:37:19 +01:00
594a8b9030
[Bugfix] Fix the issue where the model name is empty string, causing no response with the model name. (#15938 )
Chauncey
2025-04-02 21:33:52 +08:00
44f990515b
[CI] Remove duplicate entrypoints-test (#15940 )
Kay Yan
2025-04-02 17:44:01 +08:00
252937806c
[Bugfix][Benchmarks] Ensure async_request_deepspeed_mii uses the OpenAI choices key (#15926 )
Brayden Zhong
2025-04-02 05:19:35 -04:00
51826d51fa
Add minimum version for huggingface_hub to enable Xet downloads (#15873 )
Harry Mellor
2025-04-02 10:03:36 +01:00
14e53ed11f
[V1] Fix json_object support with xgrammar (#15488 )
Russell Bryant
2025-04-02 05:00:08 -04:00
ddb94c2605
[core] Add tags parameter to wake_up() (#15500 )
Eric Tang
2025-04-02 01:59:27 -07:00
90969fb39a
[Kernel] Add more dtype support for GGUF dequantization (#15879 )
LukasBluebaum
2025-04-02 10:58:48 +02:00
101f1481f9
[Build/CI] Update lm-eval to 0.4.8 (#15912 )
Chris Thi
2025-04-02 04:47:57 -04:00
2edc87b161
[Bugfix] Fix cache block size calculation for CPU MLA (#15848 )
Thien Tran
2025-04-02 16:45:02 +08:00
4203926f10
[CI/Build] Further clean up LoRA tests (#15920 )
Jee Jee Li
2025-04-02 16:39:09 +08:00
cdb57015a7
[Misc] Replace print with logger (#15923 )
Chauncey
2025-04-02 16:37:38 +08:00
aa557e6422
[Benchmark]Fix error message (#15866 )
Li Wang and Roger Wang
2025-04-02 16:32:24 +08:00
0e00d40e4f
[V1][Bugfix] Fix typo in MoE TPU checking (#15927 )
Roger Wang
2025-04-01 23:46:42 -07:00
c920e01242
[Doc] Update rocm.inc.md (#15917 )
chun
2025-04-02 15:38:26 +09:00
274d8e8818
[V1][Minor] Enhance SpecDecoding Metrics Log in V1 (#15902 )
Woosuk Kwon
2025-04-01 23:38:02 -07:00
2039c6305b
[Bugfix] Fix imports for MoE on CPU (#15841 )
Thien Tran
2025-04-02 11:33:55 +08:00
6efb195a6e
[V1] Fix: make sure k_index is int64 for apply_top_k_only (#15907 )
Brayden Zhong
2025-04-01 22:06:44 -04:00
24b7fb455a
[Spec Decode] Fix input triton kernel for eagle (#15909 )
Ekagra Ranjan
2025-04-01 21:15:14 -04:00
58f5a59769
[Docs] Add Intel as Sponsor (#15913 )
Simon Mo
2025-04-01 17:16:55 -07:00
db9dfcfa6a
[Docs] Add Ollama meetup slides (#15905 )
Simon Mo
2025-04-01 13:58:59 -07:00
9ef98d527e
[Model][MiniMaxText01] Support MiniMaxText01 model inference (#13454 )
2025-04-02 04:23:55 +08:00
93491aefc7
[BugFix] make sure socket close (#15875 )
yihong
2025-04-02 04:10:24 +08:00
7acd539cd7
[Docs] update usage stats language (#15898 )
Simon Mo
2025-04-01 12:54:13 -07:00
e75a6301bd
[V1][Spec Decode] Implement Eagle Proposer [1/N] (#15729 )
Woosuk Kwon
2025-04-01 12:33:16 -07:00
a79cc68b3a
[V1][Metrics] Initial speculative decoding metrics (#15151 )
Mark McLoughlin
2025-04-01 18:45:04 +01:00
7e3f7a4ee7
[CI] Disable flaky structure decoding test temporarily. (#15892 )
Roger Wang
2025-04-01 10:42:34 -07:00
9ec8257914
[Model] Add module name prefixes to gemma3 (#15889 )
cloud11665 and Bartholomew Sabat
2025-04-02 02:13:40 +09:00
38327cf454
[Model] Aya Vision (#15441 )
Jennifer Zhao and Roger Wang
2025-04-01 09:30:43 -07:00
dfa82e2a3d
[CI/Build] Clean up LoRA tests (#15867 )
Jee Jee Li
2025-04-02 00:28:50 +08:00
e59ca942f5
Add option to use DeepGemm contiguous grouped gemm kernel for fused MoE operations. (#13932 )
bnellnm
2025-04-01 12:07:43 -04:00
a57a3044aa
[ROCm][Build][Bugfix] Bring the base dockerfile in sync with the ROCm fork (#15820 )
Gregory Shtrasberg
2025-04-01 11:56:39 -04:00
4e5a0f6ae2
[Misc] Allow using OpenCV as video IO fallback (#15055 )
Isotr0py and Harry Mellor
2025-04-01 23:55:13 +08:00
Isotr0py