|
f75aa72732
|
[Neuron] Add custom_ops for neuron backend (#13246)
Signed-off-by: Liangfu Chen <liangfc@amazon.com>
Co-authored-by: George Novack <gnovack@amazon.com>
Co-authored-by: Aoyu Zhang <aoyuzhan@amazon.com>
|
2025-02-25 11:47:49 -08:00 |
|
|
Michael Goin
|
4d251ad00e
|
Fix CompressedTensorsWNA16MoE with grouped scales (#13769)
|
2025-02-25 00:17:14 -08:00 |
|
|
Tyler Michael Smith
|
1e15aaef56
|
[Bugfix][Quantization] Fix FP8 + EP (#13784)
Signed-off-by: Tyler Michael Smith <tyler@neuralmagic.com>
|
2025-02-25 10:54:17 +08:00 |
|
|
cjackal
|
51010a1807
|
[Misc] set single whitespace between log sentences (#13771)
Signed-off-by: cjackal <44624812+cjackal@users.noreply.github.com>
|
2025-02-25 10:26:12 +08:00 |
|
|
Harry Mellor
|
cdc1fa12eb
|
Remove unused kwargs from model definitions (#13555)
|
2025-02-24 17:13:52 -08:00 |
|
|
Michael Goin
|
db986c19ea
|
Fix precommit fail in fused_moe intermediate_cache2 chunking (#13772)
Signed-off-by: mgoin <mgoin64@gmail.com>
|
2025-02-24 09:25:47 -08:00 |
|
|
Zhonghua Deng
|
ccc00515fd
|
[BugFix] Illegal memory access for MoE On H20 (#13693)
|
2025-02-24 07:37:32 -08:00 |
|
|
Jongseok Park
|
781096e385
|
Expert Parallelism (EP) Support for DeepSeek V2 (#12583)
|
2025-02-24 07:33:20 -08:00 |
|
|
Mengqing Cao
|
23eca9cf68
|
[model][refactor] remove cuda hard code in models and layers (#13658)
|
2025-02-24 06:10:14 -08:00 |
|
|
Gregory Shtrasberg
|
c904fdddf6
|
[ROCm] Apply FP8 weights padding to values not divisible by 512 bytes on ROCm (#13231)
|
2025-02-22 05:54:38 -08:00 |
|
|
Cyrus Leung
|
7f6bae561c
|
[CI/Build] Fix pre-commit errors (#13696)
|
2025-02-22 00:31:26 -08:00 |
|
|
Yu Chin Fabian Lim
|
fca20841c2
|
Correction to TP logic for Mamba Mixer 2 when Num Groups not divisible by TP Size (#13660)
|
2025-02-22 00:19:10 -08:00 |
|
|
|
288cc6c234
|
[Attention] MLA with chunked prefill (#12639)
Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
Co-authored-by: Patrick Horn <patrick.horn@gmail.com>
Co-authored-by: simon-mo <xmo@berkeley.edu>
Co-authored-by: Tyler Michael Smith <tyler@neuralmagic.com>
|
2025-02-21 15:30:12 -08:00 |
|
|
chenxiaobing
|
ed6e9075d3
|
[Bugfix] Fix deepseekv3 grouped topk error (#13474)
Signed-off-by: Chen-XiaoBing <chenxb002@whu.edu.cn>
|
2025-02-20 06:47:01 -08:00 |
|
|
Divakar Verma
|
0d243f2a54
|
[ROCm][MoE] mi300 mixtral8x7B perf for specific BS (#13577)
Signed-off-by: Divakar Verma <divakar.verma@amd.com>
|
2025-02-20 04:01:02 +00:00 |
|
|
Divakar Verma
|
8aada19dfc
|
[ROCm][MoE configs] mi325 mixtral & mi300 qwen_moe (#13503)
|
2025-02-18 22:23:24 -08:00 |
|
|
Nick Hill
|
30172b4947
|
[V1] Optimize handling of sampling metadata and req_ids list (#13244)
Signed-off-by: Nick Hill <nhill@redhat.com>
|
2025-02-18 12:15:33 -08:00 |
|
|
Harry Mellor
|
efbe854448
|
[Misc] Remove dangling references to SamplingType.BEAM (#13402)
|
2025-02-17 19:52:35 -08:00 |
|
|
Tyler Michael Smith
|
b3942e157e
|
[Bugfix][CI][V1] Work around V1 + CUDA Graph + torch._scaled_mm fallback issue (#13425)
Signed-off-by: Tyler Michael Smith <tyler@neuralmagic.com>
|
2025-02-18 00:32:48 +00:00 |
|
|
Tyler Michael SmithandYu Chin Fabian Lim
|
1f69c4a892
|
[Model] Support Mamba2 (Codestral Mamba) (#9292)
Signed-off-by: Tyler Michael Smith <tyler@neuralmagic.com>
Co-authored-by: Yu Chin Fabian Lim <flim@sg.ibm.com>
|
2025-02-17 20:17:50 +08:00 |
|
|
Kyle Sayers
|
12913d17ba
|
[Quant] Add SupportsQuant to phi3 and clip (#13104)
|
2025-02-15 19:28:33 -08:00 |
|
|
rasmith
|
ed0de3e4b8
|
[AMD] [Model] DeepSeek tunings (#13199)
|
2025-02-15 03:58:09 -08:00 |
|
|
|
3bcb8c75da
|
[Core] Reduce TTFT with concurrent partial prefills (#10235)
Signed-off-by: Joe Runde <Joseph.Runde@ibm.com>
Signed-off-by: Prashant Gupta <prashantgupta@us.ibm.com>
Co-authored-by: Prashant Gupta <prashantgupta@us.ibm.com>
Co-authored-by: Cody Yu <hao.yu.cody@gmail.com>
|
2025-02-14 15:36:07 -08:00 |
|
|
Michael Goin
|
5e5c8e091e
|
[Quant][Perf] Use moe_wna16 kernel by default for MoEs with many experts (#13236)
Signed-off-by: mgoin <mgoin64@gmail.com>
|
2025-02-14 12:53:42 -08:00 |
|
|
Tyler Michael Smith
|
c1e37bf71b
|
[Kernel][Bugfix] Refactor and Fix CUTLASS 2:4 Sparse Kernels (#13198)
Signed-off-by: Tyler Michael Smith <tyler@neuralmagic.com>
|
2025-02-14 00:01:14 +00:00 |
|
|
Michael Goin
|
2344192a55
|
Optimize moe_align_block_size for deepseek_v3 (#12850)
Signed-off-by: mgoin <mgoin64@gmail.com>
|
2025-02-13 18:43:37 -05:00 |
|
|
Daniel Han
|
cb944d5818
|
Allow Unsloth Dynamic 4bit BnB quants to work (#12974)
|
2025-02-12 23:13:08 -08:00 |
|
|
Michael Goin
|
09972e716c
|
[Bugfix] Allow fallback to AWQ from AWQMarlin at per-layer granularity (#13119)
|
2025-02-12 09:19:53 -08:00 |
|
|
Qubitium-ModelCloud
|
36a08630e8
|
[CORE] [QUANT] Support for GPTQModel's dynamic quantization per module override/control (#7086)
|
2025-02-12 09:19:43 -08:00 |
|
|
Szymon Ożóg
|
2b25b7d2e1
|
Fix initializing GGUF weights for ColumnParallelLinear when using tensor parallel > 1 (#13023)
|
2025-02-11 08:38:48 -08:00 |
|
|
Sanju C Sudhakaran
|
2880e21e3d
|
[Hardware][Intel-Gaudi] Enable long-contexts + LoRA support for Intel Gaudi (#12812)
Signed-off-by: Sanju C Sudhakaran <scsudhakaran@habana.ai>
|
2025-02-08 17:15:30 +08:00 |
|
|
TJian
|
eaa92d4437
|
[ROCm] [Feature] [Doc] [Dockerfile] [BugFix] Support Per-Token-Activation Per-Channel-Weight FP8 Quantization Inferencing (#12501)
|
2025-02-07 08:13:43 -08:00 |
|
|
Amit Garg
|
538fab93cd
|
PR #12718 (#12718)
|
2025-02-07 06:22:37 -08:00 |
|
|
Lu Fang
|
741429a4cd
|
[MISC] Check space in the file names in the pre commit checks (#12804)
Signed-off-by: Lu Fang <lufang@fb.com>
|
2025-02-06 15:36:21 -08:00 |
|
|
Yu Chin Fabian LimandTyler Michael Smith
|
aff404571b
|
Add Bamba Model (#10909)
Signed-off-by: Yu Chin Fabian Lim <flim@sg.ibm.com>
Signed-off-by: Tyler Michael Smith <tyler@neuralmagic.com>
Co-authored-by: Tyler Michael Smith <tyler@neuralmagic.com>
|
2025-02-06 15:22:42 -08:00 |
|
|
Varun Sundar RabindranathandVarun Sundar Rabindranath
|
467a96a541
|
[V1] LoRA Support (#10957)
Signed-off-by: Varun Sundar Rabindranath <varun@neuralmagic.com>
Co-authored-by: Varun Sundar Rabindranath <varun@neuralmagic.com>
|
2025-02-06 09:32:51 -08:00 |
|
|
Dipika Sikka
|
7ca9934fe7
|
[Misc] Update w2 scale loading for GPTQMarlinMoE (#12757)
|
2025-02-06 01:02:14 -08:00 |
|
|
Lu Fang
|
9cdea30b4f
|
[Misc][Easy] Remove the space from the file name
|
2025-02-05 19:23:35 -08:00 |
|
|
Lucas Wilkinson
|
76abd0c881
|
[Bugfix] Better FP8 supported defaults
|
2025-02-05 19:22:19 -08:00 |
|
|
Roger Wang
|
bf3b79efb8
|
[VLM] Qwen2.5-VL
|
2025-02-05 13:31:38 -08:00 |
|
|
Rahul Tuli
|
3b2005e1db
|
Add: Support for Sparse24Bitmask Compressed Models
|
2025-02-05 13:30:43 -08:00 |
|
|
Kyle Sayersandmgoin
|
7ff7a638b6
|
[Model][Quant] Fix GLM, Fix fused module mappings for quantization (#12634)
Signed-off-by: mgoin <michael@neuralmagic.com>
Signed-off-by: Kyle Sayers <kylesayrs@gmail.com>
Co-authored-by: mgoin <michael@neuralmagic.com>
|
2025-02-05 05:32:06 +00:00 |
|
|
Harry Mellor
|
249824c3bf
|
Refactor Linear handling in TransformersModel (#12727)
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
|
2025-02-05 04:31:12 +00:00 |
|
|
Aviv Keshet
|
b3a0d01e45
|
[Core] add and implement VLLM_LOGITS_PROCESSOR_THREADS (#12368)
Signed-off-by: Aviv Keshet <akeshet@scaledcognition.com>
|
2025-02-04 18:46:26 -08:00 |
|
|
Hongxia YangandMatthew Wong
|
c36ac98d01
|
[AMD][ROCm] Enable DeepSeek model on ROCm (#12662)
Signed-off-by: Hongxia Yang <hongxia.yang@amd.com>
Co-authored-by: Matthew Wong <Matthew.Wong2@amd.com>
|
2025-02-04 08:24:11 +00:00 |
|
|
Kyle Sayers
|
4896d0c2dd
|
[Quant] Fix use_mla TypeError and support loading pure-sparsity Compressed Tensors configs (#12711)
|
2025-02-03 23:27:11 -08:00 |
|
|
kushanam
|
4797dad3ec
|
[Model] Add Deepseek V3 fp8_w8a8 configs for B200 (#12707)
|
2025-02-03 13:30:39 -08:00 |
|
|
+24
|
b9986454fe
|
Fix for attention layers to remain unquantized during moe_wn16 quant (#12570)
Fix to AWQ quant loading of the new R1 model
The new optimized MoE kernels for a large number of experts `moe_wn16`
uses AWQ quant which requires the attention layers to be in 16bit
The current merge has broken this, and the `get_quant_method` must
return None for it to work correctly again
---------
Signed-off-by: Srikanth Srinivas <srikanth@astrum.ai>
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Signed-off-by: Beim <beim2015@outlook.com>
Signed-off-by: rshaw@neuralmagic.com <rshaw@neuralmagic.com>
Signed-off-by: mgoin <michael@neuralmagic.com>
Signed-off-by: npanpaliya <nishidha.panpaliya@partner.ibm.com>
Signed-off-by: Aleksandr Malyshev <maleksan@amd.com>
Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>
Signed-off-by: simon-mo <xmo@berkeley.edu>
Signed-off-by: Cody Yu <hao.yu.cody@gmail.com>
Signed-off-by: Chen Zhang <zhangch99@outlook.com>
Signed-off-by: Tyler Michael Smith <tyler@neuralmagic.com>
Signed-off-by: Ryan N <ryan.nguyen@centml.ai>
Signed-off-by: Brian Dellabetta <bdellabe@redhat.com>
Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>
Signed-off-by: Rahul Tuli <rahul@neuralmagic.com>
Signed-off-by: Russell Bryant <rbryant@redhat.com>
Signed-off-by: simon-mo <simon.mo@hey.com>
Signed-off-by: Vicente Herrera <vicenteherrera@vicenteherrera.com>
Signed-off-by: Jinzhen Lin <linjinzhen@hotmail.com>
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
Signed-off-by: Shawn Du <shawnd200@outlook.com>
Signed-off-by: Kunshang Ji <kunshang.ji@intel.com>
Signed-off-by: youkaichao <youkaichao@gmail.com>
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Beim <805908499@qq.com>
Co-authored-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com>
Co-authored-by: mgoin <michael@neuralmagic.com>
Co-authored-by: simon-mo <xmo@berkeley.edu>
Co-authored-by: Nishidha <nishidha.panpaliya@partner.ibm.com>
Co-authored-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com>
Co-authored-by: Aleksandr Malyshev <164964928+maleksan85@users.noreply.github.com>
Co-authored-by: Aleksandr Malyshev <maleksan@amd.com>
Co-authored-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
Co-authored-by: simon-mo <simon.mo@hey.com>
Co-authored-by: Michael Goin <mgoin64@gmail.com>
Co-authored-by: Zhuohan Li <zhuohan123@gmail.com>
Co-authored-by: Tyler Michael Smith <tysmith@redhat.com>
Co-authored-by: Alexander Matveev <59768536+alexm-neuralmagic@users.noreply.github.com>
Co-authored-by: Roger Wang <136131678+ywang96@users.noreply.github.com>
Co-authored-by: Cody Yu <hao.yu.cody@gmail.com>
Co-authored-by: Chen Zhang <zhangch99@outlook.com>
Co-authored-by: Kevin H. Luu <kevin@anyscale.com>
Co-authored-by: Tyler Michael Smith <tyler@neuralmagic.com>
Co-authored-by: Ryan Nguyen <96593302+xpbowler@users.noreply.github.com>
Co-authored-by: Brian Dellabetta <brian-dellabetta@users.noreply.github.com>
Co-authored-by: fade_away <1028552010@qq.com>
Co-authored-by: weilong.yu <weilong.yu@shopee.com>
Co-authored-by: Jee Jee Li <pandaleefree@gmail.com>
Co-authored-by: Eldar Kurtic <eldarkurtic314@gmail.com>
Co-authored-by: Rahul Tuli <rahul@neuralmagic.com>
Co-authored-by: Russell Bryant <rbryant@redhat.com>
Co-authored-by: Vicente Herrera <vicenteherrera@vicenteherrera.com>
Co-authored-by: Jinzhen Lin <linjinzhen@hotmail.com>
Co-authored-by: Shawn Du <shawnd200@outlook.com>
Co-authored-by: Kunshang Ji <kunshang.ji@intel.com>
Co-authored-by: youkaichao <youkaichao@gmail.com>
|
2025-02-03 13:46:19 +08:00 |
|
|
Eldar Kurtic
|
c5932e5dac
|
Properly check if all fused layers are in the list of targets (#12666)
Thanks @kylesayrs for catching this!
|
2025-02-03 13:42:18 +08:00 |
|
|
Yang Chen
|
95460fc513
|
[Kernel] port sgl moe_align_block_size kernels (#12574)
sgl_moe_align_block_size is based on:
https://github.com/sgl-project/sglang/commit/ded9fcd09a43d5e7d5bb31a2bc3e9fc21bf65d2a
moe_align_block_size is based on:
https://github.com/sgl-project/sglang/commit/ba5112ff691d791a9e38c6c71f59324a5fcb49d0
Signed-off-by: Yang Chen <yangche@fb.com>
|
2025-02-03 13:09:50 +08:00 |
|