77 Commits
Author SHA1 Message Date
Nathan SalaandShaun Kotek - Nvidia 0838cd6593 [Bugfix] Load and apply Nemotron Super final vision LayerNorm (#61401)
Signed-off-by: Shaun Kotek <skotek@nvidia.com>
Co-authored-by: Shaun Kotek - Nvidia <skotek@nvidia.com>
2026-10-13 12:42:29 +00:00
73a78e6f1f [SpecDecode] Add LiLiCorr drafter (#57934)
Signed-off-by: Andrii Skliar <askliar@nvidia.com>
Signed-off-by: Andrii Skliar <andreyws96@gmail.com>
Co-authored-by: Andrii Skliar <askliar@nvidia.com>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Benjamin Chislett <chislett.ben@gmail.com>
2026-09-25 10:08:28 -04:00
Harry MellorandClaude Opus 5.5 d881812500 [Chore] Use Transformers v5 names and drop redundant processor use_fast (#58550)
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-24 14:14:10 +00:00
Harry MellorandClaude Opus 5 8a5cf54387 [Multimodal] Type dummy options per modality (#57576)
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-18 17:29:11 +01:00
Taneem Ibrahim 2b8a5495a8 [Mypy] Fix mypy typing for N/O models (#54142)
Signed-off-by: Taneem Ibrahim <taneem.ibrahim@gmail.com>
2026-09-18 12:50:01 +01:00
Harry MellorandClaude Opus 5 651a88c09a [CI] Ignore ruff D209, rejoin the docstrings it split, and silence incompatible-rule warnings (#57212)
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 10:57:24 -07:00
Cyrus Leung 0b9e018fb9 [MM] Further cleanup _apply_hf_processor_main (#53610)
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
2026-09-16 10:21:00 -07:00
Harry MellorandClaude Opus 5 8c1557a79c Add pydocstyle to the ruff rules (#52136)
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Signed-off-by: Harry Mellor <hej.mellor@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 16:44:29 +00:00
danisereb a31ec3a68b [LoRA][Nemotron] Add LoRA support for Nemotron VL models (for the language model only) (#56231)
Signed-off-by: Daniel Serebrenik <daserebrenik@nvidia.com>
2026-09-15 21:20:28 -07:00
Taneem Ibrahim a0914ab7d0 [Nano-Nemotron] Fix Nano-Nemotron precomputed multimodal embeddings (#56398)
Signed-off-by: Taneem Ibrahim <taneem.ibrahim@gmail.com>
2026-09-12 09:07:10 -04:00
Cyrus LeungandQwen Code ebcd606467 [MM] Simplify prompt updates: replace PromptSeq with list[int] (#53372)
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Co-authored-by: Qwen Code <noreply@alibaba-inc.com>
2026-08-23 14:50:32 +08:00
Cyrus Leung 020829f66b [MM] Remove renderer_applies_updates flag (#53385)
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
2026-08-23 01:25:55 +08:00
Cyrus Leung 040700aaa6 [MM] Address comments on #53275 (#53364)
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
2026-08-22 10:27:59 +00:00
Cyrus Leung 0b19ebcacd [MM] Simplify _apply_hf_processor_main (#53275)
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
2026-08-22 01:49:37 -07:00
Cyrus Leung b00f475f09 [MM] Remove text components from ProcessorInputs (#53093)
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
2026-08-20 20:42:53 -07:00
Nick Hill deeeae75d0 [MM] Keep more metadata tensors on CPU (#52827)
Signed-off-by: Nick Hill <nickhill123@gmail.com>
2026-08-18 19:56:33 -07:00
Juan Pérez de Algaba 37c3bdf5a7 fix(security): enforce audio decode duration limit in NanoNemotronVL (#50221)
Signed-off-by: jperezde <jperezde@redhat.com>
2026-08-13 03:31:55 -07:00
nvbfalk 60915c972c [Feature] Add VidCom2 video token pruning (#47750)
Signed-off-by: Benedikt Falk <bfalk@nvidia.com>
2026-07-28 03:20:17 +00:00
Noa Neria ac062147fa Avoid silent weights corruption when loading Nemotron Nano VL with reusable-buffer loaders like runai distributed streaming (#42244)
Signed-off-by: Noa Neria <nneria@nvidia.com>
2026-05-11 12:03:14 +00:00
8c780943b4 Fix Nano Nemotron text-only weight loading (#41205)
Signed-off-by: sunghoon.baek <sunghoon.baek@connectfy.cloud>
Signed-off-by: Baekpica <35071468+Baekpica@users.noreply.github.com>
Signed-off-by: sunghoon.baek <seanbb93@gmail.com>
Co-authored-by: sunghoon.baek <sunghoon.baek@connectfy.cloud>
Co-authored-by: OpenAI Codex <codex@openai.com>
Co-authored-by: Netanel Haber <58652339+netanel-haber@users.noreply.github.com>
2026-05-04 21:43:07 +03:00
milesial 9ad5abe772 Fix Nano Nemotron VL static image inputs (#40724)
Signed-off-by: Alexandre Milesi <milesial@users.noreply.github.com>
2026-04-24 09:18:55 +00:00
danielafrimiandroot 8b5531933a FIX: support language_model.backbone naming in NemotronH Nano VL quantization config (#39901)
Signed-off-by: <>
Co-authored-by: root <root@lyris0144.lyris.clusters.nvidia.com>
2026-04-15 13:49:48 +00:00
milesial 270e8a4102 Nemotron Nano VL: Streamline pixel shuffle (#37580)
Signed-off-by: milesial <milesial@users.noreply.github.com>
2026-04-10 07:31:19 +00:00
Andrii SkliarandAndrii Skliar df2503e125 nemotron-nano-vl: Allow use_audio_in_video to be passed at vllm serve time (#38538)
Signed-off-by: Andrii Skliar <askliar@nvidia.com>
Co-authored-by: Andrii Skliar <askliar@nvidia.com>
2026-04-09 11:44:39 +00:00
Netanel Haber a9a0e0551f nano-nemotron-vl: get_mm_max_tokens_per_item for audio, video, image == seq_len (#38727)
Signed-off-by: Netanel Haber <58652339+netanel-haber@users.noreply.github.com>
2026-04-07 00:23:29 -07:00
Netanel Haber d56e952239 nano_nemotron_vl: fix tensor device mismatch exception when video profiling (#39029)
Signed-off-by: Netanel Haber <58652339+netanel-haber@users.noreply.github.com>
2026-04-05 22:23:45 +00:00
Netanel Haber fa9e68022d Fix Nano Nemotron VL regressions (#38655)
Signed-off-by: Netanel Haber <58652339+netanel-haber@users.noreply.github.com>
2026-04-03 15:22:06 +08:00
Lukas Geiger 4f6eed3bd4 [Core] Simplify multimodal masking (#34246)
Signed-off-by: Lukas Geiger <lukas.geiger94@gmail.com>
2026-04-01 01:18:22 -07:00
Netanel Haberandtomeras91 e812bf70bd Restore non-hf processor path for Nano-Nemotron-VL (bypass call_hf_processor_mm_only) - fixes #38018 (#38567)
Signed-off-by: Netanel Haber <58652339+netanel-haber@users.noreply.github.com>
Co-authored-by: tomeras91 <57313761+tomeras91@users.noreply.github.com>
2026-03-30 21:56:52 +00:00
Cyrus Leung ba2f0acc2d [Misc] Reorganize inputs (#35182)
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
2026-03-25 10:22:54 -07:00
Isotr0py c7f98b4d0a [Frontend] Remove librosa from audio dependency (#37058)
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>
2026-03-21 11:36:15 +08:00
Cyrus Leung 9515c20868 [Misc] Clean up processing logic (#37541)
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
2026-03-19 13:30:20 +00:00
Collin McCarthy 0b6d52629f Support temporal compression for Nemotron-3-VL videos (#36808)
Signed-off-by: Collin McCarthy <cmccarthy@nvidia.com>
2026-03-19 08:02:19 +00:00
Netanel Haber 6ae4c8d6fc chunk parakeet into 30s clips to prevent OOMs on long audios (#36671)
Signed-off-by: Netanel Haber <58652339+netanel-haber@users.noreply.github.com>
2026-03-18 14:22:24 -07:00
Cyrus Leung f340324335 [1/2] Move InternVL-based processors (#37260)
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
2026-03-17 21:50:56 +08:00
nvnbagrov b7332b058c [Model] Nano Nemotron VL - fast media preprocessing (#35657)
Signed-off-by: Natan Bagrov <nbagrov@nvidia.com>
2026-03-08 03:04:05 -07:00
+2 5d199ac8f2 Support Audio Extraction from MP4 Video for Nemotron Nano VL (#35539)
Signed-off-by: Netanel Haber <58652339+netanel-haber@users.noreply.github.com>
Signed-off-by: Andrii Skliar <askliar@nvidia.com>
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
Signed-off-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com>
Signed-off-by: wangxiyuan <wangxiyuan1007@gmail.com>
Signed-off-by: Andrii <askliar@nvidia.com>
Co-authored-by: Netanel Haber <58652339+netanel-haber@users.noreply.github.com>
Co-authored-by: Andrii Skliar <askliar@oci-nrt-cs-001-vscode-01.cm.cluster>
Co-authored-by: Andrii <askliar@nvidia.com>
Co-authored-by: root <root@pool0-03748.cm.cluster>
Co-authored-by: Roger Wang <hey@rogerw.io>
Co-authored-by: root <root@pool0-02416.cm.cluster>
Co-authored-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com>
Co-authored-by: Matthew Bonanni <mbonanni@redhat.com>
Co-authored-by: Tyler Michael Smith <tyler@neuralmagic.com>
Co-authored-by: wangxiyuan <wangxiyuan1007@gmail.com>
Co-authored-by: root <root@pool0-04880.cm.cluster>
2026-03-03 23:20:33 -08:00
Isotr0py 6e9f21e8a2 [Chore] Remove debug code in model implementation (#35883)
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>
2026-03-03 19:50:58 -08:00
Netanel HaberandRoger Wang c8aca0c9e1 Support parakeet as audio encoder for nemotron-nano-vl (#35100)
Signed-off-by: Netanel Haber <58652339+netanel-haber@users.noreply.github.com>
Co-authored-by: Roger Wang <hey@rogerw.io>
2026-02-27 11:07:38 -07:00
Cyrus Leung 987506bca6 [Refactor] Simplify dummy data generation (#35025)
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
2026-02-22 20:55:27 -08:00
Cyrus Leung 73391a1baa [Renderer] Move InputPreprocessor into Renderer (1/2) (#34510)
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Signed-off-by: Cyrus Leung <cyrus.tl.leung@gmail.com>
2026-02-14 10:14:21 -08:00
Isotr0py 0ab06100f4 [Multimodal] Expose mm_processor_kwargs for DummyInputsBuilder (#34330)
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>
2026-02-11 09:37:40 -08:00
Cyrus Leung 51550179fc [Refactor] Define MM data parser in processing info instead of processor itself (#33260)
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
2026-01-29 13:55:17 +08:00
Cyrus Leung dcd80206b7 [Chore] Update type annotation of input_ids in model forward (#33063)
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
2026-01-26 06:02:10 -08:00
Harry HuangandChen Zhang 5206e5e28c [V1][Hybrid] Mamba Prefix Caching with align mode (#30877)
Signed-off-by: huanghaoyan.hhy <huanghaoyan.hhy@alibaba-inc.com>
Signed-off-by: Chen Zhang <zhangch99@outlook.com>
Co-authored-by: Chen Zhang <zhangch99@outlook.com>
2026-01-23 09:56:48 -08:00
Cyrus Leung 2b8a38b6d6 [Model] Extend collect_children and no_init_weights contexts (#32757)
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
2026-01-22 08:20:27 +00:00
Netanel Haber 27ca95b3c9 [Bugfix] Fix Nemotron-Nano-v2-vlm static resolution (#32682)
Signed-off-by: Netanel Haber <58652339+netanel-haber@users.noreply.github.com>
2026-01-21 06:28:21 +00:00
Cyrus Leung fda3f03eb2 [4/N] Initialize MM components in context managers (M-P) (#32663)
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
2026-01-20 14:06:32 +00:00
Netanel Haber cd3ac5b797 support dynamic resolution image encoding for Nemotron Nano VL (#32121)
Signed-off-by: Netanel Haber <58652339+netanel-haber@users.noreply.github.com>
2026-01-19 18:15:58 +00:00
Cyrus Leung 9ea07b41da [1/N] Reorganize multimodal processing code (#32327)
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
2026-01-14 15:25:31 +00:00