feat: add Nemotron video-to-text capability#666
Conversation
|
Did u run make generate and make manifest? |
thanks @slin1237 for reviewing. I ran both |
pallasathena92
left a comment
There was a problem hiding this comment.
thanks for contributing, please resolve the conflict and request review again.
195f6ae to
602f0b5
Compare
|
thanks @pallasathena92 ! rebase with PR#672 to resolve the conflicts. as PR#672 added a more granular capability AUDIO_TEXT_TO_TEXT to distinguish with AUDIO_TO_TEXT, also update the nemotron rule to use AUDIO_TEXT_TO_TEXT instead. requesting review again. |
|
We decided to make the current changes as simple as possible:
|
|
update nemotron rule to append audioTextToText/videoTextToText when it matches nemotron_nano_omni, requesting review again. |
|
Hi @slin1237 can i get one more review on this PR, thanks so much! |
What this PR does
Adds the VIDEO_TEXT_TO_TEXT model capability.
Includes it in the Kubernetes API validation enum.
Maps it between model configuration and OME API capabilities.
Adds it to the NemotronH Nano capability classification rule.
Updates classification and capability-mapping tests.
Why we need it
The imported model nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16 supports video-and-text input with text output, but OME currently omits VIDEO_TEXT_TO_TEXT from its detected capabilities.
This causes the imported model metadata to represent only its image, text, and audio capabilities.
How to test
Checklist
make testpasses locally