Skip to content

feat: add Nemotron video-to-text capability#666

Open
caihui-lol wants to merge 2 commits into
ome-projects:mainfrom
caihui-lol:hotfix/waguo/nemotron-video-capability-main
Open

feat: add Nemotron video-to-text capability#666
caihui-lol wants to merge 2 commits into
ome-projects:mainfrom
caihui-lol:hotfix/waguo/nemotron-video-capability-main

Conversation

@caihui-lol

@caihui-lol caihui-lol commented Jul 13, 2026

Copy link
Copy Markdown
Collaborator

What this PR does

Adds the VIDEO_TEXT_TO_TEXT model capability.
Includes it in the Kubernetes API validation enum.
Maps it between model configuration and OME API capabilities.
Adds it to the NemotronH Nano capability classification rule.
Updates classification and capability-mapping tests.

Why we need it

The imported model nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16 supports video-and-text input with text output, but OME currently omits VIDEO_TEXT_TO_TEXT from its detected capabilities.
This causes the imported model metadata to represent only its image, text, and audio capabilities.

How to test

  • go test
  • E2E test by importing the omni model

Checklist

  • Tests added/updated (if applicable)
  • Docs updated (if applicable)
  • make test passes locally

@github-actions github-actions Bot added api API/Types changes in pkg/apis tests Test changes labels Jul 13, 2026
@caihui-lol
caihui-lol marked this pull request as ready for review July 13, 2026 21:03
@slin1237

Copy link
Copy Markdown
Collaborator

Did u run make generate and make manifest?
I think it will update them too?

@caihui-lol

caihui-lol commented Jul 13, 2026

Copy link
Copy Markdown
Collaborator Author

Did u run make generate and make manifest? I think it will update them too?

thanks @slin1237 for reviewing. I ran both make generate and make manifests. make manifests produced no tracked changes. make generate produced no capability-related output - only unrelated checksum and whitespace-only generator noise, so I excluded them from this PR for review. The current generated schema represents modelCapabilities.items as an unconstrained string, so this enum addition does not change the CRD or Helm manifests.

@pallasathena92 pallasathena92 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks for contributing, please resolve the conflict and request review again.

@caihui-lol
caihui-lol force-pushed the hotfix/waguo/nemotron-video-capability-main branch from 195f6ae to 602f0b5 Compare July 16, 2026 18:23
@caihui-lol

Copy link
Copy Markdown
Collaborator Author

thanks @pallasathena92 ! rebase with PR#672 to resolve the conflicts. as PR#672 added a more granular capability AUDIO_TEXT_TO_TEXT to distinguish with AUDIO_TO_TEXT, also update the nemotron rule to use AUDIO_TEXT_TO_TEXT instead. requesting review again.

Comment thread pkg/modelconfig/capability.go
Comment thread pkg/modelconfig/capability.go
@beiguo218

Copy link
Copy Markdown
Collaborator

We decided to make the current changes as simple as possible:

  1. Just in nemotronHNanoRule, instead of directly adding CapabilityAudioTextToText and CapabilityVideoTextToText, we do a further check for the model type - only add those 2 capabilities when it comes to nemotronh nano omni model;
  2. Keep the current rule order and omniRule unchanged.

@caihui-lol

Copy link
Copy Markdown
Collaborator Author

update nemotron rule to append audioTextToText/videoTextToText when it matches nemotron_nano_omni, requesting review again.

@caihui-lol

Copy link
Copy Markdown
Collaborator Author

Hi @slin1237 can i get one more review on this PR, thanks so much!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

api API/Types changes in pkg/apis tests Test changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants