Skip to content

[fix] Render multimodal sample prompts under the base model (#1860)#1907

Open
sunnyadn wants to merge 1 commit into
NovaSky-AI:mainfrom
sunnyadn:fix-1860-render-lora-404
Open

[fix] Render multimodal sample prompts under the base model (#1860)#1907
sunnyadn wants to merge 1 commit into
NovaSky-AI:mainfrom
sunnyadn:fix-1860-render-lora-404

Conversation

@sunnyadn

Copy link
Copy Markdown

What does this PR do?

Fixes #1860.

_render_for_sample posts the resolved model to /v1/chat/completions/render. Under uses_lora_weight_sync=True the resolved model is the LoRA adapter name, which the render endpoint does not resolve, so every image-bearing Tinker sample request 404s. Following @benrhodes26's diagnosis in #1860, the render request now always targets the base model (self.model_name): rendering (chat-template application, tokenization, multimodal preprocessing) is adapter-independent. Generation still targets the adapter on /inference/v1/generate.

The model parameter of _render_for_sample becomes unused and is removed.

Tests

New test_sample_with_image_renders_under_base_model: image chunks + uses_lora_weight_sync=True + explicit adapter model, asserts the render sub-request targets the base model while generate targets the adapter. On main it fails with AssertionError: assert 'lora-adapter' == 'base-model'; with this fix the full file passes (57 passed, pytest tests/backends/skyrl_train/inference_servers/test_remote_inference_client.py -m "not vllm").

Also verified live against vLLM 0.23.0 (the version pinned in pyproject.toml) on an L40S, serving Qwen/Qwen2.5-VL-3B-Instruct with a LoRA adapter via --enable-lora --lora-modules:

  • POST /v1/chat/completions/render with the adapter name returns 404 (The model 'lora-test' does not exist), even though /v1/models lists the adapter.
  • The same request with the base model name returns 200 with prompt token ids and image placeholder features.
  • POST /v1/chat/completions with an image under the adapter name returns 200, confirming generation resolves adapters while render does not.

On the parity point raised in #1860 (base-model render output matching what the adapter would produce): in vLLM 0.23.0 this holds by construction. The render module (vllm/entrypoints/serve/render/) has no LoRA code path, and its model check goes through OpenAIModelRegistry, documented as having "no engine client and no LoRA support". Adapters also cannot alter tokenization in this version (LoRAConfig has no extra-vocab field), so there is no adapter-dependent render output to diverge from.

…AI#1860)

vLLM's /v1/chat/completions/render does not resolve LoRA adapter names,
so _render_for_sample 404s for any image-bearing prompt when LoRA is in
use. Rendering is adapter-independent, so always target the base model;
generation still targets the adapter on /inference/v1/generate.
@gemini-code-assist

Copy link
Copy Markdown
Contributor

Warning

Gemini encountered an error creating the review. You can try again by commenting /gemini review.

@sunnyadn sunnyadn closed this Jul 16, 2026
@sunnyadn sunnyadn reopened this Jul 16, 2026
@sunnyadn sunnyadn closed this Jul 16, 2026
@sunnyadn sunnyadn reopened this Jul 16, 2026
@sunnyadn sunnyadn closed this Jul 16, 2026
@sunnyadn sunnyadn reopened this Jul 16, 2026
@sunnyadn sunnyadn closed this Jul 16, 2026
@sunnyadn sunnyadn reopened this Jul 16, 2026
@sunnyadn sunnyadn closed this Jul 16, 2026
@sunnyadn sunnyadn reopened this Jul 16, 2026
@sunnyadn sunnyadn closed this Jul 16, 2026
@sunnyadn sunnyadn reopened this Jul 16, 2026
@sunnyadn sunnyadn closed this Jul 17, 2026
@sunnyadn sunnyadn reopened this Jul 17, 2026
@sunnyadn

Copy link
Copy Markdown
Author

The close/reopen pairs on the timeline were CI retriggers during the July 16 GitHub incident; no code changed between them. Current status: all test jobs pass, and the two remaining red checks (skyrl_gpu_tests, Vercel) fail because fork PRs cannot access ANYSCALE_CLI_TOKEN and the deploy secrets.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Multimodal Tinker sampling with LoRA 404s: _render_for_sample posts the adapter name to /v1/chat/completions/render

1 participant