Add lumen-gi-enabled / lumen-reflections-enabled camera capture settings (captures match the viewport by default) - #181
Conversation
…ings UE scene captures don't render with the global illumination / reflection method the viewport uses: a capture view seeds its method from the project CVar but never blends level PostProcessVolumes, and on hardware-ray-tracing projects its rays have no acceleration-structure data unless the component opts in — so a Lumen-lit level comes out flat and washed out in captured images, with all indirect light missing. Since UE 5.5 the engine supports Lumen in scene captures when the capture's post-process settings resolve to Lumen (the renderer then maintains a dedicated Lumen scene per capture's persistent view state, which UnrealCamera already enables via bAlwaysPersistRenderingState). Add two per-capture tri-state settings, keyed off whether the jsonc key is present: - absent (default): match the viewport. Scene (RGB) captures mirror the project's r.DynamicGlobalIlluminationMethod / r.ReflectionMethod, so Lumen projects get Lumen captures out of the box; every other image type resolves to None — their output is a replacement material, so dynamic GI underneath is pure GPU/VRAM waste (and previously they silently inherited the project CVar). - lumen-gi-enabled / lumen-reflections-enabled = true: force Lumen for this capture regardless of the project default. - false: force the method to None. This is an active override — an un-overridden capture inherits the project CVar, so omitting the override is not a way to turn Lumen off. The resolved methods are pinned as explicit post-process overrides, and bUseRayTracingIfEnabled is set whenever either method resolves to Lumen so hardware-ray-traced Lumen works when the project uses it (software Lumen works without). On engines older than 5.5 the overrides are ignored for captures. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Nice feature! Thanks for your contributions @andrewjong ! A few comments:
"lumen-gi-enabled": { "type": "boolean" },
"lumen-reflections-enabled": { "type": "boolean" }
Regarding changing default cameras behavior: @jonyMarino this is your call. |
|
Will do! I'm currently focusing on ICRA so probably this will come late September. |
|
Glad to hear. Are you going to present a work using Project AirSim? |
|
Yes possibly! Would you have a recommended bibtex citation? |
…PV-caveat docs Review follow-up for iamaisim#181: - Compile the lumen resolver and both apply sites only when UE_SUPPORTS_LUMEN_SCENE_CAPTURES (UE 5.5+, the version that introduced Lumen for scene captures). Older engines hard-disable Lumen for capture views; guarding keeps their capture behavior exactly as before this feature — a no-op by construction rather than by argument. - Document the Post Process Volume limitation: capture views never blend PPVs, so the default mirrors the project's *default* methods (r.DynamicGlobalIlluminationMethod / r.ReflectionMethod), not a viewport whose method comes from a bounded or unbound volume. Levels driving GI through PPVs should set lumen-gi-enabled / lumen-reflections-enabled explicitly. Docs reworded from "match viewport" to "match project setting" accordingly. - Add the two settings to the client robot-config schema as booleans. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
From Claude Fable: UE 5.2 / version guard — I don't have a UE 5.2 engine install to test on, so rather than argue it's a no-op I made it one by construction, as you suggested: the resolver and both apply sites are compiled only under a new Post Process Volumes — agreed, and this matches what we found reading the capture path: capture views only apply the component's own Schema — added exactly as suggested to "lumen-gi-enabled": { "type": "boolean" },
"lumen-reflections-enabled": { "type": "boolean" }Non-static camera benchmark — measured on the same UE 5.7.4 / Vulkan SM6 QA-Office setup as the PR screenshots (RTX 4090, otherwise idle GPU): a non-physics robot with two 640×480 RGB (+2 depth) captures at a 0.1 s capture interval, static vs continuously moving (a ~253 s recorded indoor tour replayed through SetPose at walking pace). Each cell is a 300-frame
So in this workload camera motion did not amplify the Lumen premium — at an on-demand ~6 Hz capture cadence, walking-pace surface-cache invalidation isn't the bottleneck (both arms are actually faster while moving because the static spawn pose faces the expensive window-vista view; only the within-row on/off deltas are meaningful). Caveats for generalizing: 640×480 captures at ~6 Hz effective cadence rather than a 60 fps game loop, and walking-pace motion — a fast-flying vehicle or high-rate/high-resolution captures could invalidate more cache per frame, which is part of why the settings stay per-capture so any camera can opt out with 🤖 Generated with Claude Code |
3 participants
About
TL;DR adds Lumen support to AirSim camera images.
Camera images captured through
USceneCaptureComponent2Ddon't render with the global illumination / reflection method the viewport uses: a capture view seeds its method from the project CVars but never blends level PostProcessVolumes, and on hardware-ray-tracing projects the capture's rays have no acceleration-structure data unless the component opts in. The practical result is that a Lumen-lit level comes out flat and washed out in the sim's camera images — all indirect light missing — even though the game/editor viewport looks correct (see screenshots below).Since UE 5.5 the engine natively supports Lumen in scene captures: when a capture's post-process settings resolve to Lumen, the renderer maintains a dedicated Lumen scene for the capture's persistent view state (which
UnrealCameraalready enables viabAlwaysPersistRenderingState).This PR adds two per-capture tri-state settings, documented in
docs/sensors/camera_capture_settings.md:r.DynamicGlobalIlluminationMethod/r.ReflectionMethod, so Lumen projects get Lumen camera images out of the box. Every other image type (depth, segmentation, ...) resolves to None — their output comes from a replacement material, and previously they silently inherited the project CVar, paying Lumen cost for nothing.lumen-gi-enabled/lumen-reflections-enabled=true: force Lumen for that capture regardless of the project default.false: force the method to None. This is an active override — an un-overridden capture inherits the project CVar, so omitting the override is not an off switch.The resolved methods are pinned as explicit post-process overrides on each capture, and
bUseRayTracingIfEnabledis set whenever either method resolves to Lumen so hardware-ray-traced Lumen works (software Lumen works without it). The RGB wire format is unchanged (SCS_FinalColorLDR, 8-bit — in UE 5.5+ it runs the full viewport film tone curve). On engines older than 5.5 the overrides are ignored for scene captures, so this is a no-op there.Cost note: each Lumen-enabled capture maintains its own Lumen scene (GPU time + VRAM) — numbers below; set the knobs to
falseper capture to opt out where frame rate matters more than fidelity.How Has This Been Tested?
UE 5.7.4, Vulkan SM6, Linux (Epic
unreal-engine:dev-5.7.4container), photorealistic office scene (marketplace "QA Office" level, Lumen GI + reflections + HWRT in project settings), non-physics robot with two 640×480 RGB+depth-planar cameras at a 0.1 s capture interval:false→ matches the pre-PR captures (verified visually and by luminance statistics);true→ forces Lumen. The on/off pair was verified at four camera poses, the full tri-state matrix at one.Screenshots and videos:
Lobby — ceiling receives no direct light, only GI bounce; the marble floor gains reflections:
Lounge — ceiling, brass stanchions, chrome table legs:
Corridor — the trash can turns from flat matte to brushed metal with environment reflections:
Game-viewport reference at the same pose as the corridor pair — the Lumen-on capture matches its lighting (the yellow on-screen text is an unrelated engine debug message, viewport-only):
🤖 Generated with Claude Code