Skip to content

Update display size scale tests with video frame from video file #4614

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
haoxli wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Update display size scale tests with video frame from video file #4614

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
147 changes: 86 additions & 61 deletions src/webgpu/web_platform/external_texture/video.spec.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,6 @@ Tests for external textures from HTMLVideoElement (and other video-type sources?
TODO: consider whether external_texture and copyToTexture video tests should be in the same file
TODO(#3193): Test video in BT.2020 color space
TODO(#4364): Test camera capture with copyExternalImageToTexture (not necessarily in this file)
TODO(#4605): Test importExternalTexture with video frame display size different with coded size from
a video file
`;

import { makeTestGroup } from '../../../common/framework/test_group.js';
Expand All @@ -34,6 +32,14 @@ const kHeight = 16;
const kWidth = 16;
const kFormat = 'rgba8unorm';

const kDisplayScaleVideoNames = [
'four-colors-h264-bt601.mp4',
'four-colors-vp9-bt601.webm',
'four-colors-vp9-bt709.webm',
] as const;
type DisplayScale = 'smaller' | 'same' | 'larger';
const kDisplayScales: DisplayScale[] = ['smaller', 'same', 'larger'];
Copy link
Copy Markdown
Contributor

@shaoboyan091 shaoboyan091 Mar 19, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If thit won't expand in future then I slightly lean to keep the origin format (Using 'smaller' | 'same' | 'larger' in all three places).


export const g = makeTestGroup(TextureUploadingUtils);

function createExternalTextureSamplingTestPipeline(
Expand Down Expand Up @@ -164,67 +170,55 @@ function checkNonStandardIsZeroCopyIfAvailable(): { checkNonStandardIsZeroCopy?:
}
}

function createVideoFrameWithDisplayScale(
async function createVideoFrameWithDisplayScale(
Copy link
Copy Markdown
Contributor

@shaoboyan091 shaoboyan091 Mar 19, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would you mind to add comments similar to your commit message to simple describe we need real video file instead of canvas created ones?

t: GPUTest,
displayScale: 'smaller' | 'same' | 'larger'
): { frame: VideoFrame; displayWidth: number; displayHeight: number } {
const canvas = createCanvas(t, 'onscreen', kWidth, kHeight);
const canvasContext = canvas.getContext('2d');
videoName: (typeof kDisplayScaleVideoNames)[number],
displayScale: DisplayScale
): Promise<VideoFrame> {
let sourceFrame: VideoFrame | undefined;
let codedWidth: number | undefined;
let codedHeight: number | undefined;

const videoElement = getVideoElement(t, videoName);

await startPlayingAndWaitForVideo(videoElement, async () => {
const source = await getVideoFrameFromVideoElement(t, videoElement);
sourceFrame = source;
codedWidth = source.codedWidth;
codedHeight = source.codedHeight;
});

if (canvasContext === null) {
t.skip(' onscreen canvas 2d context not available');
if (sourceFrame === undefined || codedWidth === undefined || codedHeight === undefined) {
Copy link
Copy Markdown
Contributor

@shaoboyan091 shaoboyan091 Mar 19, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Using assert? Or skip with correct info

unreachable();
}

const ctx = canvasContext;

const rectWidth = Math.floor(kWidth / 2);
const rectHeight = Math.floor(kHeight / 2);

// Red
ctx.fillStyle = `rgba(255, 0, 0, 1.0)`;
ctx.fillRect(0, 0, rectWidth, rectHeight);
// Lime
ctx.fillStyle = `rgba(0, 255, 0, 1.0)`;
ctx.fillRect(rectWidth, 0, kWidth - rectWidth, rectHeight);
// Blue
ctx.fillStyle = `rgba(0, 0, 255, 1.0)`;
ctx.fillRect(0, rectHeight, rectWidth, kHeight - rectHeight);
// Fuchsia
ctx.fillStyle = `rgba(255, 0, 255, 1.0)`;
ctx.fillRect(rectWidth, rectHeight, kWidth - rectWidth, kHeight - rectHeight);

const imageData = ctx.getImageData(0, 0, kWidth, kHeight);

let displayWidth = kWidth;
let displayHeight = kHeight;
let displayWidth = codedWidth;
Copy link
Copy Markdown
Contributor

@shaoboyan091 shaoboyan091 Mar 19, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or just assign 0 instead of codedWidth.

let displayHeight = codedHeight;
switch (displayScale) {
case 'smaller':
displayWidth = Math.floor(kWidth / 2);
displayHeight = Math.floor(kHeight / 2);
displayWidth = Math.floor(codedWidth / 2);
displayHeight = Math.floor(codedHeight / 2);
break;
case 'same':
displayWidth = kWidth;
displayHeight = kHeight;
displayWidth = codedWidth;
displayHeight = codedHeight;
break;
case 'larger':
displayWidth = kWidth * 2;
displayHeight = kHeight * 2;
displayWidth = codedWidth * 2;
displayHeight = codedHeight * 2;
break;
default:
unreachable();
}

const frameInit: VideoFrameBufferInit = {
format: 'RGBA',
codedWidth: kWidth,
codedHeight: kHeight,
const frame = new VideoFrame(sourceFrame, {
Copy link
Copy Markdown
Collaborator

@kainino0x kainino0x Mar 19, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a way to encode this data in a video file rather than going through VideoFrame? Doing it with VideoFrame seems kind of contrived - the way people would actually run into issues in the wild is through video files that have different dimensions encoded in them.

Copy link
Copy Markdown
Contributor Author

@haoxli haoxli Mar 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It doesn't seem easy to create a video file with display size scale. I see we can use setsar=2/1 to make video display size becomes 2 times.

ffmpeg.exe -loop 1 -i .\four-colors.png -c:v libx264 -pix_fmt yuv420p -frames 50 -colorspace smpte170m -color_primaries smpte170m -color_trc smpte170m -color_range tv -vf "setsar=2/1" four-colors-h264-bt601-sar2.mp4

The display height becomes two times of coded width, but the width does not, and the four colored squares are still displayed according to the coded size.
Image

If I force DAR using setsar=2/1,setdar=4/3, the SAR will be 1:1.
ffmpeg.exe -loop 1 -i .\four-colors.png -c:v libx264 -pix_fmt yuv420p -frames 50 -colorspace smpte170m -color_primaries smpte170m -color_trc smpte170m -color_range tv -vf "setsar=2/1,setdar=4/3" four-colors-h264-bt601-sar2.mp4

Verify:
ffprobe.exe -v error -select_streams v:0 -show_entries stream=width,height,coded_width,coded_height,sample_aspect_ratio,display_aspect_ratio -of default=noprint_wrappers=1 .\four-colors-h264-bt601-sar2.mp4

width=320
height=240
coded_width=320
coded_height=240
sample_aspect_ratio=1:1
display_aspect_ratio=4:3

I will further investigate how to make it works.

If it works, I think we need to add 6 new video files at least (3 different codes/color spaces videos x 2 display scales (larger/smaller).

Copy link
Copy Markdown
Collaborator

@kainino0x kainino0x Mar 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That should be fine as long as they're not too large. Thank you for investigating!

displayWidth,
displayHeight,
timestamp: 0,
};
timestamp: sourceFrame.timestamp,
});
sourceFrame.close();

const frame = new VideoFrame(imageData.data.buffer, frameInit);
return { frame, displayWidth, displayHeight };
return frame;
}

g.test('importExternalTexture,sample')
Expand Down Expand Up @@ -446,23 +440,29 @@ Tests that we can import an VideoFrame with non-YUV pixel format into a GPUExter
]);
});

g.test('importExternalTexture,video_frame_display_size_diff_with_coded_size')
g.test('importExternalTexture,video_frame_display_size_scale')
.desc(
`
Tests that we can import a VideoFrame with display size different with its coded size, and
Tests that we can import a VideoFrame with a display size different from its coded size, and
sampling works without validation errors.

For the importExternalTexture path with scaled video frame display size, we validate scaling
using the available codec and color space assets: VP9/H.264 and bt.601/bt.709.
`
)
.params(u =>
u //
.combine('displayScale', ['smaller', 'same', 'larger'] as const)
.combine('videoName', kDisplayScaleVideoNames)
.combine('displayScale', kDisplayScales)
)
.fn(t => {
.fn(async t => {
const { videoName, displayScale } = t.params;

if (typeof VideoFrame === 'undefined') {
t.skip('WebCodec is not supported');
}

const { frame } = createVideoFrameWithDisplayScale(t, t.params.displayScale);
const frame = await createVideoFrameWithDisplayScale(t, videoName, displayScale);

const colorAttachment = t.createTextureTracked({
format: kFormat,
Expand Down Expand Up @@ -496,11 +496,30 @@ sampling works without validation errors.
passEncoder.end();
t.device.queue.submit([commandEncoder.finish()]);

// Build expected sampled colors by drawing the same source frame to a 2D canvas.
// This makes the check robust across codecs/container metadata while still validating
// that importExternalTexture sampling matches browser video rendering behavior.
const canvas = createCanvas(t, 'onscreen', kWidth, kHeight);
Copy link
Copy Markdown
Contributor

@shaoboyan091 shaoboyan091 Mar 19, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just double check, are we using kWidth, kHeight instead of using codedWidth, codedHeight for some purpose?

The origin part use kWidth and kHeight due to the generated video frame is from a kWidth and kHeight (which mapping to codedWidth and codedHeight here) canvas.

Copy link
Copy Markdown
Contributor Author

@haoxli haoxli Mar 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's not entirely accurate. I only used kWidth and kHeight to avoid having to define an additional size variable here. We also can use displayWidth and displayHeight in VideoFrame for that. In the origin PR, I used canvas becuase I found it also can reporduce the issue and I can control the sizes easily.

Copy link
Copy Markdown
Collaborator

@kainino0x kainino0x Mar 19, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are too many different widths and heights. Can you rename kWidth and kHeight to something more specific? (kCanvasWidth kCanvasHeight)

Copy link
Copy Markdown
Contributor Author

@haoxli haoxli Mar 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, I will return displayWidth and displayHegith from createVideoFrameWithDisplayScale, and use them in the cases stead of kWidth and kHeight.

const canvasContext = canvas.getContext('2d', { colorSpace: 'srgb' });
if (canvasContext === null) {
frame.close();
t.skip(' onscreen canvas 2d context not available');
Copy link
Copy Markdown
Contributor

@shaoboyan091 shaoboyan091 Mar 19, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

leading space, pls remove.

}
const ctx = canvasContext as CanvasRenderingContext2D;
ctx.drawImage(frame, 0, 0, kWidth, kHeight);
const imageData = ctx.getImageData(0, 0, kWidth, kHeight, { colorSpace: 'srgb' });
const bytes = imageData.data;
const sample = (x: number, y: number): Uint8Array => {
const xi = Math.floor(x);
const yi = Math.floor(y);
const i = (yi * kWidth + xi) * 4;
return new Uint8Array([bytes[i + 0], bytes[i + 1], bytes[i + 2], bytes[i + 3]]);
};
const expected = {
topLeft: new Uint8Array([255, 0, 0, 255]),
topRight: new Uint8Array([0, 255, 0, 255]),
bottomLeft: new Uint8Array([0, 0, 255, 255]),
bottomRight: new Uint8Array([255, 0, 255, 255]),
topLeft: sample(kWidth * 0.25, kHeight * 0.25),
topRight: sample(kWidth * 0.75, kHeight * 0.25),
bottomLeft: sample(kWidth * 0.25, kHeight * 0.75),
bottomRight: sample(kWidth * 0.75, kHeight * 0.75),
};

ttu.expectSinglePixelComparisonsAreOkInTexture(t, { texture: colorAttachment }, [
Expand Down Expand Up @@ -533,21 +552,24 @@ g.test('importExternalTexture,video_frame_display_size_from_textureDimensions')
.desc(
`
Tests that textureDimensions() for texture_external matches VideoFrame display size.

For the importExternalTexture path with scaled video frame display size, we validate scaling
using the available codec and color space assets: VP9/H.264 and bt.601/bt.709.
`
)
.params(u =>
u //
.combine('displayScale', ['smaller', 'same', 'larger'] as const)
.combine('videoName', kDisplayScaleVideoNames)
.combine('displayScale', kDisplayScales)
)
.fn(t => {
.fn(async t => {
const { videoName, displayScale } = t.params;

if (typeof VideoFrame === 'undefined') {
t.skip('WebCodec is not supported');
}

const { frame, displayWidth, displayHeight } = createVideoFrameWithDisplayScale(
t,
t.params.displayScale
);
const frame = await createVideoFrameWithDisplayScale(t, videoName, displayScale);

const externalTexture = t.device.importExternalTexture({
source: frame,
Expand Down Expand Up @@ -594,7 +616,10 @@ Tests that textureDimensions() for texture_external matches VideoFrame display s
pass.end();
t.device.queue.submit([encoder.finish()]);

t.expectGPUBufferValuesEqual(storageBuffer, new Uint32Array([displayWidth, displayHeight]));
t.expectGPUBufferValuesEqual(
storageBuffer,
new Uint32Array([frame.displayWidth, frame.displayHeight])
);

frame.close();
});
Expand Down
Loading