Spears & Munsil 4K UHD HDR Benchmark (Test Discs) May 15, 2023

[Keenan]

Originally Posted by SeeMoreDigital

Shame Amazon doesn't offer bit-rate info on their streaming service like Netflix...

There is something you can use, it's the Developer Tools Menu app. There are some images of it in use below. You'll see an initial buffer filling download and then it averages out to around 15-16 Mb/s for UHD content. The images are from "The Expanse".

I don't know why that last image is sideways, sorry about that.

[Stacey Spears]

Originally Posted by Keenan

There is something you can use, it's the Developer Tools Menu. There are some images of it in use below. You'll see an initial buffer filling download and then it averages out to around 15-16 Mb/s for UHD content. The images are from "The Expanse".

I don't know why that last image is sideways, sorry about that.

The Expanse looked good in HDR on the LG.

Thank you for sharing the images!

[Keenan]

Originally Posted by Stacey Spears

The Expanse looked good in HDR on the LG.

Thank you for sharing the images!

It does look, I'm using a DeWayne Davis calibrated Sony A9F myself.

[Stacey Spears]

Originally Posted by Balling

Netflix does use DCI-P3 or Display P3 without any BT.2020 container in-house and HDMI 2.0 does have DCI-P3 mode in EDID colorimetry block that can then be further clarified whether it is DCI-P3 or display p3 in HDMI Gamut metadata. Consumers do not have access to that, but if they got it it is supported (just like xvYCC extended gamut that is always limited range and sYCC YCBCr format with selectable quantisation, yes, YCbCr can also be full range in HDMI, ha, in Nvidia control panel digital format menu you can select it).

Next. You can use any chroma_loc you want. And there are not just 2 of em. There are six. HEVC does specify all of them and they are supported by Mediainfo (it writes 4:2:0 (Type 2), for example). We in ffmpeg also added support for all of them. Dolby Vision does redefine BT.2020 spec's chroma sitting at least for IPTPQc2 color (that is profile 5, not HDR10 compatible and it should be full range).

Hopefully this does not get posted twice. I had responded earlier, but then it said something about admin approval.

Has the 3x3 matrix been published for using P3 YCbCr? I can derive the numbers, but unless there is a spec, no two companies will convert back to RGB the same. I mean, they have trouble even when there is a spec. This is why I pushed back against supporting P3 over HDMI, when asked. I believe it is just one more thing to get wrong.

Yes, lots of chroma_loc options, but decoder vendors tend to support the minimal amount possible. As I mentioned, in my testing, they don't seem to trigger off the chroma_loc flag, but rather the color space. I would not be surprised if some just trigger off transfer function and assume HDR is chroma_loc 2 and SDR is chroma_loc 0. Xbox Series X still does this wrong. They are looking at it again after Vincent's video, so hopefully they fix it this time. (Been broken on Xbox One S/X) That reminds me, I owe them some test material for it.

There is a streaming provider that was using 709 down scaling, but flagged as chroma_loc 2 because 2020 said to use it. They did this at the time because all of the TVs used 709 up scaling. A recent post on Doom9 suggested they may still do something like this.

Because Dolby Vision on Blu-ray is only optional, there must be a proper HDR10 base layer in YCbCr 4:2:0. They can't support both chroma_loc 0 and 2 in the same file and Dolby's own tool uses chroma_loc 2 on the down scale.

FYI, Dolby informed me this issue has been logged. But filing a bug and shipping a fix are two very different things.

Sorry for being such a pessimist. That is from years of disappointment. :-)

[Pagey123]

Wow, loving this thread! I had no idea about the chroma_loc issue, but I can see how software or hardware interpreting that incorrectly could cause some nasty issues during playback/decode. Without knowing more about the process/procedures behind such a decision, I'll assume it's one of those "it sounded good on paper" things, lol. Having worked in IT/information security in banking for 20 years, I have come to appreciate the difference between theory (but it sounded good on paper!) and practice (it's broke, now you and your team just get it fixed)!

[Stacey Spears]

Originally Posted by Pagey123

Wow, loving this thread! I had no idea about the chroma_loc issue, but I can see how software or hardware interpreting that incorrectly could cause some nasty issues during playback/decode. Without knowing more about the process/procedures behind such a decision, I'll assume it's one of those "it sounded good on paper" things, lol. Having worked in IT/information security in banking for 20 years, I have come to appreciate the difference between theory (but it sounded good on paper!) and practice (it's broke, now you and your team just get it fixed)!

I am told they switched to chroma_loc 2 in BT.2020 because they thought it would reduce multi generation (4:2:2 <-> 4:2:0) loss.

[Pagey123]

Originally Posted by Stacey Spears

I am told they switched to chroma_loc 2 in BT.2020 because they thought it would reduce multi generation (4:2:2 <-> 4:2:0) loss.

I wonder if, in our lifetimes, we'll have the bandwidth and storage space necessary to simply eliminate chroma subsampling. Or maybe because of the human visual system's bias for luminance, we'll simply always compress the color component(s)...

[nathan_h]

Glad to be wrong!

Thanks for explaining it, discovering / making tests to detect it.

Makes me less sad that my Sony 800 version 1 player doesn’t even have DV.

Originally Posted by Stacey Spears

You are reading incorrectly.

Dolby and HDR10 are the same in terms of color volume. The issue here is strictly the conversion from 4:2:0 into 4:2:2 or chroma up scaling. This is a playback bug. Please read my post from 7:42 AM where I mention three different 709 vs. 2020 discussions that are possible and which one we are discussing here. Your confusion is understandable since those that wrote the 2020 doc decided to incorporate 4:2:0 pixel layout into it.

My comments are also just for BD. Since I don't have my patterns on a streaming service, I can't tell you how they perform on the playback side.

There are many different chroma layouts in 4:2:0. DVD and BD are assumed to always use chroma_loc 0. UHD BD is expected to support both chroma_loc 0 and chroma_loc 2. For DVD (601), the progressive layout is what was adopted where chroma is co-sited horizontally and interstitial vertically. This is chroma_loc 0. This did not change when we moved to HD DVD and Blu-ray (709). The fact that they decided to make this change when they introduced 2020 has been a mess. Many older TVs use 709 up scaling for 2020 content if you play a file from USB. BD was the first format to get this right, at least in some players and scenarios.



For the new disc, we have patterns to test this in HDR10, HDR10+ and Dolby Vision (all 2020) for HD and UHD resolutions. (all chroma_loc 2) And SDR in both 709 (chroma_loc 0) and 2020 (chroma_loc 2) in HD and UHD resolutions. There are a lot of ways to get this wrong, which is why we are providing so many versions for people to test with. In my limited testing, the players seem to ignore the chroma_loc flag and instead decide which method to use based on the color space flag.

[nathan_h]

If I am following this right the issue is something that needs to be fixed in players, not displays?

Or it depends on which version of DV is in use?

I thought the chroma up sampling occurred in the display (unless one forces ones player to do it) so that would be the place to implement a fix?

[Stacey Spears]

Originally Posted by nathan_h

If I am following this right the issue is something that needs to be fixed in players, not displays?

Or it depends on which version of DV is in use?

I thought the chroma up sampling occurred in the display (unless one forces ones player to do it) so that would be the place to implement a fix?

4:2:0 to 4:2:2 "usually" occurs in the player. For Dolby Vision, it always occurs in the player.

4:2:2 to 4:4:4 can occur in the player or display. For Dolby Vision, it always occurs in the display.

Dolby Vision is always transmitted at 12-bit 4:2:2. In player-led, it is output as HDR12 (12-bit 4:2:2). In TV-led, it is output as 8-bit RGB, but is really 12-bit 4:2:2. Same number of bytes. It was a way to push Dolby Vision through an AVR w/o it being messed up because 8-bit RGB is required by all HDMI transmitters and receivers. There is a special VSIF to signal Dolby Vision over HDMI.

To your point, this needs to be fixed in the player. Dolby needs to fix it in their BD SDK. Then the player manufacturers need to ship a new FW with the new SDK incorporated. And any 3rd party in between Dolby and the player manufacturer. e.g. MediaTek would need to provide an update to anyone using their part.

[nathan_h]

Originally Posted by Pagey123

I wonder if, in our lifetimes, we'll have the bandwidth and storage space necessary to simply eliminate chroma subsampling. Or maybe because of the human visual system's bias for luminance, we'll simply always compress the color component(s)...

Well wasn’t it Joe Kane who said that they should have used 444 for DVD and everything after because it is actually easier to compress and ends up smaller than 420 when using digital encoding?

That is, digital codecs are so much more efficient, even mpeg, that 444 ends up smaller than 420.

But people were so used to analog rules / bandwidth / tricks that they didn’t stop to compare?

[nathan_h]

Originally Posted by Stacey Spears

4:2:0 to 4:2:2 "usually" occurs in the player. For Dolby Vision, it always occurs in the player.

4:2:2 to 4:4:4 can occur in the player or display.

So broken displays could be okay to use if fed with a player that creates the correct 444 output?

Ah you edited your post so I can see that my amateur logic is wrong but the conclusion is similar in that players can be updated and that would fix this.

[Stacey Spears]

Originally Posted by nathan_h

Well wasn’t it Joe Kane who said that they should have used 444 for DVD and everything after because it is actually easier to compress and ends up smaller than 420 when using digital encoding?

That is, digital codecs are so much more efficient, even mpeg, that 444 ends up smaller than 420.

But people were so used to analog rules / bandwidth / tricks that they didn’t stop to compare?

4:2:0 is smaller than 4:4:4, but the difference is not that great when compressed. Joe is pushing for 16-bit half float to the display.

[Stacey Spears]

Originally Posted by nathan_h

So broken displays could be okay to use if fed with a player that creates the correct 444 output?

Ah you edited your post so I can see that my amateur logic is wrong but the conclusion is similar in that players can be updated and that would fix this.

It is a player fix since the issue is with the 4:2:0 to 4:2:2 part.

Sorry for the constant edits.

[Keenan]

Originally Posted by Stacey Spears

To your point, this needs to be fixed in the player. Dolby needs to fix it in their BD SDK. Then the player manufacturers need to ship a new FW with the new SDK incorporated. And any 3rd party in between Dolby and the player manufacturer. e.g. MediaTek would need to provide an update to anyone using their part.

So as you noted earlier, all this will happen, maybe, someday, down the road, we hope.

[Balling]

Originally Posted by Stacey Spears

Has the 3x3 matrix been published for using P3 YCbCr? I can derive the numbers, but unless there is a spec, no two companies will convert back to RGB the same.

There is no matrix. It is RGB mode and uses JPEG 2000 in files. And if you mean matrix to XYZ, it is in the standards. You do understand that to XYZ is done after linearisation?

Obviously the matrix to YCbCr would be the same as in BT.601. It is the standard matrix for sYCC and all. But it is not suitable for HDR and wide-gamut. You must use ICTCP with CL and CI.

You can use medianfo on this file to check it out.
https://trac.ffmpeg.org/raw-attachme...9d790b_cut.mxf

It is Display P3 (thus D65 adapted), uses PQ and is RGB with 24.000 frame rate.

Of course this method is not the only one Netflix uses. JPEG2000 is already outdated. There is lossless AVC/HEVC/VP9/AV1 after all and what not. JPEG 2000 is only good for XYZ data, because the use of Identity matrix for mp4 /mkv is just strange.

[Stacey Spears]

Originally Posted by Balling

There is no matrix. It is RGB mode and uses JPEG 2000 in files. And if you mean matrix to XYZ, it is in the standards. You do understand that to XYZ is done after linearisation?

Obviously the matrix to YCbCr would be the same as in BT.601. It is the standard matrix for sYCC and all. But it is not suitable for HDR and wide-gamut. You must use ICTCP with CL and CI.

You can use medianfo on this file to check it out.
hxxps://trac.ffmpeg.org/raw-attachment/ticket/9145/VIDEO_e4da5fcd-5ffc-4713-bcdd-95ea579d790b_cut.mxf

It is Display P3 (thus D65 adapted), uses PQ and is RGB with 24.000 frame rate.

Of course this method is not the only one Netflix uses. JPEG2000 is already outdated. There is lossless AVC/HEVC/VP9/AV1 after all and what not. JPEG 2000 is only good for XYZ data, because the use of Identity matrix for mp4 /mkv is just strange.

I think you misunderstood what I asked. Please let me try and clarify.

You had mentioned using P3 over HDMI. (P3-D65 or DCI-P3) That could be either YCbCr or RGB. If YCbCr, then you need a 3x3 matrix on the source side to convert from RGB into YCbCr, for transmission over HDMI, and then the inverted 3x3 to convert YCbCr back into RGB on the display side. If you are saying only RGB is used for P3, I would be surprised since the first thing a consumer TV does with RGB is to convert it into YCbCr.

I was simply asking if someone, like the ITU or SMPTE, has published a doc with the 3x3 matrix that should be used for P3.

The BT.601 3x3 matrix would not be appropriate for P3 since it is a larger color gamut and the 3x3 matrix is usually derived knowing the gamut. The only thing that should be using the BT.601 3x3 matrix is standard definition video such as NTSC and PAL.

You would also not be sending J2K over the wire. J2K works on more than just XYZ. It has been used on camera RAW BAYER (Green1, Green2, Red and Blue) sensor data, which is not in XYZ. It also supports YCbCr and RGB among others.

Most HDR is delivered in YCbCr, not ICtCp. Blu-ray does not support ICtCp at all, which is too bad. I would be surprised if any non-Dolby Vision HDR content is in ICtCp.

And as far as frame rates, both 23.976 and 24.0 are used. 23.976 is still the dominant format, but more and more 24.0 is being used. Apple TV seems to finally be supporting true 24. Hopefully Xbox Series X is next. Same for 59.94 vs. 60.0.

AVC/HEVC/VP9/AV1 are not appropriate for a mezzanine format. All of those are really delivery CODECs. A lot of effort was put into IMF for that purpose, which uses J2K. DCPs also use J2K.

[Stacey Spears]

Originally Posted by Keenan

So as you noted earlier, all this will happen, maybe, someday, down the road, we hope.

Tyler and I plan weekly videos later this year to go over the disc in more detail. We also plan to interview various people in the industry. My goal is "The View" for video. Probably won't reach that level, but we must have something to aim for! And yes, I have seen too many episodes of the View and the Talk during the pandemic.

Once the videos start, hopefully everyone will suggest topics they would like to see covered in future videos. We will not be doing reviews. We will teach people how to use the patterns to conduct reviews, but we will not get into that business. Been there, done that. Much more fun to get stuff fixed than to talk about how veils have been lifted.

I am hoping that I can convince Vincent to produce a video on the issue. He recently did a video on player vs. TV-led.

Everyone has someone they listen to. e.g. At one point VIZIO held CNET in the highest regard. If David K. said something, they would listen. Just need to find that person that Dolby follows.

Here is what we did for the chroma bug. First we went to player manufactures. They did not seem to care. Then we went to decoder vendors. Same thing. Then we published the article. At that point, Toshiba had actual customers calling and demanding their money back. Boy were they upset with us! Pioneer had similar feelings in that regard. They had plenty of notice to fix it. You know who sent us the most hate mail? Pioneer DVD player owners!

[Stacey Spears]

For those interested, here is the menu mock-up for disc 1. I like to use PowerPoint for the mock-ups because it is easy to draw boxes. David takes my mock-ups and converts them into actual menus. Thank you David!

Feel free to ask questions. I will do my best to answer them. We spent more time trying to better organize and name the patterns.

Almost every pattern has been tweaked. In the most basic case, we improved the anti-aliasing of text. Before we used a fixed transfer function for text. We anti-alias in linear light. Now we use 2084 for HDR and Gamma 2.4 for SDR, when we convert back to perceptual space. This won't mean much to most, but its a small detail we care about.

We added one new element to the scaling pattern. This was to look for moiré introduced by the anti-ringing filter (Edge) in the Panasonic BD player. This was from feedback from this forum.

On the color space evaluation pattern, we added a small section of color bars above the color space conversion check for those lucky enough to have a green only mode. This is to also check color decoding. These are the four colors specifically for use in green only mode.

We modified the ADL patterns by making sure no boxes were to the left, right top or bottom of the center box, which can cause streaking. Again, feedback from a user on this forum. That same user is also provided a lot of input for the hue shift pattern. And all of the Aspect Ratio patterns are because of his feedback.

We added the 24.0 Stock Ticker a couple weeks ago after Vincent's video on the Xbox Series X not supporting 24.0.

The Motion and Motion HFR (59.94) content takes up over half the disc space of the BD100.

The Peak Luminance pattern is based on Kunkel, T. and Daly, S. (2020), 57‐1: Spatiotemporal Noise Targets Inspired by Natural Imagery Statistics. SID Symposium Digest of Technical Papers, 51: 842-845. https://onlinelibrary.wiley.com/doi/...01?cookieSet=1. We are hoping this more accurately measures real-world peak luminance capabilities of a display. It also has the ability to measure rise-time, if you have a meter and software that supports it, such as a Klein.

The pattern is one minute and 10-seconds long. Just below the auto dimming time of an LG OLED. The first 5-seconds have a black circle in the middle that counts down in SMPTE timecode. Then a white circle for one minute and then black again for 5-seconds. The black to white transition is to measure rise-time.

The idea is you use software like Calman, that can measure and graph over time with a fast meter like Klein. Then you can see how the peak luminance fluctuates over the 60-seconds as the background image simulates real-world content. The center circle is always at 100%. The synthetic background goes up to the peak of the nit level you selected. The average signal level is 10 cd/m². The circle is well below 10% area. We made it as small as we could so that a Klein K10 would be covered on a 48" display.

There are some UHDA certified displays that can measure 1000 cd/m² in the UDHA cert, but never go over 600 nits on real-world content. We are hoping this can help show more real-world performance numbers. That is the theory!

On the last two discs we had used the Placebo preset in x264 and x265. I had noticed that in some cases the VerySlow preset produced more accurate results. (closer to lossless) I eventually figured out which settings was responsible for the difference. In the end, we encoded every test pattern 96 times to find the settings closest to lossless / lossless. This resulted in it taking a few months to actually encode the 4500 patterns. I would encode 96 times, then do a binary compare of the source to the decoded encode. Then sort and use those settings. I never want to do that again! LOL I wish Blu-ray supported true lossless encoding. Every pattern has its on spreadsheet with all the data in it.

[Balling]

Originally Posted by Stacey Spears

You had mentioned using P3 over HDMI. (P3-D65 or DCI-P3) That could be either YCbCr or RGB.

No, it cannot be. Every digital format in HDMI is either YCC (YCbCr) or RGB. DCI-P3 and Display P3 are only RGB. The difference is quite obvious, for 8 bit YCC (YCbCr) 0 and 255 are reserved for Hsync while in RGB modes no values are reserved.

Just FYI I will post the whole list of supported digital formats in Colorimetry block in EDID:

"xvYCC601",
"xvYCC709",
"sYCC601",
"opYCC601",
"opRGB",
"BT2020cYCC",
"BT2020YCC",
"BT2020RGB",
"DCI-P3",
"ICtCp". (Yes, ICtCp was defined this year and no, Dolby Vision IPTPQc2 is very different with ITU's ICtCp, cYCC is contant luminance SDR only BT.2020 format.)

Containing "YCC" are only YCC formats. And containing RGB are only RGB. DCI-P3 is also only RGB format, while ICtCp is only YCC format.

"consumer TV does with RGB is to convert it into YCbCr."

Not in PC mode, but okay.

"The BT.601 3x3 matrix would not be appropriate for P3 since it is a larger color gamut"

This is false. Absolutely. For example for 16 bit color you just use higher precision BT.601 matrix as specified in ISO standards. xvYCC601 does use BT.601 matrix (with 4 decimal points though) and is extended gamut. sYCC MUST only use BT.601. sRGB does not use any matrix, it is already RGB. The same about JPEG YCbCr, it uses BT.601 full range matrix. Same about bg-sYCC and "Adobe RGB" (also known as opYCC).

"the 3x3 matrix is usually derived knowing the gamut"

That is XYZ matrix. YCbCr matrix is... Well... it depends on dynamic range, I think. But it may be nice to read how they derived BT.2020 SDR matrix. I dunno. If you have a link...

"such as NTSC and PAL"

That is primaries. Matrix is the same (one for digital, one for analog a.k.a. YUV/YIQ).

"You would also not be sending J2K over the wire."

Sooner or later that (or better lossless HEVC, for example) will be implemented. If you think VESA and HDMI forum are very happy with LOSSY B formats Display Stream compression 1.2, you are mistaken. But to do HEVC compression on the fly will be insanly hard to implement. But everything is possible. YCgCo-R and stuff (as used in DSC 1.2) is also nice, of course, but not nice enough for us, as it is lossy

"J2K works on more than just XYZ."

That is what I said. There on that sample it is RGB tagged (think ICC) Display P3 primaries and PQ transfer. And yeah, there are a lot of color spaces in JPEG2000. From 3 flavours of YCbCr to CIEJab and e-sRGB from PIMA, YCCK and PhotoYCC are also there.

"Most HDR is delivered in YCbCr, not ICtCp"

Most HDR is delivered in IPTPQc2 as in Netflix profile 5. And Netflix wants RGB. So is Google movies.

"Apple TV seems to finally be supporting true 24."

Yeah. We discussed that on ffmpeg mailing list with Apple devs. But most of Cinema is actually 24.000. Just saying. For example Planet Earth 2. It was in 24.000 and was converted into /1.001 and 25.000...

"AVC/HEVC/VP9/AV1 are not appropriate for a mezzanine format."

Then use JPEG XS. It is much better than the very old JPEG 2000. Really, no progress in that sphere is very bad. JPEG-XS i think will make lossless video much more simple than in JPEG 2000.

"I would be surprised if any non-Dolby Vision HDR content is in ICtCp."

Dolby Vision does not use ICTCP. It uses IPTPQc2. For use of ICTCP you can just tag the right matrix on the file (see ITU-T H.273). That is already archived with zimg. See on github, there was a lengthy disscusion and a lot of bugs. ICtCp is really a beauty, just like DeltaIPT.