Spears & Munsil 4K UHD HDR Benchmark (Test Discs) May 15, 2023

[lgans316]

Originally Posted by witheygull

Slipcover. Some folks are obsessive over them and want them on every release, though I believe the above poster was being cheeky.

Yes I was. I have the previous edition and will get this one as well.

I remember reading somewhere that there was some encoding issue in one of the clip and it will be corrected in the new release. Sorry if I am incorrect.

[Stacey Spears]

Originally Posted by witheygull

Slipcover. Some folks are obsessive over them and want them on every release, though I believe the above poster was being cheeky.

Thank you for clarifying. I guess I am a little OCD about them too. I remove them from the disc and throw away along with the magnetic security stickers inside. I had no idea anyone actually kept them.

We have talked about making a cover to sell later on that would hold all four discs. Similar to the one for the Kevin Smith movies. I have one of those, though I don't put discs in it. If we did that, then we would print replacement covers for the first two discs that include the disc number on the side.

[brian9229]

Just a little clarification, can this disc be used to calibrate without color meters and such? I'd like to calibrate my bedroom and den tv's if I can. I have my main ht professionally calibrated, but cant really afford that for all of them.

[Scarriere]

Originally Posted by brian9229

Just a little clarification, can this disc be used to calibrate without color meters and such? I'd like to calibrate my bedroom and den tv's if I can. I have my main ht professionally calibrated, but cant really afford that for all of them.

Calibrate isn't the word that I'd use.
Tweak? Set-up? I'm unsure of the proper term but calibrate isn't it.

[lgans316]

Originally Posted by Scarriere

Calibrate isn't the word that I'd use.
Tweak? Set-up? I'm unsure of the proper term but calibrate isn't it.

I have used it for certain tests although I bet pro calibration involves many other aspects.

The disc is certainly useful to test jaggies and tone mapping at the very least

[Stacey Spears]

Originally Posted by brian9229

Just a little clarification, can this disc be used to calibrate without color meters and such? I'd like to calibrate my bedroom and den tv's if I can. I have my main ht professionally calibrated, but cant really afford that for all of them.

The video setup menu covers what you can do without a colorimeter, which is half a dozen patterns. The analysis (window patterns) section is what calibrators use for calibration, which does require a colorimeter. (grayscale, gamma, gamut, etc...)

We did include a new optical comparator section under video setup. You still need some hardware, but an optical comparator can be a lot cheaper than a colorimeter. We were trying to include one in the box and spent $5,000 building a prototype. It looked promising, but we could not hit the price point we needed to include it. We may revisit in the future as we have some dealers that really want one. If we could offer one for around $25, we would probably bring it to market. For now, I am not involved in that effort. You can try and hobble one together yourself using a gray card and a D65 light source.

The audio section requires an SPL meter. I still have mine from Radioshack. I don't know where one buys an SPL meter today or how well the phone app SPL meters work. We spent some time looking into using a phone for video calibration, but gave up on that too.

The evaluation section is targeted at reviewers, display manufacturers and enthusiasts looking to buy a new display. That section is the heart of the disc and where the name benchmark comes from. When I started reviewing DVD players and displays back in the late 90s, I wish I would have had these patterns. I wish this disc would have been around when UHD BD first launched, then the OPPO would be a much better player.

We call our disc a benchmark disc and not a calibration disc because calibration is just one small section. People get frustrated because the evaluation section points out flaws that they can't calibrate out. :-)

[Stacey Spears]

Originally Posted by lgans316

I have used it for certain tests although I bet pro calibration involves many other aspects.

The disc is certainly useful to test jaggies and tone mapping at the very least

In the 90s, calibration was grayscale using two points and a hidden service menu. Today it has grown to include CMS, 1D/3D LUTs, grayscale, gamut, etc... When you hire a calibrator, you don't really know what you will get. They are not all created equal and not all displays have the same level of adjustment. I think an LG OLED has the best adjustment capability with the 1D and 3D LUTs for SDR, HDR10 and Dolby Vision. I wish Sony, Panasonic and Samsung (And the many others) would offer the same level of adjustments. Just like we have Filmmaker mode, I wish we would get a standard on calibration settings.

I would guess there are people at every display manufacturer that don't like offering calibration controls. They believe that their "look" is why people buy their displays. They even feel insulated that you would want to change their look. I have no problem with them wanting a special look as long as they also allow the option of making it look accurate.

The other topic is high frame rate. They just don't understand why anyone wants to watch 24p content with judder. They love that they can remove the judder and make it look smooth. Don't get me wrong, there are plenty of people that love the high framerate look. I was surprised on another AV forum how many people did not understand why filmmakers were not all shooting high frame rate. I was most surprised because it was an enthusiast forum. There were a lot of people in that discussion that love the soap opera effect.

One of my friends shoots a lot of content that you might see at a trade show where it has the video look. I know his 8K footage has been used at several 8K launch events. He was kind enough to let us use his 8K footage at the 2021 Value Electronics shootout to evaluate the 8K displays. Phil is also responsible for most of the Apple TV screensavers, which look beautiful. The aerial LA skyline footage in our montage came from him and that was an early aerieal test shoot. His 8K aerial footage is probably the best 8K demo material around. We hired him to shoot the skin tone content on the new disc, which was shot three years ago this month. (2/29/20) Had we waited one more week, we would not have any of that content because of the COVID lockdowns.

BTW, disc 3 is being uploaded to Sony today. We have one disc left to upload, which is disc 2. Hopefully that goes up within the next week. It is the least complicated disc of the three. It has the HDR skin tone footage, montages and two patterns that did not fit on disc 1.

[rickardl]

Originally Posted by Stacey Spears

We did include a new optical comparator section under video setup. You still need some hardware, but an optical comparator can be a lot cheaper than a colorimeter. We were trying to include one in the box and spent $5,000 building a prototype. It looked promising, but we could not hit the price point we needed to include it. We may revisit in the future as we have some dealers that really want one. If we could offer one for around $25, we would probably bring it to market. For now, I am not involved in that effort. You can try and hobble one together yourself using a gray card and a D65 light source.

So this is something different than the blue filters used in the old days?

[Stacey Spears]

Originally Posted by rickardl

So this is something different than the blue filters used in the old days?

Yes.

A blue filter is used to ensure that color and tint are set correctly.

An optical comparator is used to adjust grayscale like a colorimeter. The best optical comparator was $1,200 back in the early 2000s. This is when a colorimeter was a lot more money.

It was a gray card with a D65 light source that illuminated the card. It often had a hole in it that you looked through. You compared the gray window pattern on screen to the gray of the optical comparator. You then adjusted the RGB bias / gain controls on your display to try and get the window pattern to match the optical comparator.

Here is a photo of one that the ISF made. I think they charged $300 for it in the late 90s. https://www.portrait.com/resource-ce...or-comparison/

You can't judge the size of the device in the photo. It was larger than a record or laser disc. The one we were working on was the size of a cassette tape.

There were a number of calibrators that could not afford a colorimeter and used an optical comparator and charged you $300 for the calibration. I think the Philips colorimeter, which was the gold standard for grayscale calibration, was in the $3k-$5k range. Long before people were using a Photoresearch, which was in the $15k price range back then. Today you can buy an X-Rite i1 Display for under $300 and use with your favorite calibration software.

[Wendell R. Breland]

Originally Posted by Stacey Spears

I don't know where one buys an SPL meter today or how well the phone app SPL meters work.

There are a number of inexpensive SPL meters available from Amazon and others that would most likely be fine for level matching.

If one wants a step up model then have a look at the Galaxy CM-140 (here). I have the Verified+ version that includes the response data that can be used with TrueRTA (here) or REW (here). Also have the MiniDSP UMIK-1 microphone (here).

[Geoff D]

Originally Posted by Stacey Spears

In the 90s, calibration was grayscale using two points and a hidden service menu. Today it has grown to include CMS, 1D/3D LUTs, grayscale, gamut, etc... When you hire a calibrator, you don't really know what you will get. They are not all created equal and not all displays have the same level of adjustment. I think an LG OLED has the best adjustment capability with the 1D and 3D LUTs for SDR, HDR10 and Dolby Vision. I wish Sony, Panasonic and Samsung (And the many others) would offer the same level of adjustments. Just like we have Filmmaker mode, I wish we would get a standard on calibration settings.

I would guess there are people at every display manufacturer that don't like offering calibration controls. They believe that their "look" is why people buy their displays. They even feel insulated that you would want to change their look. I have no problem with them wanting a special look as long as they also allow the option of making it look accurate.

The other topic is high frame rate. They just don't understand why anyone wants to watch 24p content with judder. They love that they can remove the judder and make it look smooth. Don't get me wrong, there are plenty of people that love the high framerate look. I was surprised on another AV forum how many people did not understand why filmmakers were not all shooting high frame rate. I was most surprised because it was an enthusiast forum. There were a lot of people in that discussion that love the soap opera effect.

One of my friends shoots a lot of content that you might see at a trade show where it has the video look. I know his 8K footage has been used at several 8K launch events. He was kind enough to let us use his 8K footage at the 2021 Value Electronics shootout to evaluate the 8K displays. Phil is also responsible for most of the Apple TV screensavers, which look beautiful. The aerial LA skyline footage in our montage came from him and that was an early aerieal test shoot. His 8K aerial footage is probably the best 8K demo material around. We hired him to shoot the skin tone content on the new disc, which was shot three years ago this month. (2/29/20) Had we waited one more week, we would not have any of that content because of the COVID lockdowns.

BTW, disc 3 is being uploaded to Sony today. We have one disc left to upload, which is disc 2. Hopefully that goes up within the next week. It is the least complicated disc of the three. It has the HDR skin tone footage, montages and two patterns that did not fit on disc 1.

It's amazing how many "enthusiasts" - often with some of the priciest gear going - don't actually give a shit about accuracy. Motion interpolation, sharpening enhancement, colour/contrast boosting, no black bars, they want everything to look how they want it to look, to provide that sweet sweet "pop" (ugh) at all times. We only get one life, sure, and it shouldn't be spent watching something that looks "bad" by our own personal standards, but it makes objective assessment of content rather difficult. Should everyone be an "objective assessor"? Of coursh not, they just want to watch and enjoy. Nothing wrong with that.

But with everyone and his brother now able to disseminate their views across the internets, enabling anyone to become a renowned "reviewer" just because they can chat any old unqualified shit on YouTube, it makes sorting the sense from the noise very difficult especially when combined with the colossal industry clusterf*ck that is HDR tone mapping, and renders the very notion of reviews obsolete. There's never been any accounting for taste, natch, as even if someone has "reference" equipment that's properly calibrated they may still not like what they're seeing, but at least it'd remove the question mark that hangs over 99% of the speculative fiction I've read/watched that masquerades as "reviews".

In some ways it makes all the efforts you've put into the S&M UHD Benchmarks a moo point, as there are so few people out there who want to adhere to any kind of objective standards. But thank you for doing it anyway, I can't wait to finally get a look at the new discs.

[panasonicst60]

@Stacey Spears

For the speaker levels, are the test tones from 500 to 2000hz or full range?

[Stacey Spears]

Originally Posted by panasonicst60

@Stacey Spears

For the speaker levels, are the test tones from 500 to 2000hz or full range?

500 Hz to 2 kHz @ -30 dBFS for the main speakers.
30 Hz to 80 Hz @ -40 dBFS for the LFE.

[Archivy]

Originally Posted by Geoff D

It's amazing how many "enthusiasts" - often with some of the priciest gear going - don't actually give a shit about accuracy. Motion interpolation, sharpening enhancement, colour/contrast boosting, no black bars, they want everything to look how they want it to look, to provide that sweet sweet "pop" (ugh) at all times. We only get one life, sure, and it shouldn't be spent watching something that looks "bad" by our own personal standards, but it makes objective assessment of content rather difficult. Should everyone be an "objective assessor"? Of coursh not, they just want to watch and enjoy. Nothing wrong with that.

But with everyone and his brother now able to disseminate their views across the internets, enabling anyone to become a renowned "reviewer" just because they can chat any old unqualified shit on YouTube, it makes sorting the sense from the noise very difficult especially when combined with the colossal industry clusterf*ck that is HDR tone mapping, and renders the very notion of reviews obsolete. There's never been any accounting for taste, natch, as even if someone has "reference" equipment that's properly calibrated they may still not like what they're seeing, but at least it'd remove the question mark that hangs over 99% of the speculative fiction I've read/watched that masquerades as "reviews".

In some ways it makes all the efforts you've put into the S&M UHD Benchmarks a moo point, as there are so few people out there who want to adhere to any kind of objective standards. But thank you for doing it anyway, I can't wait to finally get a look at the new discs.

Eh, I know what you mean, and consumers should want to uphold their end of the bargain by ensuring their playback device(s) are set to adhere to creative intent. But your pearl-clutching seems a bit much when so much contemporary content itself isn't accurate to the original source, what with endless tomfoolery regarding Atmos remixes, shifted color grades, DNR applications, compressed streams and botched encodes, and the like.

It can feel like a lost cause to spend so much effort to fine-tune our home gear only to be faced with a consumer product that itself isn't accurate.

To be clear: I'm not championing the enthusiast behavior you're decrying; rather, I'm saying I'm unsure you realize that you're fighting a battle that the content creators will not let you win.

[Geoff D]

Originally Posted by Archivy

Eh, I know what you mean, and consumers should want to uphold their end of the bargain by ensuring their playback device(s) are set to adhere to creative intent. But your pearl-clutching seems a bit much when so much contemporary content itself isn't accurate to the original source, what with endless tomfoolery regarding Atmos remixes, shifted color grades, DNR applications, compressed streams and botched encodes, and the like.

It can feel like a lost cause to spend so much effort to fine-tune our home gear only to be faced with a consumer product that itself isn't accurate.

To be clear: I'm not championing the enthusiast behavior you're decrying; rather, I'm saying I'm unsure you realize that you're fighting a battle that the content creators will not let you win.

The look of a movie is a moving target. Always has been. And sound has been going the same way in the modern multichannel era. BUT how are we to know what’s been done if everyone just sets it up how they want? In that regard objective analysis is and always shall be in play if someone wants to dedicate themselves to that pursuit, which is where me and my pearls are coming from. It’s not about “winning”, it’s about finding out how the contest was rigged in the first place.

[Stacey Spears]

All discs have now been sent to Sony. We should find out early next week on how long it will take to receive the check discs.

[bhampton]

Originally Posted by Stacey Spears

In the 90s, calibration was grayscale using two points and a hidden service menu.

In the 90's I owned a Sony VPH-1271.

Calibration involved

Physical placement
Electric/Magnetic and mechanical focus
Geometry and Convergence
Greyscale/Gamma
Primary and Secondary Colors
Sharpness, Hue, and Saturation
User Settings

All for Standard Definition with Standard Dynamic Range.
However, most refresh rate or resolution changes from a source would often result in a different memory block that had to also be calibrated for use.

It's both more and less involved now.

These days with the JVC RS3100 I use and trust the AutoCal program JVC provides.

A new calibration disc would let me test JVCs AutoCal program and it's Frame by Frame Tone Mapping.

[Sledgehamma]

Originally Posted by Stacey Spears

All discs have now been sent to Sony. We should find out early next week on how long it will take to receive the check discs.

Congrats on all involved!!!

[Stacey Spears]

I have been writing this post over several hours while working on other things, so I am not sure if it even makes sense it has become so long and I lost my train of thought multitasking with work and other projects today.

Looks like the Jason will receive the check discs around 3/17, if the date is accurate. I should have my set the following week once Jason ships them to me to start testing.



Fun Facts:

Every test pattern was encoded 96 times with different encoder settings to find the optimal settings for each pattern. This means each pattern is using one of the 96 combinations of encoder settings. Previous disc(s) used the same settings for every pattern.

It took six months of machine time to encode all of the content.

The first edition used a single frame encode in VC-1. The 2nd edition used a single frame encode in AVC. The UHD HDR edition used a 2-frame encode in HEVC, due to an issue where Samsung players could not play back single frame encoded content on BD. The new disc uses 9-frame encode in HEVC. What we saw was that different patterns take a different number of frames to reach their highest quality. Then we pause on the last frame. By using 9-frames, we would give every static pattern enough time to reach its maximum quality. If you encode as running video, the quality fluctuates and re-starts at each I-frame / GOP.

We believe the work we have done with specific GOP length and per pattern custom encoder settings make these the highest quality test patterns ever encoded. Others could certainly replicate, they just want, and need, to do the work. Will anyone notice? Probably not. Several of our tricks to get around lossy encoding are used on the Pixel Aging pattern. There is another disc that has a random noise pattern, that behaves similar to our Pixel Ageing, but it looks like it was done as a BD-J app and not as encoded content.

The PCA pattern set (to measure backlight resolution) is also something we can't wait to see being used and the numbers being reported. You measure 10 patterns and plug the levels into a spreadsheet which then outputs a single number. It will be interesting to see this used on the various FALD displays and the newer OLEDs both MLA and QD.

It took somewhere around 24 months to author the three discs. David started over at least three times to try and work around authoring software issues. The project was so large that it would stop saving and muxing after a certain point. The format supports up to 9999 files and we were only 50% of the way there.

Disc one has been authored at seven plus projects. It is usually one project per disc. The seven plus projects were then merged together into a single disc.

Disc three was authored as two projects. Disc two was authored as one project.

The HDR10+ team, from Samsung, provided the HDR10+ tools to generate and mux the metadata. Philips and Technicolor created the SL-HDR2 metadata. (HDR by Technicolor)

Dolby re-graded the new edit of the montage. Several gamut / highlight clipped shots have been fixed, such as the top of a building on the LA aerial shot. When you compare the two shots, it is obvious. The montage on the current disc was clipped at the 2020 boundary, but we used Colorfront Transkoder to re-create the clipped areas, so it looks decent. Some shots actually turned purple due to this clipping. You can see the purple on the peacock feather, QD viles, LA Aerial and Space Needle shot to name a few.

Colorfront (Bill Feightner himself) created a special pipeline / workflow for me to fix the gamut clipping from Resolve. Resolve clipped the gamut at the 2020 boundary instead of mapping colors outside of 2020 back into 2020 when exporting a ST2084 master. In order to avoid the clipping, Dolby exported a linear light OpenEXR sequence from Resolve, which contained the negative pixel values. Negative values meant it goes beyond 2020. The special Transkoder pipeline mapped the out of gamut colors back into 2020.

Colorfront is probably the leader in color processing and HDR. Their technology is what is used in the AJA FS-HDR and HDR Analyzer.

The HDR Analyzer version of the montage was created in Transkoder. We setup the analyzer to show colors that went beyond P3 as red so that you can visually see which pixels are >P3. We did the same for levels >4000 nits. It was not what I expected and turned out to be educational for me. We hope others like the HDR Analyzer montage.

For the SDR vs. HDR and the Graded Vs. Ungraded, the HDR side is based on the 1000 nit trim pass and not the 10,000. The last time around, the SDR vs. HDR used the 10,000 with SDR at 100. This time SDR is also at 203. Or rather peak white (code value 940) was mapped to 203 nits.

The ungraded side of the Graded Vs. Ungraded is what Shane saw when he first loaded the footage into Resolve. The biggest things you will see done is white balancing, raising exposure by 1-2 stops and crushing the black backgrounds on the shots with black backgrounds. Those were all shot against duvateen. Two shots had the sun replaced with a VFX sun and another shot had an entire sun added. Some tone mapping algorithms really have a hard time with that fake sun. (LG)

Four shots were replaced and some others were cut all together so that other shots could be lengthened. New audio was composed and mixed in Atmos. I still have not heard the Atmos mix myself. I will be curious to hear everyone's feedback on the montage audio in Atmos.

The Atmos tones exist in 5.1, 7.1 and 9.1 base and 2, 4 and 6 top layers. DTS:X is 5.1, 7.1 and 9.1 base with 2 and 4 top layers. There is a 6 top layer, but it is disabled by default. The reason is that in all systems we tested, the 6 channel was phantom created by the front and rear top speakers. DTS created them, so it may just be a limitation of the format. DTS took the Atmos masters and created the DTS:X from them.

The panning audio are the most interesting.

A lot of the patterns from the previous HDR disc have been tweaked based on feedback. For example, the ADL patterns were modified to not have boxes directly to the sides or above/below the center box.

The Pixel Aging is probably one of my favorite just because of all the small details that went into making it.

The Peak Luminance is probably the coolest from a technology standpoint. It is based on a paper using Spatio-Temporal noise to simulate real-world picture levels on a synthetic pattern. When a company claims they can do 1000 nits, this usually means in a 10% area window test pattern. If you were to find real content and measured it, that same display would not go over 400-600 nits. This test pattern will behave similarly. I suspect display manufactures won't be fans of it. The pattern starts and ends with a black circle and timecode counting down to 0. Once it reaches 0, then it turns white. This allows you to measure the rise-time if you have a colorimeter and software than can handle that. e.g. the Klein K10 with their software can measure rise-time. A CR with their software can do the same thing. The white lasts for 60-seconds. The idea was the pattern would run below the time the auto diming kicks in on an OLED. In Calman you use the mode that plots a graph over time. Then you can see the peak luminance go up and down during that minute. The target is just big enough to cover a Klein K10 on an 48" display. We wanted it as small as possible since it is at code value 940. (10,000 nits) There are 350, 600, 1000, 2000, 4000 and 10,000 nit versions. The nit level applies to the background level that fluctuates like real-world content. I hope reviewers will start to use this pattern to report real-world nit levels. Here is a YouTube video of it being used by someone already. https://youtube.com/shorts/oO_w5i_rXzo?feature=share

The Hue Shift pattern began life as something else but turned out to show hue shift issues when tone mapping. The Near-black Overdrive was meant to show the flicker that occurs on OLEDs coming out of black. Did not work at all for that. However, Vincent found a use for it when comparing OLED vs. QD-OLED. It exposed some issues with QD-OLED, so we left it on the disc. This is one of two patterns that are on the 2nd HDR disc, which contains the montages. It is on here because we ran out of space on disc 1.

The Sarah on a Hammock is back. The 24p version is really the 60p version. Then for the 60p version, we repeated every frame twice, to make it 120p and then played back as 60p. This allowed both versions to produce the desired behavior. The default 24p at 24p pans way to fast as does 60p at 60p. We grossly violated the ASC panning rules. The ropes moving over the rock texture pretty much breaks every soap opera mode algorithm today with some nasty artifacts.

Pricing:
MSRP - $59.94
Introductory - $47.95
Upgrade - $29.97 (Limited time offer. Proof of purchase required.)

Some cost information:
Bill of Materials - $8.05 per unit.
AACS - $3,000 a year. ($1,000 per disc per year)
BDA - $1000 a year. ($500 for 3D BD logo and $500 for UHD BD logo)
Skin Tone Content (8 models) - $15,000 for one day shoot. I own rights the rights.
Montage Music (Atmos) - $5,000 to compose new music. Rights are limited for this disc only. Future rights, such as streaming, YouTube, etc... would be a new license we would have to pay for.
Authoring - $$,$$$.
10 Check Discs - $1,000.

It took a little over year to recoup the costs of the 2nd edition disc. Given what we have spent on this disc, which is a lot more than all previous discs combined, we expect two years to break-even.

The disc will not initially be available on Amazon. It will come after the upgrade program has ended. We have had a lot of issues with Amazon. The primary being people using it as a disc rental service. You would be surprised at what people return instead of the actual disc and Amazon does not care. Given the cut they take, you would expect more from Amazon.

[panasonicst60]

@Stacey Spears. So I'm assuming this will first be available from Jason at BiasLighting.com? When is the estimated date that Jason will be selling your disc?