Quote:
Originally Posted by bailey1987
|
from the press release...'“Motion Picture Experts Group (MPEG).Testing is already underway”...
Not talking about the Technicolor solution per se, but with regards to MPEG testing in general, inquiring scientific minds wonder
what exactly is the difference between subjective consumer testing/shootouts for evaluating something visual, as compared to that at professional labs, who work on a more scientific level.
Well….glad you asked
. For an example, in order to explore whether the coding efficiency and/or the functionality of current HEVC standard could be significantly improved for HDR and WCG content by “putting on a shootout*over the past few months in determination of whether different proponent solutions could beat (achieve statistically significant superior visual quality) the HEVC Main 10 Profile based Anchor….this (below) is how it’s/was done by one such professional scientific lab with regards to the dataset (content), test environment, methodology, test planning, and just a little statistical analysis so as not to bore people to death with the statistics of it all -
Dataset
The dataset used for the subjective evaluation tests consists of five HD resolution HDR video sequences, namely, Market3, AutoWelding, ShowGirl2, WarmNight, and BalloonFestival. Each video sequence was cropped to 950 x 1080 pixels, so that the video sequences were presented side-by-side with a 20-pixels separating black border. Each video sequence was displayed at 24 fps, which is the native frame rate of the display used in the experiments (see Section 2.2), and cut to 240 frames, which corresponds to 10 seconds. Note that the Market3 sequence was played at a slower frame rate than the original content (50 fps). This solution was evaluated as visually more pleasant than playing every other frame, which created temporal distortion. The coordinates of the cropping window and selected frames are given in Table 1.
The data was stored in uncompressed 16 bit TIFF files, in 12 bit non-linearly quantized (using Dolby PQ EOTF) RGB signal representation, using the SDI data range (code values from 16 up to 4076) and BT.2020 RGB colour space.
The side-by-side video sequences were generated using the HDRMontage tool from the HDRTools package.
Test environment
The experiments were conducted at the Multimedia Signal Processing Group (MMSPG) test laboratory, which fulfills the recommendations for subjective evaluation of visual data issued by ITU-R [4]. The test room is equipped with a controlled lighting system of a 6500 K color temperature. The color of all background walls and curtains in the room is mid grey. The laboratory setup is intended to ensure the reproducibility of the subjective tests results by avoiding unintended influence of external factors. In the experiments, the luminance of the background behind the monitor was about 20 cd/m2. The ambient illumination did not directly reflect off of the display.
To display the test stimuli, a full HD (1920 × 1080 pixels) 42'' Dolby Research HDR RGB backlight dual modulation display (aka Pulsar) was used. The monitor has the following specifications: full DCI P3 color gamut, 4000 cd/m2 peak luminance, low black level (0.005 cd/m2), 12 bits/color input with accurate and reliable reproduction of color and luminance.
In every session, three subjects assessed the displayed test video content simultaneously. They were seated in one row perpendicular to the center of the monitor, at a distance of about 3.2 times the picture height, as suggested in recommendation ITU-R BT.2022 [5].
Methodology
Two video sequences were presented simultaneously in side-by-side fashion. Since only one full HD 1920 × 1080 HDR monitor was available, each video was cropped to 950 × 1080 pixels with 20 pixels of black border separating the two sequences. One of the two video sequences was always the Anchor, with a randomized position on the screen (either on the left or on the right). The other video sequence was the Proponent to be evaluated, at the same (targeted) bit rate as the Anchor.
Subjects were asked to judge which video sequence in a pair (‘left’ or ‘right’) has the best overall quality, considering fidelity of details in textured areas and color rendition. The option ‘same’ was also included to avoid random preference selections.
Test planning
Before the experiments, a consent form was handed to subjects for signature and oral instructions were provided to explain the evaluation task. A training session was organized to allow subjects to familiarize with the assessment procedure. The same contents were used in the training session as in the test session to highlight the areas where distortions can be visible. Eleven training samples were manually selected by expert viewers. First, two samples, one of high quality and one of low quality, without any difference between left and right, were selected from the AutoWelding sequence. The purpose of these two examples was that subjects could get familiar with HDR content, as this content has both dark and bright luminance levels and fast luminance temporal changes, and see the extreme levels of quality observed in the test material. Then, one sample from AutoWelding with large visible difference was presented to illustrate the main differences that can be observed between the left and right video sequences, i.e., loss of texture/details and color artifacts. Finally, for each of the remaining contents, two samples were presented (one example with large difference and one example with small differences) in the following order: Market3, BalloonFestival, ShowGirl2, and WarmNight. The training materials were presented to subjects exactly as for the test materials, thus in side-by-side fashion.
The overall experiment was split into 6 test sessions. Each test session was composed of 30-31 basic test cells (BTC), corresponding to approximately 14 minutes each. To reduce contextual effects, the stimuli orders of display were randomized, whereas the same content was never shown consecutively. The test material was randomly distributed over the six test sessions.
Each subject took part to exactly three sessions. Three dummy pairs, whose scores were not included in the results, were included at the beginning of the first session to stabilize the subjects' ratings. Between the sessions, the subjects took a 14-minute break.
A total of 48 naďve subjects (16 females and 32 males) took part in the experiments, leading to a total of 24 ratings per test sample. Subjects were between 18 and 49 years old with an average and median of 25.3 and 24 years of age, respectively. All subjects were screened for correct visual acuity and color vision using Snellen and Ishihara charts, respectively.
Statistical analysis
No outlier detection was performed on the raw scores, since there is no international recommendation or a commonly used outlier detection technique for paired comparison results.
For each test condition, i.e., combination of content, algorithm, and bit rate, the winning frequency of the Anchor, w_Ai, winning frequency of the Proponent, w_Pi, and tie frequency, t_i, are computed from the obtained subjective ratings. Note that w_Ai+w_Pi+t_i=N, where N is the number of subjects. To compute the preference probability of selecting the proponent version over the Anchor, p_P, ties are considered as being half way between the two preference options: p_P= w_Pi/N+t_i/2N.
And so on…..and so on.
All countries
Argentina
Austria
Belgium
Brazil
Bulgaria
Chile
China
Colombia
Cyprus
Czech Republic
Estonia
Denmark
Finland
Greece
Hong Kong
Hungary
Iceland
Indonesia
Ireland
Israel
India
Latvia
Lithuania
Malaysia
New Zealand
Norway
Philippines
Poland
Portugal
Romania
Russia
Singapore
South Africa
South Korea
Sweden
Switzerland
Taiwan
Thailand
Turkey
Ukraine
United Arab Emirates
, so this will take some experimentation and experience in order not to produce unrealistic, distracting or less than optimal imagery. HDR live is new territory, but nevertheless exciting. 
I received a PM regarding the pic ^ of the soccer stadium and field I posted above....
Linear Mode
