How are reviewers assigned to rate BDs for the site? Reviews reviews reviews!

[shinseiRomeo]

I'm curious how the reviewers work here. Do studios send advanced copies, does someone specialize in a genre and review those that fall in it, or does every reviewer sit in a room and whoever picks the short straw has to watch that movie?

What made me curious was that I noticed certain genres didn't have any professional reviews, such as the Best of Travel/Europe series.

[Darkthrone]

Are the reviewers over at high def digest extremely picky about their scores or what's going on? If you look at the movie Frozen, blu-ray.com gave it 4.5 for video quality and high def digest gave it 2.5 (both scores out of 5).
https://www.blu-ray.com/movies/Frozen-Blu-ray/11377/
http://bluray.highdefdigest.com/3412/frozen.html
That's a pretty big difference in the scores.

Overall a lot of movies gets lower scores from high def digest compared to blu-ray.com. I was just curious why lol

[DetroitSportsFan]

We don't discuss other forums here, that's why.

Seriously though, when it comes to reviews, everyone has an opinion. Opinions are often inconsistent.

[4K2K]

Originally Posted by Darkthrone

Overall a lot of movies gets lower scores from high def digest compared to blu-ray.com. I was just curious why lol

I think this site (blu-ray.com) shouldn't give a movie ("28 Days Later") which was 99% made on standard definition DV camcorders a 4.5 out of 5 for picture quality. I think picture quality ratings (or at least a picture quality rating - even if it's a separate one) should take into account the spatial (and temporal) resolution actually in a title. A title with "Beyond High Definition" on the front but which was actually 99% shot with standard def DV camcorders shouldn't get anywhere near a 4.5 out of 5, and shouldn't be allowed to call itself HD, never mind "Beyond High Definition".

[pro-bassoonist]

Originally Posted by 4K2K

I think this site (blu-ray.com) shouldn't give a movie ("28 Days Later") which was 99% made on standard definition DV camcorders a 4.5 out of 5 for picture quality. I think picture quality ratings (or at least a picture quality rating - even if it's a separate one) should take into account the spatial (and temporal) resolution actually in a title. A title with "Beyond High Definition" on the front but which was actually 99% shot with standard def DV camcorders shouldn't get anywhere near a 4.5 out of 5, and shouldn't be allowed to call itself HD, never mind "Beyond High Definition".

I cannot speak on behalf of my colleagues, but these are the criteria I have adopted:

1. Picture quality ratings should reflect whether the transfer accurately replicates the original content. Naturally, a good review should address whether the transfer used for a particular Blu-ray release accurately replicates the native characteristics of the original source as well as the filmmaker's intent, not a fictional standard.

2. If replicating a native 720p or 1080i content, and done right, there is absolutely no reason why a native 720p or 1080i transfer should not be eligible to receive a perfect score.

3. There is nothing problematic about softness, excessive grain, fluctuating contrast levels and color-schemes (crushed blacks, etc), if they are part of the original content. Most serious film enthusiasts cherish Blu-ray because the format allows for impressive replication of the original content, not impressive corrections to meet some fictional standard(s).

4. Films have been and are handled by people - thus negatives have been damaged, lost, etc. Therefore, when one reviews, one should consider the variety of different limitations that affect how certain films can/can't look on Blu-ray, not a fictional standard that apparently mandates how a film should look on Blu-ray.

Lastly, there is absolutely no need to discuss what other sites do/don't. If you are concerned with a particular review/score on Blu-ray.com, you are always welcome to contact the reviewer responsible for the review, or even start a thread, to get a clarification.



Pro-B

[LordoftheRings]

But do all reviewers use the same standards?

[4K2K]

So why can't we have an additional PQ (or whatever name is decided) score/scores which takes into account resolution (spatial and temporal) and ignoring directors intent etc. so that, if say a "film" was 99% shot on standard def DV camcorders, one of the PQ scores would let you know (eg. it would say something like: Average Approx Resolution: 576i or whatever the spatial resolution was measured to).

2. If replicating a native 720p or 1080i content, and done right, there is absolutely no reason why a native 720p or 1080i transfer should not be eligible to receive a perfect score.

I'm not saying there's anything wrong with either of those two, if those were the original source format. Though if the original format was shot & mastered at 1080p23.976 I don't think they should get a high score. Though the fact that both of those can allow for a higher temporal resolution on Blu-ray than 1080p23.976 or can be the correct source format for almost all European TV titles also means that they should get a high normal PQ score (higher than, say, a European TV programme released at 1080p23.976 - since that is unlikely to be the original format). Though I do think that a resolution score should tell approx how much effective spatial resolution is in a particular title, as well as temporal. So if a title is filtered (for whatever reason, including directors intent or because of interlacing) we can see which titles have the highest true resolutions.

I'm not asking for the current PQ scores to be changed, if that is what the site wants, but for additional PQ or "resolution" scores for an objective, standardised method of letting people know what the picture quality is like (a high score because the PQ is bad because the director intended it to be bad, doesn't let the buyer know).

[pro-bassoonist]

Originally Posted by 4K2K

So why can't we have an additional PQ (or whatever name is decided) score/scores which takes into account resolution (spatial and temporal) and ignoring directors intent etc. so that, if say a "film" was 99% shot on standard def DV camcorders, one of the PQ scores would let you know (eg. it would say something like: Average Approx Resolution: 576i or whatever the spatial resolution was measured to).

Because there is no need for anything else. All you need to do is read the technical analysis. And as far as the actual system and layout we have in place are concerned, our developers have made sure that we have the most advanced and sophisticated set-up of all major review sites.


Pro-B

[LordoftheRings]

Fair question.

[Troy73]

I think they play "duck, duck, GOOSE".

[Dreemworx]

It's not a bad question, Burt.

[pro-bassoonist]

Originally Posted by shinseiromeo

I'm curious how the reviewers work here.

Each staff member has assigned a group of studios.

However, we are a team, and we all work together to make sure that we have timely reviews for our community.

Pro-B

[4K2K]

Originally Posted by pro-bassoonist

Each staff member has assigned a group of studios.

However, we are a team, and we all work together to make sure that we have timely reviews for our community.

Pro-B

If the studios give the site discs to review, what's to stop the site giving favourable reviews (when the BD isn't that good) or higher than average scores for titles, so that those studios keep sending discs for review?

Shouldn't the "average" score for titles be 2.5 out of 5? Yet aren't most titles above that?

[shinseiRomeo]

Originally Posted by Troy73

I think they play "duck, duck, GOOSE".

Never thought of that one. Good thinking!

Originally Posted by 4K2K

If the studios give the site discs to review, what's to stop the site giving favourable reviews (when the BD isn't that good) or higher than average scores for titles, so that those studios keep sending discs for review?

Shouldn't the "average" score for titles be 2.5 out of 5? Yet aren't most titles above that?

I do trust the site. They have a fantastic staff and wonderful community. Many reviews receive a negative score as well as many positive. Just look at the new release Grown Ups. It doesn't come out for two weeks but Martin Liebman has reviewed it last night and gave it a fairly poor score.

[Maximus]

Originally Posted by 4K2K

If the studios give the site discs to review, what's to stop the site giving favourable reviews (when the BD isn't that good) or higher than average scores for titles, so that those studios keep sending discs for review?

Shouldn't the "average" score for titles be 2.5 out of 5? Yet aren't most titles above that?

Integrity.

If your site is big enough (like this one) the studios can't really afford to stop sending you screeners. The only way to really fall out with a studio is by giving unjustified scores, and even then they might ask you to reconsider, but you are under no obligation to do so.

As long as your score for a BD is justifiable then it's nothing to worry about. You can give 0/5 for some BDs and the studio won't bat an eyelid because they know it is bad, but if you gave Transformers across the board 0/5 because the reviewer doesn't like Michael Bay they will probably get upset as the score isn't justified.

[Ken Brown]

Originally Posted by 4K2K

If the studios give the site discs to review, what's to stop the site giving favourable reviews (when the BD isn't that good) or higher than average scores for titles, so that those studios keep sending discs for review?

I was going to say "integrity," but Maximus beat me to it

The studios send screeners regardless of our scores. Any publicity is good publicity, as the adage goes. Plus, I've been pretty rough on many a Warner, Disney and Universal release and I've never heard a word about it from the PR or studio reps. We don't receive any word or indication as to whether they like our reviews or not. We also purchase a lot of releases we don't get screeners for -- far more than any other site and far more than many of our readers probably realize. The scores these releases earn are no harsher or more forgiving than the scores we give when reviewing a studio-supplied screener.

There's an inherent integrity surrounding the relationship between reviewers and studios as well. Even if a studio did want to influence scores, the last thing they would want is their threats, promises, bribes and the like to go public. That would be far worse than the worst scores we could dole out. Even if we were shady and the studios were manipulative, to indulge in our darker desires would be mutually assured self-destruction. The entire review industry would crumble. Journalists are black-listed for just such behavior and reviewers are no different. We'd be ousted in a second if we engaged in any studio-snuggling. Personally, I would be enraged if a PR or studio rep even suggested I compromise my opinion to receive more discs, and I know the reviewers on this site and the vast majority of writers on other sites would be too.

Plus, don't think for a second PR firms and studios aren't aware when one of their titles is subpar. Reviews rarely surprise them

[Ken Brown]

Originally Posted by 4K2K

Shouldn't the "average" score for titles be 2.5 out of 5? Yet aren't most titles above that?

Imagine if the same idea were applied to children in school. If I had 15 students in a kindergarten class, 5 would fail, 5 would be average and 5 would excel. No matter how high all 15 students scored, no matter how impressive their individual results were, as a teacher I would be left with a ludicrous responsibility: 5 must fail, 5 must be dubbed average and 5 must be declared excellent. Now, if the studios were striving for average, and somehow achieved perfect mediocrity, the average score would indeed be a 2.5. However, because the studios are striving for perfection and because, for the most part, they come close on more occasions than they fall short, the overall score of titles is higher than 2.5.

The resulting average -- be it a 2, 3, 4 or 4.5 -- tells you how close the studios are coming to that ideal as a whole. It should not be 2.5 just because of some implied law of averages.

As a scoring scale for such wildly different products -- catalog releases from ten decades of film, new theatrical beasts, direct-to-video product, television releases, live concert performances -- it works because one release is not entirely dependent on another. Likewise, it doesn't assume that all scores must work out to a 2.5. As filmfans, we want perfect releases. If the studios were able to deliver perfection, but we weren't able to declare their releases perfect because of a preconceived law of averages, what use would we be when measuring the actual quality of their releases?

Anyway, hope that made sense. My advice? Don't put so much weight in a random number. We write thousands of words per review for a reason. The scores are just icing on the cake used to break our impressions down into an easily digestible nugget. To simply judge our impressions based on raw numerical data will leave you with all sorts of headaches and heartaches.

Thanks as always for posting!

[LordoftheRings]

Originally Posted by Ken Brown

Imagine if the same idea were applied to children in school. If I had 15 students in a kindergarten class, 5 would fail, 5 would be average and 5 would excel. No matter how high all 15 students scored, no matter how impressive their individual results were, as a teacher I would be left with a ludicrous responsibility: 5 must fail, 5 must be dubbed average and 5 must be declared excellent. Now, if the studios were striving for average, and somehow achieved perfect mediocrity, the average score would indeed be a 2.5. However, because the studios are striving for perfection and because, for the most part, they come close on more occasions than they fall short, the overall score of titles is higher than 2.5.

The resulting average -- be it a 2, 3, 4 or 4.5 -- tells you how close the studios are coming to that ideal as a whole. It should not be 2.5 just because of some implied law of averages.

As a scoring scale for such wildly different products -- catalog releases from ten decades of film, new theatrical beasts, direct-to-video product, television releases, live concert performances -- it works because one release is not entirely dependent on another. Likewise, it doesn't assume that all scores must work out to a 2.5. As filmfans, we want perfect releases. If the studios were able to deliver perfection, but we weren't able to declare their releases perfect because of a preconceived law of averages, what use would we be when measuring the actual quality of their releases?

Anyway, hope that made sense. My advice? Don't put so much weight in a random number. We write thousands of words per review for a reason. The scores are just icing on the cake used to break our impressions down into an easily digestible nugget. To simply judge our impressions based on raw numerical data will leave you with all sorts of headaches and heartaches.

Thanks as always for posting!

Well said Ken! ...Interesting view on those 15 children from kindergarten school! ...I guess each school have their own set of standards, and teachers are more or less in agreement with them, according to their own personal set of values in their own life's experience.
After all, we live in a free world, or country (1st amendment of the United States Constitution).

[shinseiRomeo]

What made me think of this is when I wrote a review for Best of Travel: Beautiful Japan (https://www.blu-ray.com/movies/Best-...Blu-ray/13967/) I noticed this site was quoted on the back cover. I assumed there was a reviewer for the series, but there wasn't. I then checked my copy of BoT: China and it had the same quote as well. Since we didn't have a primary reviewer here, I decided to write a thorough review, longer than I usually do, to try to cover some slack. Of course the fun part was that it took me three hours to write the review for a 50 minute documentary.


The quote on the back cover of the Best of Travel series is: "A beautiful visual experience - Blu-ray.com", but not attributed to anyone specifically, so I wonder if Questar just pulled a random comment from the forums.