Pages: [1]   Go Down

Author Topic: DXOMark: RX1R scores (v. slightly) lower than RX1.  (Read 10261 times)

KirbyKrieger

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 422
    • Kirby Krieger Pictures
DXOMark: RX1R scores (v. slightly) lower than RX1.
« on: August 23, 2013, 05:00:19 pm »

DXOMark has released their test results for the Sony RX1R.  It scores overall at 91 (RX1=93).  The difference is in the Dynamic Range, where the RX1R scores significantly (that is, scientifically significant) lower than the RX1.  Comparison results are here.  DXO says in practice the difference is negligible.  They also note that they do not test resolution.

1. Are the results valid and trustworthy?
2. Why would the removal of the low-pass filter result in a lower score?

Thanks.

BernardLanguillier

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 13983
    • http://www.flickr.com/photos/bernardlanguillier/sets/
Re: DXOMark: RX1R scores (v. slightly) lower than RX1.
« Reply #1 on: August 23, 2013, 05:46:01 pm »

DXOMark has released their test results for the Sony RX1R.  It scores overall at 91 (RX1=93).  The difference is in the Dynamic Range, where the RX1R scores significantly (that is, scientifically significant) lower than the RX1.  Comparison results are here.  DXO says in practice the difference is negligible.  They also note that they do not test resolution.

1. Are the results valid and trustworthy?
2. Why would the removal of the low-pass filter result in a lower score?

This is probably just the result of sample + measure variation, nothing to worry about.

Cheers,
Bernard

eronald

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 6642
    • My gallery on Instagram
Re: DXOMark: RX1R scores (v. slightly) lower than RX1.
« Reply #2 on: August 23, 2013, 08:30:00 pm »

This is probably just the result of sample + measure variation, nothing to worry about.

Cheers,
Bernard


Bernard,

 If you said this "might be" sample variation yadda yadda, I would agree. When you say "probably" I have issues.
 It might equally well mean a different chip plant ... in which case the two cameras would represent different realizations of the same design. Or a different filter glass on the package which might mean a difference in sensitivity of a couple of stops at certain wavelengths ...

Edmund
« Last Edit: August 23, 2013, 08:46:05 pm by eronald »
Logged
If you appreciate my blog posts help me by following on https://instagram.com/edmundronald

BernardLanguillier

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 13983
    • http://www.flickr.com/photos/bernardlanguillier/sets/
Re: DXOMark: RX1R scores (v. slightly) lower than RX1.
« Reply #3 on: August 24, 2013, 01:20:42 am »

Bernard,

 If you said this "might be" sample variation yadda yadda, I would agree. When you say "probably" I have issues.
 It might equally well mean a different chip plant ... in which case the two cameras would represent different realizations of the same design. Or a different filter glass on the package which might mean a difference in sensitivity of a couple of stops at certain wavelengths ...

Deepest apologies Edmund, I had not expected the word "probably" to generate such a level of stress. :)

I personally think there are better ways to spend that amount of cash anyway.

Cheers,
Bernard

eronald

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 6642
    • My gallery on Instagram
Re: DXOMark: RX1R scores (v. slightly) lower than RX1.
« Reply #4 on: August 24, 2013, 07:14:12 am »


I personally think there are better ways to spend that amount of cash anyway.

Cheers,
Bernard


Now I have to I agree. :)
Maybe someone should tell Sony.

Edmund
Logged
If you appreciate my blog posts help me by following on https://instagram.com/edmundronald

Bart_van_der_Wolf

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 8913
Re: DXOMark: RX1R scores (v. slightly) lower than RX1.
« Reply #5 on: August 24, 2013, 07:43:16 am »

1. Are the results valid and trustworthy?

In general, they are quite accurate. Of course it represents only a small sample of production models.

Quote
2. Why would the removal of the low-pass filter result in a lower score?

That is not the cause for the lower score only at the lowest ISO (!). It's most likely caused by the camera electronics (ADC circuits).

Cheers,
Bart
« Last Edit: August 24, 2013, 07:54:46 am by BartvanderWolf »
Logged
== If you do what you did, you'll get what you got. ==

KirbyKrieger

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 422
    • Kirby Krieger Pictures
Re: DXOMark: RX1R scores (v. slightly) lower than RX1.
« Reply #6 on: August 24, 2013, 09:25:56 am »

In general, they are quite accurate. Of course it represents only a small sample of production models.

That is not the cause for the lower score only at the lowest ISO (!). It's most likely caused by the camera electronics (ADC circuits).

Cheers,
Bart

Thanks Bart.  Appreciate the specificity   :) .

Thanks to the other responders as well.  All helpful.

Rob C

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 24074
Re: DXOMark: RX1R scores (v. slightly) lower than RX1.
« Reply #7 on: August 24, 2013, 10:17:17 am »

So, in essence, with so any minute variables it's best forgetting such 'scores' for the esoteric nonsense that they may actually be?

Not saying, just wondering.

Rob C

JayWPage

  • Full Member
  • ***
  • Offline Offline
  • Posts: 216
    • Jay W Page Photography
Re: DXOMark: RX1R scores (v. slightly) lower than RX1.
« Reply #8 on: August 24, 2013, 11:27:25 am »

A sample size of one is simply a test and cannot be used to determine some metric about the overall population.

It would be interesting to see how much variability exists for individual cameras for a whole variety of measurements, such as DR, resolution, % dead pixels, power consumption, shutter speed accuracy, etc. etc.

One supposes that the camera manufacturers quote a mean value when stating the statistics and not a tail value of some distribution when it is advantageous to do so.

In regard to the DR of the RX1 VS the RX1R, it could easily be that most RX1(s) cameras fall into a distribution that averages 13.9 ± 0.4 Evs. One would have to test a number of cameras to even begin to get a sense of the distribution.

I have never seen this level of data provided by the manufacturers, it's probably in their best self-interest to keep it proprietary.
Logged
Jay W Page

Bart_van_der_Wolf

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 8913
Re: DXOMark: RX1R scores (v. slightly) lower than RX1.
« Reply #9 on: August 24, 2013, 12:44:31 pm »

A sample size of one is simply a test and cannot be used to determine some metric about the overall population.

Hi,

You are assuming it's a sample size of one. It possibly is, be don't know.

Quote
It would be interesting to see how much variability exists for individual cameras for a whole variety of measurements, such as DR, resolution, % dead pixels, power consumption, shutter speed accuracy, etc. etc.

All camera's that I tested independently, had virtually identical DR scores (to 1/10th to 2/10th of a stop) compared to their tests. So that at least makes it a sample of 2 ;) Many independent testers also get pretty similar scores. On the rare occasion that there is an odd score, they seem to retest a camera, just to be sure.

Quote
One supposes that the camera manufacturers quote a mean value when stating the statistics and not a tail value of some distribution when it is advantageous to do so.

Camera manufacturers rarely present production model tests. We're lucky if they even publish manufacturing goals ...

Cheers,
Bart
Logged
== If you do what you did, you'll get what you got. ==

eronald

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 6642
    • My gallery on Instagram
Re: DXOMark: RX1R scores (v. slightly) lower than RX1.
« Reply #10 on: August 25, 2013, 11:00:57 am »

So, in essence, with so any minute variables it's best forgetting such 'scores' for the esoteric nonsense that they may actually be?

Not saying, just wondering.

Rob C

Rob,
 I don't call 0.7 stops minute if it corresponds to actual 0.7 visible stops. A difference of 0.7 stops is huge on location. A long time ago I took some street images of a girl with a hat to compare my then Canon 1Ds to Leica's DMR, and found that about 1/2 stop of DR was just about the *practical* difference, yielding very perceptibly different images with the hat's inside brim showing texture in one case and washed out into black and noise in the other -
 It's possible that the DR of a modern sensor is so huge that such a test today would be irrelevant, but my impression is that my old 1Ds is actually only about 1.5 to 2 practical stops or so behind my D3x, with a much better "look", and so a real-life difference of 0.7 stops would still be very relevant to many people here.
Note that I bring in the DR of the old 1Ds as our friend the Cooter has also indicated that in fact he could have just kept shooting the 1Ds right to today. What the camera manufacturers have done is move the models on an orbit of pixel numbers, different color filters, frame rates and features that present the appearance of constant improvement when they have just been going round and round.
 A couple of years ago I took the battered old 1Ds up to Montmartre and shot many of the painters there, and had exactly that issue - DR. In *practice* you never have enough of it; so frankly if I really knew that one $3K compact had 0.7 *practical* stops more DR than another, I'd say that would be worth about $1.5K of the price because compacts are typically used in outside scenes with strong contrasts, unlike studio cameras like the D3x which can expect a nice uniformly lit scene, and do well in that case.
 Of course, for most of us the difference between Kodachrome and Ektachrome is equally minute ;)
Edmund
« Last Edit: August 25, 2013, 11:27:28 am by eronald »
Logged
If you appreciate my blog posts help me by following on https://instagram.com/edmundronald

Rob C

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 24074
Re: DXOMark: RX1R scores (v. slightly) lower than RX1.
« Reply #11 on: August 25, 2013, 11:52:48 am »

Rob,
 I don't call 0.7 stops minute if it corresponds to actual 0.7 visible stops. A difference of 0.7 stops is huge on location. A long time ago I took some street images of a girl with a hat to compare my then Canon 1Ds to Leica's DMR, and found that about 1/2 stop of DR was just about the *practical* difference, yielding very perceptibly different images with the hat's inside brim showing texture in one case and washed out into black and noise in the other -
 It's possible that the DR of a modern sensor is so huge that such a test today would be irrelevant, but my impression is that my old 1Ds is actually only about 1.5 to 2 practical stops or so behind my D3x, with a much better "look", and so a real-life difference of 0.7 stops would still be very relevant to many people here.
Note that I bring in the DR of the old 1Ds as our friend the Cooter has also indicated that in fact he could have just kept shooting the 1Ds right to today. What the camera manufacturers have done is move the models on an orbit of pixel numbers, different color filters, frame rates and features that present the appearance of constant improvement when they have just been going round and round.
 A couple of years ago I took the battered old 1Ds up to Montmartre and shot many of the painters there, and had exactly that issue - DR. In *practice* you never have enough of it; so frankly if I really knew that one $3K compact had 0.7 *practical* stops more DR than another, I'd say that would be worth about $1.5K of the price because compacts are typically used in outside scenes with strong contrasts, unlike studio cameras like the D3x which can expect a nice uniformly lit scene, and do well in that case.
Of course, for most of us the difference between Kodachrome and Ektachrome is equally minute ;)
Edmund


Only if you didn't use them on the same format!

;-)

Rob C

eronald

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 6642
    • My gallery on Instagram
Re: DXOMark: RX1R scores (v. slightly) lower than RX1.
« Reply #12 on: August 25, 2013, 01:41:26 pm »


Only if you didn't use them on the same format!

;-)

Rob C

I still regret never having used Kodachrome in 120 format.

Edmund
Logged
If you appreciate my blog posts help me by following on https://instagram.com/edmundronald

Misirlou

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 711
    • http://
Re: DXOMark: RX1R scores (v. slightly) lower than RX1.
« Reply #13 on: August 26, 2013, 03:44:49 pm »

Me too. Big mistake.
Logged

digitalphotographer

  • Newbie
  • *
  • Offline Offline
  • Posts: 5
Re: DXOMark: RX1R scores (v. slightly) lower than RX1.
« Reply #14 on: August 26, 2013, 05:27:23 pm »

I don't call 0.7 stops minute if it corresponds to actual 0.7 visible stops. A difference of 0.7 stops is huge on location. A long time ago I took some street images of a girl with a hat to compare my then Canon 1Ds to Leica's DMR, and found that about 1/2 stop of DR was just about the *practical* difference, yielding very perceptibly different images with the hat's inside brim showing texture in one case and washed out into black and noise in the other -

It's possible that the DR of a modern sensor is so huge that such a test today would be irrelevant, but my impression is that my old 1Ds is actually only about 1.5 to 2 practical stops or so behind my D3x, with a much better "look", and so a real-life difference of 0.7 stops would still be very relevant to many people here.

A couple of years ago I took the battered old 1Ds up to Montmartre and shot many of the painters there, and had exactly that issue - DR. In *practice* you never have enough of it; so frankly if I really knew that one $3K compact had 0.7 *practical* stops more DR than another, I'd say that would be worth about $1.5K of the price because compacts are typically used in outside scenes with strong contrasts, unlike studio cameras like the D3x which can expect a nice uniformly lit scene, and do well in that case.

Hi all, I had to sign up here today so I could get some advice on this RX1R DR issue since there's not much help from DPR forums.  :(

I recently purchased the RX1R to use primarily as a street camera (some landscapes), but since I shoot mainly in very contrasty situations, I've found the files don't provide the "smooth highlight roll-off" look that I'm used to on the GR and D800E. In fact, RAW files are very contrasty, highlight areas are blown, even when I shoot with -0.3EV  :o

Without in-depth full reviews from DPR and field reports from Lloyd Chambers yet, I can't decide whether I've made the right choice to go for the "R" instead of RX1, especially some are attributing this to DXO's sampling issue or testing methodology.  If I had an RX1, I would be able to test them myself to see which renditions suit me, but without a control, how would I know if I'm constantly losing 0.7EV in DR (not sure if this is in highlight areas or normalized across the whole range)?  ???

I'm hoping to get some perspective here before my 14-day exchange/return window is up! This is one of the most expensive cameras I ever bought so I'm concerned. Only a few days left.  :-\
Logged

eronald

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 6642
    • My gallery on Instagram
Re: DXOMark: RX1R scores (v. slightly) lower than RX1.
« Reply #15 on: August 26, 2013, 07:25:08 pm »

Reviews are useful if you don't have the product.

You have the camera -either you like it or you don't.
Playing with the raw converter might help you make up your mind.
I think getting review info at this point is useless; you know what you want, and you can confront the actual equipment with your expectations.


However your description is interesting; it would be nice if you could get the sister camera to compare.
Smooth highlight roll-off seems to be dependent on the tuning of the antiblooming circuitry on new-gen digitals. This is firmware; however if is tuned to be forgiving, it seems to blur correctly exposed


Edmund

Hi all, I had to sign up here today so I could get some advice on this RX1R DR issue since there's not much help from DPR forums.  :(

I recently purchased the RX1R to use primarily as a street camera (some landscapes), but since I shoot mainly in very contrasty situations, I've found the files don't provide the "smooth highlight roll-off" look that I'm used to on the GR and D800E. In fact, RAW files are very contrasty, highlight areas are blown, even when I shoot with -0.3EV  :o

Without in-depth full reviews from DPR and field reports from Lloyd Chambers yet, I can't decide whether I've made the right choice to go for the "R" instead of RX1, especially some are attributing this to DXO's sampling issue or testing methodology.  If I had an RX1, I would be able to test them myself to see which renditions suit me, but without a control, how would I know if I'm constantly losing 0.7EV in DR (not sure if this is in highlight areas or normalized across the whole range)?  ???

I'm hoping to get some perspective here before my 14-day exchange/return window is up! This is one of the most expensive cameras I ever bought so I'm concerned. Only a few days left.  :-\
« Last Edit: August 26, 2013, 08:00:25 pm by eronald »
Logged
If you appreciate my blog posts help me by following on https://instagram.com/edmundronald

Dale_Cotton2

  • Full Member
  • ***
  • Offline Offline
  • Posts: 109
    • http://daystarvisions.com
Re: DXOMark: RX1R scores (v. slightly) lower than RX1.
« Reply #16 on: August 26, 2013, 08:39:10 pm »

digitalphotographer wrote:
> but without a control, how would I know if I'm constantly losing 0.7EV in DR (not sure if this is in highlight areas or normalized across the whole range)?

It's over the entire range.

> In fact, RAW files are very contrasty, highlight areas are blown, even when I shoot with -0.3EV

Not sure this is what you mean, but on DPR there's this pervasive mythology that a camera's metering can be made to prevent blown highlights by simply dialing in a permanent -X EV, where X often = 0.3 or 0.7.

In fact, few if any camera metering systems have any interest in preserving highlights. Instead their main focus is to keep the subject of the picture from being too dark or too bright. The subject being whatever is at your focus point. Even the R1 with 14.3 stops DR is going to give you blown highlights if you use this approach -- it'll just do it a little less often. If you have a dark subject in shadow, the camera is going to try to brighten that up for you, even at the expense of blowing highlights elsewhere in the frame, or even blowing out the entire sky.

Use the in-camera live histogram to keep the exposure from piling up on the right edge before you take a shot. Check after you take the shot to make sure the post-exposure histogram still shows that nothing has been clipped. One you nail the exposure in a given lighting situation you can often leave the same settings in place for a series of shots.

> I'm hoping to get some perspective here before my 14-day exchange/return window is up!

The DxO numbers have been pretty spot-on for the cameras I've owned. I'm having a hard time accepting the theory that they have made such a simple mistake as putting up such a glaring outlier number as this drop in DR is without making some effort to double-check it, either by testing another camera or by running their results by Sony. I well remember that they delayed their results for their testing of the Olympus EM-5 for many months while checking back with Olympus because they had concerns about the results they were seeing.

I very much doubt you're going to get either certain confirmation or certain contradiction of the DxO DR anomaly in the short time you have left. Another thing you don't yet know for certain is whether the presumed sharpness improvement for the R1R is particularly significant, although Michael Recihmann's take is that the diff is very minimal. FWIW: 13.6 stops is still a very respectable technical DR number that few other cameras have equalled; and notice that the R1R only deviates from the R1 at ISO 100; after that they're back in sync.

All that suggests to me that the safest course would seem to be to return it, then wait until you have enough review information to make the call.
Logged

hjulenissen

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2051
Re: DXOMark: RX1R scores (v. slightly) lower than RX1.
« Reply #17 on: August 27, 2013, 04:40:59 am »

Is it possible that the RX1R reviewed simply runs a newer/variant of the RX1 firmware that either:
1. Introduce a bug
2. Introduce a new feature/way of doing things that may or may not be a good thing for regular photography, but affects DXO testing in a nagative way

If this was the case, one may or may not expect RX1 cameras shipped from now on to have the same behaviour.

-h
Logged

KirbyKrieger

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 422
    • Kirby Krieger Pictures
Re: DXOMark: RX1R scores (v. slightly) lower than RX1.
« Reply #18 on: August 27, 2013, 12:42:28 pm »

Hi all, I had to sign up here today so I could get some advice on this RX1R DR issue since there's not much help from DPR forums.  :(

I recently purchased the RX1R to use primarily as a street camera (some landscapes), but since I shoot mainly in very contrasty situations, I've found the files don't provide the "smooth highlight roll-off" look that I'm used to on the GR and D800E. In fact, RAW files are very contrasty, highlight areas are blown, even when I shoot with -0.3EV  :o

I've had the RX1R for about three weeks, and have recorded over ... brztbzzrrrp ... 1,700 exposures: daytime, night-time, indoors, outdoors, low-light, macro, w. the built-in flash, with a Sony "20" flash, etc.  The RAW files I'm seeing have a lot of headroom and foot-room, and allow the usual adjustments even when the exposure recorded is poor.  What RAW converter are you using?  At what settings?

I do my conversion with Apple's Aperture (the converter is system wide: Apple RAW Compatibility Update 4.08).  The default settings for the RX1R were too "hot" for my needs (I decry this trend, but that's for another thread), so I created my own default:
    Boost:  0.80
    Hue Boost:  0.65
    Sharpening:  0.50
    Detail:  0.10
    Moire:  0.00
    Radius: 4.00
    De-Noise:  0.17

If anything, compared other Sony cameras (a850, a900, a77, Nex7) the RX1R tends to under-expose at the same settings.

What RAW converter are you using?  At what settings?  Have you tried any other converters or settings?

I'm no expert (but there are some here  ;) ), but those are the first places I'd look.

Here is "street" picture taken in sunlight, converted with Aperture at my default settings.  No other PP.  The lights and the darks are all "blown":
{Sorry -- can't get file to show.}

Here's a picture from the same file, cropped and with luminance adjustments applied (nothing special -- lowered the exposure and the black point):
{Sorry -- can't get file to show.}

digitalphotographer

  • Newbie
  • *
  • Offline Offline
  • Posts: 5
Re: DXOMark: RX1R scores (v. slightly) lower than RX1.
« Reply #19 on: September 11, 2013, 11:47:12 pm »

All that suggests to me that the safest course would seem to be to return it, then wait until you have enough review information to make the call.

Unfortunately on the last day of the return window, I was out shooting so couldn't return/exchange for RX1. Good news is I'm happy with most of the images from the camera.

But I would like to maximize the DR of the captures if possible. So I've been reading some of Lloyd Chamber's tests and articles on RX1R's ETTR, which indicates there is 1-2 stops of headroom buried in default metering. This brings up a few more questions about DXO's testing. Did they consider ETTR? Is their methodology limited in showing sensor performance since I seem to recall the final output is jpg?

Another interesting thread on this forum here called "Michael Reichman, ETTR and OMD EM5" at http://www.luminous-landscape.com/forum/index.php?topic=68334.0 brings up another question. EM5's performance is documented there, but it seems RX1/RX1R has been tested or discussed yet. I would really like to know what "Photographic Dynamic Range Shadow Improvement of RX1/RX1R" is.  Unfortunately, Bill Claff hasn't measured it yet (not sure if he ever will). I just think it would be awesome to know the max ISO I should raise on RX1R during ETTR shooting.

Hope I am making sense.
« Last Edit: September 12, 2013, 04:54:10 am by digitalphotographer »
Logged
Pages: [1]   Go Up