Pages: [1] 2 3 4   Go Down

Author Topic: DxO Mark  (Read 33349 times)

Guillermo Luijk

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2005
    • http://www.guillermoluijk.com
DxO Mark
« on: November 17, 2008, 12:35:34 pm »

A new site by DxO Labs has appeared: DxO Mark, comparing sensors and cameras with a rigurous approach (they claim to focus on pure RAW undemosaiced data).

Some of the statements such as: "Contrary to conventional wisdom, higher resolution actually compensates for noise" (copy & paste: www.dxomark.com/index.php/eng/Insights/More-pixels-offsets-noise!) make me thing it can become a quite interesting place for reference, contrarily to common JPEG sites such as DPreview.

Just the overall DR tests, not per-pixel DR (the Y-scale seems rounded but plots seem correct). All new FF's have similar DR but into that group the A900, with the highest pixel count, won:



BR
« Last Edit: November 17, 2008, 06:47:49 pm by GLuijk »
Logged

DarkPenguin

  • Guest
DxO Mark
« Reply #1 on: November 17, 2008, 12:48:32 pm »

I found this introduction to that site on the web.
Logged

Guillermo Luijk

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2005
    • http://www.guillermoluijk.com
DxO Mark
« Reply #2 on: November 17, 2008, 12:52:15 pm »

Quote from: DarkPenguin
I found this introduction to that site on the web.
oops! I did a search about dxomark.com on the forum but forgot to check the main page. Anyway this thread can be used for discusion.
« Last Edit: November 17, 2008, 12:52:25 pm by GLuijk »
Logged

BernardLanguillier

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 13983
    • http://www.flickr.com/photos/bernardlanguillier/sets/
DxO Mark
« Reply #3 on: November 17, 2008, 05:31:25 pm »

Quote from: GLuijk
Just the overall DR tests, not per-pixel DR (the Y-scale seems rounded but plots seem correct). All new FF's have similar DR but into that group the A900, with the highest pixel count, won:


Good performance by the A900, although this is probably within measuring errors of the other 3.

By the way, is that pixel level DR or image level DR? I would think that pixel level DR is more important since image level DR will make the larger image look better at a given print size thanks to the smoothing effect of downsizing on shadow noise, right?

Cheers,
Bernard

douglasf13

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 547
DxO Mark
« Reply #4 on: November 17, 2008, 06:15:17 pm »

Quote from: GLuijk
A new site by DxO Labs has appeared: DxO Mark, comparing sensors and cameras with a rigurous approach (they claim to focus on pure RAW undemosaiced data).

Some of the statements such as: Contrary to conventional wisdom, higher resolution actually compensates for noise! make me thing it can become a quite interesting place for reference, contrarily to common JPEG sites such as DPreview.

Just the overall DR tests, not per-pixel DR (the Y-scale seems rounded but plots seem correct). All new FF's have similar DR but into that group the A900, with the highest pixel count, won:



BR

  I noticed that this site uses the cameras lowest ISO, which they list as ISO 100 on the A900.  ISO 200 is the actual base ISO of the A900, and most tests I've seen give ISO 200 nearly a +1EV stop of DR compared to ISO 100, so the A900 may actually have even better DR than this new test shows.  What do you think?



Logged

douglasf13

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 547
DxO Mark
« Reply #5 on: November 17, 2008, 06:43:27 pm »

Quote from: douglasf13
I noticed that this site uses the cameras lowest ISO, which they list as ISO 100 on the A900.  ISO 200 is the actual base ISO of the A900, and most tests I've seen give ISO 200 nearly a +1EV stop of DR compared to ISO 100, so the A900 may actually have even better DR than this new test shows.  What do you think?

  Never mind.  They tested a range of ISOs.  Strange that ISO 100 is higher than 200
Logged

Guillermo Luijk

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2005
    • http://www.guillermoluijk.com
DxO Mark
« Reply #6 on: November 17, 2008, 06:45:15 pm »

Quote from: BernardLanguillier
By the way, is that pixel level DR or image level DR? I would think that pixel level DR is more important since image level DR will make the larger image look better at a given print size thanks to the smoothing effect of downsizing on shadow noise, right?

I think DR measures are at image level, they speak about that here: www.dxomark.com/index.php/eng/Insights/More-pixels-offsets-noise!
(just copy & paste the address including the ! in the end since it seems the LL forum engine does not allow the ! symbol for links).
Otherwhise the 1Ds3 and A900 would surely have worse per-pixel SNR and DR than the Nikons.
« Last Edit: November 17, 2008, 06:47:14 pm by GLuijk »
Logged

Ray

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 10365
DxO Mark
« Reply #7 on: November 17, 2008, 07:54:44 pm »

Quote from: GLuijk
Some of the statements such as: "Contrary to conventional wisdom, higher resolution actually compensates for noise" .........

I thought this fact had been conventional wisdom for quite some time and that only a few diehards like Gabor (Panopeeper) objected to trading resolution for lower noise.  

This new site seems an excellent idea to me. I'm all in favour of more rigorous and more meaningful testing which relates more directly to the final product, which is usually a print, or image on screen, of a particular size. You can't have a sizeless image or print.
Logged

DarkPenguin

  • Guest
DxO Mark
« Reply #8 on: November 17, 2008, 10:24:30 pm »

Iliah Borg over at dpreview doesn't seem to think a lot of the results from dxomark so far...

http://forums.dpreview.com/forums/read.asp...essage=30054057
Logged

Guillermo Luijk

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2005
    • http://www.guillermoluijk.com
DxO Mark
« Reply #9 on: November 18, 2008, 01:11:21 am »

Quote from: DarkPenguin
Iliah Borg over at dpreview doesn't seem to think a lot of the results from dxomark so far...

http://forums.dpreview.com/forums/read.asp...essage=30054057
As Emil says, it's a brand new site in beta stage and we still don't know deeply what methods they are using.

Also for example if you plot DR of the 40D vs 50D (the so much criticised review in DPreview), for both cameras DR is slightly higher for ISO200 than for ISO100. I can only think of that happening if saturation points at those ISO values vary so much to compensate SNR loss at a given RAW exposure when going from ISO100 to ISO200. Looking at the SNR plots I rather would think they didn't make a very accurate calculation in the low end (deep shadows). Perhaps they applied the same data processing to a big group of samples (they have reviewed many cameras) and the measures for some of them were not enough accurate at that critical point. Or simply they did not have data reaching the low end and some approximation was done.

See here the Canon 50D strange behaviour in the low end:




And for the case of the Nikon D90 there also seems to be an unexpected behaviour in the 4th curve (ISO 1600) that could lead to a such suspicious DR curve plot:




So surely there are things to be fine tuned in the results and methods, after all it's a massive beta version of the site, but IMO the beginning seems promising compared to other comparisions using procedures sometimes out of any logic (DPreview).

Let's see if Emil and Iliah hang around here.
« Last Edit: November 18, 2008, 01:28:20 am by GLuijk »
Logged

marcmccalmont

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1780
DxO Mark
« Reply #10 on: November 18, 2008, 01:37:51 am »

I'd like to see popular MFDB's included in the test results.
Marc
Logged
Marc McCalmont

ejmartin

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 575
DxO Mark
« Reply #11 on: November 18, 2008, 10:32:07 am »

Canon DR usually goes up from ISO 100 to 200 in current models for a combination of two reasons: (1) read noise goes down by almost a factor of two; and (2) the sensor saturates well before the max raw level 16383 at ISO 100.  This means that one loses less than a stop at the highlight end in going from ISO 100 to 200, while getting nearly a stop at the shadow end.

It does seem that there are some glitches in the measurements at low signal in their SNR plots.  I would think it better to use the measurements to fit the data to the universal noise vs signal formula, taking as many points are required to get a good chi-squared.  Then plot the universal formula with the fitted parameters as the curve, with the data points to show the goodness of fit.  And use the fit rather than the measured data to determine the DR etc, so that individual data points (especially in shadows) don't skew the results.  My guess though is that they're using the measured points directly, or with simple spline-type fits rather than fits to the universal noise formula, and so if they don't have enough data they'll have more substantial errors in the results.
Logged
emil

Tony Beach

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 452
    • http://imageevent.com/tonybeach/twelveimages
DxO Mark
« Reply #12 on: November 18, 2008, 11:12:39 am »

Since the D3 and D700 use the same sensor but get different DR evaluations, I see an obvious problem with this site's credibility.

I also wonder why anyone would care if their camera is better or worse at some extraordinarily negligible level of detail.  When cameras had clear image quality advantages, then comparisons were useful, but at this point it is not worth the efforts some are putting into trying to determine which camera is "better".
Logged

Ray

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 10365
DxO Mark
« Reply #13 on: November 18, 2008, 11:32:35 am »

Quote from: ejmartin
Canon DR usually goes up from ISO 100 to 200 in current models for a combination of two reasons: (1) read noise goes down by almost a factor of two; and (2) the sensor saturates well before the max raw level 16383 at ISO 100.  This means that one loses less than a stop at the highlight end in going from ISO 100 to 200, while getting nearly a stop at the shadow end.

Emil,
This seems to be very useful information to have. I think that most photographers would tend to assume that best results with respect to tonality, DR and noise will be achieved at base ISO. As a consequence, one might choose a merely adequate shutter speed at ISO 100 for the sake of better DR, in preference to a shutter speed twice as fast at ISO 200. The faster shutter speed is likely to produce at least marginally sharper results. However, if the lower ISO of 100 has no benefit with regard to DR, one might wonder what purpose is served by using it, unless one needs a slower shutter speed for a particular effect.
Logged

Guillermo Luijk

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2005
    • http://www.guillermoluijk.com
DxO Mark
« Reply #14 on: November 18, 2008, 11:48:53 am »

Quote from: Tony Beach
Since the D3 and D700 use the same sensor but get different DR evaluations, I see an obvious problem with this site's credibility.

I also wonder why anyone would care if their camera is better or worse at some extraordinarily negligible level of detail.  When cameras had clear image quality advantages, then comparisons were useful, but at this point it is not worth the efforts some are putting into trying to determine which camera is "better".
This DR test over the new FF cameras demonstrates that regarding DR, the differences among them are negligible. And that's a _very_ useful information to focus on other characteristics to make a choice. If you plot the result of the Canon 350D, or the Fuji S5 Pro vs these FF cameras, you will see important differences in DR.

I don't see a problem in the D3 and D700 comparision, their DR results are so close that could well be due to differences in the units under test (just look at the Y-scale). The point is not only which camera got a higher or lower DR in the test, but how much higher or how much lower. Again the conclusion is 'they have the same DR' and that allows to pay attention to other parameters (e.g. price).
« Last Edit: November 18, 2008, 12:10:24 pm by GLuijk »
Logged

Guillermo Luijk

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2005
    • http://www.guillermoluijk.com
DxO Mark
« Reply #15 on: November 18, 2008, 12:53:06 pm »

Quote from: ejmartin
Canon DR usually goes up from ISO 100 to 200 in current models for a combination of two reasons: (1) read noise goes down by almost a factor of two; and (2) the sensor saturates well before the max raw level 16383 at ISO 100.  This means that one loses less than a stop at the highlight end in going from ISO 100 to 200, while getting nearly a stop at the shadow end.
Emil, please correct me if I am wrong: what you mean with this is that:

1. These Canon cameras _almost_ double their S/read noise ratio when going from ISO100 to ISO200, and that is equivalent to having very close curve plots in the low end (never crossing curves however), and that is equivalent to admitting that almost all read noise is produced after the ISO amplification.
2. On the other side, the difference in saturation point (lower at ISO100 than at ISO200) means more extra DR in the right end than what we loose in the left end when going from ISO100 to ISO200.

Being * the saturation points for ISO100 and ISO200 this is what we are talking about:



Questions:
1. If the saturation points were the same, we would _never_ get more DR at a higher ISO no matter how much improvement in S/read noise we could get from ISO100 to ISO200. The more we could expect in that case would be to get the same DR. Right?
2. If it is true that for a given camera a higher ISO provides more DR, unless we need extra long exposure times, it would be completely stupid to use the lower ISO for any application (this is what Ray asked).
3. Why is that big difference in the Canon's saturation points? at ISO100 my cheap old 350D saturates at 4095, while the 5D saturates at 3692. I never understood this. It seems as if the output of the ISO amplification gets clipped before entering the ADC, and even that clipping point depends on the ISO set.

BR
« Last Edit: November 18, 2008, 12:57:47 pm by GLuijk »
Logged

douglasf13

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 547
DxO Mark
« Reply #16 on: November 18, 2008, 12:57:19 pm »

Quote from: GLuijk
This DR test over the new FF cameras demonstrates that regarding DR, the differences among them are negligible. And that's a _very_ useful information to focus on other characteristics to make a choice. If you plot the result of the Canon 350D, or the Fuji S5 Pro vs these FF cameras, you will see important differences in DR.

I don't see a problem in the D3 and D700 comparision, their DR results are so close that could well be due to differences in the units under test (just look at the Y-scale). The point is not only which camera got a higher or lower DR in the test, but how much higher or how much lower. Again the conclusion is 'they have the same DR' and that allows to pay attention to other parameters (e.g. price).

  Agreed.  In fact, I'd say that, other than making the choice between the lower noise of the D3 vs. the resolution of the 1Dsiii/A900, there really isn't much between the FF cameras.  It seems that price, handling, camera options and system options are the more relevant reasons to make a purchasing decision.

Logged

ejmartin

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 575
DxO Mark
« Reply #17 on: November 18, 2008, 01:11:41 pm »

Quote from: douglasf13
Agreed.  In fact, I'd say that, other than making the choice between the lower noise of the D3 vs. the resolution of the 1Dsiii/A900, there really isn't much between the FF cameras.  It seems that price, handling, camera options and system options are the more relevant reasons to make a purchasing decision.

And there you have it in a nutshell.
Logged
emil

ejmartin

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 575
DxO Mark
« Reply #18 on: November 18, 2008, 01:23:45 pm »

Quote from: GLuijk
Emil, please correct me if I am wrong: what you mean with this is that:

1. These Canon cameras _almost_ double their S/read noise ratio when going from ISO100 to ISO200, and that is equivalent to having very close curve plots in the low end (never crossing curves however), and that is equivalent to admitting that almost all read noise is produced after the ISO amplification.

Yes the curves shouldn't cross; the examples you showed where they do represent flaws in the data.

Quote
Being * the saturation points for ISO100 and ISO200 this is what we are talking about:



Questions:
1. If the saturation points were the same, we would _never_ get more DR at a higher ISO no matter how much improvement in S/read noise we could get from ISO100 to ISO200. The more we could expect in that case would be to get the same DR. Right?

Correct.

Quote
2. If it is true that for a given camera a higher ISO provides more DR, unless we need extra long exposure times, it would be completely stupid to use the lower ISO for any application (this is what Ray asked).

Because, though you lose a bit of DR at ISO 100 on Canons (at the cost of a stop in Tv/Av as well), you gain in S/N throughout the range.  Any particular feature in the image will be less noisy.

Quote
3. Why is that big difference in the Canon's saturation points? at ISO100 my cheap old 350D saturates at 4095, while the 5D saturates at 3692. I never understood this. It seems as if the output of the ISO amplification gets clipped before entering the ADC, and even that clipping point depends on the ISO set.

It's to do with the native ISO of the sensor relative to the gain applied by the variable gain amplifier.  Canon has chosen their lowest gain setting such that the sensor clips before raw saturation in the newer sensors.  Since the newer ones are more efficient, if the same amplification is used and the well capacity remains about the same, the photosite will saturate more rapidly (one might say that the "native" ISO of the sensor is higher on the newer camera sensors).  Of course these things are never kept fixed from models to model (due in particular to changing pixel pitch), but it is true that the native sensor ISO has increased and so one has the possibility that the sensor saturates earlier for a given ISO setting.
Logged
emil

Guillermo Luijk

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2005
    • http://www.guillermoluijk.com
DxO Mark
« Reply #19 on: November 18, 2008, 01:45:18 pm »

Quote from: ejmartin
It's to do with the native ISO of the sensor relative to the gain applied by the variable gain amplifier.  Canon has chosen their lowest gain setting such that the sensor clips before raw saturation in the newer sensors.  Since the newer ones are more efficient, if the same amplification is used and the well capacity remains about the same, the photosite will saturate more rapidly (one might say that the "native" ISO of the sensor is higher on the newer camera sensors).  Of course these things are never kept fixed from models to model (due in particular to changing pixel pitch), but it is true that the native sensor ISO has increased and so one has the possibility that the sensor saturates earlier for a given ISO setting.
I see, but then if a given sensor saturates at ISO100 at (let's say) ADU=13000, which is less than 1 f-stop far from RAW saturation (ADU=16383), any other ISO should clip at RAW saturation. Does this happen?
For instance the 40D which saturates at 13823 (from my tests) at ISO100, should saturate at 16383 for any other higher ISO value. Does it?
« Last Edit: November 18, 2008, 01:46:24 pm by GLuijk »
Logged
Pages: [1] 2 3 4   Go Up