Pages: 1 [2] 3 4   Go Down

Author Topic: DxO Mark  (Read 33345 times)

ejmartin

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 575
DxO Mark
« Reply #20 on: November 18, 2008, 02:44:18 pm »

Quote from: GLuijk
I see, but then if a given sensor saturates at ISO100 at (let's say) ADU=13000, which is less than 1 f-stop far from RAW saturation (ADU=16383), any other ISO should clip at RAW saturation. Does this happen?
For instance the 40D which saturates at 13823 (from my tests) at ISO100, should saturate at 16383 for any other higher ISO value. Does it?

Close enough.  John Sheehy did some analysis of the 30D a while back and the clipping levels were not exactly 4095, but close enough that it doesn't affect things on the log scale of EV.  The difference between 13823 and 16383 however is close to 1/4 stop.  I did some 40D tests which gave clipping at 16383 for the green channel at ISO 200 and up, IIRC.
Logged
emil

Panopeeper

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1805
DxO Mark
« Reply #21 on: November 18, 2008, 03:06:09 pm »

Quote from: ejmartin
Canon DR usually goes up from ISO 100 to 200 in current models for a combination of two reasons: (1) read noise goes down by almost a factor of two; and (2) the sensor saturates well before the max raw level 16383 at ISO 100.  This means that one loses less than a stop at the highlight end in going from ISO 100 to 200, while getting nearly a stop at the shadow end

1. The DR of both the Canon 40D and 50D is larger at ISO 100 than at ISO 200.

2. The saturation level plays no role in the DR; the ISO 100 values are scaled down, probably from the base ISO, which appears to be 1/4 EV over 100.
Logged
Gabor

Ray

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 10365
DxO Mark
« Reply #22 on: November 18, 2008, 06:49:24 pm »

Looks like I'll have to carry out some testing, when I have the time, comparing shadow and highlight detail at ISOs 100 and 200. I know that cameras do vary from copy to copy (of the same model), just as lenses do. Years ago, after collecting my new 5D, I wasn't happy with the severe banding in the darkest shadows. I returned it and tried another copy which was better. I wouldn't say much better, but noticeably better.
Logged

Tony Beach

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 452
    • http://imageevent.com/tonybeach/twelveimages
DxO Mark
« Reply #23 on: November 18, 2008, 07:57:56 pm »

Forgive me for continuing to piss all over this thread, but I find these discussions not to be useful to me and question their purpose.

Quote from: DarkPenguin
Iliah Borg over at dpreview doesn't seem to think a lot of the results from dxomark so far...

It's more fundamental than whether or not DxO is "accurately" measuring the characteristics of these sensors.  What is the point of dissecting RAW data absent any conversion of that data?  If fixating on differences in sensors makes any difference at all to the outcome of your photographs, then equal attention should be paid to RAW converters:  http://forums.dpreview.com/forums/read.asp...essage=30062493  You can't simply divorce the data coming from the sensor from the data that comes from conversion of that file; after all, the later is what you will ultimately be working with when you craft your photograph.  This leads me to question what some people in these photography forums are attempting to craft:  better photographs, or better ways of measuring RAW data.  For me, the tools I care about are the ones that deliver files I can use to make better photographs with, and DxO Mark is a diversion from that goal.
Logged

Ray

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 10365
DxO Mark
« Reply #24 on: November 18, 2008, 08:32:09 pm »

Tony has a point. We all have to use a RAW converter of one type or another and that choice of RAW converter might be based on lots of factors that are not directly related to getting the last ounce of detail from highlights or shadows. We might use a converter because we like the results, the work-flow, and like the ease of getting those results, or perhaps because we are simply familiar with the interface.

We should not forget that DXO are in the business of creating their own RAW converter. Their tests, which they are now sharing with everyone, are presumably necessary for their own purposes.
Logged

ejmartin

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 575
DxO Mark
« Reply #25 on: November 18, 2008, 08:49:59 pm »

Quote from: Tony Beach
Forgive me for continuing to piss all over this thread, but I find these discussions not to be useful to me and question their purpose.

It's more fundamental than whether or not DxO is "accurately" measuring the characteristics of these sensors.  What is the point of dissecting RAW data absent any conversion of that data?

Because it provides a measure of the camera's capabilities rather than the sum total of the camera's capabilities filtered through a particular raw converter's capability.  Usually the scientific method is to strip away those aspects of the subject under study that don't pertain to the subject under study.  If you got a lousy result from camera A with raw converter X, you can't know if the issue is camera A or raw converter X.  If you study the raw data of camera A without going through conversion, and you find it to be poorer quality than that of camera B, then no raw converter is going to make camera A's output look better than camera B's provided it is doing an equally competent job on each.  The latter issue should be the subject of a separate investigation, which to my knowledge is yet to be done in a scientific manner for commercial raw converters.

Better to isolate the variable under study than to lump it in with all sorts of other unknowns.

Quote
If fixating on differences in sensors makes any difference at all to the outcome of your photographs, then equal attention should be paid to RAW converters:  http://forums.dpreview.com/forums/read.asp...essage=30062493  You can't simply divorce the data coming from the sensor from the data that comes from conversion of that file; after all, the later is what you will ultimately be working with when you craft your photograph.  This leads me to question what some people in these photography forums are attempting to craft:  better photographs, or better ways of measuring RAW data.  For me, the tools I care about are the ones that deliver files I can use to make better photographs with, and DxO Mark is a diversion from that goal.

I agree that equal attention should be paid to raw converters, and that hasn't been done yet.  That doesn't mean it can't be done, and it doesn't mean that raw data analysis is without meaning or worth.  
Logged
emil

Panopeeper

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1805
DxO Mark
« Reply #26 on: November 18, 2008, 09:07:05 pm »

Quote from: Tony Beach
I find these discussions not to be useful to me and question their purpose

Well, obviously the purpose of pleasing you is failed miserably. I guess you need to ask Michael to close the thread.

Quote
What is the point of dissecting RAW data absent any conversion of that data?

It is called abstraction. The information gained this way reflects the characteristics of the camera, independently of the raw converter.

If this is not done, then one could believe, that the camera can do only as good as it appears from certain specific conversions. For example DPReview conducted tests of the 50D with the preliminary version of ACR, which contained some error (IMO it still does). The 40D raw data interpretation was wrong from the outset. I reported the error, which has been turned into another one. Measuring the DR based on that interpretation of the raw data leads to an incorrect result (but DPReview can not measure the DR anyway).

Do you find it useful to evaluate a camera based exclusively on a specific interpretation of the raw data?
Logged
Gabor

Tony Beach

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 452
    • http://imageevent.com/tonybeach/twelveimages
DxO Mark
« Reply #27 on: November 18, 2008, 11:26:59 pm »

Quote from: Panopeeper
Well, obviously the purpose of pleasing you is failed miserably. I guess you need to ask Michael to close the thread.

No need to be flippant; I'm just expressing my point of view.  No, I would never presume to ask Michael to close any thread in his forums.

Quote
If this is not done, then one could believe, that the camera can do only as good as it appears from certain specific conversions.

I'm certainly not advocating that; in fact I'm thinking you and I agree more than you think we do as far as that is concerned.  However, a camera's RAW files are only as good as what they can be converted to.  While that may require a better RAW conversion algorithm; if none exists then that becomes very problematic to a photographer who doesn't have the knowledge or means to create an ideal RAW converter -- then you move on to another camera -- but I don't think this a really a problem with the recent DSLRs and associated RAW converters we currently have available to us.

Quote
Do you find it useful to evaluate a camera based exclusively on a specific interpretation of the raw data?

Yes, specifically with the "interpretation" from the converter I would be using -- which for me would be the one that delivered the best results.  Some people complain that superior results from a particular converter are attained because of NR being applied (often they are wrong), but I don't care as what matters to me is that the level of noise and the amount of detail are optimized, and what I find using NX with my D300 is that I have both less noise and more detail than what I get with other converters.  Should I dismiss the results I know I can get because some website uses a different (putatively neutral) converter and then claims that some other camera does a better job than mine using that other converter?  Should I care if someone claims another camera gets better DR than mine because some "abstract" analysis says so, particularly when I can get what I consider extremely good results with the camera I have?
Logged

madmanchan

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2115
    • Web
DxO Mark
« Reply #28 on: November 19, 2008, 12:07:24 am »

The strength of the DxO Mark method and similar methods is that it allows for accurate characterization of a sensor and/or optical system.

The weakness of the method is that final image quality (e.g., what is seen in a print or on a display, such as in a web gallery) depends on much more than raw-level characterization.

For example, a raw-level characterization may show more promise for the Panasonic DMC-LX3's sensor compared to a Canon PowerShot G10's sensor, but raw files rendered using these makers' respective software (SilkyPix and Digital Photo Professional) may yield opposite results. This may seem initially like a bogus comparison because different software products are being used, but on the other hand it's a common scenario that will apply to many users (i.e., owners of these two cameras will simply use the software that comes with the camera).

This is the classic tradeoff of the scientific method. By isolating specific variables, you can get a very good understanding of that variable within a limited context. The downside is that it's insufficient to explain the big picture or the broader context, without a (much more difficult and time-consuming) examination of the whole context. For example, we have detailed models of the human eye, which can explain certain visual phenomena but is insufficient to model color appearance. As another example, there has been much study of the individual contributions of various nutrients (vitamin B, antioxidants, etc.) to health, which unfortunately are poor predictors in general because they do not take into account the much larger complex system (i.e., how all of these elements interact).

Don't get me wrong. I applaud DxO for their efforts in launching this site and for making the (summarized) data available. It is certainly the right way to go when developing raw conversion software. But if you don't develop raw conversion software, and you're just a good ol' photographer, you have to ask yourself if the numbers/data are going to be relevant to your workflow. Or, put another way, if the doctor tells you to eat fewer eggs because lower cholesterol means a smaller chance of a heart attack, you should ask if that is really the factor that matters.
Logged
Eric Chan

douglasf13

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 547
DxO Mark
« Reply #29 on: November 19, 2008, 12:33:07 am »


  As a Sony user, I can say that the two worst RAW converters I've seen for the A900 files are ACR and....Sony's own IDC!        Since nearly every review site uses one of those two converters for RAW development with the Sony, I think it's good to show what the sensor is capable of.  Aperture and Capture One are two examples of converters that don't mush the Sony RAWs, and I'm hoping the new Bibble acts similarly.
Logged

jani

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1624
    • Øyet
DxO Mark
« Reply #30 on: November 19, 2008, 03:34:13 am »

Quote from: Tony Beach
I'm certainly not advocating that; in fact I'm thinking you and I agree more than you think we do as far as that is concerned.  However, a camera's RAW files are only as good as what they can be converted to.  While that may require a better RAW conversion algorithm; if none exists then that becomes very problematic to a photographer who doesn't have the knowledge or means to create an ideal RAW converter -- then you move on to another camera -- but I don't think this a really a problem with the recent DSLRs and associated RAW converters we currently have available to us.
The point of an analysis without the raw converter would then be to determine the potential for improvement in raw conversion.

This kind of data may not be useful to regular photographers per se, but it could be useful to those making raw converters or otherwise working directly on raw data. It could also be useful to camera sensor makers.
Logged
Jan

Guillermo Luijk

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2005
    • http://www.guillermoluijk.com
DxO Mark
« Reply #31 on: November 19, 2008, 07:01:48 am »

Quote from: madmanchan
This is the classic tradeoff of the scientific method. By isolating specific variables, you can get a very good understanding of that variable within a limited context. The downside is that it's insufficient to explain the big picture or the broader context, without a (much more difficult and time-consuming) examination of the whole context.
I agree that results based solely on measures over RAW data providing an objective view of the hardware of the camera, need some extra interpretation effort to extrapolate results to what we can expect to achieve in our real photographs. However I prefer by far these results than those more obvious DPreview samples. After all we are not always going to use our camera to take JPEG pictures of a stamp with the queen of England.

BR
« Last Edit: November 19, 2008, 07:28:28 am by GLuijk »
Logged

Guillermo Luijk

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2005
    • http://www.guillermoluijk.com
DxO Mark
« Reply #32 on: November 19, 2008, 07:23:49 am »

Quote from: ejmartin
Yes the curves shouldn't cross; the examples you showed where they do represent flaws in the data.
Emil, I was thinking that perhaps they already corrected the saturation point of the camera at each ISO to fit 100% in the X-axis of their plots, contrarily to your plots that strictly display ADU levels on that axis where the right end is always RAW saturation (ADU=2^14-1).
In that case their curves could cross, and in fact this would show in a very intuitive way in the plots what you explained on how some cameras can reach a higher DR for a given higher ISO, but at the cost of worse SNR in most of the range.

I think it could be a good idea that you show the sat point for each ISO (typically just ISO100 should differ from 2^14-1) in your graphs, specially for proper DR measures on them.

BR
« Last Edit: November 19, 2008, 07:34:42 am by GLuijk »
Logged

ejmartin

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 575
DxO Mark
« Reply #33 on: November 19, 2008, 08:40:45 am »

Quote from: GLuijk
Emil, I was thinking that perhaps they already corrected the saturation point of the camera at each ISO to fit 100% in the X-axis of their plots, contrarily to your plots that strictly display ADU levels on that axis where the right end is always RAW saturation (ADU=2^14-1).
In that case their curves could cross, and in fact this would show in a very intuitive way in the plots what you explained on how some cameras can reach a higher DR for a given higher ISO, but at the cost of worse SNR in most of the range.

I think it could be a good idea that you show the sat point for each ISO (typically just ISO100 should differ from 2^14-1) in your graphs, specially for proper DR measures on them.

BR


I agree, but I don't know the saturation point for some of the cameras.
Logged
emil

bjanes

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 3387
DxO Mark
« Reply #34 on: November 19, 2008, 10:40:37 am »

Quote from: ejmartin
I agree, but I don't know the saturation point for some of the cameras.

Saturation and clipping are more complex than one might initially think. To test the saturation of my Nikon D3, I grossly overexposed an image of a lightbox at various ISOs with the camera set to record losslessly compressed NEFs. The indicated exposure for ISO 200 was 1/1000 s at f/2.8 and I used 2 s at f/2.8, so I am reasonably certain that all pixels are saturated. I then used Iris to separate the Green 1, Green 2, Red and Blue channels and examined the histograms of the ISO 200 and ISO 3200 images in ImageJ:

[attachment=9786:Histograms.gif]

Clipping can occur when the sensor is saturated or when the ADC overflows. The maximum value for a 14 bit data number is 16383. The green channels at ISO 200 never reach 16383. Due to PRNU (Pixel Response Nonuniformity), the pixels saturate at various levels as shown in the histograms. The camera gain is such that the ADC does not overflow. For the red channel at ISO 200, the gain is such that at pixel saturation, the ADC overflows for all pixels as shown in the histogram. For the blue channel, the gain is such that some pixels cause overflow of the ADC but others do not and PRNU is again observed.

I thought that these results were interesting and worth posting.

Bill
Logged

Guillermo Luijk

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2005
    • http://www.guillermoluijk.com
DxO Mark
« Reply #35 on: November 19, 2008, 03:47:12 pm »

Quote from: bjanes
Saturation and clipping are more complex than one might initially think.
The strangest saturation point I have found is that of the R sensor in the Fuji Super CCD. Looking at the histograms obtained with increasing exposure one could think it never saturates since blown information never concentrates at any particular point but along a wide range of values in the form of a gaussian distribution. See here the EV histogram (each of them normalised to its maximum) with subsequently increasing exposure of +1EV:




If we look at the last histogram in linear mode we see all the saturated areas concentrated in a gaussian bell distribution in the right half:




One could think that's information compression (which would mean a huge DR!), but unluckily it is not, it seems to be just noise. Yes, noise in the highlights but noise after all with no texture or detail on it. And the three RGB channels of that bell are aligned in the RAW data (i.e. prior to WB which is the way I calculated these histograms), that is why once WB is applied, R and B relative exposures increase with respect to G and this causes the typical Fuji magenta highlights for having G shortage that becomes visible when applying a strong exposure correction down (what others call highlight recovery). So when a magenta cast appears on a Fuji image it just means one thing: saturation of the R sensor (of course in this situation the S sensor saturated time ago).

These images were obtained from the same RAF file, first with ACR and strong exposure correction down by -4EV, second using a not so strong exposure correction to try to avoid the magenta cast, and last with Zero Noise (DCRAW) using as saturation point the beginning of the gaussian bell, i.e. deliberately clipping to pure white all the 'information' that could be packed into that gaussian distribution:



The Zero Noise result displays the same information as ACR with the -4EV correction down but without the channel misalignment thanks to the specific saturation point set in DCRAW. Also ACR and DCRAW show differences in the WB applied to the R sensor, when both apply the same WB to the S sensor (i.e. the shadows of any RAF).

This particular behaviour of the R sensor in the Super CCD can easily lead to being optimistic (~0.5 f-stops) in the calculation of the DR for the Fuji cameras if this effect is not properly taken into account.
« Last Edit: November 19, 2008, 05:39:31 pm by GLuijk »
Logged

bjanes

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 3387
DxO Mark
« Reply #36 on: November 22, 2008, 11:41:35 am »

Quote from: GLuijk
The strangest saturation point I have found is that of the R sensor in the Fuji Super CCD. Looking at the histograms obtained with increasing exposure one could think it never saturates since blown information never concentrates at any particular point but along a wide range of values in the form of a gaussian distribution. See here the EV histogram (each of them normalised to its maximum) with subsequently increasing exposure of +1EV:

This particular behaviour of the R sensor in the Super CCD can easily lead to being optimistic (~0.5 f-stops) in the calculation of the DR for the Fuji cameras if this effect is not properly taken into account.

Guillermo,

These data are interesting, and I am surprised that no one has commented. I'm not that familiar with the Fuji super CCD and the R sensor component, but I would imagine that it would behave similarly to other CCDs with only one type of pixel.

In performing a sensor analysis of my D3 (CMOS sensor) using paired images at various exposures using the method of Roger Clark, one can make some interesting observations. I used Iris to separate the green 1 channel and cropped the image to 201 x 201 pixels. I then subtracted one pair from the other and added an offset of 200 to avoid negative numbers and determined the standard deviations. The standard deviations of the single frames represents total noise, while the difference standard deviation removes variations due to non-uniformity of the light field (e.g. due to vignetting in the lens), dust specs on the sensor, pixel to pixel sensitivity variations and other effects.

This table shows the results of some of the data. Sigma3 is the standard deviation of the subtracted frames and it is repeated for the second pair.

[attachment=9821:Table1c.gif]

When the sensor saturates, there is an abrupt decrease in noise since all pixels are driven to their maximum value. The standard deviations of the individual frames represents mainly pixel nonuniformity and this is removed when one looks at the difference image:

[attachment=9820:ImageJ_Histo.gif]

The individual image histogram has a double Gaussian appearance since the D3 uses multiple readout channels. I think that the histogram of the subtracted frames represents quantization error.

It would be interesting if you would perform a similar analysis on your data and let us know the results. From the appearance of your histograms, I assume that you are using demosaiced data from DCRaw. It might be better to use data directly from a raw channel if this is possible.

Bill
« Last Edit: November 22, 2008, 12:06:38 pm by bjanes »
Logged

Tony Beach

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 452
    • http://imageevent.com/tonybeach/twelveimages
DxO Mark
« Reply #37 on: November 22, 2008, 09:33:15 pm »

There is something seriously wrong with the conclusions this website offers.  I was reading a post at another photography forum that referred to the D300 ISO performance on this page:  http://www.dxomark.com/index.php/eng/DxOMark-Sensor and when I went there I noticed that the D90 ISO performance (which uses the same sensor) was rated about half a stop better.  I believe people relying on this sort of information in making a DSLR purchase are going to be seriously mislead as there is simply no way the D90 is a half stop better than the D300, and this makes me highly suspicious of all of the charts offered by this website.  To put it bluntly, this site as it currently stands is simply bogus.
Logged

Ray

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 10365
DxO Mark
« Reply #38 on: November 22, 2008, 09:49:56 pm »

Quote from: Tony Beach
... there is simply no way the D90 is a half stop better than the D300, and this makes me highly suspicious of all of the charts offered by this website.  To put it bluntly, this site as it currently stands is simply bogus.

Tony,
How do you know there is no way? What inside information do you have to support this conclusion. What test comparisons have you done?
Logged

Tony Beach

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 452
    • http://imageevent.com/tonybeach/twelveimages
DxO Mark
« Reply #39 on: November 22, 2008, 09:52:45 pm »

Quote from: Ray
Tony,
How do you know there is no way? What inside information do you have to support this conclusion. What test comparisons have you done?

A reputable reviewer has done the tests:  http://www.bythom.com/nikond90review.htm

I know who I trust.
Logged
Pages: 1 [2] 3 4   Go Up