Pages: 1 [2] 3 4   Go Down

Author Topic: SPECTACULAR: 5D MKII's DR in DPreview  (Read 25595 times)

eronald

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 6642
    • My gallery on Instagram
SPECTACULAR: 5D MKII's DR in DPreview
« Reply #20 on: February 17, 2009, 11:13:10 am »

Quote from: GLuijk
NikosR, did you spend the same time as here in writing to the DPreview team to let them know they are providing non advanced users with misleading figures because they consider noise reduction as a DR enhancer? (which BTW is just another evidence of how surrealistic their analysis are).

Because I think that is by far much more important than the completeness and accuracy of this thread.

What else are the poor b******s supposed to do? NR can take many forms, viz the strange sensor in the Fuji cameras. You just take what comes out of the camera and rate it. What they could be criticized for is having a DR measurement method that can be fooled by NR. But we'll talk about that on the day somone can convince me that there is actually a meaningful way to *really* define DR.

Edmund

----------
Pay peanuts, get monkeys; be kind to strangers, get friends.
« Last Edit: February 17, 2009, 11:15:47 am by eronald »
Logged
If you appreciate my blog posts help me by following on https://instagram.com/edmundronald

NikosR

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 622
    • http://
SPECTACULAR: 5D MKII's DR in DPreview
« Reply #21 on: February 17, 2009, 11:19:01 am »

Quote from: GLuijk
NikosR, did you spend the same time as here in writing to the DPreview team to let them know they are providing non advanced users with misleading figures because they consider noise reduction as a DR enhancer? (which BTW is just another evidence of how surrealistic their analysis are).

Because I think that is by far much more important than the completeness and accuracy of this thread.


I can't be bothered with dpreview, but I'm bothered with this site. You opened a thread 'as a joke' as you said presenting some ridiculous findings on the part of dpreview. You could have taken the time to try to find a reason behind those findings. This would not have made the findings less ridiculous but, since you care about readers' education, would have educated a few.
« Last Edit: February 17, 2009, 11:21:57 am by NikosR »
Logged
Nikos

Guillermo Luijk

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2005
    • http://www.guillermoluijk.com
SPECTACULAR: 5D MKII's DR in DPreview
« Reply #22 on: February 17, 2009, 11:28:30 am »

Quote from: eronald
But we'll talk about that on the day somone can convince me that there is actually a meaningful way to *really* define DR.
There is one, and in fact is much easier to apply than DPreview's strange manoeuvres. Just calculate the RAW exposure with respect to saturation at which SNR falls below some threshold (I'd suggest 12dB). That would help a lot to obtain comparable figures among different cameras.
Of course there is still the issue of cameras applying noise reduction on RAW data, but we would be quite closer to a rigurous measurement than the DP team are now.

Quote from: NikosR
I can't be bothered with dpreview, but I'm bothered with this site. You opened a thread 'as a joke' as you said presenting some ridiculous findings on the part of dpreview. You could have taken the time to try to find a reason behind those findings. This would not have made the findings less ridiculous but, since you care about readers' education, would have educated a few.
I guess you didn't spend any time writing to the DPreview team.
« Last Edit: February 17, 2009, 11:33:36 am by GLuijk »
Logged

eronald

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 6642
    • My gallery on Instagram
SPECTACULAR: 5D MKII's DR in DPreview
« Reply #23 on: February 17, 2009, 12:09:43 pm »

Quote from: GLuijk
There is one, and in fact is much easier to apply than DPreview's strange manoeuvres. Just calculate the RAW exposure with respect to saturation at which SNR falls below some threshold (I'd suggest 12dB). That would help a lot to obtain comparable figures among different cameras.
Of course there is still the issue of cameras applying noise reduction on RAW data, but we would be quite closer to a rigurous measurement than the DP team are now.


How d'you measure *this* SNR with respect to saturation ?

Edmund
« Last Edit: February 17, 2009, 12:10:30 pm by eronald »
Logged
If you appreciate my blog posts help me by following on https://instagram.com/edmundronald

Panopeeper

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1805
SPECTACULAR: 5D MKII's DR in DPreview
« Reply #24 on: February 17, 2009, 03:06:27 pm »

Let's remove DPReview from the subject and concentrate on the underlying issues, namely the methodology of determining the dynamic range.

If a review is focusing on potentional "JPEG shooters" (as opposed to "raw shooters"), then it is logical to evaluate the camera's capability based on in-camera JPEGs. The reviewers decide the basic parameters: the acceptable noise level and the minimum intensity level. If the result of the evaluation is, that the camera throws away three stops of the DR @ ISO 100, then either

1. the methodology or the actual measurements are useless, or

2. the final verdict must be Not recommended.

However, if the review is aimed at users, who want to squeeze out the best of the camera, then the JPEG based evaluation is worthless.

Quote from: GLuijk
Just calculate the RAW exposure with respect to saturation at which SNR falls below some threshold (I'd suggest 12dB)
Why do you think that you or anyone else should determine the level of acceptable noise? Why would any specific value be declared as a "universal threshold"?

Someone may find a certain level of noise acceptable @ ISO 100 in landscapes shot in bright daylight; however, someone else may be shooting night club scenes @ ISO 1600. Why on earth would one apply the same criteria for all situations?
Logged
Gabor

NikosR

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 622
    • http://
SPECTACULAR: 5D MKII's DR in DPreview
« Reply #25 on: February 17, 2009, 03:45:05 pm »

Quote from: Panopeeper
Why do you think that you or anyone else should determine the level of acceptable noise? Why would any specific value be declared as a "universal threshold"?

Someone may find a certain level of noise acceptable @ ISO 100 in landscapes shot in bright daylight; however, someone else may be shooting night club scenes @ ISO 1600. Why on earth would one apply the same criteria for all situations?


Gabor,

I believe I understand and I quite agree with what you're saying here. Acceptable noise floor is both subjective and scene / photograph specific. Where does that leave us with respect to measuring DR, as a single reference noise floor must be established for any measurements to be of any comparative value? Maybe to Dxo Mark's definition of S/N=1?
« Last Edit: February 17, 2009, 03:45:23 pm by NikosR »
Logged
Nikos

Guillermo Luijk

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2005
    • http://www.guillermoluijk.com
SPECTACULAR: 5D MKII's DR in DPreview
« Reply #26 on: February 17, 2009, 03:46:54 pm »

Quote from: Panopeeper
Why do you think that you or anyone else should determine the level of acceptable noise? Why would any specific value be declared as a "universal threshold"?
Someone may find a certain level of noise acceptable @ ISO 100 in landscapes shot in bright daylight; however, someone else may be shooting night club scenes @ ISO 1600. Why on earth would one apply the same criteria for all situations?

It's simple Gabor, the goal of these analysis is not to determine the DR using a criteria according to any given situation (if so there should be thousands of criteria), but to get a figure that allows to compare DR on different cameras.

I chose 12dB since it seems to me a good trade-off to export the conclusions from that comparision, to real situations. But in order to stablish comparisions, any other criteria will be as good (DxO Mark use 0dB).

Personally I prefer much more to look at the complete SNR curve plots, which are the only ones that let you find situations like that reported by Ray: camera A with more DR than camera B for a given criteria, but camera B is less noisy than camera A in most of the range, which could happen to be the range of interest for my application.

BR
« Last Edit: February 17, 2009, 03:47:48 pm by GLuijk »
Logged

Daniel Browning

  • Full Member
  • ***
  • Offline Offline
  • Posts: 142
SPECTACULAR: 5D MKII's DR in DPreview
« Reply #27 on: February 17, 2009, 05:24:02 pm »

Quote from: GLuijk
Personally I prefer much more to look at the complete SNR curve plots

It's like the difference between an MTF plot vs. just one MTF-50 number.
Logged
--Daniel

Panopeeper

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1805
SPECTACULAR: 5D MKII's DR in DPreview
« Reply #28 on: February 17, 2009, 06:45:40 pm »

Quote from: NikosR
Where does that leave us with respect to measuring DR, as a single reference noise floor must be established for any measurements to be of any comparative value? Maybe to Dxo Mark's definition of S/N=1?
I don't see any reason to declare a single noise level as floor. GLuijk suggested SNR curve plots. I prefer a chart in Excel like format. The format is less interesting; the essence is, that I am producing data for many different noise levels and one can pick from the list any level and compare it to another camera of another ISO of the same camera.

I posted an example in the thread "1 Feb, 2008 - Quality vs. Value - When is Enough Enough?", post #85, and a crop of the results in #125. I make literally hundreds of measurements in an ISO serie of such shots:

EV   -> noise

9.28 -> 32.5
9.32 -> 35.1
9.41 -> 34.9
9.51 -> 41.0
9.59 -> 40.0
9.69 -> 43.2
9.75 -> 43.9
9.78 -> 44.7
9.89 -> 46.4
9.95 -> 51.7

The first number shows the intensity of a selected patch from saturation downwards, the second number is the degree of noise, as the standard deviation in percentage of the average pixel values in the selected patch. 51.7% is roughly SNR=2 at -9.95 EV. This means, that if you accept SNR=2, then the DR is 10 EV (this is the 40D at ISO 200). If you accept only SNR=3, i.e. 33% noise, the DR is somewhere between 9.3 and 9.4 EV.

I have done this for several cameras, and I will put them in a single Excel chart for easy comparison. Unfortunately, I don't have suitable raw files from the 5D2, nor from the D3X.
Logged
Gabor

marcmccalmont

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1780
SPECTACULAR: 5D MKII's DR in DPreview
« Reply #29 on: February 17, 2009, 07:24:08 pm »

Quote from: Panopeeper
I don't see any reason to declare a single noise level as floor. GLuijk suggested SNR curve plots. I prefer a chart in Excel like format. The format is less interesting; the essence is, that I am producing data for many different noise levels and one can pick from the list any level and compare it to another camera of another ISO of the same camera.

I posted an example in the thread "1 Feb, 2008 - Quality vs. Value - When is Enough Enough?", post #85, and a crop of the results in #125. I make literally hundreds of measurements in an ISO serie of such shots:

EV   -> noise

9.28 -> 32.5
9.32 -> 35.1
9.41 -> 34.9
9.51 -> 41.0
9.59 -> 40.0
9.69 -> 43.2
9.75 -> 43.9
9.78 -> 44.7
9.89 -> 46.4
9.95 -> 51.7

The first number shows the intensity of a selected patch from saturation downwards, the second number is the degree of noise, as the standard deviation in percentage of the average pixel values in the selected patch. 51.7% is roughly SNR=2 at -9.95 EV. This means, that if you accept SNR=2, then the DR is 10 EV (this is the 40D at ISO 200). If you accept only SNR=3, i.e. 33% noise, the DR is somewhere between 9.3 and 9.4 EV.

I have done this for several cameras, and I will put them in a single Excel chart for easy comparison. Unfortunately, I don't have suitable raw files from the 5D2, nor from the D3X.

I'm back home so I can supply 5DII RAWs for you please detail how you want them shot
Marc
Logged
Marc McCalmont

Panopeeper

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1805
SPECTACULAR: 5D MKII's DR in DPreview
« Reply #30 on: February 17, 2009, 08:00:26 pm »

Quote from: marcmccalmont
I'm back home so I can supply 5DII RAWs for you please detail how you want them shot
Marc
Hi Marc, I remember you wrote once that you were on a trip months long.

If you have a printer, then there is a good solution. The same issue came up with Eric (a fellow poster here). He has a color checker card, but that turned out not to be reliable (in terms of noisiness). Then he had an idea: printing a color checker card. The result is much better than the original (but it can not be used for color profiling).
You can download a color checker file from here: http://www.rags-int-inc.com/PhotoTechStuff...hTarget_Lab.tif
For the noise measurement the colors are irrelevant; the point is, that the different color squares offer many different intensity levels in all three channels.
Before printing it out, pls remove the layer with the calibration numbers.
Print it please on hard, not mat paper, which does not have any visible texture. Eric printed it on glossy paper, which he is using for proofing.

The illumination should be *very* even, and the shot underexposed by three stops at least. In order to avoid unnecessary uploading, pls make two or  three shots @ ISO 100 with different underexposure and upload them; I verify if they are all right. I would need them for all full-stop ISOs up to 3200; ISO 50 too only for demonstration, that it is a plain overexposed ISO 100 (there are some, who don't want to believe that).

Pls use a lens at an aperture, which does not create much vignetting.

Thanks
Logged
Gabor

Guillermo Luijk

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2005
    • http://www.guillermoluijk.com
SPECTACULAR: 5D MKII's DR in DPreview
« Reply #31 on: February 17, 2009, 09:09:14 pm »

2 questions Gabor:

- When you say -9.95EV, you mean 9.95EV from sensor's saturation point, or from the top end of the bitscale (2^n-1 being n the RAW bitdepth). IMO the first approach is more interesting, but you need to calculate that saturation point of course. Having the sat point informed for each camera and ISO is also very interesting.

- As Emil Martinec commented some day, being Nikon's RAWs pre-scaled in the R and B channels (pre-WB), the SNR and thus your measured stdev for a given EV will be worse in them than in the G channel. Are you taking this into account? In which channel are you measuring noise?

I would love to have that Excel datasheet, obtaining the plots can be very interesting. I was even thinking of some simple VBA app reading your Excel and allowing camera comparisions of any brand and/or ISO values.

BR
« Last Edit: February 17, 2009, 09:15:58 pm by GLuijk »
Logged

marcmccalmont

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1780
SPECTACULAR: 5D MKII's DR in DPreview
« Reply #32 on: February 17, 2009, 09:14:08 pm »

Quote from: Panopeeper
Hi Marc, I remember you wrote once that you were on a trip months long.

If you have a printer, then there is a good solution. The same issue came up with Eric (a fellow poster here). He has a color checker card, but that turned out not to be reliable (in terms of noisiness). Then he had an idea: printing a color checker card. The result is much better than the original (but it can not be used for color profiling).
You can download a color checker file from here: http://www.rags-int-inc.com/PhotoTechStuff...hTarget_Lab.tif
For the noise measurement the colors are irrelevant; the point is, that the different color squares offer many different intensity levels in all three channels.
Before printing it out, pls remove the layer with the calibration numbers.
Print it please on hard, not mat paper, which does not have any visible texture. Eric printed it on glossy paper, which he is using for proofing.

The illumination should be *very* even, and the shot underexposed by three stops at least. In order to avoid unnecessary uploading, pls make two or  three shots @ ISO 100 with different underexposure and upload them; I verify if they are all right. I would need them for all full-stop ISOs up to 3200; ISO 50 too only for demonstration, that it is a plain overexposed ISO 100 (there are some, who don't want to believe that).

Pls use a lens at an aperture, which does not create much vignetting.

Thanks

Give me a day or two
Marc
Logged
Marc McCalmont

Panopeeper

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1805
SPECTACULAR: 5D MKII's DR in DPreview
« Reply #33 on: February 17, 2009, 09:39:33 pm »

Quote from: GLuijk
- When you say -9.95EV, you mean 9.95EV from sensor's saturation point, or from the top end of the bitscale (2^n-1 being n the RAW bitdepth)
Of course I calculate with the saturation level. This questions is offensive :-)

Quote
you need to calculate that saturation point of course
Actually I do not calculate it; I read it from an overexposed shot. Although there are some variations between copies of a camera model, but that is negligable small.

Quote
being Nikon's RAWs pre-scaled in the R and B channels (pre-WB)
I don't know about this myth.

Quote
I would love to have that Excel datasheet, obtaining the plots can be very interesting
At the moment I have the results in "raw" format.  They have to be cleaned up; sometimes outlandish values occur. I uploaded now the 50D values for you, measured on a Stouffer transmission wedge: tab separated form

Sometimes I create a graph from them, but I find the numbers more useful. Attached the visualized charts of the 50D.
Logged
Gabor

Guillermo Luijk

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2005
    • http://www.guillermoluijk.com
SPECTACULAR: 5D MKII's DR in DPreview
« Reply #34 on: February 17, 2009, 09:58:04 pm »


Those data look great.

I will let Emil know about in case he wants to comment something on the Nikon pre-WB. Anyway if you have some Nikon test, it would be clarifying to recalculate separate noise datasets for each RAW channel. If all behave the same it can be concluded there was no scaling.


Panopeeper

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1805
SPECTACULAR: 5D MKII's DR in DPreview
« Reply #35 on: February 17, 2009, 10:33:05 pm »

Quote from: GLuijk
Anyway if you have some Nikon test, it would be clarifying to recalculate separate noise datasets for each RAW channel. If all behave the same it can be concluded there was no scaling.
There ARE differences, but not due to WB. Nikon published something like Canon's "white papers" are, re the D2X, and declared that the analog stage applies WB. However, this is not so. (I think I mentioned the D2H in some other thread, but that's incorrect.)
Logged
Gabor

ejmartin

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 575
SPECTACULAR: 5D MKII's DR in DPreview
« Reply #36 on: February 17, 2009, 10:50:08 pm »

Quote from: GLuijk
Those data look great.

I will let Emil know about in case he wants to comment something on the Nikon pre-WB. Anyway if you have some Nikon test, it would be clarifying to recalculate separate noise datasets for each RAW channel. If all behave the same it can be concluded there was no scaling.


Nikon does a fixed scaling of the R and B channels after quantization whose purpose remains a mystery to me, since it introduces additional sources of roundoff error and reduces DR in those channels.  It is easy to see from the RAW data, a histogram or list of populated RAW levels shows a regular pattern of gaps.  From the pattern of gaps one can infer the multiplier by taking the list of numbers (1 2 3 4 5 6 7 ...) and doing integer multiplication and division, rounding off at each step.  If the correct multiplier is chosen, then the output will be the list of populated levels.  It is easy to get in the right ballpark via the fraction of levels that are unpopulated.

The multiplier is also quite evident in the fact that the gain (e-/RAW level) differs for the three color channels, being smaller for R and B than it is for G.
Logged
emil

marcmccalmont

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1780
SPECTACULAR: 5D MKII's DR in DPreview
« Reply #37 on: February 17, 2009, 11:11:29 pm »

Quote from: marcmccalmont
Give me a day or two
Marc


In focus or slightly out of focus? and so that I get this correct let the camera meter the correct exposure then under expose by 3 stops.
Logged
Marc McCalmont

Panopeeper

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1805
SPECTACULAR: 5D MKII's DR in DPreview
« Reply #38 on: February 17, 2009, 11:37:05 pm »

Quote from: marcmccalmont
In focus or slightly out of focus?
This should not matter if your printer delivers a smooth, uniform color. Please do not touch the surface with bare fingers if you want to avoid being fingerprinted.

Quote
so that I get this correct let the camera meter the correct exposure then under expose by 3 stops
Honestly I have no idea, how auto metering would look like; that's the reason I suggested to make first at least two shows with different underexposure. The darkest patch has to be in the 12th stop; this requires huuuge underexposure. Give it a try with -3 and -4 EV and let's see.

You can pretest it in ACR: pick WB on the white patch, reset everything to 0 (blacks too!), curves linear, then the black patch should show R,G,B in the range 2 to 4.

ADDED

I am an airhead. If you make only one shot (let's say with -4 EV), then I can determine, how much more or less is necessary.
« Last Edit: February 17, 2009, 11:46:17 pm by Panopeeper »
Logged
Gabor

ErikKaffehr

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 11311
    • Echophoto
SPECTACULAR: 5D MKII's DR in DPreview
« Reply #39 on: February 18, 2009, 12:31:04 am »

Hi,

Just a comment on the CC card, I was using a mini color checker that accentuates the problem. Originally I just introduced it as a reference, being aware of a potential problem with granularity. Exchanging some e-mails with Gabor i felt I needed a smooth copy of the CC card so we came up with the idea of printing it. The first copy I had got dusty and it seems that it's also sensitive for fingerprints, so be careful with those prints.

For illumination I had a couple of halogen lights standing on a table at about 45 degrees angle on each side. The lamps were slightly below so illumination was somewhat uneven, but seemingly good enough. Using strobes would be probably better, but I have no strobes and exposure may have been harder to control.

Best regards
Erik


Quote from: marcmccalmont
Give me a day or two
Marc
Logged
Erik Kaffehr
 
Pages: 1 [2] 3 4   Go Up