Luminous Landscape Forum

Raw & Post Processing, Printing => Other Raw Converters => Topic started by: Iliah on July 16, 2013, 03:40:43 am

Title: RawDigger 0.9.18 is out, with sRAW analysis and TIFF export
Post by: Iliah on July 16, 2013, 03:40:43 am
Download (http://www.rawdigger.com/news/rawdigger-0-9-18)

Changes

New features

Data export (http://www.rawdigger.com/usermanual/data-export) as TIFF
Menu -> File -> Export TIFF)
This feature allows exporting RGB rendition, composite RAW, or single RAW channel as a TIFF file.
For more detailed discussion please see Data Export section in manual.
Better analyses of Canon sRAW data format

New settings:
Preferences -> Data Processing -> Show YCbCr data for Canon sRAW files
Preferences -> Data Processing -> Do not interpolate Cb/Cr channels data (Canon sRAW) The first setting allows to switch off conversion of YCbCr into RGB for Canon sRAW format, while the second switches off interpolation of Cb/Cr channels

New setting
Preferences -> Display Options -> Display RGB Render in RAW colors
This setting allows to switch off the conversion of RGB render into sRGB. This way the output remains in camera "color space".
For RAW data containing four different colors (RGBE, CMYG) this also switches off RGB render display. TIFF export for such RAW is still possible and results in a four-channel TIFF.

Added camera support for:

Baumer TXG14
Canon EOS C500
Nikon D5200
OmniVision OV5647 (Raspberry Pi)
Panasonic DMC-GF6
Samsung NX300, NX1100, NX2000
Sony NEX-3N

Minor changes:

CGATS output reworked to include additional data and for better standard conformance
Title: Re: RawDigger 0.9.18 is out, with sRAW analysis and TIFF export
Post by: Bart_van_der_Wolf on July 16, 2013, 04:29:35 am
Hi Iliah,

Thanks for this useful application that can help to analyze Raw data files, and gain a better understanding of what's going on under-the-hood.

If I may, there is one useful (optional) statistical metric that could be a useful addition for geeks.

When a shot is made of a featureless uniformly lit surface, the subtraction of the usually 2 Green filtered sensel responses per [R,G1,G2,B] set can provide useful info. If the subject and lighting is uniform, then the two green sensels should give the same output (except for read noise). If they don't, and it is measured over a larger area, a calibration bias may be detected (their average value is different). When their per channel standard deviation noise is subtracted in quadrature, read noise can be pretty accurately determined in a single shot. The subtraction will eliminate pattern noise because it is not random, and only shot-noise will remain. It can also detect outliers as hot or dead Green filtered sensels.

It could also be used to indirectly improve uniform lighting of a scene by minimizing the difference between the 2 green channel averages, but other channels could also be used for that.

Comparing spatial differences between sensels within a channel could also indicate a measure of focus-quality on highly detailed subjects or focus bracketing sequences.

One could also think about implementing a noise spectrum measurement, for the detection of on-sensor, or in-ADC pipeline, noise reduction.

Also estimated White-balance Raw channel multipliers would be useful.

Just some thoughts for when you run out of ideas ;)

Thanks again for making RawDigger available.

Cheers,
Bart
Title: Re: RawDigger 0.9.18 is out, with sRAW analysis and TIFF export
Post by: Iliah on July 16, 2013, 04:42:50 am
Dear Bart,

> If the subject and lighting is uniform, then the two green sensels should give the same output (except for read noise)

Let's consider this screenshot, Canon 1DMkIV, fairly uniform light. I'm under impression that there may be some other reasons for difference in green channels, not just different read noise. How to check?
Title: Re: RawDigger 0.9.18 is out, with sRAW analysis and TIFF export
Post by: Bart_van_der_Wolf on July 16, 2013, 05:43:48 am
Dear Bart,

> If the subject and lighting is uniform, then the two green sensels should give the same output (except for read noise)

Let's consider this screenshot, Canon 1DMkIV, fairly uniform light. I'm under impression that there may be some other reasons for difference in green channels, not just different read noise. How to check?

Hi Iliah,

The G1 and G2 averages are pretty close, so assuming the lighting was uniform (and there was no surface structure !), the directly neighboring sensels should subtract to zero (or with a small constant bias), except for 'noise'. So the approx. read+photon(!) noise is sqrt(20.0^2 +19.5^2) /sqrt(2) = 19.75. Since we can know the photon noise level if we know the gain that was used to produce this average level, we'd need to subtract the shot noise to be left with only the read-noise. I'm confident you guys can figure that out. Otherwise we'd just get random noise (without pattern noise which subtracts to an average of zero) almost as accurate as subtracting different exposure pairs, and that also allows to quantify the PRNU noise (for the channels involved).

I've made a simple ImageJ script available (via this thread (http://www.luminous-landscape.com/forum/index.php?topic=76446.msg614665#msg614665)) that uses a Raw data dump from DCraw to separate the channel data into separate files, and optionally subtract the G1 and G2 channels. That can IMO produce very interesting information.

Of course the spatial displacement of the 4 channels should be taken into account when image detail is involved, but for G1, G2 analysis, they are spatially very close (1.4x sensel-pitch diagonally and 2x sensel pitch orthogonally, compared 2.8x and 2 times for R and B) so especially for uniform (defocused) areas it's basically the quantified electronic differences that we measure (with a minimum of risk for e.g. amp.glow influences and vignetting or light fall-off). Between different color channels there are of course also the White-balance differences to take note of.

Hope that helps.

Cheers,
Bart
Title: Re: RawDigger 0.9.18 is out, with sRAW analysis and TIFF export
Post by: 32BT on July 17, 2013, 02:21:16 am
Okay, how about this:

1. Suppose dG = G1-G2

2. Compute sqrt(avg(dG^2))

This should likely be a good estimator for total noise. Note, you don't want to take avg(dG) which just equals avg(G1)-avg(G2) because that would always return ~zero given enough pixels.

Then to separate shot noise and read noise:
Take several measurements for different iso, create a curve. The curve should reduce as iso reduces, not to zero, but to the read-noise value.
Title: Re: RawDigger 0.9.18 is out, with sRAW analysis and TIFF export
Post by: Iliah on July 17, 2013, 02:44:37 am
Can you verify that avg(g1) - avg(g2) is zero?
Title: Re: RawDigger 0.9.18 is out, with sRAW analysis and TIFF export
Post by: Bart_van_der_Wolf on July 17, 2013, 03:00:15 am
Okay, how about this:

1. Suppose dG = G1-G2

2. Compute sqrt(avg(dG^2))

This should likely be a good estimator for total noise. Note, you don't want to take avg(dG) which just equals avg(G1)-avg(G2) because that would always return ~zero given enough pixels.

Then to separate shot noise and read noise:
Take several measurements for different iso, create a curve. The curve should reduce as iso reduces, not to zero, but to the read-noise value.

Hi Oscar,

Yes, that's the basic idea. It also allows to separate random (shot+read) noise from pattern noise (PRNU) when we subtract it from total noise. So one measurement on a single file, can give us (with a bit of extra work and a few files) the main components that make up the total noise. Could provide a lot of insight.

Basically it could also be used for the Red and Blue channels of a common Bayer CFA arrangement, but those filtered sensels are spatially a bit more distant diagonally, so they may be a bit more sensitive to picking up uneven lighting and/or vignetting and light fall-off and e.g. sensel aperture mask shading effects near corners. However, RawDigger has to also consider alternative filter arrangement, which complicates the programming effort, I realize that.

Of course it is possible to also calibrate for a gradual slope in the sampled area and calibrate that out, so the actual noise measurements gain even a bit more accuracy if the sample is not taken from the image center, and it could additionally produce a Lens Cast Calibration (LCC) file, although that might be a bit out of the scope of what RawDigger is envisioned to do.

Interesting stuff.

Cheers,
Bart
Title: Re: RawDigger 0.9.18 is out, with sRAW analysis and TIFF export
Post by: alextutubalin on July 17, 2013, 03:03:02 am
Can you verify that avg(g1) - avg(g2) is zero?

I've downloaded sample Sony A77 image from imaging-resource site. ISO200, several resolution targets and two Color Checkers on gray field.

The avg(g1) - avg(g2) is definitely not zero. For  250x250 selection: G1avg = 1616, G2avg = 1626. Another selection: 1576/1586 (both selections are made on gray area)

Also, the difference is different :) on different colors:
 A2 (orange) patch on color checker image (same camera, same sample image): 1372/1367
 A3 (deep blue) patch: 714/727 ( 2% difference!)
 B3 (green) patch: 1525/1526 (practically same on green!)
 F1 (blue) patch: 3019/3038

So, it looks like two greens are slightly different in color response too
 
Title: Re: RawDigger 0.9.18 is out, with sRAW analysis and TIFF export
Post by: 32BT on July 17, 2013, 03:14:38 am
I've downloaded sample Sony A77 image from imaging-resource site. ISO200, several resolution targets and two Color Checkers on gray field.

The avg(g1) - avg(g2) is definitely not zero. For  250x250 selection: G1avg = 1616, G2avg = 1626. Another selection: 1576/1586.

Also, the difference is different :) on different colors:
 A2 (orange) patch on color checker image (same camera, same sample image): 1372/1367
 A3 (deep blue) patch: 714/727 ( 2% difference!)
 B3 (green) patch: 1525/1526 (practically same on green!)
 F1 (blue) patch: 3019/3038

So, it looks like two greens are slightly different in color response too
 

There are probably differences in the two green channels because of different spill-overs, different read-out channels, and simple statistics. Please compute the confidence interval of the difference. Like I said the difference is approximately zero given enough pixels. It is not exactly zero. But I believe it is not a statistically relevant parameter. Taking the absolute value of the difference is far more relevant and interesting.




Title: Re: RawDigger 0.9.18 is out, with sRAW analysis and TIFF export
Post by: 32BT on July 17, 2013, 03:16:30 am
I've downloaded sample Sony A77 image from imaging-resource site. ISO200, several resolution targets and two Color Checkers on gray field.


Sony did have that funky sensor once with different colors for G1 and G2. Is the A77 one of those?
Title: Re: RawDigger 0.9.18 is out, with sRAW analysis and TIFF export
Post by: Iliah on July 17, 2013, 03:23:15 am
Sony did have that funky sensor once with different colors for G1 and G2. Is the A77 one of those?

No, it is not that extreme. But generally in my testing those 2 green channels can't be treated equally.
Title: Re: RawDigger 0.9.18 is out, with sRAW analysis and TIFF export
Post by: alextutubalin on July 17, 2013, 03:27:34 am
Sony did have that funky sensor once with different colors for G1 and G2. Is the A77 one of those?

The 'funky' RGBE (Emerald) sensor was in Sony F828 camera.

In A77 the G1/G2 are very close, but there is systematic difference:
 on blue color checker patches G1 is less than G2 (up to 1.5-2% difference)
 on green patches G1 and G2 are near the same
 on yellow (and red) patches G2 is less than G1

Title: Re: RawDigger 0.9.18 is out, with sRAW analysis and TIFF export
Post by: alextutubalin on July 17, 2013, 03:57:46 am
There are probably differences in the two green channels because of different spill-overs, different read-out channels, and simple statistics. Please compute the confidence interval of the difference.

(Standard) Deviation on relative small samples (100x100 or 250x250) is due to photon noise (for 1000 electrons generated and 1e/1ADU amplification deviation will be about 32).

So, statistically speaking, you need to shot many samples, then average them (subtract dark averaged dark frame and do other things astrophotographers do with shots).

Anyway, for Sony A77 (and for Leica M9, just tested it) the G1/G2 difference is systematic. It is very easy to see, just take color checker shot (or download ready image from Imaging resource), run RawDigger, use Grid Tool to capture Color Checker data, export it to Excel CSV file and explore.
Title: Re: RawDigger 0.9.18 is out, with sRAW analysis and TIFF export
Post by: Bart_van_der_Wolf on July 17, 2013, 04:15:48 am
So, it looks like two greens are slightly different in color response too.

Hi Alex,

That's not uncommon. I have a similar discrepancy with my 1Ds3, its G1 sensels on average give an up to 1.605 ADU higher response than the G2 sensels, also in the image center. That is already an interesting observation, because that means that Raw converters can consider the G1 and G2 samples as separate channels or as a single channel, with different effects. For example DCRaw and RawTherapee allow the user to choose how the green sensels are to be used, as the same or as different channels. It can e.g. cause luminance noise in smooth (e.g. sky) gradients which is due to relatively poor G1/G2 calibration, so it is a useful metric by itself.

There can also be other biases introduced by e.g. vignetting or sensel mask shading and tunneling effect when we get away from the center of the image, but for modest sized areas it will not impact the noise analysis too much.

However, the averages over a reasonable sample area size are usually not miles apart, and that makes the noise analysis based around their average still very useful and accurate enough. A given standard deviation based on a mean difference of 2 produces pretty much the same amplitudes.

Cheers,
Bart
Title: Re: RawDigger 0.9.18 is out, with sRAW analysis and TIFF export
Post by: Bart_van_der_Wolf on July 17, 2013, 04:22:55 am
Can you verify that avg(g1) - avg(g2) is zero?

Hi Iliah,

Ideally, for noise analysis, one would like to subtract two Raw files in their entirety, just like astronomers do to calibrate for noise per sensel, and to eliminate the influence of PRNU. That would make sure that only random noise and exposure (shutter time and sticky aperture blade) inaccuracies could be present at the sensel level. Unfortunately, we do not always have exposure pairs available for analysis.

So, as a second best option we could consider the G1/G2 statistics, but as always we need to understand what we are looking at.

Cheers,
Bart
Title: Re: RawDigger 0.9.18 is out, with sRAW analysis and TIFF export
Post by: Iliah on July 17, 2013, 04:24:55 am
Dear Bart,

I would suggest to try to determine how the exposure affects the G1/G2 ratio, say, we have a reasonably stable light source, like a halogen bulb, and a long lens covered with a styrofoam cup. Let's say aperture is fixed 2-3 stops from wide open, and only the shutter speed varies. If folks here are interested to shoot those and process we all will have some numbers to evaluate.
Title: Re: RawDigger 0.9.18 is out, with sRAW analysis and TIFF export
Post by: 32BT on July 17, 2013, 05:22:05 am
Variance is a bitch, and reducing variance takes exponential effort. One possible way to see how safe your estimate really is: plot the average difference in relation to count. It should vary wildly initially, and then vary less and less as the count increases. You would get a useful view of where variance becomes irrelevant for your purposes, and where you can relatively confidently say what the average will be.

Another option would be to simply take the minimum difference of the 4 direct neighbours of any given pixel within a single CFA plane. This is additionally useful if you later want to use differences for edge-detection. An edge occurs on the maximum of those 4 differences, so that maximum should at least exceed the average Dmin in order to be considered relevant.

Another thought: while the DCRaw code is an accomplishment in itself, the coding is excessively horrible. It might very well introduce integer errors where we least expect them. For example, I see both of the following being used and they can yield different results for signed integers:

(a+b)>>1
(a+b) / 2
Title: Re: RawDigger 0.9.18 is out, with sRAW analysis and TIFF export
Post by: alextutubalin on July 17, 2013, 05:46:14 am
Variance is a bitch, and reducing variance takes exponential effort. One possible way to see how safe your estimate really is: plot the average difference in relation to count. It should vary wildly initially, and then vary less and less as the count increases. You would get a useful view of where variance becomes irrelevant for your purposes, and where you can relatively confidently say what the average will be.
I think, it is better (and more scientific) to use t-statistics criterion. According to it, for 10k pixels (200x200 bayer sample) with average value 1000 and same standard deviation 33 (photon noise), 1-level difference is significant for 0.9 level of confidence and 2-level difference is enough for 0.99 confidence.

For example, I see both of the following being used and they can yield different results for signed integers:

(a+b)>>1
(a+b) / 2


In RAW data all values are non-negative and relatively small (10-16 bit), way smaller than largest possible signed 32-bit integer.
With this restrictions both averaging methods will produce same results.

Title: Re: RawDigger 0.9.18 is out, with sRAW analysis and TIFF export
Post by: Iliah on July 17, 2013, 06:14:44 am
Variance is a bitch

Dear Oscar,

This discussion may be a bit theoretical without experiments and measurements to support or reject theories. Experiments for this particular problem are not very difficult but very educative. I think it would be really good to study real data.
Title: Re: RawDigger 0.9.18 is out, with sRAW analysis and TIFF export
Post by: 32BT on July 17, 2013, 06:22:30 am
I think, it is better (and more scientific) to use t-statistics criterion. According to it, for 10k pixels (200x200 bayer sample) with average value 1000 and same standard deviation 33 (photon noise), 1-level difference is significant for 0.9 level of confidence and 2-level difference is enough for 0.99 confidence.

Yes, with 2 caveats:
1. you have to take the stddev of the actual samples (not a theoretical value from photon noise)
2. this criterion is meant for truly uncorrelated data, I don't know whether it can be used indiscriminately on correlated data which is what you are trying to establish: i.e. you are basically trying to answer the question "is there a determinate deviation between the green channels?" with some confidence level.

I wonder whether it is worth the effort since we are not so much interested in that difference as we are interested in the actual noise, possibly measured from the separate CFA planes which also eliminates any cross-channel correlation.

(Just wondering out loud, I am in no way dismissing the effort and usefulness of RAWDigger and all of its statistics. Besides, my statistics knowledge is too rusty to be trusted anyway.)




In RAW data all values are positive and relatively small (10-16 bit), way smaller than largest possible signed 32-bit integer. With this restrictions both averaging methods will produce same results.

Yes, except that the lossless RAW compression of some manufacturers represent difference signals, which obviously represent negative values. This then eliminates one of the restrictions mentioned. If DCRaw is doing the conversion to RGB and doing so incorrectly… (Not to mention incorrect rounding on the in-camera encoding chip to name just one of many other pitfalls).

The point i suppose is this: There is a "certain" amount of "uncertainty" involved in the numbers you use for statistics. If what you are trying measure falls too far below that certainty you should consider whether what you're trying to measure is worth the effort. You may draw incorrect conclusions because there might not be enough precision available in your data. The precision problem is not your fault. Drawing the wrong conclusions however...


 
Title: Re: RawDigger 0.9.18 is out, with sRAW analysis and TIFF export
Post by: Iliah on July 17, 2013, 06:28:15 am
> I am in no way dismissing the effort and usefulness of RAWDigger and all of its statistics

But we do not take the discussion as heading there.

> there might not be enough precision available in your data

Just in case, RawDigger does not use dcraw processing.
Title: Re: RawDigger 0.9.18 is out, with sRAW analysis and TIFF export
Post by: alextutubalin on July 17, 2013, 06:35:09 am
2. this criterion is meant for truly uncorrelated data, I don't know whether it can be used indiscriminately on correlated data which is what you are trying to establish: i.e. you are basically trying to answer the question "is there a determinate deviation between the green channels?" with some confidence level.

Not true. There is modification of t-statistics criterion for depended (correlated) samples. Of course, I've used this modification for my confidence level calculations.

Also, for low ISO standard deviation is *lower* than square root from signal. It slightly above this level at 1e/1ADU ISO (at ISO ~800 for my Canon  6D for example)

Yes, except that the lossless RAW compression of some manufacturers represent difference signals, which obviously represent negative values. This then eliminates one of the restrictions mentioned.
There are two color-difference formats: Canon sRAW and Kodak YCC. Both formats do not record two green channels, so irrelevant to our discussion.

If DCRaw is doing the conversion to RGB and doing so incorrectly…
RawDigger do not use dcraw's RGB conversion for raw data extraction.
All differences between channels (on Sony A77 in my example) are for RAW data, not for RGB.

Anyway, it is worth it to do these numerical experiments. There are many good samples on the net (imaging-resource.com is a good source) and RawDigger provides fast and easy tools to measure, for example, Color Checker shots.
Title: Re: RawDigger 0.9.18 is out, with sRAW analysis and TIFF export
Post by: 32BT on July 17, 2013, 06:44:41 am
Okay, got it.

But as far as I understand the Canon CR2 lossless compression encodes rows as a DC value with differences and DCRaw is reversing that encoding to the Bayer RGB samples. I do understand that RAWDigger isn't actually using the DCRaw demosaicing, which is not what I was referring at.

Title: Re: RawDigger 0.9.18 is out, with sRAW analysis and TIFF export
Post by: 32BT on July 17, 2013, 06:46:15 am
Okay, got it.

But as far as I understand the Canon CR2 lossless compression encodes rows as a DC value with differences and DCRaw is reversing that encoding to the Bayer RGB samples. I do understand that RAWDigger isn't actually using the DCRaw demosaicing, which is not what I was referring at.


In fact, doesn't DNG allow/use the same kind of compression?
Title: Re: RawDigger 0.9.18 is out, with sRAW analysis and TIFF export
Post by: Bart_van_der_Wolf on July 17, 2013, 06:47:50 am
Dear Bart,

I would suggest to try to determine how the exposure affects the G1/G2 ratio, say, we have a reasonably stable light source, like a halogen bulb, and a long lens covered with a styrofoam cup. Let's say aperture is fixed 2-3 stops from wide open, and only the shutter speed varies. If folks here are interested to shoot those and process we all will have some numbers to evaluate.

Hi Iliah,

Here are the results for an ISO 100 exposure pair series , on my Canon 1Ds3 with an EF 100mm f/2.8 lens used at f/5.6. I have excluded the frames for which the right tail started clipping from the linear regression calculations.

The correlation between the G1/G2 channel averages is almost perfect (CorrCoeff=1), which leads me to the conclusion that the differences are an ADC calibration effect.

Cheers,
Bart
Title: Re: RawDigger 0.9.18 is out, with sRAW analysis and TIFF export
Post by: alextutubalin on July 17, 2013, 06:51:54 am
Okay, got it.
But as far as I understand the Canon CR2 lossless compression encodes rows as a DC value with differences and DCRaw is reversing that encoding to the Bayer RGB samples. I do understand that RAWDigger isn't actually using the DCRaw demosaicing, which is not what I was referring at.

Canon CR2 format uses usual 'lossless JPEG' format. This is just Huffman encoding, nothing special. No integer rounding, just bit manipulation.

Several different decoders (dcraw, RawSpeed, Adobe's DNG converter) produces bitwise-the-same results on this data. So, I do not suspect any errors in this part of dcraw.

Title: Re: RawDigger 0.9.18 is out, with sRAW analysis and TIFF export
Post by: alextutubalin on July 17, 2013, 06:54:33 am
The correlation between the G1/G2 channel averages is almost perfect (CorrCoeff=1), which leads me to the conclusion that the differences are an ADC calibration effect.

Have not checked 1Ds3, but for 1DX colorchecker shots (from Imaging Resource) there are very interesting correlations between patch color and G1/G2 ratio.
Title: Re: RawDigger 0.9.18 is out, with sRAW analysis and TIFF export
Post by: Iliah on July 17, 2013, 07:03:27 am
Dear Bart,

My current theory is that if the colour of light and light intensity is fairly stable and the analyzed target is sufficiently uniform (same colour and reflection coefficient) analysis of G1 vs. G2 makes good sense.
Title: Re: RawDigger 0.9.18 is out, with sRAW analysis and TIFF export
Post by: Bart_van_der_Wolf on July 17, 2013, 07:15:49 am
Dear Bart,

My current theory is that if the colour of light and light intensity is fairly stable and the analyzed target is sufficiently uniform (same colour and reflection coefficient) analysis of G1 vs. G2 makes good sense.

Yep, my thoughts exactly. Hence the suggestion to consider some sort of metric based on G1 vs. G2, for those cases where we only have a single file to analyse. It's of course better to be able and use subtracted exposure pairs for noise analysis, but we're not always that lucky.

Proper testing also requires taking care of the test conditions. It's also the best way to identify mistakes in the execution of the test procedure itself, before they can lead to wrong conclusions.

Isn't learning, and getting a better understanding of how-things-work, fun!

Cheers,
Bart
Title: Re: RawDigger 0.9.18 is out, with sRAW analysis and TIFF export
Post by: Iliah on July 17, 2013, 08:20:37 am
Dear Bart, Alex just pointed out to me that you have not subtracted black. Can you please look at the numbers with black subtracted?
Title: Re: RawDigger 0.9.18 is out, with sRAW analysis and TIFF export
Post by: Bart_van_der_Wolf on July 17, 2013, 10:48:02 am
Dear Bart, Alex just pointed out to me that you have not subtracted black. Can you please look at the numbers with black subtracted?

Hi Iliah, Alex,

I've checked, and as expected, it just subtracts slightly different levels around 1023-1026 (Auto Black-level according to RawDigger) from the  averages (see attachments). The correlation stays basically the same and only the regression intercept changes as the interpolated line shifts down.

I've updated the results as attached.

Cheers,
Bart
Title: Re: RawDigger 0.9.18 is out, with sRAW analysis and TIFF export
Post by: alextutubalin on July 17, 2013, 11:20:22 am
I've checked, and as expected, it just subtracts slightly different levels around 1023-1026 (Auto Black-level according to RawDigger) from the  averages (see attachments). The correlation stays basically the same and only the regression intercept changes as the interpolated line shifts down.

It looks like you need more points in shadows (0...1023).  AFAIK, EV0 (lightmeter calibration) value for 1Ds3 is about 1000, may be 2000. So, your sample set completely ignores range below average gray.
Title: Re: RawDigger 0.9.18 is out, with sRAW analysis and TIFF export
Post by: Bart_van_der_Wolf on July 17, 2013, 12:18:23 pm
It looks like you need more points in shadows (0...1023).  AFAIK, EV0 (lightmeter calibration) value for 1Ds3 is about 1000, may be 2000. So, your sample set completely ignores range below average gray.

Hi Alex,

I didn't have lower exposures available for that series, which was shot for shot-noise analysis (didn't want to get too much read-noise obscuring that series). 

When I use a black frame pair (1/8000s, no lens and viewfinder covered) shot at a different occasion, and let Raw digger Auto-determine and subtract the black points of 1025 and 1023 for one frame and 1024 and 1023 for the other frame, I get a G1=2.2 and G2=2.7 for the entire frame, and  G1=2.2 and G2=2.8 for the center crop only, for both frames the same averages.

An issue then becomes that the lack of numerical precision, with one decimal for read-out of the averages, is starting to play a role in how exact the ratio of the averages can be calculated (the actual values are needed instead of the summaries).

I'll dig for some more low exposure files, but it still looks rather (but not perfectly) linear sofar.

Cheers,
Bart
Title: Re: RawDigger 0.9.18 is out, with sRAW analysis and TIFF export
Post by: alextutubalin on July 17, 2013, 12:33:13 pm
When I use a black frame pair (1/8000s, no lens and viewfinder covered) shot at a different occasion, and let Raw digger Auto-determine and subtract the black points of 1025 and 1023 for one frame and 1024 and 1023 for the other frame, I get a G1=2.2 and G2=2.7 for the entire frame, and  G1=2.2 and G2=2.8 for the center crop only, for both frames the same averages.

Hi, Bart!

Your regression calculations produces two different equations: G2 = 0.99*G1+18 (for black subtracted) and =0.99+30 for zero black level (I've round values to two digits).  The 2.2/2.7 values in black frame do not fit this equation.

I guess that adding more data points in shadows will significantly change regression equation  :)

Title: Re: RawDigger 0.9.18 is out, with sRAW analysis and TIFF export
Post by: bjanes on July 17, 2013, 02:53:34 pm
Yep, my thoughts exactly. Hence the suggestion to consider some sort of metric based on G1 vs. G2, for those cases where we only have a single file to analyse. It's of course better to be able and use subtracted exposure pairs for noise analysis, but we're not always that lucky.

Proper testing also requires taking care of the test conditions. It's also the best way to identify mistakes in the execution of the test procedure itself, before they can lead to wrong conclusions.

Isn't learning, and getting a better understanding of how-things-work, fun!

Cheers,
Bart

One might think that one could subtract the G1 and G2 channels from a single exposure to eliminate PRNU and determine shot noise when the exposure is near clipping and read noise is not significant, and thus avoid the need for duplicate exposures required for the usual sensor analysis as described by Roger Clark.

I used the new version of Rawdigger to extract the G1 and G2 channels from a D800e NEF flat field shot exposed just short of saturation (green pixel values ~13,085 and saved the resulting TIFFs. The TIFFs were compressed and would not open in ImagesPlus, which needs uncompressed TIFFs. The files opened in ImageJ, but the appearance of the field was quite non-uniform suggesting that ImageJ does not handle compressed TIFFs properly. To obtain uncompressed TIFFs, I opened the files in PhotoshopCC and saved in non-compressed format and then cropped to the 200 x 200 pixels in the central image for analysis.

Subtracting the G1 and G2 channels in ImagesPlus after adding an offset to one file to avoid negative numbers gave a standard deviation of 102.5 for two files and the SD for one file was 102.5/sqrt(2) or 72.5. Using the split_cfa function of Iris gave identical results.

Subtracting the green channels from duplicate images using the same exposure using ImagesPlus and Roger's method gave a standard deviation of 85.5 for two images and 60.4 for one image. This is significantly different from the results obtained by subtracting the channels from the Rawdigger single image, and I conclude that one should use the standard method as outlined by Roger for sensor analysis. I do not know the reasons for this difference.

Regards,

Bill



Title: Re: RawDigger 0.9.18 is out, with sRAW analysis and TIFF export
Post by: Bart_van_der_Wolf on July 17, 2013, 06:15:59 pm
One might think that one could subtract the G1 and G2 channels from a single exposure to eliminate PRNU and determine shot noise when the exposure is near clipping and read noise is not significant, and thus avoid the need for duplicate exposures required for the usual sensor analysis as described by Roger Clark.

Hi Bill,

That's one of the possible uses, when one is limited to a single file.

Quote
I used the new version of Rawdigger to extract the G1 and G2 channels from a D800e NEF flat field shot exposed just short of saturation (green pixel values ~13,085 and saved the resulting TIFFs. The TIFFs were compressed and would not open in ImagesPlus, which needs uncompressed TIFFs. The files opened in ImageJ, but the appearance of the field was quite non-uniform suggesting that ImageJ does not handle compressed TIFFs properly. To obtain uncompressed TIFFs, I opened the files in PhotoshopCC and saved in non-compressed format and then cropped to the 200 x 200 pixels in the central image for analysis.

Many potential sources for the introduction of an unexpected result. Did you check if the saved result from PhotoshopCC still corresponds with the RawDigger mouse pointer values for those same coordinates? Photoshop is not what I would trust blindly (15-bit and such).

When I export a Raw file data dump (without the masked pixels) with DCRaw (DCRaw -D -4 -T filename), I get the exact same readouts as Rawdigger shows, when opening that TIFF output in ImageJ. I can then use the macro I made for ImageJ to separate the G1 and G2 channels and also produce the G1-G2 result if no prior math or cropping is required. The G1-G2 difference file is in 32-bit floating point, so even no offset provision for avoiding negative integers is required.

I'll do a comparison on my Raw files tomorrow, between an exposure pair and the 2 single file sub-channels.

Cheers,
Bart
Title: Re: RawDigger 0.9.18 is out, with sRAW analysis and TIFF export
Post by: bjanes on July 17, 2013, 08:55:51 pm
Hi Bill,

That's one of the possible uses, when one is limited to a single file.

Many potential sources for the introduction of an unexpected result. Did you check if the saved result from PhotoshopCC still corresponds with the RawDigger mouse pointer values for those same coordinates? Photoshop is not what I would trust blindly (15-bit and such).

When I export a Raw file data dump (without the masked pixels) with DCRaw (DCRaw -D -4 -T filename), I get the exact same readouts as Rawdigger shows, when opening that TIFF output in ImageJ. I can then use the macro I made for ImageJ to separate the G1 and G2 channels and also produce the G1-G2 result if no prior math or cropping is required. The G1-G2 difference file is in 32-bit floating point, so even no offset provision for avoiding negative integers is required.

I'll do a comparison on my Raw files tomorrow, between an exposure pair and the 2 single file sub-channels.

Cheers,
Bart

Bart,

Another way to get the Rawdigger TIFFs into ImagesPlus is to open the TIFFs in ImageJ and save as FITS, which ImagesPlus can use. My previoius anomalous results with ImageJ were due to improper parameters in Rawdigger. One needs to use the half-size option in the Bayer processing options. I get the same results using the ImageJ route as with the PhotoshopCC route. I will await your results tomorrow.

Bill
Title: Re: RawDigger 0.9.18 is out, with sRAW analysis and TIFF export
Post by: bjanes on July 17, 2013, 11:03:51 pm
Bart,

Another way to get the Rawdigger TIFFs into ImagesPlus is to open the TIFFs in ImageJ and save as FITS, which ImagesPlus can use. My previoius anomalous results with ImageJ were due to improper parameters in Rawdigger. One needs to use the half-size option in the Bayer processing options. I get the same results using the ImageJ route as with the PhotoshopCC route. I will await your results tomorrow.

Bill

Bart,

I downloaded your ImageJ macro and used it on my NEF with the DCRaw -D -4 -T arguments and it worked like a charm and gave me the split out raw channels. Subtracting the uncropped channels in ImageJ gave nearly identical results as from the previous crops processed through PhotoshopCC.

Still awaiting your tests tomorrow.

Bill
Title: Re: RawDigger 0.9.18 is out, with sRAW analysis and TIFF export
Post by: Bart_van_der_Wolf on July 18, 2013, 02:17:00 pm
I'll do a comparison on my Raw files tomorrow, between an exposure pair and the 2 single file sub-channels.

I have extracted a center crop (200x200px) of the G1 channel from 2 files from my Canon EOS-1Ds3, extracted via DCRaw as undemosaiced Raw data numbers and split with ImageJ into separate channel files. It was verified that the channel files represent the same data numbers as RawDigger gives for the same sensel positions. Then the same Auto Blackpoints have been subtracted as RawDigger does.
ID=142_G1_BPC1023, Min=12601, Max=13411, Mean=13005.046, StdDev=93.598
ID=143_G1_BPC1023, Min=12757, Max=13539, Mean=13143.038, StdDev=95.265

Mathematically, if they are truly Random and Independent samples from a Normal distribution, one could expect a more accurate average noise estimate to be Sqrt((93.598^2+95.265^2))/Sqrt(2) = 94.435 for these two samples from the same noisy distribution.

However, when we actually subtract the channels of 2 files and take the statistics from that result we get:
Min=-492, Max=212, Mean=-137.991, StdDev=80.857 . That lower StdDev suggests that there is a non-random component in the data which reduces the Standard Deviation. To get the actual noise level we still need to divide by Sqrt(2).

-----------------------------------------------
Repeating that for the G2 channel from 2 files:
ID=142_G2_BPC1024, Min=12470, Max=13212, Mean=12864.084, StdDev=92.014
ID=143_G2_BPC1024, Min=12607, Max=13392, Mean=12999.766, StdDev=92.694

Again, mathematically, if they are truely Random and Independent samples from a Normal distribution, one could expect a
more accurate average noise estimate to be Sqrt((92.014^2+92.694^2))/Sqrt(2) = 92.355 .

However, when we actually subtract the channels of 2 files and take the statistics from that result we get:
Min=-527, Max=242, Mean=-135.682, StdDev=79.093 . That also suggests that there is a non-random component in the data which reduces the Standard Deviation. To get the actual noise level we still need to divide by Sqrt(2).

-----------------------------------------------
Now repeating that for the subtraction of the G1 and G2 channels in a single file, which may be not truely independent and may have some PRNU added to the Normal distribution noise:
ID=142_G1-G2_BPC, Min=-378, Max=638, Mean=140.963, AvgStdDev=123.317, CorrectedStdDev=87.198
and
ID=143_G1-G2_BPC, Min=-407, Max=652, Mean=143.272, AvgStdDev=124.258, CorrectedStdDev=87.864

That shows a Standard deviation that is indeed closer to the proper subtraction of exactly the same sensel positions, but not quite as low. I've added the four 16-bit TIFF files of the Black-point adjusted center-crops for experimentation, and the histograms of the single file G1-G2 subtractions for information.

I'm not exactly sure yet how to correlate that to the sample positions offset or PRNU, maybe I need to try with more samples to test it in practice.

Cheers,
Bart
Title: Re: RawDigger 0.9.18 is out, with sRAW analysis and TIFF export
Post by: bjanes on July 19, 2013, 10:02:49 am

Now repeating that for the subtraction of the G1 and G2 channels in a single file, which may be not truely independent and may have some PRNU added to the Normal distribution noise:
ID=142_G1-G2_BPC, Min=-378, Max=638, Mean=140.963, AvgStdDev=123.317, CorrectedStdDev=87.198
and
ID=143_G1-G2_BPC, Min=-407, Max=652, Mean=143.272, AvgStdDev=124.258, CorrectedStdDev=87.864

That shows a Standard deviation that is indeed closer to the proper subtraction of exactly the same sensel positions, but not quite as low. I've added the four 16-bit TIFF files of the Black-point adjusted center-crops for experimentation, and the histograms of the single file G1-G2 subtractions for information.

I'm not exactly sure yet how to correlate that to the sample positions offset or PRNU, maybe I need to try with more samples to test it in practice.


Bart,

Thank you very much for your detailed investigations. I repeated my investigation using Rawdigger to separate the green channels from two images (Image 15 and Image 16) of flat fields taken under identical conditions and exposure with the Nikon D800e. For comparison, I also used a DCRaw conversion with -D -4 -T and split out the channels with your ImageJ macro. Finally, I used IP to split out the green channel of the two identical exposures (IP combines G1 and G2 into a single file). For subtraction of images to eliminate non-random noise, I usually use ImagesPlus (IP) since it simplifies the task. Since it does not accept compressed TIFF, I used ImageJ to open the TIFFs and save them as FITs for processing by IP. With IP, I cropped the files to the central 400 x 400 pixels.

For completeness, I also performed subtractions using ImageJ using non-cropped files, since I don't know how to crop by the numbers with ImageJ--it seems to require a manual selection. The results are nearly identical to those obtained with cropped images.

The results are summarized in this table. As you found, subtracting G2 from G1 of a single image gives a larger standard deviation than subtracting either the G1 or G2 from two identical exposures.

(http://bjanes.smugmug.com/Photography/Rawdigger-Export-TIFF/i-JnDCcBh/0/O/Summary.png)

Looking at the subtraction of the green channels of a single file, I see some banding which is confirmed with FFT analysis.

(http://bjanes.smugmug.com/Photography/Rawdigger-Export-TIFF/i-NhdhfRn/0/O/FFT_Subtract%20channels%20same%20file.png)

No such banding is noted when subtracting crops from two separate files.

(http://bjanes.smugmug.com/Photography/Rawdigger-Export-TIFF/i-mTQWG8g/0/O/FFT_SubtractGreenDuplicateFiles.png)

My investigations are consistent with your theory that PRNU is not eliminated when one subtracts two green channels from the same file. For sensor analysis, one should use two images obtained under identical conditions as standard practice. The ability of Rawdigger to separate out the channels does offer an additional tool for image analysis.

Regards,

Bill


Title: Re: RawDigger 0.9.18 is out, with sRAW analysis and TIFF export
Post by: Iliah on July 19, 2013, 10:19:07 am
Dear Bill,
Banding of this type is consistent with crosstalk.
Title: Re: RawDigger 0.9.18 is out, with sRAW analysis and TIFF export
Post by: Bart_van_der_Wolf on July 19, 2013, 02:19:41 pm
Bart,

Thank you very much for your detailed investigations. I repeated my investigation using Rawdigger to separate the green channels from two images (Image 15 and Image 16) of flat fields taken under identical conditions and exposure with the Nikon D800e.

Hi Bill,

Thanks for sharing your findings, it makes me feel not too lonely ;)

I haven't had the time to compare Rawdigger channel output with the others yet, but its TIF compression is complicating matters slightly.

Quote
For completeness, I also performed subtractions using ImageJ using non-cropped files, since I don't know how to crop by the numbers with ImageJ--it seems to require a manual selection. The results are nearly identical to those obtained with cropped images.

It's simple, once you know it ... For a central crop, just use "Image>Adjust>Canvas size... Position: Center" and set the crop dimensions. Since we're shrinking the canvas, the Zero Fill option is not important.

Quote
As you found, subtracting G2 from G1 of a single image gives a larger standard deviation than subtracting either the G1 or G2 from two identical exposures.

Yes, and that is why it's always more accurate to use exposure pairs, because then we sample with the same sensels, thus eliminating one variable.

Quote
Looking at the subtraction of the green channels of a single file, I see some banding which is confirmed with FFT analysis.
[...]
No such banding is noted when subtracting crops from two separate files.

Indeed, and that is not uncommon. In fact that also demonstrates the superiority of the 2-frame subtraction method. Even with 2 exposures most of the pattern noise is eliminated, only random noise remains. Even a Fourier transform doesn't pick-up the parallel readout / crosstalk signal anymore.

That's not to say that the G1-G2 subtraction has no merit. In fact, it can be used as a differencing tool to amplify some correlated signals with two offset windows, but it still does reduce the Standard deviation so some noise is reduced, and it isn't the truly Random signal.

Quote
My investigations are consistent with your theory that PRNU is not eliminated when one subtracts two green channels from the same file. For sensor analysis, one should use two images obtained under identical conditions as standard practice. The ability of Rawdigger to separate out the channels does offer an additional tool for image analysis.

Well, some PRNU may be reduced, but other signals may be amplified (e.g. hot/dead pixels). That's the remaining puzzle, to find out what it is best used for. As I recall, Emile Martinec also used the G1/G2 for some of his noise analysis, when lacking the benefit of an exposure pair. I'll have to look it up and read it again, with the benefit of the current insights.

Anyway, nothing beats the old tried, tested, and proven subtracted-exposure-pair technique, but there may be uses for the single image G2-G1 evaluation. It e.g. detects dead/hot pixels which therefore points out that an outlier will spoil the statistical analysis if allowed to play a role in the analysis of that area.

Cheers,
Bart
Title: Re: RawDigger 0.9.18 is out, with sRAW analysis and TIFF export
Post by: Iliah on July 19, 2013, 02:23:33 pm
Here is a portion of sky, very close to the centre of the frame, taken with a camera with a sensor with stripped CFA. have forgotten to mention, no AA filter too, just a very good and clean optical glass.