Pages: 1 [2] 3   Go Down

Author Topic: RawDigger 0.9.18 is out, with sRAW analysis and TIFF export  (Read 30856 times)

Iliah

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 770
Re: RawDigger 0.9.18 is out, with sRAW analysis and TIFF export
« Reply #20 on: July 17, 2013, 06:28:15 am »

> I am in no way dismissing the effort and usefulness of RAWDigger and all of its statistics

But we do not take the discussion as heading there.

> there might not be enough precision available in your data

Just in case, RawDigger does not use dcraw processing.
Logged

alextutubalin

  • Newbie
  • *
  • Offline Offline
  • Posts: 14
Re: RawDigger 0.9.18 is out, with sRAW analysis and TIFF export
« Reply #21 on: July 17, 2013, 06:35:09 am »

2. this criterion is meant for truly uncorrelated data, I don't know whether it can be used indiscriminately on correlated data which is what you are trying to establish: i.e. you are basically trying to answer the question "is there a determinate deviation between the green channels?" with some confidence level.

Not true. There is modification of t-statistics criterion for depended (correlated) samples. Of course, I've used this modification for my confidence level calculations.

Also, for low ISO standard deviation is *lower* than square root from signal. It slightly above this level at 1e/1ADU ISO (at ISO ~800 for my Canon  6D for example)

Yes, except that the lossless RAW compression of some manufacturers represent difference signals, which obviously represent negative values. This then eliminates one of the restrictions mentioned.
There are two color-difference formats: Canon sRAW and Kodak YCC. Both formats do not record two green channels, so irrelevant to our discussion.

If DCRaw is doing the conversion to RGB and doing so incorrectly…
RawDigger do not use dcraw's RGB conversion for raw data extraction.
All differences between channels (on Sony A77 in my example) are for RAW data, not for RGB.

Anyway, it is worth it to do these numerical experiments. There are many good samples on the net (imaging-resource.com is a good source) and RawDigger provides fast and easy tools to measure, for example, Color Checker shots.
« Last Edit: July 17, 2013, 06:38:13 am by alextutubalin »
Logged
Alex Tutubalin
RawDigger:  http://www.rawdigger.com
LibRaw: http://www.libraw.org

32BT

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 3095
    • Pictures
Re: RawDigger 0.9.18 is out, with sRAW analysis and TIFF export
« Reply #22 on: July 17, 2013, 06:44:41 am »

Okay, got it.

But as far as I understand the Canon CR2 lossless compression encodes rows as a DC value with differences and DCRaw is reversing that encoding to the Bayer RGB samples. I do understand that RAWDigger isn't actually using the DCRaw demosaicing, which is not what I was referring at.

Logged
Regards,
~ O ~
If you can stomach it: pictures

32BT

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 3095
    • Pictures
Re: RawDigger 0.9.18 is out, with sRAW analysis and TIFF export
« Reply #23 on: July 17, 2013, 06:46:15 am »

Okay, got it.

But as far as I understand the Canon CR2 lossless compression encodes rows as a DC value with differences and DCRaw is reversing that encoding to the Bayer RGB samples. I do understand that RAWDigger isn't actually using the DCRaw demosaicing, which is not what I was referring at.


In fact, doesn't DNG allow/use the same kind of compression?
Logged
Regards,
~ O ~
If you can stomach it: pictures

Bart_van_der_Wolf

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 8913
Re: RawDigger 0.9.18 is out, with sRAW analysis and TIFF export
« Reply #24 on: July 17, 2013, 06:47:50 am »

Dear Bart,

I would suggest to try to determine how the exposure affects the G1/G2 ratio, say, we have a reasonably stable light source, like a halogen bulb, and a long lens covered with a styrofoam cup. Let's say aperture is fixed 2-3 stops from wide open, and only the shutter speed varies. If folks here are interested to shoot those and process we all will have some numbers to evaluate.

Hi Iliah,

Here are the results for an ISO 100 exposure pair series , on my Canon 1Ds3 with an EF 100mm f/2.8 lens used at f/5.6. I have excluded the frames for which the right tail started clipping from the linear regression calculations.

The correlation between the G1/G2 channel averages is almost perfect (CorrCoeff=1), which leads me to the conclusion that the differences are an ADC calibration effect.

Cheers,
Bart
Logged
== If you do what you did, you'll get what you got. ==

alextutubalin

  • Newbie
  • *
  • Offline Offline
  • Posts: 14
Re: RawDigger 0.9.18 is out, with sRAW analysis and TIFF export
« Reply #25 on: July 17, 2013, 06:51:54 am »

Okay, got it.
But as far as I understand the Canon CR2 lossless compression encodes rows as a DC value with differences and DCRaw is reversing that encoding to the Bayer RGB samples. I do understand that RAWDigger isn't actually using the DCRaw demosaicing, which is not what I was referring at.

Canon CR2 format uses usual 'lossless JPEG' format. This is just Huffman encoding, nothing special. No integer rounding, just bit manipulation.

Several different decoders (dcraw, RawSpeed, Adobe's DNG converter) produces bitwise-the-same results on this data. So, I do not suspect any errors in this part of dcraw.

Logged
Alex Tutubalin
RawDigger:  http://www.rawdigger.com
LibRaw: http://www.libraw.org

alextutubalin

  • Newbie
  • *
  • Offline Offline
  • Posts: 14
Re: RawDigger 0.9.18 is out, with sRAW analysis and TIFF export
« Reply #26 on: July 17, 2013, 06:54:33 am »

The correlation between the G1/G2 channel averages is almost perfect (CorrCoeff=1), which leads me to the conclusion that the differences are an ADC calibration effect.

Have not checked 1Ds3, but for 1DX colorchecker shots (from Imaging Resource) there are very interesting correlations between patch color and G1/G2 ratio.
Logged
Alex Tutubalin
RawDigger:  http://www.rawdigger.com
LibRaw: http://www.libraw.org

Iliah

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 770
Re: RawDigger 0.9.18 is out, with sRAW analysis and TIFF export
« Reply #27 on: July 17, 2013, 07:03:27 am »

Dear Bart,

My current theory is that if the colour of light and light intensity is fairly stable and the analyzed target is sufficiently uniform (same colour and reflection coefficient) analysis of G1 vs. G2 makes good sense.
Logged

Bart_van_der_Wolf

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 8913
Re: RawDigger 0.9.18 is out, with sRAW analysis and TIFF export
« Reply #28 on: July 17, 2013, 07:15:49 am »

Dear Bart,

My current theory is that if the colour of light and light intensity is fairly stable and the analyzed target is sufficiently uniform (same colour and reflection coefficient) analysis of G1 vs. G2 makes good sense.

Yep, my thoughts exactly. Hence the suggestion to consider some sort of metric based on G1 vs. G2, for those cases where we only have a single file to analyse. It's of course better to be able and use subtracted exposure pairs for noise analysis, but we're not always that lucky.

Proper testing also requires taking care of the test conditions. It's also the best way to identify mistakes in the execution of the test procedure itself, before they can lead to wrong conclusions.

Isn't learning, and getting a better understanding of how-things-work, fun!

Cheers,
Bart
Logged
== If you do what you did, you'll get what you got. ==

Iliah

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 770
Re: RawDigger 0.9.18 is out, with sRAW analysis and TIFF export
« Reply #29 on: July 17, 2013, 08:20:37 am »

Dear Bart, Alex just pointed out to me that you have not subtracted black. Can you please look at the numbers with black subtracted?
Logged

Bart_van_der_Wolf

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 8913
Re: RawDigger 0.9.18 is out, with sRAW analysis and TIFF export
« Reply #30 on: July 17, 2013, 10:48:02 am »

Dear Bart, Alex just pointed out to me that you have not subtracted black. Can you please look at the numbers with black subtracted?

Hi Iliah, Alex,

I've checked, and as expected, it just subtracts slightly different levels around 1023-1026 (Auto Black-level according to RawDigger) from the  averages (see attachments). The correlation stays basically the same and only the regression intercept changes as the interpolated line shifts down.

I've updated the results as attached.

Cheers,
Bart
Logged
== If you do what you did, you'll get what you got. ==

alextutubalin

  • Newbie
  • *
  • Offline Offline
  • Posts: 14
Re: RawDigger 0.9.18 is out, with sRAW analysis and TIFF export
« Reply #31 on: July 17, 2013, 11:20:22 am »

I've checked, and as expected, it just subtracts slightly different levels around 1023-1026 (Auto Black-level according to RawDigger) from the  averages (see attachments). The correlation stays basically the same and only the regression intercept changes as the interpolated line shifts down.

It looks like you need more points in shadows (0...1023).  AFAIK, EV0 (lightmeter calibration) value for 1Ds3 is about 1000, may be 2000. So, your sample set completely ignores range below average gray.
Logged
Alex Tutubalin
RawDigger:  http://www.rawdigger.com
LibRaw: http://www.libraw.org

Bart_van_der_Wolf

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 8913
Re: RawDigger 0.9.18 is out, with sRAW analysis and TIFF export
« Reply #32 on: July 17, 2013, 12:18:23 pm »

It looks like you need more points in shadows (0...1023).  AFAIK, EV0 (lightmeter calibration) value for 1Ds3 is about 1000, may be 2000. So, your sample set completely ignores range below average gray.

Hi Alex,

I didn't have lower exposures available for that series, which was shot for shot-noise analysis (didn't want to get too much read-noise obscuring that series). 

When I use a black frame pair (1/8000s, no lens and viewfinder covered) shot at a different occasion, and let Raw digger Auto-determine and subtract the black points of 1025 and 1023 for one frame and 1024 and 1023 for the other frame, I get a G1=2.2 and G2=2.7 for the entire frame, and  G1=2.2 and G2=2.8 for the center crop only, for both frames the same averages.

An issue then becomes that the lack of numerical precision, with one decimal for read-out of the averages, is starting to play a role in how exact the ratio of the averages can be calculated (the actual values are needed instead of the summaries).

I'll dig for some more low exposure files, but it still looks rather (but not perfectly) linear sofar.

Cheers,
Bart
Logged
== If you do what you did, you'll get what you got. ==

alextutubalin

  • Newbie
  • *
  • Offline Offline
  • Posts: 14
Re: RawDigger 0.9.18 is out, with sRAW analysis and TIFF export
« Reply #33 on: July 17, 2013, 12:33:13 pm »

When I use a black frame pair (1/8000s, no lens and viewfinder covered) shot at a different occasion, and let Raw digger Auto-determine and subtract the black points of 1025 and 1023 for one frame and 1024 and 1023 for the other frame, I get a G1=2.2 and G2=2.7 for the entire frame, and  G1=2.2 and G2=2.8 for the center crop only, for both frames the same averages.

Hi, Bart!

Your regression calculations produces two different equations: G2 = 0.99*G1+18 (for black subtracted) and =0.99+30 for zero black level (I've round values to two digits).  The 2.2/2.7 values in black frame do not fit this equation.

I guess that adding more data points in shadows will significantly change regression equation  :)

Logged
Alex Tutubalin
RawDigger:  http://www.rawdigger.com
LibRaw: http://www.libraw.org

bjanes

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 3387
Re: RawDigger 0.9.18 is out, with sRAW analysis and TIFF export
« Reply #34 on: July 17, 2013, 02:53:34 pm »

Yep, my thoughts exactly. Hence the suggestion to consider some sort of metric based on G1 vs. G2, for those cases where we only have a single file to analyse. It's of course better to be able and use subtracted exposure pairs for noise analysis, but we're not always that lucky.

Proper testing also requires taking care of the test conditions. It's also the best way to identify mistakes in the execution of the test procedure itself, before they can lead to wrong conclusions.

Isn't learning, and getting a better understanding of how-things-work, fun!

Cheers,
Bart

One might think that one could subtract the G1 and G2 channels from a single exposure to eliminate PRNU and determine shot noise when the exposure is near clipping and read noise is not significant, and thus avoid the need for duplicate exposures required for the usual sensor analysis as described by Roger Clark.

I used the new version of Rawdigger to extract the G1 and G2 channels from a D800e NEF flat field shot exposed just short of saturation (green pixel values ~13,085 and saved the resulting TIFFs. The TIFFs were compressed and would not open in ImagesPlus, which needs uncompressed TIFFs. The files opened in ImageJ, but the appearance of the field was quite non-uniform suggesting that ImageJ does not handle compressed TIFFs properly. To obtain uncompressed TIFFs, I opened the files in PhotoshopCC and saved in non-compressed format and then cropped to the 200 x 200 pixels in the central image for analysis.

Subtracting the G1 and G2 channels in ImagesPlus after adding an offset to one file to avoid negative numbers gave a standard deviation of 102.5 for two files and the SD for one file was 102.5/sqrt(2) or 72.5. Using the split_cfa function of Iris gave identical results.

Subtracting the green channels from duplicate images using the same exposure using ImagesPlus and Roger's method gave a standard deviation of 85.5 for two images and 60.4 for one image. This is significantly different from the results obtained by subtracting the channels from the Rawdigger single image, and I conclude that one should use the standard method as outlined by Roger for sensor analysis. I do not know the reasons for this difference.

Regards,

Bill



Logged

Bart_van_der_Wolf

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 8913
Re: RawDigger 0.9.18 is out, with sRAW analysis and TIFF export
« Reply #35 on: July 17, 2013, 06:15:59 pm »

One might think that one could subtract the G1 and G2 channels from a single exposure to eliminate PRNU and determine shot noise when the exposure is near clipping and read noise is not significant, and thus avoid the need for duplicate exposures required for the usual sensor analysis as described by Roger Clark.

Hi Bill,

That's one of the possible uses, when one is limited to a single file.

Quote
I used the new version of Rawdigger to extract the G1 and G2 channels from a D800e NEF flat field shot exposed just short of saturation (green pixel values ~13,085 and saved the resulting TIFFs. The TIFFs were compressed and would not open in ImagesPlus, which needs uncompressed TIFFs. The files opened in ImageJ, but the appearance of the field was quite non-uniform suggesting that ImageJ does not handle compressed TIFFs properly. To obtain uncompressed TIFFs, I opened the files in PhotoshopCC and saved in non-compressed format and then cropped to the 200 x 200 pixels in the central image for analysis.

Many potential sources for the introduction of an unexpected result. Did you check if the saved result from PhotoshopCC still corresponds with the RawDigger mouse pointer values for those same coordinates? Photoshop is not what I would trust blindly (15-bit and such).

When I export a Raw file data dump (without the masked pixels) with DCRaw (DCRaw -D -4 -T filename), I get the exact same readouts as Rawdigger shows, when opening that TIFF output in ImageJ. I can then use the macro I made for ImageJ to separate the G1 and G2 channels and also produce the G1-G2 result if no prior math or cropping is required. The G1-G2 difference file is in 32-bit floating point, so even no offset provision for avoiding negative integers is required.

I'll do a comparison on my Raw files tomorrow, between an exposure pair and the 2 single file sub-channels.

Cheers,
Bart
Logged
== If you do what you did, you'll get what you got. ==

bjanes

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 3387
Re: RawDigger 0.9.18 is out, with sRAW analysis and TIFF export
« Reply #36 on: July 17, 2013, 08:55:51 pm »

Hi Bill,

That's one of the possible uses, when one is limited to a single file.

Many potential sources for the introduction of an unexpected result. Did you check if the saved result from PhotoshopCC still corresponds with the RawDigger mouse pointer values for those same coordinates? Photoshop is not what I would trust blindly (15-bit and such).

When I export a Raw file data dump (without the masked pixels) with DCRaw (DCRaw -D -4 -T filename), I get the exact same readouts as Rawdigger shows, when opening that TIFF output in ImageJ. I can then use the macro I made for ImageJ to separate the G1 and G2 channels and also produce the G1-G2 result if no prior math or cropping is required. The G1-G2 difference file is in 32-bit floating point, so even no offset provision for avoiding negative integers is required.

I'll do a comparison on my Raw files tomorrow, between an exposure pair and the 2 single file sub-channels.

Cheers,
Bart

Bart,

Another way to get the Rawdigger TIFFs into ImagesPlus is to open the TIFFs in ImageJ and save as FITS, which ImagesPlus can use. My previoius anomalous results with ImageJ were due to improper parameters in Rawdigger. One needs to use the half-size option in the Bayer processing options. I get the same results using the ImageJ route as with the PhotoshopCC route. I will await your results tomorrow.

Bill
Logged

bjanes

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 3387
Re: RawDigger 0.9.18 is out, with sRAW analysis and TIFF export
« Reply #37 on: July 17, 2013, 11:03:51 pm »

Bart,

Another way to get the Rawdigger TIFFs into ImagesPlus is to open the TIFFs in ImageJ and save as FITS, which ImagesPlus can use. My previoius anomalous results with ImageJ were due to improper parameters in Rawdigger. One needs to use the half-size option in the Bayer processing options. I get the same results using the ImageJ route as with the PhotoshopCC route. I will await your results tomorrow.

Bill

Bart,

I downloaded your ImageJ macro and used it on my NEF with the DCRaw -D -4 -T arguments and it worked like a charm and gave me the split out raw channels. Subtracting the uncropped channels in ImageJ gave nearly identical results as from the previous crops processed through PhotoshopCC.

Still awaiting your tests tomorrow.

Bill
Logged

Bart_van_der_Wolf

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 8913
Re: RawDigger 0.9.18 is out, with sRAW analysis and TIFF export
« Reply #38 on: July 18, 2013, 02:17:00 pm »

I'll do a comparison on my Raw files tomorrow, between an exposure pair and the 2 single file sub-channels.

I have extracted a center crop (200x200px) of the G1 channel from 2 files from my Canon EOS-1Ds3, extracted via DCRaw as undemosaiced Raw data numbers and split with ImageJ into separate channel files. It was verified that the channel files represent the same data numbers as RawDigger gives for the same sensel positions. Then the same Auto Blackpoints have been subtracted as RawDigger does.
ID=142_G1_BPC1023, Min=12601, Max=13411, Mean=13005.046, StdDev=93.598
ID=143_G1_BPC1023, Min=12757, Max=13539, Mean=13143.038, StdDev=95.265

Mathematically, if they are truly Random and Independent samples from a Normal distribution, one could expect a more accurate average noise estimate to be Sqrt((93.598^2+95.265^2))/Sqrt(2) = 94.435 for these two samples from the same noisy distribution.

However, when we actually subtract the channels of 2 files and take the statistics from that result we get:
Min=-492, Max=212, Mean=-137.991, StdDev=80.857 . That lower StdDev suggests that there is a non-random component in the data which reduces the Standard Deviation. To get the actual noise level we still need to divide by Sqrt(2).

-----------------------------------------------
Repeating that for the G2 channel from 2 files:
ID=142_G2_BPC1024, Min=12470, Max=13212, Mean=12864.084, StdDev=92.014
ID=143_G2_BPC1024, Min=12607, Max=13392, Mean=12999.766, StdDev=92.694

Again, mathematically, if they are truely Random and Independent samples from a Normal distribution, one could expect a
more accurate average noise estimate to be Sqrt((92.014^2+92.694^2))/Sqrt(2) = 92.355 .

However, when we actually subtract the channels of 2 files and take the statistics from that result we get:
Min=-527, Max=242, Mean=-135.682, StdDev=79.093 . That also suggests that there is a non-random component in the data which reduces the Standard Deviation. To get the actual noise level we still need to divide by Sqrt(2).

-----------------------------------------------
Now repeating that for the subtraction of the G1 and G2 channels in a single file, which may be not truely independent and may have some PRNU added to the Normal distribution noise:
ID=142_G1-G2_BPC, Min=-378, Max=638, Mean=140.963, AvgStdDev=123.317, CorrectedStdDev=87.198
and
ID=143_G1-G2_BPC, Min=-407, Max=652, Mean=143.272, AvgStdDev=124.258, CorrectedStdDev=87.864

That shows a Standard deviation that is indeed closer to the proper subtraction of exactly the same sensel positions, but not quite as low. I've added the four 16-bit TIFF files of the Black-point adjusted center-crops for experimentation, and the histograms of the single file G1-G2 subtractions for information.

I'm not exactly sure yet how to correlate that to the sample positions offset or PRNU, maybe I need to try with more samples to test it in practice.

Cheers,
Bart
« Last Edit: July 18, 2013, 06:34:27 pm by BartvanderWolf »
Logged
== If you do what you did, you'll get what you got. ==

bjanes

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 3387
Re: RawDigger 0.9.18 is out, with sRAW analysis and TIFF export
« Reply #39 on: July 19, 2013, 10:02:49 am »


Now repeating that for the subtraction of the G1 and G2 channels in a single file, which may be not truely independent and may have some PRNU added to the Normal distribution noise:
ID=142_G1-G2_BPC, Min=-378, Max=638, Mean=140.963, AvgStdDev=123.317, CorrectedStdDev=87.198
and
ID=143_G1-G2_BPC, Min=-407, Max=652, Mean=143.272, AvgStdDev=124.258, CorrectedStdDev=87.864

That shows a Standard deviation that is indeed closer to the proper subtraction of exactly the same sensel positions, but not quite as low. I've added the four 16-bit TIFF files of the Black-point adjusted center-crops for experimentation, and the histograms of the single file G1-G2 subtractions for information.

I'm not exactly sure yet how to correlate that to the sample positions offset or PRNU, maybe I need to try with more samples to test it in practice.


Bart,

Thank you very much for your detailed investigations. I repeated my investigation using Rawdigger to separate the green channels from two images (Image 15 and Image 16) of flat fields taken under identical conditions and exposure with the Nikon D800e. For comparison, I also used a DCRaw conversion with -D -4 -T and split out the channels with your ImageJ macro. Finally, I used IP to split out the green channel of the two identical exposures (IP combines G1 and G2 into a single file). For subtraction of images to eliminate non-random noise, I usually use ImagesPlus (IP) since it simplifies the task. Since it does not accept compressed TIFF, I used ImageJ to open the TIFFs and save them as FITs for processing by IP. With IP, I cropped the files to the central 400 x 400 pixels.

For completeness, I also performed subtractions using ImageJ using non-cropped files, since I don't know how to crop by the numbers with ImageJ--it seems to require a manual selection. The results are nearly identical to those obtained with cropped images.

The results are summarized in this table. As you found, subtracting G2 from G1 of a single image gives a larger standard deviation than subtracting either the G1 or G2 from two identical exposures.



Looking at the subtraction of the green channels of a single file, I see some banding which is confirmed with FFT analysis.



No such banding is noted when subtracting crops from two separate files.



My investigations are consistent with your theory that PRNU is not eliminated when one subtracts two green channels from the same file. For sensor analysis, one should use two images obtained under identical conditions as standard practice. The ability of Rawdigger to separate out the channels does offer an additional tool for image analysis.

Regards,

Bill


Logged
Pages: 1 [2] 3   Go Up