Pages: [1] 2   Go Down

Author Topic: Histogrammar V1.0  (Read 21429 times)

Guillermo Luijk

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2005
    • http://www.guillermoluijk.com
Histogrammar V1.0
« on: July 08, 2007, 05:20:28 pm »


Always download latest version HERE.

______________________


I have just uploaded a new version of Histogrammar, a free software to plot detailed histograms from 16-bit images in their native 0..65535 range. Ideal to study histograms in deep, compare RAW developers, processings,...

It adds an option to plot the histogram in logaritmic scale that I have never seen in any other software and I find very interesting to check dynamic range of scenes and sensor (the image has to be provided in linear format for this, as the ones developed using DCRAW).

Hope you find it useful.

Regards.

Histogrammar V1.0 click on "DESCARGAR HISTOGRAMMAR V1.0"

« Last Edit: February 26, 2009, 05:15:15 pm by GLuijk »
Logged

marcmccalmont

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1780
Histogrammar V1.0
« Reply #1 on: July 09, 2007, 03:33:31 am »

Thanks for listening!
Marc
Logged
Marc McCalmont

Guillermo Luijk

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2005
    • http://www.guillermoluijk.com
Histogrammar V1.0
« Reply #2 on: July 09, 2007, 04:08:10 am »

Quote
Thanks for listening!
Marc
[a href=\"index.php?act=findpost&pid=127230\"][{POST_SNAPBACK}][/a]

Did I?  

bjanes

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 3387
Histogrammar V1.0
« Reply #3 on: July 09, 2007, 11:25:19 am »

Guillermo,

The histogram program looks very nice, but when I click on the link I just get some document ion in Spanish but not the actual executable program. Am I doing something wrong?

Bill
Logged

bjanes

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 3387
Histogrammar V1.0
« Reply #4 on: July 09, 2007, 11:44:44 am »

Quote
Guillermo,

The histogram program looks very nice, but when I click on the link I just get some document ion in Spanish but not the actual executable program. Am I doing something wrong?

Bill
[a href=\"index.php?act=findpost&pid=127274\"][{POST_SNAPBACK}][/a]

I found the link on the upper right and downloaded the program. Comments to follow.

Bill
Logged

Guillermo Luijk

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2005
    • http://www.guillermoluijk.com
Histogrammar V1.0
« Reply #5 on: July 09, 2007, 02:30:53 pm »

Quote
I found the link on the upper right and downloaded the program. Comments to follow.

Bill
[a href=\"index.php?act=findpost&pid=127276\"][{POST_SNAPBACK}][/a]

Oh yeah sorry, "DESCARGAR HISTOGRAMMAR V1.0" is the download link into that page.
I have just uploaded a new version with some minor errors corrected. Try the new one if you find problems, it should look like this:




Thanks for testing.
« Last Edit: July 09, 2007, 02:33:07 pm by GLuijk »
Logged

bjanes

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 3387
Histogrammar V1.0
« Reply #6 on: July 09, 2007, 03:19:43 pm »

Quote
I have just uploaded a new version of Histogrammar, a free software to plot detailed histograms from 16-bit images in their native 0..65535 range. Ideal to study histograms in deep, compare RAW developers, processings,...

It adds an option to plot the histogram in logaritmic scale that I have never seen in any other software and I find very interesting to check dynamic range of scenes and sensor (the image has to be provided in linear format for this, as the ones developed using DCRAW).

Hope you find it useful.

[a href=\"index.php?act=findpost&pid=127167\"][{POST_SNAPBACK}][/a]

Guillermo,

The high resolution histogram is impressive, but with 16 bit files derived from a 12 bit raw image, there are gaps in the data. Only 1 of 16 values have data and the rest are blank, since the 16 bit image is derived from the 12 bit by multiplying the latter by 4. Raw values 1,2,3,4 .. 4095 = 16, 32, 48, 64, .. 65536. It might make sense to allow the user to adjust the bin width used to construct the histogram frequencies. With a bin width of 16, you would capture all the values, but a larger width could be used so tht the histogram would fit on the width of the screen.

Bill
Logged

Guillermo Luijk

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2005
    • http://www.guillermoluijk.com
Histogrammar V1.0
« Reply #7 on: July 09, 2007, 03:39:09 pm »

Quote
Guillermo,

The high resolution histogram is impressive, but with 16 bit files derived from a 12 bit raw image, there are gaps in the data. Only 1 of 16 values have data and the rest are blank, since the 16 bit image is derived from the 12 bit by multiplying the latter by 4. Raw values 1,2,3,4 .. 4095 = 16, 32, 48, 64, .. 65536. It might make sense to allow the user to adjust the bin width used to construct the histogram frequencies. With a bin width of 16, you would capture all the values, but a larger width could be used so tht the histogram would fit on the width of the screen.

Bill
[a href=\"index.php?act=findpost&pid=127309\"][{POST_SNAPBACK}][/a]


mmmm you are a bit wrong, let me explain: RAW files are 12 bit as you say, so displayed on a 16-bit range (0..65535) should have 15-hole gaps between every two tones or levels.
But we are making a double mistake:
1. White balance completely disorganises this 1+15 distribution by scaling all levels by a non integer multiplier.
2. Second, and much more important: after developing, interpolated values are calculated in the true 16-bit range, i.e. 0..65535, so they completely fill all empty gaps we had before the development stage.

Find here some samples of linear histograms from TIFF files generated using DCRAW.
- Peaks correspond to non-interpolated levels (i.e. levels captured by the sensor). R, G and B peaks perfectly match due to absence of WB in this case (I wanted to compare pure camera histograms).
- Between peaks (and also in the peaks themselves) spread all the rest of interpolated levels, completely filling the gaps.

It's not until the gamma compensation when again holes will be generated in the lowest part of the histogram, making it look like a gruyere piece of cheese.

Histograms correspond to the same scene as viewed from:
- Canon 5D: completely linear 12-bit
- Leica M8: non-linear 8 bit
- DMR back: already 16-bit in origin so no peaks will be visible:


[span style=\'font-size:9pt;line-height:100%\']5D[/span]





________________________________________________________
[span style=\'font-size:9pt;line-height:100%\']M8[/span]





________________________________________________________
[span style=\'font-size:9pt;line-height:100%\']DMR[/span]






All those 3 RAW files were developed with no WB applied (-r 1 1 1 1 option in DCRAW) and hence that awful green colour.
In a normally developed RAW file, the WB scales differently each channel and this happens:




Non interpolated peaks now mismatch and again gaps between every two captured levels are filled with interpolated new levels.
That is why I always state that ETTR doesn't provide more tonal richness, as all levels are filled in any case, but provides more tonal precission as peaks corresponding to captured levels get closer (after the exposure correction down), and thus interpolated levels are calculated more accurately.

And this is all thanks to the 12-bit to 16-bit conversion, and I have heard no one saying this!!! In that way, ETTR in a camera with 16-bit native RAWs like the DMR back, will improve SNR as usual, but willl not improve at all the tonal precission of the capture as all overexposed levels will agregate after the exposure correction down to the same levels they would get in a non exposed to the right shot from the begining.


It is always best to start from the whole 0..65535 range, and zoom out with the X-axis zoom option provided. Pressing that Zoom out 4 times will do what you are looking for (0..65535 -> 0..4095). Every press scales down the histogram by 2, which equals to substracting one bit of the image file resolution.
« Last Edit: July 09, 2007, 06:21:47 pm by GLuijk »
Logged

allan67

  • Jr. Member
  • **
  • Offline Offline
  • Posts: 69
    • http://
Histogrammar V1.0
« Reply #8 on: July 10, 2007, 04:08:03 am »

Thanks a lot for the programme, it's a nice tool to play with and it helps to understand what's happenning with the image when you apply different conversions and modifications.


Quote
making it look like a gruyere piece of cheese
A real gruyère cheese does not have holes in it! It's swiss and in Switzerland nobody will pay for air ;-)
It's emmenthal cheese that has lots of holes.

Allan
Logged

francois

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 13769
Histogrammar V1.0
« Reply #9 on: July 10, 2007, 04:29:50 am »

Quote
Thanks a lot for the programme, it's a nice tool to play with and it helps to understand what's happenning with the image when you apply different conversions and modifications.
A real gruyère cheese does not have holes in it! It's swiss and in Switzerland nobody will pay for air ;-)
It's emmenthal cheese that has lots of holes.

Allan
Correct, in fact ideal histograms would look like Gruyère cheese!
« Last Edit: July 10, 2007, 04:30:08 am by francois »
Logged
Francois

Guillermo Luijk

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2005
    • http://www.guillermoluijk.com
Histogrammar V1.0
« Reply #10 on: July 10, 2007, 05:58:53 am »

Quote
A real gruyère cheese does not have holes in it! It's swiss and in Switzerland nobody will pay for air ;-)
It's emmenthal cheese that has lots of holes.

Allan
[{POST_SNAPBACK}][/a]


ehhhhhh what about this?
[a href=\"http://www.estvideo.com/dew/index/php-mysql/2004/09]Gruyere cheese[/url]
I think there are both, with and without holes. But you are the experts

francois

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 13769
Histogrammar V1.0
« Reply #11 on: July 10, 2007, 06:45:30 am »

Quote
ehhhhhh what about this?
Gruyere cheese
I think there are both, with and without holes. But you are the experts
[{POST_SNAPBACK}][/a]
This is [a href=\"http://fr.wikipedia.org/wiki/Emmental]Emmental[/url] cheese, with large holes (see real Gruyère, without holes)!
Gruyère village is just 60km from my office...

 
« Last Edit: July 10, 2007, 06:47:19 am by francois »
Logged
Francois

Guillermo Luijk

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2005
    • http://www.guillermoluijk.com
Histogrammar V1.0
« Reply #12 on: July 10, 2007, 08:36:28 am »

Quote
This is Emmental cheese, with large holes (see real Gruyère, without holes)!
Gruyère village is just 60km from my office...

 
[a href=\"index.php?act=findpost&pid=127412\"][{POST_SNAPBACK}][/a]

Ok, I believe you.

allan67

  • Jr. Member
  • **
  • Offline Offline
  • Posts: 69
    • http://
Histogrammar V1.0
« Reply #13 on: July 10, 2007, 10:57:36 am »

Quote
ehhhhhh what about this?
Gruyere cheese
I think there are both, with and without holes. But you are the experts
[{POST_SNAPBACK}][/a]

Just another site for real [a href=\"http://www.gruyere.com]Gruyère[/url]

Cheese in Switzerland is a BIG subject  

Allan
Logged

bjanes

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 3387
Histogrammar V1.0
« Reply #14 on: July 10, 2007, 06:50:48 pm »

Quote
mmmm you are a bit wrong, let me explain: RAW files are 12 bit as you say, so displayed on a 16-bit range (0..65535) should have 15-hole gaps between every two tones or levels.
But we are making a double mistake:
1. White balance completely disorganises this 1+15 distribution by scaling all levels by a non integer multiplier.
2. Second, and much more important: after developing, interpolated values are calculated in the true 16-bit range, i.e. 0..65535, so they completely fill all empty gaps we had before the development stage.
[a href=\"index.php?act=findpost&pid=127312\"][{POST_SNAPBACK}][/a]

Guillermo,

Thanks for the detailed reply. Floating point white multipliers do spread out the histogram somewhat. They shift the red and blue values to the right by a fixed amount (for daylight white balance), but they don't really disperse the values. Here is an example from my Nikon D200 with raw conversion done with Iris rather than DCRaw.

Here is a raw file converted into 16 bit with no white balance or gamma adjustment. There are gaps in the data and the RGB channels are aligned as you predicted:

[attachment=2792:attachment]

Here is the file with white balance. There are still gaps since the red and blue data are merely shifted to the right by a fixed amount and are no longer superimposed:

[attachment=2793:attachment]

And here is the file with white balance and gamma adjustment (gamma = 2.2):

[attachment=2794:attachment]

There are still plenty of gaps in the data. I don't really see the filling in that you described.

By the way, I did note that Adobe Camera Raw seems to do things a bit differently.
In the histogram, the channels are superimposed and there is no spreading of the data. Any thoughts on this?

[attachment=2795:attachment]

The above histogram was from the shadow area which undergoes little compression with the gamma correction. Towards the right of the histogram, the gamma causes the values to be closer together. This does not occur with linear data. I think that is why the Swiss cheese appearance appears in the shadows but not the highlights.

[attachment=2796:attachment]

Quote
That is why I always state that ETTR doesn't provide more tonal richness, as all levels are filled in any case, but provides more tonal precission as peaks corresponding to captured levels get closer (after the exposure correction down), and thus interpolated levels are calculated more accurately.

Pressing that Zoom out 4 times will do what you are looking for (0..65535 -> 0..4095). Every press scales down the histogram by 2, which equals to substracting one bit of the image file resolution.
[a href=\"index.php?act=findpost&pid=127312\"][{POST_SNAPBACK}][/a]

I don't see the degree of filling in that you describe. However, it is really a moot point since the eye can distinguish 100 levels per f/stop at most, and the 2048 levels in the brightest f/stop of a 12 bit raw image can not be seen by the human eye, which in effect blends many of them together.

Yes, I do see that the zoom control effectively does what I was asking for. All in all, a very nice tool. Now you need to make a similar tool for HDR 32 bit floating point images  

Bill
« Last Edit: July 10, 2007, 07:20:23 pm by bjanes »
Logged

Guillermo Luijk

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2005
    • http://www.guillermoluijk.com
Histogrammar V1.0
« Reply #15 on: July 10, 2007, 07:18:57 pm »

I think you don't see the degree of filling because, as a Spanish motto says, your RAW developer is "more weird than a green dog"

Please try to develop the same RAW file on DCRAW with something like:
dcraw -v -w -o 0 -q 3 -T -4 pic.nef

You will see all those levels completely filled with interpolated values. In fact, it is a bit stupid (no idea why Iris does) not to make use of the entire 16-bit range for 2 reasons:
1. 16 bits means more precision than 12 bits: if the demosaicing algorithm calculates a value that falls between two 12-bit range levels, why should we round it to one of them losing our additional precission?
2. Even if human eye cannont distinguish so many levels, most pictures will get into PS for heavy edition after developing. The more differentiated levels you have the more robust they will behave in curves, levels, and so forth adjustments to avoid solarization and other artifacts.

Another possibility for such a strange histogram is that the images you are analysing are not linear but already have a gamma compensation curve applied. Are you sure about this point? a colour profile conversion for instance implies a gamma compensation without the user having to introduce any parameter value, and a short window view will show levels equally spaced as the ones you show.

Anyway, Histogrammar well served in this case to find out how Iris soft works. It looks interesting, I will have a look at that softwarre tomorrow.

Regards.


PS: there is a new version of Histogrammar. Called the same V1.0 but with some corrections and additional information applied.

I forgot about your question about ACR version (BTW did you notice the very different % level filling between your Iris and ACR developments?), and I have developed a RAW file and got the same results as you: vertical lines with R, G and B perfectly matched:




But it has to be that way if my theory is correct: gamma compensation has expanded the shadows of the histogram, creating holes where there used to be a continous raw of RGB values. Now they look like aligned as the gamma compensation affected equally the 3 channels. It's my interpretation.

The only thing that disturbs me a bit is the absence of peaks as those found on DCRAW's -q 3 development. It seems ACR's algorithm is quite differente from that of DCRAW, as the same RAW file on DCRAW (just R channel presented for simplicity) shows:

DCRAW linear:




DCRAW gamma corrected (trough sRGB colour profile conversion):




which still shows the peaks found on the linear histogram.
« Last Edit: July 10, 2007, 08:04:17 pm by GLuijk »
Logged

bjanes

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 3387
Histogrammar V1.0
« Reply #16 on: July 10, 2007, 07:40:55 pm »

Quote
Guillermo,

Thanks for the detailed reply. Floating point white multipliers do spread out the histogram somewhat. They shift the red and blue values to the right by a fixed amount (for daylight white balance), but they don't really disperse the values. Here is an example from my Nikon D200 with raw conversion done with Iris rather than DCRaw.

Here is a raw file converted into 16 bit with no white balance or gamma adjustment. There are gaps in the data and the RGB channels are aligned as you predicted:

[attachment=2792:attachment]

Here is the file with white balance. There are still gaps since the red and blue data are merely shifted to the right by a fixed amount and are no longer superimposed:

[attachment=2793:attachment]

And here is the file with white balance and gamma adjustment (gamma = 2.2):

[attachment=2794:attachment]

There are still plenty of gaps in the data. I don't really see the filling in that you described.

By the way, I did note that Adobe Camera Raw seems to do things a bit differently.
In the histogram, the channels are superimposed and there is no spreading of the data. Any thoughts on this?

[attachment=2795:attachment]

The above histogram was from the shadow area which undergoes little compression with the gamma correction. Towards the right of the histogram, the gamma causes the values to be closer together. This does not occur with linear data. I think that is why the Swiss cheese appearance appears in the shadows but not the highlights.

[attachment=2796:attachment]

Postscript:
The degree of filling towards the right of the histgram is less with the Iris conversion with white balance and gamma of 2.2.

[attachment=2797:attachment]

I don't see the degree of filling in that you describe. However, it is really a moot point since the eye can distinguish 100 levels per f/stop at most, and the 2048 levels in the brightest f/stop of a 12 bit raw image can not be seen by the human eye, which in effect blends many of them together.

Yes, I do see that the zoom control effectively does what I was asking for. All in all, a very nice tool. Now you need to make a similar tool for HDR 32 bit floating point images   

Bill
[a href=\"index.php?act=findpost&pid=127487\"][{POST_SNAPBACK}][/a]
Logged

Andre22

  • Newbie
  • *
  • Offline Offline
  • Posts: 4
    • http://
Histogrammar V1.0
« Reply #17 on: July 11, 2007, 06:56:21 am »

Guillermo- thanks for this!

A request: would it possible to make portable versions of all your apps? No install, no (or optional) registry entries, all configurations saved as an *.ini in the program folder, instructions for manual install of any libraries etc. ?

Andre
Logged

John Sheehy

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 838
Histogrammar V1.0
« Reply #18 on: July 11, 2007, 08:44:56 am »

Quote
Anyway, Histogrammar well served in this case to find out how Iris soft works. It looks interesting, I will have a look at that softwarre tomorrow..
[a href=\"index.php?act=findpost&pid=127491\"][{POST_SNAPBACK}][/a]

IRIS is not a RAW converter.  It is a tool for astrophotography that can load RAW data literally, and perform mathematical operations on it, including the ability to interpolate full RGB from CFA images.  By default it will not get any in-between values at all for interpolation or WB, as it works on integers, and loads RAW data without gaps.  You have to multiply the data by some value first to get in-between values.  There are 3 CFA interpolation methods, one creates one intermediate value, one creates 3, and one fills the histogram.  All three methods have spikes where the original data values were.

IRIS will do linear "HDR", but you need to export the result to take advantage of better demoasaicing algorithms.  IRIS RAW decoding is based on DCRAW, but the auther has not used the AHD demosaicing from DCRAW.
« Last Edit: July 11, 2007, 08:45:39 am by John Sheehy »
Logged

bjanes

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 3387
Histogrammar V1.0
« Reply #19 on: July 11, 2007, 03:49:32 pm »

Quote
I think you don't see the degree of filling because, as a Spanish motto says, your RAW developer is "more weird than a green dog"

Please try to develop the same RAW file on DCRAW with something like:
dcraw -v -w -o 0 -q 3 -T -4 pic.nef

You will see all those levels completely filled with interpolated values. In fact, it is a bit stupid (no idea why Iris does) not to make use of the entire 16-bit range for 2 reasons:
1. 16 bits means more precision than 12 bits: if the demosaicing algorithm calculates a value that falls between two 12-bit range levels, why should we round it to one of them losing our additional precission?
2. Even if human eye cannont distinguish so many levels, most pictures will get into PS for heavy edition after developing. The more differentiated levels you have the more robust they will behave in curves, levels, and so forth adjustments to avoid solarization and other artifacts.
[a href=\"index.php?act=findpost&pid=127491\"][{POST_SNAPBACK}][/a]

I did develop the raw file with DCRaw as you suggested and the levels were completely filled in as you predicted.

I do not think that the raw converters under discussion get 16 bit precision from a 12 bit file--they are merely filling in blank spaces with interpolated data. Why else would anyone go to the trouble of developing a camera with 16 bits? It does make sense for the converter to use the 16 bit space for intermediate results, but we are not getting true 16 bit precision.

The highlight areas of a 12 bit file contain wasted bits, since such heavy duty editing that would lead to posterization is not done in real world photography. For example, Nikon cameras have a compressed raw format which reduces redundant highlight levels. The resulting file has 683 levels rather than the 4096 values in the original file, and Nikon calls it visually losless. Some people claim that they see a difference in the highlights but have produced no convincing demonstration confirming their claims.

Quote
Another possibility for such a strange histogram is that the images you are analysing are not linear but already have a gamma compensation curve applied. Are you sure about this point? a colour profile conversion for instance implies a gamma compensation without the user having to introduce any parameter value, and a short window view will show levels equally spaced as the ones you show.

Anyway, Histogrammar well served in this case to find out how Iris soft works. It looks interesting, I will have a look at that softwarre tomorrow.
[a href=\"index.php?act=findpost&pid=127491\"][{POST_SNAPBACK}][/a]

Apparently, Iris does not do the type of interpolation that fills in the blank spaces as John Sheehy pointed out. The files were not gamma corrected as evidenced by their dark appearance in the Histogrammar preview. So far as I know, Iris does not support color profiles.

Quote
I forgot about your question about ACR version (BTW did you notice the very different % level filling between your Iris and ACR developments?), and I have developed a RAW file and got the same results as you: vertical lines with R, G and B perfectly matched:

The only thing that disturbs me a bit is the absence of peaks as those found on DCRAW's -q 3 development. It seems ACR's algorithm is quite differente from that of DCRAW, as the same RAW file on DCRAW (just R channel presented for simplicity)...
[a href=\"index.php?act=findpost&pid=127491\"][{POST_SNAPBACK}][/a]

I did a conversion with Nikon Capture NX and noted the same behavior as with ACR. I also did a conversion with RawMagick, which is a well regarded converter that uses floating point math rather than integers.

Here is the shadow portion of the histogram:
[attachment=2804:attachment]

and towards the midportion:
[attachment=2805:attachment]

The missing spaces on the left are present, but irregularly spaced. It could be that they were filled in in the linear file, but the holes appeared during the gamma correction as you predict.

With Histogrammar we see all types of things that were not apparent in low resolution histograms.

Bill
Logged
Pages: [1] 2   Go Up