Luminous Landscape Forum

Equipment & Techniques => Digital Cameras & Shooting Techniques => Topic started by: Guillermo Luijk on January 07, 2008, 02:22:11 pm

Title: Camera's histogram reliable to the RAW data
Post by: Guillermo Luijk on January 07, 2008, 02:22:11 pm
We all know that camera's display in terms of histogram and blown highlight is based on a in-camera JPEG version of the RAW file, and is pesimistic with respect to the real RAW data.

One can set a low contrast value in the camera parameters, no saturation, and so on,... but those tricks have little effect in making the camera's more reliable to the RAW information.

Thanks to playing with DCRAW I learnt some time ago how critical is the white balance in transforming the RAW's histogram, by overexposing one or even 2 channels by 1 or even 1.5 f-stops.

I have tried to find out the way to neutralise my 350D's white balance setting a neutral (1.0 scaling in all 3 channels) white balance. Images will display in a wrong colour (of course RAW data remains intact), but the histogram and blown blinking highlights should improve.

Tell me what do you think.
The detailed article can be found here:
UNIWB. MAKE CAMERA DISPLAY RELIABLE (http://www.guillermoluijk.com/tutorial/uniwb/index_en.htm)


Steps:

1. I create a colour chart representing all possible RGB proportions (I chose to fix G=64). Over this chart I shot and I will try to find out which tone becomes pure gray in the RAW file when no WB at all is applied (that is camera's "true gray"):

 :

(http://www.guillermoluijk.com/tutorial/uniwb/pruebathumb.jpg)

This is the RAW magenta chart that can be used on any 350D to achieve this specificic WB: UniWB350D.cr2 (http://stats.sergiodelatorre.com/dlcount.php?id=_GUI_&url=http://www.guillermoluijk.com/download/uniwb350d.cr2)

What do you think?
Title: Camera's histogram reliable to the RAW data
Post by: bansal98 on January 08, 2008, 11:22:20 pm
I've been doing a similar thing for gamma correction. The idea being to get as close to the raw histogram as possible. See here (http://luminous-landscape.com/forum/index.php?showtopic=22062).
Title: Camera's histogram reliable to the RAW data
Post by: Panopeeper on January 08, 2008, 11:40:19 pm
GENIAL!!!

I will have the chart printed and do the same with my cameras.
Title: Camera's histogram reliable to the RAW data
Post by: Panopeeper on January 08, 2008, 11:49:26 pm
Btw, how did you make the basis color chart? If I make a graduation from (0,64,255) - (255,64,0), then I don't get any green, like you got in the upper left corner.
Title: Camera's histogram reliable to the RAW data
Post by: Guillermo Luijk on January 09, 2008, 09:28:35 am
Quote
Btw, how did you make the basis color chart? If I make a graduation from (0,64,255) - (255,64,0), then I don't get any green, like you got in the upper left corner.
[{POST_SNAPBACK}][/a] (http://index.php?act=findpost&pid=166051\")

I set (almost) arbitrarily G=64 since I knew the higher sensitivity of the sensor for the G channel would produce seeing a pure gray at some R&B dominant point.

Regarding the gradation, R and B are independent of each other: R runs 0 to 255 left to right, and B runs 0 to 255 up to down, so top left corner is pure green (0,64,0), and (64,64,64) is the gray point for this test chart. All that is clearer looking at the Hue distribution of the test chart:

(http://www.guillermoluijk.com/tutorial/uniwb/cartaprueba_hue.jpg)


Find here the test chart in 1024x1024 size: [a href=\"http://img520.imageshack.us/img520/6944/cartag64zs5.jpg]CARTA DE PRUEBA G64[/url]

I don't recommend you to print the charts but shoot them straight on your monitor instead.
This will not invalidate the procedure at all as long as the magenta chart is shot under the same conditions (monitor settings + room light conditions) as the test chart.
Thus your monitor does not need to be calibrated at all (mine is not), and your room's light conditions can also be any as long as they remain constant (be careful with sunlight entering your window, these experiments are best done at night).
You will even be more precise this way, and you will save time and ink!

BTW I have been told Nikon cameras (from D200) can do this by simply configuring the so called 'UniWB' white balance mode in the camera. No idea how difficult is that.

Regards.

PS: this is a sample shot using the custom WB achieved, comparing the result of developing the same RAW file:
- Left: with its embedded WB from camera (i.e. our specific neutral WB)
- Middle: no WB at all (forcing 1.0 multipliers in DCRAW)
- Right: Daylight WB preset

(http://img138.imageshack.us/img138/3779/testwa3.jpg)

First 2 images match very well as expected.

In my first quick tests, blinking areas in the camera's display (350D) are very close to real RAW blown areas. In fact, surprisingly they are a little bit more optimistic than a strict partial saturation detection over the RAW data. So watch out!.
Whether this happens or not depends solely on the 350D's implementation of the blinking highlights; the WB calculation procedure is conceptually flawless.
Title: Camera's histogram reliable to the RAW data
Post by: Guillermo Luijk on January 09, 2008, 09:48:58 am
Quote
I've been doing a similar thing for gamma correction. The idea being to get as close to the raw histogram as possible. See here (http://luminous-landscape.com/forum/index.php?showtopic=22062).
[a href=\"index.php?act=findpost&pid=166044\"][{POST_SNAPBACK}][/a]

I have your article in mind since I saw it some days ago. Will have a close look tonight.
Title: Camera's histogram reliable to the RAW data
Post by: Panopeeper on January 09, 2008, 05:46:56 pm
Guillermo,

I carried out the "neutral WB calibration" (I too came to the idea of doing it with the monitor). The result is as good as I can imagine it; the blinking clipping indication of the camera reflects the clipping down to a tiny clipping, barely perceivable. This is much better than what I got with setting the color temperature and color correction (that method would certainly lead to an exact adjustment, but that is very tiresome).

Now I can rely on the blinking on the camera's display. The histogram itself is only a rough indication because it is too tiny and one can't see if it really reaches the right end.

Are you the originator of this idea? I want to spread it and want to give the well-deserved credit to the originator.

Btw, I copied the template on every CF card, so that if it becomes necessary to change the custom balance, one can return to this setting any time. Of course I protected the template from accidental erasure.
Title: Camera's histogram reliable to the RAW data
Post by: Guillermo Luijk on January 09, 2008, 05:54:37 pm
Quote
I carried out the "neutral WB calibration" (I too came to the idea of doing it with the monitor). The result is as good as I can imagine it; the blinking clipping indication of the camera reflects the clipping down to a tiny clipping, barely perceivable. This is much better than what I got with setting the color temperature and color correction (that method would certainly lead to an exact adjustment, but that is very tiresome).

Now I can rely on the blinking on the camera's display. The histogram itself is only a rough indication because it is too tiny and one can't see if it really reaches the right end.

Are you the originator of this idea? I want to spread it and want to give the well-deserved credit to the originator.

It's nice, isn't it?

Strangely, I am the originator of the idea. Last Monday was bank holiday in Spain and I spent the whole Sunday night shooting my monitor lol. Didn't want to go to bed until I published the article.

Feel free to distribute the idea but from my experience, don't expect too much feedback or good vibrations, few have the knowledge to appreciate such intriguing camera fine tunings.

Just curious, what multipliers did you achieve? I was surprised at my 5,6% error, remarkably low since I was really sleepy and didn't take any care to be precise. Wanted to go to bed!
Title: Camera's histogram reliable to the RAW data
Post by: Panopeeper on January 09, 2008, 06:23:52 pm
I am not familiar with DCRaw and have not used that for the calibration; I picked the grey point from the shot of the original template with Rawnalyze, and verified the resulting grey too with it. But I was curious re the factors, so I found in DCRaw that there is an option -i for information. It displayed (1057, 1045, 1083, 1045) - this is 3.6%.

Re shooting the monitor you need to think of followings:

1. defocus and do not go too close, otherwise you record the "gaps" between the monitor pixels as well (Moire-like effect),

2. do not go too far, for the "grey to be" area (R=162, G=64 B=104 for your camera) has to cover the circle on the focusing screen. Be generous, because that area is involved in the custom WB setting. I increased the size, so that I got a larger image.

Guillermo, you deserve a big THANK for this idea.

Btw, if you want to distribute it, you can upload the final raw file for the WB setting, that depends only on the camera but not on the illumination any more. I will offer this on the Rawnalyze page, with due credit to you (I am at the middle of the new manual, it is done in a few days).
Title: Camera's histogram reliable to the RAW data
Post by: Guillermo Luijk on January 09, 2008, 06:46:47 pm
Quote
I am not familiar with DCRaw and have not used that for the calibration; I picked the grey point from the shot of the original template with Rawnalyze, and verified the resulting grey too with it. But I was curious re the factors, so I found in DCRaw that there is an option -i for information. It displayed (1057, 1045, 1083, 1045) - this is 3.6%.

Re shooting the monitor you need to think of followings:

1. defocus and do not go too close, otherwise you record the "gaps" between the monitor pixels as well (Moire-like effect),

2. do not go too far, for the "grey to be" area (R=162, G=64 B=104 for your camera) has to cover the circle on the focusing screen. Be generous, because that area is involved in the custom WB setting. I increased the size, so that I got a larger image.

Guillermo, you deserve a big THANK for this idea.

Btw, if you want to distribute it, you can upload the final raw file for the WB setting, that depends only on the camera but not on the illumination any more. I will offer this on the Rawnalyze page, with due credit to you (I am at the middle of the new manual, it is done in a few days).
[a href=\"index.php?act=findpost&pid=166215\"][{POST_SNAPBACK}][/a]

Grrr, so you beat me. What cameras do you have?
With dcraw -v -w pic.cr2 DCRAW will display the 1.0 based multipliers.

I already put the 350D UniWB RAW file for download at the end of the first post of the thread. BTW I have just been sent a chart shot using the 40D from other user and the magenta chart (we could call it 'wine chart') for that camera and conditions happened to be:

Canon 40D:
(http://img135.imageshack.us/img135/4310/huesatiw8.jpg)

R=170
G=64
B=115

My 350D was:
Canon 350D:
R=162
G=64
B=104

Of course comparing is a bit nonsense since the differences are in this case not only for the cameras but also include my and the other user's conditions (monitor + lighting conditions), so I will send him the wine chart to obtain the final UniWB40D.cr2 RAW file.
Anyway, looking at the hue transformations in both cameras, if the 350D's well know issue of getting "oranged" reds, what about the 40D? could it even be worse looking at the distribution?:

(http://img104.imageshack.us/img104/4537/anaranjadojo0.jpg)
Title: Camera's histogram reliable to the RAW data
Post by: Panopeeper on January 09, 2008, 10:53:48 pm
My "grey to be" for the 40D is (120, 64, 75). Of course, this does not mean anything.

I don't understand your "oranged red" subject. I did not make that step at all. Instead of generating a non-WBd file by DCRaw, I displayed the shot of the rainbow chart (what you posted) with Rawnalyze, no WBing, and looked for grey point (simply by clicking around, and when close, I saw from the numbers, in which direction I have to go to get even closer to true grey). I transformed the co-ordinates of that point over the original (simply by measuring it with a ruler on the screen and interpolating), so I got the RGB of the "grey to be". I created that square.

Then I shot that "winechart", though my one is not really "winy" and checked the color in Rawnalyze. It was a bit off grey, I corrected directly the winechart (for example the red was too much, so I reduced it in the winechart), and shot it again. I made three iterations and then shot a serie of the scenery outside with 1/3 stops, and compared the clipping indication with that in Rawnalyze (it produces something like ACR's clipping indication). Since then I have been happy. I don't touch it, because it can become only worse :-)

My 40D template can be downloaded from

http://www.cryptobola.com/PhotoBola/WB_00040.CR2 (http://www.cryptobola.com/PhotoBola/WB_00040.CR2)

It can be stored and used on the CF card with this name (both the 40D and the 20D are picky about accepting files with names different from the original scheme).
Title: Camera's histogram reliable to the RAW data
Post by: NikosR on January 10, 2008, 03:58:32 am
Maybe I'm missing something, but how is this method original and different than the well known Uniwb advocated by Iliah Borg on dpreview and nikoncafe for some years now and actually quite extensively used by Nikon photographers by setting up unitary WB coefficients in their Nikon cameras?

(PS. BTW, unitary wb coefficients are not handled gracefully in ACR/LR and some other raw converters so one has to use 'not-quite unitary' coefficients if one wants to work in these converters).
Title: Camera's histogram reliable to the RAW data
Post by: Guillermo Luijk on January 10, 2008, 05:25:01 am
Quote
I don't understand your "oranged red" subject. I did not make that step at all.

What I mean by the "orange reds" in the Canon is that many Canon user complain at the red colours their cameras capture, which trend to become a bit orange (this is a fact I have often read about in the forums).

And lookig at the RGB -> HSV transformations of the original (test chart) and the results captured by the camera sensors, we can notice 2 effects:

1. Displacement of the gray (R=G=B ) point: this was expected due to the different relative channel sensitivities.

2. But also a colorimetric effect, which makes tone (Hue) change. You can see that the angle of Hues that could be considered as acceptably "red" in the original test chart, becomes a much narrower angle in the 350D and specially the 40D.
Of course this was influenced by the lilghting conditions present in the scene (a calibrated monitor and 100% neutral ambience lighting would be needed to properly confirm). But anyway I wonder if also the camera sensor could have an influence on this.
After all R channel is just a band pass filter which can behave far from ideally, and could happen that in these sensors the R channel trend to have more gain on the frequency band close to oranges.

(http://img213.imageshack.us/img213/8569/orangedredshr4.jpg)


BTW your UniWB RAW yields in DCRAW multipliers  1.016330 1.000000 1.033622.
Do you mind if I offer it for download in my website and other forums?
(I will reference you of course)

One question: by opening it I see it display not only the chart but also a screenshot of PS. Does this mean the camera only takes into account a centred portion of the RAW to calculate the custom WB? I didn't know that. Do you know exactly what size (%) of the total image surface?

Thanks
Title: Camera's histogram reliable to the RAW data
Post by: Guillermo Luijk on January 10, 2008, 05:29:26 am
Quote
Maybe I'm missing something, but how is this method original and different than the well known Uniwb advocated by Iliah Borg on dpreview and nikoncafe for some years now and actually quite extensively used by Nikon photographers by setting up unitary WB coefficients in their Nikon cameras?

(PS. BTW, unitary wb coefficients are not handled gracefully in ACR/LR and some other raw converters so one has to use 'not-quite unitary' coefficients if one wants to work in these converters).
[a href=\"index.php?act=findpost&pid=166284\"][{POST_SNAPBACK}][/a]

I knew that Nikons from the D200 could be configured for UniWB, but thought they had just this config option in the camera. Is the method for Nikons the same as explained here?    then why was it assumed to be valid just for Nikons? I always thought they had something special that made UniWB much easier in them. I have a feeling I could have reinvented the wheel here hehe.


Of course the unitary converters are not to be fed into your RAW developer, that's nonsense. This is only a white balance for capture, not for developing. Commercial RAW developers find problems in handling certain WB multipliers, probably when converted they fall in too extreme Temperature/Tone parameter values.

Just curious: a friend of mine turned his old 300D into an IR camera. It worked fine in the capture, but he found that ACR was not capable to achieve the needed Temperature for a proper development. He used DCRAW instead, which has no limitations on the multipliers, and he got his pics OK then.
Title: Camera's histogram reliable to the RAW data
Post by: NikosR on January 10, 2008, 05:38:42 am
Quote
I knew that Nikons from the D200 could be configured for UniWB, but thought they had just this config option in the camera. Is the method for Nikons the same as explained here?    then why was it assumed to be valid just for Nikons? I always thought that.

Of course the unitary converters are not to be fed into your RAW developer, that's nonsense. This is only a white balance for capture, not for developing.
Just curious: a friend of mine turned his old 300D into an IR camera. It worked fine, but he found that ACR was nos capable to achieve the needed Temperature for a proper development. He used DCRAW instead, which has no limitations on the multipliers, and he got his pics OK then.
[{POST_SNAPBACK}][/a] (http://index.php?act=findpost&pid=166292\")

With Nikon cameras that support uploading the coefficients directly (e.g. D2 cameras) you can do it directly. For other cameras you just load a custom white balance (examples of which Mr. Borg and others have provided) created, I suppose, in a similar fashion as you do here.

[a href=\"http://www.pochtar.com/Uni.zip]http://www.pochtar.com/Uni.zip[/url] provides you with a UniWB custom setting and a 'not-quite-uni'WB for raw converters that cannot handle coefs of exactly 1.0



So your work here might be original with regards the specific Canon implementations but the idea is far from original. A Google search for UniWB will point you to a lot of discussions.

I'm not saying that one feeds these coefficients in the converter. I'm just saying that if a RAW file with coeffs of exactly 1.0 (at least for Nikon cameras) is fed into ACR/LR, ACR/LR will not treat them correctly when you subsequently try to change WB. This has been acknowledged by Thomas Knoll in Adobe forums. http://www.adobeforums.com/cgi-bin/webx/.3bb6a85c.3bbfce73/2 (http://www.adobeforums.com/cgi-bin/webx/.3bb6a85c.3bbfce73/2)

Other converters (e.g. Nikon Capture and NX) are known to crash if an attempt at adjusting the exposure is done in the converters. BTW, such adjustment even if possible would not lead to the benefits one might expect when using normal WB coefficients as there is no artifically clipped channels in the raw conversion as rendered by the converter in the first place! That's the benefit of UniWB.
Title: Camera's histogram reliable to the RAW data
Post by: Guillermo Luijk on January 10, 2008, 05:46:19 am
Quote
I'm not saying that one feeds these coefficients in the converter. I'm just saying that if a RAW file with coeffs of exactly 1.0 is fed into ACR/LR, ACR/LR will not treat them correctly when you subsequently try to change WB. This has been acknowledged by Thomas Knoll in Adobe forums.

I cannot find a logic in this. Even if ACR or any RAW developer cannot handle the 1.0 multipliers (which is understandable), they should not fail later to apply a correct "regular" WB since pure RAW data are not affected by the UniWB. Could you please provide me that Thomas Knoll reference?

Regarding the copyright/original idea matter, believe me, I am 0% interested in being acknowledged as the inventor of anything. I just had fun configuring my camera for a neutral WB last night, that is all.
People waste too much time in the forums discussing who is the owner or the copier of new ideas, instead of being positive and trying to get the best of our hobbies/work.
Title: Camera's histogram reliable to the RAW data
Post by: NikosR on January 10, 2008, 05:50:40 am
Quote
I cannot find a logic in this. Even if ACR or any RAW developer cannot handle the 1.0 multipliers (which is understandable), they should not fail later to apply a correct "regular" WB since pure RAW data are not affected by the UniWB. Could you please provide me that Thomas Knoll reference?

Regarding the copyright/original idea matter, believe me, I am 0% interested in being acknowledged as the inventor of anything. I just had fun configuring my camera for a neutral WB last night, that is all.
People waste too much time in the forums discussing who is the owner or the copier of new ideas, instead of being positive and trying to get the best of our hobbies/work.
[a href=\"index.php?act=findpost&pid=166294\"][{POST_SNAPBACK}][/a]

For your first comment pls. see my updated post above.

For the second, I have no reason not to believe you, but I thought that setting the record straight was the correct thing to do, since there were posters in this thread assuming exactly what you were not interested in. Blame it on my scholarly background.
Title: Camera's histogram reliable to the RAW data
Post by: Guillermo Luijk on January 10, 2008, 06:01:20 am
Quote
For your first comment pls. see my updated post above.

"Using the "UNIWB" hack is confusing Camera Raw into thinking the image is a double exposed image, and Camera Raw using a different processing path."

I understand. This is an elegant way to admit that ACR has an undesired behaviour (i.e. a bug)  
Everytime I find more happy with Dave's  DCRAW.

PS: Panopeeper, I am terribly sorry for making you believe I was the originator of this idea (if it is true someone thought of it before I did, although I couldn't find a similar procedure anywhere valid for any camera model and brand), and I hope you can forgive me some day for such an offence. lol
Title: Camera's histogram reliable to the RAW data
Post by: Panopeeper on January 10, 2008, 05:00:25 pm
Gullermo,

1. of course I have no problem if you offer anyone the 40D WB template I created,

2. I made a search for uniwb and did not find anything re how that has been achieved. There are several ways, for example there is a huge color checker box from Gretag with 120 or so colors. When one shots that from the monitor, one can find a color, which is quite close to grey without WB (non-demosaiced, of course). From there, it requires I guess perhaps five to eight iterations (only guessing) to arrive at the proper color.

The core of the idea is using an "inverted color", and for me you are the inverter.

3. Anyway, if you disseminate the method and/or the raw files, you should explain, that the contrast, saturation, sharpness and color tone have to be "neutralized". How to achieve that depends on the camera. For example the 40D shows a "0" at the appropriate places: at the left end of sharpness and at the middle of the other ones.
Title: Camera's histogram reliable to the RAW data
Post by: Guillermo Luijk on January 10, 2008, 05:18:26 pm
Quote
2. I made a search for uniwb and did not find anything re how that has been achieved. There are several ways, for example there is a huge color checker box from Gretag with 120 or so colors. When one shots that from the monitor, one can find a color, which is quite close to grey without WB (non-demosaiced, of course). From there, it requires I guess perhaps five to eight iterations (only guessing) to arrive at the proper color.

The core of the idea is using an "inverted color", and for me you are the inverter.

3. Anyway, if you disseminate the method and/or the raw files, you should explain, that the contrast, saturation, sharpness and color tone have to be "neutralized". How to achieve that depends on the camera. For example the 40D shows a "0" at the appropriate places: at the left end of sharpness and at the middle of the other ones.
[a href=\"index.php?act=findpost&pid=166387\"][{POST_SNAPBACK}][/a]

This is a good point, I want to make tests in my camera to find out since I wonder:
- Contrast: would be 0 value a good idea or a negative one? (I imagine negative is an inverted S curve).
- Saturation: I guess the right value here is 0, non-modifying saturation value.
- Sharpness: I don't think it will affect too much, but 0 should be OK.
- The big question: what about sRGB/AdobeRGB? since sRGB expands more the histogram, do you think it could be a good idea to set AdobeRGB? or we could even fall into being too optimistic with the neutral WB?

I would really like to know which is the camera's criteria to blink. I think it is partial saturation (any channel saturated) but I am not sure if there are more hidden rules.

Regarding the authority, I really don't care. I had this idea Sunday night by myself, I had fun putting it into practice, and if someone already had it before is fine.

PS: I recall this question above: One question: by opening it [your RAW winechart] I see it displays not only the chart but also a screenshot of PS. Does this mean the camera only takes into account a centred portion of the RAW to calculate the custom WB? I didn't know that. Do you know exactly what size (%) of the total image surface?
Title: Camera's histogram reliable to the RAW data
Post by: Panopeeper on January 10, 2008, 05:57:15 pm
Quote
- The big question: what about sRGB/AdobeRGB? since sRGB expands more the histogram, do you think it could be a good idea to set AdobeRGB? or we could even fall into being too optimistic with the neutral WB?

I think we will have to thest the effect.

Quote
I would really like to know which is the camera's criteria to blink. I think it is partial saturation (any channel saturated) but I am not sure if there are more hidden rules

I compared the blinking on the camera with raw clipping shown in Rawnalyze and found, that they were virtually equal (expectable due to the close to 1.0000 factors), and that has shown, that green clipping alone was enough.

It would be easy to test by extracting the embedded JPEG from the "blinking" raw file and verifying the relevant areas in PS or whatever. I could make a test for that, but I believe this is a clear-cut issue, I rather keep writing the manual.

Regarding the authority, I really don't care. I had this idea Sunday night by myself, I had fun putting it into practice, and if someone already had it before is fine.

Quote
I recall this question above: One question: by opening it [your RAW winechart] I see it displays not only the chart but also a screenshot of PS. Does this mean the camera only takes into account a centred portion of the RAW to calculate the custom WB? I didn't know that. Do you know exactly what size (%) of the total image surface?

This was not a question but I posted you what to watch for (closeness, etc.). The circle in the viewfinder has to be filled (the camera manual states this).
Title: Camera's histogram reliable to the RAW data
Post by: Guillermo Luijk on January 10, 2008, 07:33:11 pm
hehe, David Coffin has given me a crazy but very logical idea for studio shooting: to print the magenta tone on transparency slides. They could be used then as filters on D65 neutral lamps o flashes, so the RAW would be (theoretically) balanced without the need to scale the channels at all.

Also the higher exposure of the B and R channels would mean better SNR for a given exposure of the G channel on the RAW file.

"Instead of your magenta chart, you need to make lamps ofthat color. Maybe you could print the color on a transparencyand mount it over a D65 lamp. Never again would you need tostretch and clip color channels to achieve white balanace, atleast when working in a studio.", David Coffin said.

Unfortunately I don't think such a solution could not introduce any colour dominant (the print has to be perfect, on a calibrated printer).
For B&W should be a good solution to get less noise and tonal richness.
Title: Camera's histogram reliable to the RAW data
Post by: Panopeeper on January 10, 2008, 08:21:39 pm
Guillermo,

1. This is nothing new about using color filter with digital cameras.

For two months ago I ordered a magenta filter from B+W, in order to increase the dynamic range in daylight. After several weeks waiting it turned out, that B+W (Schneider) discontinued that filter (and Tiffan does not make any larger than 77mm).

2. The whole idea is nonsense. The filter changes the composition of the light and thereby the resulting *raw* histogram. The goal of this WB trick is not to change the light, nor to change the raw histogram, but to make the camera software use 1.0 as WB coefficient. Note this:

whatever values the sensels produce, they will be white balanced

Changing the composition of light might create "parallel" raw values (depending on the scenery and on the light source), but that does not help, if the camera white balances that away. We don't need "nicer" histograms, we need "like" histograms in raw and in the in-camera JPEG.

So, stick to your idea.
Title: Camera's histogram reliable to the RAW data
Post by: John Sheehy on January 11, 2008, 01:35:14 am
Quote
hehe, David Coffin has given me a crazy but very logical idea for studio shooting: to print the magenta tone on transparency slides. They could be used then as filters on D65 neutral lamps o flashes, so the RAW would be (theoretically) balanced without the need to scale the channels at all.

"Flomo" transparent magenta book report binders that I bought in a "99 cent" store here, in double thickness, balance my 550EX almost perfectly, with two sheets.

Quote
Also the higher exposure of the B and R channels would mean better SNR for a given exposure of the G channel on the RAW file.

That's full color SNR, of course.  No more "highlight recovery" per se.  The shadow noise, however, will lack the chromatic cast that it has with most light sources, even when the blackpoint is correctly determined by the converter.

Quote
"Instead of your magenta chart, you need to make lamps ofthat color. Maybe you could print the color on a transparencyand mount it over a D65 lamp. Never again would you need tostretch and clip color channels to achieve white balanace, atleast when working in a studio.", David Coffin said.

His choice of the word "instead" was probably not ideal.  "In addition to" would probably be a better choice, as the more accurate histogram could be beneficial regardless of lighting color.  Together, however, you get a more (but still not perfectly) RAW-accurate histogram with a review image that is close to real color, *and* maximum full-color DR.

Quote
Unfortunately I don't think such a solution could not introduce any colour dominant (the print has to be perfect, on a calibrated printer).
For B&W should be a good solution to get less noise and tonal richness.
[a href=\"index.php?act=findpost&pid=166406\"][{POST_SNAPBACK}][/a]

What is a "color dominant"?

Magenta lighting/filtering works very nicely, but as I said, there is no recovery of highlights in the red and blue channel (unless the highlights are saturated and on the green side of the spectrum).
Title: Camera's histogram reliable to the RAW data
Post by: Jonathan Wienke on January 11, 2008, 05:58:00 am
Quote
"Instead of your magenta chart, you need to make lamps ofthat color. Maybe you could print the color on a transparencyand mount it over a D65 lamp. Never again would you need tostretch and clip color channels to achieve white balanace, atleast when working in a studio.", David Coffin said.

Unfortunately I don't think such a solution could not introduce any colour dominant (the print has to be perfect, on a calibrated printer).
For B&W should be a good solution to get less noise and tonal richness.
[a href=\"index.php?act=findpost&pid=166406\"][{POST_SNAPBACK}][/a]

This approach has been done already by putting a magenta filter on the camera so that the levels of the RAW channels are approximately equal. This does not introduce any color casts in the final images, but does require different WB settings during RAW conversion than what one would normally encounter.
Title: Camera's histogram reliable to the RAW data
Post by: Guillermo Luijk on January 11, 2008, 06:41:20 am
What you John and Jon are telling me encourages me to experiment. I have asked a friend who has a photo studio with a RGB projection system to project the test chart (not the magenta winechart) over it and shoot it with the camera.
Analysing the RAW produced I will provide him with the corresponding magenta chart so he can use it to light the scene in a photo session. If the whole light system (PC+projector) gets correctly calibrated by doing this for the camera used, any picture taken with the camera under those lighting conditions would be already balanced in the RAW file.
Paradoxically, under these circumstances, if he sets a custom UniWB in the camera (need to generate the proper file for it), the images would be balanced on the camera display, eliminating both any magenta or green casts.

The important thing however would be to look at the RAW files produced to check if they are properly balanced. This would be close to the ideal situation of a balanced sensor: balanced in-camera RAWs and balanced camera display as well. Now just the scene is unbalanced  

Panopeeper, don't you think we can try to take this further? if there already exist magenta standard filters that work well in improving RGB balance, why this custom magenta light shouldn't be optimum? the combinations seem to be endless.

I will post the findings...
Title: Camera's histogram reliable to the RAW data
Post by: John Sheehy on January 11, 2008, 09:04:03 am
Quote
This approach has been done already by putting a magenta filter on the camera so that the levels of the RAW channels are approximately equal. This does not introduce any color casts in the final images, but does require different WB settings during RAW conversion than what one would normally encounter.
[a href=\"index.php?act=findpost&pid=166473\"][{POST_SNAPBACK}][/a]

When I first experimented with a magenta filter a few years ago, ACR was not able to white-balance the camera's native RAW color balance!  The "tint" slider could not go far enough to the left.  A later version of ACR increased the tint range.
Title: Camera's histogram reliable to the RAW data
Post by: Panopeeper on January 11, 2008, 11:24:36 am
Guillermo,

something is wrong. Please separate the two issues:

1. Increasing the DR by filtering the light: this can be achieved by filtering the light source, by creating light with different composition, or by filtering the light at the camera; no matter, which way you go.

The effect of filtering depends on the light source, and of course on the targets. I do not think that the same filtering, which is useful in daylight, would be useful in studio.

In any case, this filtering has nothing to do with the topic, namely raw-like histogram displayed in the camera.

2. The question, how to make the in-camera histogram like the raw histogram is totally independent of the light composition at the time of shooting, except when using auto WB.

There is only one parameter here (well, there are two of them): the WB factors applied by the camera (let's neglect the other settings, like saturation). In order to coax the camera into applying (1.0, 1.0), you have to provide a sample shot, which represents the inverse proportions of the raw colors from a white sample shot under certain lighting conditions. This is, what you achieved with shooting the "winechart". Shooting a white sheet under light with the color of the purplish chart would yield the same, of course.

After having established custom WB based on this shot, the WB factors are fixed.

From now on the lighting of the scenery plays no role in this issue. You can use whatever filter you want to, it will influence how the histogram looks - depending on the scenery - but it won't change the matching of the raw histogram with the in-camera histogram.

The same, from a different viewpoint: with choosing the lighting, you can achieve, that the three channels are "parallel" (of a given scenery), but it has nothing to do with how that shot is displayed in-camera.
Title: Camera's histogram reliable to the RAW data
Post by: Guillermo Luijk on January 11, 2008, 11:49:48 am
Quote
In any case, this filtering has nothing to do with the topic, namely raw-like histogram displayed in the camera.

Of course Panopeeper, this is a completely different goal, but based in the same idea of playing with different levels in the three channels.
In this case what I propose is to model the lighting conditions to obtain directly an already balanced RAW file, i.e. in this case the RAW data is gonna be changed so later WB becomes unnecessary, and having other advantages (if it works of course). And of course this would be only of application when you have control over the entire scene's lighting (studio).

It's a different point, the UniWB question for displaying a more reliable histogram is finished now. Maybe I should have opened a new topic for this.
Title: Camera's histogram reliable to the RAW data
Post by: Panopeeper on January 11, 2008, 12:16:35 pm
Quote
In this case what I propose is to model the lighting conditions to obtain directly an already balanced RAW file, i.e. in this case the RAW data is gonna be changed so later WB becomes unnecessary, and having other advantages (if it works of course

I don't see the "other advantages", not even "this" one. A WB balanced raw file offers no advantage over a not WB-balanced one. Methink you are mixing up a white balanced raw file with one, the channels of which are clipping at the same time (I called this "parallel channels").

The only advantage I see is avoiding one channel clipping much before the other channels, because that restricts the DR of the other channels. However, this depends on the actual scenery as well, not only on the lighting conditions.

Shoot a foresty scenery with clouded sky, the chance is high, that the green will be clipped. With bright, clear sky the blue may be ahead, and shoot colorful flowers under the very same lighting condition and the red may clip first.
Title: Camera's histogram reliable to the RAW data
Post by: Guillermo Luijk on January 11, 2008, 12:33:30 pm
I understand your point and I agree.

But it's usual that the R and B channels need overexposure in relation to the G channel to perform the WB, so this approach would overpexpose them in relation to the G channel and we could do a more precise ETTR for the 3 channels at the same time.
Of course how often this would be true will depend on the colours present in the scene: a deeply red object for instance could easily reach higher values in the R histogram than in the the G histogram, so in that case we would be losing DR in the G.

It's an approach to get:
1. White balanced RAW files straight from the camera.
2. USUALLY (i.e. for most scenes in average), better SNR (and hence DR) and more tonal richness.

Just an experiment, don't take it as a proposal for everyday shooting at all.

PS: I like the term "parallel channels".
Title: Camera's histogram reliable to the RAW data
Post by: Jonathan Wienke on January 11, 2008, 12:47:16 pm
I can't vouch for other RAW converters, but ACR handles UniWB settings pretty well. Here's my results with the 1Ds:

(http://www.visual-vacations.com/images/1Ds-UniWB.gif)

Left side is RawAnalyze linear histogram, right is ACR adjustments after click-WB. The WB adjustments are pretty extreme, but still within usable range.
Title: Camera's histogram reliable to the RAW data
Post by: Panopeeper on January 11, 2008, 01:17:07 pm
Quote
It's an approach to get:
1. White balanced RAW files straight from the camera

Why? There is no advantage of that.

Quote
2. USUALLY (i.e. for most scenes in average), better SNR (and hence DR) and more tonal richness

This is, what could be achieved using filters on the camera.

The problem I see is, that the fiters required to achieve this have been used only on film cameras and are not the same quality as the newer ones for digital cameras. I would put only a multicoated UV filter (if at all, for lens protection) or polar filter, from the best brand. I don't use a low quality magenta filter in order to increase the DR, if that reduces the image quality, except perhaps in special circumstances.
Title: Camera's histogram reliable to the RAW data
Post by: Guillermo Luijk on January 11, 2008, 02:23:44 pm
Quote
Why? There is no advantage of that.
This is, what could be achieved using filters on the camera.

You could be right in the end. It's all about the pleasure of getting the visual confirmation that I have understood how a camera works and I can control it, not the camera controls me. I am not a photographer and in the last months I have not even shot too much, mainly because I felt quite anxious for not knowing exactly what was going on inside the camera. I think I have reached a higher degree of understanding lately and this forum has helped specially.
Looking at your software to analise RAW files I think you share this point of view.

BTW you have a 40D. I am planning to change my 350D for a 5D instead once its substitute is in the market. Could you suggest me some advantages to choose 40D instead of 5D?
I think I prefer the 5D for being FF (bokeh and wide angle) and for NOT having all those new features such as sensor cleaning, HTP, live view,... that I reallly don't appreciate too much and don't want to pay for.

I don't like the idea to put colour filters either. I would only like them on a B&W camera (http://luminous-landscape.com/forum/index.php?showtopic=22123) lacking colour filters in the sensor, to obtain different B&W in an optical way.

Regards
Title: Camera's histogram reliable to the RAW data
Post by: Panopeeper on January 11, 2008, 02:56:55 pm
Jonathan,

I see you downloaded the newer version with black background in the histograms. I changed it from grey to increase the contrast for better visibility of thin columns, which occur typically with clipping caused by specular points.

Then I found that the white point bar hides the highest column. Even worse is the saturation colum, for that can not be moved, like the white point, and these two together made a column totally disappearing.

So, yesterday I changed the appearance of these bars. Plus, I changed the scaling of the histogram. It is still logarithmic, now going from 0.01% at the bottom to 100%. (This is the percentage  of pixels of that color within 1/512th of the sensor's range, related to all pixels of that color).

However, the very bottom row is not on the logarithmic  scale; it represents anything from 1 pixel to 0.01%, i.e. even a single pixel in that range appears as a dot in the bottommost line. Disadvantage: hot pixels.

Anyway, if you intend to use the program, you should download the newest version, I just uploaded it (there was an error in the histogram of the mapped data, that is gone now).

The markers at the side of the histograms mark *now* from the bottom upwards 0.02%, 0.05%, 0.1%, 0.2%, 0.5%, 1%, 2%, 5%, 10%, 20% and 50%.

This should be shown in the window, but I did not find any good way to show them yet. Suggestions are welcome.
Title: Camera's histogram reliable to the RAW data
Post by: Panopeeper on January 11, 2008, 08:13:48 pm
Quote
BTW you have a 40D. I am planning to change my 350D for a 5D instead once its substitute is in the market. Could you suggest me some advantages to choose 40D instead of 5D?

Well, let's turn the question around: why would you purchase a two and half years old model for probably more, than the top of the technology? Note, that the much more expensive models, even the 1DsMkII don't have better technology than the 40D.

Quote
I think I prefer the 5D for being FF (bokeh and wide angle)

I don't understand the bokeh issue. It depends on the lens alone, on nothing else.

Btw, if you want to see comparisons of bokeh between seven lenses, take a look at http://www.panopeeper.com/Bokeh (http://www.panopeeper.com/Bokeh)

The availability of more wider lenses is indisputable, just like the other side of the coin: that you don't need super expensive lenses to reach far.

Plus, the EF-S line contains the 10-22, which is said to be quite good and very wide, and the super-duper 17-55 IS, which is really super; I can confirm, that it stands up well against the EF 16-35 f/2.8 L Mk II, which is the best one for FF, and even against the 50mm f/1.4. For a few weeks ago I purchased the 16-35 and the 17-55 at the same time (telling the dealer ahead, that I will send back one of them), and I tested these against each other and against three other of my lenses (and I kept the 17-55).

Quote
and for NOT having all those new features such as sensor cleaning, HTP, live view,... that I reallly don't appreciate too much and don't want to pay for

Think once more about it:

- the sensor cleaning is a great feature

- HTP is only for JPEG shooters

- live view is great. I don't give a fig for live view, but now I can do something, what others can't: shoot with auto exposure bracketing without flapping the mirror in between; this is a side effect of live view.

Now, some positives to the cropping cameras, apart the focal length question:

- much lighter

- the selection of very good lenses is much larger, than for FF. This is not only due to the EF-S lenses, but to the cropping. Several lenses, which are mediocre on FF because of the weak edge performance, are stellar on cropping camera. For example the old "magic drainpipe", the legendary 80-200mm f/2.8L has been replaced by the 70-200mm because of the edge performance (take a look at the MTF). This lens has a renaissance on cropping cameras (I too have an excellent copy, though I plan to sell it, due to its weight). Another example: the 50mm f/1.4 is sharp at the edges at much larger apertures than on FF

- much cheaper (and calculate the lenses!)

There is a constant disadvantage: flaring, due to the unsuitable lensd hoods. One more reason to go with the 17=55mm: the hood is designed for the cropped view.

If there was a small brother of the 1DMkIII (lighter and cheaper), I would have gone for that, i.e. 1.3x cropping. But there is none.

A final remark: the dynamic range and lesser noise of the 5D is a myth, compared to the 40D. There was a thread about this question. The 5D outputs about 3570 levels; in the nineth stop there are only seven levels.
Title: Camera's histogram reliable to the RAW data
Post by: Guillermo Luijk on January 11, 2008, 08:43:30 pm
I think I understand (or already knew) all points but the live view (flapping mirror...). What do you mean with this, could you explain a bit more?

Regarding my priorities, with bokeh I meant capability of defocusing the background. My 70-200 f4L hardly diferentiates subject from background in APS-C while in FF it will be much nicer to use for portraits for instance. My 24-70 f2.8, which already produces a great bokeh in APS-C, is superb to get narrow DOF in FF.

The cleaning system, it's a matter of concept. I know this is argueably, but I prefer to open my camera and blow the sensor from time to time than having something that 'moves' my sensor.

Regarding wide angle, in the next months I am probably going deep into architecture photographing (who knows if even will purchase some tilt shift lens someday) and I want to be able to have a wider selection of wide angle lenses.

I am not likely to be interested in tele zooms. I went on a trip to Namibia with a 300mm f4L, and it was a nice experience to shoot the animals, but I don't think this is the kind of photography I will enjoy best. In fact my favourite pictures there were paradoxically among the few shots I did in wide angle (Dead Vlei (http://www.pbase.com/gluijk/image/88448861), Giraffe (http://www.pbase.com/gluijk/image/88448956)).

I have the Canon 10-22 and I am quite happy with it, but not so much to launch fireworks (Spanish expression). With the 50mm f1.4 I am not happy at all. At short distances (portraits,...) is perfect, but at long distances its sharpness decreases too quickly.

I know the 5D is a bit old now, but this does not mean it's old fashioned. It was a camera advanced to its time so it has a still great sensor today. I haven't been able to check DR, but I am sure that 40D's DR is a bit better than 5D's, I assume that. But I am developing a good technique to blend a second overexposed shot, so this is even funny for me. And anyway the 9 f-stops of DR in the 40D is not enough for indoor shooting in HDR scenes for instance.
For a really high DR I love the Super CCD concept.

The price, if I wait the substitute of the 5D comes, will surely get almost the same as the 40D's. You know how people are: as soon as there is a new camera, old models automatically become old fashioned. I like the 5D.
And I am probably travelling to the States soon, and photographic stuff there is by far cheaper than in Europe. The 5D started here at 3000 EUR ($4400), and now it is 2000 EUR ($2940). I expect it to reach close to 1500 EUR.

Thank you very much!

PS: from your bokeh samples I went into the panos. Nice stuff there. Why did you darken so much the skies in the B&W landscapes?
Title: Camera's histogram reliable to the RAW data
Post by: Panopeeper on January 11, 2008, 11:11:46 pm
Quote
I think I understand (or already knew) all points but the live view (flapping mirror...). What do you mean with this, could you explain a bit more?

If you turn on live view, you have to press the button in the middle of the big wheel to activate/deactivate it. When MLU too is turned on, the mirror flips up when activating live view. If you are using a remote control (there are very cheap wired ones from third parties), and you set up exposure bracketing, with a single press you start shooting all three shots of the bracket. Neither mirror flapping, nor button pressing occurs between the shots, which will be made in a fraction of a second.

Jonathan made a remark, that it is still not totally vobration free due to the shutter movement. This is correct, I have not verified yet the effect, I ope it won't be much. Anyway, I guess this will give the ideal basis for HDR, which may be my next project.

Quote
with bokeh I meant capability of defocusing the background

Thankx, but I do now the meaning. Once more: the quality of bokeh has to do *only* with the lens. FF or cropping plays a role only as far as the selection of focal length (and thereby of the lens) depends on cropping.

Every lens, which gives a good bokeh on FF, gives a good one on a cropping camera as well.

Quote
My 70-200 f4L hardly diferentiates subject from background in APS-C while in FF it will be much nicer to use for portraits for instance

This is a mistake. F4 is the problem, not the cropping.

Quote
My 24-70 f2.8, which already produces a great bokeh in APS-C, is superb to get narrow DOP in FF

Another mistake. The DoF does not depend on cropping either. Note, that the focal length of the lens is the same, no matter on which camera you are using it.

Quote
The cleaning system, it's a matter of concept. I know this is argueably, but I prefer to open my camera and blow the sensor from time to time than having something that 'moves' my sensor

I wish you much fun. I rather do it less often than I had to with the 20D.

Quote
Regarding wide angle, in the next months I am probably going deep into architecture photographing (who knows if even will purchase some tilt shift lens someday) and I want to be able to have a wider selection of wide angle lenses

If you are doing it professionally, under time pressure, then you need a medium format camera (you can get away already with EUR 20000 or so, plus the lenses).

Otherwise, you may think about panoramas.

There are many examples for that on http://www.panopeeper.com/panorama/Hungary.htm (http://www.panopeeper.com/panorama/Hungary.htm), or see the Mormon church on http://www.panopeeper.com/panorama/USA.htm (http://www.panopeeper.com/panorama/USA.htm)

It is unquestionably much more work than with a tilt and shift lens, but with panos

1. your options for creativity are much more,

2. the dynamic range is much higher. This needs explanation. The camera does not get better when shppting pano frames, but the DR of the scenery is sometimes much higher than the DR of the individual frames.

Examples:

Pano from 11 frames (http://www.panopeeper.com/panorama/BCFerryAground_28mm_1.jpg)

The frames have been shot 4.5 stops apart. With which camera can you do 13 stops?

Btw, I like the Dead Vlei.

Quote
Why did you darken so much the skies in the B&W landscapes?

Those sceneries are not B&W candidates in the classinc sense (you can see the same panos in color on the USA page). The B&W versions are rather gimmickry, going for very strong effect; I will try how they look on print. The sky was almost totally clear on all those panos. Though I might try to insert another, strongly cloudy sky (I do this sometimes, when the sky is not nice enough), then I would not blacken it.

(I have a stockpile of panoramic skies for this purpose.)
Title: Camera's histogram reliable to the RAW data
Post by: Guillermo Luijk on January 12, 2008, 05:19:23 am
Quote
If you turn on live view, you have to press the button in the middle of the big wheel to activate/deactivate it. When MLU too is turned on, the mirror flips up when activating live view. If you are using a remote control (there are very cheap wired ones from third parties), and you set up exposure bracketing, with a single press you start shooting all three shots of the bracket. Neither mirror flapping, nor button pressing occurs between the shots, which will be made in a fraction of a second.

Jonathan made a remark, that it is still not totally vobration free due to the shutter movement. This is correct, I have not verified yet the effect, I ope it won't be much. Anyway, I guess this will give the ideal basis for HDR, which may be my next project.
OK that's understood now. I normally do HDR (not the tone mapping, just getting the HDR data) from 2 shots: 0EV, +4EV. To obtain them I bracket -2,0,2, so that the -2 matches my desired 0EV (i.e. ETTR shot without blowing any highlight) and +2 give me the +4V sample. I use MLU and images match pixel by pixel even with my cheap tripod, it's very precise. Of course avoiding mirror flip could even be better but not necessary actually. The advantage I see in the live view is the speed at which the shots can be taken: I have to set timer+MLU so I have to wait a couple of seconds between shots, and with the 40D you don't. Good point. BTW is the bracketing in the 40D -2,0,2 or Canon finally allowed more f-stops?
I wrote an article showing some tests: HDR blending (http://www.guillermoluijk.com/article/nonoise/index_en.htm), surely nothing new for you. What are you planning to do in HDR? I am willing to see your research.



Quote
Thankx, but I do now the meaning. Once more: the quality of bokeh has to do *only* with the lens. FF or cropping plays a role only as far as the selection of focal length (and thereby of the lens) depends on cropping.

Every lens, which gives a good bokeh on FF, gives a good one on a cropping camera as well.
This is a mistake. F4 is the problem, not the cropping.
Another mistake. The DoF does not depend on cropping either. Note, that the focal length of the lens is the same, no matter on which camera you are using it.
Don't be offended, I said that in case bokeh does not mean exactly what I thought (I told you I am very new to this), or you meant quality of bokeh instead of amount of bokeh.
But then, being pragmatic, I have to disagree with you: DOF DOES depend on the cropping, i.e. on the sensor format. The same lens at the same focal length will obviously provide the same light projection over the sensor plane, but a different scene's framing (not sure if this is the right term to mean 'the portion scene falling into the sensor area') according to sensor size. So to obtain the same framing with my APS-C at 70mm f4 in the 5D, I will need to get closer to the subject or better (since distance would change the perspective) to use a larger focal length: 112mm f4. And 112mm f4 in the 5D will have less DOF than 70mm f4 in the APS-C, so we will increase the amount of bokeh for the same scene, distance and maximum aperture of the lens. So in terms of bokeh, a less luminous, lighter and cheaper lens provides more bokeh on a FF sensor.

Paradoxically if we don't change focal length nor get closer, the 5D at 70mm f4 will have more DOF than the APS-C at 70mm f4 since the reference circle of confusion for the 5D sensor is bigger than in the APS-C. Of course this comparision does not make any sense as we would be talking of different scene's framing. Once we increase the focal length on the 5D or get closer to the subject, the CoC effect is cancelled and the larger focal length or shorter distance prevails in reducing the DOF on the 5D.


Quote
If you are doing it professionally, under time pressure, then you need a medium format camera (you can get away already with EUR 20000 or so, plus the lenses.
Don't think so. I met sometime ago one important Spanish arquitecture photographer; he used to work medium format but he simply does not want to spend the huge amount of money in digital medium format at today's prices. He is NOT happy with his 5D and tilt shift lenses, but with some extra effort achieves an incredible quality, far beyond my intentions. Look at some of his work here: http://javierazurmendi.blogspot.com/ (http://javierazurmendi.blogspot.com/) (I know they are ridiculously small, but according to what he told me and the prints I saw in his studio, the quality suffices professional requirements perfectly).


Panos is one of my pending tasks for some day. What software do you recommend me? PT?
Title: Camera's histogram reliable to the RAW data
Post by: Westy on January 12, 2008, 06:28:59 am
Hi, my first post here.  I have  been watching with great interest some of the technical discussions you guys have on this site.  A few points:

I think this unity wb is a great idea, as the inaccuracy of the in-camera jpg at indicating correct clipping is one of the difficulties of ETTR.  One less thing the deriders of ettr have to complain about.

But one question I have is what exactly is the highlight warning on the lcd showing?  Is it showing clipping of any one of the three channels or is it some sort of composite?  I understand that the histogram on the lcd is usually some sort of green channel weighted composite.  Is the clipping warning working on this?  If so, is it possible to be clipping a channel (particularly red perhaps?) and it not showing blinkies on the lcd?  I guess the reason I came to this conclusion is that after implementing Guillermo's method I am still getting very similar flashies on the lcd preview between the unity wb and a scene wb.  However when developing a linear non-wb tiff I see that I still have maybe a stop of so of headroom until I get clipping.  I am assuming my custom white balance image is correct (although it looks more pink than magenta (i'm on a 5D)) as it is giving multipliers of about 1.01 for each channel in dcraw.

On the issue of DOF, Pano you are forgetting that maintaining identical FOV between different systems is usually required.  To obtain identical FOV between a 1.6x crop sensor and a FF sensor, one must either select a different focal length lens and/or change the distance to subject.  Both of these factors affect DOF.  You won't be able to capture the same lack of DOF (for the same FOV) with a 1.6x sensor as you could with for example a 50mm f1.4 on a full frame camera.

cheers, Bernie
Title: Camera's histogram reliable to the RAW data
Post by: Guillermo Luijk on January 12, 2008, 06:43:56 am
Quote
Hi, my first post here.  I have  been watching with great interest some of the technical discussions you guys have on this site.  A few points:

I think this unity wb is a great idea, as the inaccuracy of the in-camera jpg at indicating correct clipping is one of the difficulties of ETTR.  One less thing the deriders of ettr have to complain about.

But one question I have is what exactly is the highlight warning on the lcd showing?  Is it showing clipping of any one of the three channels or is it some sort of composite?  I understand that the histogram on the lcd is usually some sort of green channel weighted composite.  Is the clipping warning working on this?  If so, is it possible to be clipping a channel (particularly red perhaps?) and it not showing blinkies on the lcd?  I guess the reason I came to this conclusion is that after implementing Guillermo's method I am still getting very similar flashies on the lcd preview between the unity wb and a scene wb.  However when developing a linear non-wb tiff I see that I still have maybe a stop of so of headroom until I get clipping.  I am assuming my custom white balance image is correct (although it looks more pink than magenta (i'm on a 5D)) as it is giving multipliers of about 1.01 for each channel in dcraw.

I pointed that question in the thread, and the conclusion is that this will depend on the camera implementation. The method to cancel de Wb is flawless (the 1.0 multipliers confirm this); how it is interpreted in the camera's display will depend on your particular camera. And finding out the relation between the real RAW and the blinking camera highlights should be achieved by doing comparisions.
From the few pics I tested in my 350D, I pre-conclude that its blinking lights show any partial saturation (i.e. any channel blown makes the display blink) which is a good new. But any camera can be different.

1.01 is really very precise, you did it fine. But "after implementing Guillermo's method I am still getting very similar flashies on the lcd preview between the unity wb and a scene wb.  However when developing a linear non-wb tiff I see that I still have maybe a stop of so of headroom until I get clipping" I am a bit surprised at this (and disappointed since it can be my next camera). Can you do some more checking? or diplaying a couple of rear shots of your 5D over the same scene with and without UniWB?

Could you post the RAW file somewhere so I can check? BTW would you mind I publish it in my website? I already have 350D, Panopeeper's 40D, D300 coming and your 5D.

Regards.
Title: Camera's histogram reliable to the RAW data
Post by: Westy on January 12, 2008, 07:10:25 am
Hi Guillermo,  I have just done a bit of checking of my results and in fact all that i have rechecked the uniWB is functioning fine and matches the linear raw quite well.  The uniWB jpgs show just slightly more clipping than the raws.  I will find the one I did where I thought I had a stop or so of headroom and recheck it as well (I can't actually remember which shot it was).  And I've just now realised that I have had my jpeg settings non-neutral.  After I set more neutral rendering parameters I suspect the uni wb jpg will match even closer the linear raw.

I've also rechecked my wb multipliers and they are 1.01,1.02,1.  You can have my custom raw, but at the moment I am limited to dial up as my asdl got zapped in a storm the other day, so it might take me a few days to sort that out.

On the subject of the 5D, I think you can't go wrong with it.  Whether the 40D has better or worse high iso noise performance is debatable, but the difference is likely to be so small as to not concern yourself.  Both will give EXCEPTIONAL high iso noise performance.  Other issues to consider for low light interiors are bigger and brighter view finder of the 5D
Title: Camera's histogram reliable to the RAW data
Post by: Guillermo Luijk on January 12, 2008, 07:24:31 am
You could try some strong RED subject (a rose, coke can,...), and compare camera displays with Daylight WB JPEG, UniWB WB JPEG and RAW (dcraw -r 1 1 1 1).

OK, thanks for the RAW file (when available).
Title: Camera's histogram reliable to the RAW data
Post by: Jonathan Wienke on January 12, 2008, 08:16:52 am
I've been playing around with the UniWB, and have run across some rather surprising things. With my 1D-MkII, when shooting normal "natural" objects that would probably fall within sRGB, the RGB histogram is now pretty good at predicting single-channel RAW clipping. But the histogram can still be quite wrong in some cases. I conducted a test where I photographed some red, green, and blue LED glow lights to see how accurate channel clipping might be for highly saturated colors, and was unable to get the color channel being tested to fill more than 1/4 of the rightmost histogram segment without gross overexposure (>2 stops) where the channel clipping was obvious in the image even on the camera LCD. This behavior was almost identical with both color matrix 1 (Standard sRGB) and 4 (Adobe RGB). So while the UniWB has significantly increased the conditions in which the RGB histogram accurately reflects the RAW data, there are still a few exceptions. I'm going to do more testing and then post detailed results.
Title: Camera's histogram reliable to the RAW data
Post by: Guillermo Luijk on January 12, 2008, 08:29:04 am
Jon, so in your camera histogram and blinking highlights do not match? it seems blinking highlights are more accurate than histogram then.
Since my histogram is monochrome and that means too many blinding variables, I cannot do tests regarding this issue.

BTW would you mind to offer the UniWB for the 1D-MKII so that we can create a list of UniWB RAW files? I could settle it at the beginning of the thread.

Regards.
Title: Camera's histogram reliable to the RAW data
Post by: John Sheehy on January 12, 2008, 09:14:21 am
Quote
I've been playing around with the UniWB, and have run across some rather surprising things. With my 1D-MkII, when shooting normal "natural" objects that would probably fall within sRGB, the RGB histogram is now pretty good at predicting single-channel RAW clipping. But the histogram can still be quite wrong in some cases. I conducted a test where I photographed some red, green, and blue LED glow lights to see how accurate channel clipping might be for highly saturated colors, and was unable to get the color channel being tested to fill more than 1/4 of the rightmost histogram segment without gross overexposure (>2 stops) where the channel clipping was obvious in the image even on the camera LCD. This behavior was almost identical with both color matrix 1 (Standard sRGB) and 4 (Adobe RGB). So while the UniWB has significantly increased the conditions in which the RGB histogram accurately reflects the RAW data, there are still a few exceptions. I'm going to do more testing and then post detailed results.
[a href=\"index.php?act=findpost&pid=166694\"][{POST_SNAPBACK}][/a]

This is exactly what I have been talking about, except with red, orange and yellow  flowers as examples.

You will never get a true RAW histogram out of any RGB conversion.  The colors are altered selectively, the saturations are altered selectively, etc, etc, regardless of the JPEG settings.  It will only reflect the RAW proportionately across the channels for grayscale subjects, and maybe close for weakly saturated ones.
Title: Camera's histogram reliable to the RAW data
Post by: Jonathan Wienke on January 12, 2008, 10:16:12 am
Quote
You will never get a true RAW histogram out of any RGB conversion.  The colors are altered selectively, the saturations are altered selectively, etc, etc, regardless of the JPEG settings.  It will only reflect the RAW proportionately across the channels for grayscale subjects, and maybe close for weakly saturated ones.

That assesment is a bit pessimistic; UniWB not perfect, but certainly is a step in the right direction. Even with the caveats about super-saturated colors, the histogram is much closer to RAW than it is when using any other white balance.
Title: Camera's histogram reliable to the RAW data
Post by: Guillermo Luijk on January 12, 2008, 10:31:49 am
Quote
You will never get a true RAW histogram out of any RGB conversion.  The colors are altered selectively, the saturations are altered selectively, etc, etc, regardless of the JPEG settings.
John, this question could be considered as an offtopic, but it's a long time I wonder this: if we develop a RAW file into some colour space like sRGB or AdobRGB but linearly, i.e. we still don't apply the corresponding gamma (DCRAW does this for instance), is adjusting the exposure as simple as multiplying each channel by the same factor as when we don't convert to any color space? and would this operation alter hue/saturation?

Quote
That assesment is a bit pessimistic; UniWB not perfect, but certainly is a step in the right direction. Even with the caveats about super-saturated colors, the histogram is much closer to RAW than it is when using any other white balance.
I agree. The UniWB is, at least, a good improvement.
Title: Camera's histogram reliable to the RAW data
Post by: John Sheehy on January 12, 2008, 11:43:26 am
Quote
That assesment is a bit pessimistic; UniWB not perfect, but certainly is a step in the right direction. Even with the caveats about super-saturated colors, the histogram is much closer to RAW than it is when using any other white balance.
[a href=\"index.php?act=findpost&pid=166708\"][{POST_SNAPBACK}][/a]

Yes.  I was shooting flat magenta images a few years back, and started using it for custom WB, but what I found was that it was most accurate when the highlights were white, and guess what?  The white highlights are just as accurate with or without uniWB.  Theoretically, you might expect better ETTR of a red or blue highlight with uniWB, now that cameras have RGB histograms, but this is exactly where the color conversion messes things up.  It isn't worth the hassle to me, to look at a cyan-green preview and JPEGs, for gains which were hard to realize.

What I really want is a camera that shows the WB'ed sRGB image in the preview (at least as an option, against a linear RAW), but flashes (red, green, blue, cyan, magenta, yellow or white alternating with black) clipped highlights based on the RAW, and shows a pure RAW histogram.
Title: Camera's histogram reliable to the RAW data
Post by: John Sheehy on January 12, 2008, 11:47:04 am
Quote
John, this question could be considered as an offtopic, but it's a long time I wonder this: if we develop a RAW file into some colour space like sRGB or AdobRGB but linearly, i.e. we still don't apply the corresponding gamma (DCRAW does this for instance), is adjusting the exposure as simple as multiplying each channel by the same factor as when we don't convert to any color space? and would this operation alter hue/saturation?
[a href=\"index.php?act=findpost&pid=166716\"][{POST_SNAPBACK}][/a]

Hue and saturation are just ways of looking at R:G:B ratios.  If the data is linear, then simply scaling them (with blackpoint at zero, of course) does not change hue or saturation, except as an unscaled curve might be applied to them at conversion.
Title: Camera's histogram reliable to the RAW data
Post by: Panopeeper on January 12, 2008, 12:57:41 pm
Quote
I conducted a test where I photographed some red, green, and blue LED glow lights to see how accurate channel clipping might be for highly saturated colors, and was unable to get the color channel being tested to fill more than 1/4 of the rightmost histogram segment without gross overexposure (>2 stops) where the channel clipping was obvious in the image even on the camera LCD

I think you stretched the concept over the limit. We can practicall turn off white balancing, we can turn off contrast and saturation, but we can't turn off the de-mosaicing.

The LED light is practically single wavelength; as such, it passes one (or two) filters to a high degree, while the other (or others) let only a relative small portion through, although it appears measurably in all three colors, see spectral response. Therefor you can clip a channel with a fraction of the values in the other channels. However, the de-mosaicing will "distribute" the pixel values between the neighbours.

You mentioned "highly saturated". Led and laser lights are highly saturated, for sure - but highly saturated what? There are many different color within for example the range of "red" in the spectrum, but there is only one saturated red in the color space.

Consequently the highly saturated color you used may appear as a composite color from the point of de-mosaicing.
Title: Camera's histogram reliable to the RAW data
Post by: Jonathan Wienke on January 12, 2008, 01:25:22 pm
Quote
What I really want is a camera that shows the WB'ed sRGB image in the preview (at least as an option, against a linear RAW), but flashes (red, green, blue, cyan, magenta, yellow or white alternating with black) clipped highlights based on the RAW, and shows a pure RAW histogram.

I agree that this would be far preferable to what we have now. But for colors that fall within sRGB, UniWB delivers a closer correlation between camera histogram and RAW than any other alternative.
Title: Camera's histogram reliable to the RAW data
Post by: Jonathan Wienke on January 12, 2008, 01:31:15 pm
Quote
I think you stretched the concept over the limit. We can practicall turn off white balancing, we can turn off contrast and saturation, but we can't turn off the de-mosaicing.

The LED light is practically single wavelength; as such, it passes one (or two) filters to a high degree, while the other (or others) let only a relative small portion through, although it appears measurably in all three colors, see spectral response.

I agree this is an extreme case; that's why I tried it--to see how well UniWB worked in extreme circumstances. I don't think that this invalidates the usefulness of UniWB in most circumstances, though.
Title: Camera's histogram reliable to the RAW data
Post by: John Sheehy on January 12, 2008, 01:31:39 pm
Quote
I think you stretched the concept over the limit. We can practicall turn off white balancing, we can turn off contrast and saturation, but we can't turn off the de-mosaicing.
[a href=\"index.php?act=findpost&pid=166745\"][{POST_SNAPBACK}][/a]

I don't see how demosaicing would be relevant.  You would get the same relatively saturated capture if R, G, and B were co-located.
Title: Camera's histogram reliable to the RAW data
Post by: Panopeeper on January 12, 2008, 02:08:32 pm
Quote
I have to disagree with you: DOF DOES depend on the cropping, i.e. on the sensor format

I stated, that DoF depends on the lens, but not on the cropping; this is a fact. It is not useful to mix up concepts.

One can bring the sensel size in the equation as well; that again has nothing to do with cropping. I had similar discussion with paper tigers, whose measure of image quality is how large an image can be printed. They will debate endlessly over side-issues, like sensel size, number of pixels, etc.

One needs to see the issues more abstract, otherwise we can not conduct a factual discussion. Another such fruitless discussion was, that some people stated that the perspective depends on focal length and cropping. Plain BS. You can state, that with a different focal length you have to go to a different distance. Right, that changes the perspective, not the focal length.

Re panos in architectural photographing: it is unquestionable, that you can achieve a very good result with a wide angle lens or tilt and shift, but do you have such wide lens?

http://www.panopeeper.com/panorama/GameRoom1.jpg (http://www.panopeeper.com/panorama/GameRoom1.jpg)

Quote
Panos is one of my pending tasks for some day. What software do you recommend me? PT?

There are many stitchers for casual panomakers. For those, who are serious about it, there is only one: PT (and its descendants, working on the very same principle). However, PT is not for human consumption, you have to have a good user interface. I prefer Panorama Tools Assembler, others prefer PTGui, and there is Hugin, but not on all platforms.
Title: Camera's histogram reliable to the RAW data
Post by: Panopeeper on January 12, 2008, 02:23:04 pm
Quote
I don't see how demosaicing would be relevant.  You would get the same relatively saturated capture if R, G, and B were co-located.

You certainly would not get the same, because the colors of a single tri-color pixel would have to undergo "only" a color space transformation, while the de-mosaicing has to work with several neighbouring pixels, where the distance between them is relevant too (and the result undergoes the color space conversion).

Nevertheless, there is no saturated color on raw level. You can not find any wavelength, which would be totally filtered by two channels. Theoretically, one wavelength per filter would be acceptable, but this irrelevant, and anyway there are no such filters.

So, you can have many different color compositions from different saturated colors (single wavelengths),  which have to be transformed in different RGBs; only one of them may be transformed in a fully saturated color.
Title: Camera's histogram reliable to the RAW data
Post by: bjanes on January 12, 2008, 04:24:21 pm
Quote
You certainly would not get the same, because the colors of a single tri-color pixel would have to undergo "only" a color space transformation, while the de-mosaicing has to work with several neighbouring pixels, where the distance between them is relevant too (and the result undergoes the color space conversion).

Nevertheless, there is no saturated color on raw level. You can not find any wavelength, which would be totally filtered by two channels. Theoretically, one wavelength per filter would be acceptable, but this irrelevant, and anyway there are no such filters.

So, you can have many different color compositions from different saturated colors (single wavelengths),  which have to be transformed in different RGBs; only one of them may be transformed in a fully saturated color.
[{POST_SNAPBACK}][/a] (http://index.php?act=findpost&pid=166769\")

You guys are getting way off base here when you talk about red lasers, LEDs, and the highly saturated or monochromatic colors produced by these devices. It is not important to capture such colors since they do not occur in nature (or only very rarely) and can not be reproduced on any current display or printed. What is important for photography are the real world surface colors, which are discussed [a href=\"http://www.colour.org/tc8-05/MetricsUpdateNov01.pdf]here.[/url]
Title: Camera's histogram reliable to the RAW data
Post by: Jonathan Wienke on January 12, 2008, 04:34:03 pm
Quote
You guys are getting way off base here when you talk about red lasers, LEDs, and the highly saturated or monochromatic colors produced by these devices. It is not important to capture such colors since they do not occur in nature (or only very rarely) and can not be reproduced on any current display or printed.

Have you ever shot at a dance club, or concert? Or a car show? Highly saturated colors outside Adobe RGB aren't that uncommon. Not natural, perhaps, but that doesn't mean you'll never find them in a photograph. Knowing what to do when you do is of more than merely academic interest.
Title: Camera's histogram reliable to the RAW data
Post by: bjanes on January 12, 2008, 05:12:33 pm
Quote
Have you ever shot at a dance club, or concert? Or a car show? Highly saturated colors outside Adobe RGB aren't that uncommon. Not natural, perhaps, but that doesn't mean you'll never find them in a photograph. Knowing what to do when you do is of more than merely academic interest.
[a href=\"index.php?act=findpost&pid=166791\"][{POST_SNAPBACK}][/a]

This has been discussed many times. If you want to capture all real world surface colors, you should render into ProPhotoRBG. Also, saturation clipping is easily seen in the ACR histogram, and alerts one to use a wider color space.

Here is a plot from Gernot Hoffmann's web site showing the real world surface colors (dotted lines) along with sRGB, aRGB, and ProPhotoRGB gamuts.

(http://bjanes.smugmug.com/photos/237613131-O.png)
Title: Camera's histogram reliable to the RAW data
Post by: Panopeeper on January 12, 2008, 05:15:55 pm
I find it generally useful to perform abstract, in practice useless tests - in thought or in real - in order to understand the behaviour and the underlying principles of some phenomenon.

One could say it is useless to shoot brick walls and newspapers instead of looking at the nice picture of the favourite cat (usually a crop in 25%), as proof for the high quality of a lens. I am on the other side. I am shooting abstact and/or uninteresting subjects and peeping the non-demosaiced images, when I judge a lense (the test of bokeh is an exemption).
Title: Camera's histogram reliable to the RAW data
Post by: John Sheehy on January 12, 2008, 06:35:22 pm
Quote
You certainly would not get the same, because the colors of a single tri-color pixel would have to undergo "only" a color space transformation, while the de-mosaicing has to work with several neighbouring pixels, where the distance between them is relevant too (and the result undergoes the color space conversion).

That's still not relevant.  That's only a spatial issue (resolution), not a color issue.

Quote
Nevertheless, there is no saturated color on raw level.

Sure there is.  Have you ever shot red or blue LEDs?  You can have one color channel almost clipping, while another is down near the noise floor; the green response is weak but significant.  I have seen shadows cast from an LED in the "right" color on bulbs, viewed in the "wrong" RAW channel.  Looks like they aren't even on at all, and you have to take a shot with just that one bulb on to see it in the "wrong" channel, buried down in the noise.  But no one claimed 100% saturation.

Quote
You can not find any wavelength, which would be totally filtered by two channels. Theoretically, one wavelength per filter would be acceptable, but this irrelevant, and anyway there are no such filters.

Who said two channels?  Only one has to mostly miss it for it to be considered saturated.  How do you record a yellow or violet or cyan laser?  How do you display it?  You sometimes must depend on two colors for recording and displaying saturation (not ideal, of course).  Saturation is emulated by dropping the weaker channel more than it is really recorded in RAW, while applying a push into the ceiling" curve to the other two colors.

I have a deep blue 77mm filter that drops everything on the red side of green down into the noise.  I have a "pad" of gel samples, some of which, when looking at a spectrum through a diffraction grating, look black in certain ranges.

Quote
So, you can have many different color compositions from different saturated colors (single wavelengths),  which have to be transformed in different RGBs; only one of them may be transformed in a fully saturated color.
[a href=\"index.php?act=findpost&pid=166769\"][{POST_SNAPBACK}][/a]

Could you rephrase that, and make your point clearer.
Title: Camera's histogram reliable to the RAW data
Post by: bernie west on January 12, 2008, 06:39:36 pm
Quote
Panopeeper Posted Today, 05:08 AM
I stated, that DoF depends on the lens, but not on the cropping; this is a fact. It is not useful to mix up concepts.

True, but that is not all it depends on.  Any comparison discussion about DOF between formats must be concerned with equivalent FOV, or else it's pointless.  The question should be - can I essentially capture the same image on two different formats.  In the case of extremes of DOF range, the answer is often no.
Title: Camera's histogram reliable to the RAW data
Post by: Guillermo Luijk on January 12, 2008, 07:16:32 pm
Quote
I stated, that DoF depends on the lens, but not on the cropping; this is a fact. It is not useful to mix up concepts.
Sensor format and lens have to be mixed to find out which sensor can provide more bokeh for a given lens.
In more understandable terms: given a lens with a maximum aperture (e.g. a 85mm f1.8) and shooting over the same subject, a 5D will achieve a narrower DoF (i.e. more bokeh) than a 40D, because the 5D will allow us to get closer to the subject to obtain the same FoV over the subject, then reducing DoF.
This is what I meant.

The same story applies in wide angle lenses: wide angles lenses do not distort perspective since perspective does not depend on the lens but on the distance to the subject. But wide angle lenses allow us to get closer to the subject, and it is then when perspective changes.
Title: Camera's histogram reliable to the RAW data
Post by: Panopeeper on January 12, 2008, 08:32:58 pm
Quote
Sensor format and lens have to be mixed to find out which sensor can provide more bokeh for a given lens.

The ability to abstract details is essential for the understanding of issues.
Title: Camera's histogram reliable to the RAW data
Post by: Panopeeper on January 12, 2008, 09:04:48 pm
Quote
That's still not relevant.  That's only a spatial issue (resolution), not a color issue
The spatial distribution is not "only" when evaluating the color, but irrelevant in the current context.

Quote
Have you ever shot red or blue LEDs?  You can have one color channel almost clipping, while another is down near the noise floor; the green response is weak but significant
I shot red laser light. The blue was very low, but the green was considerable, about 25% of the red. Of course, this depends on the wavelength.

Quote
Who said two channels?  Only one has to mostly miss it for it to be considered saturated

1. In the RGB model the yellow is a composite color. There is no important difference between yellow and any other color.

2. *All colors* appear in *at least two kinds of pixels* to a meaningful degree. This fact makes it possible to differentiate between colors of the same "range", for example between red colors; otherwise the Bayer sensor would not work with multicolor.
Title: Camera's histogram reliable to the RAW data
Post by: John Sheehy on January 13, 2008, 12:30:41 am
Quote
The spatial distribution is not "only" when evaluating the color, but irrelevant in the current context.
I shot red laser light. The blue was very low, but the green was considerable, about 25% of the red. Of course, this depends on the wavelength.
1. In the RGB model the yellow is a composite color. There is no important difference between yellow and any other color.

2. *All colors* appear in *at least two kinds of pixels* to a meaningful degree. This fact makes it possible to differentiate between colors of the same "range", for example between red colors; otherwise the Bayer sensor would not work with multicolor.
[a href=\"index.php?act=findpost&pid=166833\"][{POST_SNAPBACK}][/a]

Why are you telling me all this?

Most of what you seem to be saying here is what I have already said, yet you seem to be disagreeing with me.

I have serious difficulty following a conversation with you.
Title: Camera's histogram reliable to the RAW data
Post by: Panopeeper on January 13, 2008, 12:55:48 am
Quote
Why are you telling me all this?

I have serious difficulty following a conversation with you.

Let's recapitulate it. I posted earlier:

Quote
I think you stretched the concept over the limit. We can practicall turn off white balancing, we can turn off contrast and saturation, but we can't turn off the de-mosaicing

and your opinion was, that this played no role in the phenomenon Jonathan observed:

Quote
I don't see how demosaicing would be relevant. You would get the same relatively saturated capture if R, G, and B were co-located

All this time I tried to explain you, that the de-mosaicing does change pure colors in composite ones, and thus it causes the contradiction between the raw and RGB histograms.

Another issue is, that even with tri-colored pixels a conversion would have to occur with like effect (i.e. converting pure colors in composite ones), but this does not really matter, for the sensor in the experiment is a Bayer one..

Anyway, I stand by my assertion, that the de-mosaicing is the cause of the phenomenon Jonathan observed with the LED lights.
Title: Camera's histogram reliable to the RAW data
Post by: Jonathan Wienke on January 13, 2008, 09:07:56 am
Quote
All this time I tried to explain you, that the de-mosaicing does change pure colors in composite ones, and thus it causes the contradiction between the raw and RGB histograms.

De-mosaicing has nothing to do with it. A Foveon sensor would behave similarly, with a single-wavelength light source such as an LED or laser stimulating a response in more than one channel. Converting pure colors to composite colors is a product of the frequency response of the color filter array, pure and simple. You can observe this in the RAW data before demosaicing occurs:

(http://www.visual-vacations.com/images/RedLEDRAWHisto.gif)

This histogram is from one of my red LED test photos.
Title: Camera's histogram reliable to the RAW data
Post by: John Sheehy on January 13, 2008, 09:36:14 am
Quote
Anyway, I stand by my assertion, that the de-mosaicing is the cause of the phenomenon Jonathan observed with the LED lights.
[a href=\"index.php?act=findpost&pid=166860\"][{POST_SNAPBACK}][/a]

Then you are out on a limb, IMO.  The only difference of a bayer CFA here is one of spatial resolution.  There is no color-related magic in co-location.  The only reason for demosaicing over straight, independent interpolation of each isolated color plane, is that you get to abstract per-pixel luminance.  If you wanted a 3MP conversion of a 12MP sensor, you wouldn't even need to demosaic; it would serve no purpose.  You would just interpolate each color plane (maintaining its offset), and then downsample or bin to 3 MP.
Title: Camera's histogram reliable to the RAW data
Post by: Jonathan Wienke on January 13, 2008, 10:41:33 am
FWIW, I've posted my 1Ds and 1D-Mark II UniWB RAWS online at:

http://www.visual-vacations.com/images/2008/ (http://www.visual-vacations.com/images/2008/)
Title: Camera's histogram reliable to the RAW data
Post by: Panopeeper on January 13, 2008, 02:40:39 pm
Quote
De-mosaicing has nothing to do with it. A Foveon sensor would behave similarly, with a single-wavelength light source such as an LED or laser stimulating a response in more than one channel
I stated just above, that this would occur with tri-colored pixels as well. However, creating composite colors from Bayer pixels is part of the de-mosaicing process.
Title: Camera's histogram reliable to the RAW data
Post by: Panopeeper on January 13, 2008, 05:02:40 pm
Quote
I normally do HDR (not the tone mapping, just getting the HDR data) from 2 shots: 0EV, +4EV. To obtain them I bracket -2,0,2, so that the -2 matches my desired 0EV (i.e. ETTR shot without blowing any highlight) and +2 give me the +4V sample. I use MLU and images match pixel by pixel even with my cheap tripod, it's very precise. Of course avoiding mirror flip could even be better but not necessary actually

Guillermo,

I uploaded two pairs of images, both 4 EV apart. They were shot with exposure bracketing, using a wired remote control, 2sec time delay; the tripod was on carpit. One pair shot with MLU, the other with MLU + live view.

http://www.panopeeper.com/Demo/MLU-2EV.CR2 (http://www.panopeeper.com/Demo/MLU-2EV.CR2)
http://www.panopeeper.com/Demo/MLU+2EV.CR2 (http://www.panopeeper.com/Demo/MLU+2EV.CR2)
http://www.panopeeper.com/Demo/LiveView-2EV.CR2 (http://www.panopeeper.com/Demo/LiveView-2EV.CR2)
http://www.panopeeper.com/Demo/LiveView+2EV.CR2 (http://www.panopeeper.com/Demo/LiveView+2EV.CR2)

The roof of the house can be taken for comparison, the leaves not, because of a slight breeze.

I wonder if you achieved better matching images with MLU than I did.
Title: Camera's histogram reliable to the RAW data
Post by: Ray on January 13, 2008, 11:43:38 pm
Quote
A final remark: the dynamic range and lesser noise of the 5D is a myth, compared to the 40D. There was a thread about this question. The 5D outputs about 3570 levels; in the nineth stop there are only seven levels.
[a href=\"index.php?act=findpost&pid=166616\"][{POST_SNAPBACK}][/a]

But the 5D has over 20% more pixels than the 40D. 3570x12.8/10.1=4524.

It's not clear to me what role the extra 2 bits of the 40D plays, but I recall John Sheehy mentioning that they don't appear to be producing any more real levels than 12 bits would produce.
Title: Camera's histogram reliable to the RAW data
Post by: Panopeeper on January 14, 2008, 01:12:36 am
Quote
It's not clear to me what role the extra 2 bits of the 40D plays, but I recall John Sheehy mentioning that they don't appear to be producing any more real levels than 12 bits would produce.

It is not reasonable to reduce this question to the number of bits; it's not so simple.

The 5D creates about ~3570 levels. The 20D creates ~3970 levels. The 40D creates ~12800 levels at ISO 100, and ~15200 levels at higher, full stop ISOs.

This is about 3.5 times more at ISO 100 than that of the 5D. It is open to debate, how many of the 12800 levels of the 40D are really informative, but IMO it is clear, that the 5D is far underequipped with the 3570 levels.

On the practical side: I have no problem with carrying around lots of uninformative levels, they don't disturb me the least; however, the lack of levels in the shadows is an impediment. I would not exchange my 40D for a 5D based on image quality (based on other considerations even less).

Though it would be interesting to see a detailed, pixel-peeping comparison between the 40D and 5D, with the very same lens(es).
Title: Camera's histogram reliable to the RAW data
Post by: Guillermo Luijk on January 14, 2008, 03:12:03 am
Quote
I wonder if you achieved better matching images with MLU than I did.

Probably no better, and no worse hehe. I will check that anyway.
When I have compared shots put in PS layers, the location of scene's elements was indistinguishable pixel by pixel.
Title: Camera's histogram reliable to the RAW data
Post by: Guillermo Luijk on January 14, 2008, 08:13:56 am
Quote
It is not reasonable to reduce this question to the number of bits; it's not so simple.

The 5D creates about ~3570 levels. The 20D creates ~3970 levels. The 40D creates ~12800 levels at ISO 100, and ~15200 levels at higher, full stop ISOs.

This is about 3.5 times more at ISO 100 than that of the 5D. It is open to debate, how many of the 12800 levels of the 40D are really informative, but IMO it is clear, that the 5D is far underequipped with the 3570 levels.

On the practical side: I have no problem with carrying around lots of uninformative levels, they don't disturb me the least; however, the lack of levels in the shadows is an impediment. I would not exchange my 40D for a 5D based on image quality (based on other considerations even less).

Pano, you are talking about captured levels, but don't forget that interpolated levels are 16-bit on both machines, and photographs are made both from captured and interpolated levels.

Of course, if 40D levels are more precise than 5D's levels (4 times more precise to simplify), 40D's interpolated levels will also be more precise, so we could with no doubt say that 40D's images are more exact in defining the right level values.
But regarding tonal richness, after developing what you will have is:
- 40D: pixels with 1 channel taking 14 bit values, and 2 channels taking 16 bit values.
- 5D: pixels with 1 channel taking 12 bit values, and 2 channels taking 16 bit values.

Tonal precision in 40D is higher, but tonal richness (number of total different levels achieved) can be considered the same on both machines, and thus danger of posterization or banding due to lack of levels.
Moreover in the areas where the increased tonal precision could be really enjoyed (lowest f-stops) noise makes the image unusable so 40D cannot really take advantage of its increased no. of encoding bits.

In fact Leica's M8 RAW files are 8-bit. Yes, they are very cleverly distributed in a non-linear way, but they are 8 bit after all and even in the shadows are les precise (there is more gap between each pair of captured encoded levels) than 12-bit linear. Would you say Leica's M8 enconding produces noticeably lower quality images than any 12-bit linear camera?

I would exchange a 40D for a 5D if my priorities were strong bokeh and wide angle for example. In other words, FF.
Title: Camera's histogram reliable to the RAW data
Post by: Panopeeper on January 14, 2008, 09:47:02 pm
Quote
When I have compared shots put in PS layers, the location of scene's elements was indistinguishable pixel by pixel

Do you mind posting a pair of such shots?

My aim is to achieve raw pixel by pixel coverage; what you see are de-mosaiced pixels.

Quote
Tonal precision in 40D is higher, but tonal richness (number of total different levels achieved) can be considered the same on both machines, and thus danger of posterization or banding due to lack of levels

The number of tones in the resulting image is a non-issue. However, the de-mosaicing can create tones, but it can not create true details. If two pixels of the sensor can not differentiate between two slightly different tones of the subject, then that is lost, no matter how many tones you can fae by the de-mosaicing.

Quote
Moreover in the areas where the increased tonal precision could be really enjoyed (lowest f-stops) noise makes the image unusable so 40D cannot really take advantage of its increased no. of encoding bits

According to DCReview, the DR of the 40D is 9.1 stops in ISO 100 and ISO 200 (IMO this is incorrect, it is somewhat higher in ISO 200), and that of the 5D is 8.2 stops at ISO 100. The noise may be less with the 5D, but I doubt the correctness of those evaluations.

A higher dynamic range means, that the 40D does *need* more levels, and it can utilize them better.

Anyway, I would like to see a DR comparison with Jonathan's mehod, because the DPReview test regards only the noise, not the details.

Quote
Would you say Leica's M8 enconding produces noticeably lower quality images than any 12-bit linear camera?

Conditionally, yes. The M8 produces a crippled raw; it is good for nothing but change the white balance. When you need strong adjustments, the limits of the 8 bit will come forward.

Why do you think Leica's digital back has ***16*** bit depth? Is the digital back 256 times better than the M8?

Quote
exchange a 40D for a 5D if my priorities were strong bokeh and wide angle for example. In other words, FF.

Forget about the bokeh. No camera will make a lens suitable for good bokeh, no matter of the DoF. I don't know, what "strong" bokeh is, but if you like nice bokeh, you need a suitable lens, and that will not be an F4 lens.
Title: Camera's histogram reliable to the RAW data
Post by: bjanes on January 14, 2008, 10:13:34 pm
Quote
The number of tones in the resulting image is a non-issue. However, the de-mosaicing can create tones, but it can not create true details. If two pixels of the sensor can not differentiate between two slightly different tones of the subject, then that is lost, no matter how many tones you can fae by the de-mosaicing.

[a href=\"index.php?act=findpost&pid=167207\"][{POST_SNAPBACK}][/a]

The above statements are contradictory. Tones are a non-issue. Then by implication, an insufficient number of tones will limit rendering of detail. What did you mean to say?

Actually bit depth and resolution are not highly correlated. There is often little difference in resolution  between an 8 bit JPEG and a raw file rendered into 16 bits.
Title: Camera's histogram reliable to the RAW data
Post by: Panopeeper on January 15, 2008, 01:21:37 am
Quote
The above statements are contradictory. Tones are a non-issue. Then by implication, an insufficient number of tones will limit rendering of detail. What did you mean to say?

Gullermo wrote:

regarding tonal richness, after developing what you will have is:
- 40D: pixels with 1 channel taking 14 bit values, and 2 channels taking 16 bit values.
- 5D: pixels with 1 channel taking 12 bit values, and 2 channels taking 16 bit values


We are talking about two kinds of levels: those sensed and those "created". Already the 256x256x256 tones are plenty in most cases, so the issue is not if the number of combinations from 3x12 bits are enough.

Any number of resulting levels does not help, when the levels of a channel are not enough to distinguish between details.

Quote
Actually bit depth and resolution are not highly correlated. There is often little difference in resolution  between an 8 bit JPEG and a raw file rendered into 16 bits.

I did not mention resolution. Different areas of an image may be indistinguishable because of lack of resolution OR because of lack of tonal levels.

An abstract example: half of the scenery consists of something of a constant color, the other half too, but slightly different color. If they are not different enough to be "seen as different" by the sensels, then the two halves appear identical - this has nothing to do with resolution.
Title: Camera's histogram reliable to the RAW data
Post by: Ray on January 15, 2008, 02:23:26 am
Quote
An abstract example: half of the scenery consists of something of a constant color, the other half too, but slightly different color. If they are not different enough to be "seen as different" by the sensels, then the two halves appear identical - this has nothing to do with resolution.
[a href=\"index.php?act=findpost&pid=167252\"][{POST_SNAPBACK}][/a]

Interesting example. Would this be possible to demonstrate visually? I can appreciate that from a purely technical point of view, one could have the two halves of an image a very slightly different hue so that in order to technically (at the machine level) distinguish between them, you might need 14 bit or 16 bit capture instead of 12 bit, but would the eye be able to distinguish between such subtle differences that required this huge increase in the number of levels afforded by 14 or 16 bit processing?

We already know, for example, that the last stop of an ETTR exposure, in 12 bit, produces a far greater number of levels than the eye can detect, and maybe the penultimate stop too.
Title: Camera's histogram reliable to the RAW data
Post by: Guillermo Luijk on January 15, 2008, 08:18:21 am
Quote
Forget about the bokeh. No camera will make a lens suitable for good bokeh, no matter of the DoF. I don't know, what "strong" bokeh is, but if you like nice bokeh, you need a suitable lens, and that will not be an F4 lens.

I cannot Panopeeper, I see clear that the same lens capturing the same scene (same FoV over the subject) will provide at maximum aperture a shorter DoF in the 5D than in the 40D. This is what I call, maybe wrongly, strong bokeh. And it's what I am looking for in FF (apart from wide angle), a higher capability of differentiating the subject from the background/foreground.

I agree with the rest of your post although I think none of the statements will mean a real and noticeably improvement on 40D's images. 9 f-stops is simply not enough to really NEED extra bits, 12-bit cameras have showed to be able to capture with a reasonable good definition 9 f-stops (the Sony A700 for instance).
I haven't done tests over the 5D's DR at ISO100, but I assume it is lower than 40D's, so in the 5D the extra bits are even less necessary.
For a good DR in high contrast scenes, both machines need extra exposures. Any possible advantage of 14 bits is definitively gone in these circumstances.

I will try to post a couple of overlapping shoots tonight. Didn't have time to check yours sorry.
Title: Camera's histogram reliable to the RAW data
Post by: John Sheehy on January 15, 2008, 08:35:21 am
Quote
It is not reasonable to reduce this question to the number of bits; it's not so simple.

The 5D creates about ~3570 levels. The 20D creates ~3970 levels. The 40D creates ~12800 levels at ISO 100, and ~15200 levels at higher, full stop ISOs.

This is about 3.5 times more at ISO 100 than that of the 5D. It is open to debate, how many of the 12800 levels of the 40D are really informative, but IMO it is clear, that the 5D is far underequipped with the 3570 levels.
[a href=\"index.php?act=findpost&pid=167015\"][{POST_SNAPBACK}][/a]


The 5D needs about 2500 levels at ISO 100 and 1090 levels at ISO 1600 for normal (non-stacking) use.  That's for the RAW data; for conversion, of course, it is always good to force extra precision if the converter loads RAW data aligned by the LSB.
Any bit depth (or number of levels) that brings the standard deviation of a black frame above 1.4 ADU is inefficient for storage purposes.
Title: Camera's histogram reliable to the RAW data
Post by: Jonathan Wienke on January 15, 2008, 04:01:13 pm
Quote
Interesting example. Would this be possible to demonstrate visually? I can appreciate that from a purely technical point of view, one could have the two halves of an image a very slightly different hue so that in order to technically (at the machine level) distinguish between them, you might need 14 bit or 16 bit capture instead of 12 bit, but would the eye be able to distinguish between such subtle differences that required this huge increase in the number of levels afforded by 14 or 16 bit processing?

The level count can cause low-contrast detail to disappear. Try loading my DR test chart in PS and reducing the number of levels and see how many you can throw out before the smallest, lowest-contrast text starts disappearing.
Title: Camera's histogram reliable to the RAW data
Post by: John Sheehy on January 15, 2008, 08:33:16 pm
Quote
The level count can cause low-contrast detail to disappear. Try loading my DR test chart in PS and reducing the number of levels and see how many you can throw out before the smallest, lowest-contrast text starts disappearing.
[a href=\"index.php?act=findpost&pid=167396\"][{POST_SNAPBACK}][/a]

Such a test would be totally irrelevant, if the decrease in levels weren't in the original RAW data.  Conversions deteriorate much more quickly from quantization.
Title: Camera's histogram reliable to the RAW data
Post by: Jonathan Wienke on January 16, 2008, 08:05:32 am
Quote
Such a test would be totally irrelevant, if the decrease in levels weren't in the original RAW data.

The point of such a test would be to get a rough idea of how many levels per stop are needed to avoid visually apparent banding/posterization and the disappearance of low-contrast fine detail.
Title: Camera's histogram reliable to the RAW data
Post by: John Sheehy on January 16, 2008, 08:32:18 am
Quote
The point of such a test would be to get a rough idea of how many levels per stop are needed to avoid visually apparent banding/posterization and the disappearance of low-contrast fine detail.
[a href=\"index.php?act=findpost&pid=167516\"][{POST_SNAPBACK}][/a]

But of what?  You replied to Ray, who was talking about RAW bit depth, and consequently, RAW levels.  Quantizing them is not the same thing as quantizing finished conversions.  RAW data is far more quantization-resistant than full RGB images with realistic tone curves.
Title: Camera's histogram reliable to the RAW data
Post by: Jonathan Wienke on January 16, 2008, 08:58:13 am
Quote
RAW data is far more quantization-resistant than full RGB images with realistic tone curves.

WTF??? In the highlights, yes, but certainly not in the shadows. Try zeroing out all the lower-order bits of a RAW file so that it is effectively 8-bit, and run it through any RAW converter. Quantization will be much worse than if you convert to 8-bit after conversion especially in the shadows.
Title: Camera's histogram reliable to the RAW data
Post by: Panopeeper on January 16, 2008, 03:09:09 pm
Quote
I can appreciate that from a purely technical point of view, one could have the two halves of an image a very slightly different hue so that in order to technically (at the machine level) distinguish between them, you might need 14 bit or 16 bit capture instead of 12 bit, but would the eye be able to distinguish between such subtle differences that required this huge increase in the number of levels afforded by 14 or 16 bit processing?
It depends on the absolute luminousity of these values as well. The minimum difference between two luminousities (in proportion, not in absolute value) depends on the range these luminousities are in (in absolute values). See the paper James has linked to a few posts above.
Title: Camera's histogram reliable to the RAW data
Post by: Panopeeper on January 16, 2008, 03:20:24 pm
Quote
I see clear that the same lens capturing the same scene (same FoV over the subject) will provide at maximum aperture a shorter DoF in the 5D than in the 40D. This is what I call, maybe wrongly, strong bokeh

Well, "strong bokeh" is not a generally used term, at least I don't know that, so you can use it in whatever sense you want to. However, "nice bokeh" is a well-known term, and even though it is not well defined, most people are in agreement in the judgement of bokehs.

The "quality" of bokeh depends primarily on the lens; it is a special quality of the lens. Larger DoF does not guarantee nicer bokeh. For example the Canon 80-200mm f/2.8L shot in my bokeh collection shows, that that lens is not a "bokeh lens". The 50mm f/1.4 yields a medium good bokeeh only, no matter if at f/1.4 or f/2.8.

The reasons for creating a nice or bad bokeh are often discussed and guessed, but as far as I see it, this is rather shamanry. I read already, that some of the MTF curves indicate the bokeh quality, but I don't see, why. However, the number and shape of the aperture blades appear to be important, except with the maximum aperture, I guess.
Title: Camera's histogram reliable to the RAW data
Post by: John Sheehy on January 16, 2008, 03:26:05 pm
Quote
WTF??? In the highlights, yes, but certainly not in the shadows. Try zeroing out all the lower-order bits of a RAW file so that it is effectively 8-bit,

All zeros is not the proper way to do it, unless you're also going to move the blackpoint down to compensate.  Using "1000" (for 12-bit) is what you need to replace with.

Quote
and run it through any RAW converter. Quantization will be much worse than if you convert to 8-bit after conversion especially in the shadows.
[a href=\"index.php?act=findpost&pid=167533\"][{POST_SNAPBACK}][/a]

I was thinking more along the lines of 16 vs 14 vs 12 etc.  If you go to 8 significant bits, you're going to see the quantization (especially with low ISOs).
Title: Camera's histogram reliable to the RAW data
Post by: Panopeeper on January 16, 2008, 03:31:46 pm
Quote
The 5D needs about 2500 levels at ISO 100 and 1090 levels at ISO 1600 for normal (non-stacking) use.  That's for the RAW data; for conversion, of course, it is always good to force extra precision if the converter loads RAW data aligned by the LSB

With only 2500 levels, quite a few of the 256 RGB levels would be wasted.

Quote
Any bit depth (or number of levels) that brings the standard deviation of a black frame above 1.4 ADU is inefficient for storage purposes.

I wonder how you calculate this, specifically for the 5D.
Title: Camera's histogram reliable to the RAW data
Post by: Panopeeper on January 16, 2008, 03:36:14 pm
Quote
Try zeroing out all the lower-order bits of a RAW file so that it is effectively 8-bit, and run it through any RAW converter. Quantization will be much worse than if you convert to 8-bit after conversion especially in the shadows.

If you read The Dialog (http://www.cryptobola.com/PhotoBola/Dialog.htm), the first and largest section of the new Rawnalyze manual, you find how to zero out some bits (chapter Changing the original raw data). However, shifting out bits instead of zeroing is better with Canon cameras, because of the black level (explanation inside).
Title: Camera's histogram reliable to the RAW data
Post by: John Sheehy on January 16, 2008, 04:44:52 pm
Quote
With only 2500 levels, quite a few of the 256 RGB levels would be wasted.

About 1000 of the ~3590 levels currently used are already wasted by read noise.

Quote
I wonder how you calculate this, specifically for the 5D.
[a href=\"index.php?act=findpost&pid=167618\"][{POST_SNAPBACK}][/a]

"1.4 ADU" is not specific to the 5D.

It is a general figure based upon my experiments with quantizing RAW data to various bit depths, and observing at what read noise level, in ADUs, is the number of levels sufficient.  Anything resulting in much less than about 1.4 ADUs starts to get quantized visibly, and anything above 1.4 ADU looks the same as anything around 1.4 ADU.

I'm using 1.4 as a generous safety margin; 1.25 works quite well, as can be seen in the cameras that have 14 bits @ about 5 ADU.
Title: Camera's histogram reliable to the RAW data
Post by: Panopeeper on January 16, 2008, 05:21:32 pm
Quote
About 1000 of the ~3590 levels currently used are already wasted by read noise

This does not depend on noise, nor on DR. It depends on the transform function and on the target range. With sRGB this is not a real issue, but with Adobe RGB 98 one would need much more (over 10000) levels in order to utilize all 256 target levels (and where is that from 10-bit printers and monitors?).

Quote
1.4 ADU" is not specific to the 5D

I know. I asked for a specific example of your calculation, with raw values and noise levels.
Title: Camera's histogram reliable to the RAW data
Post by: bjanes on January 16, 2008, 07:03:22 pm
Quote
About 1000 of the ~3590 levels currently used are already wasted by read noise.
[{POST_SNAPBACK}][/a] (http://index.php?act=findpost&pid=167636\")

I would like to see data supporting this assertion. As Roger Clark points out, noise in DSLR cameras consists mostly of shot noise until you get fairly deeply into the shadows. Here is a noise model taken from [a href=\"http://www.clarkvision.com/imagedetail/evaluation-1d2/index.html]Roger's analysis of the 1D MII[/url]. I think you overemphasize read noise.

The table shows shot noise and read noise for the 1DMII at ISO 100 with shot and read noise expressed in electrons on the left and DNs on the right. A 12 stop range is covered, but if the darkest f/stop needs 8 levels, the effective DR is limited to 9 stops by bit depth considerations alone, disregarding noise, and as shown by Norman Koren. (http://www.normankoren.com/digital_tonality.html). The analysis assumes that the camera places the highlights at 4095, but in practice it may be lower as you indicate.

In zones 0 through 7, the noise is predominately shot noise, and these zones are considered shot noise limited. There are 4080 levels. Only zone 8 is read noise lilmited, and it contains 8 levels. If some posterization is acceptable, you could include 7 more levels for a total of 15. Where do you get the value of 1000?

Of course, DR may be limited by noise as well as posterization, but the S:N of 1.49 in the deepest shadows, while quite low, does contain image information.

(http://bjanes.smugmug.com/photos/244177013-O-3.gif)
Title: Camera's histogram reliable to the RAW data
Post by: John Sheehy on January 17, 2008, 03:35:54 pm
Quote
I would like to see data supporting this assertion. As Roger Clark points out, noise in DSLR cameras consists mostly of shot noise until you get fairly deeply into the shadows.

Define "fairly deep(ly)".  Do you realize how far below saturation "middle blue" is in tungsten WB?

Also, if someone wants to make a statement to say current cameras are shot noise limited, it's the kind of thing that can't be proven right or wrong easily, because the point of reference is arbitrary.  My question is, why would someone even say that?  What purpose does a statemnt like that serve except to be academic fluff or filler?

And, as it turns out, shot noise is not the only noise in highlight area of cameras.  You should be seeing Emil Martin's updates to his web pages soon, where he discusses another form of noise present in highlights (usually only visible in the top 1 to 2 stops of the sensor's DR).

Quote
Here is a noise model taken from Roger's analysis of the 1D MII (http://www.clarkvision.com/imagedetail/evaluation-1d2/index.html). I think you overemphasize read noise.

Unlike you and Roger, I have actually investigated what pure shot noise would look like in the deep shadows, and it is a relatively beautiful thing, compared to the reality of read noise.  Blacks are actually black.

Quote
The table shows shot noise and read noise for the 1DMII at ISO 100 with shot and read noise expressed in electrons on the left and DNs on the right. A 12 stop range is covered, but if the darkest f/stop needs 8 levels, the effective DR is limited to 9 stops by bit depth considerations alone, disregarding noise, and as shown by Norman Koren. (http://www.normankoren.com/digital_tonality.html). The analysis assumes that the camera places the highlights at 4095, but in practice it may be lower as you indicate.

In zones 0 through 7, the noise is predominately shot noise, and these zones are considered shot noise limited.

And that is significant because ...?  Would anyone really expect SNR limits to occur in the highlights?

Neither the sensor nor the camera are shot noise limited.

Quote
There are 4080 levels. Only zone 8 is read noise lilmited, and it contains 8 levels. If some posterization is acceptable, you could include 7 more levels for a total of 15. Where do you get the value of 1000?

I wrote 1000 for ISO 1600, not ISO 100, which I said would be served well by 2500 levels at ISO 100.

(Edit: I thought you were referring to my statement about ~1000 levels being enough.  Now that I think of it, you may be talking about the "1000 levels already wasted" - that referred to the fact that about 3500 are used in the 5D at ISO 100, and  only about 2500 are needed (not including some room for negative read noise).  I do not mean to imply by this that the levels should requantized to this amount; I simply mean that the original capture could have been at this many levels in the original digitization, for a more compressible file size, with infinitessimal loss of signal.)

The basic idea is that if you only need enough linear levels for the read noise to be at least 1.4 ADU, then you can scale down the number of levels by 1.4/readnoise, to get a baseline for the minimum number of levels for digitizing.

Quote
Of course, DR may be limited by noise as well as posterization, but the S:N of 1.49 in the deepest shadows, while quite low, does contain image information. [a href=\"index.php?act=findpost&pid=167668\"][{POST_SNAPBACK}][/a]

Of course it does.  You can't improve on it by having more than a certain number of levels, though, as the signal is just an average of lots of noise.  Rounding the noise values out to infinitessimally more accurate values with more levels does not help the signal come through the noise.

I have measured this visually, with extreme stretching of the levels, with RAW data from numerous cameras, and simulations, and 1.4 ADU is my safe limit.
Title: Camera's histogram reliable to the RAW data
Post by: bjanes on January 17, 2008, 04:30:11 pm
Quote
Unlike you and Roger, I have actually investigated what pure shot noise would look like in the deep shadows, and it is a relatively beautiful thing, compared to the reality of read noise.  Blacks are actually black.
And that is significant because ...?  Would anyone really expect SNR limits to occur in the highlights?

Neither the sensor nor the camera are shot noise limited.
I wrote 1000 for ISO 1600, not ISO 100, which I said would be served well by 2500 levels at ISO 100.
[a href=\"index.php?act=findpost&pid=167839\"][{POST_SNAPBACK}][/a]

OK, here is the same type of analysis showing read and shot noise for the 1DMII at ISO 1600. Again I would like to see your data supporting your assertions, not some blanket statement from high.

(http://bjanes.smugmug.com/photos/244477539-O.gif)
Title: Camera's histogram reliable to the RAW data
Post by: John Sheehy on January 17, 2008, 06:05:55 pm
You replied about 1 minute too soon.  I just edited the post you replied to, as there were two figures of 1000 or close in my post, and I apparently assumed the wrong reference in the post you replied to.

Quote
John,

You talk a lot, but your logic is questionable and you present absolutely no data to back up your assertions. Could you give us a link to your data?
[a href=\"index.php?act=findpost&pid=167849\"][{POST_SNAPBACK}][/a]

You need to be a bit clearer about what you don't believe.  I felt a very vague feeling in your previous reply.

So let me state the core of what I am saying, and then you can object specifically to something or ask for proof.  It's as if I am supposed to know what is apparently illogical to you and defend it.  Nothing I've written recently is illogical to me, so I can't figure out what you think is questionable.

Here is a summary of what I have been saying:
"In the face of all the analog noises involved in the readout of a sensor, quantization of any practical significance can only occur when the number of linear levels used is such that the blackframe read noise, in those ADUs, falls significantly below 1.4, for single-exposure RAW images."

As I've stated previously, 1.4 is a conservative value.  We can get away with read noise ADUs as low as 1.1 without incident (and that implies even less levels needed).

The tools that I am using to look at these matters are horrible, in terms of the workflow involved in compositing images for comparisons.  You can verify what I say for yourself, quite easily, though.  Just open two instances of IRIS, and in one, select a RAW from a camera whose read noise is known at the ISO.  Then, calculate the division needed to bring that read noise down to 1.4 ADU.  Set the threshold sliders to some window down in the shadows, crop an area of interest, and then save out the .fit file.  In the second instance of IRIS, load the crop.  In this second instance, divide the image by the factor needed to scale the read noise down to 1.4 ADU.  Multiply back by that number again (or rescale the threshold sliders, if that is more convenient).  What you will see, is exactly the same thing, as far as your eyes can tell, in both instances.  Now, reload the crop into the second instance of IRIS, and divide by the read noise in original ADUs (to make the new noise 1.0 ADU).  *NOW*, you can see a little bit of quantization.  Try again, bringing the read noise down to 0.8, and 0.7 ADU and things fall apart very rapidly.  This is true regardless of what level of read noise you started at; you only start running out of useful levels when you get down to 1.4 ADU of noise.

Now, try similar things with highlights.  Find the brightest area in a smooth, OOF gradient, and measure its sigma.  Then, do the same as before, to bring this down to 1.4 ADU.  The bright area that you got the shot noise deviation from will look like it lost no smoothness, but the darker areas that might be in the image will show quantization.  Try again with 1.0, 0.8, 0.7, etc.  Same principal keeps applying.

It is easiest to do this with single color channels, because they are easier to crop.  You can verify, however that the same principle applies to color by carefully cropping so that the RGB CFA patterns in the crops are unaltered, or if you have plenty of RAM, just don't crop at all and just window-in the areas to compare (quantization must be before color conversion, though).

One day you will realize that noise is a hard ruler of appreciable levels, and all the anecdotes about levels and levels per stop and such is usually irrelevant in today's noisy digital photography.  All those anecdotes come from noiseless, synthetic graphics and are totally meaningless in digital photography.
Title: Camera's histogram reliable to the RAW data
Post by: John Sheehy on January 17, 2008, 06:09:37 pm
Quote
OK, here is the same type of analysis showing read and shot noise for the 1DMII at ISO 1600. Again I would like to see your data supporting your assertions, not some blanket statement from high.
[a href=\"index.php?act=findpost&pid=167849\"][{POST_SNAPBACK}][/a]

I am not telepathic.  What assertion do you believe I am making, which needs to be proven?

In any event, I don't see the relevance of the chart you posted to the topic at hand.  We're talking about necessary levels in digitization, are we not?
Title: Camera's histogram reliable to the RAW data
Post by: bjanes on January 17, 2008, 06:30:25 pm
Quote
I am not telepathic.  What assertion do you believe I am making, which needs to be proven?
[a href=\"index.php?act=findpost&pid=167864\"][{POST_SNAPBACK}][/a]

That 1000 levels are lost to read noise.

Quote
In any event, I don't see the relevance of the chart you posted to the topic at hand.  We're talking about necessary levels in digitization, are we not?
[a href=\"index.php?act=findpost&pid=167864\"][{POST_SNAPBACK}][/a]


The chart clearly shows the relative contributions of read and shot noise for the exposure zones, and shot noise predominates in the green area of the chart, which comprises 4080 levels, whereas read noise dominates only in the very deep shadows, comprising 15 levels at most. How can you lose 1000 levels to read noise, when it predominates in only 15?
Title: Camera's histogram reliable to the RAW data
Post by: John Sheehy on January 17, 2008, 07:42:39 pm
Quote
That 1000 levels are lost to read noise.
The chart clearly shows the relative contributions of read and shot noise for the exposure zones, and shot noise predominates in the green area of the chart, which comprises 4080 levels, whereas read noise dominates only in the very deep shadows, comprising 15 levels at most. How can you lose 1000 levels to read noise, when it predominates in only 15?
[a href=\"index.php?act=findpost&pid=167868\"][{POST_SNAPBACK}][/a]

OK, now your objection is clearer.

First of all, the issue I addressed has nothing to do with how much of the recorded range has more read noise than shot noise (and read noise is significant far above this range, especially if there is 1-dimensional or "banding" noise, which is much more visually potent than 2-D noise).  The issue is *what* the level of read noise is in ADUs, as this will be the minimum noise found anywhere in the image, in a linear sense, relative to the steps of the digitized levels.  If quantization doesn't happen where there is only read noise, it shouldn't happen at any other range, where shot noise increases the absolute noise level.

The premise is that with a read noise of 2.0 ADU at ISO 100 in the 5D (not the same as the 1Dmk2), the number of levels used can be divided by 2.0/1.4, or 1.43, without incurring any practical quantization in the RAW data.  The other way to look at it is that the steps can become 1.43x their current size.  In this case, that would mean that the number of levels between black and RAW saturation in the 5D at ISO 100 could just as well have been recorded with about 1000 less levels (about 2500 instead of about 3500).  It is not about clipping levels away from the shadows (or highlights, or anywhere).
Title: Camera's histogram reliable to the RAW data
Post by: Guillermo Luijk on January 20, 2008, 09:37:07 am
Just curious: http://blog.lexa.ru/2008/01/16/balans_belo...inim_gisto.html (http://blog.lexa.ru/2008/01/16/balans_belogo_dlja_snimajuschix_v_raw_chinim_gisto.html)

This guy performs the Hue/Sat analyse with PS. Russian document.
Title: Camera's histogram reliable to the RAW data
Post by: Guillermo Luijk on April 12, 2008, 03:38:16 pm
Sorry to recall this post, but a forum member of POTN forum has depicted (http://photography-on-the.net/forum/showthread.php?t=485349 (http://photography-on-the.net/forum/showthread.php?t=485349)) a much easier way to obtain a UniWB on any camera than my whole procedure: the point is that what we are seeking for is an image which is already balanced (R=G=B ), so that when the camera tries to calculate the WB multipliers over it, they will be the closes possible to 1.0 1.0 1.0.

And which is the easiest to produce already balanced image? any shot where all three channels are saturated on every pixel, i.e. pure white, and that's as simple as shooting your camera to the sky or any bright area with the lens wide open for some seconds. Once we use the resulting image for in-camera WB calculation, the multipliers will be the closest to UniWB.

I have done this in my camera and achieved multipliers:

multipliers 1.006856 1.000000 1.005877 1.000000

I cannont improve that, surely because some hot pixel or whatever. A a max error of 0.69% for the R channel is simply outstanding.

Regards.
Title: Camera's histogram reliable to the RAW data
Post by: ride5000 on February 20, 2009, 04:18:27 pm
Quote from: GLuijk
Sorry to recall this post, but a forum member of POTN forum has depicted (http://photography-on-the.net/forum/showthread.php?t=485349 (http://photography-on-the.net/forum/showthread.php?t=485349)) a much easier way to obtain a UniWB on any camera than my whole procedure: the point is that what we are seeking for is an image which is already balanced (R=G=B ), so that when the camera tries to calculate the WB multipliers over it, they will be the closes possible to 1.0 1.0 1.0.

And which is the easiest to produce already balanced image? any shot where all three channels are saturated on every pixel, i.e. pure white, and that's as simple as shooting your camera to the sky or any bright area with the lens wide open for some seconds. Once we use the resulting image for in-camera WB calculation, the multipliers will be the closest to UniWB.

I have done this in my camera and achieved multipliers:

multipliers 1.006856 1.000000 1.005877 1.000000

I cannont improve that, surely because some hot pixel or whatever. A a max error of 0.69% for the R channel is simply outstanding.

Regards.

thanks for the update Guillermo!

-ken
Title: Camera's histogram reliable to the RAW data
Post by: erick.boileau on December 19, 2009, 02:29:53 am
I am having a 5D Mark II (and a MAC)   what can I use instead of DCRAW ?
Title: Camera's histogram reliable to the RAW data
Post by: Guillermo Luijk on December 19, 2009, 09:11:39 am
Quote from: erick.boileau
I am having a 5D Mark II (and a MAC)   what can I use instead of DCRAW ?
DCRAW works for Mac and the 5D2. There is a download link here (http://www.insflug.org/raw/).
Title: Camera's histogram reliable to the RAW data
Post by: erick.boileau on December 19, 2009, 09:22:52 am


thank you
Title: Camera's histogram reliable to the RAW data
Post by: erick.boileau on December 19, 2009, 10:37:35 am
it is what you get directly with a Digital Back PhaseOne P45

(http://www.urpix.fr/files/0yvh84eqhk7374cto.jpg)