Camera Profiling - DNG, ICC and alternative methods

[l_d_allan]

I'm wondering what are we going to do then after proving that the spectral response of cameras do differ unit to unit.

Several things come to mind ...

• Better info to judge whether camera profiles are worth doing for a particular model that was good vs average vs poor on sensor consistency. Is a "generic" likely to be "good enough"?
• If valid info came out that the top-of-the line NiCan D123 had a really consistent sensor, and the top CaNik 321D had a very inconsistent sensor, perhaps CaNik might "see the error of their ways" and respond to competitive pressure. Stranger things have happened.
• Like when concerns came out about iffy "follow focus" with an earlier Canon 1D#, they caught a lot of flak to improve. Or not?
• Or now that sensors from Sony are ahead in DSLR DR. Pressure on Canon (and I suppose Nikon also) to improve their sensors.

[Hening Bettermann]

> Hening, what do you think of the results of my test with the various settings in Irident Developer? Have I misinterpreted Brian's comments on how settings work?

I have not thought, but tried to leave that to others who are more qualified :-) I have alerted Brian to your post, but he is about having a new baby (I hope this is not considered too indiscrete?) Tonight, I'll go through your settings again and compare them to the ID help text more closely than I have so far.

There have been a couple of new postings on the Shooting Color Targets thread which I feel belong more in THIS thread. Wonder if we should try to move them?

[Hening Bettermann]

Well, my verbose interpretation of your findings would be:
1-Use of embedded DNG profile throws off colors, Embedded Curves checked or not
2-The Shadows Fine Tune Slider has no effect.
 
In your shots # 1...4, the 2 darkest patches have on my screen the same Lab readouts in the Mac DigitalColorMeter:

patch D5: 35,625/-0,266/-1,293 ; patch D6: 20,589/-1,027/-0,891

I'm afraid Brian has overlooked something, but he is on paternity leave the next 2 months or so, so we have to be patient...

[Hening Bettermann]

Hi
I just got an e-mail from Brian with a long reply to the above comparison between ID and ACR/LR, and the go ahead to post it here. Like Vladimirovich, Brian prefers to have it posted directly, split up in several posts, and so be it. Below however a zip in addition for those who prefer that.
Kind regards - Hening.


From: Iridient Digital Tech Support <support@iridientdigital.com>
Date: 13. March 2013 20:37:03 GMT+01:00
 
[Comparing Iridient Developer and ACR/LR] part 1 of 3

 I have no idea what Adobe really does in their software and how various versions of ACR/Lightroom may or may not match up with what they do in the DNG SDK. My quick tests with ACR version 6.7 and the Adobe 2010 process selected under the Camera Calibration tab seemed to show similar behavior with the DNG SDK and the "Blacks" slider adjustment found there, but I haven't done a real thorough investigation yet...

I read the notes on Shadows Fine Tune. Based on my quick tests this should be working as expected under the following circumstances.

1) When using "DNG Camera Profile..." options in Input Profile menu AND when the "Use embedded..." checkbox is checked on the Camera Curve pane the Shadows Fine Tune Slider will adjust the DNG SDKs exposure curves "shadows" variable which adjusts the contrast of the dark shadows region.

Where SFT = shadow fine tune control

Setting SFT to +100 in Iridient Developer will pass a value of 0 to the DNG SDK and lightens the dark shadows region compared to larger values.
Setting SFT to 0 in Iridient Developer will pass a value of 5 (the default used by DNG SDK v1.4 which also just happens to match up with the default "Blacks" slider available in ACR 6.7 when using the Adobe 2010 process under Camera Calibration pane. Not sure about other versions of ACR or Lightroom. Lightroom anyway seems to act differently when choosing the 2010 process and doesn't completely switch its controls at least not in my very brief investigation so far... the result here should be slightly darker shadows.
Setting SFT to -100 will pass a value of 20 and again this should go even darker with the shadow tones.

Old versions of ACR/Lightroom had a "Blacks" slider with a range of 0-100 and a default value of 5. This OLD control seems to act much like the "Shadows" variable available with the exposure curves in the DNG SDK v1.4. This is based on VERY quick visual tests of a couple images.

[Hening Bettermann]

[Comparing Iridient Developer and ACR/LR] part 2 of 3

*** NOTE *** Lightroom 4 does not seem to behave like ACR in this regard, at least not ACR v6.7.

*** NOTE *** The latest Adobe "Blacks" slider seems to have zero (or little) relation to the "Blacks" slider in Lightroom 4 which now defaults to value of 0.

2) I have no idea what Adobe actually does in their software or how it may or may not match up with the DNG SDK's processing for the exposure curves and for that matter for DNG Camera Profiles in general. My guess is they behave somewhat similarly, but based on labeling of functions in the SDK , even very latest 1.4 release, some of the processing in DNG SDK would appear to be "out of date" with their most recent software and is labeled "ACR3". Sseems to act somewhat like old versions, my guess was the 2010 process, but this was just based on a couple quick tests with a couple images and the controls in ACR v6.7.

3) I believe (again based on just visual inspection of adjustments, no actual knowledge of their processing in ACR or Lightroom) that Adobe tends to use a curve function for their exposure adjustments. It may be possible to get a linear or close to linear "curve" with the right settings options, but this seems to have changed with various versions of ACR/Lightroom and I don't know what settings may or may not introduce non-linearities into the adjustment.

samueljohnchia was using Exposure adjustments in ACR/LR which could introduce non-linearities based on their use of curves for some of these adjustments and this could account for differences in the grayscale rendering.

In other words an Exposure adjustment of for example 0.66 in Iridient Developer may behave totally differently than an exposure adjustment of 0.66 in Adobe software.

4) If doing a comparison between programs, probably best to use DNG format files where black point, white point, baseline exposure, linearization curves and white balance are all documented and should match up between Iridient Developer and Adobe software.

With native camera manufacturer RAW formats could have some minor variations in default black/white/exposure levels that would throw off the default grayscale between programs...

[Hening Bettermann]

[Comparing Iridient Developer and ACR/LR] part 3 of 3

NOTE however due to some differences with my handling of "baseline exposure", white point determination and black point determination and what Adobe ends up using for  baseline exposure/white/black in their DNG conversions (and presumably in their software as well...) I have overrides for all of these values available in Iridient Developer. Using these overrides can allow for a better match between DNG conversions and use of camera manufacturer native RAW formats.

Adobe DNG conversion support  for new cameras may come some time following new camera support being added to Iridient Developer. There is no way for me to accurately determine what Adobe may use for these values prior to them updating their DNG converter software so in some cases my defaults will not match up with theirs.

So if you are using DNG format to make comparisons between Iridient Developer and Adobe software AND you are using the DNG Camera Profile option AND using the DNG Camera Profiles embedded (or DNG SDK default) exposure/tone curve you also need to be sure that in Iridient Developer on the Camera Adv pane that the Baseline Exposure, Black Level and White Level are all set to "Default". This will use the values exactly as specified in the DNG file.

In most cases using the DNG file defaults is in fact the camera default, but not all....

5) When not using DNG Camera Profiles (using ICC profiles or DNG matrix color) or with the "Use embedded..." option unchecked in the Camera Curve pane then the Shadow Fine Tune adjustment should be set to 0 to give no change to shadow values as defined by the baseline exposure, white level, black level, camera tone curve and exposure adjustments in Iridient Developer.

Best regards,
Brian Griffith
Iridient Digital

[digitaldog]

[Comparing Iridient Developer and ACR/LR] part 2 of 3

I'm not sure of all the little details and ramifications but what I see is this: If I open a DNG with a custom DNG profile in LR 4 (yes, PV2012) and save everything, then open it in ID and set the profile for DNG Camera Profile (Use DNG Metadata), the differences in color and tone are rather subtle and I suspect to be expected. What am I missing?

[samueljohnchia]

Well, my verbose interpretation of your findings would be:
1-Use of embedded DNG profile throws off colors, Embedded Curves checked or not
2-The Shadows Fine Tune Slider has no effect.
 
In your shots # 1...4, the 2 darkest patches have on my screen the same Lab readouts in the Mac DigitalColorMeter:

patch D5: 35,625/-0,266/-1,293 ; patch D6: 20,589/-1,027/-0,891

Hening, what do you mean by the use of the embedded DNG profile throws off colors? As far as I know all camera profiles distort colors to a certain degree. Is that what you are referring to?

I disagree that the Shadows Fine Tune Slider has no effect. It does make a change in the shadow regions, both visibly and measurably with Photoshop's info readouts. I'm not sure why the Mac DigitalColorMeter is giving you the same readouts. There are definitely different in the gray patches you refer to.

I'll be looking closely at Brian's comments. Thanks for sharing the emails from Brian.

[samueljohnchia]

I'm not sure of all the little details and ramifications but what I see is this: If I open a DNG with a custom DNG profile in LR 4 (yes, PV2012) and save everything, then open it in ID and set the profile for DNG Camera Profile (Use DNG Metadata), the differences in color and tone are rather subtle and I suspect to be expected. What am I missing?

I'm not sure what you are missing Andrew, but then the subtlety between different raw converters handling of tone and color is not any less significant than the subtle differences between ICC and DNG profiles, which I may have put too fine a point on in previous posts. If there are differences, as per Brian's post, they should be minimized for comparisons. I'll need to look closer at ID again.

[samueljohnchia]

samueljohnchia was using Exposure adjustments in ACR/LR which could introduce non-linearities based on their use of curves for some of these adjustments and this could account for differences in the grayscale rendering.

I think that the exposure slider in ACR PV 2010/2003 is linear enough for our comparison purposes. As long as there is no channel clipping.

[Hening Bettermann]

Hi Samuel,

> what do you mean by the use of the embedded DNG profile throws off colors?

Your examples # 3 and 4 show a gross difference between actual and theoretical patch colors. These are the images with the embedded DNG curve checked.

> I disagree that the Shadows Fine Tune Slider has no effect. It does make a change in the shadow regions, both visibly and measurably with Photoshop's info readouts. I'm not sure why the Mac DigitalColorMeter is giving you the same readouts. There are definitely different in the gray patches you refer to.

Ooops! I measured the front (bottom-left) patches, thinking THESE were the actual patch colors! But I see now that they are the reference colors, since the rear (top-right) colors show in fact a difference. Sorry!

[samueljohnchia]

Well, my verbose interpretation of your findings would be:
1-Use of embedded DNG profile throws off colors, Embedded Curves checked or not

Your examples # 3 and 4 show a gross difference between actual and theoretical patch colors. These are the images with the embedded DNG curve checked.

Hi Hening, if I misunderstand you please correct me.

Examples #1, #2, #3,#4 and #5 all use the embedded DNG profile, the same one that I used for the ACR comparison.

Examples #3 and #4 has ID set to use the embedded tone curve in the DNG profile.

I think you did not mean that using the embedded DNG profile throws colors off, but checking the "Use embedded curve" checkbox under the Camera Curve tab in ID throws off the color patches. Patch luminosity seems to be affected to a greater degree than its hue and saturation.

Thanks for checking out the gray patches again!

[Hening Bettermann]

> Examples #3 and #4 has ID set to use the embedded tone curve in the DNG profile.

> I think you did not mean that using the embedded DNG profile throws colors off, but checking the "Use embedded curve" checkbox under the Camera Curve tab in ID throws off the color patches.

It looks to me that I thought 'The profiles are in use only when checked'.

[Hening Bettermann]

Hi

I am exploring ColorPerfect. Attached is a screen shot that shows the B1 orange field of the CC24, from left to right: 1-Bruce Lindblooms synthetic target for ProPhoto; 2-my shot of my CC24, processed with ColorPerfects MakeTiff as raw converter, displayed in Photoshop with the ColorPerfect plug-in; 3-the same shot, displayed in ACR CS5 with Adobe Standard camera profile, everything else zero. ProPhoto is the color space in ACR and PS.
The monitor is calibrated to 5800 Kelvin and L*.
As seen, 1 and 2 are relatively close to each other, ACR seems off - but it is the latter which is very close visually to the physical CC24 viewed under one of the same Solux lamps that was used for the shot, whereas the other 2 seem reddish.
What is going on?

Confused - Hening

[Tim Lookingbill]

Hi

I am exploring ColorPerfect. Attached is a screen shot that shows the B1 orange field of the CC24, from left to right: 1-Bruce Lindblooms synthetic target for ProPhoto; 2-my shot of my CC24, processed with ColorPerfects MakeTiff as raw converter, displayed in PhotoLine with the ColorPerfect plug-in; 3-the same shot, displayed in ACR CS5 with Adobe Standard camera profile, everything else zero. ProPhoto is the color space in ACR and PL.
The monitor is calibrated to 5800 Kelvin and L*.
As seen, 1 and 2 are relatively close to each other, ACR seems off - but it is the latter which is very close visually to the physical CC24 viewed under one of the same Solux lamps that was used for the shot, whereas the other 2 seem reddish.
What is going on?

Confused - Hening

My guess from photographing the same target with my 4700K Solux lamp and getting similar results is that you're seeing why the Solux wasn't designed as a photographic light but a print viewing light. I noticed my Pentax K100D kept getting an overall reddish amber bias which tended to saturate warm colors due to the halogen underpinnings of the Solux.

Combine that with DNG's color correction processes as being a bit on the light handed side adjusting for this red spectrum bias and the confusion should subside, hopefully. Just my observation, though. Sensors don't see the same way as our eyes on all colors some more than others.

I'm just so surprised how my old CC chart I've got next to my display still looks exactly as Lindbloom's synthetic patches you've posted even viewing under my daylight balanced T8 flotube desk lamp. It's uncanny.

And thanks for confirming your version of Adobe Standard behaves the same as mine for my Pentax which applies a consistent neutralization curve throughout the entire color table of an image which tends to lighten blues as evident in your CCchart rendering.

[Hening Bettermann]

Hi Tim

thank you for your reply.

I'm afraid the Solux lamp is not the whole story: I failed to post that with the CC24 viewed in sunshine that patch looks largely the same (even though the direct side-by-side comparison was not possible; but the difference is so gross that I can remember it from my desktop to my porch).

It may look like the Adobe Standard profile is trying to anticipate a bleaching that will occur in print?

Maybe I should try to get me one of these uncanny T8 tubes? And use flash for shooting the target?

Also worth noting is that the difference is not present in the white/gray patches, neither visually nor by the numbers.

[samueljohnchia]

Hi Hening, I also have similar observations, although I think that the Adobe Standard profile renders the orange patch too greenish, and the ColorPerfect too reddish, in your example. I find Bruce's target to be quite close to viewing the actual CC in daylight.

Tim's comments are right on the ball regarding the lower saturation of most of the color patches with Adobe Standard. Adobe made some changes to the V2 profiles, one of which toned down the saturation of colors because users complained that the V1 profiles were too punchy. I'm not sure if the V4 profiles received any additional tweaks to the rendering of color. The neutral patches should all be nice and neutral if the WB is correctly set.

How do the other color patches compare?

[Hening Bettermann]

Hi

Samuel, thank you, too, for your reply.

I have fiddlet a little more with this and have learned a thing or two.
1-The comparison to the physical chart is so inconsistent depending on viewing light that I dropped it. It was just the Adobe Profile that confused me.
2- I made a quick and dirty shot of the CC24 in daylight (16:15 h winter time, clear sunshine), with a custom white balance taken right before the shot, based on the second-whitest patch of the CC. When displayed on the monitor, WB as shot, the orange looked even more reddish than before. Then I had the bright idea of using only the blue-yellow balance of the as-shot-WB, and set the Tint to 0. And look! some difference! Thus inspired, I did the same to my Solux shot, and look, that improved, too, if not by that much.

What can I do to come even closer - flash?

Here is a comparison; left to right:
1-Synthetic, 2-Daylight Tint 0, 3-Solux Tint 0, 4-Solux Tint 0 5-Solux as shot.
In case you're interested to see the whole charts, here is a zip.

[Tim Lookingbill]

(even though the direct side-by-side comparison was not possible; but the difference is so gross that I can remember it from my desktop to my porch).

I'm not seeing such a "gross" difference between the first shot (exposed under the Solux) second from the Lindbloom's synth version.

In my case to check consistency I shot three (morning/noon/afternoon) 24 patch CCchart exposed under direct sunlight I used to build several DNG profiles processed with the DNG Profile Editor Wizard using ACR's default settings and (ACR 4.4) canned profile.

All three exhibited somewhat slightly different renderings in ACR under default settings. None were colorimetrically accurate according to the Lab numbers. I had to zero out contrast, black point, switch to Linear curve and adjust exposure to get the white patch to read 96L Lab readings to get near perfect Lab numbers on every patch.

Unfortunately the overall image looked like a machine took the shot because it was flat and dim looking, not the vibrant, crisp and full of life vision I saw under a sunny day. Also the noon shoot (brightest and most dull part of the day) made the second to the right white patch look sort of pinkish so when I clicked to establish R=G=B it suddenly made the overall image appear to look greenish yellow but got rid of the reddish saturated look of the gold patch I see in your shot while zeroing out all contrast inducing settings did most of the color correction.

Maybe I should try to get me one of these uncanny T8 tubes? And use flash for shooting the target?

That's a misunderstanding. I was making the point that the eyes are much more forgiving (adaptive) compared to how sensors record color under a wide range of neutral looking light sources. Sensors will pick up on odd spectra reflected off even the CC chart we don't even see.

Editing software was never designed to make photography an exacting color reproduction process though it is possible but requires work. The default settings along with ACR/LR's color engine were designed to somewhat emulate the behavior of film with regard to the relationship of contrast to saturation/hue changes, but in my experience it doesn't take a lot of work to get exacting numbers if that's the goal. It's just the overall look of the image will resemble what I've seen operating a graphics camera as a copy stand reproduction setup where the scene dynamic range has to match the output (i.e. reproduction of a painting viewed in a museum) which can be quite dim and flat by comparison to bright sunny day.

It may look like the Adobe Standard profile is trying to anticipate a bleaching that will occur in print?

I believe why that gold patch looks so yellow from applying the Adobe Standard profile may be due to hue twist that some have suspected and mentioned in the past. I get the same effect with a lot of sunset scene that push the saturation of golden colors caused by a custom DNG profile that tends to clip/flatten detail. I use it sometimes to bring out definition in upper mids to highlights shot under extreme lighting situations often delivering "desirable" results over accuracy. See the comparison below.

[Tim Lookingbill]

Hi

Samuel, thank you, too, for your reply.

I have fiddlet a little more with this and have learned a thing or two.
1-The comparison to the physical chart is so inconsistent depending on viewing light that I dropped it. It was just the Adobe Profile that confused me.
2- I made a quick and dirty shot of the CC24 in daylight (16:15 h winter time, clear sunshine), with a custom white balance taken right before the shot, based on the second-whitest patch of the CC. When displayed on the monitor, WB as shot, the orange looked even more reddish than before. Then I had the bright idea of using only the blue-yellow balance of the as-shot-WB, and set the Tint to 0. And look! some difference! Thus inspired, I did the same to my Solux shot, and look, that improved, too, if not by that much.

What can I do to come even closer - flash?

Here is a comparison; left to right:
1-Synthetic, 2-Daylight Tint 0, 3-Solux Tint 0, 4-Solux Tint 0 5-Solux as shot.
In case you're interested to see the whole charts, here is a zip.

I checked your "Solux As Shot" version agains the Lab numbers I have derived from actual measurements of real CCcharts and it shows you're WAY within specs. In fact you can be as far off as 5 numbers in any of L,a,b readings and still have it look accurate enough. In fact your blue patch is spot on, but your caucasian skin looks a bit on the pinkish side but still within "5" number specs with the Solux shot.

If you're seeing something that says differently then I suspect your calibration is way off. In fact I'ld like to know why your png screengrabs have "Display" as the name of the embedded profile when I open them in Photoshop CS3. Your custom display profile should be embedded or you should've converted to sRGB. Do you have a custom display profile?

What's up with that?

Here's my screengrab showing how close you are to the Lab numbers.