Camera Profiling - DNG, ICC and alternative methods

[Hening Bettermann]

Related to this subject, on the Apple Colorsync list a discussion is going on after Ben Goren wrote an article on exposure, threadname:
"Primer on photographic exposure, etc."

Ben Goren's  article:
http://trumpetpower.com/photos/Exposure

The thread is as interesting as the article.

--

Hi Ernst,
thank you for this link, interesting reading indeed. Even though I don't quite understand: The first part sounds much like the ColorPerfect theory of just getting the grays right; but then he uses 600(+) color patches for a target.
Hening.

[Ernst Dinkla]

Hening,

There is even more added to the Colorsync thread like this link to a Wayne Bretl presentation file:

http://trumpetpower.com/files/trumpetpower.com/Wayne_Bretl_Theoretical_and_Practical_Limits_to_Wide_Color_Gamut.ppsx

I'm not even trying to understand all that, my intuition tells me that there is nothing to gain for me there. Swamp territory where scientists knit the loose ends with creativity and (I think) where artists shouldn't try to knit the grand unified theory of color management. My point of view that is older than this thread:
http://www.luminous-landscape.com/forum/index.php?topic=75235.15;wap2


--
Met vriendelijke groet, Ernst

http://www.pigment-print.com/spectralplots/spectrumviz_1.htm
December 2012, 500+ inkjet media white spectral plots.

[Hening Bettermann]

Hi Ernst,

thanks for the new link as well, but I can not open it. - I have now subscribed to the ColorSync mailing list and will try to understand what little I can. I can not "knit the grand unified theory of color management", but I have to understand a little to be able to deconstruct the default renderings of various established implementations of color management. As a landscaper, I do not need the precision level required for art reproduction. But I want something that is BASICALLY accurate (natural) and not just pretty. And in all due modesty, I find that my efforts so far already give me images that are visibly closer to that goal than mentioned defaults.

Kind regards - Hening.

[samueljohnchia]

Thanks to everyone who has contributed to this thread and keeping the discussion alive and civil. I have learned a great deal along the way. I haven't found the time to read everything on the colorsync list and on the trumpetpower website but I will. I have briefly glanced through the powerpoint slides that Ernst has shared here and I now know that the on-landscape conclusions at the end of this post are misleading. I also learned that wider band pass filters contribute to an increase in color noise! Very interesting!

At this point I'm pretty convinced that getting accurate colorimetric color out of the camera by using photographs of color targets will never happen. It may be an improvement, as Hening and myself have found, by giving us a different (and to our minds, better) starting point. However, many tools in the raw converters that we use thereafter, throw wrenches into the mix of color, which distort, compress and expand color in other ways as we tweak our images to what we like. At the end of the day, it ain't as "accurate" as we had hoped it would be.

What is very interesting, to me at least, is the work by Jun Jiang, Dengyu Liu, Jinwei Gu and Sabine Süsstrunk that Tim has graciously shared earlier in this thread. Their experiments show that it is possible to come very close to recovering the spectral sensitivities of camera sensors, and they show examples of image correction as well. Certainly it is entirely possible to achieve nearly the same visual results with good knowledge of the available tools, a good eye for color and good aesthetic judgement.

At this point I would not yet abandon camera profiling altogether. Again, as Hening and I have found, we like this new starting point for editing images. I have also noted that I needed less intervention into localized zones of color to fix any problematic hues. The LUT tables in DNG profiles do tweak color in a rather detailed manner, that sliders can't. Since DNG profiles made with the DNG PE still use the same camera matrices as the original Adobe Standard profiles, and they provide a different "flavor" by tweaking the LUT tables, and I have less issues with my custom versions, I'll stick with them for now until something better shows up. But I'll stop calling this process more "accurate".

[solarj]

samueljohnchia, I believe that exact matching of each color patch is possible, but first we must look at most fundamental elements in color information record and conversion, without blend in tone curve, LUT and such kind of after processing

In my understanding, if lighting is D65 and tone curve is linear, a calibrated camera color checker shot will always show the exactly number as reference value, if the sensor is enough good

Last time I discovered that DNGPE don't change the matrix, and I don't like LUT (it will make large area of gradualy changed color unnatural), if a good sensor act very close to human eye's cone spectral response, a single matrix should fix all the matching point on the gamut. In reality the sensor can not be perfect, so some kind of error has to be tolerated, but anyway without a LUT most of the transfer between colors will be gradual and natural

My past experience showed that if the sensor have a high quality, with a very good matrix, you can almost get all the color patches correct: Blue/Green/Red patch can be fixed 100% accurate, and the rest of them might have some error, and you adjust the matrix to reduce the error on other patches and increase the error on B/G/R patch until all of them fall into similar error level

I just picked up the CC24 and did some further testing. To my surprise, under a noon sunlight and 360 degree blue sky at northern europe (Standard definition of D65), the color temp indicated by camera shows 5800K, and the CC shot in ACR get a white balance of 5500K if I pick the second grey patch to do a white balance

So I get some difficulty: It seems that the daylight (sunlight + bluesky) is far from Illuminate D65, it is possible that D65 is a mixture of shadow (9000K) and direct sunlight (5000K), so I can not get a D65 lighting casted on the color checker. Maybe it is still May, I will try it in mid-summer to see if there are some difference, but since the sunlight is already very strong, I doubt even in June it will not be 6500K, maximum 5900K

[solarj]

I just did a test, I opened a raw file, applied two different camera profile (one is only matrix and another one is same matrix + HSD table) and save them as different dng file. Then I opened each of these dng files in xrite software and generated two dcp camera profile

The generated camera profiles have different matrix and HSD table

I then tried another test, applied same camera profile, but different tone curve in ACR and save as different DNG file, then open each of them and use xrite software to generate two dcp camera profile

This time the generated dcp camera profiles are identical

So it becomes a loop, with a camera profile in dng file at the first place, you won't be able to generate an accurate camera profile since the result is depend on the existing camera profile in dng file

And add to these confusing, I discovered that switching the camera profile will affect the "as shot" white balance reading, so I suppose that white balance data is also closely related to the matrix in camera profile, but how is that calculated?

I used to work with SIGMA X3F raw files, with that file, if white balance and matrix data are fixed, all the value will be fixed, crystal clear. It seems with ACR there are many layers of processing, this makes a reliable reproduce of a certain color very difficult  

[Ernst Dinkla]

Well the discussions on that subject in other forums did not dampen down either, it may be interesting to see the latest entries in the Colorsync forum thread: Colorimetric Accuracy in the Field. Some would rather bury both ICC and RAW development CM together.
http://prod.lists.apple.com/archives/colorsync-users/2013/Jun/msg00067.html

--
Met vriendelijke groet, Ernst

http://www.pigment-print.com/spectralplots/spectrumviz_1.htm
December 2012, 500+ inkjet media white spectral plots.

[hjulenissen]

Did anyone see this:?
http://chromasoft.blogspot.no/2009/02/visualizing-dng-camera-profiles-part-1.html


I think it might be interesting to visualize the Adobe default, manufacturer and my own profiles for my camera in this manner, tracing a few "colors". I suspect that the largest spread is going to be in the reds, meaning that they "disagree" on how red should be interpreted.

-h

[solarj]

Did anyone see this:?
http://chromasoft.blogspot.no/2009/02/visualizing-dng-camera-profiles-part-1.html


I think it might be interesting to visualize the Adobe default, manufacturer and my own profiles for my camera in this manner, tracing a few "colors". I suspect that the largest spread is going to be in the reds, meaning that they "disagree" on how red should be interpreted.

-h

Yes, read that some time ago, an interesting view, not scientific anyway

As my above post discovered, the dng camera profile itself is not the original data stored in RAW file, different camera manufacturer hide the most critical data like whitebalance gain and colorconversion matrix inside their proprietary RAW format, Adobe only get to access part of that data, so all the color matching at ACR level is dependant on the critical data inside the RAW file

This indicated that even if you can get very accurate color in one of the RAW file, it is not a guarantee that you will get same good color for another RAW file with the same camera profile, because some of the data (for example white balance gain) in RAW file changed between shooting. To have consistant result, you must be able to fix those critical data in camera raw file, and a typical practice is always shoot the photo with same custom white balance

[hjulenissen]

Yes, read that some time ago, an interesting view, not scientific anyway

As my above post discovered, the dng camera profile itself is not the original data stored in RAW file, different camera manufacturer hide the most critical data like whitebalance gain and colorconversion matrix inside their proprietary RAW format, Adobe only get to access part of that data, so all the color matching at ACR level is dependant on the critical data inside the RAW file

This indicated that even if you can get very accurate color in one of the RAW file, it is not a guarantee that you will get same good color for another RAW file with the same camera profile, because some of the data (for example white balance gain) in RAW file changed between shooting. To have consistant result, you must be able to fix those critical data in camera raw file, and a typical practice is always shoot the photo with same custom white balance

Not sure that I quite follow.

I never touch the WB of my camera (default AWB), and I was under the impression that Lightroom did not read it either? I usually have to fiddle with WB in images that I care about.

I dont think that it is a goal for Lightroom to look exactly like Canons interpretation. I would rather that color response was equalized between Canon and Nikon image in my library to satisfy some "neutral" definition. After that, I'll be applying my own subjective editing.

-h

[solarj]

I found that people can still use dcraw to decode the raw files, I will give it a try. Now that many camera profiles have a large look up table to manually adjust the behavior of each individual color zone, all the color becomes artificial

[solarj]

Not sure that I quite follow.

I never touch the WB of my camera (default AWB), and I was under the impression that Lightroom did not read it either? I usually have to fiddle with WB in images that I care about.

I dont think that it is a goal for Lightroom to look exactly like Canons interpretation. I would rather that color response was equalized between Canon and Nikon image in my library to satisfy some "neutral" definition. After that, I'll be applying my own subjective editing.

-h

The strange thing is, even with AWB, the white balance value in ACR changes if you switch to another camera profile, and we all know that if white blance value changed, the color performance will also change. By definition, white balance is the gain for each RGB channel, should be constant for each individual RAW file

[madmanchan]

Only the "as shot" WB gain (scale factors) are a constant for a given image, as these were pre-determined by the camera and stored in the metadata.  In practice users can (and will) change the WB setting (hence the scale factors) to whatever they want for a given image.  DNG camera profiles will map the colors based on what this (possibly user-modified) WB setting is.

[solarj]

Only the "as shot" WB gain (scale factors) are a constant for a given image, as these were pre-determined by the camera and stored in the metadata.  In practice users can (and will) change the WB setting (hence the scale factors) to whatever they want for a given image.  DNG camera profiles will map the colors based on what this (possibly user-modified) WB setting is.

Even if you always select "as shot" WB, when you apply another camera profile (other than Adobe provided ones), that temperature and tint value still change, this is the most confusing part of ACR

I just ran dcraw for my nex-5r raw file
-----------------------------------------------------

D:\photo\5R\2013\06>dcraw -i -v DSC00425.ARW

Filename: DSC00425.ARW
Timestamp: Mon Jun 10 17:28:53 2013
Camera: SONY NEX-5R
ISO speed: 200
Shutter: 1/640.0 sec
Aperture: f/4.5
Focal length: 27.0 mm
Embedded ICC profile: no
Number of raw images: 1
Thumb size:  1616 x 1080
Full size:   4928 x 3276
Image size:  4928 x 3276
Output size: 4928 x 3276
Raw colors: 3
Filter pattern: RGGBRGGBRGGBRGGB
Daylight multipliers: 2.614211 0.926724 1.255255
Camera multipliers: 2984.000000 1024.000000 1472.000000 1024.000000

Since I always shoot with a fixed custom WB, the Camera multipliers never changes between different shots, but the read out on temperature and tint value in ACR still changes if I apply different camera profiles, so I guess the temperature and tint value is matrix dependant, not calculated from the original camera multipliers

[madmanchan]

That depends on how you've set the white balance settings.

What you're referring to is the As Shot white balance setting, which is the default.  This simply uses the WB values as recorded originally by the camera and written into the raw file.  WB in this case is recorded as a set of gain factors (or equivalently, "camera neutral" values).  The translation between these neutral/gain values and white points (or temperature and tint slider values, which is what you see in the UI) is dependent on the camera profile (color matrix values).  So when you switch between different camera profiles, the translation between the camera-recorded gain values and user-visible temperature/tint values can change, and this is expected behavior.  When switching between Adobe-provided profiles (e.g., Adobe Standard and Camera Standard) for a given camera, you generally don't see these values change because we use the same ColorMatrix tags for all the profiles of a given camera model.

When you set a custom white balance setting (e.g., Daylight from the popup, or set your own temp/tint slider values) then ACR applies white balance using your chosen white point, not by using the camera-recorded neutral/gain values.  Therefore the temp/tint slider values remain the same regardless of which camera profile you choose.