Camera Calibration profiles: think of them as color palettes not as absolutes

[AlterEgo]

in recent Adobe Standard profiles the HSV Lut part has indeed become larger,
and unlike with former Adobe Standard profiles it can't be completely removed anymore via the DNG PE Chart Wizard + Clear Color Adjustments,
but then the Lut is still getting downsized that way, it is different from the starting Lut with Adobe Standard, which finally makes it a different profile
- as confirmed by looking inside the profiles with the dcpTool, or from what I see in practice in LR/ACR.

my point was that before you can just leave the purely matrix part from the original dcp file and it was quite OK for non challenging scenarios (LUTs were just fixing minor issues, I may say so) and nowadays you can't really

[AlterEgo]

I just did

if I am not mistaken your camera (Pentax) is quite old and Adobe profiles for it are not exactly what are they now, so keep that in mind - whatever works for you is not necessarily the case for somebody who has a shiny new camera with a different construction of Adobe profiles for it

[Tim Lookingbill]

if I am not mistaken your camera (Pentax) is quite old and Adobe profiles for it are not exactly what are they now, so keep that in mind - whatever works for you is not necessarily the case for somebody who has a shiny new camera with a different construction of Adobe profiles for it

I'll have to take your word on that since I can't tell how, when or where LUTs affect the preview. So if this is the case with more modern versions of Adobe Standard as a starting point, why not pick another Base Profile that doesn't have LUTs or doesn't screw up the starting point preview of the CCchart image. I have no idea what Base Profiles that have LUTs do to Raw images. It would help to see this in action and actually be able to connect that the LUTs are causing it.

I've opened downloaded Raw sample images from more modern Canon & Nikon models and noticed the huge myriad of choices of Adobe supplied camera profiles. And every one of them do funky things to the preview both in overall contrast, role off into shadows AND hue/sat distortions in individual colors. How this affects the performance of creating CCchart Wizard profile to get more accurate results is a mystery.

So am I to understand on a more modern camera Raw file that if Adobe Standard is embedded (thus chosen) in the converted DNG CCchart target image and first opened in the DNG PE app that the preview will change when selecting "Clear Color Adjustments"?

The reasons for my specificity on this subject was to address posters like "mouse" who made this comment...

Immediately after opening an image, in the right panel, there is a drop down list for "Base Profile".  Here one can select from a number of Adobe profiles for your camera.  After one makes this selection. the next step is to select the "Chart" tab and proceed from there to construct the profile.  What I have found is that the selection of the "Base Profile" has an visible effect (sometimes very significant) on the resulting profile produced by the editor.

Choosing different "Base Profiles" in the drop down menu in the "Color Tables" panel in DNG PE will change the preview the same as if you'ld done it first BEFORE converting to DNG format in ACR/LR. Change the starting preview by changing the "Base Profile" in DNG PE will cause the Chart Wizard to adjust more or less depending how wacked the newly selected Base Profile changes the preview. The farther away from accurate the newly chosen Base Profile messes with the preview the more the Wizard has to fix which can introduce distortions mainly because this free app is meant to do small and subtle movements to Color Tables and not act as a color restoration tool.
 

[Peter_DL]

my point was that before you can just leave the purely matrix part from the original dcp file and it was quite OK for non challenging scenarios (LUTs were just fixing minor issues, I may say so) and nowadays you can't really

and you could be right (actually you are right), however,
the suggested procedure via the Chart Wizard + Clear Color Adjustments still works fine also with recent Adobe Standard profiles,
whether or not or even because said ""baseline matrix profile"" goes beyond a pure matrix, and I should just replace this term in my earlier posts #14 and #16 by something broader like "core profile".

Any quotes or insights on the background of this change in design with Adobe Standard ?

It seems to contradict what we heard earlier from Eric Chan:
quote >> all I meant to say in my post was that some more recent sensors are not too far off from satisfying the Luther-Ives condition, and hence good scene referred profiles can be obtained with a simple 3x3 matrix.<<


More in response to Tim, - the differences between the different profiles, Adobe Standard old/new, before/after the Chart Wizard, with or without step 5 Clear Color Adjustments  seem to correlate well with the .dcp file size [kB] as a basic indicator, in line with what we see inside with the dcpTool or finally in LR/ACR (I would not trust the DNG-PE preview only)… - if of help.

--

[AlterEgo]

It seems to contradict what we heard earlier from Eric Chan:
quote >> all I meant to say in my post was that some more recent sensors are not too far off from satisfying the Luther-Ives condition, and hence good scene referred profiles can be obtained with a simple 3x3 matrix.<<

he said that in 2009 and dcp profiles were evolving - hence when quoting him you always need to check the timeframe... yes, you could and can have a quite good color transform using just a matrix (linear) conversion in majority of cases, but people were looking to get good results in more extreme/as much as possible situations hence dcp moved to using LUTs heavily instead of relying on using LUTs as some kind of postprocessing part to correct issues after a normal matrix color transform... that's my uneducated understanding of the situation...

[Tim Lookingbill]

More in response to Tim, - the differences between the different profiles, Adobe Standard old/new, before/after the Chart Wizard, with or without step 5 Clear Color Adjustments  seem to correlate well with the .dcp file size [kB] as a basic indicator, in line with what we see inside with the dcpTool or finally in LR/ACR (I would not trust the DNG-PE preview only)… - if of help.

You have to trust the preview because that's the reason we create custom profiles using the CCchart Wizard and then attempt further minor tweaks to HSL in the Color Table panel. We're using our eyes to perform those edits!

The ACR/LR preview with a starting base profile (pick one) before & after converting to DNG format looks the same when first opened in DNG PE with no edits or Wizards. Do a screengrab with both previews (ACR/LR & DNG PE) of the CCchart target image and check the lab numbers of screengrab file in Photoshop. They should be identical or at least within 5 points in either of the three Lab channels.

Then run the Wizard as usual and see how it changes. Do a screengrab. Then go back in ACR/LR and open the CCchart target image and start off with a different Base Profile, convert to DNG and run the Wizard and do a screengrab. Compare the Lab numbers of the two CCchart target image screengrabs (that had different starting Base Profiles) in Photoshop.

If there are wildly different Lab numbers between the two (affected by starting out with different Base Profile starting points) then you can confirm that the increase of file size in the .dcp is taking into account these extra color matrices/tables/LUTs and that they are affecting the results.

[Peter_DL]

he said that in 2009 and dcp profiles were evolving - hence when quoting him you always need to check the timeframe...

and the early date of the quote actually supports the point:  sensors do have improved since that time,
and a best-fit-matrix should suit better than ever in the past (what I can confirm from my camera history).
It is the design of the Adobe Standard profile which has changed.

yes, you could and can have a quite good color transform using just a matrix (linear) conversion in majority of cases, but people were looking to get good results in more extreme/as much as possible situations hence dcp moved to using LUTs heavily instead of relying on using LUTs as some kind of postprocessing part to correct issues after a normal matrix color transform...

With the recent Adobe Standard profiles, from what I see, all the tweaks are now in the HueSatDelta table which is applied "straight" on top of the matrix (two matrices and two HueSatDelta tables according the two illuminants, to be precise).  Whereas the former Adobe Standard profiles included a single LookTable which was reported to be applied at a later stage, after the exposure control in terms of former PV2010.  Seems that the idea of having lightness-dependent hue twists in the profile was abandoned.

--
Late edit:  just realized that the recent Adobe Standard profiles indeed include both:
the new HueSatDelta tables as well as a LookTable.  Nicely illustrated here.

[Tim Lookingbill]

With the recent Adobe Standard profiles, from what I see, all the tweaks are now in the HueSatDelta table which is applied "straight" on top of the matrix (two matrices and two HueSatDelta tables according the two illuminants, to be precise).  Whereas the former Adobe Standard profiles included a single LookTable which was reported to be applied at a later stage, after the exposure control in terms of former PV2010.  Seems that the idea of having lightness-dependent hue twists in the profile was abandoned.

After reading the dcpTool site links I couldn't find any real images (not color patches) that showed what you're talking about, Peter. But your mentioning PV2010 vs PV2012 sparked my curiosity to try this out. Maybe you can confirm in the LR4.4 screengrabs below if this is what you're describing.

I opened the dual illuminant custom DNG embedded ACR 6.7 PV2010 edited pumpkin image in LR4.4 and switched to Adobe Standard and then switched to PV2012 (clicking on warning icon) where I got the "Before/After" previews which darkened and added more definition and contrast and then took a screengrab. Then I just increased exposure from +70 to +1.36 and took another screengrab. (The left side "Before" top & bottom previews are the same.)

Note the lack of saturated yellow on the right after increasing exposure where I'm assuming LR4.4 is using a different Adobe Standard for my camera as well.

[Tim Lookingbill]

I did some further experimenting using the same DNG CCchart target image I shot back in 2008 when building a custom profile for ACR 4.6 PV2003 and using an older DNG PE Chart Wizard and it appears the latest DNG PE app builds profiles that render with less saturation and yellowish hues to orange colors. (See the ACR 6.7 PV2010 screengrabs below. Same results in LR4.4 PV2012.)

It also appears choosing the starting Base Profile (either Adobe Standard or ACR 4.4) in the Color Table panel in the current version of DNG PE doesn't affect the final CCchart Wizard profile. I made two with each starting Base Profile and applied both profiles in PV2010 ACR 6.7 and LR4.4 PV2012 on the same pumpkin image and got the same preview. However, switching back & forth sampling the ProPhoto RGB numbers in bright orange highlights and mids shows the green channel shift one to two numbers up for ACR 4.4 and down for Adobe Standard.

[Peter_DL]

... and it appears the latest DNG PE app builds profiles that render with less saturation and yellowish hues to orange colors.

as noted among the unexpected factors of influence,
(point 2, post #14), the version of the DNG PE.

It also appears choosing the starting Base Profile (either Adobe Standard or ACR 4.4) in the Color Table panel in the current version of DNG PE doesn't affect the final CCchart Wizard profile.

With the former Adobe Standard profiles, the LookTable got completely discarded, automatically by the Chart Wizard,
before the color adjustments as derived from the ColorChecker were set on top of the matrices.

--
Supplementary, added by edit,
I guess this is the quote I was looking for:

Eric Chan wrote (Aug. 2011):  >>When using the Chart Wizard feature of DNG PE, the color matrices are taken from the choice of the Base Profile (ColorMatrix* and ForwardMatrix* DNG tags), and the color tables (HueSatMap1 and HueSatMap2 DNG tags) are replaced by the ones calculated by the Chart Wizard. If the Base Profile has a look table (LookTable DNG tag), it is removed.<<

http://forums.adobe.com/message/3861785#3861785
http://exdeejjjaaaa.blogspot.de/2013/07/ec-others-forumsadobecom.html

[Tim Lookingbill]

Eric Chan wrote (Aug. 2011):  >>When using the Chart Wizard feature of DNG PE, the color matrices are taken from the choice of the Base Profile (ColorMatrix* and ForwardMatrix* DNG tags), and the color tables (HueSatMap1 and HueSatMap2 DNG tags) are replaced by the ones calculated by the Chart Wizard. If the Base Profile has a look table (LookTable DNG tag), it is removed.<<

Can't any of you code fiddlers provide a series of sample image previews that relate to each color manipulation term embedded in these Base Profiles and .dcp recipes that affects the preview so we know what to look for if things are working wrong or right? There's so much stuff going on under the hood for what little changes you'ld expect among all the app version upgrades not to mention added PV version influences that it makes me wonder if it's working as it should. And I don't need someone telling me "Oh your camera's different from mine so that's why that is happening on your end". That doesn't help anyone!

For example if the Base Profile has a look table (LookTable DNG tag) and it is removed what will that look like with or without it in a preview. I need to know where color ground zero is when I convert my Raw to DNG and not have to wonder if there are "color thingys" getting embedded, utilized and/or removed when creating a custom profile using the DNG PE.

I haven't made a custom profile since 2008 because I thought everything was working fine and I accepted the over saturation. Now that I've bought into 3 PV versions and notice preview changes some slight and some just downright odd I'm left wondering how does one tell if things are working as they should.

Case in point when I'm switching from PV2010 to PV2012 in ACR 6.7 the Linear Point Curve gets a custom curve. (See below). Of course I switch back to PV2010 by canceling out of ACR and everything goes back to the original edits of the embedded xmp sidecar. That custom curve does not get applied to all images doing the same conversion and some images going from PV2003 to PV2012 in LR4 get a different shaped curve that may or may not replace an already existing curve.

I know about this point curve replacement by reading about complaints online about it but I'm not frustrated about that. I accept this. I am frustrated about all the doubt that's created in these types of discussions where no one SHOWS what all this .dcp code jargon does to the preview and whether it's necessary to be concerned about it. Or whether these custom replacement PV curves in ACR/LR become part of and/or influence in some way the processing pipeline when going through the trouble of making a custom DNG profile.

Telling someone they should use this version app because "color taggy table kadiddle-hop is now in front and not behind the process" (whatever that means) so use the current version does not help remove the doubt. Then the doubt compounds when you get the current version and you hardly see a difference and are left wondering about whether "color taggy table kadiddle-hop" really made an improvement with my camera but too bad it's not doing anything to your old camera. You need to get a newer camera. That's just horseshit! And it only proves what little these discussions do to help people when the techno talk doesn't get put into action to prove what's going on.

[Peter_DL]

 
Not sure if you really expect (or appreciate) a response, Tim, based on your rant.
Aren't we all in the same boat here.

For example if the Base Profile has a look table (LookTable DNG tag) and it is removed what will that look like with or without it in a preview. I need to know where color ground zero is ...

you can easily check into this on your own, -  happy to explain again:
just do step 5 as I had suggested it. The complete procedure how to extract the core profile from the Adobe Standard profile is compiled below. Not particularly difficult at all.

With the Pentax K100D (if this is your camera) the Adobe Standard profile is of the former type, just including the matrices and a LookTable. When you strip off the LookTable according the suggested procedure, the remaining core profile is what I had called the baseline matrix profile.

Case in point when I'm switching from PV2010 to PV2012 in ACR 6.7 the Linear Point Curve gets a custom curve. (See below).

The definition for zero/Linear has changed in the UI from PV2010 to PV2012:
PV2010 Medium Contrast curve -> PV2012 Linear
PV2010 Linear -> PV2012 inverse of the Medium Contrast curve as shown in your post.

To this point it is solely a UI thing, and it should not be responsible for any differences in tonality when converting from PV2010 to PV2012.
Feel free to open a separate thread on this subject

--
--
How to extract the core profile from the Adobe Standard profile,
with the DNG Profile Editor:
1.  open a DNG of the ColorChecker chart, Base Profile Adobe Standard
(no need to take particular care of the lighting, etc. when shooting the target).
2.  place color circles over corner patches
3.  select: Both color tables
4.  Create Color Table
5.  Edit > Clear Color Adjustments
6.  optional: change Base Tone Curve to Camera Raw Default
7.  File > Export Profile

-

[Peter_DL]

I've been a long time advocate of creating a custom profile for your camera …

That's fine and dandy but here's the point: I want to encourage you to at least try the other profiles to see which basic color palette (the "Profile") works best before you start adjusting any of Lightroom or ACR's sliders. You might discover something you like.

I think the recommendation is particularly valuable when the choice of other profiles comprises
an Adobe Standard profile of the recent type (larger dcp file size), i.e. with a "newer" camera.

I had ignored the Adobe Standard profile for some time (didn't like it in the past),
however, upon re-evaluating it now I’m considering the recent type as a clear improvement,
whether from a technical point of view (less and less work left for the Chart Wizard, in continuation of the previous trend of "matrix getting better"),
or, judging by perception in LR/ACR.

Sorry in case we got too much into the "acres of the thorny brambles, the charts and equations" of DNG profiling here.

Peter

--

[Tim Lookingbill]

Peter, show me, don't tell me procedures you've already outlined.

You won't show in one of your images this "new" Adobe Standard and what it does or doesn't do compared to the old. Yours or anyone else's input on this subject has not been helpful to me at all.

[mouse]

Tim,

I think you and other poster in this thread have drifted too far from Ellis (the OP) original intent.  He is simply suggesting that one use their own camera, make custom profiles, and employ them in the raw converter of their choice.  Then use the profile that gives the best results; which is a purely subjective decision. 

I don't see how displaying images which we have created, for comparison purposes, is really useful.   One simply has to go through the process (in the comfort of your own home) and arrive at one's own subjective conclusion.

[Tim Lookingbill]

Tim,

I think you and other poster in this thread have drifted too far from Ellis (the OP) original intent.  He is simply suggesting that one use their own camera, make custom profiles, and employ them in the raw converter of their choice.  Then use the profile that gives the best results; which is a purely subjective decision.  

I don't see how displaying images which we have created, for comparison purposes, is really useful.   One simply has to go through the process (in the comfort of your own home) and arrive at one's own subjective conclusion.

I'll have to disagree with you that what's being discussed about camera profile dcp color table's affect on color rendering as being off topic. Ellis offered a broad title for interpretation, a word you'll find used several times on that dcpTool site. Anecdotal color issues presented by random users posted on Adobe forum discussions is not evidence that points to any verifiable cause.

This issue affects folks who shoot furniture, jewelry and other products who want to maintain consistency and accuracy in workflow and need to rely on good information that assures this without introducing a lot of doubt and complexity by having to understand "interpretations" of what is happening under the hood with both the image editing tools and the building of custom camera profiles.

There are so many variables that influence the Raw preview during editing that delving into the code mechanics of dcp camera profiles which offer no proof of their influence needs better evidence to determine the source of color anomalies which I feel are the reasons Ellis chose such a fitting title to this thread.

In fact I've been able to duplicate and fix the color anomalies attributed to changes in profiles and Adobe processing engines going back to PV2003 through PV2012 in particular to using the Adobe Standard profile whose suspected version changes have been attributed to changes to the Raw preview as well as affecting the results form the creation of custom DNG profiles when used as a Base Profile.

Here's the variables:

Changing lighting conditions scene to scene having similar colors (yellows, oranges, reds) don't change consistently. Note the pumpkin image I posted where the edits are unknown compared to the another image of furniture (see below) where I'll be specific with the edits as they are applied using PV2010 slider behavior differences, "supposedly" fixed in PV2012 LR4 where I put back in using LR4's "Whites" slider. Way too many variables to attribute to one cause like color tables or is it slider behavior?

The first screengrab is of an image shot in mixed lighting (2800K CFL & window light), choosing Adobe Standard and setting WB by clicking on white paper on top of the dresser drawer. It demonstrates what others complained about in the past about the different affect on color using Exposure slider (more pop and clarity) VS using Brightness (flat). Note the hue changes in yellow and oranges where it looks like white balance has skewed to green.

The second image shows the "Before & After" edit importing a copy of the PV2010 image edits in LR4 and just switching to PV2012 without editing. The third image shows I could duplicate the same behavior WB skew in PV2010 using the "White" slider. Again...Way too many variables.

This does not happen on every image with yellows, oranges and reds using Adobe Standard. Again more variables due to a lack of predictable consistency.