DCamProf - a new camera profiling tool

[AlterEgo]

So which target would you recommend?

naturally the one that you 1) can measure yourself carefully, accounting for/correcting by reheating the possible spectrometer drift, averaging, etc and 2) can shoot yourself carefully avoiding improper reflections (like the difference between SG anc Classic - one semigloss and one quite matte)

or you can decide that differences does not matter vs other factors for you and just ignore the #1... a lot of people do ignore both #1 and even #2 and "we" like their pictures still

[AlterEgo]

To make a broad appeal product the right thing would be to make a good GUI for the scanning part though

or there might be some 3rd party with a proper "ruki.sys" like the author of DisplayCAL who will build the GUI front-end

[jrp]

You can adjust those aspects. The curve use in the default workflow is Adobe's default curve which many find pleasing, but you can design an own lower contrast one to open up shadows and compress highlights less. The neutral+ has a slight saturation increase over neutral, you can find that in its configuration file and disable it. In fact for those that always do color adjustments manually having a slightly desaturated profile can be an advantage, which indeed is an approach Adobe has chosen lately for many cameras.

The neutral+ profile has some desaturation of neutrals, but also keeps highlight color higher up than a typical Adobe profile which could explain your experience of the skies. This too can be adjusted.

[:]

Personally I don't use the ACR default curve in my profiles, but I've instead matched it with the camera's native rendering and then fine-adjusted to taste.

Thanks.  I've processed a few more images with a neutral+ profile with the ACR adjustment that is meant to avoid the need to adjust color balance when changing profiles.

Generally, it is an improvement over the packaged Adobe Standard Leica SL profile.  The oranges are not unpleasant and it works as advertised: foliage looks as if the sun has hit it, close to neutrals neutralised (this is a tad strong, for my taste, unless you are going to use the Margulis PPW to whack up the colours later), skies are blue, not white, etc.

The camera also produces a profile embedded in each DNG (which I assume is a fairly crude one, as it seems to over-emphasise the reds, for example).  I don't know what that profile contains or whether it would provide a short-cut to getting a dcamprof-rolled profile that has better colours (eg, spectral data, curves, etc)?

[torger]

The camera also produces a profile embedded in each DNG (which I assume is a fairly crude one, as it seems to over-emphasise the reds, for example).  I don't know what that profile contains or whether it would provide a short-cut to getting a dcamprof-rolled profile that has better colours (eg, spectral data, curves, etc)?

You can extract the profile to JSON text format with

dcamprof dcp2json rawfile.dng

the profile looks like this:

{
  "UniqueCameraModel": "LEICA SL (Typ 601)",
  "CalibrationIlluminant1": "StdA",
  "CalibrationIlluminant2": "D50",
  "ColorMatrix1": [
    [  1.258700, -0.523200, -0.149600 ],
    [ -0.361000,  1.084100,  0.027700 ],
    [ -0.091100,  0.167400,  0.207000 ]
  ],
  "ColorMatrix2": [
    [  0.838900, -0.319800, -0.101900 ],
    [ -0.383400,  1.022200,  0.066100 ],
    [ -0.112200,  0.190000,  0.341500 ]
  ]
}

That is a dual-illuminant matrix-only profile without separate forward matrices, which is typical for embedded profiles. As no curve is embedded ACR will add the default curve to this on top rather than presenting it as a linear profile.

Indeed this is very crude. You could make a new full-featured DCamProf profile based on those matrices. If the matrices are designed for a good scene-referred match the result could be quite good I guess. You would derive the forward matrices using CAT and then use the neutral tone reproduction operator when applying a curve. Any decent shot of a CC24 is a better start though...

[jrp]

Thanks.

I've enjoyed trying this out and even getting a good profile out of it. Better than either the XRite or the Adobe DNG Profiler which, surprisingly, starting from the same image produce completely different profiles.

What are your plans for DCampProf?  To me, the most potentially productive next step would be to provide facilities for tweaking the profile  in ways that go beyond having to edit a text file, regenerate profiles and restart lightroom, and repeat.

I quite like the Natural+ profile; it does what you say it is designed to do.  But I would sometimes prefer different emphasis (no lime green foliage, no neutralising of near neutrals, but better separation of colours, eg).  Having something that would allow me to adjust visually would be ideal.

[torger]

Thanks.

I've enjoyed trying this out and even getting a good profile out of it. Better than either the XRite or the Adobe DNG Profiler which, surprisingly, starting from the same image produce completely different profiles.

What are your plans for DCampProf?  To me, the most potentially productive next step would be to provide facilities for tweaking the profile  in ways that go beyond having to edit a text file, regenerate profiles and restart lightroom, and repeat.

I quite like the Natural+ profile; it does what you say it is designed to do.  But I would sometimes prefer different emphasis (no lime green foliage, no neutralising of near neutrals, but better separation of colours, eg).  Having something that would allow me to adjust visually would be ideal.

A GUI for making subjective look adjustments is what I'd like to have the most too, the trial-error process with editing the text file is really tedious. I would put effort into that before making a GUI for say patch matching. However I think that would be a six month project at least and the older I get the tougher it it becomes to both have a full-time job and make additional coding on top.

There's also some strong reasons not to do it, while it would make custom design much easier it's something you would do once for every camera you buy or even less. It would be a pretty advanced GUI for something you use extremely rarely.

I really want it myself though, and having a powerful GUI would most likely give some further insights into how camera colors work in practice, so we'll see.

Other core aspects I'm more curious about investigating further is precision of high saturation colors, and work more with gamut compression. I'm also curious about looking into making a custom color space adapted specifically for general-purpose camera profiling, I have a few ideas there I'd like to test to see if they're feasible. The things that interest me the most is the unmeasurable "psychovisual" aspects of color which plays a central role when you make profiles for general-purpose photography rather than reproduction.

[AlterEgo]

A GUI for making subjective look adjustments is what I'd like to have the most too, the trial-error process with editing the text file is really tedious. I would put effort into that before making a GUI for say patch matching. However I think that would be a six month project at least and the older I get the tougher it it becomes to both have a full-time job and make additional coding on top.

There's also some strong reasons not to do it, while it would make custom design much easier it's something you would do once for every camera you buy or even less. It would be a pretty advanced GUI for something you use extremely rarely.

I really want it myself though, and having a powerful GUI would most likely give some further insights into how camera colors work in practice, so we'll see.

Other core aspects I'm more curious about investigating further is precision of high saturation colors, and work more with gamut compression. I'm also curious about looking into making a custom color space adapted specifically for general-purpose camera profiling, I have a few ideas there I'd like to test to see if they're feasible. The things that interest me the most is the unmeasurable "psychovisual" aspects of color which plays a central role when you make profiles for general-purpose photography rather than reproduction.

I'd say that for may a mass (which is of course a strange thing for a command line tool, but still) user some specific command line options/keys to generate profiles aimed for either @ "landscape" (properly "green" grass and properly "blue" skies + proper constructing LUTs in the areas of such tones) or @ "portraiture" (properly "colored" skin + proper constructing LUTs in the areas of such tones)... granted you can achieve that with a lot of existing options, but I mean specifically a "hoi polloi" options like -aim-skin or -aim-landscape type of... $0.02

[sebbe]

I'd say that for may a mass (which is of course a strange thing for a command line tool, but still) user some specific command line options/keys to generate profiles aimed for either @ "landscape" (properly "green" grass and properly "blue" skies + proper constructing LUTs in the areas of such tones) or @ "portraiture" (properly "colored" skin + proper constructing LUTs in the areas of such tones)... granted you can achieve that with a lot of existing options, but I mean specifically a "hoi polloi" options like -aim-skin or -aim-landscape type of... $0.02

This would be great. The last weeks I tried to figure out the -a paramater on make-target without much success. When I set "AdjustJCh" for one field, this has an impact also on the other field. That's logic to me. But I really don't get, what values I have to set. And with about 5 fields (e.g all skin color related fields), I'm completely lost. Let's say I have A1 (3.05 DE LCh +2.05 -2.08 0.90) and A2 (3.90 DE LCh +2.55 +1.78 2.36), what would be the "AdjustJCh" for these two fields to get them close to 0?

The idea of AlterEgo would be a good workaround for most situations: "for landscape", "for portrait", "for general use".
Or if I can priorise the fields with rankings. Or the hue values are more important, than lightness and chroma.
And of course all within a GUI, where I can the impact directly (after calculations), instead of using "test-profile" each time.

[jrp]

There appear to be two conflicting objectives:

 = make the GUI usable by the masses for intuitive profile refinement, a functional, results-oriented tool
 = make the GUI powerful, giving full direct access to the capabilities of dcamprof, for research purposes

It may be that you need to build the latter in order to get to the former.

I don't think that profiling would be something that you would just use once (we change cameras  every 1-2 years, and we want a range of different "look"s for different types of shooting).

[Alexey.Danilchenko]

I really want it myself though, and having a powerful GUI would most likely give some further insights into how camera colors work in practice, so we'll see.

If you are thinking about going the GUI way and perhaps even cross platform, I'd recommend QT library. A couple of years ago had to do this myself (some cross platform tools for Kodak ProBacks) and doing it to work simultaneously on Mac and Windows was a really easy exercise. Also ported a few Photoshop plugins using QT from Windows to Mac and it was relatively easy task. The learning curve (if you have not used it) is also not that steep...

[torger]

This would be great. The last weeks I tried to figure out the -a paramater on make-target without much success. When I set "AdjustJCh" for one field, this has an impact also on the other field. That's logic to me. But I really don't get, what values I have to set. And with about 5 fields (e.g all skin color related fields), I'm completely lost. Let's say I have A1 (3.05 DE LCh +2.05 -2.08 0.90) and A2 (3.90 DE LCh +2.55 +1.78 2.36), what would be the "AdjustJCh" for these two fields to get them close to 0?

I guess you mean -a on make-profile, that is provide a target adjustment file. This file is used to change the values in the target to the "wrong ones", but in a subjective way. The main purpose is as a side effect to control the result of the matrix optimizer. A typical thing would be to say that the deep blue CC24 patch is lighter than it actually is if you want to avoid to strong blue subtraction in the matrix (but usually -y parameter can solve that problem).

A problem with matrix optimization is that the problem is "unsolvable", if you make one patch match better another match worse and an optimizer tends to pull towards certain solutions regardless of weights (because they're still the least bad ones despite other weights), so to really change the matrix result more directly one have to adjust the target, and that's where the -a comes in. But even when adjusting the target the results can seem somewhat "random" at times, simply because the linear matrix can only fit the data that good.

An advanced technique which I sometimes use is to develop the matrix first, using target adjustment, and then store the matrix and do another run with the LUT against a target without adjustment. Unlike the matrix optimizer the LUT responds very well to weighting, -w and -l, and the LUT can pull errors to zero if you want that (unless you have several patches on the same chromaticity), but some relax is recommended, it's only a target we're matching after all.

Anyway, before jumping into -a you should probably play around with the LUT and -l.

[torger]

If you are thinking about going the GUI way and perhaps even cross platform, I'd recommend QT library. A couple of years ago had to do this myself (some cross platform tools for Kodak ProBacks) and doing it to work simultaneously on Mac and Windows was a really easy exercise. Also ported a few Photoshop plugins using QT from Windows to Mac and it was relatively easy task. The learning curve (if you have not used it) is also not that steep...

I'm more used to GTK via other projects (like RawTherapee), but still I would probably do it in QT as it's better maintained.

[torger]

There appear to be two conflicting objectives:

 = make the GUI usable by the masses for intuitive profile refinement, a functional, results-oriented tool
 = make the GUI powerful, giving full direct access to the capabilities of dcamprof, for research purposes

It may be that you need to build the latter in order to get to the former.

Yes I agree, I think I need to do the later to be able to do the former in the best way. That is by making a powerful GUI one can learn more about "what matters" and put that into a simplified GUI. And for myself I rather have a powerful complicated GUI that almost only I can use and learn a lot from that, than only a simple GUI that doesn't contribute to developing my understanding of camera color.

[Alexey.Danilchenko]

I'm more used to GTK via other projects (like RawTherapee), but still I would probably do it in QT as it's better maintained.

Personally I just find QT a lot easier from portability prospective - GTK after all was not designed to be crossplatform from scratch...

[torger]

Personally I just find QT a lot easier from portability prospective - GTK after all was not designed to be crossplatform from scratch...

QT vs GTK is much like a Mac vs PC or Nikon vs Canon thing in the open-source community, but usually it's politics rather than features that is discussed. QT is the better designed toolkit, but the ownership jumping around between different companies with unclear directions have not been good for the credibility. As  free community-developed open-source the GTK project has better credibility, but it's development/maintenance has not been satisfactory. For the moment it seems to have gained a bit of speed though.

Anyway, I'm not starting this project tomorrow...

[jrp]

Well I made a couple of further profiles and the result is, to me, quite interesting. If I use the -m option to copy the white balance out of the Adobe profile in many pictures there is almost no change between the Adobe profile and the DCamProf ones (natural, natural+).

This is with the latest version of Lightroom; I don't know whether they updated the profile for the Leica SL. Yes, there are differences in very dark, saturated blues, violets and oranges, but these tend not to be critical, in the sense that if they are off, no one will complain, in general.  So, assuming that Adobe have created their profiles using some expensive equipment, DCamProf has more or less matched it and the tweaks in the + profile can be attractive. The DCamProf profile darkens blue skies a bit, for example.

It is striking that these profiles are almost invariable better than ones created with the XRite ColorChecker software, or even Adobe's profile editor.  I should be able to use the latter to identify where the various profiles differ, but haven't got around to figuring it out yet.

There are a couple of things that stand out, however. Grey, overcast clouds seem to render much bluer than they appear in real life;  setting them grey, with white balance, makes everything else look far too warm. This is with both Adobe and DCampProf DCP profiles.  I don't know whether this has something to do with the UV filters that I use, rather than a profile issue.

[sebbe]

There are a couple of things that stand out, however. Grey, overcast clouds seem to render much bluer than they appear in real life;  setting them grey, with white balance, makes everything else look far too warm. This is with both Adobe and DCampProf DCP profiles.  I don't know whether this has something to do with the UV filters that I use, rather than a profile issue.

They are never grey. Pollution filters light and short waves are more affected than longer. That's why anything tend to blue in a certain distance. Our brain does adapt this. So we "think to see", that they are grey.

But beside that, did you check your profile with test-profile? You can look at the errors.tif to see the differences visually. How about to white balance with the preset "cloudy" or a grey card (instead of white balance on the clouds).

[torger]

Well I made a couple of further profiles and the result is, to me, quite interesting. If I use the -m option to copy the white balance out of the Adobe profile in many pictures there is almost no change between the Adobe profile and the DCamProf ones (natural, natural+).

That is good, it simply means that all those profiles have the intention to produce natural and realistic colors. Profiles that are designed for that will all look very similar, it's only when you study details you discover differences. Probably all profiles are using the ACR default curve too which helps in the similar look as the contrast is the same.

Adobe's profiles are not all designed in the same way or even for the same look, so there's great variation depending on camera on what you get. Some profiles are good, some are not so...

[sebbe]

Anyway, before jumping into -a you should probably play around with the LUT and -l.

I'm using both now (-a and -l) and made three profiles for different purposes (good general match, great general match (except A1, which is too bright and not that saturated) and one that reduces very saturated orange to magenta).
It's great to have all these possibilities to tweak a profile.

[jrp]

A question about including gamut compression in the generated profile.

With Lightroom is it preferable to

(a) include prophotoRGB gamut compression and then get Lightroom to compress down further to sRGB or your printer's RGB profile when exporting/printing; or
(b) include sRGB gamut compression in the profile itself?