DCP: Using LookTable and HSV with saturation of zero

[rohitp]

I have posted this in Adobe's DNG sdk forum but no replies

I am currently looking at the DCP spec and the DNG SDK, specifically using the LookTable to encode an HSV conversion that does not necessarily preserve grays.
But the way these tables are setup, there does not seem to be a way to increase saturation of a pixel whose initial saturation is 0 (zero), since the values stored are saturation scales.
For example, below are the input HSV values (using DNG SDK's encoding range) and the value it needs to be transformed to.
 
Input= Hue: 2.5 Sat: 0 Val: 0.0666667
Output= Hue: 3.96332 Sat: 0.285995 Val: 0.0977313
 

So, the question I am hoping someone has an answer to is, is this even possible using a DCP file? Other tables in the DCP? If yes, any pointers would be appreciated!

Thank you!

[torger]

Through my software development work with DCamProf, Lumariver HDR and RawTherapee I have quite good experience of the DNG and DCP format.

DCP has some limitations by design, and you just discovered one of them.

A DCP cannot alter the color of gray, and as a side effect a DCP cannot be used to correct white balance. ICC does not have this limitation.

The reason is exactly what you describe, the LUT operates in RGB-HSV space and while hue is modified by addition, saturation and value is modified by multiplication, meaning that if saturation is zero you cannot increase it.

Although not a limitation in the LUT itself, the DNG spec also mandates that the LUT entries for S = 0 or V = 0 must be HSV 0,1,1, that is you cannot scale the grayscale axis in value either. In fact, Adobe's own software like Lightroom does not even look at the LUT for the 0/0 entries (I've tested), it just assumes they're 0,1,1 and renders it like that. This makes all of those entries in the LUT redundant.

And no, I don't think this is good design, but it is what it is.

[rohitp]

Thank you for your reply! I was looking at your program dcamprof and the documentation. Great work!

I have used ICC profiles a lot (am a contributor to the SampleICC project) and as you said, it does not have the limitations that the DNG spec does.

So, as a follow up question, do you have a recommended way of using DNG/DCP files to encode a look that does NOT preserve grays?

Thanks again!

[torger]

It's impossible with DNG profiles, as said the LUT entries for 0/0 is always 0,0,1 and you can't change that without changing Adobe Camera Raw.

You can't change it with the matrix either, it must map white to white (as stated in the DNG spec), if it doesn't Adobe Camera Raw / Lightroom will refuse to load the profile (I've tried).

It's a hard-coded limitation of the DNG profile format :-/

[torger]

I found out another limitation today, namely that due to a bug (or feature depending on how you see it) in the DNG pipeline, you can't have hue shift discontinuity. The DNG pipeline thinks the average of a hue adjustment of -170 and +160 is +5 and not +175 as it should be.

For normal hue corrections it's generally not a problem, but if you design a look you quite easily run into this limitation.

[rohitp]

I have realized that there is the spec document which leaves out a lot of implementation details and then there is the Adobe provided dng_sdk that implements things in a way that do not exactly match the spec. I am assuming you are referring to the dng_sdk implementation of the hue adjustment?

After discussing the colorizing of grays limitation with Adobe it has become clear to me that the spec was/is meant to provide a means of colorimetric interpretation of scene data (thru the camera) rather than providing the ability to impart "preferred" look, which the ICC spec allows with it's saturation and perceptual intents. Moreover with the ability to encode floating point data and custom processing elements with updated version 4 of the ICC profile spec, I see it as a VERY compelling option to use for camera profiles. If only Lightroom supported camera ICC profiles (like capture one does)!

[AlterEgo]

If only Lightroom supported camera ICC profiles (like capture one does)!

and you can end up w/ the situation when ACR/LR 'd be using ICC container, but limiting how the data stored there (guiding the color transforms)  might be actually used... so it is really not about what you can store in the icc or dcp container, but what the prevailing raw converter on the market does.

[rohitp]

True, there is the data storage part and then there is the data processing part. It is the later where differences start to creep up between different implementations. Some of it can be negated with "reference" processing implementations as done by the dng_sdk for the DNG spec and SampleICC for the ICC spec.

[torger]

If DNG profiles were really used for scene-referred colorimetric profiles, then I'd understand it, but even Adobe doesn't use them like that.

As soon as you apply a tone curve you're making it output-referred and Adobe does that on all their profiles. And then they have the "LookTable" which even according to the spec is intended for making a subjective look. I think the problem is that they did not really have full understanding of all aspects of camera color management when the format was designed.

It has some strong advantages over ICC too though, such as that you can define a higher resolution LUT with fewer elements.

But the saying that "ICC is bad for cameras and DCP is great" which has been said over and over again in various DCP vs ICC articles is simply not true. Yes ICC has some issues, ICCv2 being integer only is really messy to work with, and that there is no standard of how to apply it to raw data is pretty severe. However while DCP solves some issues it adds some new too.

Here's a list of issues from the top of my head:
- can't alter hue/saturation of neutral axis
- LUT can't alter value of neutral axis (not problem of format, problem of pipeline)
- can't handle hue discontinuity (not problem of format, problem of pipeline)
- baseline exposure split between DNG file and DCP
- blackrender not clearly defined how it affects rendering
- matrices must be white-point preserving (generally what you want, but there are other ways to optimize a matrix)
- color space cropped to ProPhoto primaries, which is smaller than human gamut (can be issue for scientific applications)
- white balance model built into the profile format, it's not a bad model but it's not so wise if you want others to adopt the format (which of course already have their own white balance models)

[torger]

Before we get too excited by ICC though -- I still prefer working with DNG profiles as it's at least reasonably defined and documented.

The major obstacle with ICC is all the proprietary undocumented raw pipelines that apply ICC profiles in different ways, which means that the first part of making an ICC profile is to reverse-engineer the software you're making it for. You need to figure out which type of pre-processing that is made before the ICC profile is applied, and what post-processing that is required afterwards.

Capture One is not even compatible with itself, ICC profiles for the Leaf digital backs are applied in a different way than for the others.

A very neat thing with DNG profiles is that you have two LUTs, one for colorimetric correction and one for look (it's not so well carried through as it could be though). Although you could achieve the same with ICC by having two profiles, one colorimetric and one for look, all ICC raw converters I've come by apply only one profile, and typically towards the end of the pipeline.

[AlterEgo]

I think the problem is that they did not really have full understanding of all aspects of camera color management when the format was designed.

just wait for Schewe  he knows better ....

[AlterEgo]

Before we get too excited by ICC though -- I still prefer working with DNG profiles as it's at least reasonably defined and documented.

and also to Adobe's credit their people (some, E.Chan notabley) were more forthcoming publicly about how things are working vs for example P1 (not that they didn't, they did sometimes here for example - but much less vs Adobe).

[torger]

Yes I do appreciate Adobe's relative openness, mr Chan has helped me out with some tech questions,and that they've released the DNG SDK open source -- it's fantastic.

And indeed, Phase One is quite the opposite, hard to get any information from although indeed some small bits and pieces have leaked. It's not like any Japanese company in any case...

I don't blame Adobe for not having made the perfect camera profile format, it's a difficult task and I'd guess it's only one or two people at Adobe that has had input on it during design. Future DNG revisions could improve it. Adobe would probably have put more work into it if the industry has been more accepting of the format. Now it's more or less just a Adobe Camera Raw format only fulfilling the needs of Adobe's color pipeline.