DCamProf - a new camera profiling tool

[Iliah]

but then @ any angle that curve shall be still smoother than others, because it has a better "starting point /at 0 degrees/", no ?

The problem here is to have good mixing after the filter to have reasonably constant SPD across the beam. The setup starts to be bulky and expensive (compared to a single fresnel lens).

[Alexey.Danilchenko]

Lee are publishing pretty accurate Y-curves on their site.
For the purpose of making the light to monochromator more even I would be looking at 281 and 202, maybe double layer.

Thanks will try this combination (I have a Lee swatchbook and that is enough for light source fiber optic lead).

[torger]

This tone curve thing is giving me an headache...

I've attached two images to demonstrate. Both are using the same profile. The first with linear curve, and the other a typical S-curve, Adobe's default. The curve brighten the picture a lot so I have reduced the exposure of the second to make it easier to compare.

The first has rather accurate colors when I compare the calibrated screen with the real thing, as expected. The second has not only increased saturation quite far, but also pushed the reds into some other hue. I know one reason, the curve is applied in Prophoto space (as DNG pipeline does), pushing green to 0 in sRGB. Adobe Lightroom (used here) has applied some sort of gamut mapping though so it's not as bad as it could be. I've tried to apply the curve in sRGB space instead when using RawTherapee, better result. Even without space clipping there are saturation increases and hue shifts though.

In theory the tone curve could been seen as just a compensator for the Hunt and Stevens effects (eg the dimmer screen needs more contrast and saturation to perceptually match a brighter real scene), but it's generally going way over the top. I'd like it not to be the profile's job to deal with side effects of the tone curves, but anyone that's going to use default Adobe curves with DCamProf profiles are going to think that the profiles suck, and indeed they kind of do.

Currently I'm currently out of ideas how to deal with this. For myself using RawTherapee I can apply contrast using a CIECAM02 lightness curve and have good realistic results, but that's not what the general user will be doing, so I guess I need to come up with something. I'm not totally out of ideas, but there are only the painful ones left...

There are a zillion of tone curve models out there in the research world, which all are better than a simple RGB curve (ICC profiles) or Adobe's RGB curve twist (DNG profiles), but DCamProf can just make the profile, not affect which tone curve that's being used, and current color pipelines are locked to simplistic RGB curves.

[torger]

It seems like this tone curve thing is what makes camera color "difficult".

As an RGB curve is so massively bad at doing anything perceptually sane, you need to make a profile that rather than making accurate colors make a precompensation for the tonecurve errors, so you end up at something decent.

The commercial profiles seems to be a mix of subjective look and compensation of tone curve errors. It differs a lot how saturated reds are dealt with for example.

I shall continue with some testing though. If I'm lucky a good compensation can be some gradual pre-desaturation of colors, that is low saturation colors are kept almost unchanged while high saturation colors are desaturated more, and out of sRGB/AdobeRGB colors even more. If that works the profile can stay 2.5D.

In the painful case I need to make the profile 3D with the sole purpose of compensating the tonecurve's wrongdoings. In a way that could be a cleaner design though. In that case I'd keep subjectivity out of the design process and the native profile format, it would always be linear, and tonecurve compensation would only be an option in the make-dcp command. It would then just add a 3D LookTable on top which compensates for the tonecurve. It would be quite simple to make a "mathematically exact" compensation this way, if you know what the end result should be. A 3D LUT would be less robust though, if you change the curve the compensation would no longer match, but maybe that's not a big issue.

With ICC profile you cannot add a separate look-table on top, but one can merge everything into the same 3D CLUT, for the user it will be no difference.

To do the 3D way I need to decide on what the desired tonecurve look should be... work in progress...!

[Hening Bettermann]

Hi Torger

Many of the technical details in this thread are far above my head. Yet I dare to come up with a little comment. I attach 3 screen shots. 1-shows your linear jpeg opened in PhotoLine. 2-with an S-curve in RGB, which shows the same color shift as yours. And 3- the same S-curve, but applied in HSV, using the V channel only. No color shift (visually judged). It seems to me the solution to the curve problem should not be searched in the profile. But I can see, it is something the USER would have to do...

Best regards - Hening.

[torger]

Hi Torger

Many of the technical details in this thread are far above my head. Yet I dare to come up with a little comment. I attach 3 screen shots. 1-shows your linear jpeg opened in PhotoLine. 2-with an S-curve in RGB, which shows the same color shift as yours. And 3- the same S-curve, but applied in HSV, using the V channel only. No color shift (visually judged). It seems to me the solution to the curve problem should not be searched in the profile. But I can see, it is something the USER would have to do...

Thanks for the feedback. Indeed, the easy way out is to say "the profile gives you accurate colors, but the raw converter messes it up with its ancient RGB curve, you need to use some other method to add contrast". I can certainly live with that myself, but in terms of making profiles for wider use it's probably not so good strategy. The sad thing is that a neutral accurate profile will in Lightroom, Capture One etc produce less realistic colors (at least for the saturated ones) than some of the bundled ones, as the bundled ones are designed with the tone curve in mind. Unless you lock it to a linear curve, but for a decent look you need to add at least a little contrast.

Designing for a specific curve and contrast makes the profile less generic and means that you cannot use it for reproduction work, so it's really not a nice option. But the fact is there -- most current raw converters use primitive color pipelines from the 1990s and that makes it impossible to have one profile with neutral accurate color that can be used with different amounts of contrast at still produce realistic colors.

People tend to think that you can use a profile with any "film curve" (in Capture One you can select it separately), but the profiles are clearly designed for the default film curve. Capture One's profiles produce too desaturated results when used with a linear curve, because they're not designed for that. DCamProf's profiles produce accurate results with linear curve, but then instead too saturated and sometimes visibly hue-shifted results when used with a curve.

I don't have a finished answer on what's the state of the art contrast curve today, but preliminary I've got very nice results with a CIECAM02 lightness curve. CIELab lightness which in theory would be the same thing as CIECAM02 lightness is not good at all, it desaturates color, not realistic result. The reason being that the CIECAM02 is simply a more recent and better color appearance model.

What raw converters should be doing is have a modern color pipeline designed for digital photography rather than being nostalgically stuck with "film looks", in that way the profile would not be involved in tone reproduction or tone mapping at all but you would control that in the raw converter and you could choose both realism and make a subjective look. The only raw converter that allows this type of workflow I know of is RawTherapee (the oldschool way is the default though). My goal with DCamProf is to work with popular widely used raw converters though so I need to adapt to their workflows.

[torger]

I wonder what Adobe were thinking when they made the DNG pipeline apply the tone curve in prophoto space. To me that seems to be a rather bad idea. Any fairly saturated color will be clipped due to this, and Lightroom "repairs" it with some sort of gamut mapping. It makes the look really unpredictable so I understand that people need to handtune their profiles.

Capture One applies their RGB curve in some smaller space, which it is seem to depend on camera. That's probably a better approach, but still not well-defined.

[Alexey.Danilchenko]

I wonder what Adobe were thinking when they made the DNG pipeline apply the tone curve in prophoto space. To me that seems to be a rather bad idea. Any fairly saturated color will be clipped due to this, and Lightroom "repairs" it with some sort of gamut mapping. It makes the look really unpredictable so I understand that people need to handtune their profiles.

Capture One applies their RGB curve in some smaller space, which it is seem to depend on camera. That's probably a better approach, but still not well-defined.

A small correction, they don't apply it in just ProPhoto - they apply it nearly at the end of the pipeline in linear ProPhoto. That is before converting to gamma corrected ProPhoto.

[torger]

A small correction, they don't apply it in just ProPhoto - they apply it nearly at the end of the pipeline in linear ProPhoto. That is before converting to gamma corrected ProPhoto.

Indeed, I am often a bit sloppy when talking about color spaces. Used to floating point since ten years gamma has lost its use, so I meant prophoto primaries in linear space. I shall try to be clearer, it's not the first time I confuse people with my sloppy color space terminology :-)

[Hening Bettermann]

[...] The only raw converter that allows this type of workflow I know of is RawTherapee [...]

Hm - I use(d) Iridient that way. And if memory serves me, one could do it in ACR as well. I use(d) the linear camera-own profile both as an in and out profile. This requires a gamma matrix profile, which is built with the assumption of a linear sensor response. This should be theoretically OK with the 5D2. And I have naïvely done the same with the a7r, and become first now aware of that the compressed raw means the sensor is not linear. I have not examined this any further yet.

edit: Ooops, no, ACR does not allow you to choose a linear out profile.

[Alexey.Danilchenko]

And I have naïvely done the same with the a7r, and become first now aware of that the compressed raw means the sensor is not linear. I have not examined this any further yet.

For all intents and purpouses the raw is linear. The tonal curve applied to compress (lossy compression) the raw. Raw converters then apply it  in reverse to unpack the raw data. The raw values are linear because otherwise you would not be able to white balance it easily.

[Hening Bettermann]

Oh thank you! That is certainly good news to me!

[AlterEgo]

Oh thank you! That is certainly good news to me!

and when things are getting non linear you hence have WB issues - like deeeeeep shadows.

[Hening Bettermann]

and when things are getting non linear you hence have WB issues - like deeeeeep shadows.

And hiiiighlights? Obviously this is the problem - but what do I make of this? Is the response of the sensor+raw converter linear, or not? Does the raw converter apply white balance before or after linearising? Is this the correct question? The practical question for me is: Can I make a gamma matrix profile for the a7r based on the assumption of linear sensor response? And can I continue to use this both as the in and out profile? Or do I need to make 2 LUT profiles? What I want is a linear output profile.

Are we side tracking the thread, Torger?

[AlterEgo]

Is the response of the sensor+raw converter linear, or not?

you are not dealing with "sensor", you are dealing with the numbers that firmware writes in raw file... so for profiling purposes you try to get those numbers in a "linear fashion", means you expose so that any non linearities can be avoided ... that is you do not underexpose and you do not get close to clipping - based on how particular camera behaves here and there.

Does the raw converter apply white balance before or after linearising?

naturally after

[Hening Bettermann]

thank you for clarifying!

[Iliah]

>> Does the raw converter apply white balance before or after linearising?
> naturally after

Yes, if linearization is a part of raw unpacking and decoding
No, if you mean full well limited cameras, like some Panasonics are.
There are more woodpeckers than one can possibly count.

[AlterEgo]

No, if you mean full well limited cameras, like some Panasonics are.

you mean that with Panasonic you can have each sensel data recorded in raw to the max possible DN /no adjustment by firmware to so max limit/, so you have non linear data if you take more then one sensel at once for your purposes - they /sensels/ all will have (might have) different well capacities... hence  non linearity.

[Iliah]

you mean that with Panasonic you can have each sensel data recorded in raw to the max possible DN /no adjustment by firmware to so max limit/, so you have non linear data if you take more then one sensel at once for your purposes - they /sensels/ all will have (might have) different well capacities... hence  non linearity.

_Some_ Panasonic cameras at the lowest ISO setting are full well limited, so each sensel charged close to max (within 1 stop of saturation) is non-linear, Panasonic has the linearity limit in makernotes, and we informed Phil of that tag - so it is covered in exiftool.

[torger]

Searched some more on tonecurve stuff, and it's not that easy to find any obvious solution to the problem. In the research world this is called "tone reproduction operator", and they have since 10+ years left the simplistic one global tone curve approach and are more into spatially variable operators, that is what we usually call tonemapping. A profile can't do scene-dependent corrections so we can't use that kind of stuff.

The traditional RGB tone curve is not only about creating a subjective look, but also to fix appearance issues, mainly Stevens and Hunt effects (contrast and colorfulness appears higher in a real bright scene than if reproduced with lower luminance on a screen). Digging down you also have Helmholtz-Kohlrausch effect and Bezold-Brücke hue shift effects. All these effects are scene-dependent of course, but you could say a fixed tone curve roughly matches a sunny day, and it's probably good approach. It will make duller scenes look more contrasty than the real deal but that's probably how people will want to make it look anyway.

The big name raw converters are in any case not designed for scene-dependent appearance modeling, we have this fixed tonecurve to relate to. And then my idea is that I want a tone curve which changes contrast, but colors appears to be of the same hue (and saturation) as before. It's there the current raw converters with their RGB curves fail miserably, making a linear colorimetric profile produce bad colors. But a profile can always compensate to give you any result you'd like, and it's exactly what the big names do (with varying success). My plan is to make it possible to make tone curve compensation with DCamProf profiles as well, but using some good appearance model rather than relying on golden-eye subjective hand-tuning which many commercial raw converters seem to be doing.