DCamProf - a new camera profiling tool

[GWGill]

And in that case the input curve may only be there to make a mapping trick to get better integer precision (ICCv2 is an old format integer math unfortunately)

Little to do with integer encoding (it's rarely an issue), but to reduce the impact of the limited resolution of cLUT tables. Practical cLUT tables sizes are limited by memory/disk space, net bandwidth & CPU speed.

The ICCv2 standard show two types of input profiles 1) XYZ+TRC tags (matrix plus shaper curves), 2) A2B0 only. The mix of both 1&2 is some non-standard stuff by Capture One as far as I understand, which may be because they are used both as camera profiles and can also be attached to a TIFF file just like sRGB.icm or AdobeRGB.icm.

It's a tag format, so you can add almost any tags you want, including both matrix profile and cLUT tags. To be legal though, the PCS needs to be XYZ if the matrix profile tags are included. I have seen illegal profiles that label PCS as L*a*b* for the cLUT and then includes matrix profile elements, but not every CMM will cope with such a Frankenstein.

[torger]

Little to do with integer encoding (it's rarely an issue), but to reduce the impact of the limited resolution of cLUT tables. Practical cLUT tables sizes are limited by memory/disk space, net bandwidth & CPU speed.It's a tag format, so you can add almost any tags you want, including both matrix profile and cLUT tags. To be legal though, the PCS needs to be XYZ if the matrix profile tags are included. I have seen illegal profiles that label PCS as L*a*b* for the cLUT and then includes matrix profile elements, but not every CMM will cope with such a Frankenstein.

Thanks for the info. I'll have to figure out the cLUT mapping/resolution issue when I implement cLUT support. I think I have an idea of how to do it...

Anyway, that makes Phase One profiles illegal (what a surprise...), as they contain matrix profile tags plus lab PCS for the A2B0. LittleCMS seems to swallow it though as it can display TIFFs with these profiles embedded, and it looks right to my eyes. I don't know which tags it looks at in this displaying case, it would be interesting to know. I guess the A2B0 tag needs to be used somehow otherwise it would not display right.

I've noted that some of the newer profiles don't have TRC and XYZ tags though, but only the A2B0 tag.

(I've got my matrix profiles sort of working in Capture One now by parsing out the TIFFTAG_TRANSFERFUNCTION from their profiling tiffs and applying that as TRC shaper curve. Brightness/contrast looks right, but the colors are too saturated... I have so far no idea why.)

[torger]

Attached a plot of the transfer functions as reported in the profiling tiff by Capture One. A pure gamma 1.8 curve is used as reference (yellow), and then the two curves "linear scientific" (almost gamma 1.8  ), and then "film standard".

This is for the P45+.

Note that linear scientific does not reach 1.0, which I assume is intended to clip away noise close to clipping.

[Alexey.Danilchenko]

Anders,

Thanks for the new version. It does not seem to parse v4 ICC profiles. Had a look at the code and it looks like you parsing the ICC file yourself - any reason why? The DCamProf already requires lcms2 and it has a full loader and acces to all profile internals already there (and handles v2 and v4).

[torger]

Thanks for the new version. It does not seem to parse v4 ICC profiles. Had a look at the code and it looks like you parsing the ICC file yourself - any reason why? The DCamProf already requires lcms2 and it has a full loader and acces to all profile internals already there (and handles v2 and v4).

The reason why is because the ICC standard is very big and if I parse anything DCamProf needs to handle anything. It's much better to block things early so the software internals doesn't have to see things it doesn't understand. To actually generate ICC profiles I need to know exactly how they work, which tags and headers that are included etc. I have deliberately narrowed down DCamProf's ICC handling to only camera profiles. Getting into the huge LCMS2 ICC API, which is huge because it's supporting all aspects of ICC standard, is probably not much easier than making the parser for ICCv2 profiles, and when making the parser I learnt a lot about the format which I needed to know anyway. Only the LCMS2 header file is longer than the complete ICC parsing code in DCamProf. DCamProf's internal ICC representation is very scaled down, only contains the elements required for cameras, it's not a full generic ICC.

I don't love ICC that much that I want to spend lots of time to implement JSON conversion for printer profiles or handle and generate ICCv4 if no popular raw converter requires it

That different raw converters handle ICCs differently, and even throws around invalid ICCs (like Capture One) makes ICC a pain to work with. I just like to have as little pain as possible

(there's still some bugs in the icc2json and icc lut stuff which will be fixed to the next patch release.)

[Alexey.Danilchenko]

The reason why is because the ICC standard is very big and if I parse anything DCamProf needs to handle anything. It's much better to block things early so the software internals doesn't have to see things it doesn't understand. To actually generate ICC profiles I need to know exactly how they work, which tags and headers that are included etc. I have deliberately narrowed down DCamProf's ICC handling to only camera profiles. Getting into the huge LCMS2 ICC API, which is huge because it's supporting all aspects of ICC standard, is probably not much easier than making the parser for ICCv2 profiles, and when making the parser I learnt a lot about the format which I needed to know anyway. Only the LCMS2 header file is longer than the complete ICC parsing code in DCamProf. DCamProf's internal ICC representation is very scaled down, only contains the elements required for cameras, it's not a full generic ICC.

I don't love ICC that much that I want to spend lots of time to implement JSON conversion for printer profiles or handle and generate ICCv4 if no popular raw converter requires it

That different raw converters handle ICCs differently, and even throws around invalid ICCs (like Capture One) makes ICC a pain to work with. I just like to have as little pain as possible

The LCMS2 profile hanling API - that is the one that accesses tags and checks the version is very simple. So is creational one (have a look at dcp2icc sources it creates ICC profile there via LCMS calls and writes LUT to it). I was not suggesting to use it in other capacities or other profiles - so I am not really sure I understang the reasons to code the parsing and reading in full. The only API you need to access profile data is to open profile from a file and to get the tag data. The writing is a bit more since it involves calling various set methods for the header but still not the full CMS API.

[torger]

The code in DCamProf ICC parsing has an error btw - the logic where it determines version of the profile is incorrect and valid V2 profile with version encoded as 0x22000000 is producing error stating that it is V4 profile. These lines

Which software has produced that profile? The version check should not be incorrect according to the standard document, the first byte is major, the second byte is minor + patch, ie 0x22000000 means ICCv22.0.0 not ICCv2.2.0. I'm grateful for help finding bugs, but think it's better to deal with typo bugs in private messages as most Lula readers are not programmers

[GWGill]

Getting into the huge LCMS2 ICC API, which is huge because it's supporting all aspects of ICC standard, is probably not much easier than making the parser for ICCv2 profiles, and when making the parser I learnt a lot about the format which I needed to know anyway.

I really don't recommend cooking up your own (the Firefox guys did that, and made a pigs ear of it). It has taken some of us rather a long time to figure out some of the subtleties, best practices, and interoperability gotchas - far better to leverage Marti's experience and support.

[torger]

I really don't recommend cooking up your own (the Firefox guys did that, and made a pigs ear of it). It has taken some of us rather a long time to figure out some of the subtleties, best practices, and interoperability gotchas - far better to leverage Marti's experience and support.

Making ICCs work for raw converters is full of that regardless, it's a whole different space. The most popular ICC-supporting raw converter, ie Capture One, uses illegal ICC profiles and does some pre-processing outside the ICC's intention. I can't just go for basic use cases and say Capture One is broken, sorry guys. Oh well, I could, but then I could just skip ICC support all-together. I've implemented support for various ICC profiles in RawTherapee and there are at least four specific incompatible types I've had to handle. I'm not going to support them all with DCamProf though.

I'm not into re-inventing the wheel, but it's ~500 lines of fairly trivial code with fixed scenarios, the hardest part is converting to from the various fixed point formats. I may actually switch back to LCMS2 later on (it is a good idea using existing code) and replace those 500 lines, but now this is the fastest way forward. The code is there. I first looked at using LCMS2 but I got very frustrated that there were no easy to find example code for the use cases I needed, made a quickslam hack with own code just to get going and learnt/reapeated a lot of important aspects of the format meanwhile.

LCMS2 will almost certainly be used when the test-profile command gets ICC support, ie for running colors through the profile.

I plan to support Capture One, excluding Leaf-style ICCs, and then any raw converter with basic linear pre-processing, possibly add some trivial thing on request. I won't do Nikon-style ICCs, I won't do v4 ICCs.

[AlterEgo]

Attached a plot of the transfer functions as reported in the profiling tiff by Capture One. A pure gamma 1.8 curve is used as reference (yellow), and then the two curves "linear scientific" (almost gamma 1.8  ), and then "film standard".

interesting to compare that with curves data in .fcrv files from C1 itself

x/y coordinates [0.0...1.0] pairs are coded as 4 bytes x 2 containing an UINT (4 bytes, 32bit, little endian) which shall be divided by 4294967295 ( 0xFFFFFFFF ), for example 4 bytes = "0x86" "0xF2" "0x1A" "0x0A" = 169538182 and / 4294967295 = 0.03947368 = 0.039474

the content (partial) of .fcrv files

[torger]

Without checking I would guess that they are the same, seems logical. I want to reverse-engineer as little as possible of Capture One though, they don't deserve this attention for making it hard to make ICC profiles for them .

interesting to compare that with curves data in .fcrv files from C1 itself

[AlterEgo]

Without checking I would guess that they are the same, seems logical.

but they are not... linear and linear scientific are linear curves as coded in .fcrv files [0.0,0.0] and [1.0,1.0] - at least in recent cameras... so whatever your see is C1 code doing something with each curve when writing the tag

[torger]

but they are not... linear and linear scientific are linear curves as coded in .fcrv files [0.0,0.0] and [1.0,1.0]... so whatever your see is C1 code doing something with each curve when writing the tag

Have you checked the P45+ curves? It was those I plotted. Maybe they are different per camera? I haven't checked but I know there are different sets of curves available depending on camera chosen.

[AlterEgo]

Have you checked the P45+ curves? It was those I plotted. Maybe they are different per camera? I haven't checked but I know there are different sets of curves available depending on camera chosen.

true - may be that camera indeed has different curve - I was checking the very recent ones, with .fcrv = 128 bytes... it seems that older camera are not that linear  

PhaseOneP45+-Linear Scientific.fcrv = 128 bytes and the content indicates 2 points for the curve, so it is linear there
PhaseOneP45+-Linear Response.fcrv = 9 points for the curve - not linear

for comparison

PhaseOneP65+-Linear Response.fcrv
PhaseOneP65+-Linear Scientific.fcrv

are both 128 bytes with 2 points for curves and the only one byte (content) difference, which apparently is a tag that tells C1 that if Linear Scientific then it shall not invent data to paint clipped highlights in a nice way

[torger]

Thanks for the info Alterego, those things help in figuring out what's happening in the C1 ICC conversion pipeline.

[Alexey.Danilchenko]

Which software has produced that profile? The version check should not be incorrect according to the standard document, the first byte is major, the second byte is minor + patch, ie 0x22000000 means ICCv22.0.0 not ICCv2.2.0. I'm grateful for help finding bugs, but think it's better to deal with typo bugs in private messages as most Lula readers are not programmers

My fault - you right, and your code is correct. It is not my profile but some of the users on russian forums. I'll try to get more information about how it was made.

[torger]

Anyway I agree that it's a good idea using LCMS2 for parsing, but I had my reasons not to and now I'm not going to change it this moment. I'm already on the next challenge . Cleaning up code comes later (=usually meaning never )

[Alexey.Danilchenko]

Anyway I agree that it's a good idea using LCMS2 for parsing, but I had my reasons not to and now I'm not going to change it this moment. I'm already on the next challenge . Cleaning up code comes later (=usually meaning never )

I can have a go at doing this but closer to the end of May ...

[torger]

I think I have it working on C1 now, can hopefully release in the next few days.

The saturation issue was simply because the P45+ raw data looks different (more saturated) out from C1 than from Adobe DNG / DCraw / Anyone Else. I don't know why. Maybe C1 makes a pre-matrixing to reduce color space to get a smaller space for the ICC to work on, which probably is a good idea as the LUT has limited resolution.

It would be interesting to know what it is though, because without being able to revert it it won't be possible to make profiles for C1 directly from SSFs.

I haven't checked how it is with other cameras though, the P45+ could be an exception...

[AlterEgo]

I think I have it working on C1 now, can hopefully release in the next few days.

The saturation issue was simply because the P45+ raw data looks different (more saturated) out from C1 than from Adobe DNG / DCraw / Anyone Else. I don't know why. Maybe C1 makes a pre-matrixing to reduce color space to get a smaller space for the ICC to work on, which probably is a good idea as the LUT has limited resolution.

It would be interesting to know what it is though, because without being able to revert it it won't be possible to make profiles for C1 directly from SSFs.

I haven't checked how it is with other cameras though, the P45+ could be an exception...

may be C1 sees its own back and simple does something extra hardcoded there... I mean more than they do for regular dSLR/dSLM/P&S