DCamProf - a new camera profiling tool

[torger]

A few C1 findings:

- tifftag_transferfunction seems to be it concerning linearizing the profiling tiff data
- C1 linear is very similar to but not exactly the same as gamma 1.8
- some more preprocessing than the film curve is applied before ICC, sometimes extra clip range, and possibly some matrixing (P45+), more investigation can be done, not required for DCamProf though, unless someone wants to make C1 profiles from SSFs.
- C1's own profiles applies a slight S-curve also in the ICC Lab LUT (probed several profiles, always the same), ie linear curve won't be linear for C1 profiles, but generated profiles that are truly linear and apply the tifftag_transferfunction should be linear as far as I can see.
- The only way to see which curve C1's native profile applies so to run data through the LUT and see what you get. The curves embedded in the profile say nothing about the end result.

I have some issues with ultrasaturared blues in my P45+ test profile, may be a clipping issue.

[AlterEgo]

is it possible for the sake of saving some (quite some) time during the (multiple !) experiments to augment "make-dcp" command with options to write 1) profile name and 2) baseline exposure offset  instead of repeating dumping, editing those fields, compiling back ?

[torger]

is it possible for the sake of saving some (quite some) time during the (multiple !) experiments to augment "make-dcp" command with options to write 1) profile name and 2) baseline exposure offset  instead of repeating dumping, editing those fields, compiling back ?

I'll probably look into streamlining the workflow a bit later on and add more options when the program has matured a bit. Meanwhile you could make a script for that using your favourite script language

[AlterEgo]

I'll probably look into streamlining the workflow a bit later on and add more options when the program has matured a bit. Meanwhile you could make a script for that using your favourite script language

that is certainly doable in many ways, yes - so willl do this for now... another question is - may be instructions for DCamProf can be updated that it is possible to use rawdigger (albeit profile edition is not free) to extract raw RGB values to CGATS and then use txt2ti3 to produce .ti3 from that and .cie instead of scanin route ( and you can always position the grid on target better manually )... plus scanin does not put spectral data in .ti3 by itself ... additional advantage of involving rawdigger ( www.rawdigger.com ) is to do flatfielding and txt2ti3 is to provide spectral data from .cie (if they are there of course) when making .ti3 in one step.

[torger]

Thanks for the tip, I'll update the docs when/if I test the workflow myself.

scanin will include spectral data in the .ti3 if the .cie file contains it.

[AlterEgo]

scanin will include spectral data in the .ti3 if the .cie file contains it.

does it ? it seems that it didn't... I just tried with .cie file that has XYZ, LAB, CH and spectral data and .ti3 only has

SAMPLE_ID XYZ_X XYZ_Y XYZ_Z RGB_R RGB_G RGB_B STDEV_R STDEV_G STDEV_B

but no spectral data in it

next I removed XYZ LAB CH data from .cie and left only spectral and scanin says

"D:\argyll\bin\scanin.EXE: Error - Input file 'qpnm.cie' doesn't contain field XYZ_X or LAB_L"

am I missing something then ?

[torger]

does it ?

It probably does not understand the patchtool CGATS format, it the spectral data comes in Argyll's own format it will include it. See the provided cc24_ref.cie as an example.

[AlterEgo]

It probably does not understand the patchtool CGATS format, it the spectral data comes in Argyll's own format it will include it. See the provided cc24_ref.cie as an example.

but txt2ti3 does understand the same format in the same file w/ no issues... so that's something that Graeme G. can comment - how come that scanin does not understand what txt2ti3 does for the purpose of creating the same ti3 file...

anyways will check cc24_ref.cie then, thank you for the note - it did not happen to me to think about that difference

[torger]

I've just released 0.6.1, which can make ICC profiles with LUT, and also supports Capture One.

http://www.ludd.ltu.se/~torger/dcamprof.html

I also made a small adjustment to the native LUT such that whitepoint is always preserved.

I haven't tested it a whole lot, I hope some Capture One users will help out with some ICC testing

[AlterEgo]

I've just released 0.6.1, which can make ICC profiles with LUT, and also supports Capture One.

http://www.ludd.ltu.se/~torger/dcamprof.html

I also made a small adjustment to the native LUT such that whitepoint is always preserved.

I haven't tested it a whole lot, I hope some Capture One users will help out with some ICC testing

ouch.... now to build libtiff - is there a place where people just share prebuilt static libraries  ?

[torger]

ouch.... now to build libtiff - is there a place where people just share prebuilt static libraries   ?

RawTherapee uses libtiff and a bunch of other builds, pre-built stuff for windows is found at, not sure how libtiff is configured though.
http://www.rawtherapee.com/shared/builds/windows/dependencies_for_creating_builds/
Otherwise the compile.txt for RawTherapee can help when it comes building yourself:
https://code.google.com/p/rawtherapee/source/browse/COMPILE.txt
The Cygwin environment has libtiff there from the beginning so there it's just "make" to build it, but of course a pure Mingw build is nicer.

Libtiff is currently only needed for reading the TIFFTAG_TRANSFERFUNCTION that Capture One workflow needs, but in the future I may add other TIFF functionalities.

[AlterEgo]

RawTherapee uses libtiff and a bunch of other builds, pre-built stuff for windows is found at, not sure how libtiff is configured though.
http://www.rawtherapee.com/shared/builds/windows/dependencies_for_creating_builds/
Otherwise the compile.txt for RawTherapee can help when it comes building yourself:
https://code.google.com/p/rawtherapee/source/browse/COMPILE.txt
The Cygwin environment has libtiff there from the beginning so there it's just "make" to build it, but of course a pure Mingw build is nicer.

Libtiff is currently only needed for reading the TIFFTAG_TRANSFERFUNCTION that Capture One workflow needs, but in the future I may add other TIFF functionalities.

thank you... I prefer to have just as few pieces as possible so I want to use mingw of course.

[torger]

Some comments about ICC LUT; this really is a weak spot of ICC. The LUT maps from camera RAW RGB triplets to LabD50 triplets. A problem is that a very large space of the LUT will be meaningless values, a camera never registers 1,0,1 for example, you nearly always have some signal on all three channels.

Capture One seems to have worked around this a little by pre-matrixing to reduce the "raw" color space and thus make more use of the LUT space.

Another issue is that it always have to be 3D even if you have 2.5D data, which means that you have to have quite low resolution or else the ICC file will be huge, a 33x33x33 cube could have been a 190x190 2.5D square instead. With a suitable input curve the low resolution seems to work alright in practice though.

An ICC profile is typically applied after exposure correction (DCP HSM is applied before), and that's fortunate otherwise resolution could suffer if you push a file a lot.

If you bring up a DCamProf ICC in a viewer and look at the Lab gamut you will see a large "box", the reason is that the native profile have information that extends all the way up to +/-128 ICC clip point. Those extreme colors will never be used in a real scene though (that wasted LUT space I mentioned) so don't worry that it looks a little rough and boxy. Capture One's native profiles aren't that boxy, probably because they do some sort of gamut mapping in the profile. DCamProf won't do any gamut mapping and I don't plan to add that either as I don't think it's the job of the profile, if you have out of gamut colors just reduce saturation etc in the raw converter with your color tools until it fits.

[AlterEgo]

0.6.1 build for Windows (mingw = dcamprof.exe + libgomp_64-1.dll) : https://app.box.com/s/h2lwvnzi96c9jkdzv6m7gs9drrsrdfdb

[AlterEgo]

DCamProf-generated icc profile(s) do not work for C1 (v8.2, PC/Win8.1x64)... I used the same json that was technically OK for dcp profile in ACR 9 (I mean that ACR recognized, showed and I was able to select it in ACR UI) and C1 does not recognize or show it.. for simplicity I used just a pure matrix/g1 profile... also a regular tools like http://www.color.org/wxProfileDump.zip or http://www.color.org/ICCProfileInspector2_4.zip can't open it  

CORRECTION - it does recognize them, I simply put in a wrong directory - however the tools do not open profiles still (probably some important tags are still missing ?)

CORRECTION 2 - not so simple... for example when I put a custom made (with argyll) icc profile for my camera in either "C:\Users\<user>\AppData\Local\CaptureOne\Color Profiles" or even directly in "C:\Windows\System32\spool\drivers\color" then C1 sees it, but not the profile I generate with dcamprof - it only can be seen when it goes to where OEM profiles are ("C:\Program Files\Phase One\Capture One 8\Color Profiles\DSLR")... which is certainly wrong, user supplied profiles for C1 shall go to "C:\Users\<user>\AppData\Local\CaptureOne\Color Profiles" location.

[GWGill]

but txt2ti3 does understand the same format in the same file w/ no issues... so that's something that Graeme G. can comment - how come that scanin does not understand what txt2ti3 does for the purpose of creating the same ti3 file...

txt2ti3's whole purpose is to understand a foreign CGATS format and translate them into ArgyllCMS format. For every tool to have all that code in it to deal with extra multiple matching input files (early LOGO/Gretag/X-Rite tools kept device and measured values separate) and do the translation, as well as recognize and make sense of the variations in ArgyllCMS format would make the code totally unwieldy.

[GWGill]

The LUT maps from camera RAW RGB triplets to LabD50 triplets. A problem is that a very large space of the LUT will be meaningless values, a camera never registers 1,0,1 for example, you nearly always have some signal on all three channels.

Yep - that's another reason for the per channel Luts - you can restrict the box size. If you are in XYZ PCS you can also use the matrix to better fit your box around the gamut.

Another issue is that it always have to be 3D even if you have 2.5D data, which means that you have to have quite low resolution or else the ICC file will be huge, a 33x33x33 cube could have been a 190x190 2.5D square instead.

I think ICCV4 allows different resolutions per channel.

[torger]

Thanks for testing.

I have only tested C1 by installing the profile, ie it gets to c:\windows spool color directory like printer profiles, I thought it was like that you should do. It works for me then in C1 7. The profile should not miss any tags in terms of standard, but maybe C1 needs more tags if put in the directory you describe. I shall investigate. Meanwhile just install by right click => install.

I have only tested with C1 7, and Mac OS X default ICC tool (it's really nice), and iccexamin and Argyll's iccdump. Sounds strange that they wouldn't work in color.org tools, I shall investigate that too.

The DCamProf LUT profiles don't have the XYZ or TRC tags like most C1 native profiles have, whose function is unknown to me. If the XYZ and TRC are involved in pre-matrixing/pre-processing somehow it maybe won't work in their native profiles directory.

[torger]

Tested a profile with the color.org tools, it worked. I get one warning in ProfileDump validator though - unknown CMM, but it should not warn on that, the standard clearly states 0x0 should be used if no specific CMM is preferred, and that is also what many other profiles (including C1) use.

The profile was generated by the Linux build though, maybe there was some issue with the windows code, there are some specifics there. Investigating...

Edit: Tested your windows compiled version and indeed it makes invalid profiles, investigating why...

Edit2: the htonl/nthol macro code I copied was bad, it assumes the compiler defines BYTE_ORDER LITTLE_ENDIAN etc, and if it doesn't the code is written such that big endian becomes default ie all words in the icc gets in the wrong order, it doesn't look like that but C preprocessing has some traps... will fix and release a patch soon.

Edit3: updatated code and uploaded, hasn't been out for many hours so didn't care to change patch level so it's still at http://www.ludd.ltu.se/~torger/files/dcamprof-0.6.1.tar.bz2
I don't know if mingw defines __BYTE_ORDER__ like gcc do, but if it doesn't you should get an error message not silent accept big endian like before... if you do get an error with Mingw please let me know what macros it uses for byte order identification.

[AlterEgo]

Tested a profile with the color.org tools, it worked. I get one warning in ProfileDump validator though - unknown CMM, but it should not warn on that, the standard clearly states 0x0 should be used if no specific CMM is preferred, and that is also what many other profiles (including C1) use.

The profile was generated by the Linux build though, maybe there was some issue with the windows code, there are some specifics there. Investigating...

Edit: Tested your windows compiled version and indeed it makes invalid profiles, investigating why...

Edit2: the htonl/nthol macro code I copied was bad, it assumes the compiler defines BYTE_ORDER LITTLE_ENDIAN etc, and if it doesn't the code is written such that big endian becomes default ie all words in the icc gets in the wrong order, it doesn't look like that but C preprocessing has some traps... will fix and release a patch soon.

Edit3: updatated code and uploaded, hasn't been out for many hours so didn't care to change patch level so it's still at http://www.ludd.ltu.se/~torger/files/dcamprof-0.6.1.tar.bz2
I don't know if mingw defines __BYTE_ORDER__ like gcc do, but if it doesn't you should get an error message not silent accept big endian like before... if you do get an error with Mingw please let me know what macros it uses for byte order identification.

ok, I will redo my build then - thank you.