DCamProf - a new camera profiling tool

[markanini]

I've considered the option of "rebuilding" existing DCPs with DCamProf. The usage case would be profiles that already are colorimetric accurate profile but lack a desired tone curve or look operators BLE, black subtraction etc. Another could be for cross checking.

I'm assuming this would require a script to read the source DCPs data and re-format it into the DCamProfs native .json format?

[torger]

I've considered the option of "rebuilding" existing DCPs with DCamProf. The usage case would be profiles that already are colorimetric accurate profile but lack a desired tone curve or look operators BLE, black subtraction etc. Another could be for cross checking.

I'm assuming this would require a script to read the source DCPs data and re-format it into the DCamProfs native .json format?

If I understand you correctly you want to take any DCP (not necessarily made by DCamProf) which has a good colorimetric base but then has the standard Adobe curve slapped on top without caring about the color appearance distortions it makes, and then make a new profile which uses that colorimetric base but uses DCamProf's netural tone reproduction operator to add the curve (and possibly add look operators too).

To support all cases you would need to make something that transforms DCPs to the native .json format, which for LUTs would be quite complicated and a 100% 1:1 match is not always possible.

However in some cases you can do it with some hand-editing. The key is that the original DCP is structured such that the colorimetric base is in the HueSatMap and the LookTable is not used (or if used applies a crap look you want to strip away). Then you make a DCamProf profile, for any camera, with the desired look. All tone reproduction and look operator stuff will be stored in the LookTable + tone curve.

Then you copy the LookTable and Curve elements into the original profile, but keep the HueSatMap and matrices. By using dcp2json and json2dcp you can edit them as text files.

Some/many DCP profiles are however not made in that way, but there is no HueSatMap but instead only a LookTable. You could then just rename LookTable to HueSatMap and merge as above, but keep in mind that the HueSatMap is applied before exposure adjustment and LookTable after but it in most cases that will probably not matter.

(Using the Looktable instead of the HueSatMap is often used to mimic an ICC profile pipeline, as ICCs are generally applied after exposure adjustment rather than before. Adobe Lightroom has this design for most of their "brand-specific" profiles which I think is made to mimic in-camera JPEG looks, not sure though as I have never compared them myself).

[markanini]

Went perfectly thanks to your clear instructions. Have a nice weekend.

[AlterEgo]

You could then just rename LookTable to HueSatMap and merge as above, but keep in mind that the HueSatMap is applied before exposure adjustment and LookTable after but it in most cases that will probably not matter.

dcptool will move LookTable table(s) into HueSatMap table(s)

[markanini]

dcptool will move LookTable table(s) into HueSatMap table(s)

Would I do that by using dcptool to make a profile untwisted?

EDIT:Turns out the answer is to make the profile invariate rather than untwisting. I cant see any difference when basing on a untwisted profile from an invariate. Perhaps it depends on the profile.

[torger]

A comment about "hue twists" -- this is often presented as something bad and distorting, and if you want constant color appearance regardless of exposure you must remove them.

This ignores the fact that general-purpose profiles have tone curves, and if you want constant color appearance, like DCamProf's neutral tone reproduction operator strives for, there's all sorts of non-linear tricks you need to do in the LookTable LUT to counteract the effects of the curve, which will show up as "hue twists". The LookTable is applied after exposure and the curve directly after that so they follow each-other, that is the LookTable corrections will always match the curve.

That is "untwisting" a profile only makes sense if you intend to use the profile without tone curve. One use case for that would be to prepare files for HDR merging, but then it's better to merge in raw (like in my own Lumariver HDR), or if you don't have software for that use a pure matrix profile as even an untwisted LUT can have other nonlinearities left.

Profiles like Adobe's own also add subjective elements to it, probably stuff like cooling down shadows and warming up highlights (a common adjustment) which is a "true" hue twist, but as long as it's in the LookTable and thus applied after exposure it makes sense.

Another aspect is that if your profiler works with a different colorspace like say CIECAM02 JCh it will not perfectly match up with RGB-HSV, so you get some minor "hue twists" in the HSV space despite that the color is kept perceptually stable in the other space. DCamProf uses CIECAM02 quite extensively.

[markanini]

I wonder if my results are what is expected.
Basically I'm doing:
dcp2json profile.dcp profile.json
dcamprof make-dcp  profile.json new-profile.dcp

I've tried using profile.json in different states
As is
ProfileLookTableDims changed to ProfileHueSatMapDims and ProfileLookTable cahnged to ProfileHueSatMap1
Made invariate by dcptool
Made untwisted by dcptool
Made untwisted by dcptool +ProfileLookTableDims changed to ProfileHueSatMapDims and ProfileLookTable cahnged to ProfileHueSatMap1

All of the above look identical to me and the historgram. The look is mostly simmilar to a matrix only profile.

[torger]

Make-dcp expects a native profile. Use json2dcp instead to convert from dcp in json format to dcp in binary. Make-dcp probably only reads the forwardmatrix, I had not realised that it was even possible to feed it with a dcp json file.

[markanini]

So my final steps were:
make the source dcp invariate with dcptool
convert the new dcp to json
copy "ProfileLookTableDims" and "ProfileLookTableEncoding" "ProfileLookTable" and "ProfileToneCurve" from an existing DCP/json
convert the edited json to dcp

Now I have the original look but reds are no longer oversaturated etc

Mission accomplished!

EDIT:
A more detailed report on my results if anyone's interested.

On a colorimetric profile it was easy to add the tone curve and neutral+ look operators on top and the results looked excellent.
An early batch Adobe Standard profile required making the the profile invariate first with dcptool which converts looktables to huesatmap tables so that you can add on the tone curve and neutral+ look operators. Also excellent results to my eyes.
On a camera based profile I lost smoothness in skin tones from making it invariate with dcptool. Making invariate looked better than simply renaming looktables to huesatmap tables. Untwisting first gave me hue shifts. In the end the profile in it's original state looked best, even with oversaturated colors so in this case it was a lost effort.

[torger]

I have now released version 1.0.0. It's basically just a re-branded 0.10.5 (no new features) plus a much needed cleanup of the docs.

DCamProf was first released about a year ago, then followed a intensive feature upgrade and bug fix period for six months, and after that it's been resting and tested by users all around. Now I think it's mature enough to be promoted into version 1.0, which is saying that it's ready for a broader audience.

http://www.ludd.ltu.se/~torger/dcamprof.html

[torger]

On a camera based profile...

I don't know how Adobe makes the camera based profiles, but I suspect that they shoot a target and render a profile from the in-camera JPEG. This cannot be split into a colorimetric base + look, but will only be a LookTable (+ curve) which contains a combination of both. It will be just as non-linear as the in-camera rendering is, and thus very hard to reverse into a linear curve profile an adding another tone reproduction operator on top.

It would be the same problem to reverse DCamProf's profiles if all was put into the LookTable, however the DCP's DCamProf makes split them in two parts, colorimetric base (matrix + HSM) + tone reproduction on top (LookTable + Curve).

[howardm]

Macintosh build of dcamprof-v1.0.0 now available at below URL

[AlterEgo]

1.0.0 build for Windows ( mingw = dcamprof.exe + libgomp_64-1.dll + manual, tutorial, pre-release history log / = copies of Torger's web pages / in 3 formats : IE archive .mht, Mozilla archive .maff and regular .pdf ) : https://app.box.com/DCamProf

brief instructions how to build DCamProf yourself on Windows platform :

    download and install TDM GCC from http://tdm-gcc.tdragon.net/download ( tdm64-gcc-5.1.0-2.exe at the moment ), don't forget to check 'openmp' option during installation.

    download and install MSYS ("...MSYS is a collection of GNU utilities such as bash, make, gawk and grep to allow building of applications and programs which depend on traditionally UNIX tools to be present..." ) from https://sourceforge.net/projects/mingw/files/MSYS/Base/msys-core/msys-1.0.11/MSYS-1.0.11.exe/download .

    download LCMS by Marti Maria from https://github.com/mm2/Little-CMS or from https://sourceforge.net/projects/lcms ( http://www.littlecms.com ) and build it = run msys.bat, then go to your LCMS root folder and do './configure', then do 'mingw32-make' -> lcms library will be 'src\.libs\liblcms2.a' .

    download LibTIFF from http://www.libtiff.org  and build it = run msys.bat, then go to your LibTIFF root folder and do './configure', then do 'mingw32-make' -> libtiff library will be 'libtiff\.libs\libtiff.a' .

    download DCamProf from http://www.ludd.ltu.se/~torger/dcamprof.html#download , copy liblcms2.a + lcms2.h from LCMS and libtiff.a + tiff.h + tiffconf.h + tiffio.h + tiffvers.h from LibTIFF to the source code folder, make sure that 'makefile' has the proper library names (replace '-llcms2 -ltiff' with 'liblcms2.a libtiff.a') and add -D_POSIX_THREAD_SAFE_FUNCTIONS to CFLAGS in 'makefile' to account for 'localtime_r' and 'asctime_r' with TDM GCC and build it = run msys.bat, then go to your DCamProf root folder and do 'mingw32-make'... copy dcamprof.exe and libgomp_64-1.dll from TDM GCC to where you want to keep the binaries.

[sebbe]

I have now released version 1.0.0. It's basically just a re-branded 0.10.5 (no new features) plus a much needed cleanup of the docs.

DCamProf was first released about a year ago, then followed a intensive feature upgrade and bug fix period for six months, and after that it's been resting and tested by users all around. Now I think it's mature enough to be promoted into version 1.0, which is saying that it's ready for a broader audience.

http://www.ludd.ltu.se/~torger/dcamprof.html

Thanks for your good work and the helping people around. I'm very happy with my profiles made by your program.

[markanini]

Thanks for your good work and the helping people around. I'm very happy with my profiles made by your program.

I totally agree. With the Neutral+ operators I get a excellent refined look for general photography and a good starting point for further processing. I realise DCamProf has become an indispensable tool for me where previously included lightroom profiles and JPEG colors would leave me frustrated.

[jrp]

I have now released version 0.10.5, ...
I've also included a reference file for the CC24 manufactured November 2014 and later, the data provided by AlterEgo

In case it helps, Xrite have published data here http://xritephoto.com/ph_product_overview.aspx?ID=1192&Action=Support&SupportID=5884&catid=28

[AlterEgo]

In case it helps, Xrite have published data here http://xritephoto.com/ph_product_overview.aspx?ID=1192&Action=Support&SupportID=5884&catid=28

yes, was reported here on Oct 30, 2015 = http://forum.luminous-landscape.com/index.php?topic=100015.msg864549#msg864549

the issue was/is - X-Rite does not publish ___spectral___ data...

[Hening Bettermann]

Macintosh build of dcamprof-v1.0.0 now available at below URL

Hi Howard,
thanks again for your work! - Strangely, this one won't download. If I click the Download button, it says 'Your download has started', but it hasn't, at least I can't see it in the Downloads folder. I have no firewall active, and app downloads are allowed from everywhere in the Safari prefs. Would you have any clue to what could be wrong? I'm on OS X 10.10.5.
Thanks.

[howardm]

Not yet but I do see some weirdness.  I'll try contacting them tmrw unless it auto-fixes itself by then.

[jrp]

Are there any fuller instructions around on how to extract the target image correctly for use with dcamprof.

If I extract it from the raw linear TIFF using Photoshop, for example, is Photoshop going to apply a profile (unless I say it should be unmanaged)?

I would use RawTherapee, as suggested, but the interface is fairly non-standard.  Before I dive into the manual, is there a quick recipe for getting it to do the necessary, if that is preferable to using DCRaw?

Why when running the scanin command do I not use -G 1.0 to tell it that it is a linear file?  Or -p to compensate for the inevitable for perspective distortion? 

Thanks.