DCamProf - a new camera profiling tool

[Alexey.Danilchenko]

I was trying with Visual C++ (not the most recent) - does not work. Will have go with mingw.

Had a good look through the sources and your code does seem to fit very well to C++ (it is already kind of structured that way with data structures and supporting functions for them).

[torger]

C++ with its flexibility suits pretty much everything, it's a great language. I am first and foremost a C systems programmer on Unix though so when I want fast results I use the tools I know and enjoy best. In my professional life it sometimes feels that more time is spent learning new languages and tools of the day than solving actual problems. When I do stuff for myself I rarely do it to become better at some language, then I just use the tools I like and know best and get on to problem solving.

That Microsoft fails to support the C99 standard, which is a known issue, is pretty much a scandal but it's not uncommon for the big software companies to ignore open standards and rather make up their own. Dot net and C sharp is their thing nowadays. Mac OS X doesn't support OpenMP, another well-established open standard, but it looks a bit better there though as they ride on open source and the clang compiler will be including it "soon".

Let us know if it works out compiling with mingw64. I'll remove the long double code to next patch release since it's not needed.

[Alexey.Danilchenko]

All I was saying is that your code is very well structured (so converting it to C++ for someone who wants to do so would be an easy task )

Mac OS clang support for OpenMP (lacking of it) is diappointing. I needed it for some of the things I did so the way I solved it was to build GCC version with it and create Xcode templates to use it from Xcode directly. It does not work with Mac OS native libraries but for me where UI was built with QT (from ground up) it worked quite nicely. I do with they start adding it though with clang to avoid all that.

[AlterEgo]

Let us know if it works out compiling with mingw64. I'll remove the long double code to next patch release since it's not needed.

Torger, can you share the windows binary that you compiled, if you don't mind ? thank you

[torger]

Torger, can you share the windows binary that you compiled, if you don't mind ? thank you

I don't think it works without the Cygwin environment installed, a mingw build can be made standalone statically linked I suppose. The advantage of having cygwin is that you get a command line terminal that actually is not painful to work with, unlike the cmd.exe in windows.

With cygwin you just install from www.cygwin.com, install all devel-packages then just open the command line and

1. get: $ wget http://www.ludd.ltu.se/~torger/files/dcamprof-0.5.3.tar.bz2
2. unpack: $ tar xjf dcamprof-0.5.3.tar.bz2
3. change dir: $ cd dcamprof-0.5.3
4. compile: $ make
5. run: $ ./dcamprof.exe

To shoot targets you'll need Argyll too.

I just released v0.5.3, with some minor additions. Added a txt2ti3 conversion command so you can read raw text spectral data which spectral databases are commonly distributed as. I also added a link to Lippmann2000 spectral database, which unfortunately only may be used for research purposes, but they do have lots of human spectral data.

[AlterEgo]

I don't think it works without the Cygwin environment installed, a mingw build can be made standalone statically linked I suppose.

I last did anything like this (building the executable that way) with C/C++ code several years ago (myself - I am not a software developer by trade), so I was hoping to use a shortcut 

[Alexey.Danilchenko]

I don't think it works without the Cygwin environment installed

It will - all it will need is cygwin.dll bundled with executable.

Here is a version compiled for 64 bit Windows with all needed cygwin dependencies to execute standalone.

[torger]

Thanks Alexey for the info and the build!

[kirkt]

I built v 0.5.3 on my mac (Mavericks) without openMP.  It now generates DCPs that are visible and recognized by ACR and Iridient Developer.  I will also build on my work machine (Yosemite).

Now to test.

kirk

[Alexey.Danilchenko]

Thanks Alexey for the info and the build!

No problem. It could potentially be compiled with mingw but  some posix stuff needs customising (I can probably fix it with a few ifdefs - if it won't be too ugly, I'll send the changes to you if that's ok).

[torger]

No problem. It could potentially be compiled with mingw but  some posix stuff needs customising (I can probably fix it with a few ifdefs - if it won't be too ugly, I'll send the changes to you if that's ok).

Please do

[AlterEgo]

No problem. It could potentially be compiled with mingw but  some posix stuff needs customising (I can probably fix it with a few ifdefs - if it won't be too ugly, I'll send the changes to you if that's ok).

ok, years w/o programming taking a tall... I finally managed to find a stupid elementary school level error when using mingw... of course fopen shall be used with wb/rb flags on windows platform for binary i/o (profio.c file), damn... we were taught this like 25 years ago and I forgot  :-(

so the things necessary to build DCamProf with mingw are :

1) replace mkdir(dir, 0777) with mkdir(dir) in dcamprof.c

2) add #include <pthread.h> in argyllio.c to get localtime_r and asctime_r

3) replace #include <alloca.h> with #include <malloc.h> in tpc.c

4) and damn it ! in profio.c use fopen with wb and rb flags to read/write .dcp (binary) files

that's it

Administrator@CLEVO /z/dcamprof-0.5.3
$ make
gcc -o dcamprof.o -c -I. -Wall -std=c99 -g -O2 -D_XOPEN_SOURCE=600 -D__EXTENSIONS__ -fopenmp  -DDCAMPROF_VERSION=\"0.5.3\" dcamprof.c
gcc -o nmsimplex.o -c -I. -Wall -std=c99 -g -O2 -D_XOPEN_SOURCE=600 -D__EXTENSIONS__ -fopenmp  -DDCAMPROF_VERSION=\"0.5.3\" nmsimplex.c
gcc -o profio.o -c -I. -Wall -std=c99 -g -O2 -D_XOPEN_SOURCE=600 -D__EXTENSIONS__ -fopenmp  -DDCAMPROF_VERSION=\"0.5.3\" profio.c
gcc -o argyllio.o -c -I. -Wall -std=c99 -g -O2 -D_XOPEN_SOURCE=600 -D__EXTENSIONS__ -fopenmp  -DDCAMPROF_VERSION=\"0.5.3\" argyllio.c
gcc -o tps.o -c -I. -Wall -std=c99 -g -O2 -D_XOPEN_SOURCE=600 -D__EXTENSIONS__ -fopenmp  -DDCAMPROF_VERSION=\"0.5.3\" tps.c
gcc -o colmath.o -c -I. -Wall -std=c99 -g -O2 -D_XOPEN_SOURCE=600 -D__EXTENSIONS__ -fopenmp  -DDCAMPROF_VERSION=\"0.5.3\" colmath.c
gcc -o matopt.o -c -I. -Wall -std=c99 -g -O2 -D_XOPEN_SOURCE=600 -D__EXTENSIONS__ -fopenmp  -DDCAMPROF_VERSION=\"0.5.3\" matopt.c
matopt.c: In function 'matopt_find_matrix':
matopt.c:270:12: warning: variable 'de' set but not used [-Wunused-but-set-variable]
     double de;
            ^
gcc -o lut.o -c -I. -Wall -std=c99 -g -O2 -D_XOPEN_SOURCE=600 -D__EXTENSIONS__ -fopenmp  -DDCAMPROF_VERSION=\"0.5.3\" lut.c
gcc -o dngref.o -c -I. -Wall -std=c99 -g -O2 -D_XOPEN_SOURCE=600 -D__EXTENSIONS__ -fopenmp  -DDCAMPROF_VERSION=\"0.5.3\" dngref.c
dngref.c:470:1: warning: 'dngref_XYtoXYZ' defined but not used [-Wunused-function]
 dngref_XYtoXYZ(const double XY[2])
 ^
gcc -o spectrum.o -c -I. -Wall -std=c99 -g -O2 -D_XOPEN_SOURCE=600 -D__EXTENSIONS__ -fopenmp  -DDCAMPROF_VERSION=\"0.5.3\" spectrum.c
gcc -o jsonio.o -c -I. -Wall -std=c99 -g -O2 -D_XOPEN_SOURCE=600 -D__EXTENSIONS__ -fopenmp  -DDCAMPROF_VERSION=\"0.5.3\" jsonio.c
jsonio.c:77:1: warning: 'json_print' defined but not used [-Wunused-function]
 json_print(cJSON *js)
 ^
gcc -o cJSON.o -c -I. -Wall -std=c99 -g -O2 -D_XOPEN_SOURCE=600 -D__EXTENSIONS__ -fopenmp  -DDCAMPROF_VERSION=\"0.5.3\" cJSON.c
gcc -o target.o -c -I. -Wall -std=c99 -g -O2 -D_XOPEN_SOURCE=600 -D__EXTENSIONS__ -fopenmp  -DDCAMPROF_VERSION=\"0.5.3\" target.c
gcc -o xyz2spec.o -c -I. -Wall -std=c99 -g -O2 -D_XOPEN_SOURCE=600 -D__EXTENSIONS__ -fopenmp  -DDCAMPROF_VERSION=\"0.5.3\" xyz2spec.c
gcc -o observers.o -c -I. -Wall -std=c99 -g -O2 -D_XOPEN_SOURCE=600 -D__EXTENSIONS__ -fopenmp  -DDCAMPROF_VERSION=\"0.5.3\" observers.c
gcc -o spectraldb.o -c -I. -Wall -std=c99 -g -O2 -D_XOPEN_SOURCE=600 -D__EXTENSIONS__ -fopenmp  -DDCAMPROF_VERSION=\"0.5.3\" spectraldb.c
gcc -o spectraldb_munsell.o -c -I. -Wall -std=c99 -g -O2 -D_XOPEN_SOURCE=600 -D__EXTENSIONS__ -fopenmp  -DDCAMPROF_VERSION=\"0.5.3\" spectraldb_munsell.c
gcc -o spectraldb_cc24.o -c -I. -Wall -std=c99 -g -O2 -D_XOPEN_SOURCE=600 -D__EXTENSIONS__ -fopenmp  -DDCAMPROF_VERSION=\"0.5.3\" spectraldb_cc24.c
gcc -o spectraldb_kuopio_natural.o -c -I. -Wall -std=c99 -g -O2 -D_XOPEN_SOURCE=600 -D__EXTENSIONS__ -fopenmp  -DDCAMPROF_VERSION=\"0.5.3\" spectraldb_kuopio_na
tural.c
gcc -o dcamprof -fopenmp dcamprof.o nmsimplex.o profio.o argyllio.o tps.o colmath.o matopt.o lut.o dngref.o spectrum.o jsonio.o cJSON.o target.o xyz2spec.o obse
rvers.o spectraldb.o spectraldb_munsell.o spectraldb_cc24.o spectraldb_kuopio_natural.o liblcms2.a -lm

Administrator@CLEVO /z/dcamprof-0.5.3
$

compiled .exe = https://app.box.com/s/mqv2dwhv0voswlc9sf58z8e1va21yb7l   ( dcamprof.exe and libgomp_64-1.dll )

[ErikKaffehr]

Hi,

One interesting aspect of DCamProf is that it can generate DCP profiles using different targets. I have earlier used one of Wolfgan Faust's IT-8 charts for checking rendition accuracy of my P45+ back.

With DCamProf it is possible to generate a DCP profile using different targets while Adobe DNG Profile Editor and ColorChecker Passport are limited to the ColorChecker.

I got a DCP profile foe my P45+ from Anders Torger that was based on test shot I made earlier. What I found is that the profile is pretty accurate, probably better than any of my profiles. BUT it needs a different WB setting in Lightroom. Setting WB on a ColorChecker gives a different WB with the DCamProf profile than with Adobe Standard Profile. Once WB is set correctly, the profile is very accurate.

There is an issue, that some of the light grey samples are to dark.

Excellent job from Anders. I hope he will continue to develop the tool.

Best regards
Erik

[torger]

Thanks AlterEgo, will fix to next patch release.

ok, years w/o programming taking a tall...

[torger]

Thanks for testing, I shall look at why it renders darker. In principle DCamProf does not care about absolute lightness as it's a 2.5D correction it makes, but it looks like there may be some relative lightness error too.

The WB thing is interesting. Few cameras have their preset "daylight" white balance at exactly D50, and DCamProf will correct for exactly D50 if instructed to do so, this means that it calculates which custom white balance is required to fulfill that. Conveying that information is not supported by DCP though so one have to set white balance using color pickers etc. The preset white balance will as it doesn't exactly match D50 not render neutral. For varying light conditions like in landscape this generally does not matter as it will be off anyway.

I may look into the possibility to calibrate not for D50 but for the camera's preset white balance so one will get neutral white with the preset. You would still call the light "D50" in the profile probably as there is no EXIF name that can match any light source, and the actual calibration illuminant used when lighting the target may be slightly different, this does not affect the calibration precision though, just the temp/tint information which indeed will be more wrong with this type of calibration so I'm still not sure it's a good idea. I haven't checked but I'm quite sure Lightroom ignores the DCP matrices for temp/tint numbers and always uses the builtin though. Otherwise the temp/tint numbers would change slightly when you switch profiles.

Hi,

One interesting aspect of DCamProf is that it can generate DCP profiles using different targets. I have earlier used one of Wolfgan Faust's IT-8 charts for checking rendition accuracy of my P45+ back.

With DCamProf it is possible to generate a DCP profile using different targets while Adobe DNG Profile Editor and ColorChecker Passport are limited to the ColorChecker.

I got a DCP profile foe my P45+ from Anders Torger that was based on test shot I made earlier. What I found is that the profile is pretty accurate, probably better than any of my profiles. BUT it needs a different WB setting in Lightroom. Setting WB on a ColorChecker gives a different WB with the DCamProf profile than with Adobe Standard Profile. Once WB is set correctly, the profile is very accurate.

There is an issue, that some of the light grey samples are to dark.

Excellent job from Anders. I hope he will continue to develop the tool.

Best regards
Erik

[AlterEgo]

I got a DCP profile foe my P45+ from Anders Torger that was based on test shot I made earlier. What I found is that the profile is pretty accurate, probably better than any of my profiles. BUT it needs a different WB setting in Lightroom. Setting WB on a ColorChecker gives a different WB with the DCamProf profile than with Adobe Standard Profile. Once WB is set correctly, the profile is very accurate.

here is a purely matrix (single illuminant) profile (no LUTs) made from IT8 target (not individually measured), illuminated by halogen (no 80A filter on light or lens), w/o flatfielding (light was quite uneven), applied quickly in ACR (process 2010), with a raw containing a different target (not IT8, but colorchecker SG), w/o again measurements for that one and raw with ColorChecker SG was from imaging-resource where the light is naturally uneven on target and light was ~5xxxK something (not 2800-3000K halogen for IT8)... so a lot of handicaps here, still :



PS: I was told (by IBorg) that if errors are noticeable (say more than 2dE-whatever) it is more fair to use dE94, not dE2000... dE2000 makes your results better than they are (numbers-wise)

here is dE2000 which makes things better looking



and visually - again, profile made off one target (IT8) applied to a totally different target (ColorCheckerSG) :

[AlterEgo]

Otherwise the temp/tint numbers would change slightly when you switch profiles.

Adobe's profiles for a given camera model /at least for recent generations/ have always the same CMs...

[torger]

Erik, I haven't used patchtool so I don't know how it works. How is the output from "my" profile made? Looking at the grayscale range at the bottom of the IT-8 target the patchtool image shows the brightest patch from "my" profile way too dark, even darker than the second brightest.

If I look with my eyes at the actual output from say RawTherapee there is no such error. I've attached an image of a linear rendering of the actual test target shot with the p45plus.dcp applied. As you can see the brightest patch down to the left is brighter than the second brightest unlike in the patch tool image.

[torger]

Adobe's profiles for a given camera model /at least for recent generations/ have always the same CMs...

But what happens if you change to a custom profile, such as one generated by DCamProf? If I remember correctly the temp/tint will stay the same in Lightroom, which means that it derives those numbers from builtin matrices rather than from the profile. Most likely though if the custom profile is dual illuminant it uses those matrices to get the temperature which is used for matrix/LUT mixing but it doesn't show it in the GUI, so the temp/tint shown in the GUI is just for informational purposes, and I guess Adobe think those cannot be improved upon.

In the actual low level pipeline the white balance end up being multipliers for raw RGB channels, the temp/tint is just a transform using the CM(s)and the precision is of course limited as the camera doesn't measure the full spectrum. It becomes even more approximate if the illuminant used during profiling doesn't exactly match D50 or D65 or whatever the profile is tagged with. Most likely Adobe uses SSFs to derive the CMs, in that case the illuminants can be made exact and CMs will be better at predicting actual temperature.

[ErikKaffehr]

Hi Anders,

I may have messed the conversion in Lightroom, to many sliders...

More testing today...

Best regards
Erik

Erik, I haven't used patchtool so I don't know how it works. How is the output from "my" profile made? Looking at the grayscale range at the bottom of the IT-8 target the patchtool image shows the brightest patch from "my" profile way too dark, even darker than the second brightest.

If I look with my eyes at the actual output from say RawTherapee there is no such error. I've attached an image of a linear rendering of the actual test target shot with the p45plus.dcp applied. As you can see the brightest patch down to the left is brighter than the second brightest unlike in the patch tool image.