DCamProf - a new camera profiling tool

[torger]

As Argyll can't make DCPs and the commercial software seemed limited and have issues with smoothness I decided to make my own software, and I have now just released a first version (it's free and open-source):

http://www.ludd.ltu.se/~torger/dcamprof.html

It's command line only and quite technical, but if you can use Argyll you can use my DCamProf.

Hope you enjoy it.

Now I need some rest and start living again. I've been working nonstop on my spare time for almost two months with this...

[AlterEgo]

thank you

[torger]

There are a few unique features in the software, but perhaps the biggest thing is that you can make profiles directly from the camera's spectral sensitivity functions (=color filter responses), just like many manufacturers do.

You can make it the traditional way too from photos of test targets of course. The ability to experiment with SSFs is a great learning experience though if you want to get a feel of how camera color work. I have links to some sources of SSFs that have been made public through academic work, unfortunately it's hard to come by for very recent cameras.

There's the color spectral data too. I currently have builtin spectral data for cc24, munsell glossy set and spectra from nordic nature. I'd like to have a skin tone database too and more. If any of you know about some useful public spectral databases I'd like to know so I can consider them for inclusion. There are a few commercial databases out there, such as directly from CIE, but those cannot be redistributed in free software...

[TRANTOR]

Great work.

[torger]

Thanks. I was inspired by your work Trantor which showed me the power of using SSFs directly.

[kirkt]

Wow Torger - a lot to think about and learn.  Thank you for the effort!

kirk

[AlterEgo]

Now I need some rest and start living again.

can you (after some rest) add an option to imitate matrix/trc model (like in icc containers) where you have a dcp basically with CM/FM for a color transform & WB operations for ACR/LR and HueSatMap (so it is before exposure adjustments done by a user in ACR/LR UI) that is just imitating (and nothing else) what in "icc" case TRCs (in a simplest case gamma != 1) 'd do per each RGB channel on WB/demosaicked data before matrix conversion into PCS (cieXYZ/D50)... something like "-g <gamma>" option for a simple scenario (more complex might pickup curve(s) data from some text/xml/etc file)

[Bart_van_der_Wolf]

As Argyll can't make DCPs and the commercial software seemed limited and have issues with smoothness I decided to make my own software, and I have now just released a first version (it's free and open-source):

http://www.ludd.ltu.se/~torger/dcamprof.html

It's command line only and quite technical, but if you can use Argyll you can use my DCamProf.

Hope you enjoy it.

Now I need some rest and start living again. I've been working nonstop on my spare time for almost two months with this...

Hi Anders, do get some rest. I know how much it takes do put something like this together. Huge amounts of research, even before the programming effort can begin, and once you are encoding, you discover all sorts of new things that need research.

The effort and making the result available is much appreciated. Unfortunately for this version, I'm an ICC Camera profile user (Capture One), so DCPs are not my main interest. But if you get around to adding ICC profile generation, I'll surely try to compile a version for Windows if I can find some time myself.

Cheers,
Bart

[AlterEgo]

But if you get around to adding ICC profile generation

wasn't there somewhere dcp2icc free conversion tool

[AlterEgo]

I'd like to have a skin tone database too

so everybody here = get naked with your i1s (or whatever) in hands and donate !

[torger]

There are skin databases out there: http://files.cie.co.at/699_Report%20for%20CIE%20R1-56.pdf haven't had times to research yet into if there's someone that can be used in open-source software.

I may look into making ICC profile generator for Capture One (and others) later. The original intention was to make a format-agnostic profile maker, and although DCamProf does store a "color matrix" and "forward matrix" (DCP names) it's generic concepts and it stores a generic format so ICC making should be possible. I need this to settle for a while first, get some early adopters to test etc. There's quite much new stuff to learn for me when it comes to ICC about how the LUTs work (it will probably be a camera RGB directly to Lab D50), and they're a bit tricky to work with as they're integer-based so you need to think it through to not break precision.

AlterEgo, I'm not sure if I follow your feature request. DCamProf can generate profiles with our without HueSatMap. I don't understand why you would want to have some sort of gamma feature? DCPs work in linear floating point space. When it comes to ICC I need to think about gamma of course as some raw converters (like Capture One) feeds the ICC profile with gamma-processed rather than linear image, so when generating ICCs there will have to be some options regarding those aspects.

[AlterEgo]

AlterEgo, I'm not sure if I follow your feature request. DCamProf can generate profiles with our without HueSatMap. I don't understand why you would want to have some sort of gamma feature? DCPs work in linear floating point space. When it comes to ICC I need to think about gamma of course as some raw converters (like Capture One) feeds the ICC profile with gamma-processed rather than linear image, so when generating ICCs there will have to be some options regarding those aspects.

just experimenting between converters ! you can put a tone curve (probably a wrong wording for that purpose) in dcp profile that is applied after exposure adjustment operations, but if you want (for example) to do this before then HueSatMap trick is the only way, no ?

[torger]

just experimenting between converters ! you can put a tone curve (probably a wrong wording for that purpose) in dcp profile that is applied after exposure adjustment operations, but if you want (for example) to do this before then HueSatMap trick is the only way, no ?

Ah, ok. Yes if you want to apply a curve before exposure adjustments you need to incorporate that in HueSatMap.

[AlterEgo]

Ah, ok. Yes if you want to apply a curve before exposure adjustments you need to incorporate that in HueSatMap.

yes, so if you will find some strength after the deserved rest for the experimentation sake please

[kirkt]

I have stepped through the basic workflow outlined in your web page "Basic workflow for making a profile using a test target".

I used a shot of the 24 patch section of my CCPassport, cropped it, checked the crop with the diagnostic image from scanin, etc.  The output from scanin looks good, the whole process runs great, etc.  This is from my own build with -fopenmp removed from the makefile.  I'm on a Mac, running 10.10.3.  I made the scanin input with the ColorChecker spectral data file that you provide with the downloaded archive.

Fair enough.  I made the profile with the appropriate camera specific identifier I copied verbatim from the Adobe Standard profile, using dcp2json.

When I place the newly created DCP in the user level camera profiles folder, it down not show up in LR/ACR.  When I use Iridient Developer to specify the newly created DCP, it gives me a little more info:

"Unable to load a valid 3 color DNG camera profile from the selected file."

I decompiled the newly made DCP using dcpTool to see if there was any difference between the json and the decompiled XMP version of the file.  There does not appear to be.  I compared it to the structure of the Adobe Standard profile for the Canon 5DIII and all seems correct.

I'm not sure what is preventing the profile from being used.  Any thoughts?

Thanks,

kirk

[torger]

Thanks for testing.

I've done most of my testing in RawTherapee, just generated a few profiles including a dual-illuminant one. It was a while since I tested in Lightroom though and lots of code has been written since. I shall make a new test in Lightroom and see so there's not some bug there preventing them from working. From the error message you're getting it does sound like a bug.

The only mistake one is supposed to be able to make is to name the camera incorrectly, so you can double-check that the UniqueCameraModel is *exactly* the same as in a profile that works with your camera.

[AlterEgo]

Any thoughts?

may be you can just attach the dcp file or its dump from dcptool - makes it easier for somebody to notice any errors (if any)

[kirkt]

Here is a log of the process, from Terminal output:

KT-Laptop-3:Debug kirkt$ dcraw -v -r 1 1 1 1 -o 0 -H 0 -T -6 -W -g 1 1 /Users/kirkt/Desktop/dcamprof/Debug/_MG_0044.CR2
Loading Canon EOS 5D Mark III image from /Users/kirkt/Desktop/dcamprof/Debug/_MG_0044.CR2 ...
Scaling with darkness 2047, saturation 15488, and
multipliers 1.000000 1.000000 1.000000 1.000000
AHD interpolation...
Building histograms...
Writing data to /Users/kirkt/Desktop/dcamprof/Debug/_MG_0044.tiff ...


KT-Laptop-3:Debug kirkt$ scanin -v -dipn target.tif ColorChecker.cht cc24_ref.cie
TIFFFetchNormalTag: Warning, ASCII value for tag "ImageDescription" contains null byte in value; value incorrectly truncated during reading due to implementation limitations.
TIFFFetchNormalTag: Warning, ASCII value for tag "Make" contains null byte in value; value incorrectly truncated during reading due to implementation limitations.
TIFFFetchNormalTag: Warning, ASCII value for tag "Model" contains null byte in value; value incorrectly truncated during reading due to implementation limitations.
TIFFFetchNormalTag: Warning, Incompatible type for "RichTIFFIPTC"; tag ignored.
Input file 'target.tif': w=2356, h=1562, d = 3, bpp = 16
Data input file 'cc24_ref.cie'
Data output file 'target.ti3'
Chart reference file 'ColorChecker.cht'
Creating diagnostic tiff file 'diag.tif'
About to allocate scanrd_ object
Verbosity = 2, flags = 0x42a01
About to read input tiff file and discover groups
adivval = 1.000000
About to calculate edge lines
379 useful edges out of 482
About to calculate rotation
Mean angle = -0.220790
Standard deviation = 1.164465
Robust mean angle = -0.189701 from 356 lines
About to calculate feature information
About to read reference feature information
Read of chart reference file succeeded
About to match features
Checking xx
Checking yy
Checking xy
Checking yx
Checking xix
Checking yiy
Checking xiy
Checking yix
Axis matches for each possible orientation:
  0: xx  = 0.320776, yy  = 0.306004, xx.sc  = 0.134631, yy.sc  = 0.135603
 90: xiy = 0.225432, yx  = 0.351580, xiy.sc = 0.389820, yx.sc  = 0.088742
180: xix = 0.301947, yiy = 0.347327, xix.sc = 0.134340, yiy.sc = 0.134642
270: xy  = 0.224873, yix = 0.351576, xy.sc  = 0.257560, yix.sc = 0.088742
r0 = 0.440145, r90 = 0.095076, r180 = 0.459195, r270 = 0.143794
There are 2 candidate rotations:
cc = 0.440145, irot = -0.189701, xoff = -49.567580, yoff = -74.423086, xscale = 7.427721, yscale = 7.374475
cc = 0.459195, irot = 179.810299, xoff = -2416.435540, yoff = -1602.952441, xscale = 7.443805, yscale = 7.427127
About to compute match transform for rotation -0.189701 deg.
About to setup value scanrdg boxes
About to read raster values
About to compute expected value correlation
About to compute match transform for rotation 179.810299 deg.
About to setup value scanrdg boxes
About to read raster values
About to compute expected value correlation
Expected value distance values are:
0, rot -0.189701: 2410.510929
1, rot 179.810299: 3842.704025
Chosen rotation -0.189701 deg. as best
About to compute final match transform
Improve match
About to setup value scanrdg boxes
About to read raster values
About to write diag file
Writing output values to file 'target.ti3'


KT-Laptop-3:Debug kirkt$ dcamprof make-profile target.ti3 profile.json
Reading target and generating values for the calibration illuminant D50...
Re-generating target reference XYZ values for illuminant D50...
Making camera profile...
Finding a camera raw RGB to CIE XYZ matrix for illuminant D50...
Whitest patch in target differs DE 1.18 from calibration illuminant,
  close enough to calculate whitepoint preservation.
Inverting to get ColorMatrix:
  {
    "ColorMatrix1": [
      [  0.729155, -0.155578, -0.066452 ],
      [ -0.482378,  1.364639,  0.172933 ],
      [ -0.111882,  0.237018,  0.685179 ]
    ]
  }
Matrix patch match average DE 1.38, DE LCh 0.61 0.63 0.97
                      mean DE 1.24, DE LCh 0.50 0.51 0.69
                       p90 DE 2.32, DE LCh 1.28 1.15 1.90
                       max DE 4.06, DE LCh 1.92 1.90 3.94
ColorMatrix optimal white balance for target: 0.478142,1,0.660402
Using previously calculated RGB to XYZ D50 matrix.
Applying white-balance to get ForwardMatrix:
  {
    "ForwardMatrix1": [
      [  0.738497,  0.156947,  0.072765 ],
      [  0.257033,  0.850257, -0.107290 ],
      [  0.031675, -0.268494,  1.049609 ]
    ]
  }
Matrix patch match average DE 1.58, DE LCh 0.96 0.59 0.98
                      mean DE 1.48, DE LCh 0.91 0.60 0.69
                       p90 DE 2.58, DE LCh 1.83 1.05 1.91
                       max DE 4.23, DE LCh 2.40 1.78 3.95
ForwardMatrix optimal white balance for target: 0.478142,1,0.660402
Making 2.5D chromaticity-addressed lookup table for XYZ correction...
25.00% of the patches was put in a chromaticity group due to nearby neighbor.
  Largest chromaticity group contains 6 patches. Patch count reduced from
  24 to 18. Note that patch matching cannot reach 100% when chromaticity
  groups are formed, as the LUT matches the average within a group.
Native LUT patch match average DE 0.37, DE LCh 0.14 0.15 0.23
                          mean DE 0.00, DE LCh 0.00 0.00 0.00
                           p90 DE 1.21, DE LCh 0.48 0.60 0.81
                           max DE 1.89, DE LCh 1.19 0.95 1.80
5 worst patches for Overall DE:
       0 RGB 0.077 0.106 0.054 => XYZ 0.127 0.111 0.055 (0.130 0.114 0.054) DE 1.89 DE LCh 0.57 0.04 1.80 (dark brown)
      18 RGB 0.484 1.000 0.652 => XYZ 0.965 1.000 0.779 (0.948 0.978 0.773) DE 1.40 DE LCh 0.48 0.95 0.90 (white)
      23 RGB 0.019 0.041 0.027 => XYZ 0.034 0.035 0.029 (0.039 0.040 0.033) DE 1.21 DE LCh 1.19 0.23 0.05 (gray 20%)
      20 RGB 0.194 0.408 0.267 => XYZ 0.380 0.394 0.320 (0.382 0.398 0.318) DE 1.02 DE LCh 0.19 0.89 0.46 (gray 70%)
      22 RGB 0.048 0.103 0.068 => XYZ 0.094 0.098 0.081 (0.096 0.101 0.082) DE 1.02 DE LCh 0.42 0.60 0.71 (gray 40%)
5 worst patches for Lightness DE:
      23 RGB 0.019 0.041 0.027 => XYZ 0.034 0.035 0.029 (0.039 0.040 0.033) DE 1.21 DE LCh 1.19 0.23 0.05 (gray 20%)
       0 RGB 0.077 0.106 0.054 => XYZ 0.127 0.111 0.055 (0.130 0.114 0.054) DE 1.89 DE LCh 0.57 0.04 1.80 (dark brown)
      18 RGB 0.484 1.000 0.652 => XYZ 0.965 1.000 0.779 (0.948 0.978 0.773) DE 1.40 DE LCh 0.48 0.95 0.90 (white)
      22 RGB 0.048 0.103 0.068 => XYZ 0.094 0.098 0.081 (0.096 0.101 0.082) DE 1.02 DE LCh 0.42 0.60 0.71 (gray 40%)
      19 RGB 0.319 0.671 0.440 => XYZ 0.622 0.645 0.521 (0.628 0.654 0.525) DE 0.64 DE LCh 0.28 0.58 0.03 (gray 80%)
5 worst patches for Chroma DE:
      18 RGB 0.484 1.000 0.652 => XYZ 0.965 1.000 0.779 (0.948 0.978 0.773) DE 1.40 DE LCh 0.48 0.95 0.90 (white)
      20 RGB 0.194 0.408 0.267 => XYZ 0.380 0.394 0.320 (0.382 0.398 0.318) DE 1.02 DE LCh 0.19 0.89 0.46 (gray 70%)
      22 RGB 0.048 0.103 0.068 => XYZ 0.094 0.098 0.081 (0.096 0.101 0.082) DE 1.02 DE LCh 0.42 0.60 0.71 (gray 40%)
      19 RGB 0.319 0.671 0.440 => XYZ 0.622 0.645 0.521 (0.628 0.654 0.525) DE 0.64 DE LCh 0.28 0.58 0.03 (gray 80%)
      21 RGB 0.102 0.215 0.141 => XYZ 0.202 0.210 0.171 (0.202 0.210 0.168) DE 0.78 DE LCh 0.06 0.32 0.71 (gray 50%)
5 worst patches for Hue DE:
       0 RGB 0.077 0.106 0.054 => XYZ 0.127 0.111 0.055 (0.130 0.114 0.054) DE 1.89 DE LCh 0.57 0.04 1.80 (dark brown)
      18 RGB 0.484 1.000 0.652 => XYZ 0.965 1.000 0.779 (0.948 0.978 0.773) DE 1.40 DE LCh 0.48 0.95 0.90 (white)
       1 RGB 0.253 0.363 0.201 => XYZ 0.424 0.382 0.204 (0.422 0.378 0.206) DE 0.84 DE LCh 0.22 0.03 0.81 (red)
      21 RGB 0.102 0.215 0.141 => XYZ 0.202 0.210 0.171 (0.202 0.210 0.168) DE 0.78 DE LCh 0.06 0.32 0.71 (gray 50%)
      22 RGB 0.048 0.103 0.068 => XYZ 0.094 0.098 0.081 (0.096 0.101 0.082) DE 1.02 DE LCh 0.42 0.60 0.71 (gray 40%)
5 best patches for Overall DE:
      16 RGB 0.187 0.186 0.183 => XYZ 0.325 0.223 0.259 (0.325 0.223 0.259) DE 0.00 DE LCh 0.00 0.00 0.00 (purple-red)
       3 RGB 0.071 0.154 0.065 => XYZ 0.123 0.141 0.056 (0.123 0.141 0.056) DE 0.00 DE LCh 0.00 0.00 0.00 (yellow-green)
       2 RGB 0.089 0.249 0.219 => XYZ 0.189 0.211 0.284 (0.189 0.211 0.284) DE 0.00 DE LCh 0.00 0.00 0.00 (purple-blue)
      14 RGB 0.152 0.093 0.045 => XYZ 0.224 0.134 0.041 (0.224 0.134 0.041) DE 0.00 DE LCh 0.00 0.00 0.00 (red)
       9 RGB 0.048 0.078 0.084 => XYZ 0.090 0.073 0.117 (0.090 0.073 0.117) DE 0.00 DE LCh 0.00 0.00 0.00 (dark purple)
Writing output to "profile.json"...
Complete!

dcamprof make-dcp -c "Canon EOS 5D Mark III"  profile.json testprofile.dcp

In the next post, I will post a Dropbox link with a ZIP archive of all of the files I used in this process.

kirk

[torger]

I'm about to do some Lightroom tests myself... I just have some issues with my windows virtual box here so it can be an hour or two...

[kirkt]

Here is a link to the archive of files I used in this example:

https://www.dropbox.com/s/xq5w8whoi8sq1av/dcamprofDebugArchive.zip?dl=0

it is uploading now (1400 hr eastern daylight time) so give it a few minutes if the download fails.

The archive includes the source raw, the dcraw conversion, a crop of the conversion, the scanin target chart in file, etc. 

thanks in advance!

kirk