Luminous Landscape Forum

Raw & Post Processing, Printing => Colour Management => Topic started by: torger on April 30, 2015, 01:26:32 pm

Title: DCamProf - a new camera profiling tool
Post by: torger on April 30, 2015, 01:26:32 pm
As Argyll can't make DCPs and the commercial software seemed limited and have issues with smoothness I decided to make my own software, and I have now just released a first version (it's free and open-source):

http://www.ludd.ltu.se/~torger/dcamprof.html

It's command line only and quite technical, but if you can use Argyll you can use my DCamProf.

Hope you enjoy it.

Now I need some rest and start living again. I've been working nonstop on my spare time for almost two months with this...
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on April 30, 2015, 01:58:33 pm
thank you
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on May 01, 2015, 03:30:01 am
There are a few unique features in the software, but perhaps the biggest thing is that you can make profiles directly from the camera's spectral sensitivity functions (=color filter responses), just like many manufacturers do.

You can make it the traditional way too from photos of test targets of course. The ability to experiment with SSFs is a great learning experience though if you want to get a feel of how camera color work. I have links to some sources of SSFs that have been made public through academic work, unfortunately it's hard to come by for very recent cameras.

There's the color spectral data too. I currently have builtin spectral data for cc24, munsell glossy set and spectra from nordic nature. I'd like to have a skin tone database too and more. If any of you know about some useful public spectral databases I'd like to know so I can consider them for inclusion. There are a few commercial databases out there, such as directly from CIE, but those cannot be redistributed in free software...
Title: Re: DCamProf - a new camera profiling tool
Post by: TRANTOR on May 01, 2015, 06:44:18 am
Great work.
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on May 01, 2015, 07:27:27 am
Thanks. I was inspired by your work Trantor which showed me the power of using SSFs directly.
Title: Re: DCamProf - a new camera profiling tool
Post by: kirkt on May 01, 2015, 09:01:20 am
Wow Torger - a lot to think about and learn.  Thank you for the effort!

kirk
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on May 01, 2015, 09:38:39 am
Now I need some rest and start living again.

can you (after some rest) add an option to imitate matrix/trc model (like in icc containers) where you have a dcp basically with CM/FM for a color transform & WB operations for ACR/LR and HueSatMap (so it is before exposure adjustments done by a user in ACR/LR UI) that is just imitating (and nothing else) what in "icc" case TRCs (in a simplest case gamma != 1) 'd do per each RGB channel on WB/demosaicked data before matrix conversion into PCS (cieXYZ/D50)... something like "-g <gamma>" option for a simple scenario (more complex might pickup curve(s) data from some text/xml/etc file)


Title: Re: DCamProf - a new camera profiling tool
Post by: Bart_van_der_Wolf on May 01, 2015, 09:59:44 am
As Argyll can't make DCPs and the commercial software seemed limited and have issues with smoothness I decided to make my own software, and I have now just released a first version (it's free and open-source):

http://www.ludd.ltu.se/~torger/dcamprof.html

It's command line only and quite technical, but if you can use Argyll you can use my DCamProf.

Hope you enjoy it.

Now I need some rest and start living again. I've been working nonstop on my spare time for almost two months with this...

Hi Anders, do get some rest. I know how much it takes do put something like this together. Huge amounts of research, even before the programming effort can begin, and once you are encoding, you discover all sorts of new things that need research.

The effort and making the result available is much appreciated. Unfortunately for this version, I'm an ICC Camera profile user (Capture One), so DCPs are not my main interest. But if you get around to adding ICC profile generation, I'll surely try to compile a version for Windows if I can find some time myself.

Cheers,
Bart
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on May 01, 2015, 10:08:03 am
But if you get around to adding ICC profile generation
wasn't there somewhere dcp2icc free conversion tool
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on May 01, 2015, 10:11:53 am
I'd like to have a skin tone database too
so everybody here = get naked with your i1s (or whatever) in hands and donate :D !
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on May 01, 2015, 10:28:49 am
There are skin databases out there: http://files.cie.co.at/699_Report%20for%20CIE%20R1-56.pdf haven't had times to research yet into if there's someone that can be used in open-source software.

I may look into making ICC profile generator for Capture One (and others) later. The original intention was to make a format-agnostic profile maker, and although DCamProf does store a "color matrix" and "forward matrix" (DCP names) it's generic concepts and it stores a generic format so ICC making should be possible. I need this to settle for a while first, get some early adopters to test etc. There's quite much new stuff to learn for me when it comes to ICC about how the LUTs work (it will probably be a camera RGB directly to Lab D50), and they're a bit tricky to work with as they're integer-based so you need to think it through to not break precision.

AlterEgo, I'm not sure if I follow your feature request. DCamProf can generate profiles with our without HueSatMap. I don't understand why you would want to have some sort of gamma feature? DCPs work in linear floating point space. When it comes to ICC I need to think about gamma of course as some raw converters (like Capture One) feeds the ICC profile with gamma-processed rather than linear image, so when generating ICCs there will have to be some options regarding those aspects.
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on May 01, 2015, 10:35:36 am
AlterEgo, I'm not sure if I follow your feature request. DCamProf can generate profiles with our without HueSatMap. I don't understand why you would want to have some sort of gamma feature? DCPs work in linear floating point space. When it comes to ICC I need to think about gamma of course as some raw converters (like Capture One) feeds the ICC profile with gamma-processed rather than linear image, so when generating ICCs there will have to be some options regarding those aspects.

just experimenting between converters ! you can put a tone curve (probably a wrong wording for that purpose) in dcp profile that is applied after exposure adjustment operations, but if you want (for example) to do this before then HueSatMap trick is the only way, no ?

Title: Re: DCamProf - a new camera profiling tool
Post by: torger on May 01, 2015, 10:50:27 am
just experimenting between converters ! you can put a tone curve (probably a wrong wording for that purpose) in dcp profile that is applied after exposure adjustment operations, but if you want (for example) to do this before then HueSatMap trick is the only way, no ?

Ah, ok. Yes if you want to apply a curve before exposure adjustments you need to incorporate that in HueSatMap.
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on May 01, 2015, 11:11:22 am
Ah, ok. Yes if you want to apply a curve before exposure adjustments you need to incorporate that in HueSatMap.
yes, so if you will find some strength after the deserved rest for the experimentation sake please
Title: Re: DCamProf - a new camera profiling tool
Post by: kirkt on May 01, 2015, 12:29:58 pm
I have stepped through the basic workflow outlined in your web page "Basic workflow for making a profile using a test target".

I used a shot of the 24 patch section of my CCPassport, cropped it, checked the crop with the diagnostic image from scanin, etc.  The output from scanin looks good, the whole process runs great, etc.  This is from my own build with -fopenmp removed from the makefile.  I'm on a Mac, running 10.10.3.  I made the scanin input with the ColorChecker spectral data file that you provide with the downloaded archive.

Fair enough.  I made the profile with the appropriate camera specific identifier I copied verbatim from the Adobe Standard profile, using dcp2json.

When I place the newly created DCP in the user level camera profiles folder, it down not show up in LR/ACR.  When I use Iridient Developer to specify the newly created DCP, it gives me a little more info:

"Unable to load a valid 3 color DNG camera profile from the selected file."

I decompiled the newly made DCP using dcpTool to see if there was any difference between the json and the decompiled XMP version of the file.  There does not appear to be.  I compared it to the structure of the Adobe Standard profile for the Canon 5DIII and all seems correct.

I'm not sure what is preventing the profile from being used.  Any thoughts?

Thanks,

kirk

Title: Re: DCamProf - a new camera profiling tool
Post by: torger on May 01, 2015, 12:49:35 pm
Thanks for testing.

I've done most of my testing in RawTherapee, just generated a few profiles including a dual-illuminant one. It was a while since I tested in Lightroom though and lots of code has been written since. I shall make a new test in Lightroom and see so there's not some bug there preventing them from working. From the error message you're getting it does sound like a bug.

The only mistake one is supposed to be able to make is to name the camera incorrectly, so you can double-check that the UniqueCameraModel is *exactly* the same as in a profile that works with your camera.
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on May 01, 2015, 01:13:39 pm
Any thoughts?
may be you can just attach the dcp file or its dump from dcptool - makes it easier for somebody to notice any errors (if any)
Title: Re: DCamProf - a new camera profiling tool
Post by: kirkt on May 01, 2015, 01:49:31 pm
Here is a log of the process, from Terminal output:

KT-Laptop-3:Debug kirkt$ dcraw -v -r 1 1 1 1 -o 0 -H 0 -T -6 -W -g 1 1 /Users/kirkt/Desktop/dcamprof/Debug/_MG_0044.CR2
Loading Canon EOS 5D Mark III image from /Users/kirkt/Desktop/dcamprof/Debug/_MG_0044.CR2 ...
Scaling with darkness 2047, saturation 15488, and
multipliers 1.000000 1.000000 1.000000 1.000000
AHD interpolation...
Building histograms...
Writing data to /Users/kirkt/Desktop/dcamprof/Debug/_MG_0044.tiff ...


KT-Laptop-3:Debug kirkt$ scanin -v -dipn target.tif ColorChecker.cht cc24_ref.cie
TIFFFetchNormalTag: Warning, ASCII value for tag "ImageDescription" contains null byte in value; value incorrectly truncated during reading due to implementation limitations.
TIFFFetchNormalTag: Warning, ASCII value for tag "Make" contains null byte in value; value incorrectly truncated during reading due to implementation limitations.
TIFFFetchNormalTag: Warning, ASCII value for tag "Model" contains null byte in value; value incorrectly truncated during reading due to implementation limitations.
TIFFFetchNormalTag: Warning, Incompatible type for "RichTIFFIPTC"; tag ignored.
Input file 'target.tif': w=2356, h=1562, d = 3, bpp = 16
Data input file 'cc24_ref.cie'
Data output file 'target.ti3'
Chart reference file 'ColorChecker.cht'
Creating diagnostic tiff file 'diag.tif'
About to allocate scanrd_ object
Verbosity = 2, flags = 0x42a01
About to read input tiff file and discover groups
adivval = 1.000000
About to calculate edge lines
379 useful edges out of 482
About to calculate rotation
Mean angle = -0.220790
Standard deviation = 1.164465
Robust mean angle = -0.189701 from 356 lines
About to calculate feature information
About to read reference feature information
Read of chart reference file succeeded
About to match features
Checking xx
Checking yy
Checking xy
Checking yx
Checking xix
Checking yiy
Checking xiy
Checking yix
Axis matches for each possible orientation:
  0: xx  = 0.320776, yy  = 0.306004, xx.sc  = 0.134631, yy.sc  = 0.135603
 90: xiy = 0.225432, yx  = 0.351580, xiy.sc = 0.389820, yx.sc  = 0.088742
180: xix = 0.301947, yiy = 0.347327, xix.sc = 0.134340, yiy.sc = 0.134642
270: xy  = 0.224873, yix = 0.351576, xy.sc  = 0.257560, yix.sc = 0.088742
r0 = 0.440145, r90 = 0.095076, r180 = 0.459195, r270 = 0.143794
There are 2 candidate rotations:
cc = 0.440145, irot = -0.189701, xoff = -49.567580, yoff = -74.423086, xscale = 7.427721, yscale = 7.374475
cc = 0.459195, irot = 179.810299, xoff = -2416.435540, yoff = -1602.952441, xscale = 7.443805, yscale = 7.427127
About to compute match transform for rotation -0.189701 deg.
About to setup value scanrdg boxes
About to read raster values
About to compute expected value correlation
About to compute match transform for rotation 179.810299 deg.
About to setup value scanrdg boxes
About to read raster values
About to compute expected value correlation
Expected value distance values are:
0, rot -0.189701: 2410.510929
1, rot 179.810299: 3842.704025
Chosen rotation -0.189701 deg. as best
About to compute final match transform
Improve match
About to setup value scanrdg boxes
About to read raster values
About to write diag file
Writing output values to file 'target.ti3'


KT-Laptop-3:Debug kirkt$ dcamprof make-profile target.ti3 profile.json
Reading target and generating values for the calibration illuminant D50...
Re-generating target reference XYZ values for illuminant D50...
Making camera profile...
Finding a camera raw RGB to CIE XYZ matrix for illuminant D50...
Whitest patch in target differs DE 1.18 from calibration illuminant,
  close enough to calculate whitepoint preservation.
Inverting to get ColorMatrix:
  {
    "ColorMatrix1": [
      [  0.729155, -0.155578, -0.066452 ],
      [ -0.482378,  1.364639,  0.172933 ],
      [ -0.111882,  0.237018,  0.685179 ]
    ]
  }
Matrix patch match average DE 1.38, DE LCh 0.61 0.63 0.97
                      mean DE 1.24, DE LCh 0.50 0.51 0.69
                       p90 DE 2.32, DE LCh 1.28 1.15 1.90
                       max DE 4.06, DE LCh 1.92 1.90 3.94
ColorMatrix optimal white balance for target: 0.478142,1,0.660402
Using previously calculated RGB to XYZ D50 matrix.
Applying white-balance to get ForwardMatrix:
  {
    "ForwardMatrix1": [
      [  0.738497,  0.156947,  0.072765 ],
      [  0.257033,  0.850257, -0.107290 ],
      [  0.031675, -0.268494,  1.049609 ]
    ]
  }
Matrix patch match average DE 1.58, DE LCh 0.96 0.59 0.98
                      mean DE 1.48, DE LCh 0.91 0.60 0.69
                       p90 DE 2.58, DE LCh 1.83 1.05 1.91
                       max DE 4.23, DE LCh 2.40 1.78 3.95
ForwardMatrix optimal white balance for target: 0.478142,1,0.660402
Making 2.5D chromaticity-addressed lookup table for XYZ correction...
25.00% of the patches was put in a chromaticity group due to nearby neighbor.
  Largest chromaticity group contains 6 patches. Patch count reduced from
  24 to 18. Note that patch matching cannot reach 100% when chromaticity
  groups are formed, as the LUT matches the average within a group.
Native LUT patch match average DE 0.37, DE LCh 0.14 0.15 0.23
                          mean DE 0.00, DE LCh 0.00 0.00 0.00
                           p90 DE 1.21, DE LCh 0.48 0.60 0.81
                           max DE 1.89, DE LCh 1.19 0.95 1.80
5 worst patches for Overall DE:
       0 RGB 0.077 0.106 0.054 => XYZ 0.127 0.111 0.055 (0.130 0.114 0.054) DE 1.89 DE LCh 0.57 0.04 1.80 (dark brown)
      18 RGB 0.484 1.000 0.652 => XYZ 0.965 1.000 0.779 (0.948 0.978 0.773) DE 1.40 DE LCh 0.48 0.95 0.90 (white)
      23 RGB 0.019 0.041 0.027 => XYZ 0.034 0.035 0.029 (0.039 0.040 0.033) DE 1.21 DE LCh 1.19 0.23 0.05 (gray 20%)
      20 RGB 0.194 0.408 0.267 => XYZ 0.380 0.394 0.320 (0.382 0.398 0.318) DE 1.02 DE LCh 0.19 0.89 0.46 (gray 70%)
      22 RGB 0.048 0.103 0.068 => XYZ 0.094 0.098 0.081 (0.096 0.101 0.082) DE 1.02 DE LCh 0.42 0.60 0.71 (gray 40%)
5 worst patches for Lightness DE:
      23 RGB 0.019 0.041 0.027 => XYZ 0.034 0.035 0.029 (0.039 0.040 0.033) DE 1.21 DE LCh 1.19 0.23 0.05 (gray 20%)
       0 RGB 0.077 0.106 0.054 => XYZ 0.127 0.111 0.055 (0.130 0.114 0.054) DE 1.89 DE LCh 0.57 0.04 1.80 (dark brown)
      18 RGB 0.484 1.000 0.652 => XYZ 0.965 1.000 0.779 (0.948 0.978 0.773) DE 1.40 DE LCh 0.48 0.95 0.90 (white)
      22 RGB 0.048 0.103 0.068 => XYZ 0.094 0.098 0.081 (0.096 0.101 0.082) DE 1.02 DE LCh 0.42 0.60 0.71 (gray 40%)
      19 RGB 0.319 0.671 0.440 => XYZ 0.622 0.645 0.521 (0.628 0.654 0.525) DE 0.64 DE LCh 0.28 0.58 0.03 (gray 80%)
5 worst patches for Chroma DE:
      18 RGB 0.484 1.000 0.652 => XYZ 0.965 1.000 0.779 (0.948 0.978 0.773) DE 1.40 DE LCh 0.48 0.95 0.90 (white)
      20 RGB 0.194 0.408 0.267 => XYZ 0.380 0.394 0.320 (0.382 0.398 0.318) DE 1.02 DE LCh 0.19 0.89 0.46 (gray 70%)
      22 RGB 0.048 0.103 0.068 => XYZ 0.094 0.098 0.081 (0.096 0.101 0.082) DE 1.02 DE LCh 0.42 0.60 0.71 (gray 40%)
      19 RGB 0.319 0.671 0.440 => XYZ 0.622 0.645 0.521 (0.628 0.654 0.525) DE 0.64 DE LCh 0.28 0.58 0.03 (gray 80%)
      21 RGB 0.102 0.215 0.141 => XYZ 0.202 0.210 0.171 (0.202 0.210 0.168) DE 0.78 DE LCh 0.06 0.32 0.71 (gray 50%)
5 worst patches for Hue DE:
       0 RGB 0.077 0.106 0.054 => XYZ 0.127 0.111 0.055 (0.130 0.114 0.054) DE 1.89 DE LCh 0.57 0.04 1.80 (dark brown)
      18 RGB 0.484 1.000 0.652 => XYZ 0.965 1.000 0.779 (0.948 0.978 0.773) DE 1.40 DE LCh 0.48 0.95 0.90 (white)
       1 RGB 0.253 0.363 0.201 => XYZ 0.424 0.382 0.204 (0.422 0.378 0.206) DE 0.84 DE LCh 0.22 0.03 0.81 (red)
      21 RGB 0.102 0.215 0.141 => XYZ 0.202 0.210 0.171 (0.202 0.210 0.168) DE 0.78 DE LCh 0.06 0.32 0.71 (gray 50%)
      22 RGB 0.048 0.103 0.068 => XYZ 0.094 0.098 0.081 (0.096 0.101 0.082) DE 1.02 DE LCh 0.42 0.60 0.71 (gray 40%)
5 best patches for Overall DE:
      16 RGB 0.187 0.186 0.183 => XYZ 0.325 0.223 0.259 (0.325 0.223 0.259) DE 0.00 DE LCh 0.00 0.00 0.00 (purple-red)
       3 RGB 0.071 0.154 0.065 => XYZ 0.123 0.141 0.056 (0.123 0.141 0.056) DE 0.00 DE LCh 0.00 0.00 0.00 (yellow-green)
       2 RGB 0.089 0.249 0.219 => XYZ 0.189 0.211 0.284 (0.189 0.211 0.284) DE 0.00 DE LCh 0.00 0.00 0.00 (purple-blue)
      14 RGB 0.152 0.093 0.045 => XYZ 0.224 0.134 0.041 (0.224 0.134 0.041) DE 0.00 DE LCh 0.00 0.00 0.00 (red)
       9 RGB 0.048 0.078 0.084 => XYZ 0.090 0.073 0.117 (0.090 0.073 0.117) DE 0.00 DE LCh 0.00 0.00 0.00 (dark purple)
Writing output to "profile.json"...
Complete!

dcamprof make-dcp -c "Canon EOS 5D Mark III"  profile.json testprofile.dcp


In the next post, I will post a Dropbox link with a ZIP archive of all of the files I used in this process.

kirk
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on May 01, 2015, 01:58:13 pm
I'm about to do some Lightroom tests myself... I just have some issues with my windows virtual box here so it can be an hour or two...
Title: Re: DCamProf - a new camera profiling tool
Post by: kirkt on May 01, 2015, 02:03:57 pm
Here is a link to the archive of files I used in this example:

https://www.dropbox.com/s/xq5w8whoi8sq1av/dcamprofDebugArchive.zip?dl=0

it is uploading now (1400 hr eastern daylight time) so give it a few minutes if the download fails.

The archive includes the source raw, the dcraw conversion, a crop of the conversion, the scanin target chart in file, etc. 

thanks in advance!

kirk
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on May 01, 2015, 02:21:15 pm
Here is a link to the archive of files I used in this example:

https://www.dropbox.com/s/xq5w8whoi8sq1av/dcamprofDebugArchive.zip?dl=0

it is uploading now (1400 hr eastern daylight time) so give it a few minutes if the download fails.

The archive includes the source raw, the dcraw conversion, a crop of the conversion, the scanin target chart in file, etc.  

thanks in advance!

kirk

something is wrong with CM/FM inside DCP... if you replace a pair of CM/FM with the one from Adobe OEM - then ACR see the profile and can use it... so the way CM/FM were calculated put them against something that ACR code expects (it does some checks of course, you can't just feed any matrices)
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on May 01, 2015, 02:23:44 pm
Seems like I've messed up with something, I'll report back when I have a fix.
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on May 01, 2015, 02:30:59 pm
Seems like I've messed up with something, I'll report back when I have a fix.
check how you do FM - if you leave everything but use FM from Adobe OEM then ACR code can accept the profile... replace FM with yours and ACR does not accept it
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on May 01, 2015, 02:32:14 pm
check how you do FM - if you leave everything but use FM from Adobe OEM then ACR code can accept the profile... replace FM with yours and ACR does not accept it

Thanks for the help, I'll have a look.
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on May 01, 2015, 02:34:12 pm
Thanks for the help, I'll have a look.

and CM seems to operate properly in terms of how ACR works with WB - visual impression, so CM is calculated OK I guess...
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on May 01, 2015, 02:56:40 pm
Got it working now, I'll soon make a new release. Just need to really make sure what the error was.

Edit: there where two errors:

1) Lightroom cannot handle FM with too high precision in the rational numbers, they use 1/10000, I have 1/1000000000 but then it goes haywire
2) Lightroom cannot handle other observer than 1931_2, DCamProf uses the more realistic 2006_10. By some (stupid) reason Lightroom makes a check that FM x 1,1,1 = D50, and D50 varies slightly depending on observer, and thus if you have not used 1931_2 it breaks.

I'll add a workaround for this and publish in an hour or so, but I can't say I think Lightroom does it the right way. There's no reason to "sanity-check" the D50 whitepoint this way, and why they can handle high precision rational numbers is a bit strange too.
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on May 01, 2015, 03:18:49 pm
Got it working now, I'll soon make a new release. Just need to really make sure what the error was.

Edit: there where two errors:

1) Lightroom cannot handle FM with too high precision in the rational numbers, they use 1/10000, I have 1/1000000000 but then it goes haywire
2) Lightroom cannot handle other observer than 1931_2, DCamProf uses the more realistic 2006_10. By some (stupid) reason Lightroom makes a check that FM x 1,1,1 = D50, and D50 varies slightly depending on observer, and thus if you have not used 1931_2 it breaks.

I'll add a workaround for this and publish in an hour or so, but I can't say I think Lightroom does it the right way. There's no reason to "sanity-check" the D50 whitepoint this way, and why they can handle high precision rational numbers is a bit strange too.

the first issue is kind of funny... too much precision :-)
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on May 01, 2015, 03:35:49 pm
The observer thing is a tricky one... in a way I understand why they check it, as their XYZ to prophoto matrix is hardcoded and expect 1931_2 D50 whitepoint. The easy way out is just that I change the default from 2006_10 to 1931_2, but I think that is worse design, I'd like it to be possible to use more realistic/accurate observers.

Maybe I could just do some whitepoint remapping trick... I need to think.
Title: Re: DCamProf - a new camera profiling tool
Post by: TRANTOR on May 01, 2015, 03:43:01 pm
10-degrees standard observer is not more accurate, it's a just different. Simply do not use it.
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on May 01, 2015, 03:55:28 pm
10-degrees standard observer is not more accurate, it's a just different. Simply do not use it.

I'm quite sure the 2006 versions, http://www.cvrl.org/cmfs.htm, are more accurate. When it comes to 2 vs 10 degree I agree, it's more a matter of taste, but AFAIK the 2006_2 is more accurate than 1931_2 and 2006_10 is more accurate than 1964_10. The reason those new versions are made is mainly to improve accuracy.

If it matters is another aspect though, probably not I guess.

I think I will do an auto-wp-remapping in the make-dcp step and maybe change default to 1931_2.
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on May 01, 2015, 03:55:52 pm
Maybe I could just do some whitepoint remapping trick... I need to think.

if you are so set on "10 degrees" then just supply a command line parameter for an end user to select and provide some explanantion in a manual and then may be it makes sense to default (if nothing is on the command line) to "2 degrees" - I 'd assume that all popular (for masses) software expects that and not "10 degrees" based on what I can read
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on May 01, 2015, 03:57:46 pm
if you are so set on "10 degrees" then just supply a command line parameter for an end user to select and provide some explanantion in a manual and then may be it makes sense to default (if nothing is on the command line) to "2 degrees" - I 'd assume that all popular (for masses) software expects that and not "10 degrees" based on what I can read

Command line parameter is already there, you can do "-o 1931_2", so if it weren't for the 1/10000000 thing it would have worked with the current version. Well, no more talk, I need to code :-)
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on May 01, 2015, 04:49:36 pm
I have now made a new release, and this time actually tested it on Lightroom :-)

http://www.ludd.ltu.se/~torger/files/dcamprof-0.5.1.tar.bz2
http://www.ludd.ltu.se/~torger/dcamprof.html

I added an auto-remapping of whitepoint in the make-dcp tool so regardless of observer used during profile creation it will be remapped to match what Lightroom and probably many other DCP-compatible converters are expecting.

I also changed the default observer to 1931_2 so the remapping is not really needed in the default case. Although I like the 2006_10 (or 2006_2) better I don't have the knowledge now to know if it's better to make profile with say 2006_2 and then remap the white, or if the supposedly slight improvements are nullified due to remapping and hard-coded prophoto matrices later in the pipeline which I can't control. So now by using 1931_2 it's done the way "everyone else" does it, which I suppose is a reasonable default anyway.

Kirk - now it should work for you too. Just compile the new version and run the make-profile and make-dcp steps again.
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on May 01, 2015, 05:59:09 pm
Prophoto is defined against 1931_2, and as DNG works in that space it was natural that the pipeline was hardwired to that. sRGB, AdobeRGB etc are all related to 1931_2. I missed that aspect. Learning something new every day.

This means that it will be tricky to make use of alternate observers in a proper way, seems to me to do it right one would have to make a conversion LUT between the different observers too, quite messy. So using the 1931_2 as default for now is certainly the right choice.
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on May 02, 2015, 05:54:46 am
Done some reading on observers and did some further observer tests this morning, released a very minor update v0.5.2 which tunes the observer remapping a bit. This is however only relevant if experimenting with a different observer than the nowadays default 1931_2.

There's no great method to transform say 2006_10 XYZ to 1931_2 XYZ (found this paper while searching: http://www.rit.edu/cos/colorscience/re_WptNormalization.php), but as the difference is quite small I've tested with the Bradford transform and it seems to make a good result in practice, so good that any other inherent errors in profiling should be much larger.

I made some examples between 2006_10/2006_2 and 1931_2. The 2 and 10 degree observers render near zero visible difference, while it's quite large difference between 2006 and 1931. For the particular camera I tested this morning (Pentax K10D) I preferred the 1931_2 colors, they looked more on the spot to my eyes, so having to change the default to 1931_2 doesn't feel that bad :)
Title: Re: DCamProf - a new camera profiling tool
Post by: kirkt on May 02, 2015, 10:07:40 am
Thanks!  I'll try recompiling again and report back.

Kirk
Title: Re: DCamProf - a new camera profiling tool
Post by: Alexey.Danilchenko on May 02, 2015, 03:17:55 pm
> By some (stupid) reason Lightroom makes a check that FM x 1,1,1 = D50, and D50 varies slightly depending on observer, and thus if you have not used 1931_2 it breaks.

That is not that stupid. FM is constructed in a way to map camera white balanced white point to XYZ D50 (that is according to DNG spec) and that implies this mapping exactly (white balanced camera space will have 1,1,1 as white point). FM is actually a very specific case of ICC profile matrix in this case.
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on May 02, 2015, 03:37:28 pm
Yes I agree after some analysis that it's not that stupid, although unnecessary. The pipeline does not break if there's a small white balance shift, and the reference code doesn't do this type of check.

At some point I'm probably going to again review if one can do something useful with the more modern observers, for now it seems like 1931_2 is the way to go to get best results.
Title: Re: DCamProf - a new camera profiling tool
Post by: Alexey.Danilchenko on May 02, 2015, 03:49:09 pm
It is necessary at least to calculate those matrixes. The profiles (and correction matrices) are build on the that assumption. Nobody prevents to use the matrices on imperfectly white balanced camera space (this is what creative WB is for - I for one never completely white balance tungsten lighting and prefer warmer colours).

What is limiting in DNG is that it has to map to D50 - unlike ICC which can have any white point.

Great tool - thanks for sharing it. I was writing something custom for my own purpouses but will attempt to make use of DCamProf now. Have not managed to compile it on Windows yet - it does not seem to accept gcc short syntax for initialising structs.
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on May 02, 2015, 04:41:36 pm
I just tried compiling with cygwin on windows, it worked but you have to remove long double stuff in blackbody code which probably is overkill anyway. I use c99 syntax, Microsoft compilers have been sloooow adopting that, but it's only a 16 year old standard  :)
Title: Re: DCamProf - a new camera profiling tool
Post by: Alexey.Danilchenko on May 02, 2015, 05:59:11 pm
I was trying with Visual C++ (not the most recent) - does not work. Will have go with mingw.

Had a good look through the sources and your code does seem to fit very well to C++ (it is already kind of structured that way with data structures and supporting functions for them).
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on May 03, 2015, 02:58:24 am
C++ with its flexibility suits pretty much everything, it's a great language. I am first and foremost a C systems programmer on Unix though so when I want fast results I use the tools I know and enjoy best. In my professional life it sometimes feels that more time is spent learning new languages and tools of the day than solving actual problems. When I do stuff for myself I rarely do it to become better at some language, then I just use the tools I like and know best and get on to problem solving.

That Microsoft fails to support the C99 standard, which is a known issue, is pretty much a scandal but it's not uncommon for the big software companies to ignore open standards and rather make up their own. Dot net and C sharp is their thing nowadays. Mac OS X doesn't support OpenMP, another well-established open standard, but it looks a bit better there though as they ride on open source and the clang compiler will be including it "soon".

Let us know if it works out compiling with mingw64. I'll remove the long double code to next patch release since it's not needed.
Title: Re: DCamProf - a new camera profiling tool
Post by: Alexey.Danilchenko on May 03, 2015, 05:49:48 am
All I was saying is that your code is very well structured (so converting it to C++ for someone who wants to do so would be an easy task ;))

Mac OS clang support for OpenMP (lacking of it) is diappointing. I needed it for some of the things I did so the way I solved it was to build GCC version with it and create Xcode templates to use it from Xcode directly. It does not work with Mac OS native libraries but for me where UI was built with QT (from ground up) it worked quite nicely. I do with they start adding it though with clang to avoid all that.
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on May 03, 2015, 11:07:13 am
Let us know if it works out compiling with mingw64. I'll remove the long double code to next patch release since it's not needed.
Torger, can you share the windows binary that you compiled, if you don't mind ? thank you
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on May 03, 2015, 12:48:02 pm
Torger, can you share the windows binary that you compiled, if you don't mind ? thank you

I don't think it works without the Cygwin environment installed, a mingw build can be made standalone statically linked I suppose. The advantage of having cygwin is that you get a command line terminal that actually is not painful to work with, unlike the cmd.exe in windows.

With cygwin you just install from www.cygwin.com, install all devel-packages then just open the command line and

1. get: $ wget http://www.ludd.ltu.se/~torger/files/dcamprof-0.5.3.tar.bz2
2. unpack: $ tar xjf dcamprof-0.5.3.tar.bz2
3. change dir: $ cd dcamprof-0.5.3
4. compile: $ make
5. run: $ ./dcamprof.exe

To shoot targets you'll need Argyll too.

I just released v0.5.3, with some minor additions. Added a txt2ti3 conversion command so you can read raw text spectral data which spectral databases are commonly distributed as. I also added a link to Lippmann2000 spectral database, which unfortunately only may be used for research purposes, but they do have lots of human spectral data.
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on May 03, 2015, 01:03:14 pm
I don't think it works without the Cygwin environment installed, a mingw build can be made standalone statically linked I suppose.

I last did anything like this (building the executable that way) with C/C++ code several years ago (myself - I am not a software developer by trade), so I was hoping to use a shortcut  :D
Title: Re: DCamProf - a new camera profiling tool
Post by: Alexey.Danilchenko on May 03, 2015, 03:22:47 pm
I don't think it works without the Cygwin environment installed
It will - all it will need is cygwin.dll bundled with executable.

Here is a version compiled for 64 bit Windows (https://drive.google.com/file/d/0Bw2ZohnbXtyAQmFPVjBSXzFNODg/view?usp=sharing) with all needed cygwin dependencies to execute standalone.
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on May 04, 2015, 03:39:36 am
Thanks Alexey for the info and the build!
Title: Re: DCamProf - a new camera profiling tool
Post by: kirkt on May 04, 2015, 09:16:02 am
I built v 0.5.3 on my mac (Mavericks) without openMP.  It now generates DCPs that are visible and recognized by ACR and Iridient Developer.  I will also build on my work machine (Yosemite).

Now to test.

kirk
Title: Re: DCamProf - a new camera profiling tool
Post by: Alexey.Danilchenko on May 04, 2015, 11:25:46 am
Thanks Alexey for the info and the build!

No problem. It could potentially be compiled with mingw but  some posix stuff needs customising (I can probably fix it with a few ifdefs - if it won't be too ugly, I'll send the changes to you if that's ok).
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on May 04, 2015, 02:03:47 pm
No problem. It could potentially be compiled with mingw but  some posix stuff needs customising (I can probably fix it with a few ifdefs - if it won't be too ugly, I'll send the changes to you if that's ok).

Please do
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on May 04, 2015, 02:33:51 pm
No problem. It could potentially be compiled with mingw but  some posix stuff needs customising (I can probably fix it with a few ifdefs - if it won't be too ugly, I'll send the changes to you if that's ok).
ok, years w/o programming taking a tall... I finally managed to find a stupid elementary school level error when using mingw... of course fopen shall be used with wb/rb flags on windows platform for binary i/o (profio.c file), damn... we were taught this like 25 years ago and I forgot  :-(

so the things necessary to build DCamProf with mingw are :

1) replace mkdir(dir, 0777) with mkdir(dir) in dcamprof.c

2) add #include <pthread.h> in argyllio.c to get localtime_r and asctime_r

3) replace #include <alloca.h> with #include <malloc.h> in tpc.c

4) and damn it ! in profio.c use fopen with wb and rb flags to read/write .dcp (binary) files

that's it

Quote

Administrator@CLEVO /z/dcamprof-0.5.3
$ make
gcc -o dcamprof.o -c -I. -Wall -std=c99 -g -O2 -D_XOPEN_SOURCE=600 -D__EXTENSIONS__ -fopenmp  -DDCAMPROF_VERSION=\"0.5.3\" dcamprof.c
gcc -o nmsimplex.o -c -I. -Wall -std=c99 -g -O2 -D_XOPEN_SOURCE=600 -D__EXTENSIONS__ -fopenmp  -DDCAMPROF_VERSION=\"0.5.3\" nmsimplex.c
gcc -o profio.o -c -I. -Wall -std=c99 -g -O2 -D_XOPEN_SOURCE=600 -D__EXTENSIONS__ -fopenmp  -DDCAMPROF_VERSION=\"0.5.3\" profio.c
gcc -o argyllio.o -c -I. -Wall -std=c99 -g -O2 -D_XOPEN_SOURCE=600 -D__EXTENSIONS__ -fopenmp  -DDCAMPROF_VERSION=\"0.5.3\" argyllio.c
gcc -o tps.o -c -I. -Wall -std=c99 -g -O2 -D_XOPEN_SOURCE=600 -D__EXTENSIONS__ -fopenmp  -DDCAMPROF_VERSION=\"0.5.3\" tps.c
gcc -o colmath.o -c -I. -Wall -std=c99 -g -O2 -D_XOPEN_SOURCE=600 -D__EXTENSIONS__ -fopenmp  -DDCAMPROF_VERSION=\"0.5.3\" colmath.c
gcc -o matopt.o -c -I. -Wall -std=c99 -g -O2 -D_XOPEN_SOURCE=600 -D__EXTENSIONS__ -fopenmp  -DDCAMPROF_VERSION=\"0.5.3\" matopt.c
matopt.c: In function 'matopt_find_matrix':
matopt.c:270:12: warning: variable 'de' set but not used [-Wunused-but-set-variable]
     double de;
            ^
gcc -o lut.o -c -I. -Wall -std=c99 -g -O2 -D_XOPEN_SOURCE=600 -D__EXTENSIONS__ -fopenmp  -DDCAMPROF_VERSION=\"0.5.3\" lut.c
gcc -o dngref.o -c -I. -Wall -std=c99 -g -O2 -D_XOPEN_SOURCE=600 -D__EXTENSIONS__ -fopenmp  -DDCAMPROF_VERSION=\"0.5.3\" dngref.c
dngref.c:470:1: warning: 'dngref_XYtoXYZ' defined but not used [-Wunused-function]
 dngref_XYtoXYZ(const double XY[2])
 ^
gcc -o spectrum.o -c -I. -Wall -std=c99 -g -O2 -D_XOPEN_SOURCE=600 -D__EXTENSIONS__ -fopenmp  -DDCAMPROF_VERSION=\"0.5.3\" spectrum.c
gcc -o jsonio.o -c -I. -Wall -std=c99 -g -O2 -D_XOPEN_SOURCE=600 -D__EXTENSIONS__ -fopenmp  -DDCAMPROF_VERSION=\"0.5.3\" jsonio.c
jsonio.c:77:1: warning: 'json_print' defined but not used [-Wunused-function]
 json_print(cJSON *js)
 ^
gcc -o cJSON.o -c -I. -Wall -std=c99 -g -O2 -D_XOPEN_SOURCE=600 -D__EXTENSIONS__ -fopenmp  -DDCAMPROF_VERSION=\"0.5.3\" cJSON.c
gcc -o target.o -c -I. -Wall -std=c99 -g -O2 -D_XOPEN_SOURCE=600 -D__EXTENSIONS__ -fopenmp  -DDCAMPROF_VERSION=\"0.5.3\" target.c
gcc -o xyz2spec.o -c -I. -Wall -std=c99 -g -O2 -D_XOPEN_SOURCE=600 -D__EXTENSIONS__ -fopenmp  -DDCAMPROF_VERSION=\"0.5.3\" xyz2spec.c
gcc -o observers.o -c -I. -Wall -std=c99 -g -O2 -D_XOPEN_SOURCE=600 -D__EXTENSIONS__ -fopenmp  -DDCAMPROF_VERSION=\"0.5.3\" observers.c
gcc -o spectraldb.o -c -I. -Wall -std=c99 -g -O2 -D_XOPEN_SOURCE=600 -D__EXTENSIONS__ -fopenmp  -DDCAMPROF_VERSION=\"0.5.3\" spectraldb.c
gcc -o spectraldb_munsell.o -c -I. -Wall -std=c99 -g -O2 -D_XOPEN_SOURCE=600 -D__EXTENSIONS__ -fopenmp  -DDCAMPROF_VERSION=\"0.5.3\" spectraldb_munsell.c
gcc -o spectraldb_cc24.o -c -I. -Wall -std=c99 -g -O2 -D_XOPEN_SOURCE=600 -D__EXTENSIONS__ -fopenmp  -DDCAMPROF_VERSION=\"0.5.3\" spectraldb_cc24.c
gcc -o spectraldb_kuopio_natural.o -c -I. -Wall -std=c99 -g -O2 -D_XOPEN_SOURCE=600 -D__EXTENSIONS__ -fopenmp  -DDCAMPROF_VERSION=\"0.5.3\" spectraldb_kuopio_na
tural.c
gcc -o dcamprof -fopenmp dcamprof.o nmsimplex.o profio.o argyllio.o tps.o colmath.o matopt.o lut.o dngref.o spectrum.o jsonio.o cJSON.o target.o xyz2spec.o obse
rvers.o spectraldb.o spectraldb_munsell.o spectraldb_cc24.o spectraldb_kuopio_natural.o liblcms2.a -lm

Administrator@CLEVO /z/dcamprof-0.5.3
$



compiled .exe = https://app.box.com/s/mqv2dwhv0voswlc9sf58z8e1va21yb7l   ( dcamprof.exe and libgomp_64-1.dll )
Title: Re: DCamProf - a new camera profiling tool
Post by: ErikKaffehr on May 04, 2015, 04:10:52 pm
Hi,

One interesting aspect of DCamProf is that it can generate DCP profiles using different targets. I have earlier used one of Wolfgan Faust's IT-8 charts for checking rendition accuracy of my P45+ back.

With DCamProf it is possible to generate a DCP profile using different targets while Adobe DNG Profile Editor and ColorChecker Passport are limited to the ColorChecker.

I got a DCP profile foe my P45+ from Anders Torger that was based on test shot I made earlier. What I found is that the profile is pretty accurate, probably better than any of my profiles. BUT it needs a different WB setting in Lightroom. Setting WB on a ColorChecker gives a different WB with the DCamProf profile than with Adobe Standard Profile. Once WB is set correctly, the profile is very accurate.

There is an issue, that some of the light grey samples are to dark.

Excellent job from Anders. I hope he will continue to develop the tool.

Best regards
Erik
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on May 05, 2015, 02:16:22 am
Thanks AlterEgo, will fix to next patch release.

ok, years w/o programming taking a tall...
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on May 05, 2015, 02:52:09 am
Thanks for testing, I shall look at why it renders darker. In principle DCamProf does not care about absolute lightness as it's a 2.5D correction it makes, but it looks like there may be some relative lightness error too.

The WB thing is interesting. Few cameras have their preset "daylight" white balance at exactly D50, and DCamProf will correct for exactly D50 if instructed to do so, this means that it calculates which custom white balance is required to fulfill that. Conveying that information is not supported by DCP though so one have to set white balance using color pickers etc. The preset white balance will as it doesn't exactly match D50 not render neutral. For varying light conditions like in landscape this generally does not matter as it will be off anyway.

I may look into the possibility to calibrate not for D50 but for the camera's preset white balance so one will get neutral white with the preset. You would still call the light "D50" in the profile probably as there is no EXIF name that can match any light source, and the actual calibration illuminant used when lighting the target may be slightly different, this does not affect the calibration precision though, just the temp/tint information which indeed will be more wrong with this type of calibration so I'm still not sure it's a good idea. I haven't checked but I'm quite sure Lightroom ignores the DCP matrices for temp/tint numbers and always uses the builtin though. Otherwise the temp/tint numbers would change slightly when you switch profiles.

Hi,

One interesting aspect of DCamProf is that it can generate DCP profiles using different targets. I have earlier used one of Wolfgan Faust's IT-8 charts for checking rendition accuracy of my P45+ back.

With DCamProf it is possible to generate a DCP profile using different targets while Adobe DNG Profile Editor and ColorChecker Passport are limited to the ColorChecker.

I got a DCP profile foe my P45+ from Anders Torger that was based on test shot I made earlier. What I found is that the profile is pretty accurate, probably better than any of my profiles. BUT it needs a different WB setting in Lightroom. Setting WB on a ColorChecker gives a different WB with the DCamProf profile than with Adobe Standard Profile. Once WB is set correctly, the profile is very accurate.

There is an issue, that some of the light grey samples are to dark.

Excellent job from Anders. I hope he will continue to develop the tool.

Best regards
Erik
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on May 05, 2015, 03:04:36 am

I got a DCP profile foe my P45+ from Anders Torger that was based on test shot I made earlier. What I found is that the profile is pretty accurate, probably better than any of my profiles. BUT it needs a different WB setting in Lightroom. Setting WB on a ColorChecker gives a different WB with the DCamProf profile than with Adobe Standard Profile. Once WB is set correctly, the profile is very accurate.

here is a purely matrix (single illuminant) profile (no LUTs) made from IT8 target (not individually measured), illuminated by halogen (no 80A filter on light or lens), w/o flatfielding (light was quite uneven), applied quickly in ACR (process 2010), with a raw containing a different target (not IT8, but colorchecker SG), w/o again measurements for that one and raw with ColorChecker SG was from imaging-resource where the light is naturally uneven on target and light was ~5xxxK something (not 2800-3000K halogen for IT8)... so a lot of handicaps here, still :

(http://s2.postimg.org/a54ubrj6x/A7_IT8_SG.jpg)

PS: I was told (by IBorg) that if errors are noticeable (say more than 2dE-whatever) it is more fair to use dE94, not dE2000... dE2000 makes your results better than they are (numbers-wise)

here is dE2000 which makes things better looking

(http://s1.postimg.org/wa2ktfagv/d_E2_K.jpg)

and visually - again, profile made off one target (IT8) applied to a totally different target (ColorCheckerSG) :

(http://s21.postimg.org/haj6b4m47/visual.jpg)
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on May 05, 2015, 03:22:46 am
Otherwise the temp/tint numbers would change slightly when you switch profiles.
Adobe's profiles for a given camera model /at least for recent generations/ have always the same CMs...
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on May 05, 2015, 04:14:55 am
Erik, I haven't used patchtool so I don't know how it works. How is the output from "my" profile made? Looking at the grayscale range at the bottom of the IT-8 target the patchtool image shows the brightest patch from "my" profile way too dark, even darker than the second brightest.

If I look with my eyes at the actual output from say RawTherapee there is no such error. I've attached an image of a linear rendering of the actual test target shot with the p45plus.dcp applied. As you can see the brightest patch down to the left is brighter than the second brightest unlike in the patch tool image.
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on May 05, 2015, 05:00:55 am
Adobe's profiles for a given camera model /at least for recent generations/ have always the same CMs...

But what happens if you change to a custom profile, such as one generated by DCamProf? If I remember correctly the temp/tint will stay the same in Lightroom, which means that it derives those numbers from builtin matrices rather than from the profile. Most likely though if the custom profile is dual illuminant it uses those matrices to get the temperature which is used for matrix/LUT mixing but it doesn't show it in the GUI, so the temp/tint shown in the GUI is just for informational purposes, and I guess Adobe think those cannot be improved upon.

In the actual low level pipeline the white balance end up being multipliers for raw RGB channels, the temp/tint is just a transform using the CM(s)and the precision is of course limited as the camera doesn't measure the full spectrum. It becomes even more approximate if the illuminant used during profiling doesn't exactly match D50 or D65 or whatever the profile is tagged with. Most likely Adobe uses SSFs to derive the CMs, in that case the illuminants can be made exact and CMs will be better at predicting actual temperature.
Title: Re: DCamProf - a new camera profiling tool
Post by: ErikKaffehr on May 05, 2015, 05:43:18 am
Hi Anders,

I may have messed the conversion in Lightroom, to many sliders...

More testing today...

Best regards
Erik


Erik, I haven't used patchtool so I don't know how it works. How is the output from "my" profile made? Looking at the grayscale range at the bottom of the IT-8 target the patchtool image shows the brightest patch from "my" profile way too dark, even darker than the second brightest.

If I look with my eyes at the actual output from say RawTherapee there is no such error. I've attached an image of a linear rendering of the actual test target shot with the p45plus.dcp applied. As you can see the brightest patch down to the left is brighter than the second brightest unlike in the patch tool image.
Title: Re: DCamProf - a new camera profiling tool
Post by: rubencarmona on May 05, 2015, 05:45:12 am
I opted for spydercheckr with 24 patches. It's not that expensive or complicated and good enough for my purposes:
https://www.youtube.com/watch?v=rAG7pyA0V_8

Title: Re: DCamProf - a new camera profiling tool
Post by: torger on May 05, 2015, 05:53:05 am
PS: I was told (by IBorg) that if errors are noticeable (say more than 2dE-whatever) it is more fair to use dE94, not dE2000... dE2000 makes your results better than they are (numbers-wise)

Thanks for testing.

Concerning Delta E the idea of it is that color difference should be "perceptually uniform", that is DE 5 should *to the eye* look equally large difference between two yellows as between two reds. Today CIEDE2000 (dE2000) is the current best concerning perceptual uniformity, meaning that it sometimes produces larger numbers than older standards and sometimes smaller. That dE94 would be more "fair" because it produces larger numbers for larger errors is a strange thing to say, DE is not about fairness, it's about perceptual uniformity. If de94 exaggerates the error then de2000 is more correct.

dE2000 is like any color standard not perfect though, but I haven't read/heard anything about that dE94 would in some ranges produce more perceptually uniform numbers than dE2000.

This is the reason why DCamProf exclusively use dE2000 when optimizing. You can adjust the k-weights with the -w parameter though, so you can for example say that you're more interested in hue accuracy than in lightness.
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on May 05, 2015, 06:10:11 am
I opted for spydercheckr with 24 patches. It's not that expensive or complicated and good enough for my purposes:
https://www.youtube.com/watch?v=rAG7pyA0V_8

Those are fine easy-to-use products. Although you could use it like that, DCamProf is not intended to be a replacement for those, it's a much more advanced tool aimed at users with special interest or need in camera profiling. It's not easy to use and it doesn't hurt to know a bit color science before digging in.

That said the basic workflow using a macbeth colorchecker 24 and just defaults is relatively easy to get through.

The main reason I made the tool is because there where so many unanswered questions about profiling and camera performance that the available software could not provide answers to. Without being able to process SSFs a lot of things remain unknown, how accurate is a matrix profile, how good is cc24-based profiles, what is the limits of camera color separation etc. Using DCamProf all those questions can get an answer. I haven't really had time to get that far though, but at some point I'm probably going to write an article on camera profiling and camera color that answers a few of those questions I have myself.

That and because the profiles the commercial tools I've tested seemed to have issues with smoothness and did not allow me to control tradeoff between smoothness and accuracy (haven't tested spydrchecker though). With DCamProf you can control the smoothness/accuracy tradeoff, and you can also do things like saying you're more interested in hue accuracy than lightness accuracy, and that way gain more smoothness.
Title: Re: DCamProf - a new camera profiling tool
Post by: Alexey.Danilchenko on May 05, 2015, 07:05:58 am
But what happens if you change to a custom profile, such as one generated by DCamProf? If I remember correctly the temp/tint will stay the same in Lightroom, which means that it derives those numbers from builtin matrices rather than from the profile. Most likely though if the custom profile is dual illuminant it uses those matrices to get the temperature which is used for matrix/LUT mixing but it doesn't show it in the GUI, so the temp/tint shown in the GUI is just for informational purposes, and I guess Adobe think those cannot be improved upon.

In the actual low level pipeline the white balance end up being multipliers for raw RGB channels, the temp/tint is just a transform using the CM(s)and the precision is of course limited as the camera doesn't measure the full spectrum. It becomes even more approximate if the illuminant used during profiling doesn't exactly match D50 or D65 or whatever the profile is tagged with. Most likely Adobe uses SSFs to derive the CMs, in that case the illuminants can be made exact and CMs will be better at predicting actual temperature.

It is even worse - the only white balance that stays the same across various DCP profiles is "As Shot" one. This is where it goes from multipliers to colour temperature/tint (and that will be different for different profiles). For all other presets or direct changes of colour temperature/tint in Lightroom/ACR - the path is backwards, from temperature to the multipliers (or more precisely mapping from XYZ to camera unbalanced to get white levels for that colour temperature/tint using CM or its approximation from the two given ones). And it is the colour temperature/tint that is considered primary in Lightroom/ACR so this is the parameter that is preserved with raw development - not the chosen corresponding white level in raw (which would be more correct and work across profile changes).

The wb does not have to match D50 - it is chromatically adapted to that (by white balancing and applying the forward matrix). As far as the DCP is concerned the matter is in whitebalancing the scene. The idea with two matrixes for different illuminants was that the right colour temperature is determined from the WB of the scene (and considered to be a scene illuminant), then the approximated matrixes are calculated for the scene illuminant (proportionally being between illuminants of two matrixes in the profile) and that the forward matrix obtained that way will be chromatically adapted to D50 XYZ (with WB of the raw when applied will have 1,1,1 mapped to D50 with the FM obtained this way). I am yet to be convinced that this is any better than ICC profiles which are effectively a single FM without D50 XYZ limitation (if we consider only matrix with shapers type of ICC for the moment). RPP uses (in most cases) matrix shaper ICC profiles without any problems.

Title: Re: DCamProf - a new camera profiling tool
Post by: affu933 on May 05, 2015, 07:11:47 am
thanks bro
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on May 05, 2015, 07:27:54 am
It is even worse

Thanks for the info, very interesting. Dual-illuminant profiles are really only a user convenience feature, so you don't need to switch  between appropriate single illuminant profiles. I'm also doubtful that they actually provide any meaningful extra accuracy in the range between StdA and D65 which is the typical pair used. With DCamProf and SSF simulations you can answer that question though, you can render a dual-illuminant profile and then a single-illuminant for inbetweener and compare their performance. This is one of the experiments I intend to do in the coming weeks.

I'd like to add ICC support too, but it's quite massive work so it will probably be a while. The boring thing with ICC is that preprocessing is not standardized so Capture One ICCs, RT ICCs, Leaf ICCs, Phocus ICCs and I guess RPP ICCs are not really compatible with eachother, so I would have to add some "pre-processing specification" to support ICC. Argyll have some problem generating ICCs for say Capture One as Argyll expects linear input and when it gets curve-distorted inputs it makes less than ideal optimizations. For linearly preprocessed pipeline, like RT for example, I guess Argyll makes a really good job already, but it's optimization engine seems more targeted towards the scanner use case than the camera.
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on May 05, 2015, 07:41:53 am
RPP uses (in most cases) matrix shaper ICC profiles without any problems.

From the limited experiments I've done so far it seems like matrix-only profiles for modern cameras are generally quite good performers for the colorcheckers. There's even a standardized measure for that, the Sensitivity Metamerism Index that DxOMark includes, the higher the number the better matrix-only match is possible. When it comes to high saturation colors outside the typical test target it seems though like a LUT can improve things significantly.

If the preprocessing step before ICC is applied is linear the shaper (ie linearization curves) step should not really add anything as far as I can see. Cameras today are linear, and I don't think shaper curves can do anything LUT-like, but maybe I'm wrong? For preprocessing like Capture One does shaping is required, but it would probably be better to use a fixed linearization shaper than trying to optimize something without knowing how the preprocessing is made. At least it seems so from the brief tests I've made using Argyll on Capture One.
Title: Re: DCamProf - a new camera profiling tool
Post by: Bart_van_der_Wolf on May 05, 2015, 08:22:30 am
I'd like to add ICC support too, but it's quite massive work so it will probably be a while.

Anders, that would be hugely appreciated, as is the current tour-de-force. I'm impressed with what you've already done so far, thanks.

Quote
The boring thing with ICC is that preprocessing is not standardized so Capture One ICCs, RT ICCs, Leaf ICCs, Phocus ICCs and I guess RPP ICCs are not really compatible with eachother, so I would have to add some "pre-processing specification" to support ICC.

Yes, but then DCamProf would immediately become the best solution for camera profiling, and fill a huge gap left by other 'solutions'. I've noticed some brightness shift issues in Capture One Pro with the Argyll based wrapper (MakeInputICC) that Iliah Borg made available, when I select a linear tone curve (the default ICC profile for my camera has gamma 1.9-ish TLCs, but not pure gamma). So being able to adjust for that in the profile creation stage would be fabulous.

Cheers,
Bart
Title: Re: DCamProf - a new camera profiling tool
Post by: Alexey.Danilchenko on May 05, 2015, 08:39:44 am
Thanks for the info, very interesting. Dual-illuminant profiles are really only a user convenience feature, so you don't need to switch  between appropriate single illuminant profiles. I'm also doubtful that they actually provide any meaningful extra accuracy in the range between StdA and D65 which is the typical pair used. With DCamProf and SSF simulations you can answer that question though, you can render a dual-illuminant profile and then a single-illuminant for inbetweener and compare their performance. This is one of the experiments I intend to do in the coming weeks.
Eventually perhaps - I am trying to get my own monochromator setup (not quite there yet) to obtain spectrum for my Kodak cameras/backs. Then I'll do the experimanting. For now single matrix profiles perform just as good.

I'd like to add ICC support too, but it's quite massive work so it will probably be a while.
You are already using lcms2 and it has capabilities to create profile from scratch and write it out to the file. The dcp2icc tool uses it like that.

The boring thing with ICC is that preprocessing is not standardized so Capture One ICCs, RT ICCs, Leaf ICCs, Phocus ICCs and I guess RPP ICCs are not really compatible with eachother, so I would have to add some "pre-processing specification" to support ICC. Argyll have some problem generating ICCs for say Capture One as Argyll expects linear input and when it gets curve-distorted inputs it makes less than ideal optimizations. For linearly preprocessed pipeline, like RT for example, I guess Argyll makes a really good job already, but it's optimization engine seems more targeted towards the scanner use case than the camera.

True - but then from prospective of getting the accuare colour none of that would not matter. This only start becoming a problem when profile adds some nonlinearity (as in LUT type of corrections which could be dependent on exposure corrections and so on).

From the limited experiments I've done so far it seems like matrix-only profiles for modern cameras are generally quite good performers for the colorcheckers. There's even a standardized measure for that, the Sensitivity Metamerism Index that DxOMark includes, the higher the number the better matrix-only match is possible. When it comes to high saturation colors outside the typical test target it seems though like a LUT can improve things significantly.

If the preprocessing step before ICC is applied is linear the shaper (ie linearization curves) step should not really add anything as far as I can see. Cameras today are linear, and I don't think shaper curves can do anything LUT-like, but maybe I'm wrong? For preprocessing like Capture One does shaping is required, but it would probably be better to use a fixed linearization shaper than trying to optimize something without knowing how the preprocessing is made. At least it seems so from the brief tests I've made using Argyll on Capture One.
My understanding is that LUT based profiles can introduce non linearity and as such result in colour shift (and transformation noise), matrix preserves linearity and is essentially better (though possibly less precise).
Title: Re: DCamProf - a new camera profiling tool
Post by: Alexey.Danilchenko on May 05, 2015, 08:42:32 am
I've noticed some brightness shift issues in Capture One Pro with the Argyll based wrapper (MakeInputICC) that Iliah Borg made available, when I select a linear tone curve (the default ICC profile for my camera has gamma 1.9-ish TLCs, but not pure gamma). So being able to adjust for that in the profile creation stage would be fabulous.

MakeInputICC will calculate the gamma from the meaurements. So if you are using RawDigger to get the target measurement, it will be close to the one you select there when saving the samples grid (at least that was my experience).
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on May 05, 2015, 09:13:53 am
My understanding is that LUT based profiles can introduce non linearity and as such result in colour shift (and transformation noise), matrix preserves linearity and is essentially better (though possibly less precise).

Yes for things like HDR merging a matrix-only profile is preferable thanks to the linearity, but if merging takes place on the raw level (like Lumariver HDR can do) it doesn't matter. Otherwise my personal opinion is that matrix profiles is not preferable, the linearity is a sort of mathematical beauty but I find that improved accuracy especially for high saturation colors is preferable. LUTs have a bit bad reputation though as software before has used them to correct without thinking about smoothness, and is often used to create "looks" which one may not agree with. I hope DCamProf can improve this reputation later on.

I have some ICC code since before, I'll see if I'll use lcms2 or not for parsing/writing, lcms2 supports the whole large ICC standard and DCamProf will use only a small subset, it may be less work sourcing in my old code than "dumbing down" lcms2. That part is anyway only say 20% of what needs to be done to get full support.
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on May 05, 2015, 09:25:26 am
Thanks for testing.

Concerning Delta E the idea of it is that color difference should be "perceptually uniform", that is DE 5 should *to the eye* look equally large difference between two yellows as between two reds. Today CIEDE2000 (dE2000) is the current best concerning perceptual uniformity, meaning that it sometimes produces larger numbers than older standards and sometimes smaller. That dE94 would be more "fair" because it produces larger numbers for larger errors is a strange thing to say, DE is not about fairness, it's about perceptual uniformity. If de94 exaggerates the error then de2000 is more correct.

dE2000 is like any color standard not perfect though, but I haven't read/heard anything about that dE94 would in some ranges produce more perceptually uniform numbers than dE2000.

This is the reason why DCamProf exclusively use dE2000 when optimizing. You can adjust the k-weights with the -w parameter though, so you can for example say that you're more interested in hue accuracy than in lightness.

well, you shall be better off asking him directly, here is what he said not so long ago, about why dE94 is better when we evaluate the output using custom profiles vs simulations using .cie (with BabelColor Patchtool) :


Quote
> why dE94 is better ?

Because dE2000 by design is intended for evaluation of smaller differences. dE94 is a good middle ground.

Title: Re: DCamProf - a new camera profiling tool
Post by: Alexey.Danilchenko on May 05, 2015, 09:36:40 am
Yes for things like HDR merging a matrix-only profile is preferable thanks to the linearity, but if merging takes place on the raw level (like Lumariver HDR can do) it doesn't matter. Otherwise my personal opinion is that matrix profiles is not preferable, the linearity is a sort of mathematical beauty but I find that improved accuracy especially for high saturation colors is preferable. LUTs have a bit bad reputation though as software before has used them to correct without thinking about smoothness, and is often used to create "looks" which one may not agree with. I hope DCamProf can improve this reputation later on.
My understanding is that the accuracy will be better with matrix simply because the linearity is preserved. LUT profiles will introduce colour shifts because they can introduce non linearity and hue shifts
Title: Re: DCamProf - a new camera profiling tool
Post by: Bart_van_der_Wolf on May 05, 2015, 09:46:06 am
MakeInputICC will calculate the gamma from the meaurements. So if you are using RawDigger to get the target measurement, it will be close to the one you select there when saving the samples grid (at least that was my experience).

Hi Alexey,

Yes, but (besides the mostly linear RawDigger to CGATS) the LUT gamma is not a pure gamma, when I look at the CLUT slope of the default EOS-1Ds Mark III profile that comes with C1 , and try to fit a function. So if that's what C1 expects, then the profile should be more flexible than having a simple Matrix and/or gamma based CLUTs. I therefore see the difficulty that that poses for Anders, given that different assumptions are being used by different applications.

Cheers,
Bart
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on May 05, 2015, 09:51:11 am
My understanding is that the accuracy will be better with matrix simply because the linearity is preserved. LUT profiles will introduce colour shifts because they can introduce non linearity and hue shifts
LUT profiles also can be obtained from camera's SSFs, spectrum of illumination, etc... so while they can LUT is not a bad thing if done in a proper manner... guns do kill too, not that I am pro-NRA.
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on May 05, 2015, 09:54:59 am
Hi Alexey,

Yes, but (besides the mostly linear RawDigger to CGATS) the LUT gamma is not a pure gamma, when I look at the CLUT slope of the default EOS-1Ds Mark III profile that comes with C1 , and try to fit a function. So if that's what C1 expects, then the profile should be more flexible than having a simple Matrix and/or gamma based CLUTs. I therefore see the difficulty that that poses for Anders, given that different assumptions are being used by different applications.

Cheers,
Bart

well, you can always use C1 itself to make the tiff and take it from there, instead of using RD (or lobby for RD v2.0, long overdue, to include some more complex processing when generating cgats in addition to simple gamma)

PS: there is also linear scientific curve .fcrv that can be used instead of linear curve (just rename it) - for example the content of linear scientific curve .fcrv will switch off blown highligts correction in C1... try it

Title: Re: DCamProf - a new camera profiling tool
Post by: Alexey.Danilchenko on May 05, 2015, 10:13:05 am
LUT profiles also can be obtained from camera's SSFs, spectrum of illumination, etc... so while they can LUT is not a bad thing if done in a proper manner... guns do kill too, not that I am pro-NRA.

You are missing the point - it does not matter how you build them and with what precision. LUT is a grid of space to space mapping in the nodes and between the nodes it gets linearly interpolated. This interpolation can add nonlinearities especially if LUT nodes are chosen to correct some of the colours camera cannot distinguish. In these cases we are in with the hue shifts when exposure correction is applied.
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on May 05, 2015, 10:17:31 am
You are missing the point - it does not matter how you build them and with what precision. LUT is a grid of space to space mapping in the nodes and between the nodes it gets linearly interpolated. This interpolation can add nonlinearities especially if LUT nodes are chosen to correct some of the colours camera cannot distinguish. In these cases we are in with the hue shifts when exposure correction is applied.
well, you can find out what "colours camera cannot distinguish" and then it is not going to be worse than with matrix in terms of what you do with camera RGB input there... why shifts if exposure correction is after color transform (not RPP) ?
Title: Re: DCamProf - a new camera profiling tool
Post by: Alexey.Danilchenko on May 05, 2015, 10:25:08 am
well, you can find out what "colours camera cannot distinguish" and then it is not going to be worse than with matrix in terms of what you do with camera RGB input there
In what way it is not going to be worse? My understanding is that it will always be worse when your exposure is not spot on (at least it will increase the chances of those shifts).
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on May 05, 2015, 10:41:07 am
DCamProf uses a 2.5D LUT, this means that regardless of lightness the same correction is applied. This means that there is no exposure-dependent hue-shift in the profile. In very rigid copy-style setups with known objects to shoot a 3D LUT can be better (this is disussed in the docs), but in all other cases a 2.5D LUT is wiser. Adobe uses 3D LUT "LookTable" in most their profiles but that is to create a subjective Adobe-look(tm) which is a different thing.

There are other sources of non-linearities and hue shifts though, S-contrast curve, desaturation of highlights to mimic film behavior etc.

DCamProf's LUT nodes are the patches, after patches with (almost) the same chromaticity has been grouped together, and then depending on weighting those nodes can be relaxed towards neutral to minimize stretch/compression/bend. You can plot the LUT using gnuplot and adjust weights as desired.

By setting grouping distance to zero (-d 0) you can get the LUT correct each patch spot on, but that can lead to crazy bends, and as same color can be represented by different spectra (and will produce different errors) it does not really make sense to get spot on.

Any fairly recent camera has overlapping filters and is relatively good at color separation. There are still minimas where they don't separate too well, but if that happens to be in one of your patches I don't really see how that would be a problem. The LUT will then make an average correction for that general group of colors.

It is a problem if the LUT makes extreme stretches, but if that happens then something is generally very wrong, bad lighting of test target, bad reading of test target, bad reference file etc.
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on May 05, 2015, 11:27:35 am
In what way it is not going to be worse?
because you always can do around the same transform as matrix profile does around those areas, no ? matrix is LUT of sort too, just the one that does purely linear transform (multiplication).
Title: Re: DCamProf - a new camera profiling tool
Post by: Alexey.Danilchenko on May 05, 2015, 11:34:40 am
because you always can do around the same transform as matrix profile does around those areas, no ? matrix is LUT of sort too, just the one that does purely linear transform (multiplication).

You can use matrix to generate the same nodes as LUT yes. But whether the results of applying such profile for colours that located between those nodes will match the matrix one remains debatable.
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on May 05, 2015, 11:41:26 am
You can use matrix to generate the same nodes as LUT yes. But whether the results of applying such profile for colours that located between those nodes will match the matrix one remains debatable.

so Alex, what is going to happen if you (hopefully) get your monochromaror + sphere + ...  working ? you will get SSF and then you will build matrix (plus possible gamma/s/ or shaper/s/) profile or you will go for LUT ?
Title: Re: DCamProf - a new camera profiling tool
Post by: Alexey.Danilchenko on May 05, 2015, 11:51:36 am
so Alex, what is going to happen if you (hopefully) get your monochromaror + sphere + ...  working ? you will get SSF and then you will build matrix (plus possible gamma/s/ or shaper/s/) profile or you will go for LUT ?

I will build  my perfect matrix profile (not relying on CC24 or other targets)  ;D

I was not even interested in LUTs. If you need specifics I am interested in finding the better fitting matrix for (a) different illuminants and (b) for better matching the digital back with specific IR filter (the one I have namely). LUT would be interesting for me if I had nonlinearities (perhaps like those in D800) to correct for but as it stands I don't need it.

DCamProf is a perfect tool for me to experiment with these.
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on May 05, 2015, 12:16:30 pm
I will build  my perfect matrix profile
waiting to see the results then !
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on May 05, 2015, 12:18:20 pm
Just to show an example, this is how a LUT stretching can look like, the test target is an IT8 which has been cleaned up a bit to exclude most darker patches, the camera is a P45+.

Colored squares show the target positions, arrows show the LUT stretch from matrix position to end position (generally close to the target). The grid shows the LUT thin plate spline in action. There's some stretch in the lightness direction too (not seen in this view) but it's very small.

There are indeed quite large stretches, but no bad bends as there are no contradicting stretches.

It shows very typical behavior in that for low saturation colors there are small stretches (ie the matrix comes close to target), and for high saturation colors there are larger stretches. The u'v' diagram is not perfectly uniform though so it exaggerates that effect a bit.

This is natural, more saturation means more narrow band, which means that the difference between the camera's SSFs and the observer's CMFs are enlarged and becomes harder to correct with a linear equation.

A general comment on the IT8 target, the many repeats of the darker shades does not provide that much value, but the target does reach considerable higher saturation than a cc24, some patches even outside Pointer's gamut, so it looks like it's quite a good target.
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on May 05, 2015, 12:56:43 pm
A general comment on the IT8 target, the many repeats of the darker shades does not provide that much value
I'd love to see some further discussion, becuse just recently somebody was stating even that intentional bracketing is worthwhile - see http://forum.luminous-landscape.com/index.php?topic=22471.msg818606#msg818606
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on May 05, 2015, 01:15:16 pm
so it looks like it's quite a good target.
I had this one from WFaust for like 7 years and it was somewhere far away in the attic... but I extracted it back couple of days ago and liked (to my subjective taste) some results, I am certainly not an expert to render a scientific grade judgement - just my personal impression (so far)... for further experimenting I am assembling some additional gear - like it was suggested 80A filters for halogen light, waiting to get them... easel to mount targets on a proper surface (metal with magnets) - again as for example IBorg suggested
Title: Re: DCamProf - a new camera profiling tool
Post by: Alexey.Danilchenko on May 05, 2015, 02:16:00 pm
waiting to see the results then !

Sure - if I make the monochromator work...
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on May 05, 2015, 02:31:51 pm
I'd love to see some further discussion, becuse just recently somebody was stating even that intentional bracketing is worthwhile - see http://forum.luminous-landscape.com/index.php?topic=22471.msg818606#msg818606

I think Iliah is referring to bracketing as a method to keep down noise. I don't think that is worthwhile either though. If by principle there will be relatively large errors in the results, say "5%" it doesn't provide much value to reduce measurement error from 0.5% to 0.1%. There is such a thing called overkill.

Darker patches provide some value in that they are probably printed using a different colorant mix so you get a slightly different spectrum shape than a lighter color of the same hue and chroma, that way you can make some more average correction. For very dark patches I would guess they contribute more noise than meaningful correction though. And yes I guess in that case bracketing would help, but why not have a test target with brighter colors instead I think.

Anyway, if you try to make exact correction for each shade, you are in 3D LUT space and pretending that the camera will be used as a scanner, with fixed light, fixed exposure *and* fixed media. The IT8 target steps is nice for a slide film target for a slide film scanner, I actually have one and have made Argyll ICC profiles for a slide scanner using it. As long as the slides you're scanning are made on the same film as the slide target you're getting very accurate results.

With a camera used in general conditions it won't be like that. Let's pretend for a while that the light is fixed and the media is fixed, but not the exposure. Then the 3D LUT correction still breaks, since when a color is exposed brighter than it was for the LUT it will be corrected wrong.

That is to DCamProf with a 2.5D LUT the only thing the darker shades provide is slight variation, but then I think a neighboring chromaticity of similar lightness would be better use of target space.
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on May 05, 2015, 02:37:11 pm
Sure - if I make the monochromator work...

I'm thinking about getting a monochromator too. You can get one new for about $1200. Most use a spectroradiometer ($10000) to measure the output, and many an integrating sphere to smoothen the light ($2000 I suppose). So naturally one would like to skip the spectroradiometer and integrating sphere.

I'm thinking that one could use a halogen lamp, measure the spectra with a consumer spectrometer, then hope it will stay stable enough while filtering it through the monochromator and doing measurements, and you shoot the slit directly, without any diffuser. I think it could work, but I don't know, may be issues with reflections. If one uses a simple white card to reflect the light shooting becomes simpler but measuring the reference becomes harder, then one needs a spectroradiometer rather than a consumer spectrometer.

What's your plan?

Here below a high end setup:
http://www.cis.rit.edu/jwgu/research/camspec/img/camspec_database.png
Title: Re: DCamProf - a new camera profiling tool
Post by: ErikKaffehr on May 05, 2015, 02:51:58 pm
Hi Anders,

I have checked back on this and I now believe it is Lightroom that applies some kind of shoulder on the highlights. So it is not a profile issue.

Best regards
Erik

Hi,

....

There is an issue, that some of the light grey samples are to dark.

Excellent job from Anders. I hope he will continue to develop the tool.

Best regards
Erik
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on May 05, 2015, 03:04:40 pm
With a camera used in general conditions it won't be like that. Let's pretend for a while that the light is fixed and the media is fixed, but not the exposure. Then the 3D LUT correction still breaks, since when a color is exposed brighter than it was for the LUT it will be corrected wrong.

what is the difference between more exposure time and more light (intensity wise = brighter patches) in this example ?  you will not see the difference in the raw file (let us ignore sensor heating)

Title: Re: DCamProf - a new camera profiling tool
Post by: Alexey.Danilchenko on May 05, 2015, 03:07:22 pm
I'm thinking that one could use a halogen lamp, measure the spectra with a consumer spectrometer, then hope it will stay stable enough while filtering it through the monochromator and doing measurements, and you shoot the slit directly, without any diffuser. I think it could work, but I don't know, may be issues with reflections. If one uses a simple white card to reflect the light shooting becomes simpler but measuring the reference becomes harder, then one needs a spectroradiometer rather than a consumer spectrometer.

What's your plan?


Much more modest - I got the full spectrum monochromator (manual one with light source) but no power supply (and a weird connectors for PSU). Not sure if it will work and it will need some calibrating/checking. I also will have an integrating sphere (albeit a small one) though I am half way though building my own (not so high spec of course as spectralon coated ones). Will update with progress if I make it work.

I am not sure how accurate the measurement without the sphere will be since you need to measure the irradiation (intensity) to adjust the camera responses and without sphere those readings will be hit or miss.

These discussions should perhaps go to a different thread - this one is about DCamProf after all.
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on May 05, 2015, 03:10:43 pm
Hi Anders,

I have checked back on this and I now believe it is Lightroom that applies some kind of shoulder on the highlights. So it is not a profile issue.

Best regards
Erik


generally it shall be process 2010 in ACR/LR and also just in case infuse a linear tone curve (applied post exposure adjustments) in profile as
 
  <ToneCurve Size="2">
    <Element N="0" h="0.000000" v="0.000000"/>
    <Element N="1" h="1.000000" v="1.000000"/>
  </ToneCurve>


in dcptool speak
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on May 05, 2015, 03:15:02 pm
What's your plan?
have to rob nearby university... or enroll in MSc  :D (repeat for each new camera)
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on May 05, 2015, 03:15:33 pm
generally it shall be process 2010 in ACR/LR and also just in case infuse a linear tone curve (applied post exposure adjustments) in profile as
 
  <ToneCurve Size="2">
    <Element N="0" h="0.000000" v="0.000000"/>
    <Element N="1" h="1.000000" v="1.000000"/>
  </ToneCurve>


in dcptool speak

I should probably add a function to add a curve to the DCP. If there is no curve, like DCamProf DCPs are, some raw converters seem to add a default curve which may not be what you want. Meanwhile one can add one manually with dcp2json / json2dcp (or dcptool I suppose)
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on May 05, 2015, 03:27:42 pm
what is the difference between more exposure time and more light (intensity wise = brighter patches) in this example ?  you will not see the difference in the raw file (let us ignore sensor heating)

No difference. But I was describing an example when exposure changes for a fixed light condition.

The difference then is that the LUT was designed with the patches at a different lightness. With a 2.5D LUT as DCamProf uses it does not matter, as the index into the LUT is only chromaticity, not lightness. If there's a 3D LUT index is hue, saturation (chromaticity) plus lightness. To match the correct patch the exposure must be the same as when the LUT was designed.

When lightness is not a parameter into the LUT there is much less value in making steps, the only value is in adding slight variation in terms of chromaticity and spectral shapes.
Title: Re: DCamProf - a new camera profiling tool
Post by: Alexey.Danilchenko on May 05, 2015, 06:49:27 pm
I should probably add a function to add a curve to the DCP. If there is no curve, like DCamProf DCPs are, some raw converters seem to add a default curve which may not be what you want. Meanwhile one can add one manually with dcp2json / json2dcp (or dcptool I suppose)

Lightroom/ACR add default one which is mix between gamma and contrast curve. So if linear profile is needed, then curve should be explicitly stated.
Title: Re: DCamProf - a new camera profiling tool
Post by: kirkt on May 06, 2015, 01:49:25 pm
Just a bug (maybe) in the "txt2ti3" command of dcamprof.

I assume that the input is a text file, whitespace delimited, that contains the spectral data from, for example, PatchReader.  Nothing else.  My PatchReader data is in 10nm increments, in rows, from 380nm to 730nm.  I cut and paste the spectral data from the PatchReader output text file into a text document called "spectral_data.txt" and issue:

$ dcamprof txt2ti3 -f 380,730,10 -l rows spectral_data.txt spectral_data.ti3

The range of the spectral elements is correct (380-730) but the increments are still in 5nm (the default) not 10nm (my command line argument to the -f flag).  Attached is a test file - spectraltest.txt - that is a copy of the whitespace delimited input file.

Still, it beats editing the PatchReader output by hand to conform to the reference CIE file you include with your distribution!  

kirk

EDIT -Is this command supposed to reissue/recalculate/interpolate my input spectra into 5nm increments, regardless of what the specified input spacing is?  If so, sorry, I misunderstood what the output was supposed to be!

EDIT2 - The scaling seems to be buggy though - I noticed in the reference CIE file that you include, the range of spectral values is 0-100 (not 0.0-1.0).  Given this reference range,  I output my values from PatchReader as 0-100.  When I run txt2ti3, I do not specify a scaling value (I assume the default, -s 1.0 according to the help message).  However, my spectral values get scaled by 100 in the output file.  If I use "-s 0.01" then the scaling is not applied and the output values are identical to the input values.  Is there a multiplication of 100 going on somewhere?  Or is the default value for the -s flag 100 and not 1.0?
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on May 06, 2015, 02:34:35 pm
Currently the txt2ti3 always resamples to 5nm increments and 0-100 scale. I'll probably change to not resample, I just thought it would be easier to manually cut and paste if always in 5 nm, as a single file must have the same sampling.

The 0-100 scaling takes place because the file is supposed to look like an Argyll file and Argyll uses 0-100 as default scale for spectra, ie 100 means 1.0. When reading the .ti3 file DCamProf translates to 1.0 internally.

I suppose the problem is that you want to use the .ti3 file in some other software than DCamProf or Argyll and therefore want a 0-1 range? The "hack" with providing 0.01 as scale I guess works then, or do I need to add some feature?
Title: Re: DCamProf - a new camera profiling tool
Post by: kirkt on May 06, 2015, 05:05:57 pm
Hi Anders,

No problem- I am using it with dcamprof.  The problem was I exported from patch reader already scaled 0-100. I thought the txt2ti3 "-s" flag was a scaling value to use when computing the output. That is, because my values are already 0-100 I would want to use "-s 1.0" (don't scale). I guess the scale value is the scale rNge that exists in the in the input  file?

Either way, please don't wory about it!

Kirk
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on May 07, 2015, 03:29:49 am
Alexey, concerning the use of monochromator to make the best matrix profile I think you're in overkill space.

The thing is that to make a good matrix fit the matching task must not be too difficult. With modern cameras the CC24 is matched pretty good with a matrix, but that's because there are no extreme saturation colors. It can be seen that this good matching of the CC24 is made at cost of extreme colors.

If you instead try to make a matrix that will match also extremely saturated colors, with the help of a glossy target or indeed SSF and spectral data, you will get a much poorer match so the CC24-range of colors will suffer. You can try this with DCamProf now using the provided SSF example. Pure matrix profiles are best when optimized only for normal range colors, and for normal range colors a standard test target procedure will do just as good as SSF, possibly even better as I think it's harder to make an exact monochromator measurement.

I also know that SSF is not really something *all* manufacturers are using, some design the profiles the traditional way using targets and subjective tunings, so one cannot really say that SSF will make better profiles. It's probably easier to make dual-illuminant DCPs with better color temp predictions though as you can use the exact standardized illuminants though. Adobe is probably using SSFs.

That said I very much myself like to be able to measure SSF since once you have them you can make lot of interesting experiments, but I'm not really sure I would use an SSF-based profile for "production", haven't decided on that. It is possible to merge both SSF and traditional measurement into one .ti3 and make a profile from that too, but it's probably difficult as light sources need to match exactly.

But if I would only want a matrix profile, I would do it based on a test target.
Title: Re: DCamProf - a new camera profiling tool
Post by: Alexey.Danilchenko on May 07, 2015, 04:49:07 am
Anders,

I have build a few of the matrix profiles from larger CC24 and passport and without further editing in daylight lighting they are hit or miss to say the least. Iliah makes cracking matrix profiles for RPP but as far as I know this involve a lot of manual colour correction past shooting profile based on sample images of known colours and reprofiling again.

The idea with spectral data for a sensor is to automate that manual colour correction step because all the expected responses for the needed colours of a scene can be generated and better fitting matrix produced. So the approach is far from useless to me.

The only decent matrix profiles I have managed to build from target and without post processing were made with QPCard 203 (a version of it not glued to the cardboard holder so was very easy to achieve needed flatness). Those still weren't perfect in some areas even on cameras with good colour separation (in that particular instance Kodak Pro Back with custom IR filter).
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on May 07, 2015, 05:19:34 am
Ok, for that purpose having SSF will be very convenient of course.
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on May 07, 2015, 10:06:57 am
I also know that SSF is not really something *all* manufacturers are using, some design the profiles the traditional way using targets and subjective tunings, so one cannot really say that SSF will make better profiles.

who are those ? are they "manufacturers" of 3rd party raw converters of note or "manufacturers" of cameras that do supply either their own OEM software (or putting OEM dcp profiles in DNG files - like Ricoh/Pentax for example) ? it is interesting to know the names and compare vs others who then allegedly are using SSFs then, no ?

Title: Re: DCamProf - a new camera profiling tool
Post by: torger on May 07, 2015, 10:29:56 am
who are those ? are they "manufacturers" of 3rd party raw converters of note or "manufacturers" of cameras that do supply either their own OEM software (or putting OEM dcp profiles in DNG files - like Ricoh/Pentax for example) ? it is interesting to know the names and compare vs others who then allegedly are using SSFs then, no ?

I get some things shared in confidence so in this case I did not want to share the names, you just have to trust me ;-)
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on May 07, 2015, 10:53:41 am
you just have to trust me
it is not about trust that they do, it is about who they are, so that we can see how they fare vs others really... but it seems that you are not going to tell, unless we get some help from CIA  :D !
Title: Re: DCamProf - a new camera profiling tool
Post by: Alexey.Danilchenko on May 07, 2015, 11:24:38 am
it is not about trust that they do, it is about who they are, so that we can see how they fare vs others really... but it seems that you are not going to tell, unless we get some help from CIA  :D !

Anders simply may be bound by NDA
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on May 07, 2015, 03:08:05 pm
I have some ICC code half-working now, probably release next week. I just need to figure out some white balance confusion I'm having...

It will be matrix-only for linear pipelines only in the first release, so no Capture One. But I plan to support that eventually too.

Title: A "torture sample"
Post by: ErikKaffehr on May 07, 2015, 06:03:18 pm
Hi Anders,

This image may be an interesting sample: http://echophoto.dnsalias.net/ekr/Articles/OLS_OnColor/SimpleCase/Data/20150107-CF046070_AdobeStandard.iiq

Color samples are here:

http://echophoto.dnsalias.net/ekr/Articles/OLS_OnColor/SimpleCase/Data/GreenBlade_spectrum.txt

http://echophoto.dnsalias.net/ekr/Articles/OLS_OnColor/SimpleCase/Data/Violet.txt

http://echophoto.dnsalias.net/ekr/Articles/OLS_OnColor/SimpleCase/Data/Green.png

http://echophoto.dnsalias.net/ekr/Articles/OLS_OnColor/SimpleCase/Data/Violet.png

Best regards
Erik
Title: Re: DCamProf - a new camera profiling tool
Post by: GWGill on May 08, 2015, 01:39:09 am
If you instead try to make a matrix that will match also extremely saturated colors, with the help of a glossy target or indeed SSF and spectral data, you will get a much poorer match so the CC24-range of colors will suffer.
As I understand it, to make a useful profile using sensor spectral measurements, you need to introduce a weighted model of real world colors, so that your matrix has least errors for "typical" use (where "typical" is set by your real world color weighting).
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on May 08, 2015, 02:37:57 am
As I understand it, to make a useful profile using sensor spectral measurements, you need to introduce a weighted model of real world colors, so that your matrix has least errors for "typical" use (where "typical" is set by your real world color weighting).

Yes that's probably a good idea, and you can do that with DCamProf using the -w parameter in the make-profile step.

I haven't had time to experiment that much with matrix-only profiles but it seems like the best "weighting" is actually to exclude those highly saturated colors and only include those that you expect to see in normal situations. It seems like the extreme colors are struggling too much in a different direction that it's impossible to make a reasonable match for those without clearly degrading the normal colors. Results will vary depending on camera though.
Title: Re: A "torture sample"
Post by: torger on May 08, 2015, 06:20:19 am
The link to the IIQ doesn't work, and the greenblade spectrum text file lacks spectral data.

I'm using the provided patchtool text file examples to make the dcamprof make-target file parser a little bit more accepting on formats.

Hi Anders,

This image may be an interesting sample: http://echophoto.dnsalias.net/ekr/Articles/OLS_OnColor/SimpleCase/Data/20150107-CF046070_AdobeStandard.iiq
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on May 08, 2015, 09:28:24 pm
0.5.4 executable built for windows with mingw ( includes dcamprof.exe and libgomp_64-1.dll ) = https://app.box.com/s/l9w24lobqr2nr08439zxpzndcmvwkj70
Title: Re: A "torture sample"
Post by: ErikKaffehr on May 09, 2015, 03:10:25 am
Hi Anders,

Sorry for the mess!

I put all info here: http://echophoto.dnsalias.net/ekr/Articles/ColorProfiles/Samples

Hopefully it works! For some reason a lot of files get protected for read from time to time.

I added a sample from my Sony Alpha 99, too.

Some background:

I had some discussion with Tim Parkin. He finds that the P45+ has inherently bad reproduction of chlorophyll green, making it yellowish. I don't disagree but I wanted to find out if it is a profiling thing or not.

So I bought a small flower, cut away some samples to measure with my ColorMunkey and shot it with strobe light. The reason a bluish purple flower was chosen was that I have seen it was a colour that easily turned into blue or reddish blue in conversion.

Best regards
Erik


The link to the IIQ doesn't work, and the greenblade spectrum text file lacks spectral data.

I'm using the provided patchtool text file examples to make the dcamprof make-target file parser a little bit more accepting on formats.

Title: Re: DCamProf - a new camera profiling tool
Post by: torger on May 09, 2015, 11:16:06 am
Thanks, I will play with the files later.

The use of camera profiling and tools to do it is not so widespread so cameras and raw converters plus profiles is inseparable for most photographers. I think the camera is blamed many times when it is more about profile design, but if you do not have the tools or knowledge to make your own the color you get from the factory and bundled software is the final result.

Cameras do differ in separation ability though, but all cameras today, even older ones like the P45, has overlapping filters so they can separate colors rather well within their noise limits. So I think it is more up to profile design than many may think.

That chlorophyll green becomes yellowish can definitely be adjusted, but if there is metamerism no profile can fix it. The color separation test in DCamProf will show where color separation is weak, but SSF curves are required to do that test. I think I have curves for the H3D 39, same sensor, probably different IR filter though. I shall play around with those and see what I find. The builtin database kuopio-natural has leaves of many kinds, and there are more databases to find online.

I know Tim thinks the SMI measure (which dxo shows) correlates well with actual performance, but I am less sure about that when you have LUT profiles. More testing is required before I could form my own view though.

A camera with poor SMI will likely have better color separation ability in some color ranges than an ideal camera (a camera where SSF = observer CMF), and that could be greens.
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on May 09, 2015, 01:49:21 pm
That chlorophyll green becomes yellowish can definitely be adjusted, but if there is metamerism no profile can fix it.
what if you can move the color from the area of metameric failure with some filters though... for a particular scene
Title: Re: A "torture sample"
Post by: Tim Lookingbill on May 09, 2015, 05:05:37 pm
I had some discussion with Tim Parkin. He finds that the P45+ has inherently bad reproduction of chlorophyll green, making it yellowish. I don't disagree but I wanted to find out if it is a profiling thing or not.

Best regards
Erik

From the excellent looking images I've seen of your work off the P45+, Erik, I've noticed they do come off as having a slight green (more like an ashen blueish green) bias with a light hand on saturation. Remember my suggestion of applying a luminance/saturation increase with Hue/Sat tool in Photoshop on your gallery exhibit image of the sunlit green knoll with the background mountains you linked to in the LuLa Coffee Corner forum?

For me adaptation working too long on a landscape with a magenta-ish blue sky (cobalt blue has a magenta element) requires constant retweaking of WB to get the right looking green. I do know from years of examining sunlit green plants that the sunlit portion of a non-waxy, midtone green leaf/grass does have a yellowish bias, but ACR/LR's exact hue that makes it look right requires very gradual adjusts due to the cyan portion of the green that can make shadows appear too cool which invokes the adaptive effect of seeing green highlights as warm or a thalo green.

To filter this cyan requires moving toward magenta which slightly warms up green highlights but with a much different hue. I would never let my custom DNG profile fix this because it made the greens look a slightly dull yellowish orange green. And I didn't like moving temp slider toward blue because it desaturates everything.

After a long WB edit with ACR4.4 profile I'ld walk away and come back to see I'ld made everything too green because I kept trying to get the cyan in the green tint slider to freshen up the green. So I went back to As Shot WB (returned tweaked 0 back to +6 to +10 toward magenta), selected the custom DNG profile, walked away, took a break, came back and it looked perfect. Hue changes applying the custom DNG profile is so subtle in warming up the greens that I never considered how much WB affects the overall perception of color cast. I think one has to consider color constancy in the mix of things rather than attributing it to metamerism.

I mean I had trouble getting the right green hue shooting Live Oak trees that I did a google image search and found WB and greens all over the map...

https://www.google.com/search?newwindow=1&hl=en&site=imghp&tbm=isch&source=hp&biw=1556&bih=941&q=live+oak+in+texas&oq=live+oak+in+texas&gs_l=img.3..0i8i30l2j0i24.3707.9093.0.9964.17.16.0.1.1.0.114.1335.15j1.16.0.msedr...0...1ac.1.64.img..0.17.1334.aZKWyeTORI4

None of them are correct looking.
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on May 10, 2015, 06:20:22 am
I just became aware that Argyll's printtarg can output chart files, that is it's probably really easy to make your own targets if you have a good inkjet printer and a spectrometer to measure it. I shall test and update the docs accordingly. I think an inkjet print will have at least as good spectral qualities as an IT8 target on photo paper.

Maybe I'll add a target generator later on as Argyll's targen is more tailored for printer and scanner targets.
Title: Re: DCamProf - a new camera profiling tool
Post by: Tim Lookingbill on May 10, 2015, 12:53:15 pm
I just became aware that Argyll's printtarg can output chart files, that is it's probably really easy to make your own targets if you have a good inkjet printer and a spectrometer to measure it. I shall test and update the docs accordingly. I think an inkjet print will have at least as good spectral qualities as an IT8 target on photo paper.

Maybe I'll add a target generator later on as Argyll's targen is more tailored for printer and scanner targets.

Would the use of the spectrometer and software determine whether certain colors in a custom target are beyond a display's ability to reproduce like say Pantone 313C (coated)? A Fuji Frontier drylab inkjet can reproduce it but currently there are no reference displays. No telling what other colors there are for creating a custom target that would fit nicely and improve current photographic workflow processes.

Has someone come up with a more robust color target with a specific selection of colors that can fit within both Raw capture and display gamut workflows that would result in the least "gamut crunching" color errors?   
Title: Re: DCamProf - a new camera profiling tool
Post by: Bart_van_der_Wolf on May 10, 2015, 01:52:02 pm
Has someone come up with a more robust color target with a specific selection of colors that can fit within both Raw capture and display gamut workflows that would result in the least "gamut crunching" color errors?

Hi Tim,

Gamut crunching only happens on profile conversions. It does not affect the creation of a camera profile IMHO, unless one seeks to make a dumbed-down target that only covers the Adobe RGB colorspace (at best). But such a profile that's based on such a limited target would not be able to reliably map colors outside the limited aRGB gamut.

It's much better to use a target with relevant colors that the camera is able to 'see', and covers it full gamut as far as relevant existing colors go, and sometimes only use part of its full gamut if the subject poses a limited challenge.

Cheers,
Bart
Title: Re: DCamProf - a new camera profiling tool
Post by: Alexey.Danilchenko on May 10, 2015, 02:21:20 pm
I just became aware that Argyll's printtarg can output chart files, that is it's probably really easy to make your own targets if you have a good inkjet printer and a spectrometer to measure it. I shall test and update the docs accordingly. I think an inkjet print will have at least as good spectral qualities as an IT8 target on photo paper.

Maybe I'll add a target generator later on as Argyll's targen is more tailored for printer and scanner targets.

There is little benefit of printing your own camera targets on common 4 colours inkjets. The  number of paints used for each patch will be limited to that of the original 4.
Title: Re: DCamProf - a new camera profiling tool
Post by: Tim Lookingbill on May 10, 2015, 02:46:13 pm
Hi Tim,

Gamut crunching only happens on profile conversions. It does not affect the creation of a camera profile IMHO, unless one seeks to make a dumbed-down target that only covers the Adobe RGB colorspace (at best). But such a profile that's based on such a limited target would not be able to reliably map colors outside the limited aRGB gamut.

It's much better to use a target with relevant colors that the camera is able to 'see', and covers it full gamut as far as relevant existing colors go, and sometimes only use part of its full gamut if the subject poses a limited challenge.

Cheers,
Bart

How has it been determined what colors the camera is able to 'see'?

I'ld have to assume this optimized for digital sensor color target would have to comprise a certain gamut shape and size that would be tuned to the math that can map colors to the display. Not sure if the shape of the gamut is more important than the size. The 3D model of most inkjets show some colors go outside the AdobeRGB gamut where most others are well within.

Can distinction between subtle differences of colors be improved by using a custom target that's more tuned to the real color gamut shape that describes how a digital camera 'sees' color?
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on May 10, 2015, 03:03:16 pm
There is little benefit of printing your own camera targets on common 4 colours inkjets. The  number of paints used for each patch will be limited to that of the original 4.

I think many like me has the higher end pigment inkjets, which have several more inks. I shall do a more thorough spectral comparison later.
Title: Re: DCamProf - a new camera profiling tool
Post by: Alexey.Danilchenko on May 10, 2015, 03:15:03 pm
I think many like me has the higher end pigment inkjets, which have several more inks. I shall do a more thorough spectral comparison later.

A quote from stephen stuff (https://stephenstuff.wordpress.com/2011/09/30/targets-for-camera-profiling/) to describe some of the problems.

Quote
Spectral problems

Camera profiling targets are commonly photographed in sunlight, which includes ultraviolet wavelengths (UV). Photographic papers can contain “fluorescent whitener additives” (or “optical brightening agents”) which makes these papers appear more blue. These papers and printed camera profiling targets can show colour shifts towards blue when there is ultraviolet in the light source.

Metamerism is another problem for photographic and printed targets. The processes are optimised to produce colour, usually from just three of four colourants, that look natural to the human eye but actually might be composed of quite different spectra. This is a problem if the camera spectral sensitivities are different to the human eye. ColorChecker targets are made using multiple different pigments, giving reflective spectra that are more representative of the real world.

In fact if the solution would be as easy as to print your own target with however good colour coverage you need then anyone would not have problem obtaining good profiles. Alas this is not what I see happening yet.
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on May 10, 2015, 03:15:43 pm
DCamProf's color separation diagram gives an idea what colors a camera can "see", most modern cameras can differentiate colors in most of the human gamut, well beyond limits of say AdobeRGB.

Profiles generally distort extreme colors though and may move them outside valid space so they get clipped to a nearby color. Extremely saturated colors is a narrow special ccase so few have problems with that.

If you really want to cover an extreme gamut with your profile, designing using SSF is the answer, which you can do with DCamProf.

Spectrometer data will indeed give the profiler full information of the color. Consumer spectrometers are a little limited in range, often cover only 400-700nm which can be a problem in some special cases.
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on May 10, 2015, 03:21:01 pm
A quote from stephen stuff (https://stephenstuff.wordpress.com/2011/09/30/targets-for-camera-profiling/) to describe some of the problems.


Already well-known and discussed in DCamProf's docs. I need more testing but initial results suggest that OBA-free paper and pigment inkjets do a good job rivaling what you find in cc24 or munsell book of colors. There are probably some problem colors though,  need to look at the spectra of all patches to give a full report. The bad reputation comes from old print systems and basic consumer papers.

I'm currently travelling so I can't test that much.
Title: Re: DCamProf - a new camera profiling tool
Post by: Bart_van_der_Wolf on May 10, 2015, 03:35:18 pm
How has it been determined what colors the camera is able to 'see'?

Isn't that the purpose of DCamProf, to allow encoding colors that the camera can see? Whether and how that translates into humanly visible colors is another exercise, and whether and how our output modalities reproduce them is yet another. If we cannot adequately encode the input, there is no hope for getting good output.

Quote
I'ld have to assume this optimized for digital sensor color target would have to comprise a certain gamut shape and size that would be tuned to the math that can map colors to the display. Not sure if the shape of the gamut is more important than the size. The 3D model of most inkjets show some colors go outside the AdobeRGB gamut where most others are well within.

Yes, that's why an inkjet printed target may not be the best basis. Remember the exercise that Bruce Lindbloom did to find the parameters for his BetaRGB colorspace (http://www.brucelindbloom.com/index.html?BetaRGB.html). It comprises many important colors that may need to be encoded, but as few more extreme ones as possible, to improve the quantization step precision.

Quote
Can distinction between subtle differences of colors be improved by using a custom target that's more tuned to the real color gamut shape of a digital camera 'sees' color?

Not all colorspace coordinates represent humanly visible colors, and they are thus by definition not 'colors'. What matters is that a camera can capture and distinguish between as many 'colors' as possible. This is not going to be perfectly possible due to the Luther-Ives condition (more here (http://www.ncbi.nlm.nih.gov/pmc/articles/PMC4299059/)), so some form of perceptual mapping will ultimately be needed.

The goal of creating a camera profile is to get a solid basis for the conversions that are to follow. Accurate for important colors, smooth transition to intermediate colors, able to reduce metameric and color constancy issues. But that is more an input or scene referred profiling than the ACR centric output referred profiling that gives us hue twists and other trouble.

I prefer clean scene referred profiling, making use of an adequate but not overly large working space, and a perceptually based output processing, e.g. using a CIECAM like color appearance model for output. I am not sure if an inkjet print will be challenging enough for a camera sensor, although it does allow to produce some saturated Cyans, Yellows, and Magentas. Apparently those dyes or pigments are available in nature, so we should also be able to encode those.

Cheers,
Bart
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on May 10, 2015, 03:53:04 pm
Afaik only Argyll and now DCamProf allows custom targets, which is a cost saver if you already have good printing gear, plus that commercial targets are focused on ease of use, minimize risk of reflection and other lighting errors rather than maximizing profile performance. Custom targets are definitely worth investigating deeply.

Target design is only a small part of the problem though. Profiling is an unsolvable problem (infinite spectral variation etc), it's always a compromise and therefore just as much an art as a craft. Many (most?) people prefer "looks" rather than neutrally accurate too, so it all boils down to taste. The best design method/target for you may not be the best for me.
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on May 10, 2015, 04:00:10 pm
Already well-known and discussed in DCamProf's docs. I need more testing but initial results suggest that OBA-free paper and pigment inkjets do a good job rivaling what you find in cc24 or munsell book of colors. There are probably some problem colors though,  need to look at the spectra of all patches to give a full report. The bad reputation comes from old print systems and basic consumer papers.

I'm currently travelling so I can't test that much.

I just measured a regular IT8 target from Wolf Faust (my is quite old, but spent most of the time in dark non humid no extreme temperatures storage) with i1pro2, here are the spectral files = https://app.box.com/s/o4gwrk0d5slw7y1g0nu4bhs1fevg775d that is a bottom level of what can be printed if you have access to proper printer/ink/paper, no ?
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on May 10, 2015, 04:04:42 pm
DCamProf only cares about human gamut currently, it's not good for IR cameras for example. You can probably do SSF design with IR but color must be related to an observer, and currently only human observers are covered. How a color we can't see should look is not so easy to define ;-)

Also note that DNG profiles unlike ICC by design clips the gamut to prophoto. DCamProf's native format is unbounded though so the upcoming ICC support will not be clipped.

The prophoto clipping can be an issue in scientific applications, but hardly for ordinary photography.
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on May 10, 2015, 04:05:13 pm
Afaik only Argyll and now DCamProf allows custom targets
ProfileMaker for example too - just supply .tiff and matching .cie
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on May 10, 2015, 04:16:43 pm
Thanks AlterEgo that it8 data will be great for reference. I do expect that a pigment inkjet target will be at least as relevant as that. However the criticism above also targets photographically printed targets like Faust's It8, and suggests special printing like used for cc24 is required for adequate results.

Worth investigating.
Title: Re: DCamProf - a new camera profiling tool
Post by: GWGill on May 10, 2015, 08:34:09 pm
I just became aware that Argyll's printtarg can output chart files, that is it's probably really easy to make your own targets if you have a good inkjet printer and a spectrometer to measure it.
Note that input targets created using a printing process have their limitations compared to targets created using a wider range of pigments.

The chart ideally should represent the spectra of what you are trying to capture, and something created with 4 or so inks has limited coverage of possible metamers.

Those capturing something like original artwork have had good success in creating custom test charts composed of the types of pigments that they intend capturing.
Title: Re: DCamProf - a new camera profiling tool
Post by: GWGill on May 10, 2015, 10:05:03 pm
Would the use of the spectrometer and software determine whether certain colors in a custom target are beyond a display's ability to reproduce like say Pantone 313C (coated)?
Perfectly possible. You need to be aware of what you are assuming with regard to white point adaptation though. (i.e. because reflective colors are measured standard to D50 illuminant, and most displays are D65).
Quote
Has someone come up with a more robust color target with a specific selection of colors that can fit within both Raw capture and display gamut workflows that would result in the least "gamut crunching" color errors?   
Not following you there. The gamut of an input device test chart has nothing to do with how out of gamut colors get handled further down the chain. Limiting the input test chart merely makes the camera/scanner characterization less accurate in the part of the gamut you haven't tested. Colors beyond that will still be captured by the input device. All colors are within a cameras gamut if its exposure is properly set (i.e. a camera doesn't have a gamut in the same sense an output device has.)
Title: Re: DCamProf - a new camera profiling tool
Post by: GWGill on May 10, 2015, 10:18:17 pm
How has it been determined what colors the camera is able to 'see'?

I'ld have to assume this optimized for digital sensor color target would have to comprise a certain gamut shape and size that would be tuned to the math that can map colors to the display.
A camera doesn't have a gamut. If one of the channels hits saturation, you reduce exposure. If everything is black, you increase exposure.

What a camera has instead is a spectral mismatch to a human observer. There are colors that are metamers to a camera that are not a match to a human observer, and vise versa. Unless it's spectral sensitivities are exactly the equivalent of the standard observer, a matrix or other color profile can only approximately convert a camera RGB into the human XYZ values, and can do nothing about metameric mismatches.

A consequence of this is that any particular profile will allow a camera to produce imaginary colors, as well as there being colors that the camera can never produce (i.e. the human spectral locus and profiled camera spectral locus will not match).
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on May 11, 2015, 03:01:47 am
A camera doesn't have a gamut in the sense of a printer, as it's an input profile, but it does have limitations of what colors it can differentiate (and capture). This can be visualized with the DCamProf color separation diagram, here an example of a Canon 5Dmk2:

(http://www.ludd.ltu.se/~torger/img/ssf_csep.png)

The darker color the worse color separation (related to the standard observer), and black it can't separate at all, ie all local colors are registered as one and the same.

The camera's SSF rarely stretches out as wide as the CMF so you get limits on maximum red and maximum violet, and in the case of older cameras you don't have red/blue overlap so you have some metamerism, modern cameras have very little metamerism though although the SNR will vary of course. The SSF shapes differ greatly from the CMF so you get some minimas here and there seen as dark spots in the diagram.

The diagram doesn't involve a profile, just the SSFs. A modern camera can generally separate colors near or at the locus, but to get a match you need strongly non-linear correction (LUT) and the strong non-linearity can hurt performance of normal colors so it's generally not a good idea to try to make a profile with maximum gamut.

A profile optimized for a smaller gamut will generally cause chaos for extreme colors, even moving them into invalid positions so they are clipped away, visualized in this diagram:

(http://www.ludd.ltu.se/~torger/img/testprof2.png)

where you see short error vectors for normal colors and looong ones for extreme colors, some stretching outside the gamut. I'm sure many that has SSFs for their cameras will be tempted to make the "ultimate" profile which makes the camera perform in almost the whole human gamut but that's probably a mistake as it will most likely hurt performance of the normal colors that are 100% of the colors in 99% of the scenes you shoot.
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on May 11, 2015, 03:16:53 am
Note that input targets created using a printing process have their limitations compared to targets created using a wider range of pigments.

The chart ideally should represent the spectra of what you are trying to capture, and something created with 4 or so inks has limited coverage of possible metamers.

Those capturing something like original artwork have had good success in creating custom test charts composed of the types of pigments that they intend capturing.

I'm very aware of this, however it seems to me than noone has really tested this to depth the past 10 years or so. Pigment inkjets today doesn't have 4 inks, they like six colors and 3 neutrals, and for each new ink generation improvements have been made in metamerism etc. I shall make a thorough spectral analysis when I get some time, I'm sure there are problem colors but I'm also sure today's pigment inkjets perform much better now than consumer printers did when the criticism towards printed test targets was formed, so I think there are reasons to re-consider this option.

Also special-printed test targets like a CC24 or munsell's book of colors has limitations. I have compared with spectral databases from skin and nature and real spectra of course show much larger variation than you can capture in a few patches. With DCamProf you can profile against those spectral databases directly using camera SSF, but due to the natural limitations of camera profiling it's not that sure it will be that much better. Likewise I don't think it's that sure that a special-printed target will be that much better than a pigment inkjet printed target, and the home-made target has the advantage that it can if you desire be made much more saturated (cover a larger chromaticity gamut) than typical commercial matte test targets do.

Apart from traditional real-world testing with your eyes, with DCamProf it's possible to make comparisons virtually, you can measure your home-made target with a spectrometer, and make a target from a spectral database, and use the spectra from a CC24 or other commercial spectra for one more, and then make profiles from SSFs for all those and compare how well they match various spectra. So there's good opportunity to make some serious testing. I'm still in for making some more features in the software though so I don't have that much time to do all those tests myself.
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on May 11, 2015, 04:09:44 am
Here's attached a few patches from canon pigment lucia ink on oba-free baryta paper, CC24 spectra, some real reflectance spectra from nordic nature (leaves, flowers etc), and finally some patches from IT8 Faust spectra provided by AlterEgo.

For the Baryta it seems like we have a colorant gap around 470nm and 570nm, and it's less smooth than CC24.

I think these plots indeed does show that the CC24 does match natural spectral shapes better than printed spectra, but that the printed spectra is certainly not as bad as it could be. The printed spectra also show higher saturation colors than the CC24 and nature so it will therefore make steeper shapes. A better/easier comparison would be to try matching the CC24 colors with the printer and see how those shapes look, I may do that later when I have some time.

On a first look the IT8 spectra looks pretty good, smooth. Certainly not obvious that it would be worse than the spectral shapes provided by CC24. There's the rise of all spectra towards 720+nm, but I don't think that matters that much as the sensitivity of the observer is low there.
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on May 11, 2015, 08:13:20 am
Just released 0.6.0 which now also can make ICC profiles in addition to DNG profiles (added make-icc, icc2json and json2icc commands). It's still limited for ICCs though as it can only make matrix-only ICCs (no LUT), and only for raw converters that feed linear data to the profile. That is no Capture One support yet, but RawTherapee and probably many other ICC-using raw converters out there should work.
Title: Re: DCamProf - a new camera profiling tool
Post by: Bart_van_der_Wolf on May 11, 2015, 08:27:16 am
Just released 0.6.0 which now also can make ICC profiles in addition to DNG profiles (added make-icc, icc2json and json2icc commands). It's still limited for ICCs though as it can only make matrix-only ICCs (no LUT), and only for raw converters that feed linear data to the profile. That is no Capture One support yet, but RawTherapee and probably many other ICC-using raw converters out there should work.

Hi Anders,

Thanks. You did intend to take it a bit slower, catch a breath, and recuperate from the exhausting climb sofar, but I'm glad you didn't ... ;)
Eagerly awaiting Capture One usable ICCs, but I'm patient (I know the next step is quite a bit more complex).

Cheers,
Bart
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on May 11, 2015, 08:38:48 am
Eagerly awaiting Capture One usable ICCs
matrix profiles are perfectly usable in C1 unless you want to use color editor, which requires LUT to be present there... tiff itself can be output from C1 instead of making it with dcraw (or similar)
Title: Re: DCamProf - a new camera profiling tool
Post by: papa v2.0 on May 11, 2015, 08:54:00 am
Hi everyone.
Interesting thread.
This may be of use to some.

Iain
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on May 11, 2015, 09:00:11 am
matrix profiles are perfectly usable in C1 unless you want to use color editor, which requires LUT to be present there... tiff itself can be output from C1 instead of making it with dcraw (or similar)

Capture One makes non-linear tiffs which DCamProf doesn't handle yet. DCamProf's matrix ICC doesn't have any shaper curves so input must be linear. I'll fix that eventually :-)
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on May 11, 2015, 09:00:39 am
Just released 0.6.0

to build with mingw I had to

1) procure arpa/inet.h - borrowed from matlab (as I have it already installed)

and

2) in profio.c

#include <pthread.h> - that fixes localtime_r ()


and borrowed the following code snippet from internet

static inline uint16_t
htons (x)
     uint16_t x;
{
#if BYTE_ORDER == BIG_ENDIAN
  return x;
#elif BYTE_ORDER == LITTLE_ENDIAN
  return __bswap_16 (x);
#else
# error "What kind of system is this?"
#endif
}

#define ntohs htons

static inline uint32_t
htonl (x)
     uint32_t x;
{
#if BYTE_ORDER == BIG_ENDIAN
  return x;
#elif BYTE_ORDER == LITTLE_ENDIAN
  return __bswap_32 (x);
#else
# error "What kind of system is this?"
#endif
}

#define ntohl htonl
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on May 11, 2015, 09:08:18 am
DCamProf's matrix ICC doesn't have any shaper curves so input must be linear. I'll fix that eventually :-)
but you can quickly put pure gamma trc tags, 1.8 is working for C1... instead of calculating some curve
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on May 11, 2015, 09:11:54 am
0.6.0 binaries for Windows build with mingw ( dcamprof.exe and libgomp_64-1.dll ) = https://app.box.com/s/6k6zw0suw3bkxrtqwmo6cl1pzrg2zp19
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on May 11, 2015, 09:20:38 am
Thanks for windows info, shall update to make easier to compile again to next release.

I guess the ntohl/ntohs etc should work if you include <winsock2.h> instead of <arpa/inet.h>? I'll make an ifdef for that.
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on May 11, 2015, 09:29:26 am
but you can quickly put pure gamma trc tags, 1.8 is working for C1... instead of calculating some curve

I think I need to look at the TIFFTAG_TRANSFERFUNCTION tag which since C1 6 (or 7?) is included in the exported TIFF. It's not a pure 1.8 curve.
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on May 11, 2015, 09:35:57 am
Thanks for windows info, shall update to make easier to compile again to next release.

I guess the ntohl/ntohs etc should work if you include <winsock2.h>?

removing code snippet and replacing with <winsock2.h> results in linker errors

 undefined reference to `__imp_ntohl'
 undefined reference to `__imp_ntohs'

so you need to add some extra libraries then

Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on May 11, 2015, 09:37:10 am
I think I need to look at the TIFFTAG_TRANSFERFUNCTION tag which since C1 6 (or 7?) is included in the exported TIFF. It's not a pure 1.8 curve.

yes, that tag was mentioned somewhere here by some P1 employee, but gamma = 1.8 still works OK meanwhile - may be not for purists though...
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on May 11, 2015, 09:38:24 am
removing code snippet results in linker errors

 undefined reference to `__imp_ntohl'
 undefined reference to `__imp_ntohs'

so you need to add some extra libraries then

Uhh... I'll see, maybe I'll remove the swap dependencies to next version it's easy for me to do.
Title: Re: DCamProf - a new camera profiling tool
Post by: Bart_van_der_Wolf on May 11, 2015, 09:50:35 am
I think I need to look at the TIFFTAG_TRANSFERFUNCTION tag which since C1 6 (or 7?) is included in the exported TIFF. It's not a pure 1.8 curve.

FWIW, attached is how the Green TRC of the default Capture One profile for the EOS 1DS Mark III profile looks. Definitely not a simple gamma, and the C parameter of a curve fit is closer to 1/1.942.... So maybe something like that can be found back in the TIFF's metadata tags if saved with Camera profile.

Cheers,
Bart
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on May 11, 2015, 09:55:38 am
here is for example rawdigger/makeinputicc argyll frontend (so w/o tiff generated by C1 - albeit you can perfectly convert C1 tiff to DNG and feed to the same workflow, just don't WB samples when exporting to CGATS file and set black level = 0 in RD) with g1.8 vs (where is which profile ?) C1 v8.2 with Sony A7 "Standard" profile... I am not advocating g = ~1.8, I am just saying that meanwhile  you can live with it.... PS: profile was built using printed IT8 target illuminated by halogen light not filtered up to D50 (I did not have filters handy) by the way...

(http://s21.postimg.org/5m04s2c05/C1_IT8_RDA.jpg)

Title: Re: DCamProf - a new camera profiling tool
Post by: Alexey.Danilchenko on May 11, 2015, 11:44:27 am
removing code snippet and replacing with <winsock2.h> results in linker errors

 undefined reference to `__imp_ntohl'
 undefined reference to `__imp_ntohs'

so you need to add some extra libraries then


Winsock is a library, simple include does not make it. I guess a separate makefile for mingw is needed.
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on May 11, 2015, 12:01:35 pm
Winsock is a library, simple include does not make it. I guess a separate makefile for mingw is needed.
including the code to define the functions for mingw build works OK I guess....
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on May 11, 2015, 02:06:18 pm
Here's attached a few patches from canon pigment lucia ink on oba-free baryta paper, CC24 spectra, some real reflectance spectra from nordic nature (leaves, flowers etc), and finally some patches from IT8 Faust spectra provided by AlterEgo.
also QP202 spectral (with i1pro2 and with colormunki) = https://app.box.com/s/un6liz50nrkpn59aem0gnrg7395wbear
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on May 11, 2015, 03:08:32 pm
FWIW, attached is how the Green TRC of the default Capture One profile for the EOS 1DS Mark III profile looks. Definitely not a simple gamma, and the C parameter of a curve fit is closer to 1/1.942.... So maybe something like that can be found back in the TIFF's metadata tags if saved with Camera profile.

I don't think the TRC curves of the LUT profiles say that much, if I've understood things correctly they are not even used at all in terms of camera profiling. As far as I know if you have an A2B0 LUT the *TRC and *XYZ tags are ignored, as there is both input curves and matrix in the LUT A2B0 tag itself.

And in that case the input curve may only be there to make a mapping trick to get better integer precision (ICCv2 is an old format integer math unfortunately) and the LUT itself may cancel out that curve and apply something new, which you can't see without putting data through the profile. In other words, it's not possible to use an ICC viewer and that way figure out which pre-processing C1 does. I think the TIFFTAG_TRANSFERFUNCTION in the exported profiling TIF is the thing to look for, looking into that now.

Those TRC curves also show an inverted curve, like you say 1/1.942 when a matrix profile actually needs ~1.8 to linearize, so they really doesn't make any sense to me.

The ICCv2 standard show two types of input profiles 1) XYZ+TRC tags (matrix plus shaper curves), 2) A2B0 only. The mix of both 1&2 is some non-standard stuff by Capture One as far as I understand, which may be because they are used both as camera profiles and can also be attached to a TIFF file just like sRGB.icm or AdobeRGB.icm.

If any ICC guru knows how it all fits together, please let me know...
Title: Re: A "torture sample"
Post by: ErikKaffehr on May 12, 2015, 01:14:45 am
Hi Tim,

I tried to your advice, but this is a season to taste thing.

BTW, those mountain images were all with the Sonys. No P45+ image made it to the wall. I don't know why. I have just had it for something like 22 months, so that matters. Also, I didn't use it on the workshop, for different reasons.

I am going to look into your recommendations, they are much appreciated.

Best regards
Erik






From the excellent looking images I've seen of your work off the P45+, Erik, I've noticed they do come off as having a slight green (more like an ashen blueish green) bias with a light hand on saturation. Remember my suggestion of applying a luminance/saturation increase with Hue/Sat tool in Photoshop on your gallery exhibit image of the sunlit green knoll with the background mountains you linked to in the LuLa Coffee Corner forum?

For me adaptation working too long on a landscape with a magenta-ish blue sky (cobalt blue has a magenta element) requires constant retweaking of WB to get the right looking green. I do know from years of examining sunlit green plants that the sunlit portion of a non-waxy, midtone green leaf/grass does have a yellowish bias, but ACR/LR's exact hue that makes it look right requires very gradual adjusts due to the cyan portion of the green that can make shadows appear too cool which invokes the adaptive effect of seeing green highlights as warm or a thalo green.

To filter this cyan requires moving toward magenta which slightly warms up green highlights but with a much different hue. I would never let my custom DNG profile fix this because it made the greens look a slightly dull yellowish orange green. And I didn't like moving temp slider toward blue because it desaturates everything.

After a long WB edit with ACR4.4 profile I'ld walk away and come back to see I'ld made everything too green because I kept trying to get the cyan in the green tint slider to freshen up the green. So I went back to As Shot WB (returned tweaked 0 back to +6 to +10 toward magenta), selected the custom DNG profile, walked away, took a break, came back and it looked perfect. Hue changes applying the custom DNG profile is so subtle in warming up the greens that I never considered how much WB affects the overall perception of color cast. I think one has to consider color constancy in the mix of things rather than attributing it to metamerism.

I mean I had trouble getting the right green hue shooting Live Oak trees that I did a google image search and found WB and greens all over the map...

https://www.google.com/search?newwindow=1&hl=en&site=imghp&tbm=isch&source=hp&biw=1556&bih=941&q=live+oak+in+texas&oq=live+oak+in+texas&gs_l=img.3..0i8i30l2j0i24.3707.9093.0.9964.17.16.0.1.1.0.114.1335.15j1.16.0.msedr...0...1ac.1.64.img..0.17.1334.aZKWyeTORI4

None of them are correct looking.
Title: Re: DCamProf - a new camera profiling tool
Post by: GWGill on May 12, 2015, 02:56:56 am
And in that case the input curve may only be there to make a mapping trick to get better integer precision (ICCv2 is an old format integer math unfortunately)
Little to do with integer encoding (it's rarely an issue), but to reduce the impact of the limited resolution of cLUT tables. Practical cLUT tables sizes are limited by memory/disk space, net bandwidth & CPU speed.
Quote
The ICCv2 standard show two types of input profiles 1) XYZ+TRC tags (matrix plus shaper curves), 2) A2B0 only. The mix of both 1&2 is some non-standard stuff by Capture One as far as I understand, which may be because they are used both as camera profiles and can also be attached to a TIFF file just like sRGB.icm or AdobeRGB.icm.
It's a tag format, so you can add almost any tags you want, including both matrix profile and cLUT tags. To be legal though, the PCS needs to be XYZ if the matrix profile tags are included. I have seen illegal profiles that label PCS as L*a*b* for the cLUT and then includes matrix profile elements, but not every CMM will cope with such a Frankenstein.
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on May 12, 2015, 03:02:06 am
Little to do with integer encoding (it's rarely an issue), but to reduce the impact of the limited resolution of cLUT tables. Practical cLUT tables sizes are limited by memory/disk space, net bandwidth & CPU speed.It's a tag format, so you can add almost any tags you want, including both matrix profile and cLUT tags. To be legal though, the PCS needs to be XYZ if the matrix profile tags are included. I have seen illegal profiles that label PCS as L*a*b* for the cLUT and then includes matrix profile elements, but not every CMM will cope with such a Frankenstein.

Thanks for the info. I'll have to figure out the cLUT mapping/resolution issue when I implement cLUT support. I think I have an idea of how to do it...

Anyway, that makes Phase One profiles illegal (what a surprise...), as they contain matrix profile tags plus lab PCS for the A2B0. LittleCMS seems to swallow it though as it can display TIFFs with these profiles embedded, and it looks right to my eyes. I don't know which tags it looks at in this displaying case, it would be interesting to know. I guess the A2B0 tag needs to be used somehow otherwise it would not display right.

I've noted that some of the newer profiles don't have TRC and XYZ tags though, but only the A2B0 tag.

(I've got my matrix profiles sort of working in Capture One now by parsing out the TIFFTAG_TRANSFERFUNCTION from their profiling tiffs and applying that as TRC shaper curve. Brightness/contrast looks right, but the colors are too saturated... I have so far no idea why.)
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on May 12, 2015, 03:39:24 am
Attached a plot of the transfer functions as reported in the profiling tiff by Capture One. A pure gamma 1.8 curve is used as reference (yellow), and then the two curves "linear scientific" (almost gamma 1.8  ), and then "film standard".

This is for the P45+.

Note that linear scientific does not reach 1.0, which I assume is intended to clip away noise close to clipping.
Title: Re: DCamProf - a new camera profiling tool
Post by: Alexey.Danilchenko on May 12, 2015, 04:31:49 am
Anders,

Thanks for the new version. It does not seem to parse v4 ICC profiles. Had a look at the code and it looks like you parsing the ICC file yourself - any reason why? The DCamProf already requires lcms2 and it has a full loader and acces to all profile internals already there (and handles v2 and v4).
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on May 12, 2015, 07:20:57 am
Thanks for the new version. It does not seem to parse v4 ICC profiles. Had a look at the code and it looks like you parsing the ICC file yourself - any reason why? The DCamProf already requires lcms2 and it has a full loader and acces to all profile internals already there (and handles v2 and v4).

The reason why is because the ICC standard is very big and if I parse anything DCamProf needs to handle anything. It's much better to block things early so the software internals doesn't have to see things it doesn't understand. To actually generate ICC profiles I need to know exactly how they work, which tags and headers that are included etc. I have deliberately narrowed down DCamProf's ICC handling to only camera profiles. Getting into the huge LCMS2 ICC API, which is huge because it's supporting all aspects of ICC standard, is probably not much easier than making the parser for ICCv2 profiles, and when making the parser I learnt a lot about the format which I needed to know anyway. Only the LCMS2 header file is longer than the complete ICC parsing code in DCamProf. DCamProf's internal ICC representation is very scaled down, only contains the elements required for cameras, it's not a full generic ICC.

I don't love ICC that much that I want to spend lots of time to implement JSON conversion for printer profiles or handle and generate ICCv4 if no popular raw converter requires it :)

That different raw converters handle ICCs differently, and even throws around invalid ICCs (like Capture One) makes ICC a pain to work with. I just like to have as little pain as possible :)

(there's still some bugs in the icc2json and icc lut stuff which will be fixed to the next patch release.)
Title: Re: DCamProf - a new camera profiling tool
Post by: Alexey.Danilchenko on May 12, 2015, 07:30:48 am
The reason why is because the ICC standard is very big and if I parse anything DCamProf needs to handle anything. It's much better to block things early so the software internals doesn't have to see things it doesn't understand. To actually generate ICC profiles I need to know exactly how they work, which tags and headers that are included etc. I have deliberately narrowed down DCamProf's ICC handling to only camera profiles. Getting into the huge LCMS2 ICC API, which is huge because it's supporting all aspects of ICC standard, is probably not much easier than making the parser for ICCv2 profiles, and when making the parser I learnt a lot about the format which I needed to know anyway. Only the LCMS2 header file is longer than the complete ICC parsing code in DCamProf. DCamProf's internal ICC representation is very scaled down, only contains the elements required for cameras, it's not a full generic ICC.

I don't love ICC that much that I want to spend lots of time to implement JSON conversion for printer profiles or handle and generate ICCv4 if no popular raw converter requires it :)

That different raw converters handle ICCs differently, and even throws around invalid ICCs (like Capture One) makes ICC a pain to work with. I just like to have as little pain as possible :)

The LCMS2 profile hanling API - that is the one that accesses tags and checks the version is very simple. So is creational one (have a look at dcp2icc sources it creates ICC profile there via LCMS calls and writes LUT to it). I was not suggesting to use it in other capacities or other profiles - so I am not really sure I understang the reasons to code the parsing and reading in full. The only API you need to access profile data is to open profile from a file and to get the tag data. The writing is a bit more since it involves calling various set methods for the header but still not the full CMS API.


Title: Re: DCamProf - a new camera profiling tool
Post by: torger on May 12, 2015, 08:38:07 am
The code in DCamProf ICC parsing has an error btw - the logic where it determines version of the profile is incorrect and valid V2 profile with version encoded as 0x22000000 is producing error stating that it is V4 profile. These lines

Which software has produced that profile? The version check should not be incorrect according to the standard document, the first byte is major, the second byte is minor + patch, ie 0x22000000 means ICCv22.0.0 not ICCv2.2.0. I'm grateful for help finding bugs, but think it's better to deal with typo bugs in private messages as most Lula readers are not programmers :)
Title: Re: DCamProf - a new camera profiling tool
Post by: GWGill on May 12, 2015, 08:47:32 am
Getting into the huge LCMS2 ICC API, which is huge because it's supporting all aspects of ICC standard, is probably not much easier than making the parser for ICCv2 profiles, and when making the parser I learnt a lot about the format which I needed to know anyway.
I really don't recommend cooking up your own (the Firefox guys did that, and made a pigs ear of it). It has taken some of us rather a long time to figure out some of the subtleties, best practices, and interoperability gotchas - far better to leverage Marti's experience and support.

Title: Re: DCamProf - a new camera profiling tool
Post by: torger on May 12, 2015, 09:10:29 am
I really don't recommend cooking up your own (the Firefox guys did that, and made a pigs ear of it). It has taken some of us rather a long time to figure out some of the subtleties, best practices, and interoperability gotchas - far better to leverage Marti's experience and support.

Making ICCs work for raw converters is full of that regardless, it's a whole different space. The most popular ICC-supporting raw converter, ie Capture One, uses illegal ICC profiles and does some pre-processing outside the ICC's intention. I can't just go for basic use cases and say Capture One is broken, sorry guys. Oh well, I could, but then I could just skip ICC support all-together. I've implemented support for various ICC profiles in RawTherapee and there are at least four specific incompatible types I've had to handle. I'm not going to support them all with DCamProf though.

I'm not into re-inventing the wheel, but it's ~500 lines of fairly trivial code with fixed scenarios, the hardest part is converting to from the various fixed point formats. I may actually switch back to LCMS2 later on (it is a good idea using existing code) and replace those 500 lines, but now this is the fastest way forward. The code is there. I first looked at using LCMS2 but I got very frustrated that there were no easy to find example code for the use cases I needed, made a quickslam hack with own code just to get going and learnt/reapeated a lot of important aspects of the format meanwhile.

LCMS2 will almost certainly be used when the test-profile command gets ICC support, ie for running colors through the profile.

I plan to support Capture One, excluding Leaf-style ICCs, and then any raw converter with basic linear pre-processing, possibly add some trivial thing on request. I won't do Nikon-style ICCs, I won't do v4 ICCs.
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on May 12, 2015, 09:25:57 am
Attached a plot of the transfer functions as reported in the profiling tiff by Capture One. A pure gamma 1.8 curve is used as reference (yellow), and then the two curves "linear scientific" (almost gamma 1.8  ), and then "film standard".

interesting to compare that with curves data in .fcrv files from C1 itself

x/y coordinates [0.0...1.0] pairs are coded as 4 bytes x 2 containing an UINT (4 bytes, 32bit, little endian) which shall be divided by 4294967295 ( 0xFFFFFFFF ), for example 4 bytes = "0x86" "0xF2" "0x1A" "0x0A" = 169538182 and / 4294967295 = 0.03947368 = 0.039474

the content (partial) of .fcrv files

(http://1.bp.blogspot.com/-OrghB5FurJI/VRsm9EJuoxI/AAAAAAAAHtA/LZNa-W3HNV0/s1600/1.jpg)

(http://3.bp.blogspot.com/-AF97oeshlqI/VRswMIR0GTI/AAAAAAAAHtQ/SBIXi4sdXG4/s1600/2.jpg)

(http://3.bp.blogspot.com/-HofjW71Jmzc/VRs4CsghyQI/AAAAAAAAHtg/H9gPIr4L9W0/s1600/3.jpg)

Title: Re: DCamProf - a new camera profiling tool
Post by: torger on May 12, 2015, 09:29:56 am
Without checking I would guess that they are the same, seems logical. I want to reverse-engineer as little as possible of Capture One though, they don't deserve this attention for making it hard to make ICC profiles for them :).

interesting to compare that with curves data in .fcrv files from C1 itself
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on May 12, 2015, 09:41:35 am
Without checking I would guess that they are the same, seems logical.
but they are not... linear and linear scientific are linear curves as coded in .fcrv files [0.0,0.0] and [1.0,1.0] - at least in recent cameras... so whatever your see is C1 code doing something with each curve when writing the tag
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on May 12, 2015, 09:46:07 am
but they are not... linear and linear scientific are linear curves as coded in .fcrv files [0.0,0.0] and [1.0,1.0]... so whatever your see is C1 code doing something with each curve when writing the tag

Have you checked the P45+ curves? It was those I plotted. Maybe they are different per camera? I haven't checked but I know there are different sets of curves available depending on camera chosen.
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on May 12, 2015, 09:48:11 am
Have you checked the P45+ curves? It was those I plotted. Maybe they are different per camera? I haven't checked but I know there are different sets of curves available depending on camera chosen.

true - may be that camera indeed has different curve - I was checking the very recent ones, with .fcrv = 128 bytes... it seems that older camera are not that linear   :D

PhaseOneP45+-Linear Scientific.fcrv = 128 bytes and the content indicates 2 points for the curve, so it is linear there
PhaseOneP45+-Linear Response.fcrv = 9 points for the curve - not linear

for comparison

PhaseOneP65+-Linear Response.fcrv
PhaseOneP65+-Linear Scientific.fcrv

are both 128 bytes with 2 points for curves and the only one byte (content) difference, which apparently is a tag that tells C1 that if Linear Scientific then it shall not invent data to paint clipped highlights in a nice way
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on May 12, 2015, 09:56:28 am
Thanks for the info Alterego, those things help in figuring out what's happening in the C1 ICC conversion pipeline.
Title: Re: DCamProf - a new camera profiling tool
Post by: Alexey.Danilchenko on May 12, 2015, 11:00:42 am
Which software has produced that profile? The version check should not be incorrect according to the standard document, the first byte is major, the second byte is minor + patch, ie 0x22000000 means ICCv22.0.0 not ICCv2.2.0. I'm grateful for help finding bugs, but think it's better to deal with typo bugs in private messages as most Lula readers are not programmers :)

My fault - you right, and your code is correct. It is not my profile but some of the users on russian forums. I'll try to get more information about how it was made.
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on May 12, 2015, 11:05:57 am
Anyway I agree that it's a good idea using LCMS2 for parsing, but I had my reasons not to and now I'm not going to change it this moment. I'm already on the next challenge :). Cleaning up code comes later (=usually meaning never ;) )
Title: Re: DCamProf - a new camera profiling tool
Post by: Alexey.Danilchenko on May 12, 2015, 12:01:00 pm
Anyway I agree that it's a good idea using LCMS2 for parsing, but I had my reasons not to and now I'm not going to change it this moment. I'm already on the next challenge :). Cleaning up code comes later (=usually meaning never ;) )
I can have a go at doing this but closer to the end of May ...
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on May 12, 2015, 03:26:41 pm
I think I have it working on C1 now, can hopefully release in the next few days.

The saturation issue was simply because the P45+ raw data looks different (more saturated) out from C1 than from Adobe DNG / DCraw / Anyone Else. I don't know why. Maybe C1 makes a pre-matrixing to reduce color space to get a smaller space for the ICC to work on, which probably is a good idea as the LUT has limited resolution.

It would be interesting to know what it is though, because without being able to revert it it won't be possible to make profiles for C1 directly from SSFs.

I haven't checked how it is with other cameras though, the P45+ could be an exception...
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on May 12, 2015, 03:46:56 pm
I think I have it working on C1 now, can hopefully release in the next few days.

The saturation issue was simply because the P45+ raw data looks different (more saturated) out from C1 than from Adobe DNG / DCraw / Anyone Else. I don't know why. Maybe C1 makes a pre-matrixing to reduce color space to get a smaller space for the ICC to work on, which probably is a good idea as the LUT has limited resolution.

It would be interesting to know what it is though, because without being able to revert it it won't be possible to make profiles for C1 directly from SSFs.

I haven't checked how it is with other cameras though, the P45+ could be an exception...

may be C1 sees its own back and simple does something extra hardcoded there... I mean more than they do for regular dSLR/dSLM/P&S
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on May 12, 2015, 04:40:34 pm
A few C1 findings:

- tifftag_transferfunction seems to be it concerning linearizing the profiling tiff data
- C1 linear is very similar to but not exactly the same as gamma 1.8
- some more preprocessing than the film curve is applied before ICC, sometimes extra clip range, and possibly some matrixing (P45+), more investigation can be done, not required for DCamProf though, unless someone wants to make C1 profiles from SSFs.
- C1's own profiles applies a slight S-curve also in the ICC Lab LUT (probed several profiles, always the same), ie linear curve won't be linear for C1 profiles, but generated profiles that are truly linear and apply the tifftag_transferfunction should be linear as far as I can see.
- The only way to see which curve C1's native profile applies so to run data through the LUT and see what you get. The curves embedded in the profile say nothing about the end result.

I have some issues with ultrasaturared blues in my P45+ test profile, may be a clipping issue.
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on May 13, 2015, 02:31:41 am
is it possible for the sake of saving some (quite some) time during the (multiple !) experiments to augment "make-dcp" command with options to write 1) profile name and 2) baseline exposure offset  instead of repeating dumping, editing those fields, compiling back ?
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on May 13, 2015, 04:05:03 am
is it possible for the sake of saving some (quite some) time during the (multiple !) experiments to augment "make-dcp" command with options to write 1) profile name and 2) baseline exposure offset  instead of repeating dumping, editing those fields, compiling back ?

I'll probably look into streamlining the workflow a bit later on and add more options when the program has matured a bit. Meanwhile you could make a script for that using your favourite script language :)
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on May 13, 2015, 09:58:16 am
I'll probably look into streamlining the workflow a bit later on and add more options when the program has matured a bit. Meanwhile you could make a script for that using your favourite script language :)
that is certainly doable in many ways, yes - so willl do this for now... another question is - may be instructions for DCamProf can be updated that it is possible to use rawdigger (albeit profile edition is not free) to extract raw RGB values to CGATS and then use txt2ti3 to produce .ti3 from that and .cie instead of scanin route ( and you can always position the grid on target better manually )... plus scanin does not put spectral data in .ti3 by itself ... additional advantage of involving rawdigger ( www.rawdigger.com ) is to do flatfielding and txt2ti3 is to provide spectral data from .cie (if they are there of course) when making .ti3 in one step.
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on May 13, 2015, 11:29:43 am
Thanks for the tip, I'll update the docs when/if I test the workflow myself.

scanin will include spectral data in the .ti3 if the .cie file contains it.
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on May 13, 2015, 11:45:00 am
scanin will include spectral data in the .ti3 if the .cie file contains it.
does it ? it seems that it didn't... I just tried with .cie file that has XYZ, LAB, CH and spectral data and .ti3 only has

SAMPLE_ID XYZ_X XYZ_Y XYZ_Z RGB_R RGB_G RGB_B STDEV_R STDEV_G STDEV_B

but no spectral data in it

next I removed XYZ LAB CH data from .cie and left only spectral and scanin says

"D:\argyll\bin\scanin.EXE: Error - Input file 'qpnm.cie' doesn't contain field XYZ_X or LAB_L"

am I missing something then ?
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on May 13, 2015, 12:46:24 pm
does it ?

It probably does not understand the patchtool CGATS format, it the spectral data comes in Argyll's own format it will include it. See the provided cc24_ref.cie as an example.
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on May 13, 2015, 01:23:46 pm
It probably does not understand the patchtool CGATS format, it the spectral data comes in Argyll's own format it will include it. See the provided cc24_ref.cie as an example.
but txt2ti3 does understand the same format in the same file w/ no issues... so that's something that Graeme G. can comment - how come that scanin does not understand what txt2ti3 does for the purpose of creating the same ti3 file...

anyways will check cc24_ref.cie then, thank you for the note - it did not happen to me to think about that difference
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on May 13, 2015, 01:29:07 pm
I've just released 0.6.1, which can make ICC profiles with LUT, and also supports Capture One.

http://www.ludd.ltu.se/~torger/dcamprof.html

I also made a small adjustment to the native LUT such that whitepoint is always preserved.

I haven't tested it a whole lot, I hope some Capture One users will help out with some ICC testing :)
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on May 13, 2015, 02:17:22 pm
I've just released 0.6.1, which can make ICC profiles with LUT, and also supports Capture One.

http://www.ludd.ltu.se/~torger/dcamprof.html

I also made a small adjustment to the native LUT such that whitepoint is always preserved.

I haven't tested it a whole lot, I hope some Capture One users will help out with some ICC testing :)

ouch.... now to build libtiff - is there a place where people just share prebuilt static libraries  :( ?
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on May 13, 2015, 02:36:15 pm
ouch.... now to build libtiff - is there a place where people just share prebuilt static libraries  :( ?

RawTherapee uses libtiff and a bunch of other builds, pre-built stuff for windows is found at, not sure how libtiff is configured though.
http://www.rawtherapee.com/shared/builds/windows/dependencies_for_creating_builds/
Otherwise the compile.txt for RawTherapee can help when it comes building yourself:
https://code.google.com/p/rawtherapee/source/browse/COMPILE.txt
The Cygwin environment has libtiff there from the beginning so there it's just "make" to build it, but of course a pure Mingw build is nicer.

Libtiff is currently only needed for reading the TIFFTAG_TRANSFERFUNCTION that Capture One workflow needs, but in the future I may add other TIFF functionalities.
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on May 13, 2015, 03:40:54 pm
RawTherapee uses libtiff and a bunch of other builds, pre-built stuff for windows is found at, not sure how libtiff is configured though.
http://www.rawtherapee.com/shared/builds/windows/dependencies_for_creating_builds/
Otherwise the compile.txt for RawTherapee can help when it comes building yourself:
https://code.google.com/p/rawtherapee/source/browse/COMPILE.txt
The Cygwin environment has libtiff there from the beginning so there it's just "make" to build it, but of course a pure Mingw build is nicer.

Libtiff is currently only needed for reading the TIFFTAG_TRANSFERFUNCTION that Capture One workflow needs, but in the future I may add other TIFF functionalities.

thank you... I prefer to have just as few pieces as possible so I want to use mingw of course.
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on May 13, 2015, 04:17:50 pm
Some comments about ICC LUT; this really is a weak spot of ICC. The LUT maps from camera RAW RGB triplets to LabD50 triplets. A problem is that a very large space of the LUT will be meaningless values, a camera never registers 1,0,1 for example, you nearly always have some signal on all three channels.

Capture One seems to have worked around this a little by pre-matrixing to reduce the "raw" color space and thus make more use of the LUT space.

Another issue is that it always have to be 3D even if you have 2.5D data, which means that you have to have quite low resolution or else the ICC file will be huge, a 33x33x33 cube could have been a 190x190 2.5D square instead. With a suitable input curve the low resolution seems to work alright in practice though.

An ICC profile is typically applied after exposure correction (DCP HSM is applied before), and that's fortunate otherwise resolution could suffer if you push a file a lot.

If you bring up a DCamProf ICC in a viewer and look at the Lab gamut you will see a large "box", the reason is that the native profile have information that extends all the way up to +/-128 ICC clip point. Those extreme colors will never be used in a real scene though (that wasted LUT space I mentioned) so don't worry that it looks a little rough and boxy. Capture One's native profiles aren't that boxy, probably because they do some sort of gamut mapping in the profile. DCamProf won't do any gamut mapping and I don't plan to add that either as I don't think it's the job of the profile, if you have out of gamut colors just reduce saturation etc in the raw converter with your color tools until it fits.
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on May 13, 2015, 08:01:59 pm
0.6.1 build for Windows (mingw = dcamprof.exe + libgomp_64-1.dll) : https://app.box.com/s/h2lwvnzi96c9jkdzv6m7gs9drrsrdfdb
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on May 13, 2015, 10:09:30 pm
DCamProf-generated icc profile(s) do not work for C1 (v8.2, PC/Win8.1x64)... I used the same json that was technically OK for dcp profile in ACR 9 (I mean that ACR recognized, showed and I was able to select it in ACR UI) and C1 does not recognize or show it.. for simplicity I used just a pure matrix/g1 profile... also a regular tools like http://www.color.org/wxProfileDump.zip or http://www.color.org/ICCProfileInspector2_4.zip can't open it  ???

CORRECTION - it does recognize them, I simply put in a wrong directory - however the tools do not open profiles still (probably some important tags are still missing ?)

CORRECTION 2 - not so simple... for example when I put a custom made (with argyll) icc profile for my camera in either "C:\Users\<user>\AppData\Local\CaptureOne\Color Profiles" or even directly in "C:\Windows\System32\spool\drivers\color" then C1 sees it, but not the profile I generate with dcamprof - it only can be seen when it goes to where OEM profiles are ("C:\Program Files\Phase One\Capture One 8\Color Profiles\DSLR")... which is certainly wrong, user supplied profiles for C1 shall go to "C:\Users\<user>\AppData\Local\CaptureOne\Color Profiles" location.
Title: Re: DCamProf - a new camera profiling tool
Post by: GWGill on May 13, 2015, 11:02:57 pm
but txt2ti3 does understand the same format in the same file w/ no issues... so that's something that Graeme G. can comment - how come that scanin does not understand what txt2ti3 does for the purpose of creating the same ti3 file...
txt2ti3's whole purpose is to understand a foreign CGATS format and translate them into ArgyllCMS format. For every tool to have all that code in it to deal with extra multiple matching input files (early LOGO/Gretag/X-Rite tools kept device and measured values separate) and do the translation, as well as recognize and make sense of the variations in ArgyllCMS format would make the code totally unwieldy.
Title: Re: DCamProf - a new camera profiling tool
Post by: GWGill on May 13, 2015, 11:07:38 pm
The LUT maps from camera RAW RGB triplets to LabD50 triplets. A problem is that a very large space of the LUT will be meaningless values, a camera never registers 1,0,1 for example, you nearly always have some signal on all three channels.
Yep - that's another reason for the per channel Luts - you can restrict the box size. If you are in XYZ PCS you can also use the matrix to better fit your box around the gamut.
Quote
Another issue is that it always have to be 3D even if you have 2.5D data, which means that you have to have quite low resolution or else the ICC file will be huge, a 33x33x33 cube could have been a 190x190 2.5D square instead.
I think ICCV4 allows different resolutions per channel.
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on May 14, 2015, 03:06:35 am
Thanks for testing.

I have only tested C1 by installing the profile, ie it gets to c:\windows spool color directory like printer profiles, I thought it was like that you should do. It works for me then in C1 7. The profile should not miss any tags in terms of standard, but maybe C1 needs more tags if put in the directory you describe. I shall investigate. Meanwhile just install by right click => install.

I have only tested with C1 7, and Mac OS X default ICC tool (it's really nice), and iccexamin and Argyll's iccdump. Sounds strange that they wouldn't work in color.org tools, I shall investigate that too.

The DCamProf LUT profiles don't have the XYZ or TRC tags like most C1 native profiles have, whose function is unknown to me. If the XYZ and TRC are involved in pre-matrixing/pre-processing somehow it maybe won't work in their native profiles directory.
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on May 14, 2015, 03:58:13 am
Tested a profile with the color.org tools, it worked. I get one warning in ProfileDump validator though - unknown CMM, but it should not warn on that, the standard clearly states 0x0 should be used if no specific CMM is preferred, and that is also what many other profiles (including C1) use.

The profile was generated by the Linux build though, maybe there was some issue with the windows code, there are some specifics there. Investigating...

Edit: Tested your windows compiled version and indeed it makes invalid profiles, investigating why...

Edit2: the htonl/nthol macro code I copied was bad, it assumes the compiler defines BYTE_ORDER LITTLE_ENDIAN etc, and if it doesn't the code is written such that big endian becomes default ie all words in the icc gets in the wrong order, it doesn't look like that but C preprocessing has some traps... will fix and release a patch soon.

Edit3: updatated code and uploaded, hasn't been out for many hours so didn't care to change patch level so it's still at http://www.ludd.ltu.se/~torger/files/dcamprof-0.6.1.tar.bz2
I don't know if mingw defines __BYTE_ORDER__ like gcc do, but if it doesn't you should get an error message not silent accept big endian like before... if you do get an error with Mingw please let me know what macros it uses for byte order identification.
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on May 14, 2015, 09:13:49 am
Tested a profile with the color.org tools, it worked. I get one warning in ProfileDump validator though - unknown CMM, but it should not warn on that, the standard clearly states 0x0 should be used if no specific CMM is preferred, and that is also what many other profiles (including C1) use.

The profile was generated by the Linux build though, maybe there was some issue with the windows code, there are some specifics there. Investigating...

Edit: Tested your windows compiled version and indeed it makes invalid profiles, investigating why...

Edit2: the htonl/nthol macro code I copied was bad, it assumes the compiler defines BYTE_ORDER LITTLE_ENDIAN etc, and if it doesn't the code is written such that big endian becomes default ie all words in the icc gets in the wrong order, it doesn't look like that but C preprocessing has some traps... will fix and release a patch soon.

Edit3: updatated code and uploaded, hasn't been out for many hours so didn't care to change patch level so it's still at http://www.ludd.ltu.se/~torger/files/dcamprof-0.6.1.tar.bz2
I don't know if mingw defines __BYTE_ORDER__ like gcc do, but if it doesn't you should get an error message not silent accept big endian like before... if you do get an error with Mingw please let me know what macros it uses for byte order identification.

ok, I will redo my build then - thank you.
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on May 14, 2015, 10:16:53 am
0.6.1 recompiled for Windows (mingw) = https://app.box.com/s/mcdk72i5hgrvbgpnrpb8pp8xs1n55bfh
Title: Re: DCamProf - a new camera profiling tool
Post by: Iliah on May 14, 2015, 05:37:21 pm
lobby for RD v2.0, long overdue, to include some more complex processing when generating cgats in addition to simple gamma

A simple script or even a spreadsheet can take the linear CGATS or CSV and apply a curve of your liking. It can be done within a profiling engine as well. IMHO RawDigger proper is not the place where it should be done as the variety of needs is quite overwhelming.
Title: Re: DCamProf - a new camera profiling tool
Post by: GWGill on May 14, 2015, 08:43:40 pm
Edit2: the htonl/nthol macro code I copied was bad, it assumes the compiler defines BYTE_ORDER LITTLE_ENDIAN etc, and if it doesn't the code is written such that big endian becomes default ie all words in the icc gets in the wrong order, it doesn't look like that but C preprocessing has some traps... will fix and release a patch soon.
Another reason to be using lcms ?

[ I personally prefer writing endian independent code, rather than relying on compile time endian detection and byte swapping. ]
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on May 14, 2015, 09:40:36 pm
A simple script or even a spreadsheet can take the linear CGATS or CSV and apply a curve of your liking. It can be done within a profiling engine as well. IMHO RawDigger proper is not the place where it should be done as the variety of needs is quite overwhelming.
certainly... but it is a commercial product and so customers as expected are wishing all kind of things, overwhelming or not :) ...
Title: Re: DCamProf - a new camera profiling tool
Post by: Iliah on May 14, 2015, 10:23:34 pm
certainly... but it is a commercial product and so customers as expected are wishing all kind of things, overwhelming or not :) ...
I can't even start on what functions I want to be added to exposure meters. Nobody manufacturers listen ;)
That's not to say we are not willing to add features, but fulfilling some requests lead to more dissatisfied users, as they want the same but in yellow and with a grey strip ;)
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on May 15, 2015, 02:31:59 am
Another reason to be using lcms ?

[ I personally prefer writing endian independent code, rather than relying on compile time endian detection and byte swapping. ]


I've written 13000 lines of codes in about two months in addition to having a full-time job, and now it's documented and freshly released to the public. If everyone's going to make points for each bug that is found this will be a very tiresome thread.
Title: Re: DCamProf - a new camera profiling tool
Post by: Alexey.Danilchenko on May 15, 2015, 04:28:07 am
I've written 13000 lines of codes in about two months in addition to having a full-time job, and now it's documented and freshly released to the public. If everyone's going to make points for each bug that is found this will be a very tiresome thread.
As promised I will have a go at converting it to LCMS but in a few weeks time (I'll send you the code when it will work).
Title: Re: DCamProf - a new camera profiling tool
Post by: Tim Lookingbill on May 15, 2015, 06:55:08 am
I've written 13000 lines of codes in about two months in addition to having a full-time job, and now it's documented and freshly released to the public. If everyone's going to make points for each bug that is found this will be a very tiresome thread.

I don't write code, don't know a lick about programming, but it will be a tiresome thread for me if I don't get to see results from all this hard work in the form of an A/B comparison photo showing the code works.

I look forward to seeing this. In a sense the anticipation of what's been accomplished in this discussion is keeping me from seeing this thread as tiresome.
Title: Re: DCamProf - a new camera profiling tool
Post by: Alexey.Danilchenko on May 15, 2015, 07:17:42 am
I don't write code, don't know a lick about programming, but it will be a tiresome thread for me if I don't get to see results from all this hard work in the form of an A/B comparison photo showing the code works.
The results of DCamProf can hardly be judged in such a primitive way as A/B photo comparison. The code is also not producing any photos so not really surte what you expect as  "showing the code works"
Title: Re: DCamProf - a new camera profiling tool
Post by: Bart_van_der_Wolf on May 15, 2015, 07:40:07 am
I've written 13000 lines of codes in about two months in addition to having a full-time job, and now it's documented and freshly released to the public. If everyone's going to make points for each bug that is found this will be a very tiresome thread.

Thanks for the huge effort and research that goes with such a project.

There will always be things to change/improve/hindsight/etc. Don't take it as criticism, but as a pat on the back because it's apparently perceived as worthwhile enough to care about and build on, and a add a bit more functionality e.g. for Capture One (although not originally intended). But I also understand that there are limits to the resources for such a labor of love, thanks.

Cheers,
Bart
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on May 15, 2015, 07:51:55 am
As promised I will have a go at converting it to LCMS but in a few weeks time (I'll send you the code when it will work).

Thanks but you don't need to, your valuable time is probably better spent on some other project than rewriting code that now works (hopefully ;) ). It seems like some think that I've written custom code for a major part of the software which LCMS already does, but that's not the case. If I strip out the code LCMS could do the ICC I/O stuff is ~500 lines out of those ~13000. LCMS is already used for LUT color conversions, CIEDE2000 and other complex code. Even if reading/writing is converted to LCMS code I still need/want the internal simplified profio_icc format as it will complexify the code dealing with a generic do-it-all ICC in the internals so the difference will be very small. Just as little as if Argyll would switch to using LCMS for reading and writing profiles DcamProf won't automatically gain ICCv4 support or any other feature.

Another point is that although I'm very grateful for your will to contribute to my personal project, I rather keep the code close to my own chest for now it's like my own child ;), patches with bug fixes yes, rewriting the code, no.
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on May 15, 2015, 08:05:54 am
Thanks for the huge effort and research that goes with such a project.

There will always be things to change/improve/hindsight/etc. Don't take it as criticism, but as a pat on the back because it's apparently perceived as worthwhile enough to care about and build on, and a add a bit more functionality e.g. for Capture One (although not originally intended). But I also understand that there are limits to the resources for such a labor of love, thanks.

Capture One support was originally intended, just not this soon :-).
Title: Re: DCamProf - a new camera profiling tool
Post by: Alexey.Danilchenko on May 15, 2015, 08:06:30 am
Fair enough.

Even if reading/writing is converted to LCMS code I still need/want the internal simplified profio_icc format as it will complexify the code dealing with a generic do-it-all ICC in the internals so the difference will be very small. Just as little as if Argyll would switch to using LCMS for reading and writing profiles DcamProf won't automatically gain ICCv4

I don't think we are still on the same page here. LCMS2 usage is not about complexity dealing with all ICC stuff it is about not writing the code to parse ICC. Whether you use it to access more complex areas is up to you. Opening a profile, checking the version and getting a matrix out of it for example is done in 3 calls. Creating profile from scratch is also simple. I kept pointing to dcp2icc tool - here is a snippet from there that creates a complete LUT profile (the lut creation are done with callbacks to dcp2iccc routines so these are all LCMS2 calls):


int CreateICC(char *fname, Sampler *smp, char *name, char *model)
{
    cmsHPROFILE h;
    LPLUT clut;

    h=cmsOpenProfileFromFile(fname,"w");
    if (!h) {
        printf("Error creating '%s'\n", fname);
        return -1;
    }
    cmsSetDeviceClass(h, icSigInputClass);
    cmsSetColorSpace(h, icSigRgbData);
    cmsSetPCS(h, icSigXYZData);
    cmsSetRenderingIntent(h, INTENT_PERCEPTUAL);
    cmsAddTag(h, icSigProfileDescriptionTag, name);
    cmsAddTag(h, icSigCopyrightTag, "(c) Nobody");
    cmsAddTag(h, icSigDeviceModelDescTag, model);
    clut=cmsAllocLUT();
    cmsAlloc3DGrid(clut, 33, 3, 3);

    if (!cmsSample3DGrid(clut, dcpTest1, smp, 0)) {
        printf("Error creating CLUT\n");
    }

    if (!cmsSample3DGrid(clut, dcpTest2, smp, 0)) {
        printf("Error creating CLUT\n");
    }

    if (!cmsSample3DGrid(clut, dcpSampler, smp, 0)) {
        printf("Error creating CLUT\n");
    }

    if (!cmsAddTag(h,icSigAToB0Tag, clut)) {
        printf("Error adding CLUT\n");
    }

    cmsFreeLUT(clut);
    cmsCloseProfile(h);
}
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on May 15, 2015, 08:23:58 am
I don't write code, don't know a lick about programming, but it will be a tiresome thread for me if I don't get to see results from all this hard work in the form of an A/B comparison photo showing the code works.

I look forward to seeing this. In a sense the anticipation of what's been accomplished in this discussion is keeping me from seeing this thread as tiresome.

There many things you can do A/B tests for, but each test will be quite narrow and quite much work to perform. You don't see much A/B images for Argyll either (although I actually have one on my Argyll print workflow page).

I will myself be busy with maintaining the code and fixing improving for some while and will unfortunately not spend much time for these types of tests, except for those I do myself during development but those are not really presentable directly. At some point I'll probably write an article more in-depth about profiling and then there can be these type of A/B examples, but that's some time away now.

I'll maybe drop in some picture at some point if I come up with something interesting and presentable. So far I mostly do things "oh yes this seems to work" sanity check types of things, plus plot a lot of things.

Some ideas to test:
* A/B test between Ligthroom/C1 standard and DCamProf ICC/DCP on some typical scenes
* A/B test between files generated with different type of targets, CC24 vs IT8
* A/B test SSF-profile vs target profile, see if for example skin tone spectral data helps or not

it's difficult to show everything in a picture though as it cannot contain all colors, so plots are useful too.

There are still to me many undiscovered aspects of profiling, and the previously existing tools did not have the features I needed to allow me to research them, DCamProf has that and hopefully I'll actually do the tests now when I can, but writing the code turned out to be interesting too :)
Title: Re: DCamProf - a new camera profiling tool
Post by: Tim Lookingbill on May 15, 2015, 03:16:30 pm
The results of DCamProf can hardly be judged in such a primitive way as A/B photo comparison. The code is also not producing any photos so not really surte what you expect as  "showing the code works"

Works in regard to color management and how it profiles a camera's response recording color which requires we see this in a photo compared to other camera profiling packages that use different coding and color descriptors to define how a camera records color.

Does that make it more clear for you?
Title: Re: DCamProf - a new camera profiling tool
Post by: Tim Lookingbill on May 15, 2015, 03:19:18 pm
There many things you can do A/B tests for, but each test will be quite narrow and quite much work to perform. You don't see much A/B images for Argyll either (although I actually have one on my Argyll print workflow page).

I will myself be busy with maintaining the code and fixing improving for some while and will unfortunately not spend much time for these types of tests, except for those I do myself during development but those are not really presentable directly. At some point I'll probably write an article more in-depth about profiling and then there can be these type of A/B examples, but that's some time away now.

I'll maybe drop in some picture at some point if I come up with something interesting and presentable. So far I mostly do things "oh yes this seems to work" sanity check types of things, plus plot a lot of things.

Some ideas to test:
* A/B test between Ligthroom/C1 standard and DCamProf ICC/DCP on some typical scenes
* A/B test between files generated with different type of targets, CC24 vs IT8
* A/B test SSF-profile vs target profile, see if for example skin tone spectral data helps or not

it's difficult to show everything in a picture though as it cannot contain all colors, so plots are useful too.

There are still to me many undiscovered aspects of profiling, and the previously existing tools did not have the features I needed to allow me to research them, DCamProf has that and hopefully I'll actually do the tests now when I can, but writing the code turned out to be interesting too :)

I'm getting the sense there will be no photo comparisons. Just say it so I can move on.
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on May 15, 2015, 04:53:30 pm
I'm getting the sense there will be no photo comparisons. Just say it so I can move on.

You can do them yourself too you know ;). Erik showed some patchtool dumps a while back. I don't know what other users will do, I will myself prioritize the todo list on software features rather than making demonstrations though. I'd love to do both but I think the early adopters prefer I focus on filling in the gaps.

Next up is ICC test-profile and plotting capability to wrap up the ICC support. I'm also thinking of adjusting the whitepoint preservation handling, seems like there's often problems with targets that have a scale of neutrals, which give contradicting corrections (possibly due to precision issues in the target reference files) and this can lead to some local stretching badness of the LUT when whitepoint is preserved. May need to do some special case handling for that.
Title: Re: DCamProf - a new camera profiling tool
Post by: GWGill on May 16, 2015, 04:00:21 am
If everyone's going to make points for each bug that is found this will be a very tiresome thread.
I'm not sure what you mean by "make points for each bug" -
my comment was intended just as food for thought - something to contemplate as you consider your longer term development plans.
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on May 16, 2015, 07:05:06 am
I'm not sure what you mean by "make points for each bug" -
my comment was intended just as food for thought - something to contemplate as you consider your longer term development plans.

Fine, I probably overreacted, no problem. There's been a bit too much focus on a thing that I don't think matter. DCP and DNG ref code is custom too, but I better shut up about that... Oops :-)
Title: Re: DCamProf - a new camera profiling tool
Post by: Alexey.Danilchenko on May 16, 2015, 11:45:21 am
Fine, I probably overreacted, no problem. There's been a bit too much focus on a thing that I don't think matter. DCP and DNG ref code is custom too, but I better shut up about that... Oops :-)
DNG SDK is nice to have but it does not suite you for other reasons - it is C++ (and you are not using that) and hard to compile (too many dependencies that one does not need).
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on May 16, 2015, 12:20:16 pm
Just released v0.6.2, now test-profile supports ICC and you can make ICC plots, that wraps up the ICC support which is now at the same level as DCP. In the usual place: http://www.ludd.luth.se/~torger/dcamprof.html
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on May 16, 2015, 05:46:37 pm
0.6.2 build for Windows (mingw = dcamprof.exe + libgomp_64-1.dll) : https://app.box.com/s/t851von5samj5tq8hzyw5qtontd8z3zo
Title: Re: DCamProf - a new camera profiling tool
Post by: Bart_van_der_Wolf on May 16, 2015, 05:54:46 pm
Thanks to both Anders and 'AlterEgo' for the updated code and binaries.

Cheers,
Bart
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on May 17, 2015, 05:01:20 am
Noticed I broke some of the old plot code in 0.6.2 when ICC plots were added. Will fix that and make some whitepoint handling improvements to next patch release.
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on May 17, 2015, 01:31:12 pm
Just released v0.6.3, fixed the plot bug, added two more error vector plots, better whitepoint preservation handling of the LUT.

Now I don't have any particular urgent things on the todo list left, so hopefully I can keep it in just-fixing-bugs mode for a while.
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on May 17, 2015, 02:54:42 pm
0.6.3 build for Windows (mingw = dcamprof.exe + libgomp_64-1.dll) : https://app.box.com/s/gtli840m59r5yd1yi9bzjgaub1bo4gdf
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on May 18, 2015, 08:32:45 am
Slow day at work so I've done some testing of matrix optimization, found a bug in the overall printed stats which is fixed in 0.6.4 now out. I also adjusted the matrix optimization very slightly and updated the docs on weighting a bit.

In practice weighting does not seem to have that big impact on low patch targets like the CC24, weighting results can for those appear a bit random, eg if you reduce importance on say lightness it may sometimes actually improve lightness accuracy rather than the other way around. For those the "weighting" is sort of already done from the choice of patches, by excluding colors you get better precision on the colors you include.

Weighting behaves more predictably on larger patch count targets with higher saturation patches like a glossy IT-8 target.

One interesting result I found is that it's often a good idea to reduce importance of chroma (saturation), this will naturally push the matrix towards lower saturation as it gives more room to fine-tune hue.
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on May 18, 2015, 09:36:49 am
0.6.4 build for Windows (mingw = dcamprof.exe + libgomp_64-1.dll) : https://app.box.com/s/bervrhj671j70l7e2fuca9q9ug9sw9zp
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on May 19, 2015, 03:16:30 pm
There is some problem with the LUT causing some purple colors to be pulled towards deep blue. I'm investigating. Sounds like kind of the reverse of "blue turns purple" lab problem, but there should be no such stuff... but well, it's something weird going on. Matrix profiles seems unaffected.

I've also noted that DCamProf produces saturated profiles compared to typical bundled profiles. I'm not sure why. If saturation is correct is hard to verify by eye, but it seems like DCamProf could be the more correct, while bundled profiles are often desaturated to not become too saturated when a film-curve is applied. I haven't thought about this before.

Maybe I'll have to add a feature that makes it possible to adapt the profile to a certain film-curve. You can't get perfect accuracy with that as it will depend on where on the S-curve the color is how saturated it will be, but I guess one would adapt assuming the color is in the midtone range.

Has anyone any input on this, that is how profiles should relate to the film curve? Today DCamProf makes "corrrect" profiles for linear curve, which means saturation will increase to higher levels than real-world when an S-curve is applied.
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on May 20, 2015, 08:07:15 am
There is some problem with the LUT causing some purple colors to be pulled towards deep blue. I'm investigating. Sounds like kind of the reverse of "blue turns purple" lab problem, but there should be no such stuff... but well, it's something weird going on. Matrix profiles seems unaffected.

There was no LUT problem, what happened was that I got some extreme stretching in the LUT, and then bad things can happen if the white balance is set to a bit different from what the profile was designed for. When white balance is changed, the input colors to the LUT shift positions and if you have some extreme local stretches it can suddenly get into the position of a neighboring color and get stretched away.

I'm experimenting with high saturation custom targets when I got this problem. I'm still a bit surprised that the LUT had to stretch as much to hit the colors, I'm getting a quite different result when using SSF on the same target, so I still have some things to investigate.

Anyway, moving the white balance back and forth in the raw converter while looking how the color change is a good "sanity check" of a LUT profile. Preferably use a shot with many colors, a color checker of some sort. If one or more colors suddenly changes a lot faster than others you probably have too strong stretch in the LUT. Of course you can also plot the LUT to see these problems.
Title: Re: DCamProf - a new camera profiling tool
Post by: Tim Lookingbill on May 20, 2015, 03:55:19 pm
There is some problem with the LUT causing some purple colors to be pulled towards deep blue. I'm investigating. Sounds like kind of the reverse of "blue turns purple" lab problem, but there should be no such stuff... but well, it's something weird going on. Matrix profiles seems unaffected.

I've also noted that DCamProf produces saturated profiles compared to typical bundled profiles. I'm not sure why. If saturation is correct is hard to verify by eye, but it seems like DCamProf could be the more correct, while bundled profiles are often desaturated to not become too saturated when a film-curve is applied. I haven't thought about this before.

Maybe I'll have to add a feature that makes it possible to adapt the profile to a certain film-curve. You can't get perfect accuracy with that as it will depend on where on the S-curve the color is how saturated it will be, but I guess one would adapt assuming the color is in the midtone range.

Has anyone any input on this, that is how profiles should relate to the film curve? Today DCamProf makes "corrrect" profiles for linear curve, which means saturation will increase to higher levels than real-world when an S-curve is applied.

Since you're describing visually what DCamProf does to saturation and hue shifting in purples toward blue, have you considered constructs within the display profile as influences?

Haven't seen this mentioned here.

Also your point about uneven saturation levels across a wide range of colors as a sanity check also happens with my custom DNG profiles on some scenes. One slight tweak to WB and things shift unexpectedly saturation and luminance wise on certain colors which is why I've questioned ICC display profile influences.

I've often thought there needs to be more accurate descriptors of saturation appearance connected to the math under the hood because your description indicates a pattern of behavior with certain colors that suggest complexities that haven't been defined thoroughly mathematically. Maybe there needs to be a mathematical formula that takes into account human adaptation on saturation appearance separate from WB influences.
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on May 21, 2015, 02:17:02 am
Screens with profiles, or factory builtin profiles (to emulate sRGB for example), can indeed sometimes have ugly nonlinearities that can make images appear to have problems when there are none, I've seen that. However I move between several computers, some without screen profiles, and I also know the profile quite well of my own screen (shaper profile designed with Argyll) so there was no such problem here.

The problem was as described in the last post, due to that there was a poor fit the LUT had to stretch a lot, and then the results become very white-balance dependent. If the white balance is not exactly as the profile expects the LUT can drag the wrong colors which happened in this case.

The next problem is why there was a poor fit and I'm still investigating that. I think there was some measurement problem, either in my XYZ reference measurements or in the test target photo, or both. I'm starting to suspect that high saturation patches as I have on my test target may not be a good idea when you shoot a target as it seems like the impact of small measurement errors becomes large, while lower saturation targets are much more robust. This is still just a theory though which I need to verify.
Title: Re: DCamProf - a new camera profiling tool
Post by: jwlimages on May 21, 2015, 12:21:30 pm
If this is too far off topic, my apologies…

Seems like an opportunity for someone here: create a good DNG profiling application, put a Mac interface on it (please!), allow for the user to load custom measurements from his/her Color Checker or CC Passport target, make it robust enough to handle larger-size DNG's (Pentax 645z). I would pay good $$ for this, and I suspect many others would, too.

Seems like there would be no competition for such a product - X-rite's Passport software is not only showing its age, but apparently will not receive any further development (users over on Pentax user forum claim to have been told this directly by X-rite staff).

John

JWL Images
Emeryville, CA
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on May 21, 2015, 12:31:35 pm
If this is too far off topic, my apologies…

Seems like an opportunity for someone here: create a good DNG profiling application, put a Mac interface on it (please!), allow for the user to load custom measurements from his/her Color Checker or CC Passport target, make it robust enough to handle larger-size DNG's (Pentax 645z). I would pay good $$ for this, and I suspect many others would, too.

Seems like there would be no competition for such a product - X-rite's Passport software is not only showing its age, but apparently will not receive any further development (users over on Pentax user forum claim to have been told this directly by X-rite staff).

John

JWL Images
Emeryville, CA

and why kind of UI you are missing exactly - because everything else is already there and works (on Mac too) ?
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on May 21, 2015, 12:48:17 pm
If this is too far off topic, my apologies…

Seems like an opportunity for someone here: create a good DNG profiling application, put a Mac interface on it (please!), allow for the user to load custom measurements from his/her Color Checker or CC Passport target, make it robust enough to handle larger-size DNG's (Pentax 645z). I would pay good $$ for this, and I suspect many others would, too.

Seems like there would be no competition for such a product - X-rite's Passport software is not only showing its age, but apparently will not receive any further development (users over on Pentax user forum claim to have been told this directly by X-rite staff).

It's probably not a bad idea, in a distant future (1 year plus) I might make a commercial GUI version of DCamProf (keeping command line open source), with the added feature to make custom hand-tuning to create "looks" (for that I think a GUI is really needed). I personally don't like having the look in the profile but the majority of users do, so to make a commercial I think that is a must-have feature. Still it would not be a huge seller, there's a reason X-rite staff is not developing their product further, there's more money to make on other products. I think most are using Adobe DNG Editor and are satisfied with that.

I also think one problem with commercializing a product is that making camera profiles beyond the very basics is relatively hard. Making a camera profile with a look requires a good eye for color and taste, and quite some technical understanding of camera limitations too. Looking at X-rite's products they are very focused on ease-of-use, so one reason they're not developing their profiling product further may be that it would be a "too complicated" product, so maybe it's not only about money-making potential...
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on May 21, 2015, 12:52:16 pm
here's more money to make on other products.
I read just recently somewhere - poster alleged that he was told by a big B&M photo shop that they sell to the tune of 10Ks xrite passports annually (???), so indeed if that is the case
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on May 21, 2015, 12:54:13 pm
Making a camera profile with a look requires a good eye for color and taste
so the business is to sell canned looks actually - a lot of companies do (even in non profile form)
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on May 21, 2015, 12:56:07 pm
there's a reason X-rite staff is not developing their product further
they killed profilemaker ! 
Title: Re: DCamProf - a new camera profiling tool
Post by: jwlimages on May 21, 2015, 08:20:38 pm
Quote
It's probably not a bad idea, in a distant future (1 year plus) I might make a commercial GUI version of DCamProf (keeping command line open source), with the added feature to make custom hand-tuning to create "looks" (for that I think a GUI is really needed).

--  Yes, I follow your reasoning here, but I too am not fond of this/would have no use for "looks" like that - kind of like all the Lightroom "presets" people are trying to sell. Yes, it's a one-button procedure, but the results generally suck, IMHO.

Yes, X-rite killed ProfileMaker, their software generally leaves a lot to be desired - I wonder if this isn't just their longtime corporate culture reasserting itself. Seems like they were always thought of as an outfit that made good hardware/not-so-good software. Once Gretag-MacBeth got gobbled up, maybe it was only a matter of time before that company's software skills (developer/engineers?) were gone.

Still I would dearly love to see a decent DNG-profile generating app, that could accept custom measurements & deal with larger DNG files. Something to help me render colors predictably, hopefully relatively "accurately" - all that business of simulating a Kodachrome look, or whatever, can easily be handled in Photoshop.

Just a thought, anyway…

Thanks for your efforts.

John
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on May 21, 2015, 08:28:07 pm
Still I would dearly love to see a decent DNG-profile generating app, that could accept custom measurements & deal with larger DNG files.
why "would" - what prevents you from using dcamprof right now this moment ?
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on May 22, 2015, 02:20:14 am
If you want neutral/"accurate" profiles DCamProf should indeed do the job. I may need to make some contrast/curve/desaturation support though to work better with film curves to give you full satisfaction, but you can start testing and playing around today. DCamProf is today very "colorimetric", accurate results with linear gamma, which may lead to a bit too saturated look when a typical film curve is applied. I shall look into that later on.

Currently I'm experimenting with inkjet printed targets; I've made some quick comparisons with real natural spectra and real skintone spectra and a homemade printed target seem to do as good or slightly better than a CC24, a Colorchecker Digital SG is slightly better on skintones (it has many patches for that) but not on nature, the skintone matching might be compensated if I actally increase density in the skintone range. The spectral shapes looks a bit better from the X-rite colorcheckers if you plot them, but in practice it doesn't seem to make that big a difference.

Grayscale patches are no good from the inkjet, black patch is good and white (=unprinted) is good, but grays don't have completely flat spectra so no good for curve calibration, my current assumption is that lens does not alter the linearity of the camera that much that curve calibration would be necessary.

With the printer on semigloss baryta paper (make sure to use OBA-free) you can make supersaturated targets, but these seem difficult to match, high saturation colors are difficult. A matte target is much more user-friendly. Matte targets can't make colors in the dark violet range though.
Title: Re: DCamProf - a new camera profiling tool
Post by: jwlimages on May 22, 2015, 01:24:05 pm
Quote
what prevents you from using dcamprof right now this moment ?

--  sad to admit it, one phrase: "command line" (same thing that has kept me away from Argyll). With all the apps I need to stay current with, my brain is too full already. Just too hard to keep adding more & more things to learn & master, I guess.

John
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on May 22, 2015, 02:48:52 pm
--  sad to admit it, one phrase: "command line" (same thing that has kept me away from Argyll). With all the apps I need to stay current with, my brain is too full already. Just too hard to keep adding more & more things to learn & master, I guess.

John

you can prepare .ti3 files with rawdigger profile edition - it is basically GUI to handle the target shot for your purposes... then it is something that you can just write once in a command file - 3 lines along the following lines, for example only, you can hardcode there all directories too  :

txt2ti3 -v -i <cgats file from rawdigger> <target .cie file, with spectral data for example> <output goes to .ti3 filename>
dcamprof make-profile -i <illuminant> <use the file from the line above .ti3 filename> <output goes to profile.json>
dcamprof make-dcp -c "your camera model" -t <curve of choice> <from the line above - profile.json> <output.dcp>

not much in a very simple case !

PS: I also modified (not in published .exe build of course !) the code for myself (.exe that I am using privately) to be able to put the baseline exposure offset for my camera and camera profile name through command line to save the time.
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on May 23, 2015, 07:05:42 am
I have now released v0.7.0, includes test chart generator and a flatfield correction command. Patch names are now kept so it's easier to track them in the reports and plots.

I've changed the default such that targets are auto-rebalanced so the whitest patch becomes perfect reference whitepont, this is because most users expect the profile's optimal white balance be exactly the same as when you color-pick the whitest patch on the target (which in reality is a tiny bit off-white). You can disable that behavior with the -B flag.

I've also updated the documentation on white balance stuff, test target shooting and design. I've noted that semi-gloss targets are tricky to shoot, but if you do it right good results is had. Doing it right means shooting indoors with controlled (halogen) light, it's virtually impossible to shoot in daylight without some glare destroying the linearity of the photo. I haven't checked but I suspect that matte target do have some glare issues too, so for the best results shooting indoors seems to be the way to go.

In the usual place: http://www.ludd.ltu.se/~torger/dcamprof.html
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on May 23, 2015, 11:06:59 am
I have a question about .sp (illumination) files - it seems that you 10nm steps are not enough ? I am getting an error message when I tested one...  and why 5nm steps in your "spectrum.json" example ? argyll spotread with -H option outputs 3.x nm steps, something like = 380.00000 383.333333 386.666667 390.000000
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on May 23, 2015, 11:21:55 am
0.7.0 build for Windows (mingw = dcamprof.exe + libgomp_64-1.dll) : https://app.box.com/s/0dk90n0h4k171f0jr6ftxsmuupzpzl4b
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on May 23, 2015, 02:22:21 pm
I have a question about .sp (illumination) files - it seems that you 10nm steps are not enough ? I am getting an error message when I tested one...  and why 5nm steps in your "spectrum.json" example ? argyll spotread with -H option outputs 3.x nm steps, something like = 380.00000 383.333333 386.666667 390.000000

DCamProf currently always resamples to 5nm, this is due to laziness and that I do a lot of merging of targets when working with SSFs, and you can only have one sample interval in a single .ti3. I'll probably fix that in an update so resampling is only made when necessary.

I use .sp files now and then and it works for me, but so far I've always used with smaller bands than 10nm, maybe there's a bug, I'll check.
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on May 24, 2015, 05:15:18 am
A few things I've learnt about camera profiling so far:

* Modern cameras make quite good matching matrix-only in D50, I can certainly see that some will satisfy with matrix-only profiles
* In Tungsten light cameras sweat more, matching is hard.
* Glossy targets are very difficult to shoot due to patch glare, dark room with directed halogen spots seems to be the way to go.
* You can't really do magic to improve matrix matching, if it's a bad match it will be bad even if you remove 50% of the highest delta E patches, in other words you don't need that many patches to get the matrix behavior.
* Targets with many closely spaced patches, like my generated one, is at risk of making conflicting adjustments. Plotting, LUT weighting/relaxation is your friend.
* If you have SSF curves, like I have for my 5Dmk2, you will see that profiles generated from targets and those from SSF look highly similar -- a good sanity check to see that it works.
* One should start with a D50/D65 profile as the camera is better at matching there, if you get a poor match it's most likely due to a bad measurement. Tungsten profiles require more tradeoffs and will likely behave badly unless you apply some LUT relaxation etc.
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on May 26, 2015, 10:35:07 am
A few things I've learnt about camera profiling so far:
a question :

for example I am building a simple matrix profile for ACR/LR -single illuminant, CM+FM+curve, nothing else... I have a target, measured (spectral data) + a light (H3200K) & filter (80A), measured (spectral data) = .sp

now my exercise ends with (a simple case) :

dcamprof make-profile -i <.sp file> <.ti3 file> <profile.json>

dcamprof make-dcp -t linear -L -c "SONY ILCE-7" <profile.json> <profile.dcp>

does  -i <.sp file> in this case bring any benefit for ForwardMatrix really in addition to having a "better" ColorMatrix which provides more logically sane temperature/tint values in ACR/LR UI (slider values) - but ColorMatrix does not do anything with color transform so to say (only ForwardMatrix does, after WB of course), it certainly helps somewhat to a user with a "proper/right/you name it" starting point for WB corrections in UI ? <.ti3 file> has raw RGB data from a target shot and spectral data from target measurments... it seems to me that ForwardMatrix is the same no matter what illuminant I use, right ?... but shall it be so ?


Title: Re: DCamProf - a new camera profiling tool
Post by: torger on May 26, 2015, 11:11:11 am
a question :

for example I am building a simple matrix profile for ACR/LR -single illuminant, CM+FM+curve, nothing else... I have a target, measured (spectral data) + a light (H3200K) & filter (80A), measured (spectral data) = .sp

now my exercise ends with (a simple case) :

dcamprof make-profile -i <.sp file> <.ti3 file> <profile.json>

dcamprof make-dcp -t linear -L -c "SONY ILCE-7" <profile.json> <profile.dcp>

does  -i <.sp file> in this case bring any benefit for ForwardMatrix really in addition to having a "better" ColorMatrix which provides more logically sane temperature/tint values in ACR/LR UI (slider values) - but ColorMatrix does not do anything with color transform so to say (only ForwardMatrix does, after WB of course), it certainly helps somewhat to a user with a "proper/right/you name it" starting point for WB corrections in UI ? <.ti3 file> has raw RGB data from a target shot and spectral data from target measurments... it seems to me that ForwardMatrix is the same no matter what illuminant I use, right ?... but shall it be so ?

Calibration illuminant has no effect on the ForwardMatrix. It will only affect the ColorMatrix and thus it's ability to predict the correlated color temperature, which is a task specific to the DNG color pipeline (ICC will only use the forward matrix and the calibration illuminant is 100% informational, it has no effect whatsoever).

If you have a single illuminant profile the DNG pipeline does not need to know which temperature the CM was designed for, as it has only one to choose from. When making dual illuminant you need to specify EXIF lightsource so the CM is tagged with a designed temperature. You can't choose any temperature, as the temp is just a table lookup on the EXIF lightsource. Run DCamProf without args to see a list of them.

Currently it's a bit messy, your workflow would be to make the profile with the spectrum (which causes the XYZ reference values to be regenerated from spectra), but the resulting DCP will then have "Other" as calibration illuminant which will work for a single illuminant profile but not for dual, so then you need to use dcp2json/json2dcp to replace that with whatever EXIF illuminant with the closest temperature. It doesn't matter if the light is called "fluorescent" or whatever, DNG only translates it to a CCT value, no spectral shape is taken into consideration.

If the measurement data is the same, the forward matrix should not change if you specify a different calibration illuminant, yes that is correct. The reason for this is that the forward matrix specifies conversion from white-balanced camera RGB to XYZ D50, that is always D50 regardless of calibration illuminant, that is the same way as ICC profiles do. The reason one always goes to D50 is that it's more well-defined to convert further, eg to RGB for your screen and printer.

That is, colormatrix is only about the profile's ability to estimate the CCT the chosen white balance corresponds to. How much of a value that is in the Lightroom user interface I guess is a personal thing. I don't use Lightroom much, but I guess you typically start off with "As shot" white balance, and the profile won't affect the WB multipliers provided by the camera, if the color matrix is off an incorrect temperature translation will show though. If you would apply a processing preset (or whatever it's called in lightroom) which has a temperature and a tint in it the profile will affect the result as Lightroom will calculated the multipliers by using the colormatrix.

When you make a dual-illuminant profile the derived CCT will be used to mix your two forward matrices, so if you have StdA 2850K and D65 6500K, and the color matrices says the white balance is 4675K (exactly inbetween) the average of both forward matrices will be used. So in that case the color matrix has some effect on the actual color conversion, but only then.

Well there's also those ancient DCP profiles which did not have any forward matrix, in that case the color matrix has a direct effect on color rendition of course, you can make such profiles with DCamProf but I don't recommend it. Forward matrices was introduced by Adobe, maybe because they realized that the ICC way was actually better... binding to D50 gives more robust color conversions.

EDIT: corrected some statements on illuminant temperatures to not confuse future readers.
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on May 26, 2015, 11:22:36 am
the resulting DCP will then have "Other" has calibration illuminant which I don't think Lightroom will like (as it can't look up a temp from that), so then you need to use dcp2json/json2dcp to replace that with whatever EXIF illuminant with the closest temperature.

in fact ACR does just fine with "255" value in illumination tag... on one other forum I posted some simple experiment

Adobe's CMs (2 from from Adobe Standard dual illuminant) vs DCamProf CM/"255", single illuminant, generated with .sp for H3200K+80A filter :

Adobe + styrofoam + no 80A = 3050К/+2
Adobe + whibal + no 80A = 3050К/+2

DCamProf + styrofoam + no 80A = 3100К/-2
DCamProf + whibal + no 80A = 3100К/-2

AND

Adobe + styrofoam + 80A = 4950К/+20
Adobe + whibal + 80A = 4900K/+19

DCamProf + styrofoam + 80A = 5200/+21
DCamProf + whibal + 80A = 5150K/+19

quite close, not completely insane at least

Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on May 26, 2015, 11:31:20 am
If the measurement data is the same, the forward matrix should not change if you specify a different calibration illuminant, yes that is correct. The reason for this is that the forward matrix specifies conversion from white-balanced camera RGB to XYZ D50, that is always D50 regardless of calibration illuminant, that is the same way as ICC profiles do.

yes, _but_ ... I have raw rgb data from the actual target shot (from raw) and the actual target measurements, sprectral, from i1pro2 (which brings its own illumination) combined in .ti3 file ... now that I have the actual illuminant in .sp then I 'd assume that the data in .ti3 file shall be adjusted somewhat to account for that, no ? raw RGB data stays the same, because it is what it is - but now that we have the spectral data of the actual illumination during the shot that shall change how target measurements now are... unless I am missing something very simple... I mean why only CM change shall be enough to account for that ?
Title: off topic
Post by: AlterEgo on May 26, 2015, 11:42:50 am
How much of a value that is in the Lightroom user interface I guess is a personal thing. I don't use Lightroom much, but I guess you typically start off with "As shot" white balance, and the profile won't affect the WB multipliers provided by the camera, if the color matrix is off an incorrect temperature translation will show though.
offtopic, but - I use ACR, my "as shot" WB is always UniWB, so I start with "auto" WB or I subjectively type something between 4500-6500 with zero tint based on whether I want it colder/warmer in general...
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on May 26, 2015, 12:59:06 pm
in fact ACR does just fine with "255" value in illumination tag... on one other forum I posted some simple experiment

Adobe's CMs (2 from from Adobe Standard dual illuminant) vs DCamProf CM/"255", single illuminant, generated with .sp for H3200K+80A filter :

Adobe + styrofoam + no 80A = 3050К/+2
Adobe + whibal + no 80A = 3050К/+2

DCamProf + styrofoam + no 80A = 3100К/-2
DCamProf + whibal + no 80A = 3100К/-2

AND

Adobe + styrofoam + 80A = 4950К/+20
Adobe + whibal + 80A = 4900K/+19

DCamProf + styrofoam + 80A = 5200/+21
DCamProf + whibal + 80A = 5150K/+19

quite close, not completely insane at least

Thanks for this, I had forgot, I needed to look into dngref.c to remember.

lsOther is fine as long as it's not dual-illuminant.

ACR will simply feed the white balance to the (inverted) color matrix so it gets XYZ and converts to a xy chromaticity coordinate which then is through a standard algorithm converted to a blackbody temp and tint. It doesn't need to know which temperature the color matrix was designed at, but it will make more accurate temperatures if the tested white balance is close to the designed.

When dual illuminant ACR needs to know which temperature each matrix was designed for, so it knows how much to mix in from each.
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on May 26, 2015, 01:18:43 pm
yes, _but_ ... I have raw rgb data from the actual target shot (from raw) and the actual target measurements, sprectral, from i1pro2 (which brings its own illumination) combined in .ti3 file ... now that I have the actual illuminant in .sp then I 'd assume that the data in .ti3 file shall be adjusted somewhat to account for that, no ? raw RGB data stays the same, because it is what it is - but now that we have the spectral data of the actual illumination during the shot that shall change how target measurements now are... unless I am missing something very simple... I mean why only CM change shall be enough to account for that ?

The CM will need XYZ reference values for your calibration illuminant, so those will be recalculated from the reflectance spectra and your provided illuminant spectrum.

The FM will need XYZ reference values for D50 so those will not change, they're always calculated for D50, so the calibration illuminant won't matter. This may seems strange like it would be always the same FM regardless light, but don't forget that the camera RGB values are affected by the light you use during test target shooting, and that affects the FM. The task for the FM is convert RGB values that have been shot under the calibration illuminant and then white balanced. But DCamProf does not need to know which light you used for that, the RGB values are already there. And the target space is always XYZ D50 for the FM.

Only if you provide camera SSF curves so the RGB values can be recalculated from scratch the provided calibration illuminant will change the FM result.

Mathematically speaking it can be seen as a problem that FM always makes XYZD50 values, if you shoot under Tungsten wouldn't we want the tristimulus values for Tungsten? Well, the problem is that to convert to RGB or any other space we need to go through D50 (profile connection space) so there would still be a conversion to D50, using some sort of CAT. Then it's better if we make the conversion when designing the profile with full knowledge of spectra.

So a profile will correct the colors to make them look as if it was shot under D50, regardless of actual light used during shooting. For extreme light temperatures the eye will not fully adapt and colors will not appear the same as in D50, so the profile won't do the "correct" job for those temperatures. Is there a way to convert say 2000K color appearance but into D50 XYZ coordinates? I don't really know, maybe CIECAM02 can do that. It can be worth investigating. I suspect that it won't be feasible though, on cannot put on a slight red tint for example as the DNG pipeline requires the FM to map to neutral white. So for now it's up to the photographer's eye and raw converter sliders to simulate any changes in look due to extreme color temperatures.
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on May 26, 2015, 02:05:28 pm
OK, after reading the notes here and there I am coming back to peace of mind then... at least I have lesser work with WB  :), so that my exercise with spectrum measurments were not totally wasted
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on May 26, 2015, 02:30:32 pm
OK, after reading the notes here and there I am coming back to peace of mind then... at least I have lesser work with WB  :), so that my exercise with spectrum measurments were not totally wasted

Yes providing the spectrum gives the best XYZ reference values when the CM is derived, so it will be as good as it can be at estimating wb temperatures. Of course it will be best at estimating temperatures close to the actual temperature used when profiling, but any reasonable modern camera should have similar matrices over at least some temperature range so it should work for quite some range. I haven't made any specific investigation on how well it works though.

If one shoot under say tungsten but provide XYZ reference values for D50 the CM will certainly be bad at wb temperature guessing, so if one doesn't have a spectrometer so one cannot provide the spectra it's preferable to be reasonable good at guessing the temperature and provide a EXIF illuminant for that when making the profile. Using Adobe's own dual illuminant profiles and make a test shot can be used as an alternative to get some idea if unsure.
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on May 26, 2015, 02:44:17 pm
and some general question, I did not even look into that area, so if it sounds stupid never mind - but given the set of real world patches (through the supplied spectral measurements) data, can software produce a target that shall be a close to "ideal" (what is the "ideal" though ? the next question) one with the given (as a parameter) number of patches ? for example I give a task to software - find me the best target with 24 patches only from the given data for patches (for example I have 200 different patches @ my disposal) that I can use and software then gives me which 24 out of those 200 I shall use to achieve the best result (may be defined as some dE over a big virtual set of patches in your database like Munsell set, etc) ? some kind of automated finding of the best target withing given limitations (# of patches in the target, set of patches that I can only use, illuminant may be and virtual set of patches that I want to aim at in terms of good conversion with that target that software suggests)
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on May 26, 2015, 03:30:32 pm
In order to solve that problem the profiler needs to know the camera's SSF (as the solution will depend on that), but if you already know the camera's SSF you can profile directly against the real spectral data.

On a related subject, I have thought a little about estimating SSFs from a large set of printed patches, but I don't think meaningful precision can be had. There has been some academic work on SSF estimation, but as far as I know no method that reach the level of precision we would be interested of has been done. There is the method of using emissive targets with narrow-band filters, Image Engineering has such systems commercially, but I don't think those are feasible for enthusiast budgets.
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on May 26, 2015, 03:47:32 pm
In order to solve that problem the profiler needs to know the camera's SSF (as the solution will depend on that), but if you already know the camera's SSF you can profile directly against the real spectral data.

that's when I do not... but why it is not possible w/o camera's SSF ? we can imagine some average, idealized SSF at least ?

On a related subject, I have thought a little about estimating SSFs from a large set of printed patches, but I don't think meaningful precision can be had. There has been some academic work on SSF estimation, but as far as I know no method that reach the level of precision we would be interested of has been done.

I modified that matlab script for my camera and IT8 target, just as an exercise

(http://s7.postimg.org/3tyoasm6j/A7_SSF.jpg)

I will be redoing it with some additional measurements though, to see if it becomes better or not

Title: Re: DCamProf - a new camera profiling tool
Post by: GWGill on May 26, 2015, 09:50:56 pm
So a profile will correct the colors to make them look as if it was shot under D50, regardless of actual light used during shooting.
Re-lighting and chromatic adaptation are different things though. I would have thought that by default a photographer would want to capture the appearance of things as they see them, in which case a chromatic adaptation is exactly the right thing to do to adapt to a different white point while maintaining the appearance.
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on May 27, 2015, 01:32:03 am
Re-lighting and chromatic adaptation are different things though. I would have thought that by default a photographer would want to capture the appearance of things as they see them, in which case a chromatic adaptation is exactly the right thing to do to adapt to a different white point while maintaining the appearance.

Hmm... maybe I should change how DCamProf does it then. Assuming we have patch spectra and we make a StdA profile it currently does like this: from spectra calculate StdA XYZ values, which is used for CM (as said this is only about tempereture identification, not color correction), and from spectra calculate D50 XYZ values which the actual correction is made with, that is a re-lighting rather than CAT.

I have assumed that the CAT is rather approximative and tries to do the same thing as re-lighting does (as I've thought that the model assumes total adaptation, ie no appearance change), but makes a worse job at it so it is better to relight when possible (if DCamProf doesn't have the spectra it applies CAT02). It's a simple change to make it always do CAT instead, but I need to figure out if it's the right thing or not.

That is the alternative would be that the D50 reference values are always generated by CAT02, first you do the XYZ calibration illuminant ref values by spectra, and then CAT02 on that to get the D50 profile connection space ref values that the color correction works at.

When surfing around I've found papers like this: http://hrcak.srce.hr/file/95370 where the "performance of CATs" is tested, and what they do is that they calculate reference data by "relighting", that is new XYZ values from spectra, and then see how well a CAT matches that, and better match means better performance there. I think that indicated that the current way DCamProf does it is the right way, that is if you can relight using spectra do that rather than using CAT. However what you say indicates that it would be better to do the alternative described above, I'm not really sure what.

My understanding of CAT is that the simpler models try to achieve the same result as a relighting but does not succeed as well as doing an actual relighting, and the more advanced like CAT02 will do the same thing, *unless* you provide some viewing condition parameters, eg dim/dark environment which seems to be adaptations for television and projector viewing (that is how colors appears on screen depending on surrounding environment), I'm not sure if that can be transformed into anything meaningful for camera calibration.

So far I have assumed that relighting is the current best choice for camera profiles.

What do you think?
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on May 27, 2015, 03:29:39 am
that's when I do not... but why it is not possible w/o camera's SSF ? we can imagine some average, idealized SSF at least ?

Yes, I guess we could. I'm not sure I understand what the purpose is though, could you clarify through a use case? Is it that you have shot a target with like 200 printed patches, and want to make a profile that matches some other spectral database as good as possible (say landscape spectra), and then pick out a subset of those 200 printed patches that would contribute the best to match that landscape spectra, and remove those that won't contribute?

If so, one method could be to just try to find as similar spectrum as possible, I have actually written a prototype of such a function, not for this task though but to compare different targets when I made my profiling target evaluation.

I'm not sure if that would improve the profile though, but one way could be to just expose that spectral matching function is DCamProf as a command so one can filter own .ti3 files and experiment and see what happens.

What can improve a matrix profile is removing colors that are too saturated for the targeted spectra, so the matrix optimizer gets a better chance to match the lower saturation colors.
Title: Re: DCamProf - a new camera profiling tool
Post by: GWGill on May 27, 2015, 04:18:35 am
Hmm... maybe I should change how DCamProf does it then. ....  (as said this is only about tempereture identification, not color correction) ...
OK, perhaps I'm taking you out of context then. Accurately re-lighting a photographed scene would be no small feat - you'd need a spectral camera for a start, and some way of distinguishing between surface colors and emissive colors - such as photographing it under two different illuminations !
Quote
When surfing around I've found papers like this: http://hrcak.srce.hr/file/95370 where the "performance of CATs" is tested, and what they do is that they calculate reference data by "relighting", that is new XYZ values from spectra, and then see how well a CAT matches that, and better match means better performance there.
Well if you want to do re-lighting and all you have is colorimetric data, then doing a chromatic adaptation is an approximation of the effect you really want. And that seems to be all the above paper was exploring. If it was testing the CATs as appearance transforms, then they would have to have done psychovisual testing.
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on May 27, 2015, 04:53:58 am
OK, perhaps I'm taking you out of context then. Accurately re-lighting a photographed scene would be no small feat - you'd need a spectral camera for a start, and some way of distinguishing between surface colors and emissive colors - such as photographing it under two different illuminations !Well if you want to do re-lighting and all you have is colorimetric data, then doing a chromatic adaptation is an approximation of the effect you really want. And that seems to be all the above paper was exploring. If it was testing the CATs as appearance transforms, then they would have to have done psychovisual testing.

I'm just trying to clarify what a camera profiler should do. We need to get into D50 space as that is the PCS (for both ICC and DCP). What we shoot is test targets, and we have the reflectance spectra, and we have the illuminant spectra. So we can do spectral calculations to get the XYZ values for any illuminant we want. The question that has come up now is if we really should.

I do know that for extreme color temperatures, like dusk in a nordic winter landscape, the white snow will not look white but bluish pink, which is an obvious case of where just white balancing won't work, but AFAIK there's also a small difference between StdA and D65, that is even if fully adapted and only an StdA light source we will see that the light is a bit yellow compared to the same situation under D65. Again, AFAIK color science is designed to work in a range of "reasonable" color temperatures and produce equal colors over those, so a camera profile is expected to be designed such that if you shoot an artwork under Tungsten with a Tungsten profile it will give you the exact same result as if you shoot an artwork under D50 with a D50 profile. This is what DCamProf strives for currently, and therefore it recalculates the XYZ reference values from spectra with the targeted illuminant whenever it can, instead of using CAT.

If however there is a way to model the slight (or large) color appearance changes caused by an illuminant much different from D50, and then convert to XYZ D50 I should at least provide that as an option to DCamProf. I have assumed that CAT is not the answer though, and that there simply is no model for that, just like there seems to be no model for converting between different observers.
Title: Re: DCamProf - a new camera profiling tool
Post by: GWGill on May 27, 2015, 07:46:12 am
I'm just trying to clarify what a camera profiler should do. We need to get into D50 space as that is the PCS (for both ICC and DCP). What we shoot is test targets, and we have the reflectance spectra, and we have the illuminant spectra. So we can do spectral calculations to get the XYZ values for any illuminant we want. The question that has come up now is if we really should.
It seems to me that there are two clear parts to this. The first is converting the camera RGB's to XYZ's - i,e. to make the camera image as colorimetric as possible. Having done that, you would then use a CAT to convert your chosen white point to D50 for interchange, since this preserves the appearance.

What you do in the first part is a different matter. If you are striving for the best possible colorimetric profile, then one approach is to key the conversion to various hints as to the original spectral composition of the colors. Obvious hints are the color itself, which you can translate to an assumed spectra of statistically likely surface colors, but the spectral content of the illuminant will affect this, as well as the white point of the illuminant shifting your whole color-to-assumed spectral mapping.

So one practical approach is to create several discrete profiles for different illuminant conditions - which is where you certainly would use spectral re-lighting of your test target - to create these discrete profiles.
Quote
Again, AFAIK color science is designed to work in a range of "reasonable" color temperatures and produce equal colors over those, so a camera profile is expected to be designed such that if you shoot an artwork under Tungsten with a Tungsten profile it will give you the exact same result as if you shoot an artwork under D50 with a D50 profile.
It's certainly reasonable that under some circumstances a photographer may want to shoot under incandescent and want the resulting image to look like they shot it under D50. But a different photographer or the same one under different circumstances may want the D50 white point image to present the same appearance as the original under incandescent. The  difference is subtle, but its certainly there.
[ The difference may be extremely subtle indeed when forced through the eye of the RGB camera spectral sensitivities. ]
Quote
I have assumed that CAT is not the answer though, and that there simply is no model for that, just like there seems to be no model for converting between different observers.
No CAT is perfect, and if you arrive at a consistent set of errors in the current CATs, that can be formulated into a different more accurate and/or flexible CAT, then you certainly have material for a technical paper.
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on May 27, 2015, 09:02:14 am
Thanks Graeme for the feedback, it's very valuable.

Let's see if I have understood you correctly;

If we would like the Tungsten profile to keep the subtle color appearance difference from a D50 light condition the best current way is to make XYZ reference values for tungsten using spectral data, and then convert to D50 XYZ using CAT02 (with average viewing condition as default) instead of ignoring the calibration illuminant and always generating our D50 XYZ values from spectra using D50 illuminant directly. Is this correct?

Prior to this my humble understanding was that a CAT, including CAT02 (with average viewing condition in/out), did not intend to model those subtle appearance differences but instead just tries to make a result as equal as possible to "relighting" (ie recalculate our D50 values from spectra in our case), and at the same time make some mathematical tradeoffs to make the transform reversible. That is a CAT result is just an approximation of the relighting result and any differences from actual relighting is not due to it tries to model slight appearance differences, but just simply limits in how precise the CAT can be. As far as I see you say that this understanding is incorrect, right?
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on May 27, 2015, 09:26:09 am
Yes, I guess we could. I'm not sure I understand what the purpose is though, could you clarify through a use case?

ok, here is what I was thinking (may be not sanely enough though)

1) no SSF
2) desire to create a profile from some "target" (see below) that will give me the best color transform for some set of spectral reflectance of some things (like the databases of skin color, etc)
3) spectral data for patches that I can use to actually create a target either physically (cut, glue) or combining raw RGB & patch spectral data from shots & measurements of my existing targets (by removing unwanted data from those)
4) I want to limit myself to some reasonable amount of patches in the "target"

so can software be fed with

1) database of spectral reflectance of things - but I do not have patches for those
2) spectral data for patches that I actually have and so can shoot
3) size of the target in patches, for example 24 patches
4) illuminant spectral data

and output what patches exactly I shall select within the said limit (for example 24) to get the best profile for the said database ?

and then I either make the target to use physically or cut&paste the data from raw RGB & patch spectral data from shots of existing targets or combine both ways

Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on May 27, 2015, 12:48:49 pm
a question

I tried to make icc profiles from SSF and used 2 different illuminant spectrums all the way for 2 profiles - one is just regular D50 (-i D50) and one is my measurements of the light I have when I am shooting targets (-i <.sp>) - which is halogen 3200K (nominal) lamp + hoya hmc 80a filter on lens

the difference though is very small (put a lot of RGB C1 color readouts - most are either identical or differ by 1 out of 255 in one or max two channels) between what C1 delivers with both profiles between them (between 2 variants one rendered with D50 profile and one with my illuminant profile - the picture itself is outdoors, ~40.44N latitude, ~4 PM EST, ~April) , is this right ?

this is my light - BLACK one :

(http://s12.postimg.org/6psxox2kt/image.jpg)

Type = File 'CM=SpotRead(A)=Direct+HoyaHMC80A=Average=2015-05-26.sp' spect 0 {Black}
XYZ = 0.945861 1.000000 0.808182, x,y = 0.343445 0.363102
D50 L*a*b* = 100.000000 -3.190716 1.360748
CMYV density = 1.898217 1.864757 1.943857 1.863647
CCT = 5096.603887, VCT = 4905.877034
CRI = 92.4
CIEDE2000 Delta E = 2.871270


Type = File 'CM=SpotRead(P)=StyroFoam+HoyaHMC80A=Average=2015-05-26.sp' spect 0 {Red}
XYZ = 0.949646 1.000000 0.805636, x,y = 0.344664 0.362939
D50 L*a*b* = 100.000000 -2.529025 1.569581
CMYV density = 1.897475 1.866200 1.946574 1.862948
CCT = 5052.127273, VCT = 4879.029968
CRI = 92.9
CIEDE2000 Delta E = 2.660874
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on May 27, 2015, 02:15:21 pm
The CCT for your halogen+filter is about the same for D50, and while the 80A filter adds some bumps and dips it's still a good illuminant so I think it's reasonable to expect that the results should be highly similar.
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on May 27, 2015, 02:44:13 pm
The CCT for your halogen+filter is about the same for D50, and while the 80A filter adds some bumps and dips it's still a good illuminant so I think it's reasonable to expect that the results should be highly similar.
somehow I was expecting that sufficiently different shape (from D50) shall manifest itself more visibly... even CCT is the same.
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on May 27, 2015, 10:08:06 pm
just an image, the difference between a pair of Adobe CMs from Adobe Standard and home-brewed dcp from IT8 printed target (which was actually done by generating SSF from a single raw, then SSF to dual illuminant DCP), the raw from I-R (Sony A7) :

(http://s13.postimg.org/52vex8jg7/delta_mid.jpg)

guessed SSF in question also attached :

Title: Re: DCamProf - a new camera profiling tool
Post by: torger on May 28, 2015, 01:55:24 am
somehow I was expecting that sufficiently different shape (from D50) shall manifest itself more visibly... even CCT is the same.

I depends on how it lines up with the reflectance spectra and then the SSF, if the bumps here and there in the spectrum doesn't change the integrated value much, then colors won't change much.

I'm also trying to simulate D50 and D65 with halogen plus filters, not really happy with the shape of the spectrum (the 80B filter is way more bumpy than I expected) but your results are encouraging. I'm most worried about that the 80B filter lets through high reds which I guess can cause the accuracy of super-saturated reds on a glossy target go bad.
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on May 28, 2015, 01:57:59 am
just an image, the difference between a pair of Adobe CMs from Adobe Standard and home-brewed dcp from IT8 printed target (which was actually done by generating SSF from a single raw, then SSF to dual illuminant DCP), the raw from I-R (Sony A7) :

guessed SSF in question also attached :

Really interesting results! Seems like I should consider trying to integrate some SSF guessing function to DCamProf afterall.
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on May 28, 2015, 03:11:43 am
I depends on how it lines up with the reflectance spectra and then the SSF, if the bumps here and there in the spectrum doesn't change the integrated value much, then colors won't change much.

I'm also trying to simulate D50 and D65 with halogen plus filters, not really happy with the shape of the spectrum (the 80B filter is way more bumpy than I expected) but your results are encouraging. I'm most worried about that the 80B filter lets through high reds which I guess can cause the accuracy of super-saturated reds on a glossy target go bad.

I just measured H3200K nominal halogen lamp ( Ushio JCV120V-1200WCH ) with Hoya HMC80A with i1pro2 (previously I did with ColorMunki)... so here they are (averaged over 10+ tries)

(http://s4.postimg.org/4dtbrf1el/measurements.jpg)

Type = File 'A=H3200K-Average.sp' spect 0 [ Black ]
XYZ = 1.069623 1.000000 0.408425, x,y = 0.431639 0.403543
D50 L*a*b* = 100.000000 17.595772 41.778263
CMYV density = 1.752698 1.938398 2.220658 1.834233
CCT = 3086.897709, VCT = 3070.559182
CRI = 99.4
CIEDE2000 Delta E = 0.774610

Type = File 'P=H3200K+HMC80A-Average.sp' spect 0 [ Red ]
XYZ = 0.946912 1.000000 0.787862, x,y = 0.346249 0.365661
D50 L*a*b* = 100.000000 -3.006881 3.039750
CMYV density = 1.894677 1.865917 1.954723 1.862778
CCT = 5003.925303, VCT = 4813.538610
CRI = 92.2
CIEDE2000 Delta E = 2.984494
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on May 28, 2015, 04:09:01 am
I've sent a question to Lee Filters and asked if they have some color balancing filters that can reshape tungsten to D50 and D65 without that localized 560nm bump, 650 dip and 700+ peak like the 80-series filters seem to have. They're good at support so I hope to get an answer, my questions tend to be a bit too technical though :)

I got a couple of Solux 4700K lamps and tried them yesterday but I don't get them to meet the specs, they should do a very good smooth 5000K on slight overdrive (really good D50 simulation), but mine gives only 3900K at 12 volts and reaches only 4600K on 16 volts, and then with a bad spectral shape. I'm currently contacting support at the seller to discuss that issue.

Ushio has the 6500K whitestar, but looking at the spec sheet it seems like it has a peaky spectrum, ie it doesn't mimic D65 well, so I don't think that is a good option. The thing with simulating high temps with tungsten is that you need to filter them, which is done in the reflectors of the Solux and Whitestar (the reflectors thus look blue). One can't get more blue (unless increasing voltage), only filter away red the total output of the filtered lamps becomes low, and I guess if one wants a smooth spectrum one needs to filter even more so the output becomes even weaker. I guess you can keep the color and get more power by letting through some green and making a dip in central red and that's why light balancing filters are the way they are. Not ideal though when we want to use them for camera calibration, although it seems to work quite well in practice anyway.
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on May 29, 2015, 09:45:00 am
Still investigating the CAT or not to CAT issue... difficult to get answers, I get contradicting messages when I read about it. There's correlated color sets, but also those performance tests when they use spectral relighting as reference, such as Bruce Lindbloom's example:

http://www.brucelindbloom.com/index.html?ChromAdaptEval.html

Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on May 29, 2015, 01:32:20 pm
...seem's like we're past the compiling stuff in the DCamProf thread by the way... at least for now...

dcamprof v0.7.0 mingw build crashes when invoked with make-testchart command (you can see that "...... done!" is printed, but .t1 file is not generated)

>dcamprof make-testchart -p 24 test.ti1
Generating 24 patches...
................done!

Win8.1x64 event viewer says :

Quote
Faulting application name: dcamprof.EXE, version: 0.0.0.0, time stamp: 0x5560a8e9
Faulting module name: ntdll.dll, version: 6.3.9600.17736, time stamp: 0x550f4336
Exception code: 0xc0000374
Fault offset: 0x00000000000f0f20
Faulting process id: 0x22b0
Faulting application start time: 0x01d09a347828b0fd
Faulting application path: D:\argyll\bin\dcamprof.EXE
Faulting module path: C:\Windows\SYSTEM32\ntdll.dll
Report Id: b60e4dbd-0627-11e5-84b5-80fa5b00a892
Faulting package full name:
Faulting package-relative application ID:

any idea that comes to mind right away ?
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on May 29, 2015, 01:52:08 pm
any idea that comes to mind right away ?

Crashes for me as well, a bug. I'll fix to next patch release. If you specify the number of lines/rows you want (say -l 4) it won't crash.

Regarding the CAT or no CAT, read even more now and I will make CAT a user option to next release. A modern CAT like CAT02 does try to render those subtle appearance differences so it's indeed not the same as relighting. However, it's not really clear how good it is at it, and it's a bit of a matter of taste and situation if you'd want to do it or not, so the default will at least in the next release be same as before, but you can then play with CAT if you like.
Title: Re: DCamProf - a new camera profiling tool
Post by: GWGill on May 29, 2015, 10:48:55 pm
Still investigating the CAT or not to CAT issue... difficult to get answers, I get contradicting messages when I read about it.
It's a topic with lots of inter-twined connections. Here are some of my thoughts on it:

Re-lighting generally has two effects :- it can change the colors of surfaces due to the details of spectral interaction between an illuminant and a colorant, but it also often shifts the white point.

The human eye and brain is always trying to extract the maximum amount of information from the light that hits the retina. One of the mechanisms to do this is chromatic adaptation. A lot of the adaptation seems to be due to the S, M & L w.l. cones independently adapting to a luminance level, explaining the success of the Von-Kries approach to modeling chromatic adaptation, but there are lots of other things going on in our visual path, including in the nerve cells in the retina, and at higher levels in our visual system.

Chromatic adaptation can be seen as a mechanism that helps achived color constancy - our visual system is striving to present an image to our minds that looks the same, independent of what the actual lighting situation is. In this sense, we are trying to "re-light" the scene in our minds. We don't have the spectral information about what's going on though, so we probably use various heuristics, on top of the adaptation that the retina does.

So a chromatic adaptation algorithm (or matrix) is an attempt to model our visual system and minds inexact algorithm for virtually re-lighting a scene.  So re-lighting a scene using exact spectral information will arrive at a result that is a lot like what we are trying to do in our eyes and minds when we look at the original scene, but it is not an exact model of it.
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on May 30, 2015, 12:20:19 am
Graeme,

txt2ti3 when fed with CGATS file with raw RGB data and .ti3 from chartread with spectral data there will not include spectral details in the output, unless I replace "SPEC_" in source .ti3 with "nm", "NM_", "SPECTRAL_NM_",  "R_" or "SPECTRAL_"... it is certainly for a good reason, but not convenient !
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on May 30, 2015, 05:21:19 am
It's a topic with lots of inter-twined connections. Here are some of my thoughts on it:

Re-lighting generally has two effects :- it can change the colors of surfaces due to the details of spectral interaction between an illuminant and a colorant, but it also often shifts the white point.

The human eye and brain is always trying to extract the maximum amount of information from the light that hits the retina. One of the mechanisms to do this is chromatic adaptation. A lot of the adaptation seems to be due to the S, M & L w.l. cones independently adapting to a luminance level, explaining the success of the Von-Kries approach to modeling chromatic adaptation, but there are lots of other things going on in our visual path, including in the nerve cells in the retina, and at higher levels in our visual system.

Chromatic adaptation can be seen as a mechanism that helps achived color constancy - our visual system is striving to present an image to our minds that looks the same, independent of what the actual lighting situation is. In this sense, we are trying to "re-light" the scene in our minds. We don't have the spectral information about what's going on though, so we probably use various heuristics, on top of the adaptation that the retina does.

So a chromatic adaptation algorithm (or matrix) is an attempt to model our visual system and minds inexact algorithm for virtually re-lighting a scene.  So re-lighting a scene using exact spectral information will arrive at a result that is a lot like what we are trying to do in our eyes and minds when we look at the original scene, but it is not an exact model of it.

Thanks! I will in the next version make it a user option to either render XYZ for the calibration illuminant and then do CAT02 to get D50 values, or re-render for D50 from spectra directly (like it does now).

When spectral info is missing DCamProf needs to use CAT to get to the right starting point, and in that case it would be best with a CAT that tries to model relighting as close as possible, rather than be based on corresponding color experiments. An extreme example, if we have a reference file without spectra and with D65 XYZ values and we shoot StdA, one would want to convert to StdA with a "spectral" CAT (bradford seems best alternative) and then back up to D50 with CAT02. Not sure if that could make sense though. Now DCamProf always uses CAT02 and will in that case convert from D65 down to D50.

From all the papers I've skimmed it seems like CAT02 is the current best of the established methods to do a "perceptual" CAT, and Bradford is the best at doing a "relighting" CAT.
Title: Re: DCamProf - a new camera profiling tool
Post by: Iliah on May 30, 2015, 08:28:57 am
The spectral data of the light source can be rather accurately guessed from white balance coefficients recorded in the raw file (one can use exiv2 or exiftool to get those; I would prefer exiftool as it is easier to patch when a new camera appears), or from external table (necessary for Nikons, as they do not put white balance coefficients into raw, they have colour data in the form of coefficients for unknown white balance calculation formula).

Chromatic adaptation is the worst case scenario in photography, and encouraging its use is far from best practice. However if it is to be used, we first do spectral adaptation of the reference data (what is called "re-lightning" here, a misleading term for a photographer) to the light used in the shot; and after the colour transform is calculated we calculate chromatic adaptation.

I do not see why not to start from normal workflow, based on 3 shots, Ill. A, 5000K, 6500K; and only after that see what is the error introduced by the methods including smaller number of shots.
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on May 30, 2015, 10:05:23 am
Just released version 0.7.1. It has the new commands average-targets (stacking for noise reduction) and match-spectra (comparing spectra between targets), adds TIFF support for flatfield.

I've now altered the use of CAT a bit. Bradford is used whenever spectral data is missing and we need to remap XYZ reference coordinates to the calibration illuminant, needed for the CM. I use Bradford instead of CAT02 as all papers I've skimmed indicate that Bradford is better than CAT02 for this particular task when you don't want to model corresponding colors.

Then in the case spectral data is available CAT02 can *optionally* (-C) be used instead of regeneration from spectra for the conversion from calibration illuminant to Profile Connection Space D50. The purpose of using CAT02 would be to simulate the subtle color appearance differences, this will work best in the StdA to D65 range as the corresponding color experiments that CAT02 is based on are primarily made in that range.

I have not yet decided if I should change the default to use CAT instead of spectral generation, for now the default is same as the old. In the default case a tungsten profile shot under tungsten light will (ideally) render the exact same result as a D65 profile shot under D65 light, and that result is equivalent to how colors appear under D50. If CAT02 is enabled, the tungsten and D65 result will look slightly different, and is supposed to look closer to how the eye experienced the original color condition.

I have not yet found any experimental data when CAT02 and spectral generation is compared concerning matching the corresponding color experiments. If I find that and see that CAT02 is significantly better at simulating the "real" look, then using CAT02 will be the new default for the next release.
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on May 30, 2015, 10:18:40 am
I do not see why not to start from normal workflow, based on 3 shots, Ill. A, 5000K, 6500K; and only after that see what is the error introduced by the methods including smaller number of shots.

Concerning DCamProf the camera shots are needed for getting the raw RGB values, but for XYZ they serve no purpose. XYZ values either comes from a manufacturer-provided CGATS reference file, and then often XYZD50 values, or ideally you have a spectrometer and can measure spectral reflectance from the patches.

So making 3 shots is not necessary, if you're making a dual-illuminant profile for StdA and D65 you only need the StdA and D65 shots. For D50 we only need XYZ values and in that case the camera cannot help us.

The question has then been how to generate those D50 values. From corresponding color experiments we know that even with a fully adapted eye colors do not appear exactly the same in StdA and D65. If we want to model that this means that the D50 XYZ values that we need to connect to the profile connection space should be different from StdA and D65, and CAT02 is the current best at modeling that. That is with spectral data available you first generate XYZ for StdA using spectral data and then remap to D50 with CAT02, and the same for D65.

What's still unclear to me is how good the CAT02 is at modeling this. Maybe it's so bad that regenerating the D50 values from spectra (and thus let them be the same for both StdA and D65) would be a smaller error than using CAT02 despite that D65 and StdA will be equalized to D50 look. Before I make CAT02 enabled per default I need to find an answer to that question.
Title: Re: DCamProf - a new camera profiling tool
Post by: Iliah on May 30, 2015, 10:33:03 am
> if you're making a dual-illuminant profile for StdA and D65 you only need the StdA and D65 shots
Well, forward matrix is for D50 white point. It is better be computed when the target is physically lit by a light source with CCT around 5000K.
The device response depends on the light spectrum, and white balance does not exactly equalize different light sources even when they are close to SPD of blackbody.

> ideally you have a spectrometer and can measure spectral reflectance from the patches
Not in my experience. Most spectrometers that are in the hands of photographers can't be used to reliably measure a target, especially when it comes to darker patches.

Actually, much more than 3 shots are needed. Black frame needs to be subtracted, division by the saturated frame needs to be performed, flat field needs to be applied. If one is to average 3 shots, he needs 3 blacks and 3 whites too, plus a flat field.
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on May 30, 2015, 10:52:51 am
Actually, much more than 3 shots are needed. Black frame needs to be subtracted, division by the saturated frame needs to be performed, flat field needs to be applied. If one is to average 3 shots, he needs 3 blacks and 3 whites too, plus a flat field.
Iliah, aren't you going into overkill mode ? because then one can say that you need black frame and saturation frame for a flatfield shot too, no ? and 3 of those then
Title: Re: DCamProf - a new camera profiling tool
Post by: Iliah on May 30, 2015, 10:57:41 am
> one can say that you need black frame and saturation frame for a flatfield shot too, no ? and 3 of those then

I think you can find a lot on the topic on astronomers' sites.
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on May 30, 2015, 11:08:45 am
0.7.1 build for Windows (mingw = dcamprof.exe + libgomp_64-1.dll) : https://app.box.com/s/x14fjbme5oc0lrrjeck6zlry3nxslavy
Title: Re: DCamProf - a new camera profiling tool
Post by: Bart_van_der_Wolf on May 30, 2015, 11:17:40 am
> one can say that you need black frame and saturation frame for a flatfield shot too, no ? and 3 of those then

I think you can find a lot on the topic on astronomers' sites.

True, but their challenge is a lack of photons. That means that each photon counts, and all sources of noise are a handicap. That's why statisticallly weighted averages are taken from many 'exposures', usually some 16 or more per type of correction (read-noise, dark noise, offset noise, and flat-field).

One of the first things a profile shot needs, is proper levels of exposure, preferably uniform, and illuminated (with a suitable spectral output) at such an angle that subject/target/surface reflections play a minor role. The flat-fielding is to eliminate un-even illumination and lens vignetting and light fall-off.

Cheers,
Bart
Title: Re: DCamProf - a new camera profiling tool
Post by: Iliah on May 30, 2015, 11:21:24 am
When we add a black traps for accurate flare compensation what good is that if the levels are wrong. Different personal standards exist, but why not to allow for high ones?
Title: Re: DCamProf - a new camera profiling tool
Post by: Bart_van_der_Wolf on May 30, 2015, 11:27:44 am
When we add a black traps for accurate flare compensation what good is that if the levels are wrong. Different personal standards exist, but why not to allow for high ones?

Nothing wrong with high standards, but they should have a meaningful effect.
When diminishing returns no longer have a measurable effect, we've gone too far.

Cheers,
Bart
Title: Re: DCamProf - a new camera profiling tool
Post by: Iliah on May 30, 2015, 11:30:44 am
Until one checks, how could he know if the effect is meaningful or not? I went to the trouble, and for me and my customers it is meaningful. So I shared my experience.
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on May 30, 2015, 11:47:00 am
I haven't seen any indications that astronomy methods need to be applied on camera color profiling, I don't think that level of precision is required. Of what I've seen so far camera profiling is the most approximate among the typical profiling tasks (screen, print, camera). Still I've added stacking in the latest release so one can experiment. Of course it does not hurt to improve the quality of input data, but at some point it becomes overkill and one is doing more work than necessary. From the results I've seen so far flatfield is necessary if you lit the target with one lamp, but if you have a quality "copy" style setup flatfield is overkill. However how large measurement errors one can accept is also a matter of taste, so different users will come to different conclusions.

As a sanity check one can make one profile with "sloppy" measurements and one with all bells and whistles and compare the results. With DCamProf you can also make experiments with SSFs (where measurement errors are nil) and do some noise-adding experiments and see how things start to deviate.

About the D50 whitepoint, the reason FM is D50 is that profile connection space is D50, to be able to get further in the color pipeline we need to get into D50. However, it doesn't necessarily mean that FM must render the color as if the target was lit by D50, you can keep the color appearance of the original illuminant if you want to. If you want to do that depends on the situation of course, if you're making a reproduction profile for Tungsten lights you probably want to make the subject look like it was shot under D50 anyway as it's better for copy applications. If you want to retain the look of Tungsten, to make colors look as they look to the eye under Tungsten, you need to do something else though. This is what the "corresponding color sets" experiments is about which CAT02 is based on. As said I need more data on the quality of its results though before I'll make it the default behavior in DCamProf.

It seems like you are interpreting the FM as it should be lit for D50 as well, that is the camera raw RGB values should be for D50 even if the profile is a StdA. This is not how DCamProf does it currently, RGB values are always for the calibration illuminant. Citing the DNG Spec:

"The use of the forward matrix tags is recommended for two reasons. First, it allows the camera profile creator to control the chromatic adaptation algorithm used to convert between the calibration illuminant and D50. Second, it causes the white bala
nce adjustment (if the user white balance does not match the calibration illuminant) to be done by scaling the camera coordinates rather than by adapting the resulting XYZ values, which has been found to work better in extreme cases."


I've interpreted this like the intention is that RGB values should stem from the calibration illuminant, and how the D50 XYZ values are derived is up to the profile designer ("control the CAT"), that is one can for example choose to re-generate from spectra for a copy profile, or use CAT02 to better preserve actual color appearance.

If we make the FM matrix based on D50 RGB values this means that the matrix will look exactly the same regardless of calibration illuminant, and that does not seem right to me.
Title: Re: DCamProf - a new camera profiling tool
Post by: Iliah on May 30, 2015, 12:05:23 pm
Well, I have no intention to interfere. Each of us do the things the way we see fit.
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on May 30, 2015, 12:12:52 pm
Until one checks, how could he know if the effect is meaningful or not? I went to the trouble, and for me and my customers it is meaningful. So I shared my experience.
right - but you just said that most of us do not have a spectrophotometer capable to properly measure all of the target with dark patches (but you have such instruments or access to such instruments), so in that case for those who do not (and i1pro2 is not good) - how does black frame and saturated frame helps when we do not have a target properly measured in the first place ? so certainly while utmost precision is a subject of desire - am I to gain ?... personal testing is good (at least as an experience if nothing else), but I 'd love to hear opinions before testing (because it takes a lot of time and I do have a day job - so I have to consider the time) from those who did... so am I going to gain when I can't do better than i1pro2 with a target ?
Title: Re: DCamProf - a new camera profiling tool
Post by: Iliah on May 30, 2015, 12:16:18 pm
But I explained already on another forum. You do not need dark patches in the target for profiling, instead you make a composite CGATS from 2 or more shots with different shutter speeds; and you scale the spectral reference accordingly.
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on May 30, 2015, 12:18:51 pm
But I explained already on another forum. You do not need dark patches in the target for profiling, instead you make a composite CGATS from 2 or more shots with different shutter speeds; and you scale the spectral reference accordingly.
yes, that was about the black patches in SG - but what if you have a target with darker patches different chromacity wise from lighter patches - then you can't use spectrum (measured with i1pro2) from lighter patches for those, right ? and not only then - w/o actually measuring properly (not possible with i1pro2) darker patches you even can't say that they are of the same chromacity as lighter patches even if manufacturer claims that ! catch22 - no ?
Title: Re: DCamProf - a new camera profiling tool
Post by: Bart_van_der_Wolf on May 30, 2015, 12:21:00 pm
I haven't seen any indications that astronomy methods need to be applied on camera color profiling, I don't think that level of precision is required. Of what I've seen so far camera profiling is the most approximate among the typical profiling tasks (screen, print, camera). Still I've added stacking in the latest release so one can experiment. Of course it does not hurt to improve the quality of input data, but at some point it becomes overkill and one is doing more work than necessary. From the results I've seen so far flatfield is necessary if you lit the target with one lamp, but if you have a quality "copy" style setup flatfield is overkill. However how large measurement errors one can accept is also a matter of taste, so different users will come to different conclusions.

Hi Anders,

I think that the averaging of multiple exposures can help in improving (in particular) the darker patch quality, so it's a useful addition. Flat-fielding is more of a must, due to illumination issues and lens vignetting/light fall-off when the target takes up a significant part of the image and/or the use of relatively wide apertures cannot be avoided.

Cheers,
Bart

P.S. I just realized myself that to eliminate most uneven illumination issues, one can shoot the target at a 45 degrees angle, and illuminate the target perpendicular to its surface with a single lightsource. This will eliminate most surface reflection issues, but of course leaves the camera/lens vignetting and light fall-off issues to deal with. Shooting with a longish focal length will reduce the lighting angle effects even further.
Title: Re: DCamProf - a new camera profiling tool
Post by: Iliah on May 30, 2015, 12:24:05 pm
> darker patches different chromacity wise from lighter patches
Different chromaticity does not matter with a target composed of 5 pigments only.
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on May 30, 2015, 12:26:08 pm
the darker patch quality
but see the note about measuring those patches - yes, you improve raw RGB data - but where you are going to get the measurements of the same accuracy ? unless you are not going to use those patches in the first place or you have a magnitude better measuring device for your target ... it is interesting to know what are the actual limits and how you can actually find out that the patch that you are trying to measure with i1pro2 or colormunki is not the one that you can reliable measure with them... does anybody has any numbers ?
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on May 30, 2015, 12:28:28 pm
> darker patches different chromacity wise from lighter patches
Different chromaticity does not matter with a target composed of 5 pigments only.
you mean target that is printed on a printer with inks, right ? and the higher end targets simply do not (except black patches) patches that are not reliably measured with i1pro2-level instruments by design ?
Title: Re: DCamProf - a new camera profiling tool
Post by: Iliah on May 30, 2015, 12:30:02 pm
No, I mean ColorChecker series. There were times when 6 pigments were in use, but one of those got into hazardous materials list.
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on May 30, 2015, 04:57:17 pm
A good way to test the performance of a test target is to measure it, make a profile with camera SSFs and then test the resulting profile against real spectral data from a database. I've done a basic test of that type in a neighboring thread.

The X-Rite targets do well, although profiling directly against the desired database you want to match is best of course. The profiling problem is sort of an impossible equation though, if you correct for one set of spectra it may produce contraproductive corrections for another set of spectra. I've seen this when working with SSFs and profiling directly against real spectra. It's not a big problem though, but the point is that one needs to make some sort of average correction for typical spectra, which means that the search for the perfect target is futile. It's not certain that a profile made against real spectra will work better than a profile made against an artifical target.

The number of pigments matters, but I guess also the spectral shape of the pigments. The X-rite CC24 have nice shapes of their spectra (you can plot it to see) probably because the pigments are broad band and more overlapping. Compared to patches printed on my inkjet which have more narrow band pigments that can achieve higher saturation, but end up with less varied spectra anyway. All that is an area worth more investigation though, from my point of view.
Title: Re: DCamProf - a new camera profiling tool
Post by: GWGill on May 30, 2015, 10:36:36 pm
txt2ti3 when fed with CGATS file with raw RGB data and .ti3 from chartread with spectral data there will not include spectral details in the output, unless I replace "SPEC_" in source .ti3 with "nm", "NM_", "SPECTRAL_NM_",  "R_" or "SPECTRAL_"... it is certainly for a good reason, but not convenient !
txt2ti3 is intended to input foreign file formats, not .tiX files.  What are you trying to achieve ? - there may be some other way of doing it that is more appropriate.
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on May 31, 2015, 12:51:12 am
What are you trying to achieve ? - there may be some other way of doing it that is more appropriate.

for example I have a cgats file with raw rgb data

case #1, I have spectral measurements produced by something else in cgats format
case #2, I have spectral measurements produced by argyll chartread (.ti3 has also XYZ and/or LAB, even I do not need to have them, no option to exclude... not a bid deal, but for consistency again it is good to be able to say that I only need spectral data)

case #1, using txt2ti3 to combine the files gives the output that I like : raw rgb from the actual shot + spectral data from the target measurements, leaving whatever software further down the chain to use the spectral data as it can/need
case #2, using txt2ti3 to combine the files gives me raw rgb + but no spectral data included, which kind of makes using chartread more difficult to use in this situation

inconsistent at least ! what I 'd love to have is to use argyll chartread to measure a target and then use txt2ti3 to combine raw rgb data with just spectral data from chartread output... w/o using text editors, etc... yes, I understand the origin how txt2ti3 was intended to be used, but

PS: and if the target measuments were not in 10nm (but for example like -H mode in 3.33x increments), why it is necessary to strip intermediate measurements from the source file and output only SPEC_380, SPEC_390, etc ?
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on May 31, 2015, 06:47:49 am
Been reading more on CATs. It seems like all CATs, including the linear Bradford, are derived from corresponding color data sets and thus try to model "color inconstancy", that is the subtle color apperance differences that still exist under various light.

There is no established model specifically designed for "relighting", that is predict XYZ coordinates for a different illuminant that gives the same result as re-intergrating the reflectance spectra with the new illuminant, although there are papers describing methods for doing that, such as this: http://www.cs.sfu.ca/~funt/2014_CIC22_MIRZAEI_GaussianPrediction.pdf and those methods make more accurate results than the established CATs. However, as a DCamProf workflow should include spectra in the normal case implementing this type of transform will not be a top priority. 0.7.1 will use Bradford rather than CAT02 when spectral data is missing for relighting, as the test data I've seen in papers show that it performs better for that particular task, but as said it's still not designed for it and there are better models for it.

The other more interesting question I'm trying to find an answer to is how well the CAT02 keeps appearance over illuminant change. The problem DCamProf faces is that profile must be made relative to D50 (the profile connection space), but the profile may have say StdA as calibration illuminant and we may want to keep those color appearances, that is convert to corresponding colors in D50 rather than relight the reflectance spectra. It's very clear that CATs are highly approximate, they will introduce errors. Therefore I wanted to find a comparison with a spectral model of perfect color constancy, that is relighting from spectra. The only paper I've found on this so far is this:
http://www.rit-mcsl.org/fairchild/PDFs/PRO28.pdf
But unfortunately the experimental data is too thin to make any conclusions, it does indicate though that the spectral color constant model might not have larger errors than CAT02 will have. It will indeed be in error by principle, show constant color when we know there should not be constant color, but the actual average DeltaE might be lower than for CAT02.

So I still don't know if the current default (use spectral color constant model) is the right one, or if I should change to CAT02. The investigation continues...
Title: Re: DCamProf - a new camera profiling tool
Post by: Bart_van_der_Wolf on May 31, 2015, 08:17:36 am
but see the note about measuring those patches - yes, you improve raw RGB data - but where you are going to get the measurements of the same accuracy ?

Just like the capture result will get improved precision from averaging, so will the target measurement improve from averaging multiple measurements. The read/electronic noise, which is a larger percentage of dark patch measurements, will reduce by the square root of the number of reads. So averaging 4 reads, will reduce the noise to 50% of a single read, 16 reads will reduce he noise to 25%. How many is enough? That depends on the level of noise and if it produces erratic readings that influence the precision notably.

Cheers,
Bart
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on May 31, 2015, 10:22:31 am
Just like the capture result will get improved precision from averaging, so will the target measurement improve from averaging multiple measurements. The read/electronic noise, which is a larger percentage of dark patch measurements, will reduce by the square root of the number of reads. So averaging 4 reads, will reduce the noise to 50% of a single read, 16 reads will reduce he noise to 25%. How many is enough? That depends on the level of noise and if it produces erratic readings that influence the precision notably.

Cheers,
Bart

I have colorchecker sg, I am measuring it with i1pro2, I am getting black patches (those along the edge) as L* = 10.x-11.x, I check the data from BabelColor Patchtool (I'd assume 1:1 from ProfileMaker) and from Iliah Borg (from makeinputicc) and I see that those patches there are L* = 6.x-7.x

Do you think that is an error that can be eliminated by many measurements ? because all my black patches like this and that is = 14 measurements, same L*'s... so do I have a different copy of CCSG (some change in manufacturing process, it differs visually from for example the one that you can see in I-R shots - there the name is on top in the middle, my has the name on top, but on the left side and "Digital SG" is black letters with white background) or does i1pro2 simply have an issue with measuring black patches that can't be eliminated by multiple measurements or something else (what) ? lighter patches way better matching L* wise between my measurements and BabelColor|Iliah.

PS: when I am measuring black trap with i1Pro2, I am getting L* reading from perfect 0.0000 to 0.1 ... so error (including operator's too) is there, but that error does not seem to be 1-s of L*
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on May 31, 2015, 10:37:09 am
Concerning relighting transform; I've compared Bradford and CAT02 with DCamProf's ability to render an artificial spectra from an XYZ coordinate, and then integrate with the new illuminant, the results are that Bradford and CAT02 has very similar results, and going through an artifical spectra is almost always more accurate, often sub-1 Delta E, while the CATs is often up at 2-4. This algorithm is similar to what is described in the paper linked earlier, although much more computationally expensive.

To the next version the default recommendation for relighting transform will be via artificial spectra, the drawback is that it's very slow though. It's still always best to have measured spectra of your target of course, but if you don't have a spectrometer and only have XYZ coordinates in your reference file and you want to calibrate for a different illuminant you need a relighting transform.

I haven't found any standard name for "relighting transform", some call it "color signal prediction", but in most contexts people say just "CAT", which is confusing as all CATs are designed to predict corresponding colors (that is which tristumulus value to use to produce the same appearance), not predict which tristimulus value the same object will have under a new illuminant. Only if you assume that color constancy is 100% perfect they will be the same, and the current established CATs do not.
Title: Re: DCamProf - a new camera profiling tool
Post by: Bart_van_der_Wolf on May 31, 2015, 10:56:47 am
I have colorchecker sg, I am measuring it with i1pro2, I am getting black patches (those along the edge) as L* = 10.x-11.x, I check the data from BabelColor Patchtool (I'd assume 1:1 from ProfileMaker) and from Iliah Borg (from makeinputicc) and I see that those patches there are L* = 6.x-7.x

Do you think that is an error that can be eliminated by many measurements ? because all my black patches like this and that is = 14 measurements, same L*'s... so do I have a different copy of CCSG (some change in manufacturing process, it differs visually from for example the one that you can see in I-R shots - there the name is on top in the middle, my has the name on top, but on the left side and "Digital SG" is black letters with white background) or does i1pro2 simply have an issue with measuring black patches that can't be eliminated by multiple measurements or something else (what) ? lighter patches way better matching L* wise between my measurements and BabelColor|Iliah.

The differences between multiple samples of the same target will gravitate towards its real mean value (somewhere between 10 and 12 in your case), as the random noise cancels out with each additional sampling of the same patch. But I think that the difference between production runs of the SG plays a role here.

Quote
PS: when I am measuring black trap with i2Pro2, I am getting L* reading from perfect 0.0000 to 0.1 (including operator's too)... so error is there, but that error does not seem to be 1-s of L*

That suggests that there is no dust/dirt or calibration issue in play here, but rather a variation in the SG targets.

Cheers,
Bart
Title: Re: DCamProf - a new camera profiling tool
Post by: Iliah on May 31, 2015, 10:57:51 am
>  haven't found any standard name for "relighting transform", some call it "color signal prediction", but in most contexts people say just "CAT"

Dr. Fairchild was using "spectral adaptation". I use sCAT as an abbreviation.
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on May 31, 2015, 11:37:08 am
sCAT sounds nice, I'll borrow that!
Title: Re: DCamProf - a new camera profiling tool
Post by: Iliah on May 31, 2015, 12:05:14 pm
Checked 3 SG targets from different manufacturing batches, Spectroscan/Spectrolino vs. i1Pro - it looks like there are no manufacturing variations to speak of. Higher values in case of i1Pro are consistent and seem to be caused by the in-instrument flare.
Title: Re: DCamProf - a new camera profiling tool
Post by: Bart_van_der_Wolf on May 31, 2015, 12:13:28 pm
Checked 3 SG targets from different manufacturing batches, Spectroscan/Spectrolino vs. i1Pro - it looks like there are no manufacturing variations to speak of. Higher values in case of i1Pro are consistent and seem to be caused by the in-instrument flare.

Hi Iliah,

Thanks for checking. I do wonder, measuring a black trap apparently gives virtually no signal, and black should not reflect much, I would not expect enough to cause such an amount of internal glare. If a lighter patch is measured in the same acceptance angle of the instrument yes, but with only 'black' in view?

Cheers,
Bart
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on May 31, 2015, 12:26:57 pm
sCAT sounds nice, I'll borrow that!

Oh, realized it's not the same thing. Fairchild's "spectral adaptation" is CAT based on full spectral data, the human adaptation is still modeled just like in an ordinary CAT. In the paper he uses a "color constant" version as reference though, which is the exact same thing as a relighting transform.
Title: Re: DCamProf - a new camera profiling tool
Post by: Iliah on May 31, 2015, 01:01:40 pm
Let's consider a workflow which was at some point SOP in (at least) one of museums for reproduction work:
- shoot a target at studio light
- measure light spectrum
- re-calculate the target reference for measured spectrum
- convert "light-adjusted" target reference to XYZ
- calculate a matrix to convert "device RGB" to "light-adjusted" XYZ
- use CAT to D50/D65/Ill.A to get matrices needed for a dcp profile

You can easily see there are serious problems with this workflow, and also they are converting to XYZ at a premature stage.

sCAT is just a term, like log (logarithm). It needs qualification as to the method used. What sCAT emphasizes is that the adaptation is based on spectra directly, not just on XYZ convolution.
Title: Re: DCamProf - a new camera profiling tool
Post by: Iliah on May 31, 2015, 01:18:25 pm
Error kicks in below L*=20, which, in the words of one of X-Rite guys on a seminar, is as deep as one may want to go for _printer_ profiling.
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on May 31, 2015, 01:43:16 pm
Very interesting on the spectrometer performance on dark patches.

DCamProf is not ideal for copy profiles as it only makes a 2.5D LUT, that is a specific chromaticity gets the same correction regardless of lightness. With say an IT-8 photographic target for copying photographs printed in the same media a 3D LUT will probably achieve a tiny bit better result.

I chose to make 2.5D profiles as I think it's better for generic profiles where exposure will vary as well as object spectra.

This means that one does not really need dark patches in the test target, except for highly saturated colors close to the line of purples that must be dark otherwise the saturation cannot be reached. (There's also a possibility that dark patches can add some spectral variation in the target by mixing in more colorants.)

From DCamProf's perspective many of the "skin tone" patches on the Colorchecker SG are redundant as they are (nearly) just darker versions of a lighter patch, and indeed the SG does not make more accurate skintone profiles than the 24 patch colorchecker when I compare with the lippmann2000 data.
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on May 31, 2015, 02:26:09 pm
the 24 patch colorchecker when I compare with the lippmann2000 data.

consider 2 cases :

1) -p cc24 -p skin-white.ti3

2) -p skin-white.ti3 -p cc24


case #1 = 2 patches from CC24 will exclude all the patches in skin-white.ti3 (resulting target = CC24 patches, 24 of them) and case #2 = all patches in skin-white.ti3 will exclude only those 2 patches from CC24 (resulting target = all 78 patches from skin-white.ti3 and 24-2 patches from CC24 = 100 patches)

so is case #2 going to be really (noticeably) better for raw with caucasian skin tone in your opinion ?

Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on May 31, 2015, 02:29:07 pm
Very interesting on the spectrometer performance on dark patches.

so may be then add a parameter to dcamprof make-profile (and may be also to dcamprof make-target) to allow (default - nothing excluded naturally) the exclusion (supply L* value for example, everything darker based on supplied target measurements / spectral data or if not XYZ/LAB / gets gutted) of dark patches (yes, we can edit  cgats with raw rgb data / target measurements, but a little bit of crude automation might be useful too ?) as "Error kicks in below L*=20, which, in the words of one of X-Rite guys on a seminar, is as deep as one may want to go for _printer_ profiling." ?
Title: Re: DCamProf - a new camera profiling tool
Post by: Iliah on May 31, 2015, 02:30:13 pm
> caucasian skin tone

Do we have significant differences in hemoglobin, melanin, carotene spectral properties between different races?
Title: Re: DCamProf - a new camera profiling tool
Post by: Iliah on May 31, 2015, 02:32:40 pm
Normally, one can't reliably print below L*=15. That may be a good default.
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on May 31, 2015, 02:37:09 pm
> caucasian skin tone

Do we have significant differences in hemoglobin, melanin, carotene spectral properties between different races?

only amount (of the same 3 organic chemicals) is different ... are you implying that we shall not bother then with all those skin spectral measurements at all ?
Title: Re: DCamProf - a new camera profiling tool
Post by: Iliah on May 31, 2015, 03:08:47 pm
Those skin samples are highly correlated, so very few patches are OK, while more than just a few may be useless or even trough things astray.
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on May 31, 2015, 03:35:30 pm
Spectral databases generally has a lot of data with similar chromaticity, including skin databases, although there are racial differences of course. In any case in terms of correction the default setting of DCamProf is to group together nearby patches and make an average correction for those. There's also a distance weighting algorithm enabled per default so it won't hurt matrix optimization either.

Anyway, in the tests I've done the CC24 skin patch gets lippmann skin sub 1 DE for 90th percentile, ie no meaningful improvement in accuracy can be had. The worst targets I've tested in terms of skin tone still gets within 1.5 DE for 90th percentile. It's hard to make any meaningful improvement on that. So no, a skin database will generally not make any meaningful improvements, not for caucasian skin.

"Excellent skin tones" in camera profiles that often comes up in the medium format forums is not so much about accuracy, but about a designed subjective look, and probably designed for foundation makeup rather than real skin. I'm quite sure that an accurate profile will not please the typical portrait photographer used to profiles with a designed look. I'm not a portrait photographer myself though so I can't contribute much when it comes to the subtleties of skin reproduction. DCamProf is not suitable for designing profiles with a look, but is rather for those of us that prefer designing our own look in the raw converter (or photoshop or whatever), with an as accurate as possible starting point.

In terms of accuracy, skin tones does not seem to be that big challenge for modern cameras, maybe there's been some consious design decision concerning camera color filters to make them good at skin tones, I don't know.
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on May 31, 2015, 03:43:27 pm
so may be then add a parameter to dcamprof make-profile (and may be also to dcamprof make-target) to allow (default - nothing excluded naturally) the exclusion (supply L* value for example, everything darker based on supplied target measurements / spectral data or if not XYZ/LAB / gets gutted) of dark patches (yes, we can edit  cgats with raw rgb data / target measurements, but a little bit of crude automation might be useful too ?) as "Error kicks in below L*=20, which, in the words of one of X-Rite guys on a seminar, is as deep as one may want to go for _printer_ profiling." ?

Yes I will consider something like that, maybe think a bit more about filtering. When doing SSF profiles one may have lots of patches so it becomes really tedious to filter out one patch at a time manually.

Looked through some target data, and for DCamProf's sake excluding all sub L*20 patches does not seem to be a loss. It's possible to cover any reasonable chromaticities with L*20 and up.
Title: Re: DCamProf - a new camera profiling tool
Post by: Iliah on May 31, 2015, 03:51:12 pm
In photography, tone is brightness and contrast (gradation), not colour. We can have excellent skin tones on black-and-white photo.
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on May 31, 2015, 04:14:37 pm
so I decided to make SSF out of just __3__ (three) patches from IT8 shot - took most saturated R, G, B that were > ~30 L*... and... with AA7hVFAI00100.ARW from I-R for Sony A7 ( SSF -> StdA & D65 virtual targets with -p cc24 -> dual illuminant .dcp -> ACR ) :

(http://s14.postimg.org/9vvou3ktd/SSF3_1.jpg)

(http://s15.postimg.org/ifem8r763/SSF3_2.jpg)

it is fair to note that it was a matrix .dcp profile : CM + FM and no LUT
Title: Re: DCamProf - a new camera profiling tool
Post by: Bart_van_der_Wolf on May 31, 2015, 04:23:36 pm
so may be then add a parameter to dcamprof make-profile (and may be also to dcamprof make-target) to allow (default - nothing excluded naturally) the exclusion (supply L* value for example, everything darker based on supplied target measurements / spectral data or if not XYZ/LAB / gets gutted) of dark patches (yes, we can edit  cgats with raw rgb data / target measurements, but a little bit of crude automation might be useful too ?) as "Error kicks in below L*=20, which, in the words of one of X-Rite guys on a seminar, is as deep as one may want to go for _printer_ profiling." ?

I don't immediately accept that L*= 20 is a/the limit, not without some more empirical evidence. A stepwedge. e.g. one from Stouffer like the R2110 (http://www.stouffer.net/Reflection.htm), might reveal a non linearity in the L* readings towards D-Max, or not. If internal glare is in play, which I somewhat doubt because that would require a bright patch and a dark patch at the same time, then the slope would not be close to the expected step delta 0.10 Optical Density . When the i1pro2 sees only a dark patch, then there is not enough light for a significant glare contribution. Maybe it's electronic noise that creates a threshold, but then a light trap would also give a signal.

Cheers,
Bart
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on May 31, 2015, 04:30:52 pm
so I decided to make SSF out of just __3__ (three) patches from IT8 shot - took most saturated R, G, B that were > ~30 L*... and... with AA7hVFAI00100.ARW from I-R for Sony A7 ( SSF -> StdA & D65 virtual targets with -p cc24 -> dual illuminant .dcp -> ACR )

How is that SSF desgined? Is it that matlab script? I have not had the time to investigate that, but my guess is that it uses a database of measured SSFs and then does some fitting against that? This probably means that with as little data as this there's an element of luck involved, that is if the SSF matches well another SSF in the database you can get a good result.
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on May 31, 2015, 04:31:52 pm
I don't immediately accept that L*= 20 is a/the limit

but that's why you shall be able to use whatever number you want as a parameter or not use that and so all patches included - makes it easier to test then... Iliah says that was said by X-Rite people, I think he 'd not/never quote somebody who was just a marketing person, right ?
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on May 31, 2015, 04:33:17 pm
Is it that matlab script?
basically yes, I did just some mods - to read CGATS from rawdigger (instead of using dcraw, etc), use my target illumination spectrum and use my target measurements from external files and as camera's database (certainly - we do not know who good were there measurements with monochromator - what it is what it is) is the same - I made eigenvectors constant... gutted also whatever junk is there.
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on May 31, 2015, 04:39:32 pm
This probably means that with as little data as this there's an element of luck involved
there is only one Sony there... if one to assume that each manufacturer stays along some "brand color" in CFA design (SSF-wise) then it is fitting well w/ the database consisting mostly of Nikon and Canon :

Canon1D Mark III
Canon20D
Canon300D
Canon40D
Canon500D
Canon50D
Canon5D Mark II
Canon600D
Canon60D
Hasselblad
Nikon3dx
NikonD200
NikonD3
NikonD300s
NikonD40
NikonD50
NikonD5100
NikonD700
NikonD80
NikonD90
NokiaN900
Olympus EPL2
PentaxK5
PentaxQ
Phase One
PointGreyGrasshopper
PointGreyGrasshopper2
Sony Nex5N



CGATS/Raw RGB :

1579.63   727.59   279.61
392.85   1445.76   2621.2
246.83   1708.88   406.09


Target (380 : 10 : 730:

0.006771   0.012186   0.019083   0.022492   0.018859   0.015078   0.01285   0.012292   0.013557   0.01626   0.02006   0.02377   0.025309   0.023362   0.020727   0.019255   0.018586   0.018996   0.02305   0.036697   0.070012   0.131161   0.215468   0.306025   0.390381   0.459813   0.513263   0.554692   0.590821   0.622936   0.652642   0.677777   0.69902   0.717879   0.739663   0.765332

0.007337   0.034702   0.090921   0.207428   0.320195   0.375344   0.384487   0.368493   0.33102   0.273935   0.212965   0.157102   0.109939   0.073874   0.051659   0.039648   0.032222   0.027431   0.026041   0.029307   0.035212   0.038126   0.03505   0.03127   0.030014   0.029921   0.0298   0.030836   0.035664   0.047068   0.067224   0.098205   0.137844   0.183507   0.236008   0.29608

0.006708   0.006364   0.008263   0.007945   0.006133   0.004683   0.003775   0.004248   0.005308   0.008668   0.016754   0.033854   0.063985   0.107077   0.154427   0.184376   0.184275   0.158114   0.120697   0.085062   0.053481   0.028621   0.013728   0.00766   0.006021   0.005722   0.005607   0.005918   0.007386   0.011355   0.019664   0.035081   0.058581   0.09012   0.130444   0.180574
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on May 31, 2015, 04:44:11 pm
then there is not enough light for a significant glare contribution. Maybe it's electronic noise that creates a threshold, but then a light trap would also give a signal.

that is beyond my knowledge to discuss, sorry... I can say that with black trap from Datacolor SpyderCube and not in a pitck dark room and with operator errors (the main again being not in a lightless room) I was never getting > 0.1 L* if I make decent effort fitting i1pro2 to the hole
Title: Re: DCamProf - a new camera profiling tool
Post by: Tim Lookingbill on May 31, 2015, 04:59:10 pm
Spectral databases generally has a lot of data with similar chromaticity, including skin databases, although there are racial differences of course. In any case in terms of correction the default setting of DCamProf is to group together nearby patches and make an average correction for those. There's also a distance weighting algorithm enabled per default so it won't hurt matrix optimization either.

Anyway, in the tests I've done the CC24 skin patch gets lippmann skin sub 1 DE for 90th percentile, ie no meaningful improvement in accuracy can be had. The worst targets I've tested in terms of skin tone still gets within 1.5 DE for 90th percentile. It's hard to make any meaningful improvement on that. So no, a skin database will generally not make any meaningful improvements, not for caucasian skin.

"Excellent skin tones" in camera profiles that often comes up in the medium format forums is not so much about accuracy, but about a designed subjective look, and probably designed for foundation makeup rather than real skin. I'm quite sure that an accurate profile will not please the typical portrait photographer used to profiles with a designed look. I'm not a portrait photographer myself though so I can't contribute much when it comes to the subtleties of skin reproduction. DCamProf is not suitable for designing profiles with a look, but is rather for those of us that prefer designing our own look in the raw converter (or photoshop or whatever), with an as accurate as possible starting point.

In terms of accuracy, skin tones does not seem to be that big challenge for modern cameras, maybe there's been some consious design decision concerning camera color filters to make them good at skin tones, I don't know.

This Capture 1 tutorial on achieving uniform skin tone confirms what you've indicated about attempting to solve this through a profile due to so many variables that affect skin tone appearance, the effects of makeup base being the tip of the iceberg. It shows that one has to go to the lengths of editing the images color tables manually using Capture 1's Color Editor Tool...

http://blog.phaseone.com/get-uniform-skintones/
Title: Re: DCamProf - a new camera profiling tool
Post by: Iliah on May 31, 2015, 05:30:43 pm
Adding more highly correlated patches normally does not help, especially with LUTs. If spectrals are known, it is usually better to eliminate highly correlated samples and patches at the very first step.

For repro work, for example, we make targets that consist only of basic pigments used in painting (a profile for every particular painting, if it is for archival and preservation purposes). Lightness and saturation do not matter - they are anyway achieved through the mix of the basic pigment with something like titan white and soot, both being spectrally very flat (hence in repro work 2.5D or 2D profiles are used more often than not if it is a serious museum).

With CC24/CCSG decent profiles can be obtained from 5-6 non-neutral patches + grey step wedge.
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on May 31, 2015, 05:37:15 pm
of basic pigments
how do you find a number of different "basic" pigments if you have a patch (one patch of whatever color) and spectrophotometer then ? is it possible ?
Title: Re: DCamProf - a new camera profiling tool
Post by: Iliah on May 31, 2015, 05:42:51 pm
There is a reason why for certain measurements Gretag used to offer polarizing filters with Spectrolino. Light is scattered back from both gloss and textured surfaces, causing flare and glare and making readings from very dark patches less reliable for semigloss and gloss substrates, and glossy black/dark pigments.
i1Pro is cost-optimized not for target measurements, but for printer profiling. IMHO it is only very logical in terms of engineering.
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on May 31, 2015, 05:45:19 pm
With CC24/CCSG decent profiles can be obtained from 5-6 non-neutral patches + grey step wedge.
so for dcamprof as a matter of automation for hoi polloi (dcamprof make-dcp/make-icc steps) it is better to find out and select(make it as a command line parameter/option of course) only non-neutral ( or for example with C* from L*C*h* > N ) patches, calculate CM/FM and possibly 2.5D LUT and then proceed (make it again as a command line parameter/option ) to neutral patches and adjust LUT (or even possibly a tone curve) ?

certainly with CC24 there are so few patches that it might be moot point (bare minimum of non neutral patches) - but for CCSG and homemade targets
Title: Re: DCamProf - a new camera profiling tool
Post by: Iliah on May 31, 2015, 05:49:46 pm
Well, for 1 patch it is possible only with the help of a database. Don't know if such databases are publicly available, and free; but OKUMURA YOSHIO made quite a study in R.I.T. on the topic.

UPD. Found one of his studies online http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.79.799&rep=rep1&type=pdf
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on May 31, 2015, 06:19:34 pm
Well, for 1 patch it is possible only with the help of a database. Don't know if such databases are publicly available, and free; but OKUMURA YOSHIO made quite a study in R.I.T. on the topic.

UPD. Found one of his studies online http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.79.799&rep=rep1&type=pdf

"...The interesting figure of the table is the volume of a inkjet printer (Canon PIXUS iP8600) because the size is not different from the volume of the Pointerʼs color gamut. The result reminds us of an innovative printing technology in which eight dye inks of the printer reproduce almost all the surface real colors introduced by Pointer [Pointer 1980]..."
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on May 31, 2015, 08:58:03 pm
one more unscientific test about dark patches, their measurements, glare on dark patches with glossy print, etc ... this IT8 target was the one measured and few saturated patches from it (actual shot) were used to build SSF profile which was then used in raw conversion of the this shot

(http://s7.postimg.org/gu100e1u3/IT8_1.jpg)
(http://s9.postimg.org/vblw09ssf/IT8_2.jpg)
(http://s13.postimg.org/lthrvwspz/IT8_3.jpg)
(http://s8.postimg.org/wgx4of2qb/IT8_4.jpg)
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on June 01, 2015, 02:06:42 am
last unscientific experiment for this night, I switched to other targets

CC Passport (1 shot - not averaged, 1 light = illumination : H3200K+80A, flat fielded), SSF from matlab using all patches from that raw, measured illumination (argyll -H mode), measured target (average of 3 measurements of the same target, argyll -H mode)

single illuminant DCP profile for measured illumination and measured target

converting the same raw with ACR using this profle

http://s14.postimg.org/y9noh3y8x/ccpp_3.jpg  ->
(http://s14.postimg.org/y9noh3y8x/ccpp_3.jpg)

(http://s17.postimg.org/66e0opvm7/ccpp_1.jpg)
.
(http://s11.postimg.org/rzxlk3s2b/ccpp_2.jpg)
.


so calculated SSF are getting better, no ?
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on June 01, 2015, 02:21:15 am
Highly correlated spectra should not hurt DCamProf profiling as it's designed to deal with such cases, but on the other hand it will not help much either. They're handled as averaging a bunch of patches into one, unless you reduce the chromaticity grouping distance to zero then it will try to correct for each patch (and result will not be good).

DCamProf currently makes no use of a neutral step wedge (it sees it just a bunch of highly correlated spectra), as it assumes the target shot is linear. Not sure yet if that's a good idea. Shots with glare issues are certainly not linear, but I've thought that those cannot be well corrected anyway so better require a glare-free shot. Even with a target shot made with proper lighting in a dark room maybe there can be some residual glare than needs correction though, or if the spectrometer measurement suffers from glare one needs to correct the other way around. Not sure though if a simple spline curve lightness correction is relevant.

Interesting to know that 2.5D LUTs are used in repro work too, and yes when the lightness adjustment is spectrally flat is should indeed work out fine. If you like DCamProf can make 2D LUTs too by relaxing the L dimension maximally (say -l 1000,0 to make-profile).
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on June 01, 2015, 03:36:06 am
I've decided to make CAT enabled per default to the next version, as I think most users will expect the look of StdA and D65 differ slightly just like in real life. CATs are far from perfect but from what I've read and some tests I've made it does seem that they are less bad than just doing relighting when it comes to modeling actual appearance.

If you do a D50 profile there will be no CAT of course as it's not needed, and you can still force "color consistent mode" (religthing instead of CAT) which is a good idea if you're making a profile for copy work, or if you just prefer that look. Color consistent means that the profile will make the images look like shot under D50 regardless of calibration illuminant.
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on June 01, 2015, 03:42:11 am
I've decided to make CAT enabled per default to the next version, as I think most users will expect the look of StdA and D65 differ slightly just like in real life. CATs are far from perfect but from what I've read and some tests I've made it does seem that they are less bad than just doing relighting when it comes to modeling actual appearance.

If you do a D50 profile there will be no CAT of course as it's not needed, and you can still force "color consistent mode" (religthing instead of CAT) which is a good idea if you're making a profile for copy work, or if you just prefer that look. Color consistent means that the profile will make the images look like shot under D50 regardless of calibration illuminant.

'd be nice to have a detailed help section on that in the blog.
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on June 01, 2015, 03:43:29 am
'd be nice to have a detailed help section on that in the blog.

Yep I'm adding a new fat section on CAT and relighting in the docs to explain the various aspects around their use.
Title: Re: DCamProf - a new camera profiling tool
Post by: Iliah on June 01, 2015, 09:30:27 am
For normal profiles I use grey step wedge only to check linearity and diagnose flare and glare. This is an important check, and important warning. Flare can't be effectively eliminated based on a tight group of neutral patches as it changes across the target; but border pattern on SG targets combined with the pair composed of the black and the white patches in the middle allows to have a TPS (I actually use 2 TPSs, one additive, and one multiplicative) to compensate for flare.

Practical recommendation was to use polarizing filters to eliminate the effect, but now it is abandoned as proper target design that includes the interleaved black and white patch pattern and math do a better (much better in fact) job.
Title: Re: DCamProf - a new camera profiling tool
Post by: GWGill on June 01, 2015, 09:33:41 am
what I 'd love to have is to use argyll chartread to measure a target and then use txt2ti3 to combine raw rgb data with just spectral data from chartread output... w/o using text editors, etc...
Normally that's all done using scanin. It creates the RGB values from the image and combines it with the reference file. How do you happen to have RGB values not created this way ?
Quote
yes, I understand the origin how txt2ti3 was intended to be used, but
I'm reluctant to clutter txt2ti3 with more file formats to recognize that it needs. It's quite complex and hard to test as it is.
Quote
PS: and if the target measurements were not in 10nm (but for example like -H mode in 3.33x increments), why it is necessary to strip intermediate measurements from the source file and output only SPEC_380, SPEC_390, etc ?
Sorry, I'm not following you. What tool, input is what ?
Title: Re: DCamProf - a new camera profiling tool
Post by: Iliah on June 01, 2015, 09:36:30 am
If you are comparing to CC24 reference, yes, it is getting better; the two sources of error as I see it are still incorrect estimation of red response in the blue part of the spectrum (may be something to do with the pigments in the target you measured) and some vail.
Title: Re: DCamProf - a new camera profiling tool
Post by: GWGill on June 01, 2015, 09:49:42 am
I haven't found any standard name for "relighting transform", some call it "color signal prediction", but in most contexts people say just "CAT", which is confusing as all CATs are designed to predict corresponding colors (that is which tristumulus value to use to produce the same appearance), not predict which tristimulus value the same object will have under a new illuminant. Only if you assume that color constancy is 100% perfect they will be the same, and the current established CATs do not.
I can't agree - there is no reason to assume that being able to spectrally accurately re-light a scene is what human mechanisms of color constancy are striving for.  They are evolved mechanisms that are useful to us, that bear the same resemblance to re-lighting as any 3-channel white balancing algorithms does.
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on June 01, 2015, 09:53:18 am
If you are comparing to CC24 reference, yes, it is getting better; the two sources of error as I see it are still incorrect estimation of red response in the blue part of the spectrum (may be something to do with the pigments in the target you measured) and some vail.
as a side note - making a target directly (not through SSF) is better, but with SSF you can can play around with extra options

here is when profile is built directly =

(http://s3.postimg.org/qwkyb9wyr/cc24p_better_1.jpg)

so

the two sources of error as I see it are still incorrect estimation of red response in the blue part of the spectrum (may be something to do with the pigments in the target you measured) and some vail.

I do have operator's errors indeed, still working out on how to shoot targets, but here (see above) I guess the matlab script and the algorithm of building SSF are to blame mostly, no ?
Title: Re: DCamProf - a new camera profiling tool
Post by: Iliah on June 01, 2015, 09:58:13 am
I can't agree - there is no reason to assume that being able to spectrally accurately re-light a scene is what human mechanisms of color constancy are striving for.  They are evolved mechanisms that are useful to us, that bear the same resemblance to re-lighting as any 3-channel white balancing algorithms does.
The issue of "re-lightning" is mainly for Illum.A and D65 matrices that Adobe are using to estimate colour temperature and tint. Those matrices are not used anymore for calculation and presentation of colour. To estimate the CCT it is better to have spectral calculations when computing those matrices, not CAT.
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on June 01, 2015, 09:58:48 am
TPS

what is TPS ?

as proper target design that includes the interleaved black and white patch pattern and math do a better (much better in fact) job.

but then why not use an extra flat field (flare field) which is printed checkerboard ? and have 3 shots - target (averaged/bracketed), FF, CB
Title: Re: DCamProf - a new camera profiling tool
Post by: GWGill on June 01, 2015, 09:59:34 am
>  haven't found any standard name for "relighting transform", some call it "color signal prediction", but in most contexts people say just "CAT"

Dr. Fairchild was using "spectral adaptation". I use sCAT as an abbreviation.
I don't think that's the same thing though.

As I understand it, "re-lighting" is creating a transform of XYZ/RGB values based on the change in values between spectral reflectance values viewed under two different illuminants.

Marks Fairchild's paper on spectral adaptation seems to be an approach to doing a white point change in spectral space, rather than tri-stimulus space. 

[ I confess that spurred by some of the discussions here, I contemplated a similar idea to that in Marks paper, based on the idea of taking the original illuminant or spectral emission values, dividing by the black body spectrum which has the source white point, and then multiplying by the black body spectrum which has the destination white point. This was based on the idea of a black body spectrum being the model of the "natural" spectral shape change with a change in color temperature. ]
Title: Re: DCamProf - a new camera profiling tool
Post by: Iliah on June 01, 2015, 10:03:42 am
From comparing SSF vs. direct, seems SSF algorithm may need some further improvements; however it may be also something with the measurements, too. The direct transform looks not bad at all (on UK scale, same as "near excellent" in US).
Title: Re: DCamProf - a new camera profiling tool
Post by: Iliah on June 01, 2015, 10:04:50 am
Flare needs to be estimated from the actual scene, as it is scene-dependent. TPS = thin plate spline.
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on June 01, 2015, 10:10:50 am
Normally that's all done using scanin. It creates the RGB values from the image and combines it with the reference file. How do you happen to have RGB values not created this way ?

I am using rawdigger to extract raw values and flatfield them... then I 'd like some automation to combine those with spectral data from chartread to be used further down the /potential/ workflow pipeline

I'm reluctant to clutter txt2ti3 with more file formats to recognize that it needs. It's quite complex and hard to test as it is.

but the issue is not with RD - the issue is that txt2ti3 treats its own (argyll) format differently !!! I have no issues to use txt2ti3 with RD and foreign formats... I want to use txt2ti3 with argyll chartread output... see the irony here ? the RD data is the same in both cases... you already recognize your own format - just make it possible to output the data which txt2ti3 currently does not output


Sorry, I'm not following you. What tool, input is what ?

chartread with -H outputs 3.3nm data... txt2ti3 when used with with such spectral data (mutated from chartead output into foreign format for the sole purpose of txt2ti3 being able to include spectral data in the output when combining  it with raw RGB) includes only 10nm steps... why no option to preserve 3.3x data ? I can certainly just use a text editor, etc... but again it will be easier to use less steps.
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on June 01, 2015, 10:15:47 am
Flare needs to be estimated from the actual scene, as it is scene-dependent.

but are we not estimating/correcting flare for the actual scene of shooting target for profile building - so that it will help with creating a profile too, no ?
Title: Re: DCamProf - a new camera profiling tool
Post by: GWGill on June 01, 2015, 10:16:42 am
Oh, realized it's not the same thing. Fairchild's "spectral adaptation" is CAT based on full spectral data, the human adaptation is still modeled just like in an ordinary CAT. In the paper he uses a "color constant" version as reference though, which is the exact same thing as a relighting transform.
I think you mis-interpret the testing in that paper. The "color constant" version was simply one of the 3 other algorithms the spectral CAT was sanity checked against in a lightweight psychophysical experiment. The following more precise Computational Model Testing used CAT02 as the reference, not the "color constant" computation, although it fared well - but note the caveat about  metamerism, which is the principle reason I don't think it should be the default in camera profiling.
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on June 01, 2015, 10:18:22 am
I can't agree - there is no reason to assume that being able to spectrally accurately re-light a scene is what human mechanisms of color constancy are striving for.  They are evolved mechanisms that are useful to us, that bear the same resemblance to re-lighting as any 3-channel white balancing algorithms does.

I probably wasn't clear, I don't suggest relighting as an alternative to CAT, although I'm interested in it's performance which is lightly touched upon in Fairchild's spectral adaptation paper. So I don't think we are in disagreement.

I don't think it's desirable to use a CAT when relighting is what you want to do. When do I want to do relighting? Say you make a profile for tungsten light, then I need XYZ reference values for tungsten, the values for the observer under this particular light for the particular target. This is calculated from spectra when possible, ie relighting. Then we need D50 XYZ reference values for the PCS, these are then created by applying CAT to those tungsten reference values to keep the color appearance.

That is CAT is used when moving to the PCS, and relighting is used when we need values for the actual calibration illuminant.

In 90% of the use cases, perhaps 95% of ICC profile use cases, the user will make a D50 profile and have a file with D50 reference values, and in this case CAT/relighting is not used.
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on June 01, 2015, 10:21:04 am
I don't think it should be the default in camera profiling.

Color constancy will not be the default in the next release. Default will be relight to calibration illuminant, use CAT to get back to PCS.

There are three scenarios for relighting in DCamProf: 1) recalculation from spectra (ideal), 2) recalculation from simulated spectra (using DCamProf's ability to render a spectrum from an XYZ coordinate), 3) using Bradford CAT. The last is "wrong" as CAT is not relighting, but among the CATs the linear bradford seems to be the least bad at the task, it performs worse than the spectral simulation though.

For the CAT, CAT02 is used.
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on June 01, 2015, 10:22:26 am
From comparing SSF vs. direct, seems SSF algorithm may need some further improvements; however it may be also something with the measurements, too. The direct transform looks not bad at all (on UK scale, same as "near excellent" in US).

and extra data to illustrate direct profiling (w/o going through SSF) results from the prev. post

attched PatchTool compare report :

(http://s16.postimg.org/jyp10k16t/cc24p_better_2.jpg)
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on June 01, 2015, 10:31:32 am
For normal profiles I use grey step wedge only to check linearity and diagnose flare and glare. This is an important check, and important warning. Flare can't be effectively eliminated based on a tight group of neutral patches as it changes across the target; but border pattern on SG targets combined with the pair composed of the black and the white patches in the middle allows to have a TPS (I actually use 2 TPSs, one additive, and one multiplicative) to compensate for flare.

Practical recommendation was to use polarizing filters to eliminate the effect, but now it is abandoned as proper target design that includes the interleaved black and white patch pattern and math do a better (much better in fact) job.

Thanks for sharing the "tricks" I really appreciate it.
Title: Re: DCamProf - a new camera profiling tool
Post by: Iliah on June 01, 2015, 10:39:41 am
If we go a step back for a moment... If we shoot a target 2 times, once with a light source filtered to 5000K, and once with a light source filtered to 6500K, we will have 2 sets of device values that are not in linear proportion to each other. Same, if we take 2 shots under daylight sunny and daylight overcast. The usual point here is that the results should not look the same anyway due to human perception. However it is a general statement, applied backwards, to justify already existing math. Contrary to scanners where the light spectrum is fairly stable, and viewing prints under D50 lights in booths, photography deals with a much more diversity in light sources, and constant complaints on wrong colour is not because photographers and clients are being capricious, but because there is a problem there indeed. This problem stems mostly from the nature of CATs.

For now, the only publicly available way to tame CATs in wrong places is to create ad hoc profiles, that is to profile for the light is the scene (easy in studio, very difficult anywhere on location). What happens when an ad hoc profile is created usually is that the CAT is an organic result of computing a transform between a shot taken under studio lights and target reference being D50/2°. Effectively it is what we here refer to as "re-lightning". But in this case we try to use not "some" lights, but lights as close to D50 as possible - and that helps to keep adaptation to minimum. Try the same with an arbitrary light - and the results are not usable. This alone demonstrates the problem we are having with CATs, "re-lightning", and profiling.
Title: Re: DCamProf - a new camera profiling tool
Post by: Iliah on June 01, 2015, 10:47:17 am
but are we not estimating/correcting flare for the actual scene of shooting target for profile building - so that it will help with creating a profile too, no ?


Sorry, I lost you here... The target is the subject in the shot for profiling, and the flare/glare needs to be estimated from the target and eliminated as much as possible from the target (smoothness here is the paramount, so a spline is a good way to do it; better to have a larger error than to sacrifice smoothness), not from some other object or other scene, as flare and glare are scene-dependent and position-dependent (varies with the scene and depends on the coordinates in the scene), right?
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on June 01, 2015, 10:52:22 am
Interesting indeed, it seems like I may need to revisit the use of CATs once more... I'll let it stay in the way described in the upcoming release though, at least for a few days ;)

How well CATs really work is indeed a bit of a mystery, hard to figure out from just reading papers. From reading, guessing and some limited numerical testing I've assumed that they are quite okay. We'll need to do some practical testing and judgements by eye too I guess. If they mostly produce useless results, I will have to return to color constancy as the default mode.

However if I understand Iliah correctly, you really need to make profiles with D50 lights to get good results, that is it's better to use a D50-designed profile even if it's used under tungsten light?
Title: Re: DCamProf - a new camera profiling tool
Post by: Iliah on June 01, 2015, 10:57:38 am
> In 90% of the use cases, perhaps 95% of ICC profile use cases, the user will make a D50 profile and have a file with D50 reference values, and in this case CAT/relighting is not used.

That's exactly what happens, and that's the wrong way of doing things. If the light in the scene is not accounted for, the transform is going to be wrong.
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on June 01, 2015, 10:59:36 am
Sorry, I lost you here... The target is the subject in the shot for profiling, and the flare/glare needs to be estimated from the target and eliminated as much as possible from the target (smoothness here is the paramount, so a spline is a good way to do it; better to have a larger error than to sacrifice smoothness), not from some other object or other scene, as flare and glare are scene-dependent and position-dependent (varies with the scene and depends on the coordinates in the scene), right?

Iliah, do you know how much of a problem glare is in a proper copy setup in a dark room? The camera outside "the family of angles" etc? Currently I assume that glare is negligible in this situation, but I'm not sure if that really is the case.
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on June 01, 2015, 11:02:33 am
Sorry, I lost you here... The target is the subject in the shot for profiling, and the flare/glare needs to be estimated from the target and eliminated as much as possible from the target (smoothness here is the paramount, so a spline is a good way to do it; better to have a larger error than to sacrifice smoothness), not from some other object or other scene, as flare and glare are scene-dependent and position-dependent (varies with the scene and depends on the coordinates in the scene), right?
yes and what is wrong with replacing b/w patches in the target (for example the one that does not have them in the first place) with b/w patches for a 2nd shot under the same light, camera position, etc... just like you do flatfielding... remove the target, place checkerboard target... advantage of checkerboard target is that you can make smaller patches (for example) and they will cover the whole area of the original target - unlike SG where you have those patches along the edge and in the center
Title: Re: DCamProf - a new camera profiling tool
Post by: Iliah on June 01, 2015, 11:04:49 am
>you really need to make profiles with D50 lights to get good results, that is it's better to use a D50-designed profile even if it's used under tungsten light?

Depends on what we consider acceptable. To create an Illum.A or Studio Tungsten (3200K) profile, I shoot under that light, use target spectral reference measured (or re-calculated) to the light spectrum in the scene, and after that use some (s)CAT to make a profile with D50 WP. Incidentally, Spectrolino measurements are taken with Illum.A light source, and D50 reference is the result of spectral re-calculation. For creating D65 reference, they have a filter attachment.
Title: Re: DCamProf - a new camera profiling tool
Post by: Iliah on June 01, 2015, 11:08:04 am
yes and what is wrong with replacing b/w patches in the target (for example the one that does not have them in the first place) with b/w patches for a 2nd shot under the same light, camera position, etc... just like you do flatfielding... remove the target, place checkerboard target... advantage of checkerboard target is that you can make smaller patches (for example) and they will cover the whole area of the original target - unlike SG where you have those patches along the edge and in the center

I see. The substitution patches need to be of the same size as patches on the target, made on the same substrate, and using the same pigments. There are minor geometry changes involved, like different number and locations of white patches will cause different flare.
Title: Re: DCamProf - a new camera profiling tool
Post by: Iliah on June 01, 2015, 11:13:06 am
Iliah, do you know how much of a problem glare is in a proper copy setup in a dark room? The camera outside "the family of angles" etc? Currently I assume that glare is negligible in this situation, but I'm not sure if that really is the case.

From measurements I took from the shots made in well-equipped studios by seasoned photographers, typically flare is robbing 1/2 of a stop of dynamic range from and SG target, and the toe portion (non-linear shadows, non-linearity being more than 3% allowed by some museum instructions) is about 1 stop long. That poses a problem, as SG target is only 7 stops DR.
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on June 01, 2015, 11:24:27 am
I see. The substitution patches need to be of the same size as patches on the target, made on the same substrate, and using the same pigments. There are minor geometry changes involved, like different number and locations of white patches will cause different flare.

I am sorry - why same pigments for example, you calculate (in your setup) flare corrections using pigments from B/W patches and applying to patches of different pigment composure, no ? so why demand different from checkerboard flarefield ?
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on June 01, 2015, 11:26:42 am
typically flare is robbing 1/2 of a stop of dynamic range from and SG target... That poses a problem, as SG target is only 7 stops DR.

you mean that when you measure SG with a really good instrument (no issues with dark patches) it is 7 stops, you do a shot and from raw rgb is 6.5 stops on the same targer (between black and white patches), right ?
Title: Re: DCamProf - a new camera profiling tool
Post by: Iliah on June 01, 2015, 11:30:54 am
you mean that when you measure SG with a really good instrument (no issues with dark patches) it is 7 stops, you do a shot and from raw rgb is 6.5 stops on the same targer (between black and white patches), right ?
Exactly.
Title: Re: DCamProf - a new camera profiling tool
Post by: Iliah on June 01, 2015, 11:33:06 am
I am sorry - why same pigments for example, you calculate (in your setup) flare corrections using pigments from B/W patches and applying to patches of different pigment composure, no ? so why demand different from checkerboard flarefield ?
Same pigments - same reflection. It is indeed a "field" problem, if one wants to recreate a magnetic field on a flat surface, it is simpler to use the same magnet sizes and same sizes and spacing than to change the distances.
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on June 01, 2015, 11:40:11 am
Same pigments - same reflection.
I see what I am missing - you mean that reflections from other patches shall be present exactly, including color patches that we in my suggestion remove to replace with flarefield, right ? but are those reflections a bigger problem vs everything else - walls, ceiling, floor, etc...  but then those are again reflecting from patches... catch22  >:( !
Title: Re: DCamProf - a new camera profiling tool
Post by: Bart_van_der_Wolf on June 01, 2015, 12:10:00 pm
From measurements I took from the shots made in well-equipped studios by seasoned photographers, typically flare is robbing 1/2 of a stop of dynamic range from and SG target, and the toe portion (non-linear shadows, non-linearity being more than 3% allowed by some museum instructions) is about 1 stop long. That poses a problem, as SG target is only 7 stops DR.

Maybe it's clearer to make a distinction.

Semi-specular surface reflection of of a target is just reflection. This is an issue that has to do with surface smoothness and angle of illumination. Glossy targets have more specular reflecting surfaces, and semi-matte targets have a more Lambertian diffuse reflection. So observing the angle of illumination versus the position of the camera/optical axis can eliminate the reflection of the lightsource better on a glossy target, but it requires a controlled environment (no other bright reflecting or light emitting surfaces that can reflect their presence to the camera lens). Diffuse reflecting surfaces will always add more environment and direct lightsource reflection (desaturation) to the patch reflection.

Reflection inside an optical system can be veiling glare (lack of perfect anti-reflection coating due to angle and wavelength and internal reflection e.g. due to non-blackened edges of lenses or barrel surfaces) which adds a uniform amount of non-image forming light mostly noticeable in the darker tones. Flare is a pattern, a (multicolored) image of the light source as formed by reflections between lens surfaces, and is more local.  This is an optical issue that can also get worse near brighter areas in the scene or image, which suggests that white and black patches of targets are best not positioned directly next to each other.

So surface reflection is target and lighting related, internal lens / instrument reflection or Glare (maybe even flare patterns) is an optical phenomenon.

And then there is electronic noise added to the signal that the sensor (camera or spectrometer) records. The signal itself is already noisy (photon shot noise) which is worse when fewer photons are available (e.g. coming from darker patches). Truely random noise can be reduced by averaging multiple images/captures/readouts.

Cheers,
Bart
Title: Re: DCamProf - a new camera profiling tool
Post by: Iliah on June 01, 2015, 12:51:09 pm
basically yes, I did just some mods - to read CGATS from rawdigger (instead of using dcraw, etc), use my target illumination spectrum and use my target measurements from external files and as camera's database (certainly - we do not know who good were there measurements with monochromator - what it is what it is) is the same - I made eigenvectors constant... gutted also whatever junk is there.

CGATS is not the best option. Switch to CSV to have pixel coordinates too, you will need those for flare/glare compensation splines - tpaps in matlab. And to assess linearrity and field flatness, too.
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on June 01, 2015, 01:02:30 pm
you will need those for flare/glare compensation splines - tpaps in matlab.
haha, I think I need some math ed. before that...
Title: Re: DCamProf - a new camera profiling tool
Post by: Iliah on June 01, 2015, 01:11:16 pm
haha, I think I need some math ed. before that...
I will help you with relevant code if you will need it; but it is so straightforward I doubt you will need any help.
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on June 01, 2015, 03:03:20 pm
I'll add some glare correction feature to the flatfield command. I need to read up a bit on how to best model glare first though. I had already started with some spline stuff, but got some issues concerning spectral flatness etc, which I need to investigate a bit first. Oh it would be nice to have Photo Research spectroradiometer now.
Title: Re: DCamProf - a new camera profiling tool
Post by: Iliah on June 01, 2015, 03:10:45 pm
Just in case, I apply flare/glare removal stage before flat field, and the goal of the flat field is mostly to get uniform white balance across the target.
Title: Re: DCamProf - a new camera profiling tool
Post by: Iliah on June 01, 2015, 03:14:54 pm
> it would be nice to have Photo Research spectroradiometer now

Maybe some University will be interested :)
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on June 01, 2015, 03:17:12 pm
Oh it would be nice to have Photo Research spectroradiometer now.
but are you not working in University or something like this yourself ? some vo... beer and voila...
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on June 02, 2015, 12:49:40 pm
I will help you with relevant code if you will need it; but it is so straightforward I doubt you will need any help.
that you, that shall wait as I still need to polish the basics...

Iliah, a question, as you saw a lot of targets, the difference between QP202 (big one) and QP203 (small one) cards... QP203 looks like patches are painted on a substrate and not that glossy at all and QP202 looks like they are cut and glued and somewhat of lesser quality (like those printed paint samples from any hardware store) and glossy (and substrate glossy too)...  and measurements are quite different (while it seems that I remember from somewhere that you mentioned that spectralwise they are similar - I think from references supplied with makeinputicc)... can you comment on that ?

attached are the measurements :

Title: Re: DCamProf - a new camera profiling tool
Post by: torger on June 02, 2015, 03:03:37 pm
Done a little bit of glare correction coding, and now a little bit of review of test shots. I'm not yet having satisfactory results, it breaks more than it fixes.

In a glossy homemade target shot indoor in a dark room with one StdA lamp outside family of angles, I get 5.57 stops between brightest and darkest color (not black, but super-deep purple, had no blacks in that target) with SSF virtual process (ie glare free) and 5.41 stops in the real shot, that is a 0.16 stop difference. Not too big DR difference, but the darkest patch is 1/3 stop too bright compared to what it should be. Measurement error in the instrument (if it's glare there too) may contribute to a smaller number than it should be.

Anyway, from the initial glare correction code tests it seems like the glare error may be too small compared to the uncertainties in the glare correction model.

Currently the code searches for neutral patches in the target by looking how flat the spectra is, then makes a reference value from provided illuminant spectrum+greenSSF (an "typical" SSF if no SSF is provided), and compares that with the actual value. This serves as input to a spline contrast correction curve. The intermediate points are estimated with TPS. Illuminant mismatch, SSF mismatch, deviation from spectral flatness, distance to nearest control point all contribute to error. However, I suspect the simplistic model of glare, ie just a contrast spline curve, contributes the most error. I'm not satisfied with that model, but haven't come up with anything better.

I've not given up yet though, I have a few things left to test.

I does look tempting though to just skip glare correction and instead relax lightness correction in ultra-saturated violets-purple as that's where the errors seem to be located.
Title: Re: DCamProf - a new camera profiling tool
Post by: Iliah on June 02, 2015, 03:21:08 pm
My glare correction method is based on comparing contrast between the neighboring black and white patches on the shot of the target to the contrast between the same patches in the reference measurement of the target. It is done on all 4 channels independently. The correction coeff. is than placed at the coordinates between the patches, and a spline is constructed. I use 2 splines, one to compensate for the overall veiling glare (bias), uniform additive model; and the second in multiplicative model, to correct for scene-dependent flare and glare - that one is not uniform.
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on June 02, 2015, 04:12:26 pm
the contrast between the same patches in the reference measurement of the target.
and contrast using XYZ is something like log2(Y from white patch / Y from black patch), right ?
Title: Re: DCamProf - a new camera profiling tool
Post by: Iliah on June 02, 2015, 04:57:20 pm
> contrast using XYZ is something like log2(Y from white patch / Y from black patch), right ?
Contrast is Y1/Y2; but if you want it to convert to photographic stops, log2(Y1/Y2) or log2(Y1)-log2(Y2). For counter-flare splines, I use just contrast, not log2
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on June 02, 2015, 04:59:22 pm
> contrast using XYZ is something like log2(Y from white patch / Y from black patch), right ?
Contrast is Y1/Y2; but if you want it to convert to photographic stops, log2(Y1/Y2) or log2(Y1)-log2(Y2).
yes, stops... thank you.

PS: out of curiosity - what the best (min) difference you achieve yourself with your own setup between the contract from measurements and the contract from your shots, before any math. corrections... ? for example I checked my shots and it seems that so far the best (in stops) difference is somewhat around what Torger mentioned (~0.17)... how small you can realistically get by setup (target, light, isolating everyting else that might throw a bad reflection to target/lens, hood, angle of light, etc) before you need to start using corrections with the data ? the mere fact that you are correcting says that you do not achieve zero, do you ? so it is interesting how far that 0.17 is from what you can do in a natural way.
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on June 02, 2015, 05:11:01 pm

attached are the measurements :


it seems I forgot about the troubles with new RAR format - here are zip files in case the issue still persists
Title: Re: DCamProf - a new camera profiling tool
Post by: Iliah on June 02, 2015, 07:34:36 pm
> what the best (min) difference you achieve yourself
 For a 7-stops SG target, lens with concave rear element, same setup:
D4s - slightly less than 0.2 EV reduction
Df - 0.3 EV reduction
D750 - 0.5 EV reduction
lens with convex rear element - approximately +0.1 EV, making D4s 0.3 EV and so on
For an 11-stops target, it starts with 0.9 EV and goes to 1.2 EV
With mirrorless camera, flare is very lens-dependent, but with 100mm Zeiss I can reliably get to 0.1 EV reduction on SG.
MF back on SG - got 0.43 EV reduction.
But if the target is shot out of studio, on location, as you've seen on the other forum, things with flare are not so controlled.
Title: Re: DCamProf - a new camera profiling tool
Post by: Iliah on June 02, 2015, 07:36:10 pm
Simply unrar'ed on Windows. I e-mailed QP guys for an explanation. My pair of 202 and 203 were very close; and the site also gives one reference for both.
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on June 02, 2015, 09:22:02 pm
Simply unrar'ed on Windows. I e-mailed QP guys for an explanation. My pair of 202 and 203 were very close; and the site also gives one reference for both.
my QP203 is 3 years old and my QP202 is just couple month old.... the difference of newer QP202 is very big vs your files (black patch specifically),  it does not look like an operator or instrument error to me (I have one more set of measurements with i1pro from 3 weeks ago and colormunki measurements from 6 weeks ago), like black patch L* = 20.4x (ColorMunki) ... 20.9x (i1Pro2) vs L* = 29.x in your files... just a note.
Title: Re: DCamProf - a new camera profiling tool
Post by: hugowolf on June 02, 2015, 11:17:18 pm
Could be the QP203 hasn't aged well. Has it seen much light?

Brian A
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on June 03, 2015, 12:22:02 am
Could be the QP203 hasn't aged well. Has it seen much light?

Brian A

the issue is with a new (2mo old) QP202, not with QP203.
Title: Re: DCamProf - a new camera profiling tool
Post by: Alexey.Danilchenko on June 03, 2015, 03:02:31 am
the issue is with a new (2mo old) QP202, not with QP203.
Have you got it from them directly or via reseller?
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on June 03, 2015, 09:31:45 am
Have you got it from them directly or via reseller?
QP202 from B&H ... do you think B&H is enterprising and making 'em in their basement ? QP203 was from QPCard directly back in 2012.
Title: Re: DCamProf - a new camera profiling tool
Post by: Alexey.Danilchenko on June 03, 2015, 09:44:42 am
QP202 from B&H ... do you think B&H is enterprising and making 'em in their basement ? QP203 was from QPCard directly back in 2012.
No - was worth a check though. I got my 203 from re-seller in UK and had to replace it - it was a bad batch apparently.
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on June 03, 2015, 10:13:06 am
No - was worth a check though. I got my 203 from re-seller in UK and had to replace it - it was a bad batch apparently.
but yours was small QP203, not big QP202 ?
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on June 03, 2015, 11:29:43 am
Done some more testing on glare and glare reduction algorithms. Core of the problem is that there are many unknowns when modeling glare so it's hard to make accurate results, and the more glare it is the harder it is to make any sane correction.

One of my conclusions is that (semi-)glossy targets produce so much glare when shot outdoors that a glare correction algorithm is bound to fail. You easily halve the dynamic range of the target in diffuse outdoor light. With that kind of input it's really hard to make good results, much better to use a matte target.

So for DCamProf the rule is that if you have a glossy target you must shoot it indoor in controlled conditions. I've done so in attached photo, lit a glossy target with a single light in a dark room, the uneven light corrected with flatfield correction (which can unlike glare be modeled which high accuracy, so really one light is ok). A minor amount of glare can be measured, but comparing the SSF results (glare free) with the target generated profile the differences are so small that my conclusion is that in the context of all inaccuracies there is in camera profiling the glare issue can be ignored. This assumes photographers will be able to follow instructions when it comes to shooting a target, which I guess is not always the case in real life. I'll be adding a "glare detection" log in the make-profile step so it least is noted when suspicious input is given.

(If there is a significant amount of glare this is seen as the SSF profile generating sane and accurate colors while the target profile will make dark patches too dark or if large amount of glare, the whole target too contrasty and saturated.)

Remaining case is matte targets shot outdoor, there I see both visible effects of glare, but fortunately also low enough glare to be able to make a reasonable correction. I shall do a little bit more testing on that.
Title: Re: DCamProf - a new camera profiling tool
Post by: Iliah on June 03, 2015, 11:41:34 am
One of better lighting setups is what Imaging Resource are using for their hVFAI shots of SG target. Checking those shots with and without applying of flare correction, the colour transforms using flare correction are 2 dE2000 more accurate.
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on June 03, 2015, 12:23:55 pm
the colour transforms using flare correction are 2 dE2000 more accurate.
2 ! I certainly need to try to write a code in Matlab then...  at least I can do this during the daylight time  :D
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on June 03, 2015, 02:54:04 pm
One of better lighting setups is what Imaging Resource are using for their hVFAI shots of SG target. Checking those shots with and without applying of flare correction, the colour transforms using flare correction are 2 dE2000 more accurate.

Thanks for the tip, I played around with animaging resource image:
http://www.imaging-resource.com/PRODS/canon-5d-mkiii/E5D3hVFAI000050.CR2.HTM

with the reference data from here: http://www.babelcolor.com/main_level/ColorChecker_content/Digital%20ColorChecker%20SG.txt

Profiling fails miserably, some dark colors pull in opposite directions compared to lighter colors. By excluding the darker, many which are redundant in terms of chromaticity anyway, a good profile is had (sanity-checked comparison with the colorchecker24 result in the same image). Can the dark color errors only be due to glare, or is the reference file bad?

(compared to the reference data it does show a quite large amount of glare, the black patches are ~1.1 stop too bright in the test shot.)
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on June 03, 2015, 03:03:05 pm
Can the dark color errors only be due to glare, or is the reference file bad?
as far as I know BabelColor Patchtool reference file is what was in ProfileMaker... so it is from the same company that makes CCSG... so if they keep the process steady since 2005 it is more or less accurate (within their tolerances) and you can also compare with what Iliah supplied with makeinputicc...
Title: Re: DCamProf - a new camera profiling tool
Post by: Iliah on June 03, 2015, 03:12:35 pm
> Profiling fails miserably
That is exactly so. Now, if you will have a look at their lighting setup, http://www.imaging-resource.com/ARTS/TESTS/HMI.HTM and some more here http://www.imaging-resource.com/ARTS/TESTS/SLMULTI.HTM you will see they do try, and from seeing what others do, Imaging Resource is among the best.

I was looking at their Canon 6D target, and flare mounts to 0.83 EV easily, dark patches (not just black, but all the dark ones, including pure blue and red) are not very usable; and plus they underexpose the targets not looking at the histograms of raw data (though they do have RawDigger and FastRawViewer).

I'm afraid it is close to impossible to make folks shoot targets properly, so we either ignore the patches where flare is about 10% of raw values and higher, or we issue warnings and refuse calculations unless explicit override is specified, or we try to take flare off (you can see it is mostly additive / bias flare, with some scene-dependent flare non-uniformity); or we do all of the above.
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on June 03, 2015, 03:56:10 pm
I just made a little sanity check, I shot a glossy printed paper white+black patch beside eachother, 7.89 stop difference on the Y channel according to the colormunki, and I got 7.28 stop on the G channel with the camera, shot in a dark room. The CCSG has a bit less DR than my baryta paper, about 7.0 stop, so the difference should be smaller there.

So it should be possible to shoot the CCSG target a fair bit better than imaging resource does, I guess they have too much light in the room.

With DCamProf, as the complex command line tool it is, I can live with putting extra requirements on shooting glossy targets.

Concerning target design, for DCamProf's 2.5D LUT making as bright colors for a given chromaticity is the way to go. The CCSG does not have an optimal design in that regard.
Title: Re: DCamProf - a new camera profiling tool
Post by: Iliah on June 03, 2015, 05:14:25 pm
Yes, they could have controlled the light better - and it is of course not about the amount of light per se, but about light missing the target, shielding the lens using proper hoods; cleaning the lenses, closing the viewfinder, etc. But we deal with what we have.

As to the amount of light, here is something to take into account. In studio we need profiles for the light we use for production shots. If it is with flashes, it is the same power as in production shots, or spectrum is different. Same with hot lights. So general additive flare is inevitable in such shots. The larger are the softboxes and the more diffused the light is, the more of the light misses the target and scatters back to the lens. As the actual production shot usually occupies the area larger than the target the flare and glare on the target are not characteristic to the scene. Scene needs different linearization.

SG target is suboptimal, as is CC24 as both are based on pigments that were created for quite a different set of spectral response curves (film), not having deep overlaps (like substantial red channel response to the blue-green range of spectrum) like CFAs do. But those are the only readily available. On top of that, CC24 has nothing useful for flat-fielding, even being matt it is still not quite to the challenge of profiling outside the studio.
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on June 04, 2015, 02:07:13 am
There is a reason why for certain measurements Gretag used to offer polarizing filters with Spectrolino. Light is scattered back from both gloss and textured surfaces, causing flare and glare and making readings from very dark patches less reliable for semigloss and gloss substrates, and glossy black/dark pigments.
i1Pro is cost-optimized not for target measurements, but for printer profiling. IMHO it is only very logical in terms of engineering.

what are the chances that this thing will work = http://www.ebay.com/itm/381275256818 ? $50+ so I took a chance.
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on June 04, 2015, 02:30:00 am
After having compared CC24 results with results of profiling directly against spectral databases I'm not particularly concerned with the spectral shapes of it. Every color can be varied in infinity in terms of spectral shapes, and it seems to me that the soft shapes the CC24 has (and also the CCSG) is a good design that works as well as one can expect. If you make the spectral shape perfectly match one color it will mismatch another. The next level is to design targets with material of what you're going to shoot, which indeed can be feasible for reproduction. Or use SSF and spectral data, but it's then only performing better for that particular set.

If we're really making a reproduction profile (a use case I'm not so much into, I made DCamProf to make generic profiles primarily), then we shouldn't really try to correct glare in the target shot, right? In that case what's important is that the shooting condition stays exactly the same, so any glare in the target shot is the same glare as when we shoot the artwork or whatever. The profile's LUT will correct the glare, if it's a 3D LUT, and it probably should be for this reason.
Title: Re: DCamProf - a new camera profiling tool
Post by: Alexey.Danilchenko on June 04, 2015, 05:56:13 am
what are the chances that this thing will work = http://www.ebay.com/itm/381275256818 ? $50+ so I took a chance.

Lucky you - I missed out on one local recently...
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on June 04, 2015, 06:15:03 am
Experimented more with glare reduction algorithms on the Imaging Resource 5Dmk3 shot of the CCSG. Glare reduction can indeed make significant improvements to the result, but it's still worse compared to the result I get when I simply exclude the dark patches when profiling. So from that result the recommended workflow would be to plot and check for the bad ones, create an exclude list for those and then make a new profile.

There's still a risk that the reference data file is not matching the dark patches very well though which makes the result worse than it otherwise would be. It would be interesting to have a CCSG target and measure it with a spectrometer, and then shoot it with lower amounts of glare.

I'll move on to check closer what glare reduction can do on matte targets on outdoor shots.
Title: Re: DCamProf - a new camera profiling tool
Post by: Bart_van_der_Wolf on June 04, 2015, 07:44:55 am
I'll move on to check closer what glare reduction can do on matte targets on outdoor shots.

Hi Anders,

You can shoot glossy targets out-door if you e.g. put them flat on the ground, but shoot them from a 45 degree angle (squaring the target is then done in software, and the resampling (if even needed) can be simple because we need uniform colored patches, not edge and other fine detail).

Since the glossy surface acts like a mirror, you need to put a sheet of black material perpendicular to the target's surface on the other side, so that only the 'black reflection' is what the camera picks up from the surface, which leaves the patch color itself relatively pure. A glossy target will pick up less ambient light reflections (because of the specular reflecting surface that only mirrors the other side at 45 degrees) than a semi-glossy or diffuse target surface (which picks up a more diffuse image of its surroundings, and thus desaturates/pollutes the colors), even with a black plane on the opposite side of the shooting angle.

These surface reflections (which can be colored from nearby walls or trees) need an additive (or subtractive) neutralization model, where as the optical vignetting and light fall-off needs a multiplicative neutralization model

Cheers,
Bart.
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on June 04, 2015, 08:48:32 am
You can shoot glossy targets out-door

Really interesting technique, but I doubt that it will work well in practice. Even in an ideal indoor situation there is some residual glare issues, and there can be glare issues in the measurement too, those values 6-7 stops down are hard to get precise. Residual issues will be larger outdoor.

Semi-glossy/glossy targets contribute nothing within the normal range of colors, they only increase risk of measurement error. For super-saturated colors they do contribute a little, but I'm quite sure that the residual error in a good outdoor setup will still be large enough to make the profile perform worse on those colors than a profile based on a matte target.

I'm satisfied with the "indoor controlled light pitch-dark room only" recommendation for semi-glossy targets, and then my results indicate that you don't need glare reduction if you do it as good as possible.

When I get back home I'll continue with tests on matte targets and see what conclusions I make from those. I'm not sure I will come to the same conclusions as Iliah, but camera profiling is an "organic" approximate process so it's natural that we can come to different conclusions on how to relate to measurement issues.

My results so far indicate that glare reduction works only in a narrow window. If glare is small, glare reduction only make things slightly worse or no significant improvement, if glare is large results will improve, but results will improve even more if just excluding the dark (=most glare-affected) patches. It seems like matte targets outdoor can be an in-between case where useful results can be had.

If making a commercial easy-to-use software users expect good results even if sub-optimally used. I've seen some bad reviews of the ColorChecker SG which probably are all glare-related. So I do understand the approach of using glare reduction to do the best of what the software get. With DCamProf I have the luxury to say "sorry, junk in, junk out", so I will only include glare reduction if it can provide significant improvements in important use cases. Actually there's only one on the list as I see it -- shooting matte targets outdoor. I'll report back on results.
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on June 04, 2015, 09:06:24 am
Lucky you - I missed out on one local recently...
come on - you have a shiny new monochromator setup now !
Title: Re: DCamProf - a new camera profiling tool
Post by: Iliah on June 04, 2015, 10:03:07 am
> After having compared CC24 results with results of profiling directly against spectral databases I'm not particularly concerned with the spectral shapes of it
In real-world photography the blue-purple shifts as well as red being too orange or too magentish are being a plague. The wrong rendition of reds touches upon yellows and causes problems in skin rendition too. In my experience it is target design issue as well as CAT issue. As I make my own targets, I can see it. One of the practical way to minimize those shifts without making new targets is profiling based on spectral response. Using spectral transforms to the latest possible stage and recording light spectrum in the scene is also one of the things I always considered better suited for profiles for photography.

Semigloss and gloss targets were appearing at the time when profiling libraries were designed with the premises that the wider is the range of colours the more accurate will be the estimation of red, green, and blue chromaticities. That started full force with CC DC, and it turned out not to work well even in studio, so Gretag made a special reference with glossy patches being removed from reference data and ignored in the device data. The other hypothesis, including patches for a LUT-oriented cube, also proved to be wrong. Another trouble is that the large white patch in the middle caused flare on the surrounding patches, and CC DC died giving place to CC SG. From the point of view of profiling, SG differs from Classic only in reflectance. But Classic also has no patches for accurate flatfielding and flare estimation. In all other aspects classic has advantages over SG even in studio. That is why we included flatfielding in RawDigger, to make Classic more usable. I always maintained that the targets need to be matt, and that in studio Classic is more practical than Passport - given one can mount the Classic geometrically flat.

I think diagnostics of flare and glare and eliminating patches with excessive amounts of those is a good working approach. Indeed, GIGO.

Profiling and linearization are different tasks, solved at different stages. I do not rely on profile LUTs for linearization (one of the reasons is that CMMs introduce too much noise and rounding errors when working with small numbers; and may even cause posterization). As to glare, I have photographic ways of reducing it, as well as object-level sampling and processing data accordingly, so in my own work glare is not an issue when it comes to profiles. In any case, flare in the final scene can't be characterized based on the shot of  device-level targets like ColorCheckers being shot for profiling.
Title: Re: DCamProf - a new camera profiling tool
Post by: Iliah on June 04, 2015, 10:04:47 am
what are the chances that this thing will work = http://www.ebay.com/itm/381275256818 ? $50+ so I took a chance.
How could I possibly know the chances? :)
This thing is important:
How do I reset my Spectrolino?

To reset your Spectrolino, hold down the measurement button on the top of the instrument for approximately 10 seconds. During this time you will hear a sequence of four tones, then two tones in rapid succession that are repeated after a pause. Continue holding the measure button until you hear a fifth long tone, indicating that the reset is complete.
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on June 04, 2015, 10:25:03 am
But Classic also has no patches for accurate flatfielding and flare estimation.
can't you DIY greyscale patches around the edges of CC Classic yourself by mounting into some frame ?
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on June 04, 2015, 10:33:11 am
On a different subject, just found this: http://www.liulabs.com/Daylight.htm a D65 simulator that actually seems to work like they should work.

Unfortunately they have no put-in-basket button. I don't know what it costs, but hopefully cheaper than buying an X-Rite Spectralight viewing booth, which also is based on precisely designed filtered halogen lights.

(I'm about to give up on Solux + standard color temperature adjustment filter, the spectrum is not nice enough, and my Solux lamps are too low temperature)

LED-based daylight simulators are the future, but I don't know if there is anything affordable yet. Filtered tungsten seems to be the current best.
Title: Re: DCamProf - a new camera profiling tool
Post by: Iliah on June 04, 2015, 11:26:44 am
> just found this: http://www.liulabs.com/Daylight.htm a D65 simulator
I saw that, but never checked it out.
Booths are adding a lot of flare.
Not sure the precision of design of the filters is that much needed, or, better to say, is placed where it needs to be. Wide and smooth spectrum light source, like a clear sky or xenon flash, normally is enough to reconstruct spectral transmission of a camera with good accuracy.

> Filtered tungsten seems to be the current best.

That's what I'm saying, too.
Title: Re: DCamProf - a new camera profiling tool
Post by: Iliah on June 04, 2015, 11:45:02 am
can't you DIY greyscale patches around the edges of CC Classic yourself by mounting into some frame ?
You can, but you need to maintain thickness and use same surface materials. There were times when Gretag were selling N-type paper in sheets. Now you can cut a Kodak grey card or something similar and make a border around Classic.
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on June 04, 2015, 03:17:30 pm
Made some glare testing of matte targets outdoor.

Typical dynamic range loss seems to be 0.14 - 0.2 stops, reducing range from 4.8 towards 4.6 stops. The loss happens in the darkest patches. In worse cases it may rise to 0.3, and if you have very dark colors in the target you may get a small visible difference. In a CC24 with a 0.14 stop difference the visible difference with glare correction (that is linearization) enabled is to my eyes negligible. I can see on the histogram that something happens to the image when I change profile, but it's very hard to see any changes in colors in the image.

Now when I've coded and integrated the linearization, with TPS and all, I will include it in the next release. But the use case seems very narrow to me. Either you have so little glare that it makes no visible difference, or you have so much glare that it's still bad after linearization, and redoing the shot or excluding problematic patches is a better choice. The inbetween case seems to be when there's been say 0.3-0.6 stop loss in dynamic range.
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on June 04, 2015, 04:24:11 pm
On a different subject, just found this: http://www.liulabs.com/Daylight.htm a D65 simulator that actually seems to work like they should work.

Mailed and asked. You can buy a D65 MR-16 lamp from them directly, 299 USD + shipping and handling. I would have preferred $60 or so, but well, they truly have something unique if it performs to spec, and for that price I expect it to.
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on June 05, 2015, 03:44:07 am
How bad do you think D65 fluorescents are by the way? The best seem to follow the D65 curve quite okay, but with those nasty narrow peaks in there. I don't like fluorescents for that reason, but I don't really know how big a problem those spectral spikes are.
Title: Re: DCamProf - a new camera profiling tool
Post by: Alexey.Danilchenko on June 05, 2015, 03:58:22 am
Mailed and asked. You can buy a D65 MR-16 lamp from them directly, 299 USD + shipping and handling. I would have preferred $60 or so, but well, they truly have something unique if it performs to spec, and for that price I expect it to.

Interestingly, their patent (https://www.google.co.uk/patents/US7378784?dq=7378784&hl=en&sa=X&ei=S1VxVfnyE8imU7aEgMgH&ved=0CCAQ6AEwAA) quoted on the site suggests that this is really just a SPD modifying filter sandwich on top of halogen lamp.
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on June 05, 2015, 04:07:44 am
Interestingly, their patent (https://www.google.co.uk/patents/US7378784?dq=7378784&hl=en&sa=X&ei=S1VxVfnyE8imU7aEgMgH&ved=0CCAQ6AEwAA) quoted on the site suggests that this is really just a SPD modifying filter sandwich on top of halogen lamp.

Yes, the principle has been known for very long time, I haven't read the patent in full so I don't know what's actually new in theirs, but I'm suspecting that the new thing is that the package is small enough so you can just clamp filters directly onto the lamp.

The Spectralight viewing booth use the same technique (but filters are away from the lamps), but liulabs product is the only one I've found where you can buy lamps separate, and 300 bucks is a bargain compared to buying a viewing booth.

The challenge is in designing the filter with a precise response to get a good spectral match. We're paying basically 300 bucks for that filter, I guess the lamp is a standard $3 MR-16.

I've experimented with Solux lamps and standard color balancing filters, and while you easily get 6500K, the spectra is not remotely close to follow a D65 shape.

Halogen can't do UV, so below 400 you don't get much output. For camera profiling this should not be a problem though.

From my experiments the main reason for getting this type of thing would be to profile glossy targets, and for me there's some added value in repeatability when developing the software. Matte targets can be profiled in free outdoor daylight without too much glare issues.
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on June 05, 2015, 08:54:21 am
Just released 0.7.2.

Not too large code additions, most of the work since last release has been research and testing. There's some more stuff in the docs.

CAT is now enabled per default for reasons discussed earlier in this thread. If you prefer color constant behavior for non-D50 profiles use the "-C" flag. Previously "-C" meant enabling CAT but now that's inverted. Before 1.0 is released there may be some of these "turbulent" changes.

Linearization is now available as a part of the testchart-ff command. I won't be using it myself much as in my tests my own results gets best if I minimize glare in the first case and then just ignore any residual glare rather than applying an approximate correction.

make-profile can now write .icc and .dcp directly (then you get default settings), useful for quick trial-and-error loops when testing how the look changes with parameters etc...
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on June 05, 2015, 09:22:21 am
0.7.2 build for Windows (mingw = dcamprof.exe + libgomp_64-1.dll) : https://app.box.com/s/7jsnnhl7orq1nlptk0w9k9qts107ow1f
Title: Re: DCamProf - a new camera profiling tool
Post by: Iliah on June 05, 2015, 09:22:30 am
I have measured some FL lights, including the top of the line ones used in booths. It was not something I would use, but here is what you may want to look at - DPR "new Studio Scene" is shot under D55 Kino Flo bank, raw files available here: http://www.dpreview.com/reviews/image-comparison
Title: Re: DCamProf - a new camera profiling tool
Post by: Iliah on June 05, 2015, 09:22:47 am
Interestingly, their patent (https://www.google.co.uk/patents/US7378784?dq=7378784&hl=en&sa=X&ei=S1VxVfnyE8imU7aEgMgH&ved=0CCAQ6AEwAA) quoted on the site suggests that this is really just a SPD modifying filter sandwich on top of halogen lamp.
Quite so.
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on June 05, 2015, 09:32:20 am
Interestingly, their patent (https://www.google.co.uk/patents/US7378784?dq=7378784&hl=en&sa=X&ei=S1VxVfnyE8imU7aEgMgH&ved=0CCAQ6AEwAA) quoted on the site suggests that this is really just a SPD modifying filter sandwich on top of halogen lamp.
actually it is seen right on the photo of the lamp too...

(http://www.liulabs.com/daylight1.JPG)
Title: Re: DCamProf - a new camera profiling tool
Post by: Alexey.Danilchenko on June 05, 2015, 09:36:30 am
actually it is seen right on the photo of the lamp too...


All you see is light and filter which does not per se lead to any conclusion. Patent reading is quite useful anyway...
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on June 05, 2015, 09:49:39 am
All you see is light and filter which does not per se lead to any conclusion. Patent reading is quite useful anyway...

sure, but they also write clearly right below

"1. Incandescent lamp with colored glass filters,
 2. Xenon arc lamp with colored glass filters, and "

w/o boasting anything special about the lamps, I'd assume manufacturing of the lamps themselves  is a way more difficult (and $$$) industrial-wise process than procuring the right set of precut to the shape glass blanks to assemble them in layered filter for a small shop like dr. Liu
Title: Re: DCamProf - a new camera profiling tool
Post by: Alexey.Danilchenko on June 05, 2015, 09:59:36 am
sure, but they also write clearly right below

"1. Incandescent lamp with colored glass filters,
 2. Xenon arc lamp with colored glass filters, and "

Yes and that gives you loads of useful info ;)

I was not criticising them or anything, just making the point that it is interesting to read how it all works. Evidently its not that interesting for everyone (it was for me :))
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on June 05, 2015, 10:18:19 am
It's very surprising that it hasn't come any LED solutions, other than the superduperexpensive.

I've seen a research paper when they made a good D65 simulator with just 8 LED channels (if you can solder it's probably a quite easy DIY project), you don't really need 20+ like the high end programmable have. But all products I've found are like 3 or 4 channels which means poor spectral properties.

If anyone know about a LED D65 simulator, please post.
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on June 05, 2015, 10:23:06 am
Yes and that gives you loads of useful info ;)

I was not criticising them or anything, just making the point that it is interesting to read how it all works. Evidently its not that interesting for everyone (it was for me :))

so practical outcome for torger is - measure his smooth halogen light, download some databases (spectral data) of filters (organic or glass) and find a combo of 2 that will filter closest enough to D65...
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on June 05, 2015, 10:33:03 am
so practical outcome for torger is - measure his smooth halogen light, download some databases (spectral data) of filters (organic or glass) and find a combo of 2 that will filter closest enough to D65...

I asked Lee filters about that, they have loads of filters, but they didn't manage to come up with a solution that works from their standard products  :-\

Here's a paper showing how to build a D65 simulator based on Lee filters (there is a full text link there on the page):
http://www.degruyter.com/view/j/msr.2011.11.issue-5/v10048-011-0025-y/v10048-011-0025-y.xml

the result is not that good though, about the same as my Solux with an 80B filter
Title: Re: DCamProf - a new camera profiling tool
Post by: Alexey.Danilchenko on June 05, 2015, 10:55:01 am
I've seen a research paper when they made a good D65 simulator with just 8 LED channels (if you can solder it's probably a quite easy DIY project), you don't really need 20+ like the high end programmable have. But all products I've found are like 3 or 4 channels which means poor spectral properties.

If anyone know about a LED D65 simulator, please post.
And that research project didn't have spikes anywhere in a8 channel LED spectrum? Do you have the link to the publication?
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on June 05, 2015, 10:58:59 am
And that research project didn't have spikes anywhere in a8 channel LED spectrum? Do you have the link to the publication?

Desperately trying to find it again, but I haven't found it :-( I'll post if I find it.
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on June 05, 2015, 11:04:39 am
I asked Lee filters about that, they have loads of filters
but you asked about _1_ filter - vs the stack of filters (2-3)...
Title: Re: DCamProf - a new camera profiling tool
Post by: Alexey.Danilchenko on June 05, 2015, 11:14:55 am
Desperately trying to find it again, but I haven't found it :-( I'll post if I find it.

I suppose this is the one where (http://colour.org.uk/Li%202007.pdf) the simulator is cited but I cannot find a free version of the cited publication (the payable article can be obtained here (http://www.ingentaconnect.com/content/ist/cgiv/2006/00002006/00000001/art00049?crawler=true))
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on June 05, 2015, 11:24:12 am
I suppose this is the one where (http://colour.org.uk/Li%202007.pdf) the simulator is cited but I cannot find a free version of the cited publication (the payable article can be obtained here (http://www.ingentaconnect.com/content/ist/cgiv/2006/00002006/00000001/art00049?crawler=true))
may be to write to that author ( ccdcli@leeds.ac.uk ) and ask for a copy ?
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on June 05, 2015, 11:36:21 am
Yes it was that, mixed up with this: http://file.scirp.org/Html/14-7600116_16626.htm
Title: Re: DCamProf - a new camera profiling tool
Post by: Iliah on June 05, 2015, 11:48:35 am
Look at second sheet here http://www.graham.auld.me.uk/406/Calculating%20CRI-CAM02UCS-v2.xls
Title: Re: DCamProf - a new camera profiling tool
Post by: nertog on June 05, 2015, 03:20:50 pm
*Disclaimer: the product in the link below is the result of a research project I work on...so I am slightly biased.*

Here you go: http://ledmotive.com/

10 LED channels, fully programmable and with spectral feedback. Although the website shows more commercial, general illumination applications, we use the light engines in several scientific studies. Multichannel LED light sources are rarely cheap as the electronics to drive the LEDs and especially the optics to ensure an homogeneous beam profile are quite advanced.

An alternative I use myself in a homemade viewing booth are spotlights with UV-pumped white LEDs. Their spectrum is smooth and continues without the pronounced blue spike of standard white LEDs.

Keep up the good work!
Wim
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on June 05, 2015, 04:03:02 pm
Thanks for the interesting link.

Indeed I was suspecting that it may be quite expensive to make a multichannel LED, more than those 300 bucks for a filtered halogen.

Could you provide a link to a UV-pumped white LED? I have never heard of them before.

Surfed a bit, the principle seems to be to have a UV led (producing invisible ultraviolet light) and then put in some phosphors there that absorbs the UV an re-emit at a smoother wider spectrum. This links shows what to expect:
http://www.open-photonics.com/featured-technologies/high-cri-leds

Better than fluorescents it seems, but we can't have D65 I guess?
Title: Re: DCamProf - a new camera profiling tool
Post by: nertog on June 05, 2015, 04:24:55 pm
http://yujiintl.com/high-cri-led-lighting/?gclid=CM6ut6-v-cUCFTHHtAodoGkAyw

Filtered halogens could be more accurate, but the MR16 halogen replacement I tested offered a much higher illuminance and better beam profile than Solux (not to mention they consumed 10x less power). Their mid-power SMD LEDs are excellent!
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on June 05, 2015, 04:30:43 pm
http://yujiintl.com/high-cri-led-lighting/?gclid=CM6ut6-v-cUCFTHHtAodoGkAyw

Filtered halogens could be more accurate, but the MR16 halogen replacement I tested offered a much higher illuminance and better beam profile than Solux (not to mention they consumed 10x less power). Their mid-power SMD LEDs are excellent!

Cool, I had just surfed up that exact site and was about to post a link :). A package for 3 5600K lamps is $125 though, so it's not for free. Cheaper than Liulabs $300 D65 simulator though.

(For the application of camera profiling I think a directional spot is a good thing, easier to get nothing but the target lit to minimize any glare issues.)
Title: Re: DCamProf - a new camera profiling tool
Post by: nertog on June 05, 2015, 04:53:51 pm
Hmm, I bought a couple of MR16 spots from them about a year ago. Maybe worth asking if they still have them?
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on June 05, 2015, 05:16:39 pm
These guys also have this kind of thing, http://high-cri-led.com/high-cri-led/led-lighting/high-cri-mr16.php
their latest product brochure was released today, and it has some 6500K products in it, on the web I find only 5600K max though.
Title: Re: DCamProf - a new camera profiling tool
Post by: nertog on June 05, 2015, 05:52:50 pm
Interesting! I will contact them and see if I can get some more info. They seem to use the standard blue pump + phosphors to achieve a high CRI. This means the spectrum most likely has a large blue spike at 450-460nm...not something you's want. Let's see.
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on June 06, 2015, 10:33:54 am
Experimented with some outdoor shooting techniques to minimize glare and flare issues.

Setup as shown in attached image; cut out the side of a cardboard box, cover it with black cloth, and put the glossy target at the bottom. Then shoot the target at an angle from the side, so all reflections is supposed to come from the blackness of the box. You need perspective correction to make the target scannable, and you also need to use the flat-field correction function to even out the light.

I have also lengthened the lens shade. The standard lens shade is designed to not vignette on the largest aperture, so when you shot at f/8 you can have a considerably longer lens shade and reduce any flare. The viewfinder is made light tight (important!). I'm using a 85mm lens as it has few lens elements which is good concerning flare.

I'm not shooting a real target here, just a glossy printer target which has both black and white patches so I can test dynamic range.

7.89 stops: spectrometer reference measurement
7.28 stops: target shot in a dark room with a tungsten light (note: not same lens used here, possibly a little bit better result can be had)
7.19 stops: outdoor with the shown setup using polarization filter
7.15 stops: outdoor with the shown setup using no polarization filter.
6.7 stops: outdoor as shown but without the cardboard box
6.3 stops: outdoor with the shown setup, being sloppy with the viewfinder

Polarization filter does not seem to make any significant difference in this type of setup, so it's probably best not using it as I think they're not fully spectrally flat.

It seems like it may be possible to shoot semi-glossy targets outdoor and get good results using this type of technique, if you're careful when setting it up. At some point I'll test it in full.
Title: Re: DCamProf - a new camera profiling tool
Post by: ah693973 on June 06, 2015, 11:04:31 am
All,

I have followed this discussion from its start and am constantly amazed at what the internet allows a few motivated people to do. I just love these kinds of globehopping collaborative projects.

In reading the latter part of the thread and the problems with flare and glare, a question keeps popping into my head; Why do you need to measure a paper target? Why not take a variable light source like the LEDMOTIVE or a homemade alternative (RGB+ LEDs and a ColorMunki?) point it at the camera and take photos as you vary the light? I guess it would look more like a version of display profiling. On that thought, maybe instead of LEDs it could even use a display for the source?

Andy
Title: Re: DCamProf - a new camera profiling tool
Post by: Iliah on June 06, 2015, 11:17:44 am
There are two ways that are currently in use; one is to use bandpass filters over a halogen or xenon stabilized light; the other is to use a target made of gel filters or an IT8 / HCT (http://hutchcolor.com/) target recorded on a slide film and positioned and spectrally measured right on the lighttable (a more stable variation of the same is to use single filters mounted into slide frames in a multiple shot technique, lighttable or slide projector). Light source must have smooth spectrum, and I have not yet seen LEDs that provide such spectrum.
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on June 06, 2015, 11:32:29 am
All,

I have followed this discussion from its start and am constantly amazed at what the internet allows a few motivated people to do. I just love these kinds of globehopping collaborative projects.

In reading the latter part of the thread and the problems with flare and glare, a question keeps popping into my head; Why do you need to measure a paper target? Why not take a variable light source like the LEDMOTIVE or a homemade alternative (RGB+ LEDs and a ColorMunki?) point it at the camera and take photos as you vary the light? I guess it would look more like a version of display profiling. On that thought, maybe instead of LEDs it could even use a display for the source?

It's probably harder and more expensive than we'd like to make a home-made programmable multi-channel LED light-source. I'm all for such a project though, but I'm not the right person to do it.

Targets should present relevant types of spectra. Homemade inkjet printed targets have some limitations discussed earlier, X-rite's targets have better spectra, but I think you can use a inkjet super-saturated target as a complement to X-Rite's matte CC24 for example. If you do copy work, such as copy paintings, making an own target by painting patches with the same colors as used in the work you copy will be a good idea.

The core of the problem is that cameras are not colorimetric devices, their SSF do not match the standard observer CMF, that is they register colors in a significantly different way than humans. This means that to make a profile that works as good as it can, we need to make it for the exact reflectance spectra that will be in the scenes we shoot. Unless we do copy work this is not really possible, and then we instead need to do it for some average spectra, then smooth rounded spectra are much better than spiky spectra as it's more similar to natural colors and works smoother over a wider range of similar colors.

Unfortunately displays are very limited in terms of spectra, it's just a red, green, and blue peak which you can mix. This can recreate tristimulus values that work well for a human, but as the camera sees colors otherwise those spectra will not make up for good profiling colors.

It's the same reason why we're so picky about the artificial light sources. The mismatch between human and camera color response makes spiky spectra unpredictable. If you have photographed in low CRI light sometime you have probably noticed that the colors come out quite different from the camera than the eye saw them, and this is a demonstration of the effect of having different color response. The spikier spectra the more the differences are seen. Therefore we want to do camera profiling with full spectrum lightsources with smooth spectra, using colors with smooth wide reflectance spectra.

Using transmissive targets is interesting though. In Image Engineering's CamSpecs product they use transmissive targets with several narrow bands to measure the camera's SSF. And I think if one is going to make some elaborate DIY project with hardware stuff, I think one should try to look into measuring SSF.
Title: Re: DCamProf - a new camera profiling tool
Post by: Iliah on June 06, 2015, 11:52:37 am
> Using transmissive targets is interesting though
And much simpler than one may think. Bonus - they can be measured as mounted on the light, right before shooting, with a German-made, half-the-PR-price JETI device http://www.jeti.com/cms/index.php/instruments-55/radiometer/specbos-1201 ; and additional focus optics is available if the patches are less than 4mm. This makes cheap gel filters very practical, and allows to virtually eliminate scene flare.
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on June 06, 2015, 12:20:14 pm
half-the-PR-price JETI device http://www.jeti.com/cms/index.php/instruments-55/radiometer/specbos-1201 ;
that is still several $Ks...
Title: Re: DCamProf - a new camera profiling tool
Post by: Iliah on June 06, 2015, 12:23:00 pm
> that is still several $Ks...
Yes, but it has no useless bells and whistles; while for a regular user i1Pro is ok in transmissive setups.
Title: Re: DCamProf - a new camera profiling tool
Post by: ah693973 on June 06, 2015, 12:42:21 pm
I have been involved in modifying a drum scanner (Howtek) in a project over at the large format forum. I started out alone with better PMTs and amps and then worked with others who generously helped to fix some of the issues that came up that were out of my "comfort zone". One of the guys I was working with is using a RGB led as his source now instead of the halogen lamp that was originally used. He is reporting good results and is technically sharp enough that I believe him.

Of course, a bayer array and a drum scanner really work on the same principle, its just that one is two dimensional and one is a point. This is what led me to believe that a more direct approach might work.

Andy
Title: Re: DCamProf - a new camera profiling tool
Post by: Iliah on June 06, 2015, 12:45:42 pm
The difference is that film spectral transmission allows for relatively narrow spectrum light sources (see colour separation filters), while the response of a camera sensor does not.
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on June 06, 2015, 12:47:38 pm
> that is still several $Ks...
Yes, but it has no useless bells and whistles; while for a regular user i1Pro is ok in transmissive setups.
so Torger can still find a use for his solux lamp by sticking gel filters along with a filter to raise CCT up on top of it and using his colormunki to measure the light directly and then shooting it directly with a camera in M mode to keep exposure the same... or alternatively find a way to mount a gel in front of styrofoam and measure the illuminated styrofoam (tele mode with colormunki) and then shoot it too, gel by gel... flare, etc will not matter much there too... now you just need to find a minimal set of gels to get results
Title: Re: DCamProf - a new camera profiling tool
Post by: Alexey.Danilchenko on June 06, 2015, 03:05:42 pm

Using transmissive targets is interesting though. In Image Engineering's CamSpecs product they use transmissive targets with several narrow bands to measure the camera's SSF. And I think if one is going to make some elaborate DIY project with hardware stuff, I think one should try to look into measuring SSF.

It would be less expensive to build proper monochromator setup to do that. I have completed mine (it needs to be calibrated) and overall it cost less than the 6500K light you were looking for. Used monochromator (from manufacturing like Luxtron/Lumasense/Xinix - like this one for example (http://www.ebay.com/itm/Monochromator-XINIX-Lumasense-Luxtron-Model-1108-Microscopy-Spectography-Testing-/301592903145?pt=LH_DefaultDomain_0&rmvSB=true&nma=true&si=W7DyBAIXW7srNNGZOow9wjIiayI%253D&orig_cvip=true&rt=nc&_trksid=p2047675.l2557)), used halogen light source with fiber optic feed (plenty on eBay) and used integrating sphere (even a simple one like this (http://www.ebay.com/itm/PHOTONIS-XP3112-PB-TUBE-HOUSING-UV-filter-integration-sphere-/271166072389?pt=LH_DefaultDomain_0&rmvSB=true) will do). Also a way to measure illumination level in a sphere - arduino + photodiode + 30mins of coding. All of that put together on a bench in a dark room gives a decent enough setup to measure SSF.
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on June 06, 2015, 03:21:50 pm
It would be less expensive to build proper monochromator setup to do that.

That's super-cool, I'd love to see some photos when you have it all working!
Title: Re: DCamProf - a new camera profiling tool
Post by: Iliah on June 06, 2015, 03:44:37 pm
> It would be less expensive to build proper monochromator setup to do that.

I doubt "less expensive" part (unless we are chasing D65, which even an advanced user will not be doing, and which is unnecessary as obtaining spectral response is a much better way to do things), but much more precise, much more stable, and much more convenient if a lot of cameras and lenses are to be characterized.

Light spheres are easy DIY project.
Title: Re: DCamProf - a new camera profiling tool
Post by: Alexey.Danilchenko on June 06, 2015, 03:45:04 pm
That's super-cool, I'd love to see some photos when you have it all working!

Sure - I'll make the setup more permanent first and will do a calibration run (with spectrophotometer and photodiode measurements to make a correlation table). I have not got much spare time lately so it all goes a bit slow at the moment. At the beginning of the week I completed light source and put it all together on a table in a dark room for a quick run - walked through the spectrum with i1Pro measuring every 10nm (thanks Graeme for Android version to make it really convenient with the phone) and  all seems to work fine.

Title: Re: DCamProf - a new camera profiling tool
Post by: Alexey.Danilchenko on June 06, 2015, 03:49:49 pm
> It would be less expensive to build proper monochromator setup to do that.

I doubt "less expensive" part (unless we are chasing D65, which even an advanced user will not be doing, and which is unnecessary as obtaining spectral response is a much better way to do things), but much more precise, much more stable, and much more convenient if a lot of cameras and lenses are to be characterized.

I meant less expensive when constructed from used parts off eBay - getting all parts shiny new is prohibitively expensive for home enthusiast. My setup was half the price of new Colorchecker SG in total.

Light spheres are easy DIY project.

I did construct a sphere using paint and was about to attempt coating it with barium sulfate when a cheap one came along made with spectralon.
Title: Re: DCamProf - a new camera profiling tool
Post by: Iliah on June 06, 2015, 03:52:04 pm
>My setup was half the price of new Colorchecker SG in total.
That is what I would expect :) But wait for calibration before celebrating ;)
All in all, you know I was suggesting this setup for quite some time.
Title: Re: DCamProf - a new camera profiling tool
Post by: Alexey.Danilchenko on June 06, 2015, 03:53:08 pm
>My setup was half the price of new Colorchecker SG in total.
That is what I would expect :) But wait for calibration before celebrating ;)
All in all, you know I was suggesting this setup for quite some time.

I am far from celebrating - the real experiments are only just beginning...
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on June 06, 2015, 04:11:03 pm
As many seems to be reading this thread it may be worth mentioning that spending lots of money to get the perfect D65 simulator "just because" it's not a wise thing to do.

When you make dual-illuminant DNG profiles for general use the defacto standard (=Adobe standard) is indeed to make them for the StdA and D65 pair. This means that in order for the EXIF lightsource tags to match the truth you need to profile under those illuminants. However, they don't really need to match. The only thing they're used for is when deciding the matrix mix, ie if the temp is 4700K 50% of the 2850K StdA matrix will be mixed with 50% of the 6500K D65 matrix. To know the mix you need to know the temperature and to now the temperature you need to know the mixed matrix, a hen and egg problem solved by an iterative loop. But only after mixing the CM is used to derive the actual temperature/tint that is shown in the white balance tool.

To summarize, you can mismatch the actual calibration illuminant and its tag with quite wide margin before it will have any real effect. Therefore it's perfectly feasible to use real daylight as "D65" even if you make a dual-illuminant profile. (On single illuminant profiles the EXIF lightsource tag has no effect at all. ICC profiles are always single illuminant.)

Having a simulator adds convenience though, especially if you make many profiles or experiment a lot. It would not need to be a D65 simulator though, you could have some other average daylight.

It's still some advantage to *know* what light that has been used as calibration illuminant, if not the spectra at least the temperature. This is needed to make the DNG profile estimate light temperatures properly. (The CAT is also affected by that, but you can of course enable color constant behavior)

I have a hunch that for dual-illuminant profiles it's an advantage to get a bit above 5000K to get some more distance from StdA. At D65 we have very limited options on artifical light sources with nice spectral properties though. The Liulabs expensive filtered halogen does not have much competition. I'm curious about the UV-pumped white LEDs though, seems like Yuji is about to release one. With those it seems to be that the higher temperature the worse spectrum, so I'm not so sure the 6500K will look that good, but probably better then a Solux with 80B filter.
Title: Re: DCamProf - a new camera profiling tool
Post by: Iliah on June 06, 2015, 04:12:17 pm
> I am far from celebrating
Not that far...
> - the real experiments are only just beginning...
That never ends ;)
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on June 07, 2015, 12:22:19 am
> contrast using XYZ is something like log2(Y from white patch / Y from black patch), right ?
Contrast is Y1/Y2; but if you want it to convert to photographic stops, log2(Y1/Y2) or log2(Y1)-log2(Y2). For counter-flare splines, I use just contrast, not log2
I have a question... I flatfielded rawdigger samples from a CC24PP shot and I am getting the following (no WB, no scaling, no gamma/gamma = 1) :


19   D1   5442.64    13193.07    8222.41
20   D2   3414.98    8434.33     5383.07
21   D3   2123.34    5291.81     3394.74
22   D4   1127.10    2778.43     1783.99
23   D5   515.34     1304.47     855.50
24   D6   179.69     439.28      287.96


so contrast is (in stops) :

R = log2( 5442.64 / 179.69 ) = 4.920724585
G = log2( 13193.07 / 439.28 ) = 4.908495689
B = log2( 8222.41 / 287.96 ) = 4.835620983



CC24PP spectral data converted to XYZ / Y :

D6 = 3.128
D1 = 92.588

log2 (92.588/3.128) = 4.887512805

comparing with flatfielded RD it is too good to be true, so where is the error might be then ?

FFD data :


19   D1   0.997140   0.999829   0.999334   0.998824
20   D2   0.997535   0.999202   0.999851   0.999649
21   D3   0.997811   1.000000   0.999594   1.000000
22   D4   1.000000   0.999567   1.000000   0.999630
23   D5   0.994782   0.993781   0.993994   0.996559
24   D6   0.975756   0.977245   0.975554   0.977036


unflatfielded samples


19   D1   5427.07   13184.19   8216.93
20   D2   3406.56   8429.48   5382.27
21   D3   2118.70   5291.81   3393.36
22   D4   1127.10   2777.32   1783.99
23   D5   512.65   1298.17   850.37
24   D6   175.34   429.24   280.92



scene is :

(http://s17.postimg.org/xl20jn5wf/DSC06299_OK.jpg)
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on June 07, 2015, 04:46:42 am
A quick question to any Yuji high CRI LED owners out there, it would be nice if you could post a spectrum measurement, for example of the 5600K. The "marketing" spectrums shown on web sites often make things look better than they really are...
Title: Re: DCamProf - a new camera profiling tool
Post by: Iliah on June 07, 2015, 10:47:33 am
Frankly, I do not see much marketing looking at the 5600K SPD on page 7 of http://www.yujiintl.com/files/high-cri-led-datasheets/VX5730-YUJI.pdf
It is not pretty for the purpose of D-series simulation for profiling. Visual comparison with a D-series SPD is not very telling, it is close to what Dr.Fairchild calls "A Metric for the Aesthetic". Kino-Flo Celeb lights are made of Yuji LEDs. An independent and reputable study of different light sources (including Celebs http://www.gtc.org.uk/media/fm/TLCI_charts_April_2015/Kino_Flo.zip ) by Alan Roberts (former BBC research engineer) is at http://www.gtc.org.uk/tlci-results/tlci-results-new-format.aspx (SPDs are presented as TIFFs, you need to download files to see those).
Title: Re: DCamProf - a new camera profiling tool
Post by: Iliah on June 07, 2015, 10:57:02 am
CC Classic was designed to cover 5 stops, flare and glare at this dynamic range is negligible. Try underexposing it by 2-3 stops and see how the error increases.
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on June 07, 2015, 11:54:00 am
I've discovered some silly bugs in v0.7.2, CAT is not run when it should and opposite, and flatfield not working properly. I will soon release a patch to fix those. I'll do a little bit more testing first.
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on June 07, 2015, 11:59:00 am
CC Classic was designed to cover 5 stops, flare and glare at this dynamic range is negligible. Try underexposing it by 2-3 stops and see how the error increases.
but it did not much, took EV-3 (vs the raw in the post above) exposed raw from the same series and

19   D1   673.49   1642.2   1017
20   D2   423.69   1044.29   666.82
21   D3   263.83   654.65   421.1
22   D4   140.29   344.66   221.82
23   D5   64.214   163.55   107.25
24   D6   22.477   55.471   36.186


R = log2( 673.49 / 22.477 ) = 4.905135127 vs 4.920724585  
G = log2( 1642.2 / 55.471 ) = 4.887752297 vs 4.908495689
B = log2( 1017 / 36.186 ) = 4.812744228 vs 4.835620983

drop by 0.02

and still strange vs measured target log2 (92.588/3.128) = 4.887512805

Title: Re: DCamProf - a new camera profiling tool
Post by: Iliah on June 07, 2015, 12:09:54 pm
but it did not much, took EV-3 (vs the raw in the post above) exposed raw from the same series and

19   D1   673.49   1642.2   1017
20   D2   423.69   1044.29   666.82
21   D3   263.83   654.65   421.1
22   D4   140.29   344.66   221.82
23   D5   64.214   163.55   107.25
24   D6   22.477   55.471   36.186


R = log2( 673.49 / 22.477 ) = 4.905135127 vs 4.920724585  
G = log2( 1642.2 / 55.471 ) = 4.887752297 vs 4.908495689
B = log2( 1017 / 36.186 ) = 4.812744228 vs 4.835620983

drop by 0.02

and still strange vs measured target log2 (92.588/3.128) = 4.887512805



Nope, not that way ;)

8 EV - log2(13193.07/55.471) for G - about 0.11 EV

Always disliked the practice of shooting targets on complex backgrounds instead of dark grey and black; having foreign objects in the scene (even if they are not showing in the frame); semi-gloss or gloss targets (SG has 5.5 stops between the "same" 2 patches, and that already breaks IR shots), shallow hoods or no hoods...
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on June 07, 2015, 12:40:05 pm
Nope, not that way ;)

8 EV - log2(13193.07/55.471) for G - about 0.11 EV

that is if you are using your approach of bracketing shots and combining target measurements with data (spectral or otherwise) adjusted to match underexposed raws... than naturally means using the data from D1 from the EV0 and the data for D6 from EV-3...


then measured target will be (using Y from spectral data divided by 2^3) log2( 92.588 / 0.391 ) = 7.887512805 and then is not 0.11EV, but ~zero, no ? unless you state that i1pro2 can't even measure ColorChecker Passport black patch (not CCSG) correctly...
Title: Re: DCamProf - a new camera profiling tool
Post by: Iliah on June 07, 2015, 01:48:28 pm
> then measured target will be (using Y from spectral data divided by 2^3) log2( 92.588 / 0.391 )

I disagree with 0.391. Measured Passport with my i1Pro, Y on black patch is 0.31
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on June 07, 2015, 02:03:28 pm
> then measured target will be (using Y from spectral data divided by 2^3) log2( 92.588 / 0.391 )

I disagree with 0.391. Measured Passport with my i1Pro, Y on black patch is 0.31

you mean you disagree not with Y = 0.391, but with Y = 3.128 from measurements ( 0.391 was derived from Y = 3.128 when I divided it by 8 = 2^3 stops to match EV-3 exposure )... well , I averaged 3 measurements a week ago (attached) - but I will repeat it again.
Title: Re: DCamProf - a new camera profiling tool
Post by: Iliah on June 07, 2015, 02:12:59 pm
I have Y = 94.79 for the white, Y = 3.1 for black, log2(94.79/3.1)+3 = 7.9344
However, in practical photography any variation below 1/6 EV (0.17 EV) is considered irrelevant. The best shutters are guaranteed to be within 1/12 EV, apertures within 1/10 EV accuracy and repeatability.
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on June 07, 2015, 02:22:12 pm
The best shutters are guaranteed to be within 1/12 EV
by the way - it sounds logical that exposure time component of exposure shall be more precise with longer exposures, no ?

Title: Re: DCamProf - a new camera profiling tool
Post by: torger on June 07, 2015, 04:45:53 pm
Just released a quick update 0.7.3 out to fix some bad bugs introduced in 0.7.2
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on June 07, 2015, 06:52:49 pm
0.7.3 build for Windows (mingw = dcamprof.exe + libgomp_64-1.dll) : https://app.box.com/s/bh3zagil9imxff5h3voxozgtgcwzmupa
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on June 08, 2015, 01:46:28 am
Been working with some elaborate workflows, seems to be one feature left to do before I can get profiles that look as they should; somehow one need to take the film curve into account. The commercial profiles do, and even some matrix-only profiles seems to be deliberately desaturated to not oversaturate when a film curve is applied. On the other hand it may just be a side-effect of using only matte targets when profiling which typically seems to lead to a slightly under-saturated profile.

This has nothing to do with reproduction style profiles which are designed to look accurate with no film curve applied. Not sure if it's suitable to rely on standard color science models at all in this case.

I think one can let highlights desaturate with the lowered S-curve contrast, but maybe one want to keep saturation in shadows, if so I need to introduce a 3D look LUT. Possibly it will look good if just compensating for the increased midtone contrast all over, that's probably what I will try first.

Adobe has a slightly different curve formula than a pure RGB curve, may need to take that into account too. If we only look at linear midtone contrast we won't though as it then is no different from a pure RGB curve.
Title: Re: DCamProf - a new camera profiling tool
Post by: nertog on June 08, 2015, 04:32:11 am
Some measured spectra of the Yuji LEDs and LEDMOTIVE multichannel source:

1. High CRI blue-pumped white LED (CCT = 5800K)

(http://s3.postimg.org/6fdv475dv/Blue.jpg) (http://postimage.org/)

2. Violet-pumped white LED (CCT = 5000K)

(http://s3.postimg.org/4q4rw4poj/Violet.jpg) (http://postimage.org/)

3. Violet-pumped LED, different phosphors (CCT = 5600K)

(http://s3.postimg.org/i5rslky6b/MR16.jpg) (http://postimage.org/)

4. LEDMOTIVE source emulating D65 spectrum (CCT = 6504K)

(http://s3.postimg.org/t2t4ackxv/Ledmotive_D65.jpg) (http://postimage.org/)


Hope this helps!
Wim
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on June 08, 2015, 04:57:39 am
Thanks! They're not really UV-pumped it seems, but on the edge of the visible range, where both the eye and a camera has some sensitivity left. Not sure how much it will disturb though.
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on June 08, 2015, 10:22:22 am
manual has a small typo error - here is a piece of text :

Quote


Here follows a description of each command available.
make-target

  dcamprof make-profile <flags, with inputs> <output.ti3>

Make a target file which contains raw camera RGB values paired with reference XYZ values, and (optionally) spectral reflectance. The file format is Argyll's .ti3, with some DCamProf extensions.
Title: Re: DCamProf - a new camera profiling tool
Post by: Iliah on June 08, 2015, 12:26:28 pm
> They're not really UV-pumped it seems, but on the edge of the visible range, where both the eye and a camera has some sensitivity left. Not sure how much it will disturb though.

If it does not cause response in the red channel it does not matter much, not adding to metameric failures. But at least with Nikon cameras based on Toshiba sensors it does.
Title: Re: DCamProf - a new camera profiling tool
Post by: GWGill on June 08, 2015, 07:31:23 pm
Thanks! They're not really UV-pumped it seems, but on the edge of the visible range, where both the eye and a camera has some sensitivity left.
Might make for more fluorescent whitener response than a "real" light source.
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on June 09, 2015, 01:38:52 am
Experimented with some different methods to adjust saturation when using the profile with a film curve. Adjusting chroma of the XYZ reference values via CIECAM02 Jab/JCh seems to work well. JCh is more linear concerning hue (that is less problems of changing hue when chroma is scaled) and overall more perceptually uniform than Lab/LCh, and a suitable adjustment seems to be in the 1-2 units range, that is very small, so hues stay stable enough. I was thinking of making a Munsell corrected space (like Bruce Lindbloom's UP Lab http://www.brucelindbloom.com/index.html?UPLab.html), but as adjustments are so small and often with more saturated colors than exist in Munsell's book of colors it's not worthwhile, and as said Jab is already a bit better than Lab.

Instead of adjusting the XYZ reference values the other alternative is to apply the curve on the RGB values, but then matrix fitting becomes badly nonlinear and I don't think the result is better. Just like the curve itself this type of adjustment is more about look-and-feel than accuracy.

I'll include possibility to adjust chroma it in the next release, and voilà DCamProf has its first subjective look-and-feel adjustment.

This is a good read regarding perceptually uniformity of various color models by the way: http://www.handprint.com/HP/WCL/color7.html
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on June 09, 2015, 08:09:32 am
any ideas how to come up with some good heuristic to speed up (or rather try in a proper directions) trying possible combinations within 28 measured SSFs to "calculate" the best "virtual" SSF for my camera ? a simple brute force is too slow because calling dcamprof executable hundreds of thousands times (to test how SSF profile generated by Matlab matches /dcamprof test-profile/ the real measured and shot target) is not an option...

so far brute force (I can only try all combo up to 5 cameras out of 28 on my notebook in several hours) gives me the around the following match :

(http://s11.postimg.org/dt49bqnn7/match.jpg)

not bad, but I want better.

dcamprof test-profile ("dcamprof test-profile -i ill.sp -C -B ccpp.ti3 ssf-profile.json", where ccpp.ti3 has real/actual raw RGB and real/actual measurements) gives for this profile :

Quote
Native LUT patch match
average DE 0.67, DE LCh 0.25 0.35 0.44
median DE 0.62, DE LCh 0.27 0.39 0.38
p90 DE 1.24, DE LCh 0.45 0.60 0.95
max DE 1.31, DE LCh 0.68 0.79 1.23

results of conversion in ACR (process 2010, etc, with WB off D2 patch ) are naturally worse than math and not matching dcamprof test-profile 1:1
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on June 09, 2015, 10:24:56 am
Nelder-Mead simplex can do such stuff, but it requires lots of finetuning of error function and start values to get good results.

It would be interesting to integrate an SSF estimation function into DCamProf but it's probably not happening for at least a few months. I'm planning to code less during the summer which is starting here now :-)
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on June 09, 2015, 10:41:15 am
Nelder-Mead simplex can do such stuff, but it requires lots of finetuning of error function and start values to get good results.

It would be interesting to integrate an SSF estimation function into DCamProf but it's probably not happening for at least a few months. I'm planning to code less during the summer which is starting here now :-)

what might be the optimization parameters, I am using max dE as the top one, then with max dE being equal I am thinking shall I consider an average/mean dE or go for maximum chromacity error (not paying attention to luminocity errors) ?
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on June 09, 2015, 10:50:56 am
what might be the optimization parameters, I am using max dE as the top one, then with max dE being equal I am thinking shall I consider an average/mean dE or go for maximum chromacity error (not paying attention to luminocity errors) ?

For Nelder-Mead to find a good minimum it needs a quite "well-behaving" error function, if it's too random it's hit and miss. That is if it changes a parameter it must see some sort of trend if it got better. Using max dE may be worse than mean for this reason. I don't have any general good guide though, trial and error is the method I've used.
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on June 09, 2015, 12:58:26 pm
How could I possibly know the chances? :)
This thing is important:
How do I reset my Spectrolino?

To reset your Spectrolino, hold down the measurement button on the top of the instrument for approximately 10 seconds. During this time you will hear a sequence of four tones, then two tones in rapid succession that are repeated after a pause. Continue holding the measure button until you hear a fifth long tone, indicating that the reset is complete.

so I shall find out tonight... whether I can measure the black patches on CCSG or not

(http://s23.postimg.org/liwxr2e7v/20150609_125138.jpg) (http://s23.postimg.org/5n83nrnnf/20150609_125254.jpg)
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on June 09, 2015, 01:18:33 pm
Just released 0.7.4. A very small update adding the subjective chroma adjustment possibility.
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on June 09, 2015, 01:23:11 pm
I'm working on a camera profiling article, still much work left, but here's a preview of an elaborate workflow to make a Dual-Illuminant DNG profile based on shots of a CC24 and a custom home-made inkjet printer target made on semi-glossy paper.

It also show a little how I work iteratively using plots as guides. I also use RawTherapee for visual evaluations when comparing the look of profiles.

    # 1. Make a custom target to top of the CC24 with some super-saturated colors
  dcamprof make-testchart -l 15 -d 14.46,12.26 -O -p 210 custom-target.ti1
  printtarg -v -S -iCM -r -h -T300 -p A4 custom-target
    # 2. Print custom target to a semi-glossy OBA-free paper
    # 3. Scan the target with a spectrometer
  chartread -v -H -T0.4 custom-target
    # 4. Create reference file
  spec2cie -v -i D50 custom-target.ti3 glossy.cie
    # 5. Setup light and measure spectrum, save to light.sp for later use
  spotread -a -H -x
    # 6. Shoot CC24 target, glossy target, and white card.
    # 7. Crop export and convert to cc24.tif, glossy.tif and ff.tif
    # 8. Apply flatfield
  dcamprof testchart-ff cc24.tif ff.tif cc24-ff.tif
  dcamprof testchart-ff glossy.tif ff.tif glossy-ff.tif
    # 9. Scan values
  scanin -v -dipn cc24-ff.tif ColorChecker.cht cc24.cie
  scanin -v -dipn glossy-ff.tif custom-target.cht glossy.cie
    # 10. Merge targets into one, letting CC24 have priority
  dcamprof make-target -p cc24-ff.ti3 -a cc24 -p glossy-ff.ti3 -a glossy -d 0.03 combo.ti3
    # 11. Make a bunch of preliminary profiles to evaluate saturation
          when used with a film curve. Possibly you will want to
          desaturate the profile a little. Do the evaluation visually
          and subjectively in a raw converter.
  dcamprof make-profile -i light.sp combo.ti3 chroma.json ; dcamprof make-dcp -t acr chroma.json chroma-0.dcp
  dcamprof make-profile -i light.sp -k -1 combo.ti3 chroma.json ; dcamprof make-dcp -t acr chroma.json chroma-1.dcp
  dcamprof make-profile -i light.sp -k -2 combo.ti3 chroma.json ; dcamprof make-dcp -t acr chroma.json chroma-2.dcp
  dcamprof make-profile -i light.sp -k -3 combo.ti3 chroma.json ; dcamprof make-dcp -t acr chroma.json chroma-3.dcp
  dcamprof make-profile -i light.sp -k -4 combo.ti3 chroma.json ; dcamprof make-dcp -t acr chroma.json chroma-4.dcp
    # 12. Using the found chroma adjustment make a preliminary
    #     profile, dumping plots. Exclude glossy from the matrix
    #     optimizer to get as good base match as possible for the
    #     important normal colors represented by cc24
  dcamprof make-profile -r dump1 -w cc24 0,1 -w glossy 0,0 -i light.sp -k -1.5 combo.ti3 preliminary.json
    # 13. Start gnuplot (cd dump1; gnuplot -background gray) and plot
    #     target and LUT, plus LUT stretch vectors and DE vectors.
    #     Use 'set view equal xyz' and 'set view equal xy' to turn
    #     on/off scaling of lightness axis, must be turned off if
    #     error vectors are viewed in 3D.
  gnuplot> splot 'nve-lut.dat' w l lc "beige", 'gmt-locus.dat' w l lw 4 lc rgb var, \
    'gmt-adobergb.dat' w l lc "red", 'gmt-pointer.dat' w l lw 2 lc rgb var, \
    'target-nve-lutve2.dat' w vec lc "black", 'target-nve-lutvm.dat' w vec lw 2 lc "olive", \
    'targetd50-xyz.dat' pt 4 lc rgb var, 'targetd50-xyz.dat' using 1:2:3:5 w labels offset 3
    # 14. Look in the plot for patches that pull in opposite directions and cause a
    #     bad bend in the LUT. Add those (typically one or two) to an
    #     exclude.txt and render
  dcamprof make-profile -r dump1 -x exclude.txt -w cc24 0,1 -w glossy 0,0 -i light.sp -k 1.5 combo.ti3 preliminary.json
    # 15. Make matrix-only and full correction profiles for sanity
    #     check comparisons later
  dcamprof make-dcp -n "Canon EOS 5D Mark II" -t acr preliminary.json no-relax.dcp
  dcamprof make-dcp -n "Canon EOS 5D Mark II" -t acr -L preliminary.json matrix.dcp
    # 16. Relax the LUT (primarily for the glossy class) to improve
    #     smoothness. Re-render and replot for each change.
    #  a) It may be worthwhile to lock the matrix before changing DE k
    #     weights: save profile to separate file matrix.json and
    #     provide -m and -f parameters.
    #  a) Try relaxing DE weight even for the important CC24, setting
    #     it to at least 1 leads often to some relax without much loss
    #     in accuracy.
    #  b) Try changing CIEDE2000 k weights, 4,1,1 good start (less
    #     weight on lightness)
    # Example result after iterating:
  dcamprof make-profile -r dump1 -x exclude.txt -f matrix.json -m matrix.json \
    -w cc24 2,1,4,1,1 -w glossy 4,0,4,4,1 -i light.sp -k -1.5 combo.ti3 final-1.json
    # 17. Make final DCP, sanity check it by comparing it with
    #     no-relax.dcp and matrix.dcp. It should have better high
    #     saturation correction than matrix.dcp, and not lose too much
    #     accuracy compared to no-relax.dcp
  dcamprof make-dcp -n "Canon EOS 5D Mark II" final-1.json final-1.dcp
    # 18. Repeat steps 5 - 17 for the second illuminant
    # 19. Merge to a dual-illuminant profile, here using a default
    #     film curve
  dcamprof make-dcp -n "Canon EOS 5D Mark II" -t acr final-1.json final-2.json final-dual.dcp
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on June 09, 2015, 01:25:43 pm
0.7.4 build for Windows (mingw = dcamprof.exe + libgomp_64-1.dll + PDF manual / copy of Torger's blog) : https://app.box.com/s/ujo64khq6blhsxpxgmxcczg8jm19vtiz
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on June 09, 2015, 07:03:19 pm
so I shall find out tonight... whether I can measure the black patches on CCSG or not

Swiss engineering... cables !!!

(http://s23.postimg.org/6591hdlgr/gm3.jpg)

it seems to be working... __HOWEVER__ my CC SG black patches are still reading in line with i1pro2 data... which is L ~10-11... what that might be now ? do I have defective Spectrolino ? light trap it reads many times perfectly 0.00xxx for L/a/b (more consistent than i1pro2, but then it fits the light trap hole more tightly)
Title: Re: DCamProf - a new camera profiling tool
Post by: Iliah on June 09, 2015, 08:49:11 pm
It was designed for old Macs, with round serial (modem/printer port on PowerMacs, pre-G3; page 69, bottom drawing of http://www.xrite.com/documents/literature/gmb/en/spectrolino_serial_5_en.pdf - get it before it is gone, it contains serial protocol commands). Makes for 1 data cable and a power cable through "T-adapter" (power injector). Mine is a bit later model, so-called "purple", and power injector is combined with serial, does not need T-adapter.

L*=10..11 instead of L*=6..7 is seriously high. Is the target clean? Any dust in Spectrolino (on your shot the measurement head was without a "N" filter)?
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on June 09, 2015, 09:51:30 pm
http://www.xrite.com/documents/literature/gmb/en/spectrolino_serial_5_en.pdf - get it before it is gone, it contains serial protocol commands).

thank you for the link !

Makes for 1 data cable and a power cable through "T-adapter" (power injector). Mine is a bit later model, so-called "purple", and power injector is combined with serial, does not need T-adapter.

the strange thing is that while Mac "T-cable" (that black thing) is working for power connection, PC "T-adapter" (silver box labeled "Gretag Adapter Mini-DIN Part 36.19.58") is not (it passes the data, but not taking in the power, so I have to use the one for Mac = hence so many connections)

L*=10..11 instead of L*=6..7 is seriously high. Is the target clean?

it is - at least black patches are, those were the first thing I measured when I switched the whole thing on (I did that 10 sec pressing button reset first with Spectrolino with "U" ("N") filter and on a calibration tile)

Any dust in Spectrolino (on your shot the measurement head was without a "N" filter)?

if not -how to clean the lens (w/ filter off it) ? just with a soft cloth ? no isopropyl alcohol ?
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on June 09, 2015, 09:57:43 pm
I fully measured CC Passport so far (attached for posterity)

1) D6 patch has "Y" = 3.116 - which is close to the number you mentioned above in this topic

2) measurements, it seems to me, are very consistent between themselves (I did 5 in a row, calibration before each, and averaged) - seems better than with i1pro2 (but again that might be an operator error when I was using i1pro2 for Passport)

*** done, 1.32 -> 1.68 *** 3) it seems that it is possible to upgade firmware for Spectrolino, my is from 1998 and v1.32 (firmware)
Title: Re: DCamProf - a new camera profiling tool
Post by: Iliah on June 09, 2015, 11:24:52 pm
Things I use to clean:
http://www.edmundoptics.com/lab-production/cleaning/compressed-air/dust-off-compressed-air/1902/
http://www.edmundoptics.com/lab-production/cleaning/gloves-brushes-swabs/cotton-tipped-swab-applicators/2922/
http://www.edmundoptics.com/lab-production/cleaning/lens-cleaners-pouches/edmund-lens-cleaner/1367/

Since Passport was measured within the specs as it seems, the measurements of the SG target may be wrong because the target is out of specs.
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on June 10, 2015, 12:13:14 am
Since Passport was measured within the specs as it seems, the measurements of the SG target may be wrong because the target is out of specs.
seems strange that spectrophotometer manufacturers always include a white calibration tile , but not something like a "black" calibration tile... so what might be the way to find the truth ? why reading light trap (which gives something like 0.0x ) does not answer the question if the device functions properly ? I took a piece of sufficiently black velvet and Spectrolino reads it as L = 7.08 and it visually looks to me way darker than black patches on my CCSG, but then it is not even semiglossy
Title: Re: DCamProf - a new camera profiling tool
Post by: Iliah on June 10, 2015, 12:24:36 am
Spectrophotometer calibration tiles are very expensive, a set can easily be more than a grand. It is cheaper to send the device to a calibration service (not necessarily X-Rite). When you read black trap there is no stray light, so the only thing you are getting is that the noise is sufficiently low. Velvet allows for some light from the sides to get to the spectrophotometer. You may also want to e-mail X-Rite with the question - why the black patches are read abnormally high. Maybe they will replace your target, or say something useful. And of course try cleaning.
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on June 10, 2015, 12:29:02 am
Velvet allows for some light from the sides to get to the spectrophotometer.
so I switched all lights in the room and it's past midnight in rural area -> result is XYZ: 0.732348 0.748855 0.654355, D50 Lab: 6.764379 0.415899 -0.691388 ...
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on June 10, 2015, 01:42:14 am
so I switched all lights in the room and it's past midnight in rural area -> result is XYZ: 0.732348 0.748855 0.654355, D50 Lab: 6.764379 0.415899 -0.691388 ...

Just so I don't miss what happened, so you got lower values now, those 6.7 instead of 11 earlier? I guess I should in the docs recommend reading glossy targets in dark rooms then, to avoid stray light issues?
Title: Re: DCamProf - a new camera profiling tool
Post by: Bart_van_der_Wolf on June 10, 2015, 07:07:50 am
Just so I don't miss what happened, so you got lower values now, those 6.7 instead of 11 earlier? I guess I should in the docs recommend reading glossy targets in dark rooms then, to avoid stray light issues?

Or shoot them at an angle, facing a black background (which is mirrored by target surface).

Cheers,
Bart
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on June 10, 2015, 07:29:07 am
Or shoot them at an angle, facing a black background (which is mirrored by target surface).

We're discussing the spectrometer measurement, not the photo :)
Title: Re: DCamProf - a new camera profiling tool
Post by: GWGill on June 10, 2015, 08:37:47 am
seems strange that spectrophotometer manufacturers always include a white calibration tile , but not something like a "black" calibration tile
They do - it's the white calibration tile. When you don't turn the illuminant on, you get black.
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on June 10, 2015, 08:38:35 am
Just so I don't miss what happened, so you got lower values now, those 6.7 instead of 11 earlier? I guess I should in the docs recommend reading glossy targets in dark rooms then, to avoid stray light issues?

no, I have or have access now to 3 different spectrophotometers

1) ColorMunki Design
2) i1Pro2
3) Spectrolino/Green

my "issue" is - can I measure the black patches on my own CCSG or I can not... a $1000 question !

so far it seems all 3 are measuring them (including operator errors too) within L* = upper 10s - lower 12s

Iliah and CCSG data from PM are saying those shall be ~7s

so my next try this evening will be - I will took a piece of black velvet (which Spectrolino apparently measures as darker than black patches on my CCSG - that is what I was referring to by saying " result is XYZ: 0.732348 0.748855 0.654355, D50 Lab: 6.764379 0.415899 -0.691388" - about that velvet w/o any ambient light), switch the light off and measure it with all 3 devices to see if there is a difference between CM, i1Pro2 and Spectrolino ...

and I will try to write a WTF letter to X-Rite too...
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on June 10, 2015, 08:40:15 am
They do - it's the white calibration tile. When you don't turn the illuminant on, you get black.
good catch, "black"
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on June 10, 2015, 08:57:06 am
Just speculating, maybe they've changed the formula on newer CCSG targets, having made the black patches brighter? That would reduce the glare problem and make the target easier to use.
Title: Re: DCamProf - a new camera profiling tool
Post by: Iliah on June 10, 2015, 11:53:57 am
Black tile is 1% reflection +/-1% accuracy. In "no light" measurement mode spectrophotometer is usually hard-pressed against the white tile, and the tile itself is usually installed in  a holder that fits the spectrophotometer head, to ensure no stray light. It is not the same as calibration against flat black tile.
Title: Re: DCamProf - a new camera profiling tool
Post by: Iliah on June 10, 2015, 12:00:26 pm
Just speculating, maybe they've changed the formula on newer CCSG targets, having made the black patches brighter? That would reduce the glare problem and make the target easier to use.
Checked Robin Mayers measurements, target production dates 2005-2007 (same generation as mine), he used Spectrolino with D65 filter, and got L*=7.5..9.5
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on June 10, 2015, 12:39:24 pm
Checked Robin Mayers measurements, target production dates 2005-2007 (same generation as mine), he used Spectrolino with D65 filter, and got L*=7.5..9.5

as I noted earlier in this thread mine is some new design (I mean where logo is printed - certainly not from 2007 or earlier) and I do not have access to other CCSGs around to check...

here is how it looks :

(http://s28.postimg.org/3kqkomqn1/ccsg.jpg)

may be X-Rite decided to make patches lighter so that i1pro level devices could measure them ? just a wild guess... I am yet to email to xrite, trying to fix a notebook after yet another series of W8.1 patches.

PS: what was the point to use D65 filter with Spectrolino to measure the target for him ?

Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on June 10, 2015, 12:45:11 pm
Just speculating, maybe they've changed the formula on newer CCSG targets, having made the black patches brighter? That would reduce the glare problem and make the target easier to use.
but at the moment I can't check - unless somebody with both CCSG can do this, I simply do not have anybody around with the old CCSG (not even new and not even with passport... but plenty of guns though - appalachia).
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on June 10, 2015, 03:13:33 pm
Here goes X-Rite support :

Quote
Dear D....,
 
In an effort to resolve your support Case, the following comment has been added.  Please review this comment, and respond as indicated.

Comment:


There are two essential things to keep in mind here.

The first one is that you can't compare LAB readings from different models of spectrophotometers and expect to see exact matches. Your i1PRO2 is the newest one of the ones you've indicated you own, and will be using the current XRGA calibration standard. So please use that device for your measurements.

The second point that is relevant is that the target measurements done in ProfileMaker were done with older formulations of the ColorChecker targets, and were also done with a different spectrophotometer than any of the ones you are testing with.

I hope that this helps to explain.

We would like you to help us continuously improve the support we provide. Click on this Survey Link to provide your input (less than 1 minute to complete).


Thank you, 


B.... W....

Technical Support Rep - Level 2

so... "the target measurements done in ProfileMaker were done with older formulations of the ColorChecker targets"

anybody with any spectrophotometer and both old (old as in www.imaging-resource.com test shots) and shiny new CC SG out there willing to test ?
Title: Re: DCamProf - a new camera profiling tool
Post by: Iliah on June 10, 2015, 03:36:27 pm
Since you already have their ear, can you please ask them where we can find the reference for the new formulation of Digital ColorChecker SG?
This one http://xritephoto.com/ph_product_overview.aspx?ID=938&Action=Support&SupportID=5158 from 2012 indicates low Lab values for the target.
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on June 10, 2015, 04:16:57 pm
Since you already have their ear, can you please ask them where we can find the reference for the new formulation of Digital ColorChecker SG?

already did and CSR did an evasive maneuver suggesting me ColorChecker Classic data ("...Yes, we do post color values for the ColorChecker Classic 24 patch target...")... so I even asked if X-Rite 'd be so kind as to suggest the official/semi-official range for black patches only... waiting
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on June 10, 2015, 04:21:18 pm
This one http://xritephoto.com/ph_product_overview.aspx?ID=938&Action=Support&SupportID=5158 from 2012 indicates low Lab values for the target.

Iliah - but this is __exactly__ the data from Profile Maker - just converted from Spectral to LAB and rounded... but I will use this in my next move (as it has the date = 2012 - so when did they allegedly change the formulation then ?).
Title: Re: DCamProf - a new camera profiling tool
Post by: Iliah on June 10, 2015, 04:59:41 pm
Iliah - but this is __exactly__ the data from Profile Maker - just converted from Spectral to LAB and rounded... but I will use this in my next move (as it has the date = 2012 - so when did they allegedly change the formulation then ?).
Yes, that data is the same as in PM5, I know that. I do not know of SG formulation ever being changed. XRGA was introduced in 2010, before the page with the link to the reference was edited. You have Spectrolino which is not XRGA-enabled by default (ColorPort 2.x may add an option to account for XRGA w/Spectrolino spectral measurements), and the measurements should conform to pre-XRGA era. I do not know of any newer official or semi-official references, but I know out-of-spec batches of targets may happen. In some regards it may be even beneficial to have higher black values (less flare), and now that you have Spectrolino and know to measure the targets in the dark environment you may take advantage of that.
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on June 10, 2015, 05:30:48 pm
e2:e4

Quote
....On the back of your Digital Color SG there will be a date with the Edition month and year. What is your Edition information on this target...

January 2014 Edition

waiting for the next move
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on June 10, 2015, 05:46:19 pm
ColorPort 2.x may add an option to account for XRGA w/Spectrolino spectral measurements
I never managed to get it working on my Windows 8.1 x 64.... i1pc.bin crashes
Title: Re: DCamProf - a new camera profiling tool
Post by: Iliah on June 10, 2015, 06:11:56 pm
I keep VMs Windows XP64 and Windows 7.
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on June 10, 2015, 06:36:29 pm
I keep VMs Windows XP64 and Windows 7.
argyll does not use XRGA corrections right ? so if one uses argyll spotread/chartread for example then the reported (by the said utilities) data from old gear and new gear will be consistent - correct ?
Title: Re: DCamProf - a new camera profiling tool
Post by: Iliah on June 10, 2015, 06:43:36 pm
argyll does not use XRGA corrections right ? so if one uses argyll spotread/chartread for example then the reported (by the said utilities) data from old gear and new gear will be consistent - correct ?
As far as I know i1Pro XRGA support is in the driver, so it may depend on what driver is in use. I guess Graeme is the person to ask ;)
Title: Re: DCamProf - a new camera profiling tool
Post by: GWGill on June 10, 2015, 08:17:03 pm
PS: what was the point to use D65 filter with Spectrolino to measure the target for him ?
Many industrial measurements standardize on D65 and the 10 degree observer. While you can compute this using reflectance data, you won't get quite the same results when there is fluorescence present, whereas the D65 filter will gives a more accurate result.
[ Interestingly one of my Spectrolino's came with a UV filter. I guess it was an option, since it isn't mentioned in the manual, nor did one come with the other Spectrolino I was given. ]
Title: Re: DCamProf - a new camera profiling tool
Post by: GWGill on June 10, 2015, 08:23:39 pm
argyll does not use XRGA corrections right ? so if one uses argyll spotread/chartread for example then the reported (by the said utilities) data from old gear and new gear will be consistent - correct ?
Yes, ArgyllCMS doesn't currently support XRGA conversion (X-Rite have never publicly revealed what that is, which is pretty poor service to their customers in my view), so you get whatever is native from the device. I believe in the case of the i1pro2, that is XRGA. My current understanding though, is that XRGA involved some massaging of the wavelength calibration and spectral reference values, so it is unlikely to have any significant  effect on the L* value.
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on June 10, 2015, 08:28:39 pm
I believe in the case of the i1pro2, that is XRGA.
BabelColor Patchtool though gives an option to select one of 2 modes of operation for i1pro2 (XRGA = M0/M1/M2 and non-XRGA = M0 only) - so either the data comes non corrected from hardware to PC/MAC side software or there is a way to tell the device to switch back to non XRGA mode...
Title: Re: DCamProf - a new camera profiling tool
Post by: GWGill on June 11, 2015, 03:08:50 am
BabelColor Patchtool though gives an option to select one of 2 modes of operation for i1pro2 (XRGA = M0/M1/M2 and non-XRGA = M0 only) - so either the data comes non corrected from hardware to PC/MAC side software or there is a way to tell the device to switch back to non XRGA mode...
There is a dll that comes with the X-Rite driver & other programs that does is used for the conversions to/from XRGA.
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on June 13, 2015, 04:06:51 pm
I keep VMs Windows XP64 and Windows 7.
Iliah how do you manager to get the instrument connected in a virtual machine ?
Title: Re: DCamProf - a new camera profiling tool
Post by: Iliah on June 13, 2015, 04:43:55 pm
Iliah how do you manager to get the instrument connected in a virtual machine ?
That probably depends on the VM you are using. For Fusion (VMWare) it is automatic, when an instrument is plugged in, it pops up with a question 'what to do' - to connect to host or to VM. This question can be switched off, too, if one of those two actions is set as default in menu item "Virtual Machine" -> USB and Bluetooth -> Advanced USB options. I also set USB 2.0 there for the guest.
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on June 13, 2015, 05:16:02 pm
That probably depends on the VM you are using. For Fusion (VMWare) it is automatic, when an instrument is plugged in, it pops up with a question 'what to do' - to connect to host or to VM. This question can be switched off, too, if one of those two actions is set as default in menu item "Virtual Machine" -> USB and Bluetooth -> Advanced USB options. I also set USB 2.0 there for the guest.
may be USB settings were the issue, I put USB 1.1 compatibility now in VMWare and can connect to i1pro2... however with Spectrolino the ColorPort v2.0.5 software only shows SpectroScan as an option - which is Spectrolino with an automated measuring table, which I don't have... does it work with Spectrolino alone ?
Title: Re: DCamProf - a new camera profiling tool
Post by: Iliah on June 13, 2015, 05:27:41 pm
may be USB settings were the issue, I put USB 1.1 compatibility now in VMWare and can connect to i1pro2... however with Spectrolino the ColorPort v2.0.5 software only shows SpectroScan as an option - which is Spectrolino with an automated measuring table, which I don't have... does it work with Spectrolino alone ?
I tried and - no, it does not work for me, sorry. Only with Spectroscan, seems only because of the automatic positioning on the calibration tile. All the rest I was able to do manually.
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on June 13, 2015, 05:48:49 pm
I tried and - no, it does not work for me, sorry. Only with Spectroscan, seems only because of the automatic positioning on the calibration tile. All the rest I was able to do manually.

OK, N/A... on the other subject - I compared a new black velvet sample measuring with i1pro2 and spectrolino : i1pro2 gives L* = ~2.9-~4.0 on various spots and spectrolino ~2.9-~3.5 on various spots... so I guess if both of them do measure noticeably lighter black patches on SG and their data is similar then so it is... new targets do have the black patches lighter... I did not hear back from X-Rite so far... so will be trying to remind them again, albeit w/o big hope.
Title: Re: DCamProf - a new camera profiling tool
Post by: Iliah on June 13, 2015, 05:56:46 pm
OK, N/A... on the other subject - I compared a new black velvet sample measuring with i1pro2 and spectrolino : i1pro2 gives L* = ~2.9-~4.0 on various spots and spectrolino ~2.9-~3.5 on various spots... so I guess if both of them do measure noticeably lighter black patches on SG and their data is similar then so it is... new targets do have the black patches lighter... I did not hear back from X-Rite so far... so will be trying to remind them again, albeit w/o big hope.

Yes, it seems they need to publish a new reference for SG. Or recognise bad batches and either replace, or publish references. One other thing - if you have polarasing filter for Spectrolino, try measuring with the filter on, using MeasureTool from ProfileMaker 5.0.10. It should not make a difference, theoretically, but does not hurt to try.

Very sorry I confused you with Spectrolino / ColorPort.
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on June 13, 2015, 08:16:55 pm
One other thing - if you have polarasing filter for Spectrolino, try measuring with the filter on, using MeasureTool from ProfileMaker 5.0.10. It should not make a difference, theoretically, but does not hurt to try.

it actually makes hell lot of difference, while w/ U filter it gives L* = 11.x, w/ P filter it gives L* = ~1.8-1.9 (???!!!)

Very sorry I confused you with Spectrolino / ColorPort.

n/p - some knowledge gained
Title: Re: DCamProf - a new camera profiling tool
Post by: Iliah on June 13, 2015, 08:19:51 pm
> it actually makes hell lot of difference, while w/ U filter it gives L* = 11.x, w/ P filter it gives L* = ~1.8-1.9 (???!!!)

Was it with MeasureTool? Does the resulting CGATS say pola filter is in use?
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on June 13, 2015, 09:04:39 pm
> it actually makes hell lot of difference, while w/ U filter it gives L* = 11.x, w/ P filter it gives L* = ~1.8-1.9 (???!!!)

Was it with MeasureTool? Does the resulting CGATS say pola filter is in use?


you asked to use ProfileMaker MeasureTool, so it was it...  yes, filters were detected

Quote
INSTRUMENTATION   "Spectrolino"
MEASUREMENT_SOURCE   "Illumination=D50   ObserverAngle=2°   WhiteBase=Abs   Filter=Pol"

strangely MeasureTool does not output spectal data - LAB/XYZ only
Title: Re: DCamProf - a new camera profiling tool
Post by: Iliah on June 13, 2015, 09:21:43 pm
> strangely MeasureTool does not output spectal data - LAB/XYZ only
It should, if asked to - click on Configuring and choose spectral.
What L does it show for white patches with pola filter? What does it show on black and white patches without a pola filter?
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on June 13, 2015, 09:54:35 pm
> strangely MeasureTool does not output spectal data - LAB/XYZ only
It should, if asked to - click on Configuring and choose spectral.
but it does not... checkbox is on, but it does not save the spectral data (I am talking about the mode when it measures Test Chart).

white patch, L* = 98.*

grey patch L* = 48.*

black patch L* = 1.9

PS: for example I measured some 3 patches again (white-grey-black) and here is the output... no spectral data

Quote
LGOROWLENGTH   1
Measurement_mode   "patch"
CREATED   "6/13/2015"  # Time: 21:58
INSTRUMENTATION   "Spectrolino"
MEASUREMENT_SOURCE   "Illumination=D50   ObserverAngle=2°   WhiteBase=Abs   Filter=Pol"
KEYWORD   "SampleID"
KEYWORD   "SAMPLE_NAME"
NUMBER_OF_FIELDS   8
BEGIN_DATA_FORMAT
SampleID   SAMPLE_NAME   XYZ_X   XYZ_Y   XYZ_Z   LAB_L   LAB_A   LAB_B
END_DATA_FORMAT
NUMBER_OF_SETS   3
BEGIN_DATA
1   A1   91.25   94.85   75.05   97.97   -0.36   2.71
2   B1   16.33   16.98   13.81   48.24   -0.25   0.53
3   C1   0.21   0.21   0.18   1.89   0.30   -0.15
END_DATA
Title: Re: DCamProf - a new camera profiling tool
Post by: Iliah on June 13, 2015, 09:58:07 pm
but it does not... checkbox is on, but it does not save the spectral data (I am talking about the mode when it measures Test Chart).

white patch, L* = 98.*

grey patch L* = 48.*

black patch L* = 1.9

Nice measurements, overall. Do you save data explicitly, File - Save As; or do you use "save measurements"?
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on June 13, 2015, 10:02:49 pm
Nice measurements, overall. Do you save data explicitly, File - Save As; or do you use "save measurements"?
the dialog window says : "export lab"

(http://s9.postimg.org/r60bpsmrj/pm1.jpg)


PS: got... it - there is a difference between that option and what is done through : File - Save As ... RTFM !


Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on June 13, 2015, 10:04:35 pm
Nice measurements, overall.

well white and grey may be - but black ?
Title: Re: DCamProf - a new camera profiling tool
Post by: Iliah on June 13, 2015, 10:05:54 pm
the dialog window says : "export lab"

Please try File - Save As. You have "spkt" mark, so it should be saving spectral data, 'cause this mark indicates the measurements were taken in spectral mode.
Title: Re: DCamProf - a new camera profiling tool
Post by: Iliah on June 13, 2015, 10:07:30 pm
As to black, take measurements of those 15 neutral patches in the middle and let's check how linear they are. I would also measure the whole target and check deltaE with the reference.
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on June 13, 2015, 10:07:35 pm
Please try File - Save As. You have "spkt" mark, so it should be saving spectral data, 'cause this mark indicates the measurements were taken in spectral mode.
yes, I found that File->Save As indeed saves the spectral data... thank you, I was thinking that both options are the same and did not bother to try.
Title: Re: DCamProf - a new camera profiling tool
Post by: Iliah on June 13, 2015, 10:08:59 pm
yes, I found that File->Save As indeed saves the spectral data... thank you, I was thinking that both options are the same and did not bother to try.
No, they are quite different. It was mentioned in manual somewhere, but not in the Help file they have with the program.
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on June 13, 2015, 10:09:26 pm
As to black, take measurements of those 15 neutral patches in the middle and let's check how linear they are. I would also measure the whole target and check deltaE with the reference.
OK, I will do this now and post then
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on June 13, 2015, 10:59:09 pm
As to black, take measurements of those 15 neutral patches in the middle and let's check how linear they are. I would also measure the whole target and check deltaE with the reference.
here is the full target, taken with P filter
Title: Re: DCamProf - a new camera profiling tool
Post by: Iliah on June 13, 2015, 11:19:48 pm
here is the full target, taken with P filter
It looks like the instrument needs cleaning, after all.
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on June 16, 2015, 12:31:36 pm
Got my Solux lamps working finally by powering them properly. Look at plots in the neighboring thread:
http://forum.luminous-landscape.com/index.php?topic=100698.msg830255#msg830255

I think the Solux 4700K looks good for simulating D50 and even D55 when pushed into overdrive.
Title: Re: DCamProf - a new camera profiling tool
Post by: Iliah on June 16, 2015, 12:44:07 pm
Got my Solux lamps working finally by powering them properly. Look at plots in the neighboring thread:
http://forum.luminous-landscape.com/index.php?topic=100698.msg830255#msg830255

I think the Solux 4700K looks good for simulating D50 and even D55 when pushed into overdrive.

I had Solux bulbs powered from their own power source, and they do not simulate D50. When powered to a higher voltage through a high grade stabilized variable voltage power supply, the time drift of SPD is quite significant. Once I measured the SPD with 2nm resolution, and it was spiky.

Solux 4700/36, measured with i1Pro, 4 times, averaged, 10nm, beam integrated over the surface:
380   1.16572925E-3
390   1.91144715E-3
400   2.50905682E-3
410   3.10437335E-3
420   3.64064472E-3
430   4.19029733E-3
440   4.51817922E-3
450   5.01060858E-3
460   5.41420793E-3
470   5.68507239E-3
480   5.91422338E-3
490   6.13182224E-3
500   6.40437938E-3
510   6.66856067E-3
520   6.87941955E-3
530   7.24500837E-3
540   7.71248853E-3
550   8.06061924E-3
560   8.29440542E-3
570   8.48474540E-3
580   8.45067855E-3
590   8.48383829E-3
600   8.30854755E-3
610   8.03543720E-3
620   7.96562620E-3
630   7.86271039E-3
640   7.66951498E-3
650   7.79212918E-3
660   7.72498781E-3
670   7.72686955E-3
680   7.82623328E-3
690   7.64010241E-3
700   7.34899333E-3
710   7.22256443E-3
720   7.06195412E-3
730   6.93874108E-3
That is less than 4300K CCT.
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on June 16, 2015, 12:50:22 pm
When powered to a higher voltage through a high grade stabilized variable voltage power supply, the time drift of SPD is quite significant.
what might be the source of the drift then... assuming that you have some lab grade power supply and voltage then can be assured within what ? like < 0.1% or something like this ?
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on June 16, 2015, 12:52:44 pm
beam integrated over the surface
does it mean that you were illuminating some surface (like styrofoam) and measuring off the reflected light ? or something else ?
Title: Re: DCamProf - a new camera profiling tool
Post by: Iliah on June 16, 2015, 12:54:58 pm
does it mean that you were illuminating some surface (like styrofoam) and measuring off the reflected light ? or something else ?
Matt spectralon panel. That gives much better idea of the beam profile.
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on June 16, 2015, 12:58:44 pm
How could the spectrum be spiky? Shouldn't it be physically impossible when you have halogen? As far as I understand the Solux is nothing else than a standard halogen lamp with a blue-filtered reflector.

Indeed, you really need a variable power supply to get a color temperature you want. It's graded to 4700K but far from all lamps reach that, halogens vary quite a bit naturally and I would not think they have put 4700K in the center of the normal distribution. Lamp fixtures also seem often seem to underpower the lamps bit, and then with an unsuitable lamp shade (especially if using the unpainted older lamps) you lose even further due to mixing in unfiltered light.

I'd be surprised if Solux halogens would drift more than any other halogen, as it is a, well, halogen lamp.
Title: Re: DCamProf - a new camera profiling tool
Post by: Iliah on June 16, 2015, 12:59:55 pm
what might be the source of the drift then... assuming that you have some lab grade power supply and voltage then can be assured within what ? like < 0.1% or something like this ?
Possibly, accelerated aging; or overheating. Anyway, Solux are using Eiko Q50MR16/CG/47/36 neodymium bulbs, and metal-doped filters produce not very natural SPDs.
Title: Re: DCamProf - a new camera profiling tool
Post by: Iliah on June 16, 2015, 01:00:51 pm
> How could the spectrum be spiky? Shouldn't it be physically impossible when you have halogen? As far as I understand the Solux is nothing else than a standard halogen lamp with a blue-filtered reflector.

If you look into how the reflector is made...
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on June 16, 2015, 01:07:46 pm
I only have the Colormunki instrument to measure with that claims 3nm at high resolution mode, no spikes turn up there (spikes do turn up very clearly on a fluorescent source for example).

This makes me think that even if there are spikes they are narrow enough to have negligible impact in camera profiling applications. What we're measuring in profiling is the camera's spectral integration, and a "worried" spectrum is no problem unless it risks shifting the energy to such an extent that it shifts the integration result. Wouldn't you agree?
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on June 16, 2015, 01:16:07 pm
Additionally, it seems to me that a typical photographic flash or xenon arc lamp would be worse concerning spikiness, and certainly shot-to-shot variation, here an example flash spectrum (wavelength is in Angstrom not nm, divide by 10):

(http://www.astrosurf.com/buil/us/spectro10/flash.gif)

I'm just presenting the "evidence" I base my own recommendations on. As I thought I had enough info on my hands to be able to recommend the Solux on overdrive as a D50 simulator for camera profiling, any claims that suggests that it's actually no good is very interesting to me.
Title: Re: DCamProf - a new camera profiling tool
Post by: Iliah on June 16, 2015, 01:24:43 pm
> a "worried" spectrum is no problem unless it risks shifting the energy to such an extent that it shifts the integration result.

I simply found that the results of colour transforms I'm getting with 5000K using normal filters over halogen lamps are better compared to Eiko/Solux, and I can use regular 3200K halogen lamps and cheap high quality filters; both components easily replaceable. And I do not need to deal with uneven distribution of CT across the beam.
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on June 16, 2015, 01:26:34 pm
I only have the Colormunki instrument to measure with that claims 3nm at high resolution mode, no spikes turn up there (spikes do turn up very clearly on a fluorescent source for example).

This makes me think that even if there are spikes they are narrow enough to have negligible impact in camera profiling applications. What we're measuring in profiling is the camera's spectral integration, and a "worried" spectrum is no problem unless it risks shifting the energy to such an extent that it shifts the integration result. Wouldn't you agree?

a small note... for example CMF that you feed into DCamProf being spiky does not affect much the profiles it generates...
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on June 16, 2015, 01:52:54 pm
Ok, thanks for sharing.

I just did a quick drift test when running it 15.8 volts at 5500K. It's not free from drift, the first 10 minutes (started cool) it drifted about 0.3DE, but when warm it seems to stabilize a bit and it drifted 0.08DE the following ten minutes (plot attached). That drift is so small it can be some related to measurement too.

Anyway, I'm going to recommend to work with lamps that's been on for 10 minutes or so, and then work relatively swiftly, should take only 5 minutes or so to complete a series of test target shots, and it that time it should be stable enough.
Title: Re: DCamProf - a new camera profiling tool
Post by: Iliah on June 16, 2015, 02:01:39 pm
> I'm going to recommend to work with lamps that's been on for 10 minutes or so

Usual is 15 minutes to 20 minutes. Now, after the bulbs are stabilized, one needs to measure the spectrum; and that defeats the purpose of having D50 simulator. To be clear, that is the case with any halogen bulbs except lab grade sources, and even with those it is still better to measure the spectrum. Point of the exercise is to get as close to 5000K blackbody as possible the cheapest way.
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on June 16, 2015, 02:12:08 pm
Usual is 15 minutes to 20 minutes. Now, after the bulbs are stabilized, one needs to measure the spectrum;
one might imagine a spectrophotometer used may be in ambient mode right when you shoot your target... for example make a hole behind something where you target is mounted to place spectrophotometer there... now you just need to sync things somehow.
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on June 16, 2015, 02:41:22 pm
My strive for a "D50 simulator" should not be taken too literally, the purpose I have is a smooth spectrum, no significant broad peaks or dips, no significant spikes, with an overall distribution similar to D50. For camera profiling there's no real value in exactly matching D50 as D50 doesn't really exist in nature either. The purpose of this is to have an alternative to going outside using actual daylight. Shooting glossy targets with minimized glare is a lot easier to do with this type of light source than using real daylight.

Now I'm pushing it to 5500K to get a little bit more spacing from 2850K in a dual-illuminant scenario.

Compared to a 3000K halogen lamp with an 80B filter all indications I get is that the overdriven Solux makes a better job. I'm sure though that Liu labs lamp is even better, but I doubt it will present any significant gain, except if we want to get closer to 6500K which the Solux can't do.

It's far from a practical setup, I have only one lamp, flatfield correction is required, temperature stabilization is required, it's preferable to measure it. I wouldn't use this for copy work or anything like that, but to shoot a target with flatfield-capable software it seems to be a good setup.

The value of actually measuring the spectrum is there when we want to make a DNG profile that can do precise whitepoint estimation, otherwise it's not that important when we make a generic "daylight" profile.
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on June 17, 2015, 02:08:03 am
On a different subject; I'm going to have to look more into how to subjectively deal with the contrast curve. The global desaturation trick did not work out as well as it first seemed, it may work for highly saturated colors but leave low saturation colors too desaturated. Possibly an increasing onset depending on saturation would work, or I need to figure out some different strategy.
Title: Re: DCamProf - a new camera profiling tool
Post by: Alexey.Danilchenko on June 17, 2015, 04:27:55 am
I simply found that the results of colour transforms I'm getting with 5000K using normal filters over halogen lamps are better compared to Eiko/Solux, and I can use regular 3200K halogen lamps and cheap high quality filters; both components easily replaceable. And I do not need to deal with uneven distribution of CT across the beam.

Iliah, do you use 80A, 80C Lee gels for correction? I had a go with Lee CT blue gels (full and half) briefly and for my halogen source they do add strange dip at the green area. Have not evaluated various lamps yet only the one that came with my source but it does not have this without filters.
Title: Re: DCamProf - a new camera profiling tool
Post by: Iliah on June 17, 2015, 09:22:03 am
Iliah, do you use 80A, 80C Lee gels for correction? I had a go with Lee CT blue gels (full and half) briefly and for my halogen source they do add strange dip at the green area. Have not evaluated various lamps yet only the one that came with my source but it does not have this without filters.
I use 8x series.

For the lights they make:
Lee 283, 201, 281, 202 (these are the main ones)
Given the sensitivity of sensor green channels, a dip in the green area may be a good thing.
 
for use with MR16 and PAR bulbs, framed glass, better greens:
Lee B64, LD201, LD202
Title: Re: DCamProf - a new camera profiling tool
Post by: Alexey.Danilchenko on June 17, 2015, 09:31:40 am
I use 8x series.

For the lights they make:
Lee 283, 201, 281, 202 (these are the main ones)
Given the sensitivity of sensor green channels, a dip in the green area may be a good thing.
 
for use with MR16 and PAR bulbs, framed glass, better greens:
Lee B64, LD201, LD202

Thanks, mine was 201 (Full CT blue) that had this dip. I tried 202 as well and it was a lot better and smoother. For now my only use for them was an attempt to even slightly blue areas of the spectrum for monochromator input hence the questions.
Title: Re: DCamProf - a new camera profiling tool
Post by: Iliah on June 17, 2015, 09:33:39 am
Thanks, mine was 201 (Full CT blue) that had this dip. I tried 202 as well and it was a lot better and smoother. For now my only use for them was an attempt to even slightly blue areas of the spectrum for monochromator input hence the questions.
Lee are publishing pretty accurate Y-curves on their site.
For the purpose of making the light to monochromator more even I would be looking at 281 and 202, maybe double layer.
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on June 17, 2015, 09:39:20 am
Lee B64, LD201, LD202
those who love smooth graphs shall like this one = B53 Blue 3 - it is a black line on the graph

(http://s4.postimg.org/guhyfng31/chart.png)
Title: Re: DCamProf - a new camera profiling tool
Post by: Iliah on June 17, 2015, 09:58:56 am
those who love smooth graphs shall like this one = B53 Blue 3 - it is a black line on the graph

(http://s4.postimg.org/guhyfng31/chart.png)
With dichroic filters, the curve is angle-dependent. If the sampling is closer, such filters usually show the transmission curve is not very smooth https://www.thorlabs.com/newgrouppage9.cfm?objectgroup_id=986
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on June 17, 2015, 10:08:25 am
With dichroic filters, the curve is angle-dependent. If the sampling is closer, such filters usually show the transmission curve is not very smooth https://www.thorlabs.com/newgrouppage9.cfm?objectgroup_id=986
but then @ any angle that curve shall be still smoother than others, because it has a better "starting point /at 0 degrees/", no ?
Title: Re: DCamProf - a new camera profiling tool
Post by: Iliah on June 17, 2015, 10:13:26 am
but then @ any angle that curve shall be still smoother than others, because it has a better "starting point /at 0 degrees/", no ?
The problem here is to have good mixing after the filter to have reasonably constant SPD across the beam. The setup starts to be bulky and expensive (compared to a single fresnel lens).
Title: Re: DCamProf - a new camera profiling tool
Post by: Alexey.Danilchenko on June 17, 2015, 10:40:50 am
Lee are publishing pretty accurate Y-curves on their site.
For the purpose of making the light to monochromator more even I would be looking at 281 and 202, maybe double layer.
Thanks will try this combination (I have a Lee swatchbook and that is enough for light source fiber optic lead).
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on June 17, 2015, 02:39:48 pm
This tone curve thing is giving me an headache...

I've attached two images to demonstrate. Both are using the same profile. The first with linear curve, and the other a typical S-curve, Adobe's default. The curve brighten the picture a lot so I have reduced the exposure of the second to make it easier to compare.

The first has rather accurate colors when I compare the calibrated screen with the real thing, as expected. The second has not only increased saturation quite far, but also pushed the reds into some other hue. I know one reason, the curve is applied in Prophoto space (as DNG pipeline does), pushing green to 0 in sRGB. Adobe Lightroom (used here) has applied some sort of gamut mapping though so it's not as bad as it could be. I've tried to apply the curve in sRGB space instead when using RawTherapee, better result. Even without space clipping there are saturation increases and hue shifts though.

In theory the tone curve could been seen as just a compensator for the Hunt and Stevens effects (eg the dimmer screen needs more contrast and saturation to perceptually match a brighter real scene), but it's generally going way over the top. I'd like it not to be the profile's job to deal with side effects of the tone curves, but anyone that's going to use default Adobe curves with DCamProf profiles are going to think that the profiles suck, and indeed they kind of do.

Currently I'm currently out of ideas how to deal with this. For myself using RawTherapee I can apply contrast using a CIECAM02 lightness curve and have good realistic results, but that's not what the general user will be doing, so I guess I need to come up with something. I'm not totally out of ideas, but there are only the painful ones left...

There are a zillion of tone curve models out there in the research world, which all are better than a simple RGB curve (ICC profiles) or Adobe's RGB curve twist (DNG profiles), but DCamProf can just make the profile, not affect which tone curve that's being used, and current color pipelines are locked to simplistic RGB curves.

Title: Re: DCamProf - a new camera profiling tool
Post by: torger on June 17, 2015, 04:40:01 pm
It seems like this tone curve thing is what makes camera color "difficult".

As an RGB curve is so massively bad at doing anything perceptually sane, you need to make a profile that rather than making accurate colors make a precompensation for the tonecurve errors, so you end up at something decent.

The commercial profiles seems to be a mix of subjective look and compensation of tone curve errors. It differs a lot how saturated reds are dealt with for example.

I shall continue with some testing though. If I'm lucky a good compensation can be some gradual pre-desaturation of colors, that is low saturation colors are kept almost unchanged while high saturation colors are desaturated more, and out of sRGB/AdobeRGB colors even more. If that works the profile can stay 2.5D.

In the painful case I need to make the profile 3D with the sole purpose of compensating the tonecurve's wrongdoings. In a way that could be a cleaner design though. In that case I'd keep subjectivity out of the design process and the native profile format, it would always be linear, and tonecurve compensation would only be an option in the make-dcp command. It would then just add a 3D LookTable on top which compensates for the tonecurve. It would be quite simple to make a "mathematically exact" compensation this way, if you know what the end result should be. A 3D LUT would be less robust though, if you change the curve the compensation would no longer match, but maybe that's not a big issue.

With ICC profile you cannot add a separate look-table on top, but one can merge everything into the same 3D CLUT, for the user it will be no difference.

To do the 3D way I need to decide on what the desired tonecurve look should be... work in progress...!
Title: Re: DCamProf - a new camera profiling tool
Post by: Hening Bettermann on June 17, 2015, 06:51:12 pm
Hi Torger

Many of the technical details in this thread are far above my head. Yet I dare to come up with a little comment. I attach 3 screen shots. 1-shows your linear jpeg opened in PhotoLine. 2-with an S-curve in RGB, which shows the same color shift as yours. And 3- the same S-curve, but applied in HSV, using the V channel only. No color shift (visually judged). It seems to me the solution to the curve problem should not be searched in the profile. But I can see, it is something the USER would have to do...

Best regards - Hening.
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on June 18, 2015, 02:39:01 am
Hi Torger

Many of the technical details in this thread are far above my head. Yet I dare to come up with a little comment. I attach 3 screen shots. 1-shows your linear jpeg opened in PhotoLine. 2-with an S-curve in RGB, which shows the same color shift as yours. And 3- the same S-curve, but applied in HSV, using the V channel only. No color shift (visually judged). It seems to me the solution to the curve problem should not be searched in the profile. But I can see, it is something the USER would have to do...

Thanks for the feedback. Indeed, the easy way out is to say "the profile gives you accurate colors, but the raw converter messes it up with its ancient RGB curve, you need to use some other method to add contrast". I can certainly live with that myself, but in terms of making profiles for wider use it's probably not so good strategy. The sad thing is that a neutral accurate profile will in Lightroom, Capture One etc produce less realistic colors (at least for the saturated ones) than some of the bundled ones, as the bundled ones are designed with the tone curve in mind. Unless you lock it to a linear curve, but for a decent look you need to add at least a little contrast.

Designing for a specific curve and contrast makes the profile less generic and means that you cannot use it for reproduction work, so it's really not a nice option. But the fact is there -- most current raw converters use primitive color pipelines from the 1990s and that makes it impossible to have one profile with neutral accurate color that can be used with different amounts of contrast at still produce realistic colors.

People tend to think that you can use a profile with any "film curve" (in Capture One you can select it separately), but the profiles are clearly designed for the default film curve. Capture One's profiles produce too desaturated results when used with a linear curve, because they're not designed for that. DCamProf's profiles produce accurate results with linear curve, but then instead too saturated and sometimes visibly hue-shifted results when used with a curve.

I don't have a finished answer on what's the state of the art contrast curve today, but preliminary I've got very nice results with a CIECAM02 lightness curve. CIELab lightness which in theory would be the same thing as CIECAM02 lightness is not good at all, it desaturates color, not realistic result. The reason being that the CIECAM02 is simply a more recent and better color appearance model.

What raw converters should be doing is have a modern color pipeline designed for digital photography rather than being nostalgically stuck with "film looks", in that way the profile would not be involved in tone reproduction or tone mapping at all but you would control that in the raw converter and you could choose both realism and make a subjective look. The only raw converter that allows this type of workflow I know of is RawTherapee (the oldschool way is the default though). My goal with DCamProf is to work with popular widely used raw converters though so I need to adapt to their workflows.
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on June 18, 2015, 03:04:46 am
I wonder what Adobe were thinking when they made the DNG pipeline apply the tone curve in prophoto space. To me that seems to be a rather bad idea. Any fairly saturated color will be clipped due to this, and Lightroom "repairs" it with some sort of gamut mapping. It makes the look really unpredictable so I understand that people need to handtune their profiles.

Capture One applies their RGB curve in some smaller space, which it is seem to depend on camera. That's probably a better approach, but still not well-defined.
Title: Re: DCamProf - a new camera profiling tool
Post by: Alexey.Danilchenko on June 18, 2015, 04:48:15 am
I wonder what Adobe were thinking when they made the DNG pipeline apply the tone curve in prophoto space. To me that seems to be a rather bad idea. Any fairly saturated color will be clipped due to this, and Lightroom "repairs" it with some sort of gamut mapping. It makes the look really unpredictable so I understand that people need to handtune their profiles.

Capture One applies their RGB curve in some smaller space, which it is seem to depend on camera. That's probably a better approach, but still not well-defined.
A small correction, they don't apply it in just ProPhoto - they apply it nearly at the end of the pipeline in linear ProPhoto. That is before converting to gamma corrected ProPhoto.
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on June 18, 2015, 05:23:31 am
A small correction, they don't apply it in just ProPhoto - they apply it nearly at the end of the pipeline in linear ProPhoto. That is before converting to gamma corrected ProPhoto.

Indeed, I am often a bit sloppy when talking about color spaces. Used to floating point since ten years gamma has lost its use, so I meant prophoto primaries in linear space. I shall try to be clearer, it's not the first time I confuse people with my sloppy color space terminology :-)
Title: Re: DCamProf - a new camera profiling tool
Post by: Hening Bettermann on June 18, 2015, 06:39:22 am
[...] The only raw converter that allows this type of workflow I know of is RawTherapee [...]

Hm - I use(d) Iridient that way. And if memory serves me, one could do it in ACR as well. I use(d) the linear camera-own profile both as an in and out profile. This requires a gamma matrix profile, which is built with the assumption of a linear sensor response. This should be theoretically OK with the 5D2. And I have naïvely done the same with the a7r, and become first now aware of that the compressed raw means the sensor is not linear. I have not examined this any further yet.

edit: Ooops, no, ACR does not allow you to choose a linear out profile.
Title: Re: DCamProf - a new camera profiling tool
Post by: Alexey.Danilchenko on June 18, 2015, 08:52:11 am
And I have naïvely done the same with the a7r, and become first now aware of that the compressed raw means the sensor is not linear. I have not examined this any further yet.
For all intents and purpouses the raw is linear. The tonal curve applied to compress (lossy compression) the raw. Raw converters then apply it  in reverse to unpack the raw data. The raw values are linear because otherwise you would not be able to white balance it easily.
Title: Re: DCamProf - a new camera profiling tool
Post by: Hening Bettermann on June 18, 2015, 10:47:55 am
Oh thank you! That is certainly good news to me!
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on June 18, 2015, 11:17:38 am
Oh thank you! That is certainly good news to me!
and when things are getting non linear you hence have WB issues - like deeeeeep shadows.
Title: Re: DCamProf - a new camera profiling tool
Post by: Hening Bettermann on June 18, 2015, 12:16:54 pm
and when things are getting non linear you hence have WB issues - like deeeeeep shadows.

And hiiiighlights? Obviously this is the problem - but what do I make of this? Is the response of the sensor+raw converter linear, or not? Does the raw converter apply white balance before or after linearising? Is this the correct question? The practical question for me is: Can I make a gamma matrix profile for the a7r based on the assumption of linear sensor response? And can I continue to use this both as the in and out profile? Or do I need to make 2 LUT profiles? What I want is a linear output profile.

Are we side tracking the thread, Torger?

Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on June 18, 2015, 01:09:23 pm
Is the response of the sensor+raw converter linear, or not?

you are not dealing with "sensor", you are dealing with the numbers that firmware writes in raw file... so for profiling purposes you try to get those numbers in a "linear fashion", means you expose so that any non linearities can be avoided ... that is you do not underexpose and you do not get close to clipping - based on how particular camera behaves here and there.

Does the raw converter apply white balance before or after linearising?

naturally after

Title: Re: DCamProf - a new camera profiling tool
Post by: Hening Bettermann on June 18, 2015, 02:21:05 pm
thank you for clarifying!
Title: Re: DCamProf - a new camera profiling tool
Post by: Iliah on June 18, 2015, 02:21:18 pm
>> Does the raw converter apply white balance before or after linearising?
> naturally after

Yes, if linearization is a part of raw unpacking and decoding :)
No, if you mean full well limited cameras, like some Panasonics are.
There are more woodpeckers than one can possibly count.
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on June 18, 2015, 03:26:45 pm
No, if you mean full well limited cameras, like some Panasonics are.
you mean that with Panasonic you can have each sensel data recorded in raw to the max possible DN /no adjustment by firmware to so max limit/, so you have non linear data if you take more then one sensel at once for your purposes - they /sensels/ all will have (might have) different well capacities... hence  non linearity.
Title: Re: DCamProf - a new camera profiling tool
Post by: Iliah on June 18, 2015, 04:47:46 pm
you mean that with Panasonic you can have each sensel data recorded in raw to the max possible DN /no adjustment by firmware to so max limit/, so you have non linear data if you take more then one sensel at once for your purposes - they /sensels/ all will have (might have) different well capacities... hence  non linearity.
_Some_ Panasonic cameras at the lowest ISO setting are full well limited, so each sensel charged close to max (within 1 stop of saturation) is non-linear, Panasonic has the linearity limit in makernotes, and we informed Phil of that tag - so it is covered in exiftool.
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on June 19, 2015, 04:55:22 am
Searched some more on tonecurve stuff, and it's not that easy to find any obvious solution to the problem. In the research world this is called "tone reproduction operator", and they have since 10+ years left the simplistic one global tone curve approach and are more into spatially variable operators, that is what we usually call tonemapping. A profile can't do scene-dependent corrections so we can't use that kind of stuff.

The traditional RGB tone curve is not only about creating a subjective look, but also to fix appearance issues, mainly Stevens and Hunt effects (contrast and colorfulness appears higher in a real bright scene than if reproduced with lower luminance on a screen). Digging down you also have Helmholtz-Kohlrausch effect and Bezold-Brücke hue shift effects. All these effects are scene-dependent of course, but you could say a fixed tone curve roughly matches a sunny day, and it's probably good approach. It will make duller scenes look more contrasty than the real deal but that's probably how people will want to make it look anyway.

The big name raw converters are in any case not designed for scene-dependent appearance modeling, we have this fixed tonecurve to relate to. And then my idea is that I want a tone curve which changes contrast, but colors appears to be of the same hue (and saturation) as before. It's there the current raw converters with their RGB curves fail miserably, making a linear colorimetric profile produce bad colors. But a profile can always compensate to give you any result you'd like, and it's exactly what the big names do (with varying success). My plan is to make it possible to make tone curve compensation with DCamProf profiles as well, but using some good appearance model rather than relying on golden-eye subjective hand-tuning which many commercial raw converters seem to be doing.
Title: Re: DCamProf - a new camera profiling tool
Post by: ErikKaffehr on June 19, 2015, 05:53:32 am
Hi,

What about just applying tone curve to the luminance channel?

My take is really that raw conversion should result in a scene referred image and different kind of "looks" could be applied that image. I don't know if this is in conflict with DCP-based processing.

Best regards
Erik

Searched some more on tonecurve stuff, and it's not that easy to find any obvious solution to the problem. In the research world this is called "tone reproduction operator", and they have since 10+ years left the simplistic one global tone curve approach and are more into spatially variable operators, that is what we usually call tonemapping. A profile can't do scene-dependent corrections so we can't use that kind of stuff.

The traditional RGB tone curve is not only about creating a subjective look, but also to fix appearance issues, mainly Stevens and Hunt effects (contrast and colorfulness appears higher in a real bright scene than if reproduced with lower luminance on a screen). Digging down you also have Helmholtz-Kohlrausch effect and Bezold-Brücke hue shift effects. All these effects are scene-dependent of course, but you could say a fixed tone curve roughly matches a sunny day, and it's probably good approach. It will make duller scenes look more contrasty than the real deal but that's probably how people will want to make it look anyway.

The big name raw converters are in any case not designed for scene-dependent appearance modeling, we have this fixed tonecurve to relate to. And then my idea is that I want a tone curve which changes contrast, but colors appears to be of the same hue (and saturation) as before. It's there the current raw converters with their RGB curves fail miserably, making a linear colorimetric profile produce bad colors. But a profile can always compensate to give you any result you'd like, and it's exactly what the big names do (with varying success). My plan is to make it possible to make tone curve compensation with DCamProf profiles as well, but using some good appearance model rather than relying on golden-eye subjective hand-tuning which many commercial raw converters seem to be doing.
Title: Re: DCamProf - a new camera profiling tool
Post by: Alexey.Danilchenko on June 19, 2015, 06:57:36 am

The big name raw converters are in any case not designed for scene-dependent appearance modeling, we have this fixed tonecurve to relate to. And then my idea is that I want a tone curve which changes contrast, but colors appears to be of the same hue (and saturation) as before.

Check RPP out
Title: Re: DCamProf - a new camera profiling tool
Post by: Bart_van_der_Wolf on June 19, 2015, 09:16:57 am
Check RPP out

Only possible on Mac ...:(

Cheers,
Bart
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on June 19, 2015, 10:45:59 am
What about just applying tone curve to the luminance channel?

My take is really that raw conversion should result in a scene referred image and different kind of "looks" could be applied that image. I don't know if this is in conflict with DCP-based processing.

Current raw converters are a bit of a mixture of old film techniques and new color appearance models. White balance or tone curves do not exist in color appearance models like CIECAM02. In color science speak white balance is a simplified chromatic adpatation transform, and an RGB tone curve is an over-simplified tone reproduction operator. Color science is busy with making realistic reproductions, even emulating eyes night vision etc (meaning that you need to know exposure level too), and taking view conditions into account. To make it state-of-the-art is very complex. Documentary video seems to be the natural applications of these types of ultra-realistic reproductions.

The first part of DCP could be said to be scene referred, but the tone curve is certainly output referred. As I want DCamProf to be a profile maker also useful for the big name raw converters I need to adapt to their workflows. Personally I'd surely prefer that a profile only did the scene referred part, and the raw converter took care of all output modeling, but that's not how it works in Lightroom and Capture One for example. They expect a curve to be there, and if the profile is linear scene referred only there are no tools to add contrast to adjust the look without having color shift effects.

In photography we seem to settle at rendering D50 bright sunny scenes fairly realistically in an average viewing condition, and then accept as a bonus the extra "pop" we get when we apply the same tone reproduction and colorfulness on other conditions. I'm fine with that. Problem is that the RGB tone curve is as discussed a very bad tone reproduction operator which don't even make the sunny scenes realistic but makes color pop like crazy, and in some circumstances cause hue shift as well. Bundled profiles are semi-compensated for that, by hand I suppose for more exclusive cameras.

I'm trying to find an automated way to compensate a profile so it can be used with RGB curves without getting crazy results.

Applying contrast to the luminance channel is a good idea, but then the next question is which model of luminance should we use? Using Lab Lightness results in a desaturated look. I'm currently experimenting with using CIECAM02 lightness and that is a better perceptual model than Lab and gives better results. However, as we apply lightness contrast to simulate a brighter outdoor condition (Stevens effect) we still get a desaturated look due to the Hunt effect, to match realistically we need to add some saturation too. It's there I am now. It seems like this problem hasn't really been solved before as those that work with CIECAM02 don't work with traditional "film curves".

A color scientist would just say it can't be done without knowing all scene parameters and viewing condition parameters, but I'm looking into using that sunny D50 scene as a reference, I think if it works for that scene we'll have a suitable "neutral" look that photographers can identify with. The bright outdoor sunny scene is really the only condition that mandates increased contrast and saturation.
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on June 19, 2015, 10:48:22 am
Only possible on Mac ...:(
no, both RPP and Iridient do work in OSX and OSX does work in VmWare (yes, EULA says you can't, but Apple can take a hike) and VmWare does work on PC/Win...
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on June 19, 2015, 10:54:58 am
RawTherapee is also a good experimental box for trying out "scene referred" workflows, in RT you have the CIECAM02 mode, where you can enter scene conditions, viewing conditions and all, simulate light dark adaptation and partial chromatic adaptation etc.

You can also apply contrast in Lab, CIECAM02 J or Q, or use their own custom "weighted standard curve" which is a quite good curve when it comes to keeping hue/saturation constant.

As said though we don't want to keep saturation constant as we have the Hunt effect to think about, if we just add contrast the impression is that we reduced saturation even if it was kept constant. We could also argue that hue should not be kept constant either due to the Bezold-Brücke effect, but I think that is overkill. So my current idea is to add contrast and saturation in some sane way and keep hue constant.

I can recommend this link for those interested in those color appearance effects: http://www.handprint.com/HP/WCL/color4.html
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on June 19, 2015, 01:09:54 pm
Small demonstration.

Same profile on all images (a matrix-only profile). First image is with linear curve (this is the reference), the others have the same contrast curve. One is CIECAM02 J lightness only (second image), hue and saturation/chroma constant. The other (third image) is the same, but with a subjective increase in chroma.

If you really just stare at the pure colors and compare you'll probably see that the constant hue/chroma matches the linear the best. If you look at the picture "globally", the constant hue/chroma looks slightly desaturated compared to the original linear curve, and then the one with slightly increased chroma may look more "right".

For reference I've added a 4th image which is a pure RGB curve in sRGB space (which here makes a better result than doing it in ProPhoto space like DNG). With sRGB and this color I don't think the RGB curve pulls it off that much, maybe a little towards yellow (hue shift is worse if done in Prophoto due to clipping). The obvious difference is that it's grossly saturated, way over the top.

It's quite subtle differences we're working with here, so it can be quite hard to compare with the images one below the other, especially since screens vary in color and contrast slightly depending on viewing angle. Downloading and flipping through them in an image viewer makes it easier.

I'm thinking that DCamProf users will want results similar to image 3, that is contrast in lightness, but with some added saturation.
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on June 19, 2015, 01:34:40 pm
For the sake of completeness, here's one with the RGB curve applied in Prophoto space, here without gamut mapping which makes the hue shift more evident.
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on June 19, 2015, 03:48:37 pm
I think it's quite clear from the little demonstration that you cannot see the profile and curve separately. That's why I'm skeptical about the practice I see many do in Lightroom and Capture One of changing to linear curve to get "more accurate color" out of the bundled profiles. Most likely the bundled profiles are designed for the default curves, and if they are they have changed the color to a big extent to work with that RGB curve, meaning that color may very well become less accurate when used with a linear curve.

I see on the forums that curves are by many seen as some minor thing you add on top to get a little contrast, but it makes a big difference in global color appearance, and for some colors hue shift is quite noticable. Striving for utmost precision in targets and measurements when making an own profile is all thrown out the window if we add a standard curve on top and think we still have accurate color.

We can always fall back on the argument that accurate color is not important, as we will subjectively tune the look to something that pops anyway, and I have no problem with that view. However, the vision I have with DCamProf is to provide to photographers that want it a neutral starting point to add their artistic intent on top, rather than getting a random starting point.

For my local runners club I shoot running competitions from time to time, and it's with bundled profiles impossible to get those supersaturated sports apparel look the same hue as in real life, which I think is remarkable that we can't do in 2015. And although I can agree that maybe it's not that important, it feels silly that my expensive gear and software can't do it. Although I add a subtle subjective look to my images, having the wrong hue on peoples clothing is not a part of my artistic intent, it just becomes like that because profiles and raw converter is not up to it. The film legacy have shaped raw converter design into making it more difficult to get realistic color than it had to be, and I'd say the lack of a proper tone reproduction operator is 95% of the problem.

If we want to do reproduction photography we're fine though, because in that case we won't be using a curve and then any raw converter with a colorimetric profile will make accurate color.
Title: Re: DCamProf - a new camera profiling tool
Post by: Alexey.Danilchenko on June 19, 2015, 05:46:50 pm
Only possible on Mac ...:(

And your point here being? I pointed to RPP because it is one of the few converters that does not use RGB of any kind as its internal working space - it uses UPLab (http://www.brucelindbloom.com/index.html?UPLab.html). It is quite possible that tonal curve is applied to the L channel there (Iliah can confirm/deny this as I am not entirely sure).
Title: Re: DCamProf - a new camera profiling tool
Post by: Alexey.Danilchenko on June 20, 2015, 05:20:46 am
I think it's quite clear from the little demonstration that you cannot see the profile and curve separately. That's why I'm skeptical about the practice I see many do in Lightroom and Capture One of changing to linear curve to get "more accurate color" out of the bundled profiles.
The approach is still perfectly possible. Matrix only profile + raw development adjustments in raw converter + assigning LUT colour correction profile in Photoshop to bring colour to accurate values. Having said that it is not that easily achievable in LR.
Title: Re: DCamProf - a new camera profiling tool
Post by: Bart_van_der_Wolf on June 20, 2015, 05:42:39 am
And your point here being?

Hi Alexey,

The point being that I'd love to try/use RPP (as would undoubtedly many others), but without creating a MAC OS environment or having a MAC, Windows users cannot use it. So suggestions that can potentially solve issues, and that only work on a 'limited' number of setups are partial solutions at best.

One of the DCamProf benefits is that it can be (and is) ported to many OS platforms, thanks to Anders' making the source code available for recompilation into complete working binaries.

Quote
I pointed to RPP because it is one of the few converters that does not use RGB of any kind as its internal working space - it uses UPLab (http://www.brucelindbloom.com/index.html?UPLab.html). It is quite possible that tonal curve is applied to the L channel there (Iliah can confirm/deny this as I am not entirely sure).

Yes, that may be useful, but I cannot test it on my Windows machines (unless I purchase and invest time in setting up a virtualization environment).

But thanks for the suggestion anyway, maybe Mac OS users can do something with it (or maybe one day there will be an RPP Windows binary?).

Cheers,
Bart
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on June 20, 2015, 06:22:13 am
I have not had that good results of applying tone curve in L in Lab, the look becomes to desaturated, but you can of course subjectively add in some saturation.

In the posted experiments above the curve is actually always run through RGB first, but only to check at what luminance it gets at then it's done in CIECAM02 J. I did that so the curve would look the same luminance-wise to be easier to compare with the RGB result.
Title: Re: DCamProf - a new camera profiling tool
Post by: Bart_van_der_Wolf on June 20, 2015, 07:26:57 am
I think it's quite clear from the little demonstration that you cannot see the profile and curve separately. That's why I'm skeptical about the practice I see many do in Lightroom and Capture One of changing to linear curve to get "more accurate color" out of the bundled profiles.

Hi Anders,

I do not think that the Linear curve users (I am one of them) in Capture One do it to get "more accurate color", but rather to have better control and rendering over e.g. highlights. With the default (until one changes that) film curve, there is a significant loss of highlight detail compared to less of a 'shoulder' roll-off designed to prevent hard-clipping.

Also, with my Exposed to the Right (ETTR) shots (as verified with Rawdigger, free of important highlights clipping), the default film curve will seemingly clip my (EOS 1Ds3) highlights by close to 1 stop. So I would need to underexpose (with more overall noise as a result) if I wanted to use the Film curve as it is encoded in Capture One.

Therefore, I much prefer using the C1 controls to adjust the tone curve (or even the profile with the ColorEditor>Save as ICC), but starting from a neutral rendering, so I can better see where the real issues are. That choice of curve shape is not because of the color 'accuracy', and C1 has excellent tools for creating pleasing colors, assuming they were not clipped and reasonably good to begin with.

Cheers,
Bart
Title: Re: DCamProf - a new camera profiling tool
Post by: ErikKaffehr on June 20, 2015, 07:53:04 am
Hi,

I made some test in an IT-8 chart and I got better accuracy in terms of Delta E (CIEDE 2000) with the linear than with default curve in C1.

I absolutely agree with Bart's observation that C1 default curve seems to yield overexposed images. I guess this may explain the reason MFDs are said to provide more DR in the highlights, the default processing and the rated ISO makes us underexpose.

Best regards
Erik


Hi Anders,

I do not think that the Linear curve users (I am one of them) in Capture One  do it to get "more accurate color", but rather to have better control and rendering over e.g. highlights. With the default (until one changes that) film curve, here is a significant loss of highlight detail compared to less of a 'shoulder'.

Also, with my Exposed to the Right (ETTR) shots (as verified with Rawdigger, free of important highlights clipping), the default film curve will seemingly clip my (EOS 1Ds3) highlights by close to 1 stop. So I would need to underexpose (with more overall noise as a result) if I wanted to use the Film curve as it is encoded in Capture One.

Therefore, I much prefer using the C1 controls to adjust the tone curve (or even the profile with the ColorEditor>Save as ICC), but starting from a neutral rendering, so I can better see where the real issues are. That choice of curve shape is not because of the color 'accuracy', and C1 has excellent tools for creating pleasing colors (assuming the were not clipped, and reasonably good to begin with).

Cheers,
Bart
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on June 20, 2015, 09:14:22 am
If you make delta E comparisons with curve you need to disregard from lightness, as the curve by nature will introduce large errors. Possibly the linear curve would show less incorrect color anyway, as the C1 profiles are created for a look its hard to know without testing. It's hard to test a curved look for accuracy as saturation need to be added for compensation of perceptual effects. Standard delta E tests don't work for testing appearance modeled color.

I'm sure though the profiles are designed to be used with the default curve. As they use an RGB tone curve they must choose to design for a specific, can't maintain color in both.
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on June 20, 2015, 09:20:55 am
Doesn't negative exposure slider settings work in C1 to bring in highlights? Haven't tried recently so I don't know.

I have noted that C1 cuts away quite some highlights per default, it varies depending on camera though
Title: Re: DCamProf - a new camera profiling tool
Post by: ErikKaffehr on June 20, 2015, 09:46:52 am
Hi,

I presume you are right on all issues. For me, the liner curve have mostly been a way to get rid of Delta E errors induced by he non linearity of the L-component.

What I think I have seen on the C1-profiles that they are over saturating blues and may be greens, so they actually go out of gamut for say Adobe RGB on the IT-8 target.

What I have seen is very large deviation on blue patches, like Delta E (CIEDE 2000) around 17 that were widely exceeding Adobe RGB gamut, but still looking very good on screen.

:-) Erik :-)



If you make delta E comparisons with curve you need to disregard from lightness, as the curve by nature will introduce large errors. Possibly the linear curve would show less incorrect color anyway, as the C1 profiles are created for a look its hard to know without testing. It's hard to test a curved look for accuracy as saturation need to be added for compensation of perceptual effects. Standard delta E tests don't work for testing appearance modeled color.

I'm sure though the profiles are designed to be used with the default curve. As they use an RGB tone curve they must choose to design for a specific, can't maintain color in both.
Title: Re: DCamProf - a new camera profiling tool
Post by: Alexey.Danilchenko on June 20, 2015, 09:56:06 am
The point being that I'd love to try/use RPP (as would undoubtedly many others), but without creating a MAC OS environment or having a MAC, Windows users cannot use it. So suggestions that can potentially solve issues, and that only work on a 'limited' number of setups are partial solutions at best.

Everyone decides what instruments they need - that includes both cameras and equipment to process the images. I was pointing out that the instrument does exist out there - whether you want to make an effort to use of course is up to you. To me it is just a matter of the choice - same as choosing the camera or lens.

Anyway I am not sure it does exactly what Anders is trying to achieve - perhaps Iliah can clarify it.
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on June 20, 2015, 10:31:56 am
If you make delta E comparisons with curve you need to disregard from lightness, as the curve by nature will introduce large errors. Possibly the linear curve would show less incorrect color anyway, as the C1 profiles are created for a look its hard to know without testing. It's hard to test a curved look for accuracy as saturation need to be added for compensation of perceptual effects. Standard delta E tests don't work for testing appearance modeled color.

I'm sure though the profiles are designed to be used with the default curve. As they use an RGB tone curve they must choose to design for a specific, can't maintain color in both.

C1 CE edition was supplied by profiles (and curve files) supposedly for reproduction work... both profiles and curves can be used in a regular edition (unfortunately it seems that Lab readouts can't - or at least I did not find a way)
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on June 20, 2015, 10:33:03 am
I absolutely agree with Bart's observation that C1 default curve seems to yield overexposed images.
I think that curve files in C1 do carry both curves and exposure correction commands inside .frv files.
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on June 21, 2015, 11:12:20 am
And your point here being? I pointed to RPP because it is one of the few converters that does not use RGB of any kind as its internal working space - it uses UPLab (http://www.brucelindbloom.com/index.html?UPLab.html). It is quite possible that tonal curve is applied to the L channel there (Iliah can confirm/deny this as I am not entirely sure).

if you are talking about the curve that you select in RPP's UI = "Curve Type : Film-like, L*, Gamma, Colorimetric Gamma" , then it is applied to raw channels post per channel multiplication for WB and before demosaicking... hence before a color transform guided by a "camera profile"
Title: Re: DCamProf - a new camera profiling tool
Post by: Alexey.Danilchenko on June 21, 2015, 04:21:33 pm
if you are talking about the curve that you select in RPP's UI = "Curve Type : Film-like, L*, Gamma, Colorimetric Gamma" , then it is applied to raw channels post per channel multiplication for WB and before demosaicking... hence before a color transform guided by a "camera profile"

You absolutely sure it applies before camera profile?
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on June 21, 2015, 09:31:13 pm
You absolutely sure it applies before camera profile?
well, I collected a number of quotes from the authors from raw-rpp.l...j......l.com, plus you can select an output to a .tiff w/o applying any color transform and this control is still enabled - so it works before that.
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on June 22, 2015, 02:53:29 am
I've worked with reproduction curves now for a while and I think I'm closing in on something that works.

Using faces is unbeatable when it comes to subjectively evaluating color, much easier to see subtle differences there than in other color ranges.

My goal is to make a tone reproduction curve where the contrast itself is a subjective/artistic matter, but hue and saturation is kept perceptually stable, ie the same as with a linear curve. That is you get to subjectively choose a contrast curve, and then color is automatically adapted to match.

The current prototype uses RGB in linear prophoto to calculate luminance change, this is to make the curve "compatible" with RGB tone curves, ie brightness/contrast becomes the same. The actual working space is CIECAM02 JCh though. Hue is kept at the original value, J (lightness) adjusted according to the curve, and C (chroma) is increased as a function of midtone contrast. Using multiplication rather than addition seems to work best for chroma increase, that is more saturated colors get more increase than less saturated colors. Some fine-tuning left to do though, and then implement in DCamProf.

It's a bit scary to do this as one have to trust one's eyes, there's nothing to verify against if the result is "correct" :)

RawTherapee already has this type of curve, "weighted standard" it's called (try it out if you like), it often produces good results but is a bit low on saturation easily noted on skin tones. In the RT team we're looking at adding a new curve with better properties. Not sure my curve will do then as it's a bit slow, at least in my current implementation. We'll see...
Title: Re: DCamProf - a new camera profiling tool
Post by: Alexey.Danilchenko on June 22, 2015, 03:15:05 am
well, I collected a number of quotes from the authors from raw-rpp.l...j......l.com, plus you can select an output to a .tiff w/o applying any color transform and this control is still enabled - so it works before that.
I thought that at least for matrix profiles the input needs to be linear.
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on June 22, 2015, 07:56:19 am
I thought that at least for matrix profiles the input needs to be linear.
still, AT = http://raw-rpp.livejournal.com/6180.html?thread=105252#t105252
Title: Re: DCamProf - a new camera profiling tool
Post by: Alexey.Danilchenko on June 22, 2015, 08:36:30 am
still, AT = http://raw-rpp.livejournal.com/6180.html?thread=105252#t105252
He does not say anything about profile - only demosaic though I can see how that can be implied.
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on June 22, 2015, 09:52:13 am
He does not say anything about profile - only demosaic though I can see how that can be implied.
you are not saying that color transform is applied in RPP to vectors like (r, 0, 0, 0) or (0, g1, 0, 0) or (0, 0, b, 0) or (0, 0, 0, g2), are you ? so that is how it is implied and I think I have some more quotes along those lines...
Title: Re: DCamProf - a new camera profiling tool
Post by: Alexey.Danilchenko on June 22, 2015, 10:36:08 am
you are not saying that color transform is applied in RPP to vectors like (r, 0, 0, 0) or (0, g1, 0, 0) or (0, 0, b, 0) or (0, 0, 0, g2), are you ? so that is how it is implied and I think I have some more quotes along those lines...


I am saying I don't know and that you are probably right. I'll have a look at it under Hopper a bit later
Title: Re: DCamProf - a new camera profiling tool
Post by: Iliah on June 22, 2015, 11:30:38 am
> I thought that at least for matrix profiles the input needs to be linear.

Depends on CMM.
Yes, user-selectable curves are applied before assigning profiles in RPP; and yet another curve is applied after the profile is assigned to compensate for certain ColorSync weakness - except if the selected curve is "colorimetric gamma". Simulation (film) profiles are a different story.
Title: Re: DCamProf - a new camera profiling tool
Post by: Alexey.Danilchenko on June 22, 2015, 11:39:23 am
> I thought that at least for matrix profiles the input needs to be linear.

Depends on CMM.
Yes, user-selectable curves are applied before assigning profiles in RPP; and yet another curve is applied after the profile is assigned to compensate for certain ColorSync weakness - except if the selected curve is "colorimetric gamma".
Thanks for confirmation Iliah. I was looking (quite some time ago) at a binary with Hopper but have not dug deep to understand differences bewteen curves applied at various stages. I was not referring to simulation profiles at all (those are quite different beasts even between generation 1 and 2 from what I could see).
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on June 22, 2015, 12:05:56 pm
yet another curve is applied after the profile is assigned to compensate for certain ColorSync weakness
isn't that increasing some math errors, curve after curve ...
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on June 22, 2015, 12:10:22 pm
Thanks for confirmatrion Iliah.
may be they, knowing what is going to happen when ColorSync CMM will apply the trc(s) from some camera profile - which is known at that moment from what user selects, do some pre-compensation in advance to negate Colorsync and then only apply the real curve  8) ...
Title: Re: DCamProf - a new camera profiling tool
Post by: Iliah on June 22, 2015, 12:55:46 pm
> isn't that increasing some math errors, curve after curve
We know what we are doing ;)
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on June 25, 2015, 03:17:15 pm
Okay now I'm hating Adobe for their DNG profile format :)

Their DNG Profile processor requires that all 3D LUT entries for saturation=0 or value=0 must be 0,1,1 that is no scaling of saturation or value can happen for neutral values. That saturation (and thus hue) cannot be changed is sort of logical, as saturation for neutral gray is zero and since it's scaled with multiplication it can't change from zero.

However, Adobe has taken this a step further meaning that you can't change value of grays either. That is you cannot make a middle gray a little brighter or darker, it stays at 1.0. This is less logical, as it is possible to scale those values with multiplication, and indeed RT's DCP code does it, but not Lightroom and I just checked not the DNG reference code either.

I should have checked this earlier. I've just completed a tone reproduction operator implemented in a LookTable (and as the table applies a curve it must scale neutrals), but it won't work in Lightroom... ***arrgghh***. It should be possible to work around though by combining the use of a curve and a table, but it got a bit messier...
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on June 26, 2015, 07:49:43 am
I have now released version 0.8.0.

The big thing in this release is the neutral tone reproduction operator, that is it's now possible to make a profile that has a tone-curve but still retains neutral color. I will probably fine-tune the tone reproduction operator in coming releases, but it won't change that much. The current version has some issues with blue channel clipping that in some cases can cause some transition issues. Thanks to the blue handling blue skies perform much better than typical profiles so I can't just compress the blue channel without doing something smart. We'll see.

Currently you can only apply the neutral tone reproduction operator on DNG profiles, but I intend to add it for ICC profiles eventually. Although the ICC LUT is simpler to work with than the DNG profile LUTs, it's not well-defined which color-space the curve will be applied in or if it's applied in the ICC or separately, so it's quite some effort to get it in there. It's summer here now so programming will be slower, unless the weather is bad.

If you have no idea what I'm talking about or just want to get some further info I suggest reading the new section on tone curves in the manual, it also contain example images so you get an idea of how it looks and what difference it makes: http://www.ludd.ltu.se/~torger/dcamprof.html#tone_curves

Other smaller features I know some will appreciate is that you can set more DCP tags directly from the command line, like baseline exposure offset and copyright string. Go to the home page to get a full list of changes.
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on June 26, 2015, 09:47:23 am
0.8.0 build for Windows (mingw = dcamprof.exe + libgomp_64-1.dll + PDF manual / = copy of Torger's web page converted to PDF /) : https://app.box.com/s/97elm0weiiyackz0dopbwk9e6pw04eap
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on June 26, 2015, 10:45:39 am
those who love smooth graphs shall like this one = B53 Blue 3 - it is a black line on the graph

(http://s4.postimg.org/guhyfng31/chart.png)

I decided to try to play with that

(http://s1.postimg.org/dnrldx14f/leedf.jpg)
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on July 01, 2015, 02:29:50 pm
It looks like the instrument needs cleaning, after all.

so _4_ spectrophotometers later (colormunki, i1Pro2, 2 x Spectrolino - I got one more just in case) here goes XRite

Quote
We've been informed that ColorChecker SG charts (January 2014 production date only!) contain a production failure, especially in the black patches (Lightness and Hue shift). These charts can be replaced under warranty (even, if they are out of warranty) via RMA.

Please send us a digital picture of the reverse of your ColorChecker SG target that will show us the production date of your target. The only production date that is affected is January 2014.

no, they are still not willing to share the numbers, but it seems they are willing to exchange the target...
Title: Re: DCamProf - a new camera profiling tool
Post by: Iliah on July 01, 2015, 03:35:10 pm
so _4_ spectrophotometers later (colormunki, i1Pro2, 2 x Spectrolino - I got one more just in case) here goes XRite

no, they are still not willing to share the numbers, but it seems they are willing to exchange the target...
Jan 2014 production only? Or maybe not...
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on July 01, 2015, 03:45:45 pm
Jan 2014 production only? Or maybe not...
speaking about the proverbial patch (tuft) of wool from a sheep...
Title: Re: DCamProf - a new camera profiling tool
Post by: Iliah on July 01, 2015, 03:55:31 pm
speaking about the proverbial patch (tuft) of wool from a sheep...
Is yours from Jan 2014?
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on July 01, 2015, 04:07:09 pm
Is yours from Jan 2014?
yes and "they" (CSR at least - it is the same name since the beginning) know that - they asked about the edition even during the first round - this is 3rd round of exchange about the matter already, as I asked somebody else couple of weeks ago to help me to get to the right people in X-Rite (and may be that /person/ helped actually /just in case - thank you, Joshua/)

Title: Re: DCamProf - a new camera profiling tool
Post by: Iliah on July 01, 2015, 04:13:29 pm
yes and "they" (CSR at least - it is the same name since the beginning) know that - they asked about the edition even during the first round - this is 3rd round of exchange about the matter already, as I asked somebody else couple of weeks ago to help me to get to the right people in X-Rite (and may be that /person/ helped actually /just in case - thank you, Joshua/)

So, they knew it is Jan 2014 beforehand. Does not add much credibility to their "only".
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on July 01, 2015, 04:19:16 pm
So, they knew it is Jan 2014 beforehand. Does not add much credibility to their "only".
at least it seems I have a chance to get a "proper" target, will see how it measures then... albeit if they state that the formulation was changed it still leaves the question about some /data/ basis vs which to compare your own measurements to see if something is out of the ordinary (unless they have a stock of legacy targets from the old days somewhere), like with this one from Jan 2014 - I am curious to see how all the devices I can use will measure the black patches if I /hopefully/ receive a replacement...
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on July 03, 2015, 12:02:40 pm
Released v0.8.1, fixed a critical bug in the tone reproduction operator and improved its extreme highlight rendering
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on July 03, 2015, 01:02:57 pm
0.8.1 build for Windows (mingw = dcamprof.exe + libgomp_64-1.dll + HTML & PDF manual / = copy of Torger's web page and the same converted to PDF /) : https://app.box.com/s/sx1f8asu3ze7zxmnwa96cixjcwceqsec
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on July 06, 2015, 07:56:27 am
Released v0.8.2 where I added the neutral tone reproduction support also for ICC profiles.
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on July 06, 2015, 10:18:17 am
Released v0.8.2 where I added the neutral tone reproduction support also for ICC profiles.
JFYI for few users my windows binaries, I will not be able to build this till this evening (EST)...
Title: Re: DCamProf - a new camera profiling tool
Post by: Bart_van_der_Wolf on July 06, 2015, 11:22:40 am
JFYI for few users my windows binaries, I will not be able to build this till this evening (EST)...

Even with the delay, much appreciated. And thank you Anders, for the ICC support.

Cheers,
Bart
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on July 06, 2015, 03:01:04 pm
Even with the delay, much appreciated. And thank you Anders, for the ICC support.

I haven't been able to test it yet in Capture One as my license is on a computer in a different location, and there was some mess with trial version on this box so I gave it up. It should work though, otherwise I'll fix it when I get the opportunity.

If you want an ICC profile designed with the neutral tone reproduction operator it will currently always be applied directly in the LUT, which for C1 means that you must always use the profile with the linear curve as the curve is applied in the LUT itself. I've tested it in RT and it works well there, and I think it should work well also for C1 but I'm not 100% sure as I've not been able to test yet. I may in the future make it possible to not include the curve in the LUT, but in that case one have to know which working space C1 is using for applying the curve and that seems to vary between cameras in some way I don't fully know how it works.

When mentioning RT (RawTherapee) where I'm also a contributor I've today pushed a patch that makes looktables be applied after exposure as they should. Before RT could not handle looktables properly. This means that if you get a build made July 6 or later you will get excellent control of DCamProf DNG profiles, in one and the same profile you can have a matrix-only profile, refined with a HusSatMap to a more accurate colorimetric result, and then a curve and looktable on top for the neutral tone operator, each element can be toggled directly inside RT depending on how you want to use the profile.
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on July 06, 2015, 05:37:43 pm
0.8.2 build for Windows (mingw = dcamprof.exe + libgomp_64-1.dll + HTML & PDF manual / = copy of Torger's web page and the same converted to PDF /) : https://app.box.com/s/t0b7hd7ufvccsvgntr3vt6e3vtouxmsj
Title: Re: DCamProf - a new camera profiling tool
Post by: Alexey.Danilchenko on July 07, 2015, 06:53:35 am
That's super-cool, I'd love to see some photos when you have it all working!

I assembled my monochromator setup at last with all bits (it took long time to complete and test). Yesterday made a test run to calibrate and took spectral sensitivities of the Kodak ProBack645 (lots of data to process and not sure that it went ok yet - I think I need some adjustments to my light measurement device on Arduino), the transmittance spectra of all filters (IR, AA etc) and transmittance spectra of almost all lenses. I now need to process it all and build a library and experiment with simulation for the camera readings of a target.

That's how the setup looks (sorry for the pictures - took them with mobile).

Monochromator with the integrating sphere and with a lightmeter on Arduino:

(http://www.aletan.com/img/s1/v22/p1305760524-5.jpg) (http://www.aletan.com/img/s1/v22/p1305760524-6.jpg)

(http://www.aletan.com/img/s7/v154/p1305757851-5.jpg) (http://www.aletan.com/img/s7/v154/p1305757851-6.jpg)


And switched on (light is very dim - it was too dark without any light for a photo and nothing visible with full light on):

(http://www.aletan.com/img/s8/v0/p1305757819-5.jpg) (http://www.aletan.com/img/s8/v0/p1305757819-6.jpg)
(http://www.aletan.com/img/s3/v7/p1305757943-5.jpg) (http://www.aletan.com/img/s3/v7/p1305757943-6.jpg)
(http://www.aletan.com/img/s5/v127/p1305757849-5.jpg) (http://www.aletan.com/img/s5/v127/p1305757849-6.jpg)

The monochromator itself (http://www.optometrics.com/monochromators) used in my setup is expensive if bought from new but they could be found as a part of the industrial assembly of Luxtron/Lumasense 1108 (http://www.ebay.co.uk/itm/170625275474) (they can be found on ebay quite cheap every now and then). The integrating sphere was part of the assembly from Photonis XP3112/PB (http://www.ebay.co.uk/itm/271166072389). Arduino device I built and coded myself and as for the light source - any with goose neck fiber optic cable will do (plenty cheap on eBay). The whole setup generally cost a lot less than a new ColorChecker SG.

The calibration procedure is simple. I pre-heat the light source (halogen 150W Philips) to stabilize, turn on my instrument, and then walk through the spectrum (400-730nm) at 5nm intervals measuring spectra with i1Pro in high res mode in Argyll (all in a dark room with insulation of i1Pro and integrating sphere) and record the readings off the Arduino meter (just upper ones - it is just a reading of the full visible range photodiode). All this is then placed in a calibration table and used with readings of the light meter when taking the shots with camera to adjust the light source drifts (and there are definitely drifts there as I discovered).

Knowing the transmission spectra of lenses and filters, it should be possible to make the simulation of a target shot by camera in a certain light (will need to do that experiment with real target).
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on July 07, 2015, 09:32:19 am
I assembled my monochromator setup at last
beautiful !
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on July 07, 2015, 10:52:26 am
Wow Alexey, great to see!
Title: Re: DCamProf - a new camera profiling tool
Post by: Iliah on July 07, 2015, 02:22:48 pm
Since you already employed Arduino, I would be adding a rotary encoder (like those Sparkfun are carrying) to log the wavelengths. Maybe a stepper motor too.
Saying "record the readings off the Arduino meter (just upper ones - it is just a reading of the full visible range photodiode)" - how do you mean "upper" - those "Ful 28" on the second image?
Title: Re: DCamProf - a new camera profiling tool
Post by: Alexey.Danilchenko on July 07, 2015, 03:10:00 pm
Since you already employed Arduino, I would be adding a rotary encoder (like those Sparkfun are carrying) to log the wavelengths. Maybe a stepper motor too.
Saying "record the readings off the Arduino meter (just upper ones - it is just a reading of the full visible range photodiode)" - how do you mean "upper" - those "Ful 28" on the second image?

Perhaps one day - I'd like to get the feeling of this whole thing first :). For now I employ a very "sophisticated" method of using pocket torch between the measurements to adjust the dial ;)

Re photodiode - I used Adafruit lux sensor (same as sparkfun uxing tao chip) that has 2 diodes in a package (full visible and ir). I used calculations from their library to calculate 4 values: full range(raw reading), ir,fill-ir and calculated lux (just for fun). I'm using fuĺl range reading - is that wrong?
Title: Re: DCamProf - a new camera profiling tool
Post by: Iliah on July 07, 2015, 06:29:15 pm
> Perhaps one day - I'd like to get the feeling of this whole thing first

I agree. Just saying how I do those things (though I use Linux boards, not Arduino, unless I need real time - and that case I use small Atmel or PIC boards or chips as slaves to some Linux board; humidity sensors in my film drying cabinet would be one of the examples, power supply control to stabilize light is another one).

> I used Adafruit lux sensor (same as sparkfun uxing tao chip) that has 2 diodes in a package (full visible and ir). I used calculations from their library to calculate 4 values: full range(raw reading), ir,fill-ir and calculated lux (just for fun). I'm using fuĺl range reading - is that wrong?

Not at all, it is perfectly OK. I was just asking because your reference to "upper" did not came clear to me. What you did with a photodiode is a classic setup, same one I use - though I use a different diode. Never occurred to me to try a TAOS chip in this application.
Title: Re: DCamProf - a new camera profiling tool
Post by: Alexey.Danilchenko on July 07, 2015, 07:46:47 pm
> I agree. Just saying how I do those things (though I use Linux boards, not Arduino, unless I need real time - and that case I use small Atmel or PIC boards
> or chips as slaves to some Linux board; humidity sensors in my film drying cabinet would be one of the examples, power supply control to stabilize light is another one).

I am not that experienced with hardware so Arduino seemed a no brainer for me ;).

> power supply control to stabilize light is another one).

I am quite interested in that (assuming you are talking about halogen sources) - would you point me in a right direction?

> Not at all, it is perfectly OK. I was just asking because your reference to "upper" did not came clear to me. What you did with a photodiode is a classic setup,
> same one I use - though I use a different diode. Never occurred to me to try a TAOS chip in this application.

I had the chip assembly left from experiments with SLR/n nonlinearities in the last 1/6 stop (we discussed it a while ago if you remember) so it was an easy choice.

I had extracted my ProBack spectras tonight and they are rubbish. The first attempt went astray so I will need to redo it. I'd need to correct a few things as well:
- I used Kodak firmware capacity to subtract black on long exposures but it still generates a lot of noise so will need to do black subtraction myself
- have not closed viewfinder (I know its stupid but I forgot) for the 400-450 range so blue is captured quite badly
- I need to invent something to connect camera to integrating sphere more tightly to prevent even slightest leaks
- And add baffle to either entry from monochromator or photodiode (the latter seem to be influenced a bit by entry light positioning and sits opposite exit from monochromator albeit offset a bit).
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on July 08, 2015, 05:53:47 am
Thought I'd post an example. I haven't actually made that many finished profiles as I've always been busy fixing something or adding some new feature.

Attached is an example dual-illuminant DNG profile for Canon 5D mark II, made using DCamProf v0.8.2 with the workflow outlined in the code block. It uses a CC24 for base colors combined with a homemade semi-glossy inkjet target for extreme colors. Light sources are halogen with voltage control, a normal lamp for StdA and a Solux on overdrive for D55. The uneven light (only one point light) has been corrected with DCamProf's flatfield correction function.

A curve was designed "by eye" in RawTherapee to reasonably match the camera's JPEG brightness/contrast, so we get predictable result when using the camera's auto exposure. The Canon does quite some black subtraction to blacken shadows in its JPEGs, I've chosen to have lower contrast shadows and keep detail. Then this curve is applied when generating the profile using DCamProf's neutral tone reproduction operator.

CAT02 was used for StdA apperance rendering, while Solux "D55" was deemed to be close enough to D50 so no CAT was used there (-C).

The DCP contains two pairs of matrices (StdA and D55) which aim to make best match for the normal range of colors (represented by the CC24), the glossy target has been excluded from matrix correction. Then there are two 2.5D HueSatMaps (StdA/D55) for further colorimetric correction, and which also stretches the borders to decently match the glossy colors. Weighting and relaxation has been used to (my) taste to make a tradeoff between smoothness and accuracy.

Then we have the Looktable+Tone Curve which implements the subjective contrast increase and with maintained neutral reproduction. The Looktable has its value dimension gamma-encoded for better perceptual resolution. The looktable is what makes the profile fairly big, as it must be a 3D LUT. Finally there's a baseline exposure offset of -0.3, which is supposed to be added on top of Adobe DNG Converter's baseline exposure of 0.4 stops it adds for 5Dmk2 files. If you don't use Adobe-generated DNGs (ie have 0 baseline exposure) that should be changed to +0.1 stops.

If you have "toggable" DCP handling like a RawTherapee built after July 6, you can turn off the Looktable+Curve to get a colormetric profile, and then turn off the HueSatMap to get a pure matrix profile. The elements are sort of layered, first matrix, then huesatmap on top and then looktable and then curve. I like that aspect of DNG profiles, but unfortunately this aspect is rarely used in profile design, and I think only RT supports toggling the elements in it.

   # 1. Make a custom target to top of the CC24 with some super-saturated colors
  dcamprof make-testchart -l 15 -d 14.46,12.26 -O -p 210 custom-target.ti1
  printtarg -v -S -iCM -r -h -T300 -p A4 custom-target
    # 2. Print custom target to a semi-glossy OBA-free paper
    # 3. Scan the target with a spectrometer
  chartread -v -H -T0.4 custom-target
    # 4. Create reference file
  spec2cie -v -i D50 custom-target.ti3 glossy.cie
    # 5. Setup light and measure spectrum, save to light.sp for later use
  spotread -a -H -x
    # 6. Shoot CC24 target, glossy target, and white card.
    # 7. Crop export and convert to cc24.tif, glossy.tif and ff.tif
    # 8. Apply flatfield
  dcamprof testchart-ff cc24.tif ff.tif cc24-ff.tif
  dcamprof testchart-ff glossy.tif ff.tif glossy-ff.tif
    # 9. Scan values
  scanin -v -dipn cc24-ff.tif ColorChecker.cht cc24.cie
  scanin -v -dipn glossy-ff.tif custom-target.cht glossy.cie
    # 10. Merge targets into one, letting CC24 have priority
  dcamprof make-target -p cc24-ff.ti3 -a cc24 -p glossy-ff.ti3 -a glossy -d 0.03 combo.ti3
    # 11. Make a preliminary profile, dumping plots. Exclude glossy
    #     from the matrix optimizer to get as good base match as
    #     possible for the important normal colors represented by cc24
    #     Add -C (no CAT) to all make-profile below if your light is
    #     close to D50.
  dcamprof make-profile -r dump1 -w cc24 0,1 -w glossy 0,0 -i light.sp combo.ti3 preliminary.json
    # 12. Start gnuplot (cd dump1; gnuplot -background gray) and plot
    #     target and LUT, plus LUT stretch vectors and DE vectors.
    #     Use 'set view equal xyz' and 'set view equal xy' to turn
    #     on/off scaling of lightness axis, must be turned off if
    #     error vectors are viewed in 3D.
  gnuplot> splot 'nve-lut.dat' w l lc "beige", 'gmt-locus.dat' w l lw 4 lc rgb var, \
    'gmt-adobergb.dat' w l lc "red", 'gmt-pointer.dat' w l lw 2 lc rgb var, \
    'target-nve-lutve2.dat' w vec lc "black", 'target-nve-lutvm.dat' w vec lw 2 lc "olive", \
    'targetd50-xyz.dat' pt 4 lc rgb var, 'targetd50-xyz.dat' using 1:2:3:5 w labels offset 3
    # 13. Look in the plot for patches that pull in opposite directions and cause a
    #     bad bend in the LUT. Add those (typically one or two) to an
    #     exclude.txt and render
  dcamprof make-profile -r dump1 -x exclude.txt -w cc24 0,1 -w glossy 0,0 -i light.sp combo.ti3 preliminary.json
    # 14. Make matrix-only and full correction profiles for sanity
    #     check comparisons later
  dcamprof make-dcp -n "Canon EOS 5D Mark II" preliminary.json no-relax.dcp
  dcamprof make-dcp -n "Canon EOS 5D Mark II" -L preliminary.json matrix.dcp
    # 15. Relax the LUT (primarily for the glossy class) to improve
    #     smoothness. Re-render and replot for each change.
    #  a) It may be worthwhile to lock the matrix before changing DE k
    #     weights: save profile to separate file matrix.json and
    #     provide -m and -f parameters.
    #  a) Try relaxing DE weight even for the important CC24, setting
    #     it to at least 1 leads often to some relax without much loss
    #     in accuracy.
    #  b) Try changing CIEDE2000 k weights, 4,1,1 good start (less
    #     weight on lightness)
    # Example result after iterating:
  dcamprof make-profile -r dump1 -x exclude.txt -f matrix.json -m matrix.json \
    -w cc24 1,1,4,1,1 -w glossy 2,0,4,4,1 -i light.sp combo.ti3 final-1.json
    # 16. Make final DCP, sanity check it by comparing it with
    #     no-relax.dcp and matrix.dcp. It should have better high
    #     saturation correction than matrix.dcp, and not lose too much
    #     accuracy compared to no-relax.dcp
  dcamprof make-dcp -n "Canon EOS 5D Mark II" -d "Final 1" final-1.json final-1.dcp
    # 17. Repeat steps 5 - 16 for the second illuminant
    # 18. Decide which tone curve to use, which baseline exposure
    #     offset if any, and if black subtraction should be
    #     automatically applied or not. Often the default "acr" curve
    #     with no baseline exposure offset provides what you want,
    #     unless you more closely want to match camera's JPEGs. If you
    #     prefer to make visual accuracy comparisons with tone curve
    #     applied, you can apply the curve earlier in the process.
    # 19. Merge to a dual-illuminant profile and apply your curve with
    #     DCamProf's neutral tone reproduction operator (enabled per
    #     default).
  dcamprof make-dcp -n "Canon EOS 5D Mark II" -d "My Profile" -i StdA -I D50 -b 0.1 -t curve.rtc final-1.json final-2.json final-dual.dcp
Title: Re: DCamProf - a new camera profiling tool
Post by: Iliah on July 08, 2015, 10:36:07 am
>> power supply control to stabilize light is another one).

> I am quite interested in that (assuming you are talking about halogen sources) - would you point me in a right direction?

Much depends on the power supply you are using currently. If it is not a stabilizing one, you can use a dimmer with a sine wave output in front of it and control the dimmer using the feedback from the diode. If it is stabilizing, you may want to replace it with a regulated power supply, or introduce a dimmer right after it.
Title: Re: DCamProf - a new camera profiling tool
Post by: Alexey.Danilchenko on July 08, 2015, 11:03:16 am
> I am quite interested in that (assuming you are talking about halogen sources) - would you point me in a right direction?

Much depends on the power supply you are using currently. If it is not a stabilizing one, you can use a dimmer with a sine wave output in front of it and control the dimmer using the feedback from the diode. If it is stabilizing, you may want to replace it with a regulated power supply, or introduce a dimmer right after it.

No idea really, the one that is in the light source and it seems a rather primitive one: transformer with small board and regulator which provides 0..+17.6V on lamp connector.
Title: Re: DCamProf - a new camera profiling tool
Post by: Iliah on July 08, 2015, 11:07:33 am
No idea really, the one that is in the light source and it seems a rather primitive one: transformer with small board and regulator which provides 0..+17.6V on lamp connector.

Than you may want to see if you can control the regulator, if it is a variable resistor you may be able to replace it with either analog output of Arduino, or with a digital potentiometer like https://www.sparkfun.com/products/10613
Title: Re: DCamProf - a new camera profiling tool
Post by: Alexey.Danilchenko on July 08, 2015, 11:22:45 am
Than you may want to see if you can control the regulator, if it is a variable resistor you may be able to replace it with either analog output of Arduino, or with a digital potentiometer like https://www.sparkfun.com/products/10613

Yep its a variable resistor (old and large). At first attempt I was trying not to change the light intensity - but set it to max output at 17.6V (I have Philips halogen bulb - this one (http://www.lighting.philips.com/main/prof/lamps/special-lamps/optical-medical-equipment/halogen/halogen-reflector/923915819104_EU/product)), measure all SPDs and photodiode output with 5nm steps, then take a series of raws and correct them following the measured SPDs (to max SPD and adjusting by photodiode measurement deviation). The light stayed the same throughout the experiment.

If I understand you correctly you are saying that I need to build a calibration table and then attempt to control the light output with it using photodiode as a feedback adjusting to the same level (presumably blue one as it is the weakest)? If this is correct then I'd need that rotary encoder to know what wavelength it is at since photodiode output although linear is different in different parts of the spectrum.
Title: Re: DCamProf - a new camera profiling tool
Post by: Iliah on July 08, 2015, 11:44:10 am
First, I would inspect the resistor (may take de-soldering), read the markings, see how it is used (potentiometer, rheostat with two pins connected together, or one pin free). It is also important to measure the current through the resistor. Next, I would figure out how to replace it with a processor-controllable element.

The goal is to maintain the light flux. This is usually achieved through a simple PID regulator (many implementations are available for microprocessor, it is bread-and-butter code - an example http://playground.arduino.cc/Code/PIDLibrary ). The input to the regulator is the measurement of the light (diode reading), the output is the "variable resistor" control signal.
Title: Re: DCamProf - a new camera profiling tool
Post by: Alexey.Danilchenko on July 08, 2015, 11:59:48 am
Thanks Iliah will do that
The goal is to maintain the light flux.

Do you mean luminous flux or radiant flux? The photodiode does not allow the radiance hence the tables and SPD or am missing something and the simple alignement  of light output to have the same photodiode readings will do?
Title: Re: DCamProf - a new camera profiling tool
Post by: Iliah on July 08, 2015, 12:10:32 pm
> the simple alignement  of light output to have the same photodiode readings will do

Yes, that will do, works from the times of early "colour stabilizers" in enlargers. If one wants to go overkill, TCS3471x / TCS3472x and its variations are just what the doctor ordered :)
Title: Re: DCamProf - a new camera profiling tool
Post by: Alexey.Danilchenko on July 08, 2015, 12:24:01 pm
> the simple alignement  of light output to have the same photodiode readings will do

Yes, that will do, works from the times of early "colour stabilizers" in enlargers. If one wants to go overkill, TCS3471x / TCS3472x and its variations are just what the doctor ordered :)
So with this then there is no need to scale raw values to adjust for intensity difference (provided all exposures were the same)? If so great - that is easier for me than taking 66 i2pro measurements and processing them all. Thanks Iliah for the help
Title: Re: DCamProf - a new camera profiling tool
Post by: Iliah on July 08, 2015, 12:37:42 pm
> So with this then there is no need to scale raw values to adjust for intensity difference (provided all exposures were the same)?
Generally, yes; subject to periodic testing, taking SPD, and calculating CCT to log the aging of the lamp (I do it after each 10 hours of use approximately).
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on July 09, 2015, 09:58:23 am
at least it seems I have a chance to get a "proper" target, will see how it measures then... albeit if they state that the formulation was changed it still leaves the question about some /data/ basis vs which to compare your own measurements to see if something is out of the ordinary (unless they have a stock of legacy targets from the old days somewhere), like with this one from Jan 2014 - I am curious to see how all the devices I can use will measure the black patches if I /hopefully/ receive a replacement...

so after some ping-pong with X-Rite's customer service about RMA & proof of purchase I received today a replacement for my ColorChecker SG target... the one that was sent by X-Rite is "November 2014" edition (as printed on the back of the target) now, so this evening I will try to measure it and see how the patches are there (or rather how my spectrophotometers will be metering them).
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on July 10, 2015, 09:50:31 am
So, they knew it is Jan 2014 beforehand. Does not add much credibility to their "only".

I did one measurement (did not have time to do several to average yet) with a newer out of 2 spectrolinos that I have = attached

Results looks "better", at least black patches are darker in November 2014 edition of CC SG.
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on July 26, 2015, 02:55:47 pm
I've just released version 0.8.3: http://www.ludd.ltu.se/~torger/dcamprof.html

The main thing is that I've reworked the tone reproduction operator as a part of the work of providing a realtime version for RawTherapee (which I just committed too). It works along the same principles and the result of it is almost the same as before, but there are some slight improvements especially concerning highlight rolloff. I've had some extra expert eyes to help me fine-tune weights. It has a bit better skin tone performance now.
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on July 26, 2015, 09:28:20 pm
0.8.3 build for Windows (mingw = dcamprof.exe + libgomp_64-1.dll + HTML & PDF manual / = copy of Torger's web page and the same converted to PDF /) : https://app.box.com/s/6dbmzwrvucjz6tprdhvit2cnjpdn3mo1
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on July 27, 2015, 03:31:09 am
Seems like tone reproduction can keep one busy for a while.

With DCamProf the intention has from the start to make neutral profiles, but as soon as contrast should be applied it's impossible not to bring some taste to the table.

To keep color appearence with increased contrast I've noted that non-linear 3D tables are required. So far I keep hue stable, but saturation is changed depending on contrast, and also depending on lightness (more saturation increase of dark colors) and source saturation. This is not to create a "look", but to keep color appearance as constant as possible.

With the latest release I've made it possible for the user to tune all these weighting parameters with a JSON configuration file.

Another big thing when it comes to tone reproduction is highlight rolloff. It's often possible to retain the correct color close to clipping, but I've noted that there will then be problems with the look. We can't change that the brightest spot on the media is white, so making a nice transition to the whitepoint is an important aspect, even if it hurts color precision.

As said DCamProf doesn't attempt to do any subjective looks so far. I might add features for that at some point, but it will be difficult to use without a GUI, I would have to provide pre-made recipes that the user can adjust to taste. A typical base recipe is to warm up highlights/cool down shadows, limit saturation of high saturation colors, increase saturation of mid-saturated colors (being careful with skintones). I think it's better to control the look in the raw converter, but unfortunately few raw converters have tools that allow these types of fine adjustments.

The attached image shows a comparison between a DCamProf profile as used in RT and Hasselblad's native Phocus rendering and illustrates a neutral vs subjective look, the camera is a H4D-50. At first glance the images are very similar, and one reason for that is that I have matched the contrast curve quite well (I like my profiles to provide the same contrast as the native camera images so default rendering becomes predictable). Another reason is that Hasselblad's color is quite sane, not crazy pop as you can see in some consumer cameras, and is thus fairly close to the neutral result.

However looking closer we see some differences, the main one that it seems like the Hassy image has a warmer white balance. This is not the case though, the white balance multipliers are exactly the same, and if we look closer at neutrals and shadows the white balance seems the same, what the Hassy look does is to warm up some of the saturated tones. The Hassy look has a little bit more saturation overall, but the difference is certainly not huge. Warming up of tones does cause some notable hue errors, here seen in the red pants that look more yellow-red rather than the actual wine-red.

The example image is not good for comparing skin tones, but anyway I've looked at other more suitable pictures for that and quite big differences can seen there, much more than in the attached example. Hassy's subjective look make caucasian skin quite a lot more yellow/golden, while a neutral look has more red as there is in reality. Golden skin looks smoother though so I'd expect many prefer that look, and that's why Hassy have hard-coded it into their profile.

Since I've published DCamProf I've got quite many requests on making profiles for camera XYZ as the manufacturer provided profile "is no good". I've been a bit reluctant to help as I know there is a big chunk of subjectivity involved. Making a neutral realistic profile is now quite easy with the tone operator in place, but I'm not so sure that is what most people want especially when it comes to skin tones.
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on July 30, 2015, 11:11:28 am
I got a request to be able to disable the lightness correction all-together, and with the new 0.8.4 release this is now possible, just specify "-l -1,0"  to make-profile and the LUT will only work with chromaticity.

As a sidenote this is how Adobe's DNG Profile Editor does, the LUT only corrects hue and saturation and leave lightness (actually RGB-HSV value) unchanged, or rather leave it at the result provided by the matrix. There is some robustness advantages to this, measurement errors in lightness is larger (I think) than in chroma and hue and there seems to be some robustness advantages in handling of clipped highlights.
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on July 30, 2015, 12:15:21 pm
0.8.4 build for Windows (mingw = dcamprof.exe + libgomp_64-1.dll + HTML & PDF manual / = copy of Torger's web page and the same converted to PDF /) : https://app.box.com/s/if132xhpl94u32jb4kg9h43oaz197xc9
Title: Re: DCamProf - a new camera profiling tool
Post by: Bart_van_der_Wolf on July 30, 2015, 01:05:33 pm
Thanks guys, for the ongoing improvements, and Windows binaries.

Cheers,
Bart
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on August 04, 2015, 06:10:30 pm
Here's a guess game for you;

I'm working with subjective tuning features in DCamProf so you can make the profile render "flattering" skin tones rather than accurate (and any other subjective tunings you may want to do). It's not exactly easy tuning without a GUI, but it can be done. You write a specification of the subjective adjustments in a JSON config file.

The attached image shows three images, one is a neutral DCamProf profile with a tone curve using the neutral tone reproduction operator, one is my work in progress concerning hand-tuned skin tones using DCamProf, and one is a Hasselblad Phocus original rendering. The camera is a H4D-50.

So which one is which? And which skin tone rendering do you prefer?
Title: Re: DCamProf - a new camera profiling tool
Post by: Tim Lookingbill on August 04, 2015, 09:06:46 pm
Can't tell you which one is which but I prefer 'A''s skin tone.

Based on 40 years of observation I can say all skin has some yellow in it at any given time of day, shade or sunshine unless the subject is an albino.

'B' & 'C' are a bit too pinky where I see the tuft of hair just above the forehead has a magenta tinge. Being able to get rid of any Rosacea or other skin conditions that show pinkish hues in a profile is a big plus. I have to make hue adjustments constantly using ACR/LR's HSL.

Good work, torger. Thanks for posting samples. It helps me make sense of this thread.
Title: Re: DCamProf - a new camera profiling tool
Post by: Iliah on August 04, 2015, 11:13:41 pm
Agree with Tim, skin needs to contain a little yellow, even for Northern European children. Hemoglobin + Melanin + skin thickness does that. "A" for me, too.
Title: Re: DCamProf - a new camera profiling tool
Post by: ErikKaffehr on August 04, 2015, 11:24:18 pm
Hi,

I guess I like 'A' best followed by 'B'.

I agree with Tim and Iliah on what has been said. But, I have little experience with skin.

Best regards
Erik


Can't tell you which one is which but I prefer 'A''s skin tone.

Based on 40 years of observation I can say all skin has some yellow in it at any given time of day, shade or sunshine unless the subject is an albino.

'B' & 'C' are a bit too pinky where I see the tuft of hair just above the forehead has a magenta tinge. Being able to get rid of any Rosacea or other skin conditions that show pinkish hues in a profile is a big plus. I have to make hue adjustments constantly using ACR/LR's HSL.

Good work, torger. Thanks for posting samples. It helps me make sense of this thread.
Title: Re: DCamProf - a new camera profiling tool
Post by: ah693973 on August 04, 2015, 11:40:08 pm
My preference would be "B" as the one I perceive is most accurate.

"A" looks nicer to me, but the lighting looks pretty cool and the skin tone looks warmer (almost unnaturally so). "A" looks more like what I would expect with the sun out (more yellow, less blue).

Andy
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on August 05, 2015, 03:16:41 am
Here's the answer:

A: DCamProf with the neutral tone reproduction operator, that is the most "accurate" colors, no manual tunings (except for the smoothness/accuracy tradeoff during colorimetric design)
B: DCamProf using the upcoming feature set that allows hand-tuning colors, here with my own skin-tone tunings
C: Hasselblad Phocus original rendering

Personally I think Hasselblad puts too much pink/magenta into skin tones, but it will depend on the light how they appear. Fixing things in a profile is a crude way to do things as it will be static adjustments which will work better in some images and worse in others so I'm not really sure it's a good idea at all. It's how every commercial raw converter does it so I thought it would be interesting to provide the possibility to do the same with DCamProf.

The adjustments I've made: compressed the skin tone hue range so that reds and yellows are brought closer together, adjusted the overall tone, added a tad more saturation on low saturation skins (to avoid grayish look on pale people). The range that fall on lips (and often cheeks) has got a tiny magenta and lightness boost (less so than Hassy's rendering) which is supposed to separate lips more and add a "freshness" to the skin.

The midtones and highlights has got a warming up, but that's for landscape (one can see it on the tone of the wall), I'm not really happy with that adjustment yet so I may revert/change that part and adjust skin tone range accordingly so it stays about the same as now. I'm quite pleased with the current skin tone result, but I'm not sure I'd use it myself over the original neutral rendering. I just want to test and see that it can be done and have an example file before I release this functionality.

The white balance is same for all (daylight) but since it's in the shade the look is a little bit on the cool side, maybe should have had a shade white balance here but it was a bit too yellow I thought. I've used several other shots in other light conditions as well when doing this tuning. I took this one as an example to show as the differences where a bit easier to see here. In bright sunlight the results are a bit closer. The wall is in reality not really neutral gray but has a bit of blue in it which may amplify the cool look.

It's interesting that most of you prefer the A rendering. Maybe the secret to skin tones may just be to render them as accurately as possible :-)
Title: Re: DCamProf - a new camera profiling tool
Post by: Torbjörn Tapani on August 05, 2015, 03:41:56 am
I was going to say I think A is hasselblad and C neutral. But clearly that was wrong. B has that tinge to the hair that made me think it is a work in progress.
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on August 05, 2015, 04:15:46 am
I was going to say I think A is hasselblad and C neutral. But clearly that was wrong. B has that tinge to the hair that made me think it is a work in progress.

Well spotted, I'd stared so much on the skin I did not notice that. I thought it was a moire effect at first, but it's most likely the lip/cheek magenta boost that taints the hair. It's the generic problem of subjective profiles, no local adjustments are possible so any colors in the same range will be modified even if it's not skin.

Another issue is that subjective profile adjustments are also very sensitive to white balance setting as it moves around the colors, that is the skin tone hue range varies a bit depending on white balance.
Title: Re: DCamProf - a new camera profiling tool
Post by: Iliah on August 05, 2015, 09:52:02 am
> Maybe the secret to skin tones may just be to render them as accurately as possible :-)

It is a cultural thing. My Far East customers always prefer less saturated skin tones on their portraits.
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on August 05, 2015, 10:09:04 am
> Maybe the secret to skin tones may just be to render them as accurately as possible :-)

It is a cultural thing. My Far East customers always prefer less saturated skin tones on their portraits.

did they ask you to photoshop their eyes too :) ?
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on August 05, 2015, 11:15:25 am
I've just released v0.9.0, the big new things is of course the "look operators" feature so you can design a look. It's really not intended to make crazy "filters", but really just very subtle adjustments from the default neutral look.

Probably it is like only myself that will ever use it :) , because it's really tedious to work with without a GUI, but if you're the patient type of person you can do it.

I've included one example look, which is an updated version of the one posted in an example a few messages back. It should be used as a template example only, as adjustments are really fine it will probably vary if it will look well or not on a profile made for a different camera, and also depending on the chosen contrast curve.

It's also a time-wasting experience to tune these looks, you can get stuck like forever on making an almost invisible adjustment, so I just said "enough is enough" and made the release instead :). For the specific test images I've had I think my look became more pleasing than Hasselblad's own, but it's of course a matter of taste :).
Title: Re: DCamProf - a new camera profiling tool
Post by: Bart_van_der_Wolf on August 05, 2015, 11:24:44 am
did they ask you to photoshop their eyes too :) ?

Don't know about their eyes, but I saw a report that they do go to the beach at night, to avoid a suntan...

And to get back on topic, I feel an urge rising within myself for some sort of GUI, but I do not have the time right now :(
Maybe there are other takers?

Cheers,
Bart
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on August 05, 2015, 11:30:12 am
0.9.0 build for Windows (mingw = dcamprof.exe + libgomp_64-1.dll + HTML & PDF manual / = copy of Torger's web page and the same converted to PDF /) : https://app.box.com/s/z5zu2zsw84tbiof65esgn66d8uz6cz2u
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on August 05, 2015, 11:40:28 am
And to get back on topic, I feel an urge rising within myself for some sort of GUI, but I do not have the time right now :(
Maybe there are other takers?
too few users I think... I can see that for example the 0.8.4 build for Windows was downloaded 12 times... OK, some are on Mac (or even Linux/whatever) and some do build themselves... but I 'd guess not more than 50 users at all in a best scenario... fewer are actually using it so regularly...
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on August 05, 2015, 12:00:50 pm
When it comes to GUIs I'd prefer to make it myself in my company and in that case it would be a commercial product, but of course anyone's free to make a GUI for the current program following the GPL terms.

Looking at hits on the web page I'd say the interest is quite broad, but few users have even seen a command line so those that actually proceed to download are quite few. If I'd really make a GUI for the complete workflow, which is quite a big job, I think it would see quite many users.

But it's a speciality tool you wouldn't use often, and still requires quite some skill and patience to use even with a GUI so it wouldn't be any big volume software for sure.

The command-line only Argyll has a stable following though, so we'll see in a few years. DCamProf is still new software.

The only part that is truly tedious without a GUI is the subjective design, all other aspects would not really be that much simpler with a GUI. Concerning the design aspect I do it in RawTherapee where you easily can load new profiles. I render with monochrome selections to fine-tune which gamut area an operator should apply on, and once I've done that I start tuning the actual operator values.
Title: Re: DCamProf - a new camera profiling tool
Post by: viahorizon on August 06, 2015, 06:09:04 am
Just a note that there might be more people who watch the thread but don't download/participate just yet. Like myself.

It seems like a terrific tool and I'd like to compile it for Mac in the future.
Title: Re: DCamProf - a new camera profiling tool
Post by: WayneLarmon on August 06, 2015, 09:25:49 am
Quote
Just a note that there might be more people who watch the thread but don't download/participate just yet. Like myself.

I'm in this category.   I've been watching this thread like a hawk, but haven't done anything with DCamProf yet because I'm still working on getting a handle on Argyll recipes (http://forum.luminous-landscape.com/index.php?topic=97262.0).

Yeah, once you have the recipes worked out, there is no real need for a GUI.  (At least for Argyll.)

Wayne

Title: Re: DCamProf - a new camera profiling tool
Post by: torger on August 06, 2015, 03:01:18 pm
I am working on a tutorial article which is supposed to provide a step-by-step guide of how to design a high end profile using DCamProf, but it's still a fairly long way to go. Hopefully within a few months. There's so much to investigate while doing it, sometimes ending up in having to develop yet another feature in the software, that is what makes the time fly...
Title: Re: DCamProf - a new camera profiling tool
Post by: one/and/a/half on August 07, 2015, 01:45:52 am
I've compiled the sources on a Mac with OpenMP. Feel free to use it and link it directly.

http://projectsbin.com/resources/dcamprof (http://projectsbin.com/resources/dcamprof)
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on August 11, 2015, 08:22:41 am
Working on a new release with some new features, but now just ran into a pretty severe limitation of the DNG profile format, or actually the DNG pipeline which applies it.

The DCP LUT is HSV-based. Hue is changed by adding an offset. To interpolate between table entries you calculate an average of the neighbors.

Say if entry A says hue -30 degrees, and entry B says +40 degrees and we want to interpolate exactly inbetween, then we get (-30 +40) / 2 = 5 degrees, just as we would expect. But say entry A is -170 degrees and entry B is +160 degrees, then we get (-170 + 160) / 2 = 5 degrees, which indeed numerically the average but as hue goes around 360 degrees the average should be (360-170 + 160) / 2 = 175 degrees in his case.

These cases could easily be handled when the LUT is applied, the problem is that the DNG reference code doesn't do that, and I assume Adobe Lightroom doesn't either (haven't tested Lightroom yet). This means in practice that hue rotation is limited to the range -90 to +90, which indeed should be enough for making colorimetric corrections, but is an issue when designing looks, even with very mild looks there is a significant risk to run into this discontinuity. The reason for this is that for lower saturation colors hue rotation can be quite huge in the RGB-HSV space despite that there is a small adjustment in visible space.

It's actually not a limitation of the LUT format itself, but how the LUT is applied. Forgetting about hue angle discontinuity is a classic bug when working with a coordinate system like HSV, JCh or other that has hue defined as an angle; been there, done that. It's a bit unfortunate that Adobe has limited their DNG profile format by having this bug in the DNG pipeline, and I'm quite sure that they'd call it a "feature" themselves as you can optimize LUT application if you don't need to care about discontinuity.

I have not yet figured out how to deal with this. If you don't use the look operators it's quite unlikely that you run into it though.

EDIT: you're not limited to +/-90 degrees (although that will guarantee that you are safe), what you need to avoid is discontinuity, that is a jump from say +179 to -179 between table entry neighbors.
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on August 11, 2015, 03:55:04 pm
Just released version 0.9.1

The hue discontinuity problem can't really be worked around, but it's a quite small problem. By adding an RGB blend mode and HSV axis in the look operators it's quite easy to make subjective looks that doesn't trigger hue discontinuity.

DCamProf will now detect hue discontinuity in the make-dcp command and abort if found.

I've added TIFF file versions of the patch matching reports so you can see those diagonally split square patches like various patch tools show.

The test-profile command can now be run without a target if you just want to render plot files, and a new generated gradient file which can be used for visual inspection of smoothness. It's very useful when designing a look using look operators.
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on August 11, 2015, 05:36:37 pm
0.9.1 build for Windows (mingw = dcamprof.exe + libgomp_64-1.dll + HTML & PDF manual / = copy of Torger's web page and the same converted to PDF /) : https://app.box.com/s/lr054ihziq8iv248a780q67fveo7boqx
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on August 12, 2015, 04:25:44 pm
Done some experiments with Capture One again (it was a while ago). Anyway I've discovered an issue, well I kind of new about it but had forgot;

When using DCamProf profiles in C1 high saturation blues sometimes clips to black.

This doesn't happen when the same profile is used in RawTherapee.

I think I know what the reason is. The ICC profile defines colors with a LUT from Raw RGB to CIELab. Then C1 converts Lab (which is virtually boundless) to some sort of RGB for output. If the Lab color is outside the target RGB gamut it will clip, and for blue this seems meaning to go from 100% blue down to 0%. Why this happens in C1 and not RT (except if you artificially push it) is because C1 is using a small RGB space and RT is using ProPhoto.

This means that for Capture One the ICC profile must do some sort of gamut mapping to avoid clipping the output space.

The problem is that I don't know what Capture One's output RGB space is (so I don't know what to gamut map against), and if it's the same for all camera models. Anyone that knows?
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on August 13, 2015, 10:19:34 am
The problem is that I don't know what Capture One's output RGB space is (so I don't know what to gamut map against), and if it's the same for all camera models. Anyone that knows?
do you mean that C1 does the final output in 2 steps : -> some unknown intermediate RGB space with smallish gamut -> RGB space designated by you, yourself, for output in C1's UI ?
Title: Re: DCamProf - a new camera profiling tool
Post by: brntoki on August 13, 2015, 11:54:16 am
Done some experiments with Capture One again (it was a while ago). Anyway I've discovered an issue, well I kind of new about it but had forgot;

When using DCamProf profiles in C1 high saturation blues sometimes clips to black.

This doesn't happen when the same profile is used in RawTherapee.

I think I know what the reason is. The ICC profile defines colors with a LUT from Raw RGB to CIELab. Then C1 converts Lab (which is virtually boundless) to some sort of RGB for output. If the Lab color is outside the target RGB gamut it will clip, and for blue this seems meaning to go from 100% blue down to 0%. Why this happens in C1 and not RT (except if you artificially push it) is because C1 is using a small RGB space and RT is using ProPhoto.

This means that for Capture One the ICC profile must do some sort of gamut mapping to avoid clipping the output space.

The problem is that I don't know what Capture One's output RGB space is (so I don't know what to gamut map against), and if it's the same for all camera models. Anyone that knows?

I'm pretty sure that C1's native color space is ProPhoto, if that is what you're looking for.
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on August 13, 2015, 12:11:22 pm
I'm pretty sure that C1's native color space is ProPhoto, if that is what you're looking for.
what is the basis of being "pretty sure" though ?

for example http://help.phaseone.com/en/co6/output/learn-more-about-file-formats/colors-in-capture-one.aspx?p=1

Quote
Capture One works in a very large color space, similar to that captured by camera sensors. A large color space ensures that little clipping of the color data can occur. Clipping is the loss of image information in a region of an image. Clipping appears when one or more color values are larger than the histogram (color space of the output file).

At the end of the workflow, the RAW data has to be processed to pixel based image files, in defined color spaces. These spaces are smaller than the internal color space used by Capture One.

you can output to ProPhoto RGB for example, but it is claimed to be smaller ;) than whatever C1 is using...

so do we have any P1 people on record or something of the same quality ?

Title: Re: DCamProf - a new camera profiling tool
Post by: torger on August 13, 2015, 12:15:20 pm
I'll back off a little from the claim that it's a RGB space clipping problem. The profile itself does clip to black in the blue area as it goes outside the valid XYZ range, for strange input like raw R = 0, G = 0, B = 100. I suspect that if I smooth away the "black hole" in the LUT it will work fine in C1 regardless of their output space. Still would be nice to know what it is.

Smoothing/interpolating away the black hole is easier said than done though... I'll experiment some more.
Title: Re: DCamProf - a new camera profiling tool
Post by: Alexey.Danilchenko on August 14, 2015, 04:23:18 am
what is the basis of being "pretty sure" though ?

for example http://help.phaseone.com/en/co6/output/learn-more-about-file-formats/colors-in-capture-one.aspx?p=1

you can output to ProPhoto RGB for example, but it is claimed to be smaller ;) than whatever C1 is using...

Not directly though I can see how that conclusion can be made. I would have thought that they talk about ERIMM (linearised ProPhoto eseentially proposed by Kodak). But if the above reading is correct then I guess they might just use LAB in some shape (normal LAB, UpLab etc) to have a "very large color space, similar to that captured by camera sensors".
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on August 14, 2015, 07:47:50 am
Just released 0.9.2, which has a fix for the ICC blue-goes-black problem.

When fixing the "black hole" the problem in C1 indeed disappeared, the easiest way was to interpolate from defined neighbors. Thus my initial assumption was incorrect, I don't need to know the Capture One output color space is, fortunately.

I have now tested a complete workflow in Capture One 8 (as it was a long time ago I tested) making both linear profiles and with a tone curve using DCamProf's neutral tone reproduction operator. It all seems to work as intended.

I've updated the basic workflow docs both for the ICC and DNG workflow to show typical weighting parameters and the use of tone curves, that is making the basic workflows a bit more informative so the casual user can get more out of DCamProf.
Title: Re: DCamProf - a new camera profiling tool
Post by: Alexey.Danilchenko on August 14, 2015, 08:03:28 am
Just released 0.9.2, which has a fix for the ICC blue-goes-black problem.
The downloads section still references 0.9.1
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on August 14, 2015, 08:51:11 am
The downloads section still references 0.9.1

Ooops! Fixed the link. Thanks for reporting.
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on August 14, 2015, 09:30:14 am
0.9.2 build for Windows (mingw = dcamprof.exe + libgomp_64-1.dll + HTML & PDF manual / = copy of Torger's web page and the same converted to PDF /) : https://app.box.com/s/xkiv79m5shimsrw7alq8cxdvvczjfh7x
Title: Re: DCamProf - a new camera profiling tool
Post by: brntoki on August 14, 2015, 12:25:53 pm
what is the basis of being "pretty sure" though ?

for example http://help.phaseone.com/en/co6/output/learn-more-about-file-formats/colors-in-capture-one.aspx?p=1

you can output to ProPhoto RGB for example, but it is claimed to be smaller ;) than whatever C1 is using...

so do we have any P1 people on record or something of the same quality ?



I'm sorry. If I could remember I would have provided that information. If memory serves, and there is a real good chance it isn't in this case, I heard that from a C1 guy on a forum. I would have thought this one, but if no one else is sure, that probably isn't true.
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on August 14, 2015, 04:55:29 pm
Played some more with Capture One tonight.

I wanted to make sure that their film curve really is a pure RGB curve, and indeed you can quite easily "copy" the curve by replicating it in the RGB curve tool. As we know RGB curves has some color shift issues. However they avoid the worst shift (and over-saturation) issues by not going below y=x in the shadow range of the S, the curve is above the linear curve throughout the range. It also seems overall a bit low on saturation (at least the P45+ which I used in testing), which does keep the colors more stable when applying an RGB curve.

It's still quite clear that the profile is designed for the standard curve, which causes some undersaturation and shifts when using the linear curve. Yellow-orange-red tones are good for spotting issues.

Today DCamProf will apply the curve to the ICC LUT, that is you must always use the "linear response" curve in Capture One (don't use linear scientific as it disables highlight reconstruction). I've done some tests and I haven't so far seen any disadvantages of the approach. I made some brief experiments to reverse the curve so you could have profiles designed like C1, that is no curve in the LUT but the right colors only appear when the intended film curve is applied. I didn't get it to work though, and I'm not sure it ever will, but I may try a bit more.

The thing is that C1, just like Hassy's Phocus, seems to have embraced the "RGB curve look" to some extent. If you like DCamProf have a different tone curve model then it means the RGB curve works very much against you, meaning that you need to "pre-distort" colors to quite much to make the intended look after an RGB curve has been applied. This may cause the profile look outright bad with a linear curve. The C1 bundled profiles still look fine with a linear curve as they haven't tried to work too much against the RGB curve quirks.

That is even if I succeed making DCamProf ICCs without tonecurve embedded for C1 it will likely still be a bad idea as it's not really an RGB style profile and is thus not as flexible as the bundled ones concerning changing curve. I will most likely not spend time on making an "RGB-friendly" tone reproduction operator as I'm not into the RGB look myself.
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on August 15, 2015, 04:34:49 am
Here's an example to show the differences between different type of tone reproduction. I've borrowed one of Erik's P45+ samples as it had a perfect strong but plain blue gradient sky for showing the differences more clearly.

The reference is a low contrast linear rendering, which shows the original color as is. I made it low contrast and brighter to make it more easy to compare with the S-curve renderings. Then we see how the RGB curve modulates the color, you get a bit of yellow into the sky. The reason is that blue is compressed more than the other channels that is the color gets less blue which means more yellow.

Adobe identified this problem and made a special variant of the RGB curve for Lightroom/ACR/DNG. It makes sure that RGB-HSV hue is kept constant. This way you don't get the color shift towards yellow. However if you make a gradient along RGB-HSV/HSL from blue to white it will not really look hue constant (see attached example), as it's not really a perceptually uniform space. This leads to that you typically get a little magenta feel to the sky.

DCamProf's tone curve (tone reproduction operator) uses CIECAM02 with some extra special tricks. Pure CIECAM02 will give you extreme highlight rendering issues. The camera clips so we must eventually blend into white which will be too abrupt with a too strong perceptual uniform model. With DCamProf I've tried to find a balance in the rolloff by mixing CIECAM02 with RGB-HSV. I'm thinking about adjusting the rolloff and actually make it non-uniform over tones. The skin-tone range seem to gain from a longer rolloff with more desaturation, while the sky blue range seems to gain from a shorter rolloff. For the blend-into-whitepoint-problem there's not much in color science models to find (as far as I know), so you have to come up with your own perceptual transition.

The examples here are rendered in RT but the RGB yellowish sky is also seen in Capture One's native rendering - that is Phase One uses the RGB color shift as a look feature and don't compensate for it that much, which makes their profiles work less bad with linear response than they otherwise would. Obviously if I make a DCamProf profile that "pre-distorts" the colors to end up right after applied RGB curve the colors will look bad with a linear response. Thus applying the curve directly to the ICC LUT is the right thing to do for every profiling software that doesn't embrace the RGB curve look.
Title: Re: DCamProf - a new camera profiling tool
Post by: Bart_van_der_Wolf on August 15, 2015, 06:18:27 am
Here's an example to show the differences between different type of tone reproduction. I've borrowed one of Erik's P45+ samples as it had a perfect strong but plain blue gradient sky for showing the differences more clearly.

The reference is a low contrast linear rendering, which shows the original color as is. I made it low contrast and brighter to make it more easy to compare with the S-curve renderings. Then we see how the RGB curve modulates the color, you get a bit of yellow into the sky. The reason is that blue is compressed more than the other channels that is the color gets less blue which means more yellow.

Hi Anders,

I'm sorry that I do not have the time to analyze in detail, so what I'm going to say may be off the mark (if so just ignore).

All RGB manipulations of brightness/contrast, should be done in linear gamma space as a minimum safeguard against unbalanced shadow/highlight changes. So not linear tone curve, but linear gamma, which means that any tone curve must first be removed/reversed then brightness/contrast adjusted for all channels, then tone-curve reapplied.

That of course still does not decouple Chrominance and Luminance, which would require conversion from RGB to e.g. HSL or HSV or XYZ.
As long as calculations are performed in floating point precision, the round-off errors from multiple colorspace conversions should not become an issue.

Cheers,
Bart
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on August 15, 2015, 06:42:27 am
Not sure I follow completely, but in any case all the images above are rendered in floating point linear gamma space. I started with color stuff quite late after most things had already become linear floating point so I am quite unfamiliar with working in gamma spaces.
Title: Re: DCamProf - a new camera profiling tool
Post by: Bip on August 18, 2015, 04:18:07 pm
Hello,

First, thank you Torger, it is a very good job with Dcamprof, thank you also for the publication of AlterEgo with versions under Windows.
A question about to make a dcp profile, how do you do to make a “tiff“ of the  target for Lr? (The ideal is to generate a profile from a DNG file created with Lightroom, no?)
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on August 18, 2015, 04:41:23 pm
A question about to make a dcp profile, how do you do to make a “tiff“ of the  target for Lr? (The ideal is to generate a profile from a DNG file created with Lightroom, no?)

what do you mean ? Torger suggests using dcraw binary or rawtherapee to make .tif for argyll utilities (you can as well use rpp or libraw utilities too) ... open http://www.ludd.ltu.se/~torger/dcamprof.html and search for example for text beginning with "Basic workflow for making a DNG profile using a test target"

I am using rawdigger (profile edition) as it is more convenient (for me).
Title: Re: DCamProf - a new camera profiling tool
Post by: Bip on August 18, 2015, 04:52:58 pm
what do you mean ? Torger suggests using dcraw binary or rawtherapee to make .tif for argyll utilities (you can as well use rpp or libraw utilities too) ... open http://www.ludd.ltu.se/~torger/dcamprof.html and search for example for text beginning with "Basic workflow for making a DNG profile using a test target"

I am using rawdigger (profile edition) as it is more convenient (for me).

Yes for Dcraw or Rawtherapee, but it is better to make profil with the software (Lr for instance) that performs the "tiff" (or DNG), the color corrections are done better.
(I use Rawdigger to check clipping RGB values on the raw)
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on August 18, 2015, 05:40:02 pm
Yes for Dcraw or Rawtherapee, but it is better to make profil with the software (Lr for instance) that performs the "tiff" (or DNG), the color corrections are done better.
(I use Rawdigger to check clipping RGB values on the raw)

If you're asking how to make a tiff file that Argyll scanin can read, then you can't use Lightroom as it cannot export a linear tiff without white balancing.

What can be desirable though if you intend to use the finished profile in Lightroom is that you convert to DNG first, and then use RawTherapee or DCRaw to make the tiff for Argyll's scanin. Usually this is not necessary, but some raw formats contain calibration data which may be applied differently by Adobe's DNG converter and DCRaw/RawTherapee
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on August 18, 2015, 05:48:31 pm
I have now released v0.9.3.

I've made the neutral tone operator more flexible on the rolloff parameter so it can be specified per hue. I've seen advantages in subjective profiles to have a shorter rolloff for the skies range, and a longer rolloff for the skintones range. This maintains better sky color in landscapes, and render high key portraits with a more pleasing and nautral-looking transition to the whitepoint.

Then I've digged deeper down the Capture One converter and added some features required for making general-purpose profiles that work in the same way as their bundled.

C1's profiles are a bit messy, it seems like they apply the tone curve separately, which they do, but the also have a residual curve in the LUT. This means that "Linear Response" still means that you have some residual S-curve left. The purpose of this as far as I can tell is to minimize the RGB curve color shift. The separate curve is never a true "S" so it doesn't distort color as much, and then the LUT curve is probably added with some other technique, perhaps a Lab Lightness curve. This split approach makes the profiles work better than they otherwise would with different curves. It explains the mystery howcome C1 can produce good results despite using RGB curves as tone curves.

You can now design this type of profiles for Capture One using DCamProf, by following the workflow described in the docs. It will then get its look optimized for the default curve, but will look ok with the others, just as native profiles. Perfectionists should still design one profile per curve, as always.

What still may be left in terms of C1 support is gamut mapping. DCamProf currently doesn't do any, meaning that high saturation colors indeed get high saturation and may clip. There are ways in C1 to get them in gamut again but it seems like their approach is to have gamut compression in the profile itself. Context sensitive dynamic gamut mapping is a zillion times better of course, but the 1990s way to do it was in the profile, and it seems C1 is still there :-/.
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on August 18, 2015, 11:57:24 pm
I have now released v0.9.3.
it seems that you didn't update the webpage @ http://www.ludd.ltu.se/~torger/dcamprof.html with the text matching to what is in the dcamprof.html from the distribution
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on August 19, 2015, 12:03:13 am
0.9.3 build for Windows (mingw = dcamprof.exe + libgomp_64-1.dll + HTML manual, no PDF this time) : https://app.box.com/s/74jijebztt5nupgypmvqzqq2i5p4i4yo
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on August 19, 2015, 01:55:05 am
it seems that you didn't update the webpage @ http://www.ludd.ltu.se/~torger/dcamprof.html with the text matching to what is in the dcamprof.html from the distribution

Just looked, it matches here. Could be that the there was some cached stuff showing?
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on August 19, 2015, 02:21:26 am
Just looked, it matches here. Could be that the there was some cached stuff showing?
may be - now it shows.
Title: Re: DCamProf - a new camera profiling tool
Post by: Bip on August 19, 2015, 03:48:04 am
If you're asking how to make a tiff file that Argyll scanin can read, then you can't use Lightroom as it cannot export a linear tiff without white balancing.

What can be desirable though if you intend to use the finished profile in Lightroom is that you convert to DNG first, and then use RawTherapee or DCRaw to make the tiff for Argyll's scanin. Usually this is not necessary, but some raw formats contain calibration data which may be applied differently by Adobe's DNG converter and DCRaw/RawTherapee

Ok, thank you for the answer, I thought to do as you indicate.
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on August 19, 2015, 12:08:36 pm
What still may be left in terms of C1 support is gamut mapping.

Hmm... it's probably not gamut mapping I need, I just need to shape up the handling of those "undefined" areas of the RGB->Lab LUT, the current implementation doesn't do it good enough. It's still a difficult problem, but I have some new ideas to try. Until then the ICC profiles may have some issues with super-saturated colors.
Title: Re: DCamProf - a new camera profiling tool
Post by: Alexey.Danilchenko on August 20, 2015, 04:41:21 am
I had extracted my ProBack spectras tonight and they are rubbish. The first attempt went astray so I will need to redo it. I'd need to correct a few things as well:
- I need to invent something to connect camera to integrating sphere more tightly to prevent even slightest leaks
...
Finally managed to get all the parts and get a more secure connection camera-to-integrating sphere. Did another run of taking readings for the cameras I have at 5nm steps. Results are below.

I used Kodak ProBack 645 and Kodak SLR/n. ProBack was used without IR filter (transmittance of several IR filters measured separately so usage of them can be simulated). Both cameras were used in the mode where they are taking dark shot aftre exposure ans subtract it to reduce the noise (SLR/n uses temperature weighted subtraction).

The SLR/n spectral responses are closely repeating the shape published in the sensor documentation but are more smooth. I managed to build very decent matrix profile with them in dcamprof using only CC24.

The ProBack spectral responses however did came out quite jagged at first but sampling different areas of raw file produced smoother spectra.

All the spectral data in the form of CSV files and JSON files (for dcamprof usage) could be downloaded here (https://drive.google.com/file/d/0Bw2ZohnbXtyAbEJWNEttYVBiNEk/view?usp=sharing).

Spectral responses of Kodak SLR/n:

(http://www.aletan.com/img/s1/v21/p1410111954.jpg) (http://www.aletan.com/img/s1/v21/p1410111954.jpg)

Spectral responses of Kodak ProBack 645 + Kodak IR filter:

(http://www.aletan.com/img/s3/v41/p1411755938.jpg) (http://www.aletan.com/img/s3/v41/p1411755938.jpg)

Spectral responses of Kodak ProBack 645 + KnightOptics IR filter (it allows a bit more red but within Kodak IR filter specs):

(http://www.aletan.com/img/s2/v61/p1411755885.jpg) (http://www.aletan.com/img/s2/v61/p1411755885.jpg)


Title: Re: DCamProf - a new camera profiling tool
Post by: Bip on August 22, 2015, 11:03:32 am
Hi
After our exchanges, I tried to make a profile for Lr from a raw demosaiced with RT (Rawtherapee) by making a linear tif.
The dcp (or icc) profiles  give very good results with RT (very low delta E 2000)

With LR, dcp profile doesn't work well, there is a shift in the temperature color display and hue for the white balance, I tested dcp linear profile and with the TRC "ACR" without better results.

The command lines used:
scanin -v - G 1.0 -dipn target.tif target.ti3
dcamprof make-profile –i D55 (1) –I D50 (2) -b P11 (3) target.ti3 profil.json
dcamprof make-dcp -n "Nikon D700" -d "Profile name" –t acr profil.json profil.dcp

(1) The pattern is taken with flash
(2) The data of the test pattern are D50
(3) P11 is the most neutral patch (a and b values of L * a * b *) of the target

If you have any comments on the process ... thank you
Someone made profiles for Lr? (With what procedure)

Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on August 22, 2015, 11:25:23 am
Nikon D700

dear, dear... you have SSF/CMF for the camera ( credit = https://www.nikonschool.it/experience/infrarosso-dslr4.php ) , why bother with targets ? this graph says that with IR-Cut filter (hence not removed)

(https://www.nikonschool.it/images/infrarosso-dslr/big/sensibilita-D700-con-IRcut.jpg)

and this will help you to digitize the graph http://arohatgi.info/WebPlotDigitizer/download.html

voila...

PS: or our friend Trantor created some already if you want just to use some = http://wowcamera.info/forum/viewtopic.php?t=967&start=40
Title: Re: DCamProf - a new camera profiling tool
Post by: Bip on August 22, 2015, 12:04:38 pm
I tried with http://www.cis.rit.edu/jwgu/research/camspec/db.php (http://www.cis.rit.edu/jwgu/research/camspec/db.php) (the link is broken today), there were the SSF data for D700.

The profile is lower than that achieved with a pattern, why? I'm not sure if it was the SSF data or the way I do, I have kept the SSF data and I would have time I will make a new test

For "Trantor's profil"  thank you, I look at
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on August 22, 2015, 12:11:04 pm
The profile is lower than that achieved with a pattern, why?
what "lower" means ? low contrast ? low saturation ?

PS: if you have a loss of contrast when shooting a target to get a raw file to use in profile creation then it is natural for a profiling software to compensate (based on target measurements, where it sees that contrast has to be higher)
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on August 22, 2015, 12:31:26 pm
I tried with http://www.cis.rit.edu/jwgu/research/camspec/db.php (http://www.cis.rit.edu/jwgu/research/camspec/db.php) (the link is broken today)
I think they simply removed the stuff, I had it saved

RED:

0.00177749581440642
       0.00274445751769572
        0.0487655203252541
        0.0556496138020373
        0.0398169097399976
        0.0313482706319957
        0.0266321977506102
        0.0281100251101365
        0.0324368880182042
        0.0298626577750321
        0.0316728595832234
        0.0383465440299556
        0.0621504452454818
        0.0753729257442601
        0.0394324854435398
        0.0219264580025719
        0.0252601293128276
        0.0953911590772382
         0.494608494836299
         0.743715011217136
         0.711720027647541
          0.61752326479373
         0.512359892198394
         0.411283156067318
         0.319804683055095
         0.242359166236931
         0.165468210996257
        0.0821160418097345
        0.0262800791741362
       0.00739375747881804
       0.00285676663650002
       0.00111688551607914
      0.000471281415204309

GREEN :

0.00143354304586119
       0.00187398911552759
        0.0230803142653007
        0.0433564348087553
        0.0599136084381471
        0.0823639826379627
         0.133560455257778
         0.244984139558533
         0.359608596552274
         0.409368145681599
         0.576987109214872
         0.766192716804414
         0.900613338014063
                         1
         0.954947098200156
         0.887281275289021
         0.747135409589881
         0.624904015610591
          0.45115151851569
         0.286898667071245
         0.144885824038473
        0.0650367429493248
        0.0319470263207801
        0.0194491179982937
         0.012242204930572
       0.00814265442363087
       0.00593576724862311
       0.00422657598707906
       0.00205652351596989
      0.000881204570773891
      0.000480143028493641
      0.000237596621203383
      0.000117563261068653

BLUE :

0.00591568345365685
        0.0143952300672906
         0.374546467413338
         0.652706391660757
         0.753092472546832
          0.90218044480169
         0.911669627000415
         0.867874807032218
         0.815767336811676
         0.644026296824005
         0.460406091187772
         0.277858930228507
         0.130722325247247
        0.0640848940275101
        0.0317483362505199
        0.0150751490571673
       0.00743144412683003
       0.00536619979974889
       0.00411640917428182
       0.00265646535288544
       0.00126895149591723
      0.000756851904319394
      0.000506425955964308
      0.000332196038026167
      0.000305303155847557
        0.0003917628185241
      0.000235394162743942
      0.000188476692220806
      0.0000804704363975458
      0.0000457342402698008
      0.0000345036366345881
      0.0000279619189396018
      0.0000246591433308175
Title: Re: DCamProf - a new camera profiling tool
Post by: Bip on August 22, 2015, 12:42:12 pm
what "lower" means ? low contrast ? low saturation ?

PS: if you have a loss of contrast when shooting a target to get a raw file to use in profile creation then it is natural for a profiling software to compensate (based on target measurements, where it sees that contrast has to be higher)

Lower* : less good, less accurate profile.

*, sorry for my english...
Title: Re: DCamProf - a new camera profiling tool
Post by: Bip on August 22, 2015, 12:44:44 pm
I think they simply removed the stuff, I had it saved

Me too (thank a lot)
Title: Re: DCamProf - a new camera profiling tool
Post by: Bip on August 22, 2015, 01:04:05 pm
I tried with http://www.cis.rit.edu/jwgu/research/camspec/db.php (http://www.cis.rit.edu/jwgu/research/camspec/db.php) (the link is broken today), there were the SSF data for D700.

The profile is lower than that achieved with a pattern, why? I'm not sure if it was the SSF data or the way I do, I have kept the SSF data and I would have time I will make a new test

For "Trantor's profil"  thank you, I look at

dcp Strantor's profiles do not work well with Lr, Lr has the same behavior with the profiles of Strantor and mine.
To make a good profile, it is essential to be able to generate the linear tiff (or dng) with the soft for which the profile is made.
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on August 22, 2015, 01:06:48 pm
Lower* : less good, less accurate profile.
and how do you test for "accuracy" ?
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on August 22, 2015, 01:12:05 pm
To make a good profile, it is essential to be able to generate the linear tiff (or dng) with the soft for which the profile is made.
that again begs the question - how do you test exactly ?

here is what I get when SSF/CMF approximated (for example, not the best approximation that I made) using the matlab script from RIT (so less precise than a quality measurement with monochromator setup) and dcp profile built based on that for ACR used for a shot (raw) of an individually measurement target (passport) vs the measurements :

(http://s13.postimg.org/ih0qhg4fr/de125_1.jpg)

is it precise enough to your eye ? the test is with babelcolor patchtool :

(http://s9.postimg.org/nr68fpihb/de125_2.jpg)

no mess with "essential to be able to generate the linear tiff (or dng) with the soft for which the profile is made" in ACR case for Sony A7 (mark I).

now when I am sure that I can get that close with the SSF/CMF I approximate with ACR settings specific for such comparison (process 2010, etc) then I can, if I want, go into some creative mode with extra features that DCamProf provides (but I don't - leaving all that for postprocessing in photoshop)
Title: Re: DCamProf - a new camera profiling tool
Post by: Iliah on August 22, 2015, 02:21:16 pm
I tried with http://www.cis.rit.edu/jwgu/research/camspec/db.php (http://www.cis.rit.edu/jwgu/research/camspec/db.php) (the link is broken today)

https://web.archive.org/web/20140107051347/http://www.cis.rit.edu/jwgu/research/camspec/camspec_database.txt
The original file is still alive in the previous location too, http://www.cis.rit.edu/~dxl5849/projects/camspec/database.txt
All the projects by Dengyu Liu, including camspec database - scroll to the bottom - http://www.cis.rit.edu/~dxl5849/projects/camspec/
Title: Re: DCamProf - a new camera profiling tool
Post by: Bip on August 22, 2015, 03:44:10 pm
Like you, with Patch tool and look at the gamuts and curves profiles Copra3.
The shape and the absence of breakage is also important that the volume.
I plan to go to Color Think.
Title: Re: DCamProf - a new camera profiling tool
Post by: Bip on August 22, 2015, 04:11:39 pm
[...]

is it precise enough to your eye ?

[...]

Yes, it is really very good result.

I made a few icc profiles with SSF data D700, with a simulation of the color digital SG , but I have a doubt about the values of this target (in memory,  I took the values given on the website of Babelcolor). I kept all the files and I'll try on dcp profiles (I still work).

Thank you for your help.
Title: Re: DCamProf - a new camera profiling tool
Post by: Bip on August 22, 2015, 04:20:14 pm
https://web.archive.org/web/20140107051347/http://www.cis.rit.edu/jwgu/research/camspec/camspec_database.txt
The original file is still alive in the previous location too, http://www.cis.rit.edu/~dxl5849/projects/camspec/database.txt
All the projects by Dengyu Liu, including camspec database - scroll to the bottom - http://www.cis.rit.edu/~dxl5849/projects/camspec/

Thank a lot for links
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on August 24, 2015, 08:38:07 am
Hi
After our exchanges, I tried to make a profile for Lr from a raw demosaiced with RT (Rawtherapee) by making a linear tif.
The dcp (or icc) profiles  give very good results with RT (very low delta E 2000)

With LR, dcp profile doesn't work well, there is a shift in the temperature color display and hue for the white balance, I tested dcp linear profile and with the TRC "ACR" without better results.

The command lines used:
scanin -v - G 1.0 -dipn target.tif target.ti3
dcamprof make-profile –i D55 (1) –I D50 (2) -b P11 (3) target.ti3 profil.json
dcamprof make-dcp -n "Nikon D700" -d "Profile name" –t acr profil.json profil.dcp

(1) The pattern is taken with flash
(2) The data of the test pattern are D50
(3) P11 is the most neutral patch (a and b values of L * a * b *) of the target

If you have any comments on the process ... thank you
Someone made profiles for Lr? (With what procedure)

When you use "-t acr" you will embed a tone curve which will activate the neutral tone reproduction operator which tries to make a "perceptual" color appearance match, it's then no longer a colorimetric profile whose accuracy can be measured with standard methods.

Without tone curve it should be colorimetric though, and it sounds strange that you've got different results from RT compared to Lr. The results should be the same. In earlier versions of DCamProf no linear curve was embedded per default which lead to that Lr added a default curve, destroying the colorimetric property. However, later version should add a linear curve per default, unless you disable it (or I have broken it, it was a while since I tested that aspect), and then you should get the same result as in say RT.

You could check using dcp2json command if there is a linear curve in there, or simply look at Lr if it renders with a contrast curve or not when applying the profile.
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on August 24, 2015, 08:55:35 am
https://web.archive.org/web/20140107051347/http://www.cis.rit.edu/jwgu/research/camspec/camspec_database.txt
The original file is still alive in the previous location too, http://www.cis.rit.edu/~dxl5849/projects/camspec/database.txt
All the projects by Dengyu Liu, including camspec database - scroll to the bottom - http://www.cis.rit.edu/~dxl5849/projects/camspec/

Thanks, will update the link on the web page to the next release.
Title: Re: DCamProf - a new camera profiling tool
Post by: Bip on August 24, 2015, 06:05:29 pm
When you use "-t acr" you will embed a tone curve which will activate the neutral tone reproduction operator which tries to make a "perceptual" color appearance match, it's then no longer a colorimetric profile whose accuracy can be measured with standard methods.

Without tone curve it should be colorimetric though, and it sounds strange that you've got different results from RT compared to Lr. The results should be the same. In earlier versions of DCamProf no linear curve was embedded per default which lead to that Lr added a default curve, destroying the colorimetric property. However, later version should add a linear curve per default, unless you disable it (or I have broken it, it was a while since I tested that aspect), and then you should get the same result as in say RT.


I tested with "t acr" and without in RT, although I found the addition of the tone curve (in fact it is possible to disable it with RT, and to see the difference).
With Lr, it is difficult to tell by looking whether there is or is not the curve, although I see difference, but it is not possible to tell the origin. With Lr, I see a shif of white balance, 5500k  Normally, and Lr displays 9900K and a hue to + 150, when I make a white balance with a pipette (I do not know how to say ...  :D :the measuring probe WB on the patch), the value turns to 7500K and always on the hue +150.

The profile is good with RT, not usable with LR.
I will try to redo a profile with shooting with white RGB values slightly lower. I found that LR has a management of the high lights different that RT.
I'll let you know


You could check using dcp2json command if there is a linear curve in there, or simply look at Lr if it renders with a contrast curve or not when applying the profile.

I will also try. Thank a lot
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on August 25, 2015, 03:54:56 am
For over-exposed images Lr will crunch the image to look more like over-exposed film, meaning applying some desaturation overall to make a smoother transition into the blown areas. RT doesn't do that automatically.

However for normally exposed images it should be the same. RT is much smoother to work with when it comes to DCP testing so I quite rarely fire up Lr and test there (I work almost exclusively on Linux boxes too which make using Lr a bit messy), which means that I may have introduced some bug that has passed undetected. I'm a bit busy with other stuff now so I don't have the opportunity to do much Lr testing.

Last time I checked I got the same result though.

However note that the white balance in RT is not using the DCP, it's always using the built-in matrices, and is having a different WB model than the one specified in DNG. This means that you cannot compare the temp/tint numbers in RT with what is shown in Lr. If you want to set the same WB in both either use the "as shot" balance as is, or use a WB color picker and click at some well-defined white/gray area.

That you get 7500k +150 tint in Lr does sound suspicious though, I'm starting to suspect some bug in DCamProf, unless there are some problem in your workflow.
Title: Re: DCamProf - a new camera profiling tool
Post by: Bip on August 25, 2015, 08:31:44 am
Thank you for the information (difference between white balance with Lr and RT)


That you get 7500k +150 tint in Lr does sound suspicious though, I'm starting to suspect some bug in DCamProf, unless there are some problem in your workflow.


Bug? May be dcamprof, but it is possible that it is my workflow?

Here is the beginning of dcp profile file decoded with dcp2json:
==>
I find that the coefficients of the color matrix are odd (?)
Is that the forward matrix (D50) is correct?


Color matrix

{
  "UniqueCameraModel": "Nikon D700",
  "ProfileName": "Df-5500k_dt7-6-031",
  "ProfileCopyright": "Copyright, the creator of this profile",
  "ProfileEmbedPolicy": "No restrictions",
  "CalibrationIlluminant1": "D55",
  "ColorMatrix1": [
    [  1.430500, -0.307400, -0.122100 ],
    [ -0.598500,  1.364700,  0.204700 ],
    [ -0.100300,  0.190100,  1.008600 ]
   
  ],
  "ForwardMatrix1": [
    [  0.748500,  0.180100,  0.035600 ],
    [  0.324700,  0.810600, -0.135300 ],
    [  0.027200, -0.129500,  0.927400 ]
  ],
  "DefaultBlackRender": "None",
  "ProfileHueSatMapDims": [ 90, 30, 1 ],
  "ProfileHueSatMap1": [
    { "HueDiv":  0, "SatDiv":  0, "ValDiv":  0, "HueShift":   0.000000, "SatScale": 1.000000, "ValScale": 1.000000 },
    { "HueDiv":  0, "SatDiv":  1, "ValDiv":  0, "HueShift":  -3.063314, "SatScale": 0.971598, "ValScale": 0.997364 },
    { "HueDiv":  0, "SatDiv":  2, "ValDiv":  0, "HueShift":  -2.713044, "SatScale": 0.983901, "ValScale": 0.996794 },
  [...]


Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on August 25, 2015, 09:17:06 am
Bug? May be dcamprof, but it is possible that it is my workflow?
may be you can also post a sequence of commands (dcamprof command line paramereters for each call) how do you build a profile too...
Title: Re: DCamProf - a new camera profiling tool
Post by: Bip on August 25, 2015, 09:53:12 am
The command lines used for a dcp profile (mono illuminant):

scanin -v - G 1.0 -dipn target.tif target.ti3
dcamprof make-profile –i D55 (1) –I D50 (2) -b P11 (3) target.ti3 profil.json
dcamprof make-dcp -n "Nikon D700" -d "Profile name" profil.json profil.dcp

(1) The target is taken with flash
(2) The data of the test pattern are in D50
(3) P11 is the most neutral patch (a and b values of  L * a * b *) of the target

For an icc profile,the last line is:

dcamprof make-icc -n "Nikon D700" -p lablut (or xyzlut) profil.json profil.icc

Title: Re: DCamProf - a new camera profiling tool
Post by: Bip on August 25, 2015, 06:00:19 pm
With this command line :

dcamprof make-dcp -n "Nikon D700" -d "Profile name" -t linear -h 90,35,15 profil.json profil.dcp

The profile is linear, does not seem linear by "default" (I looked in the file "profile".json), but there are always a matter the white balance in LR et ACR (I verified for ACR), why, I dn't undestand (?)

I had made a dcp profile in June (dcampprof V7 or V8), I tested it and it has the same behavior.

A question that has no link: Why there are the -i and -I options with the command dcamprof make dcp?
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on August 26, 2015, 02:11:40 am
oops... the linear curve should be default, that's a bug, will fix that to next release. Anyway for now you can enable it with -t linear for now. Lr will apply a default curve to the profile if the profile doesn't have any, and RT doesn't do that. So to get a well-defined result one have to embed the curve.

The make-dcp has -i and -I options to make it possible to specify the EXIF illuminants for illuminant 1 and 2. Say if you have profiled with a custom light and recorded its spectrum, there is no EXIF name for that so the native format will say "Other" on the exif tag. When you make a dual-illuminant DCP you need to have defined light sources, so then you manually pick something close from the available EXIF illuminants.

I still can't figure out why you have a white balance difference between RT and Lightroom though. Since I got your report I haven't yet been able to run a Lr test so I haven't been able to verify myself. Maybe someone else in this thread can confirm if they've had it working in Lr recently?
Title: Re: DCamProf - a new camera profiling tool
Post by: Bip on August 26, 2015, 12:19:24 pm
oops... the linear curve should be default, that's a bug, will fix that to next release. Anyway for now you can enable it with -t linear for now. Lr will apply a default curve to the profile if the profile doesn't have any, and RT doesn't do that. So to get a well-defined result one have to embed the curve.

The make-dcp has -i and -I options to make it possible to specify the EXIF illuminants for illuminant 1 and 2. Say if you have profiled with a custom light and recorded its spectrum, there is no EXIF name for that so the native format will say "Other" on the exif tag. When you make a dual-illuminant DCP you need to have defined light sources, so then you manually pick something close from the available EXIF illuminants.

I still can't figure out why you have a white balance difference between RT and Lightroom though. Since I got your report I haven't yet been able to run a Lr test so I haven't been able to verify myself. Maybe someone else in this thread can confirm if they've had it working in Lr recently?

Hi Torger,

Thank you for the answers.
I have the idea that the dcp profile (made with dcamprof) does not run with lr (white balance problem) because Lr (or Acr) wants to have a bi-illumimant profile, this would cause the calculation error by Lr of white balance.
I opened (with dcp2json) dcp profiles mono illuminant made with DNG profile editor, and, inside, they have this structure: 2 illuminants (StdA and D65), 2 colors matrix, 2 forwards  matrix and 2 ProfileHueSatMap.

I tried to manually edit a dcp profile with a similar structure, but it does not work properly yet with Lr, I continue.


Title: Re: DCamProf - a new camera profiling tool
Post by: torger on August 26, 2015, 12:47:37 pm
FYI I just ran through a workflow making a lightroom profile and it works for me, single illuminant. I get the same result in RT as in Lightroom. The actual temperature number shown in the white balance widget is different though, but that's normal as explained a few posts back.
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on August 26, 2015, 01:03:39 pm
I continue.

I created a simple single illuminant / linear curve dcp profile from available D700 SSF data, it works OK... beats me why you are refusing to use SSF data in D700 case to create profiles... why do you think that using a target makes it better - you really think that it, for example, will help you to account for a specific lens mounted on your camera ?
Title: Re: DCamProf - a new camera profiling tool
Post by: Bip on August 26, 2015, 05:15:00 pm
FYI I just ran through a workflow making a lightroom profile and it works for me, single illuminant. I get the same result in RT as in Lightroom. The actual temperature number shown in the white balance widget is different though, but that's normal as explained a few posts back.

Thank you. You refer to this:

However note that the white balance in RT is not using the DCP, it's always using the built-in matrices, and is having a different WB model than the one specified in DNG. This means that you cannot compare the temp/tint numbers in RT with what is shown in Lr. If you want to set the same WB in both either use the "as shot" balance as is, or use a WB color picker and click at some well-defined white/gray area.

That you get 7500k +150 tint in Lr does sound suspicious though, I'm starting to suspect some bug in DCamProf, unless there are some problem in your workflow.

I agree about the difference calculation between RT and LR, but but it doesn't seem normal (for me) to have a white balance to 7000k and 150 of hue (witjh color picker on neutral patch, white balance is about 7300k and always 150 of hue).
One more, with profil made with DNG adobe editor, this shift is not present.

I continued my tests this afternoon, the white balance shift is not a bi-illuminant profile issue, I found that this shift came from the color matrix, why, I don't know?
Title: Re: DCamProf - a new camera profiling tool
Post by: Bip on August 26, 2015, 05:30:02 pm
I created a simple single illuminant / linear curve dcp profile from available D700 SSF data, it works OK... beats me why you are refusing to use SSF data in D700 case to create profiles... why do you think that using a target makes it better - you really think that it, for example, will help you to account for a specific lens mounted on your camera ?

I created a simple single illuminant / linear curve dcp profile from available D700 SSF data, it works OK... beats me why you are refusing to use SSF data in D700 case to create profiles...

This will be the next step (profile with SSF), when I found a matter, I try to understand (and solve if possible).

why do you think that using a target makes it better -
No, with target,  it is difficult to find the right target with the good "inks" (reflectance matter), without to speak about the shot...



- you really think that it, for example, will help you to account for a specific lens mounted on your camera ?

Neither, I am not at this level of detail  ;)


Title: Re: DCamProf - a new camera profiling tool
Post by: torger on August 27, 2015, 03:39:44 am
I did find a suspiciously large deviation in white balance temp/tint calculation (which comes from the color matrix, the actual colors comes from the forward matrix and LUTs), the Adobe bundled profiles said ~4850, my profile said ~5500, so maybe there's some problem with the color matrix. I'll investigate that further when I get some time. It's a bit slower period now though so it may take a while.
Title: Re: DCamProf - a new camera profiling tool
Post by: Bip on August 27, 2015, 05:45:23 am
I did find a suspiciously large deviation in white balance temp/tint calculation (which comes from the color matrix, the actual colors comes from the forward matrix and LUTs), the Adobe bundled profiles said ~4850, my profile said ~5500, so maybe there's some problem with the color matrix. I'll investigate that further when I get some time. It's a bit slower period now though so it may take a while.

Ok thank you.

A slight shift of the white balance is normal with a process DNG, from about 200 to 500K, because Adobe made another calculation of raw demosaicing.

What is amazing is that this does not affect RT. I think Adobe to do a special treatment, I am trying to read back the DNG specification to try to understand, but there are not all the information, including limit values for ColorMatrix and ForwardMatrix.
I continue my trials and if I found something, I tell you.
Title: Re: DCamProf - a new camera profiling tool
Post by: Bip on August 27, 2015, 05:57:58 am
beats me why you are refusing to use SSF data in D700 case to create profiles...
What kind of virtual target  (CC24, CC digital SG* or other) you use?  (when you do profile with the SSF data from the camera)
Do you take generic or measured spectral datas to simulate virtual target?

For CC Digital SG, I dn't find the spectral datas. (It seems that Xrite did not given them)
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on August 27, 2015, 08:01:03 am
Ok thank you.

A slight shift of the white balance is normal with a process DNG, from about 200 to 500K, because Adobe made another calculation of raw demosaicing.

What is amazing is that this does not affect RT. I think Adobe to do a special treatment, I am trying to read back the DNG specification to try to understand, but there are not all the information, including limit values for ColorMatrix and ForwardMatrix.
I continue my trials and if I found something, I tell you.

For historical reasons RT always use its hard-coded color matrices for the white balance calculations, never the DCP color matrix (the RT white balance model needs some re-work, but noone's had time to do it yet). Adobe uses the DCP of course. Still if you use "as shot" you should see the same result in RT as in Lr, even if the stated color/temp is different, and this is the result I get. Have you tried "As Shot" white balance? Do you get the same result in RT as in Lr, as I do? If not you may have some other issue.

Only when you have set a manual temp/tint Lr will use the color matrix "in reverse" to figure out the multipliers and then the color matrix will matter, that is a different color matrix will result in a different tint of the image.

By using json2dcp/dcp2json and a text editor you can experiment with extracting the color matrix from Adobe's bundled profile and put that into the DCamProf profile. Then you should get the exact same white balance as the Adobe profile, but the color rendition is still completely DCamProf, as that is only affected by forward matrix and LUTs.

Really old DCPs had only a color matrix (no forward matrix), and in that case the color matrix also affected the color rendition, but as soon as you have a forward matrix, the color matrix is only used for calculating white balance multpliers from temp/tint or calculating temp/tint from the raw-embedded as-shot multipliers.
Title: Re: DCamProf - a new camera profiling tool
Post by: parhelic14 on August 27, 2015, 10:13:39 am
I am new to DCAMPROF.

I just follow the instructions below, X100.DNG is from DNG converter.

dcraw -v -r 1 1 1 1 -o 0 -H 0 -T -W -g 1 1 X100.DNG
scanin -v -G 1.0 -dipn X100.tif ColorChecker.cht cc24_ref.cie
dcamprof make-profile -w all 1.5,1,8,2,1 -l 0.1,0.1 X100.ti3 profile.json
dcamprof make-dcp -n "Fujifilm Finepix X100" -d "DCAMPROF" -t linear profile.json DCAMPROF_X100.dcp

Finally,I import DCAMPROF_X100.dcp to LR, and export to tif.
I make crop to show CC24 area. Is this ok for the calibration results? White balance looks fine here.
But it looked somewhat low contrast, dark and flat. Anything I am doing wrong here??
Thanks for help.
(http://i.imgur.com/uSaejXQ.jpg)
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on August 27, 2015, 01:53:56 pm
I am new to DCAMPROF.

I just follow the instructions below, X100.DNG is from DNG converter.

dcraw -v -r 1 1 1 1 -o 0 -H 0 -T -W -g 1 1 X100.DNG
scanin -v -G 1.0 -dipn X100.tif ColorChecker.cht cc24_ref.cie
dcamprof make-profile -w all 1.5,1,8,2,1 -l 0.1,0.1 X100.ti3 profile.json
dcamprof make-dcp -n "Fujifilm Finepix X100" -d "DCAMPROF" -t linear profile.json DCAMPROF_X100.dcp

Finally,I import DCAMPROF_X100.dcp to LR, and export to tif.
I make crop to show CC24 area. Is this ok for the calibration results? White balance looks fine here.
But it looked somewhat low contrast, dark and flat. Anything I am doing wrong here??
Thanks for help.
(http://i.imgur.com/uSaejXQ.jpg)

By my eye it seems correct. However as you do "-t linear" you get a colorimetric profile without a curve, which indeed looks low contrast, dark and flat. A flat colorimetric profile is good when you do reproduction work (copy artwork etc), but not so nice for general purpose photography.

To get a default curve embedded change your make-dcp command to:

dcamprof make-dcp -n "Fujifilm Finepix X100" -d "DCAMPROF" -t acr profile.json DCAMPROF_X100.dcp

If you want to fine-tune to match the contrast and brightness of the bundled profile you can "steal" the curve and baseline exposure from the bundled profile, or you can design your own.
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on August 27, 2015, 02:07:16 pm
Here's a more detailed description of matching the curve / baseline exposure. It's from a tutorial I'm working on but haven't published yet:

Choose a tone curve and exposure offsets

When we reproduce a scene on screen or on paper the luminance is typically lower which leads to that the eye experience the image as lower contrast, a bit dull even, this even if the (mid-tone) contrast is 100% accurate. This is a normal perceptual phenomenon and the solution is as old as photography: we apply an S-shaped contrast
curve. In digital photography there's more compression of highlights than shadows as that suits the linear sensor behavior better, but the principle is the same. By compressing highlights and shadows we get increased midtone contrast so the whole image seems to have higher contrast, and we can get a better perceptual match with the original scene.

A less known side effect of applying contrast is that the appearance of colors change. In human vision contrast and color appearance is tightly connected which means that if we want to retain the original color appearance when changing global contrast we must make some adjustments to the colors. Broadly speaking higher contrast requires higher saturation, and increased saturation is a natural side-effect of a basic RGB curve which is what's been traditionally used in digital photography. However an RGB curve will overdo saturation and it will also distort color so it's far from perfect, in fact there is still today no broadly used standard curve that is reasonably
perceptually accurate. (It's actually not possible to make it 100% accurate, as that would require image-dependent local adjustments which a camera profile can't do.)

DCamProf provides its own custom curve, a "neutral tone reproduction operator" to make it possible to embed a curve without distorting color appearance, and I think this is one of the more important features. Without that it doesn't matter how accurate we make our profile, as soon as we apply a curve in the raw converter the perceptual accuracy goes down the drain. And no, applying contrast in the Lab lightness channel or HSV value channel will not cut it (although it may work well for smaller adjustments or for creative effect). As I also contribute to the RawTherapee project, there's now actually a "perceptual" curve there which is based on DCamProf's
neutral tone reproduction operator, so for RawTherapee you can provide a profile without a curve and instead apply it using the builtin curve adjustment, but as far as I know RawTherapee is unique in this aspect. For convenience you may still want to embed it in the profile though.

For a longer more detailed discussion on how tone curves affect color, see http://www.ludd.ltu.se/~torger/dcamprof.html#tone_curves tone curve section in DCamProf's reference manual.

Anyway, in order to make our profile fit for all-around photography we need to have an S-shaped tone curve to apply some contrast. So which shape to choose? With Adobe Camera Raw and the DNG reference code comes a standard curve which is used in many of Adobe's own profiles and it's also available as a built-in choice in DCamProf. It will provide a good result and if you don't have some special need I suggest using that (<tt>-t acr</tt> provided to make-dcp).

The typical reason to want some other curve is to better match the look of in-camera JPEGs. This can indeed be important. With Adobe's curve the look may become significantly brighter or darker than the in-camera JPEG, and while it's okay if you will "expose to the right" (ETTR) manually anyway (common for planned photography shot from a tripod), the camera's auto exposure is of course tuned for the in-camera JPEG curve. That is to get good results without adjustments you need the curve to match what the camera expects.

For DNG profiles there are actually three parameters for this, the curve itself, a baseline exposure offset and a black render tag. The DNG file itself can also contain a fixed baseline exposure which is added to the offset in the profile, at least if you're using Adobe's products. The exposure offsets and black render tag may or may not be
supported by your favorite raw converter, so they may not be usable. The curve is generally supported though if the converter has a decent DNG profile support.

The baseline exposure offset is sort of redundant as a curve can also include an exposure offset (it's just the average slope of it), but if the offset is large the curve gets an extreme shape which is hard to design so in that case it can be easier to separate them. The same can be said about the black render tag. That tag does not quantify an offset, it just tells the raw converter if it should make an automatic black subtraction (that is clip away darkest shadows if there is no detail there) or leave as is.

Matching the curve used by the camera

How to get the curve from your camera? Obviously we could shoot a backlit step wedge and make a precise measurement-based match that way, but there's no real need to make an exact match, we won't have the exact camera appearance anyway concerning color. We just need to make a curve that has about the same brightness and contrast as the camera so when we use the camera's auto exposure we get a good result.

To do this I think visual matching works well, and you can use RawTherapee for that.

  * If possible set the camera to write JPEGs together with the raw.
      * If possible set color space to sRGB, unless you know what
        you're doing and can display the JPEG properly with AdobeRGB (or
        whatever the other color space is).
  * Shoot basic sunny outdoor scenes with both highlights and shadows.
      * It should be contrasty, but avoid back light.
      * Avoid extremely saurated colors as they will exaggerate
      differences between profiles and make it harder to match
      curves.
      * Preferably shoot more than one so you have a few to test.
  * If you're using a DNG workflow, convert the raw to DNG using
  your chosen DNG converter (it may add an baseline exposure offset
  tag).
      * If you use DNG files with non-zero baseline exposure and you
        intend to honor that (as Adobe Camera Raw does), make sure
        you apply corresponding exposure in RawTherapee (at the time
        of writing RawTherapee ignores any baseline exposure in the
        DNG file).
  * If you couldn't get JPEGs directly, use for example exiftool to
  extract the embedded preview image from the raw. If the color space
  is not sRGB you may need to attach a profile to it (usually AdobeRGB)
  which can also be done with exiftool.
      * Make sure you get the camera's preview, not a re-rendered
      preview made by your DNG converter if you're using that.
      * exiftool -b -PreviewImage -w _preview.jpg _MG_0715.CR2
      * exiftool "-icc_profile<=AdobeRGB.icc" _MG_0715_preview.jpg
  * Open the raw file in RawTherapee and bring up the JPEG side by
  side on screen, either by launching another RawTherapee instance and
  open the JPEG there or bring it up in an image viewer.
  * Apply (Neutral) processing profile so you get a clean start.
  * Fold out "Tone curve 1" by choosing "Custom" (which is a spline curve).
      * Curve type can be any, some are more saturated than others,
      try one that matches the look of the camera JPEG the best (makes
      it easier to match curve, optionally use the saturation slider
      to make a better match.
      * You can make the panel wider to make the curve larger and
      easier to fine-tune.
  * Add three control points to the curve, and make an S-shape
  similar to the one shown in the screen shot, it's a good starting
  point.
  * Look at the darkest shadows in the JPEG and see if it's likely
  some black subtraction has been made. If so match as well as
  you can with the "Black" slider.
  * Is the image considerably darker? Adjust "Exposure compensation"
  to make a better match.
      * Small offsets like 0.1-0.2 stops can typcially be solved by
      brightening using the curve, while above that then it's better
      to have an exposure compensation.
  * Fine-tune the curve to match by moving and adding handles. Try
  to keep down the number of handles, the more you have the harder it
  becomes to make it smooth.
  * Save the curve to an .rtc file.
      * The .rtc file can be used directly by DCamProf.
  * Write down if you had to adjust the black slider and how much
  exposure adjustment you needed if any.

It is indeed hard to make an exact match, but it's not important. This is just about getting predictable results from the camera's auto exposure. It's more important that you like the shape of the curve, that it has suitable contrast and shadow compression. I often find it desirable to keep a little more shadow detail than camera JPEGs do for example.

If you had to adjust black subtraction and/or exposure you have some choices to make. Either you make a processing profile for your favorite raw converter that contains these presets, or you include the settings in your DNG profile. What is best depends on the feature set of the raw converter (if it supports the offset DNG profile tags or
not, and how presets are managed) and what you prefer.

You can also try to reshape the curve to compensate for any black subtraction and exposure offset.

If we start with black subtraction this can be 100% mirrored with a curve, but it will then unrecoverably cut shadows. It's better to just add an extra handle for the darkest shadows and compress a bit more there. Your will undoubltly get a less contrasty look in the shadow range this way, but also more shadow detail which you may
prefer anyway.

The exposure offset can also be mirrored with the curve but again not without unrecoverable clipping, so you could instead just increase highlight compression to increase brightness overall to match.

In general I think embedding an exposure offset in the profile makes more sense than black subtraction, especially since black subtraction cannot be set as a specific number but the result will instead vary between raw converters. My recommendation is thus combine the curve with an exposure offset if it makes sense, but try to avoid black subtraction.

The total baseline exposure should rather not be negative. That is the file's baseline exposure plus the profile's baseline exposure offset should be zero or positive. If you make it negative you will force the raw converter into showing potentially clipped highlights which is not a good default.

Notes on baseline exposure

The DNG specification has two baseline exposures, one that is stored in the DNG file itself, "baseline exposure", and one in the profile "baseline exposure offset". The latter was introduced in version 1.4 of the standard, prior to that baseline exposure could only be embedded in the DNG.

This is an unfortunate situation, if you ask me it's a poor design choice made by Adobe. The baseline exposure is 100% related to the profile, as it will depend on the curve shape which number you want. Naturally Adobe's DNG converter will embed a baseline exposure tag with its DNG files that is adapted to work with Abobe's proprietary profiles, and it may not really suit your profile.

It would be much better if the DNG profile itself specified all the baseline exposure, which it can with the new baseline exposure offset tag which is stored in the profile. Unfortunately the specification says that it shouldn't override the DNG file tag, but just adds an offset to it and that is what happens in Adobe's products. This means that you still need to know what value your DNG converter will put there, and if your raw converter cares about the value. Many raw converters can do both DNG and native raw. The native raw file has no baseline exposure offset, meaning that a DNG profile may need a different baseline exposure offset when used with native raws than when used with Adobe's DNGs.

I recommend to test your favorite raw converter to see how it reacts to baseline exposure, it's not certain that it will care about the value in the DNG file. It's not unlikely that you may need one offset for Adobe's products and a different offset for others.
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on August 27, 2015, 02:27:54 pm
than when used with Adobe's DNGs.
shall be added "or with Adobe converters"... they (ACR/LR) have that component also hardcoded in their code.
Title: Re: DCamProf - a new camera profiling tool
Post by: Iliah on August 27, 2015, 05:00:26 pm
Usual way to get the curve is to take a series of of bracketed shots of some more or less uniformly lit smooth surface (a computer screen out of focus is a good target), and the curve can be constructed easily from that.
Title: Re: DCamProf - a new camera profiling tool
Post by: Bip on August 27, 2015, 05:13:10 pm
For historical reasons RT always use its hard-coded color matrices for the white balance calculations, never the DCP color matrix (the RT white balance model needs some re-work, but noone's had time to do it yet).
For RT, I think, since version 3.xx, white balance is calculated on a range from 4000k to 25000k, § White balance http://50.87.144.65/~rt/w/index.php?title=Color_Management_addon (http://50.87.144.65/~rt/w/index.php?title=Color_Management_addon)

Adobe uses the DCP of course. Still if you use "as shot" you should see the same result in RT as in Lr, even if the stated color/temp is different, and this is the result I get. Have you tried "As Shot" white balance? Do you get the same result in RT as in Lr, as I do? If not you may have some other issue.

Yes, with "as chot" the display of white balance with RT(any dcp (or icc) profil ) and Lr (profil Adobe Std) is the same (delta 30k, it is negligible) ,  with the profil made with Dcamprof the value of white balance change (~7000k et hue 150).


Only when you have set a manual temp/tint Lr will use the color matrix "in reverse" to figure out the multipliers and then the color matrix will matter, that is a different color matrix will result in a different tint of the image.

I undestand the same thing as you,  except that :

-With a "normal ColorMatrix" dcp profil with Lr, it is ok
-with profil made dcamprof, the white balance is done, but hue stay saturated.

By using json2dcp/dcp2json and a text editor you can experiment with extracting the color matrix from Adobe's bundled profile and put that into the DCamProf profile. Then you should get the exact same white balance as the Adobe profile, but the color rendition is still completely DCamProf, as that is only affected by forward matrix and LUTs.

Really old DCPs had only a color matrix (no forward matrix), and in that case the color matrix also affected the color rendition, but as soon as you have a forward matrix, the color matrix is only used for calculating white balance multpliers from temp/tint or calculating temp/tint from the raw-embedded as-shot multipliers.
yes I am agree. I do a profil dcp with DNG profil editor with a cc24 shot with the same light (flash). I swap the ColorMatrix in the profil made with Dcamprof, now it is ok, white balance and hue run normally.

Below you will find the matrix.

ColorMatrix Dcamprof that dn't run
  "ColorMatrix 1": [
    [  1.455000, -0.315800, -0.121900 ],
    [ -0.607600,  1.379900,  0.208200 ],
    [ -0.135800,  0.218600,  1.022600 ]

ColorMatrix DNG adobe editor (D65)
"ColorMatrix1": [
    [  0.813900, -0.217100, -0.066400 ],
    [ -0.874800,  1.654100,  0.229600 ],
    [ -0.192400,  0.200800,  0.809300 ]

I dn't know, if this matrices can help you to understand the matter (?). I searched without finding, may be, there are an overflow with the matrix calculated with Dcamprof?
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on August 28, 2015, 02:52:46 am
Usual way to get the curve is to take a series of of bracketed shots of some more or less uniformly lit smooth surface (a computer screen out of focus is a good target), and the curve can be constructed easily from that.

Interesting! I'll try that technique.
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on August 28, 2015, 02:55:58 am
Thanks for testing Bip, I'll do some debugging of the color matrix calculator in DCamProf when I get time and opportunity, hopefully soon.

As the color matrix is not used at all when rendering ICC profiles, or when using (single-illuminant) DCPs in RawTherapee and I'm not a Lightroom/ACR user I haven't given it much test so far.
Title: Re: DCamProf - a new camera profiling tool
Post by: Bip on August 28, 2015, 04:05:42 am
Thanks for testing Bip, I'll do some debugging of the color matrix calculator in DCamProf when I get time and opportunity, hopefully soon.

As the color matrix is not used at all when rendering ICC profiles, or when using (single-illuminant) DCPs in RawTherapee and I'm not a Lightroom/ACR user I haven't given it much test so far.
May be, it is not a bug of Dcamprof, but an incompatibility with Lr / Acr.
Thank you for your help to find a solution.
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on August 28, 2015, 07:29:32 am
May be, it is not a bug of Dcamprof, but an incompatibility with Lr / Acr.
Thank you for your help to find a solution.

I've done a few experiments, not running on Lr yet though as I have that on a different box in a different house. Anyway I haven't so far discovered any error.

In any case even if there is an error and I fix it, the color matrix result can differ quite much. Although the color matrix is white-point preserving (that is no compromise is made to match the white-point, it's the only color that is matched exactly), the raw RGB values for white will differ depending on the exact shape of the illuminant spectrum. (The actual numbers between two matrices can differ quite a lot too, even if they would map white to the exact same number, as the matrices map other colors too and can do that very much different.)

In any case as we cannot get the exact setup Adobe has used, DCamProf will come to a different conclusion, and thus estimate temp/tint a bit differently. This means that in Lr/ACR for all other white balances than "as shot" you will get a white balance shift. If you don't want that to happen there's an easy fix though, just copy Adobe's color matrix and put it into DCamProf's json profile, and re-run make-dcp.

What surprised me and made me suspect a bug is that here is quite large differences in temperature estimation, like in my example 4800K for Adobe and 5500K for DCamProf. I would have guessed the temperature difference should be smaller, so it's that point I'm investigating. But as said, you can't get a 100% match, if there is a bug I would still expect that at least +/-200K would differ in the normal case. So if you don't want your new custom profile to affect the white balance of your old adjusted photos when you change profile, you need to do the copy trick.
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on August 28, 2015, 10:05:32 am
An example for a 5D mark II;

For a daylight shot:
As shot temperature, according to ACR with bundled profile: 4803K (tint 1)
As shot temperature, according to DCamProf profile generated with D50 and SSF: 5762K (tint 20)
As shot temperature, according to DCamProf profile generated with Solux ~5500K and test targets: 5123K (tint 6)

Canon themselves call the "as shot" temperature "5200K", and the white balance multipliers are 2.14,1.00,1.68

Looking at DxOmark measurements, the D50 white balance is stated as 2.57,1.00,1.44
DCamProf D50 white balance based on SSFs is 1.93,1.00,1.78
DCamProf D50 Solux/targets: 2.12,1.00,1.72
Adobe bundled profile D50: 2.22,1,1.68

The values are all over the map. DxOMark measurement indicates that the as shot daylight settings is a lower temperature than D50. The SSFs I've got leads me to much different result (the quality of the SSF is unclear though), and my DCamProf profile made with traditional methods is in most agreement with Canon's own labeled temperature.

So far I have not found any error in the calculation process, and if it where I would suspect gross color errors as the forward matrix is calculated the same way as the color matrix.

Could it be that all those differences are related to different calibration illuminants?
Title: Re: DCamProf - a new camera profiling tool
Post by: Iliah on August 28, 2015, 10:40:40 am
The numbers of CCT and Tint strongly depend on the light spectrum, and the way the matrices are calculated;  _never_ I saw a CCT in a converter in good agreement with direct measurements, or good inter-converter agreement for that matter. CCT and Tint are convolutions, and as it is with the ordinary maps, they may be relatively accurate (direct measurements of distances/angles) for a close proximity, but not so for a larger vicinity.
WB coefficients can be easily close for very different light, but at least they are directly transportable between converters (that is, if a converter allows such input). That is why we don not use CCT in RPP, in spite of all user demands. Why chase something something that does not make any sense?
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on August 28, 2015, 11:16:33 am
The numbers of CCT and Tint strongly depend on the light spectrum, and the way the matrices are calculated;  _never_ I saw a CCT in a converter in good agreement with direct measurements, or good inter-converter agreement for that matter. CCT and Tint are convolutions, and as it is with the ordinary maps, they may be relatively accurate (direct measurements of distances/angles) for a close proximity, but not so for a larger vicinity.
WB coefficients can be easily close for very different light, but at least they are directly transportable between converters (that is, if a converter allows such input). That is why we don not use CCT in RPP, in spite of all user demands. Why chase something something that does not make any sense?

Sounds reasonable. I've noticed this issue with temperatures before, but I haven't really known exactly how unreliable it is.

The problem with Lr/ACR is that if you have edited a file with a custom white balance, they store the temp and tint number, not the white balance multipliers. So when you change profile, it probes the profile with the stored temp/tint to find out which white balance multipliers that lead up to that temp/tint for that profile, which as shown before lead to entirely different multipliers and you get a white balance shift.

Seems to me that this is a design problem with DCP/ACR which Lightroom/ACR users just have to live with. I think I'll document the work-around (copy color matrix from existing profiles) in the documentation.
Title: Re: DCamProf - a new camera profiling tool
Post by: Iliah on August 28, 2015, 11:19:39 am
> The problem with Lr/ACR is that if you have edited a file with a custom white balance, they store the temp and tint number, not the white balance multipliers. So when you change profile, it probes the profile with the stored temp/tint to find out which white balance multipliers that lead up to that temp/tint for that profile, which as shown before lead to entirely different multipliers and you get a white balance shift.

Yes, that's how it is.

A plugin to Lr that adds WB coeffs to XMP files and reads those coeffs back is possible.
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on August 28, 2015, 12:01:24 pm
I think I'll document the work-around (copy color matrix from existing profiles) in the documentation.
\
may be add a command line parameter to dcamprof itself to automate that
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on August 28, 2015, 02:21:55 pm
I've just released 0.9.4

I did not find any bug in the color matrix calculation so I assume that the fairly large differences we see depends on other natural factors. So no changes regarding that.

As suggested by AlterEgo I did add a new parameter, -m, to make-dcp so you can automatically copy color matrix/matrices from an old profile if you want to keep the exact same color temp estimation.

The big work is on ICC profile LUT generation which has been reworked to avoid high saturation color artifacts mainly in Capture One. Although the algorithm is really slow and not so elegant in terms of implementation, it does seem to work just fine. So to the best of my knowledge DCamProf is now mature enough to make really high end Capture One ICC profiles that work in the same way as the native profiles.

Get it at the usual place.
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on August 28, 2015, 02:47:03 pm
0.9.4 build for Windows (mingw = dcamprof.exe + libgomp_64-1.dll + HTML & PDF manual / = copy of Torger's web page and the same converted to PDF /) : https://app.box.com/s/hxy4q0rzi59jhxdv4aqrefl5gd1p8xu1
Title: Re: DCamProf - a new camera profiling tool
Post by: Bip on August 28, 2015, 04:43:00 pm
I've just released 0.9.4

I did not find any bug in the color matrix calculation so I assume that the fairly large differences we see depends on other natural factors. So no changes regarding that.

As suggested by AlterEgo I did add a new parameter, -m, to make-dcp so you can automatically copy color matrix/matrices from an old profile if you want to keep the exact same color temp estimation.

The big work is on ICC profile LUT generation which has been reworked to avoid high saturation color artifacts mainly in Capture One. Although the algorithm is really slow and not so elegant in terms of implementation, it does seem to work just fine. So to the best of my knowledge DCamProf is now mature enough to make really high end Capture One ICC profiles that work in the same way as the native profiles.

Get it at the usual place.

Thank you, I downloaded the 9.4
Title: Re: DCamProf - a new camera profiling tool
Post by: Bip on August 28, 2015, 04:46:32 pm
@ AlterEgo

Can you say me :

What kind of virtual target  (CC24, CC digital SG* or other) you use? when you do profile with the SSF data from the camera.
Do you take generic or measured spectral datas to simulate virtual target?

For CC Digital SG, I dn't find the spectral datas. (It seems that Xrite did not given them)

Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on August 28, 2015, 05:37:35 pm
For CC Digital SG, I dn't find the spectral datas. (It seems that Xrite did not given them)

GM data from PM attached
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on August 28, 2015, 05:41:17 pm
What kind of virtual target  (CC24, CC digital SG* or other) you use?

for a start you can use CC24 data embedded into DCamProf itself... I test on that small virtual target initially.
Title: Re: DCamProf - a new camera profiling tool
Post by: Bip on August 28, 2015, 05:52:47 pm
GM data from PM attached
Thank you.

for a start you can use CC24 data embedded into DCamProf itself... I test on that small virtual target initially.
yes, to begin, it is alright.
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on August 28, 2015, 05:55:17 pm
yes, to begin, it is alright.
but then it is totally up to you... I might be happy even with adobergb-grid target for my purposes.
Title: Re: DCamProf - a new camera profiling tool
Post by: Bip on August 29, 2015, 06:56:46 am
but then it is totally up to you... I might be happy even with adobergb-grid target for my purposes.
yes, a profile with adobe RGB is already well. I will return my ideas when I have a few profiles with SSF method.
Title: Re: DCamProf - a new camera profiling tool
Post by: Bip on August 29, 2015, 11:53:19 am


Do you know why the gamut of icc profiles (for APN) defined in the XYZ space are larger ones defined in space CIE Lab?
I found that with several profilers without understanding even if the equations give the possibility to move from one space to another.
The difference between the 2 spaces being that CIE Lab is perceptual intent rendering.
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on August 29, 2015, 03:26:16 pm
Do you know why the gamut of icc profiles (for APN) defined in the XYZ space are larger ones defined in space CIE Lab?
I found that with several profilers without understanding even if the equations give the possibility to move from one space to another.
The difference between the 2 spaces being that CIE Lab is perceptual intent rendering.

The ICC gamut for a camera does not really say much about what real colors the camera actually can capture. A camera is an input device, and you can't see the gamut the same way as for an output device such as a printer or a display.

A ICC Lab LUT will take raw RGB inputs and convert to Lab output, that is 3 values in 3 values out -- a 3D LUT. The LUT needs to cover all combinations of input values (with a limited resolution). As a camera have overlapping color filters it's impossible to get an input signal with say 100% green an 0% red and blue, but that still needs to have some output value in the LUT.

Actually this has been a very hard problem for when designing DCamProf, say 30% of all LUT values are invalid colors and since the calibration makes a model based on real colors and interpolate the rest, the impossible input value combinations get thrown out to crazy positions way outside the human locus, negative lightness and such things. First I just clipped those values, but in practice you then get an "unstable" ICC profile which can lead to strange color effects when you apply an extreme white balance (which may make some of those "impossible" input combinations arise). So I had to fake the impossible values by a special type of interpolation that makes sure that you get sane color output also for insane inputs. It seems like most other profile designers does the same thing in one way or another.

Try pulling the white balance slider to the lowest temperature possible (that will make the whole image blue) and see how the ICC profile behaves. If you have a profile that haven't cared to interpolate outputs the "impossible" inputs, you will probably see colors get clipped to black and possibly get thrown into totally different colors like green, magenta or yellow rather than staying blue.

A LUT can also have XYZ output rather than Lab, but it's not as common so I assume when you refer to XYZ you think about matrix profiles. In this case there's just a simple matrix multiplication that makes the colors, and the matrix has been optimized to produce sane colors in the range it was profiled for, typically a CC24 or similar. It means that for extreme input values you will get crazy output values. The corners of the "gamut triangle" you see in an ICC viewer are 100% green, 100% red and 100% blue, and you can't really reach those points with real inputs, and even if you could the matrix will map those to crazy positions for sure as it has been optimized to make normal colors correct, and that will cause highly saturated colors to get to bad positions.

So in other words, the gamut you see in a camera profile doesn't say anything about the camera's ability to capture colors.
With DCamProf I made a color separation analysis function that can give you some insights into how well the camera can separate colors: http://www.ludd.ltu.se/~torger/dcamprof.html#ssf_csep

It's common to say that "a camera doesn't have a gamut", and sure it does not have a gamut in the same well-defined way as a printer+paper, but it does have limitations concerning color separation. There is no standardized way to define those limits though, and unfortunately you can't get any of that information from the ICC profile.

Of course the ICC profile won't output values outside it's gamut, so you know what values you can't get, but you cannot see which values that represent those "impossible" combinations and thus are meaningless parts of the gamut.
Title: Re: DCamProf - a new camera profiling tool
Post by: Bip on August 29, 2015, 05:33:27 pm
The ICC gamut for a camera does not really say much about what real colors the camera actually can capture. A camera is an input device, and you can't see the gamut the same way as for an output device such as a printer or a display.

[...]

Torger, thank you for this detailed answer. I did not think asking this question the answer would lead you to give as much precision on the design profiler, but I understand the ideas you give. It is very interesting and there is not these details in books.

When I spoke gamut size (L * a * b / XYZ), I thought of the -p option of make-icc and selects Lablut, XYZlut or matrix. With Copra3 (similar to Colorthink, 3D / 2D viewer), when I look at the size of gamut, I noticed that the volume (3D) or surface (2D) colors (gamut) is more important with a XYZlut profile than Lablut profile.

I also find that the matrix profiles are a little less accurate for correcting delta E 2000, but gives made more smooth (more natural colors) on the pictures.
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on August 30, 2015, 04:15:24 am
Torger, thank you for this detailed answer. I did not think asking this question the answer would lead you to give as much precision on the design profiler, but I understand the ideas you give. It is very interesting and there is not these details in books.

When I spoke gamut size (L * a * b / XYZ), I thought of the -p option of make-icc and selects Lablut, XYZlut or matrix. With Copra3 (similar to Colorthink, 3D / 2D viewer), when I look at the size of gamut, I noticed that the volume (3D) or surface (2D) colors (gamut) is more important with a XYZlut profile than Lablut profile.

I also find that the matrix profiles are a little less accurate for correcting delta E 2000, but gives made more smooth (more natural colors) on the pictures.

When the ICC LUT for XYZ and Lab is generated in the same way in DCamProf, but I think the XYZ has a bit larger range before clipping which may be the reason the gamut looks larger. It's the Lab LUT I've concentrated my testing on and testing that it handles strange white balance settings etc, while the XYZ LUT I have just briefly tested that it renders the correct colors in normal conditions. I would not be surprised if the Lab LUT has better behavior for extreme white balance settings than the XYZ LUT, I haven't tested that though.

A matrix-only profile can often be quite accurate for a CC24, but will not be able to match high saturation colors so well. Concerning LUTs DCamProf will per default try to match colors as good as possible (with the natural limits of a 2.5D LUT), and this often leads to a less than smooth profile. To get a smooth LUT profile you generally need to relax it a bit, I've added a few recipes in the basic workflows in the docs.

If you look at a consumer profiler like Adobe's DNG profile editor the LUT it generates doesn't correct lightness at all (-l -1,0 parameter with DCamProf). I think they do it because it's quite difficult to shoot a target without glare and uneven light issues, so correcting lightness for a typical CC24 target shot introduces sharp bends in the LUT without adding much accuracy. It seems like many bundled commercial profiles don't correct lightness either. I'm not 100% sure but I think Hasselblad has only chromaticity correction (hue+saturation) on their bundled profiles, and when looking at the bundled P45+ profile in Capture One it seems to be the same there. Adobe's bundled profiles has some lightness corrections though.

I have not myself really decided if skipping lightness correction in full is a good idea or not for "high end" profiles, but the less sure you are that your CC24 shot is of high quality the more you should consider smoothing lightness or even disabling it all-together.

I suspect that lightness corrections are the most likely to cause problems with gradients, while hue/saturation corrections are less likely to cause problems but I haven't made any deeper research on that yet.

When I make my own profiles I do correct lightness, but I work quite thoroughly with smoothing, looking at plots etc to fix bends where I find them.
Title: Re: DCamProf - a new camera profiling tool
Post by: Iliah on August 30, 2015, 03:13:38 pm
yes, a profile with adobe RGB is already well. I will return my ideas when I have a few profiles with SSF method.

Depends, here are some screenshots of a plot of Adobe RGB vs. colours used in household, measured with a spectroradiometer
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on August 31, 2015, 01:59:37 am
just in case it might be of interest/use for somebody or simply for collection ...

I measured my copy (one copy, not multiple samples, sorry) of DataColor SpyderCheckr 24 with i1Pro2 (one device used) using BabelColor, both in XRGA and non XRGA mode, 5 spectral measurements each time... the attached .zip has 2 x 5 and averaged measurements.

this target =

(http://spyder.datacolor.com/wp-content/gallery/spyder-checkr-24/140626_spydercheckr24_chart_front.jpg)
Title: Re: DCamProf - a new camera profiling tool
Post by: Bip on September 01, 2015, 05:16:05 pm
The ICC gamut for a camera does not really say much about what real colors the camera actually can capture. A camera is an input device, and you can't see the gamut the same way as for an output device such as a printer or a display.

A ICC Lab LUT will take raw RGB inputs and convert to Lab output, that is 3 values in 3 values out -- a 3D LUT. The LUT needs to cover all combinations of input values (with a limited resolution). As a camera have overlapping color filters it's impossible to get an input signal with say 100% green an 0% red and blue, but that still needs to have some output value in the LUT.

Actually this has been a very hard problem for when designing DCamProf, say 30% of all LUT values are invalid colors and since the calibration makes a model based on real colors and interpolate the rest, the impossible input value combinations get thrown out to crazy positions way outside the human locus, negative lightness and such things. First I just clipped those values, but in practice you then get an "unstable" ICC profile which can lead to strange color effects when you apply an extreme white balance (which may make some of those "impossible" input combinations arise). So I had to fake the impossible values by a special type of interpolation that makes sure that you get sane color output also for insane inputs. It seems like most other profile designers does the same thing in one way or another.

Try pulling the white balance slider to the lowest temperature possible (that will make the whole image blue) and see how the ICC profile behaves. If you have a profile that haven't cared to interpolate outputs the "impossible" inputs, you will probably see colors get clipped to black and possibly get thrown into totally different colors like green, magenta or yellow rather than staying blue.

A LUT can also have XYZ output rather than Lab, but it's not as common so I assume when you refer to XYZ you think about matrix profiles. In this case there's just a simple matrix multiplication that makes the colors, and the matrix has been optimized to produce sane colors in the range it was profiled for, typically a CC24 or similar. It means that for extreme input values you will get crazy output values. The corners of the "gamut triangle" you see in an ICC viewer are 100% green, 100% red and 100% blue, and you can't really reach those points with real inputs, and even if you could the matrix will map those to crazy positions for sure as it has been optimized to make normal colors correct, and that will cause highly saturated colors to get to bad positions.

So in other words, the gamut you see in a camera profile doesn't say anything about the camera's ability to capture colors.
With DCamProf I made a color separation analysis function that can give you some insights into how well the camera can separate colors: http://www.ludd.ltu.se/~torger/dcamprof.html#ssf_csep

It's common to say that "a camera doesn't have a gamut", and sure it does not have a gamut in the same well-defined way as a printer+paper, but it does have limitations concerning color separation. There is no standardized way to define those limits though, and unfortunately you can't get any of that information from the ICC profile.

Of course the ICC profile won't output values outside it's gamut, so you know what values you can't get, but you cannot see which values that represent those "impossible" combinations and thus are meaningless parts of the gamut.


Thanks a lot for these informations.
I made 2 dcp and icc profiles from the ssf D700 and spectral datas of CC24.
With the ICC profile, the gamut is good, in 2D, the behavior seems correct profile for the high and low lights, the 3D shape is regular.

With Lr,  I watched the behavior of these profiles with pictures, visually they are as good as the profile made last week with a semi gloss target with 570 patches without the flare and glare with the shooting. All this remains to be confirmed with measurements (Delta E). (The white balance is ok)

I will continue with CCDigital SG.


Title: Re: DCamProf - a new camera profiling tool
Post by: Bip on September 01, 2015, 05:21:37 pm
Depends, here are some screenshots of a plot of Adobe RGB vs. colours used in household, measured with a spectroradiometer



I see some colors outside the gamut Adobe RGB (mesh), is that right must be understood?
Title: Re: DCamProf - a new camera profiling tool
Post by: Bip on September 01, 2015, 05:28:27 pm
just in case it might be of interest/use for somebody or simply for collection ...

I measured my copy (one copy, not multiple samples, sorry) of DataColor SpyderCheckr 24 with i1Pro2 (one device used) using BabelColor, both in XRGA and non XRGA mode, 5 spectral measurements each time... the attached .zip has 2 x 5 and averaged measurements.

this target =

(http://spyder.datacolor.com/wp-content/gallery/spyder-checkr-24/140626_spydercheckr24_chart_front.jpg)

Is there a difference between the target CC24 Xrite and Datacolor?
Is what he has one of the best?

(I have not had time to look at the spectral data)

==> I confirm that made a profile with ssf method is easier (if they have the ssf of camera)
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on September 01, 2015, 08:12:06 pm
Is there a difference between the target CC24 Xrite and Datacolor?

some comparison was done time ago = http://www.rmimaging.com/information/SpyderCheckr_Technical_Report.pdf
Title: Re: DCamProf - a new camera profiling tool
Post by: Iliah on September 01, 2015, 09:46:43 pm

I see some colors outside the gamut Adobe RGB (mesh), is that right must be understood?

If I understood your question correctly, yes, some colours fall off the AdobeRGB gamut.
Title: Re: DCamProf - a new camera profiling tool
Post by: professorbalrog on September 03, 2015, 01:32:29 pm
Hey guys, I am fascinated by this discussion (and pls forgive the crosspost) but I am over my head in trying to get this to compile correctly on OSX. At AlterEgo's suggestion, I'm adding a reply here with my request from this post:

http://forum.luminous-landscape.com/index.php?topic=103561

Quote
Would any of you who are smarter than me using Argyll and DCamProf on a Mac be willing to help me out with it, or point me in the right direction? I'm not at all scared of command-line utilities but I feel like I have no idea how to even get these to compile correctly. I was able to get Argyll to run the scanin command based on the instructions I found here (and after moving the chart reference files to the bin folder):

http://www.trumpetpower.com/photos/Exposure#Normalizing_exposure

But that's about it. DCamProf's instructions say "It should also be relatively easy to build on Mac OS X". I have no idea what to do with that information :/ All I want to do is generate a profile that's roughly as accurate as what the X-rite software can create for use in ACR but one that I can use in Capture One instead (so, ICC right?). I don't need to build a custom target or anything crazy (yet) just the most basic of camera calibrations.

Help? Please?

Again, any help would be HUGELY appreciated, this looks like an awesome and sorely needed tool but I'm not savvy enough to get it running just yet. I suspect I'm not the only one who want's to get their hands on this but can't figure it out.
Title: Re: DCamProf - a new camera profiling tool
Post by: Iliah on September 03, 2015, 01:49:12 pm
OS X executable:
http://s3.amazonaws.com/IliahBorg/dcamprof.0.9.4.zip

BTW, in profio.c / dto_u1fixed15 line 1144
if (frac > 0x7FFF) frac = 0x7FFF
This may cause an error.
Maybe to compare with zero?
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on September 03, 2015, 03:22:38 pm
I've just released 0.9.5, very minor updates.

Included the fix of the suspicious overflow comparison noted by Iliah. The encoding/decoding of ICC values can be written much more elegant than my quick hack code, may replace it with LCMS2 at some point as was debated early in this thread :).

Anyway the bigger news is that I've completed a tutorial for making a camera profile using DCamProf, it's a separate document found at:
http://www.ludd.ltu.se/~torger/photography/camera-profiling.html

I plan to maintain that like my other photography articles and have it separate from the DCamProf distributition itself.

DCamProf is closing version 1.0. I don't have any pressing features left to implement. The goal was to make a software that would enable me and others to make profiles for general-purpose photography that can work in the same way as bundled profiles (that is work with embedded curves, have some subjective adjustments) in the large well-known raw converters, and I think I've reached that goal now.
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on September 03, 2015, 04:14:40 pm
0.9.5 build for Windows (mingw = dcamprof.exe + libgomp_64-1.dll + both manual AND tutorial / = copies of Torger's web pages and the same converted to PDF /) : https://app.box.com/s/l1ci6j8ybh9t438fnkhlbqwdr9ooxnaq (https://app.box.com/s/l1ci6j8ybh9t438fnkhlbqwdr9ooxnaq)
Title: Re: DCamProf - a new camera profiling tool
Post by: Iliah on September 03, 2015, 04:36:10 pm
OS X executable
http://s3.amazonaws.com/IliahBorg/dcamprof.0.9.5.zip
Title: Re: DCamProf - a new camera profiling tool
Post by: Bip on September 03, 2015, 05:08:54 pm
some comparison was done time ago = http://www.rmimaging.com/information/SpyderCheckr_Technical_Report.pdf
I looked at, but it is difficult to do an opinion. I downloaded the spectral datas and, a day, I will try to do a profile.
Title: Re: DCamProf - a new camera profiling tool
Post by: Bip on September 03, 2015, 05:10:30 pm
If I understood your question correctly [...],
Yes, thank you
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on September 03, 2015, 05:12:37 pm
I looked at, but it is difficult to do an opinion.
I am for sure way less qualified than Robyn Myers, I just shared my measurements for posterity...
Title: Re: DCamProf - a new camera profiling tool
Post by: Bip on September 03, 2015, 05:31:45 pm

Versions follow and I don't follow ... :D
Thank you for everything, and for this new guide "Making a camera profile with DCamProf". I have a good read.

I did the dcp profile with the CC digital SG from the ssf datas of the D700, it works great on images, I have the Delta E measures to do when I have the time.
Title: Re: DCamProf - a new camera profiling tool
Post by: Frederic_H on September 05, 2015, 02:06:09 pm
Anders,

How would you design the "make close-to-gray more gray" look ? Something like ScaleChroma < 1 applied to [0,10] chroma range ?

In your experience, is the W Faust C1 IT8.7 saturated enough to use alongside a CC24, or a target printed on glossy paper would be needed ?

Thanks.
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on September 05, 2015, 02:20:17 pm
How would you design the "make close-to-gray more gray" look ? Something like ScaleChroma < 1 applied to [0,10] chroma range ?

In your experience, is the W Faust C1 IT8.7 saturated enough to use alongside a CC24, or a target printed on glossy paper would be needed?

I think the C1 IT8 is glossy enough if I remember correctly, so it's probably a good complement. You can think about that later though, the difference between a CC24 profile only and one strengthened with glossy colors is not big.

Your recipe for making close-to-gray more gray seems fine to me. It takes some trial-and-error to nail down the range and value of course. You'd want to fade out the effect towards higher saturation, using a roundedstep curve is good for that.

(As reported in another thread there's currently a bug in the ICC LUT generator that makes darkest shadows too light and posterized. I'm working on a fix for that.)
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on September 05, 2015, 02:24:27 pm
In this older message:
http://forum.luminous-landscape.com/index.php?topic=100015.msg820035#msg820035
you can see a u'v' chromaticity diagram with IT8 patches plotted compared to the pointer's gamut and adobeRGB. As seen the colors are pretty saturated.
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on September 05, 2015, 02:40:27 pm
Just released v0.9.6, fixed a critical bug in ICC LUT generation that caused bad shadows. It doesn't affect DNG profiles, but ICC profiles which uses the neutral tone reproduction operator.
Title: Re: DCamProf - a new camera profiling tool
Post by: Frederic_H on September 05, 2015, 02:49:09 pm
Great, thanks.

We'll get the IT8.7 this week and shoot all the targets at the same time with the Aptus II-7, IQ260 and A7rII.
Title: Re: DCamProf - a new camera profiling tool
Post by: Iliah on September 05, 2015, 03:08:40 pm
May I suggest changing line 266 in colmath.c to a simple "else"

OS X executable:
http://s3.amazonaws.com/IliahBorg/dcamprof.0.9.6.zip
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on September 06, 2015, 01:36:31 am
0.9.6 build for Windows ( mingw = dcamprof.exe + libgomp_64-1.dll + both manual & tutorial / = copies of Torger's web pages / in 3 formats : IE archive .mht, Mozilla archive .maff and regular .pdf ) : https://app.box.com/s/nhqbo5znht2htv0fuwcrh4dxgjdtziby
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on September 06, 2015, 04:41:48 pm
Yet another release, 0.9.7, yet another fix for the ICC LUT. Hopefully I got it right this time, but I can't promise it. This is one of those test for 30 seconds then release... not sure how much time I have next week to code so I just wanted to get something out before I go to bed.

http://www.ludd.ltu.se/~torger/files/dcamprof-0.9.7.tar.bz2
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on September 06, 2015, 07:46:32 pm
0.9.7 build for Windows ( mingw = dcamprof.exe + libgomp_64-1.dll + both manual & tutorial / = copies of Torger's web pages / in 3 formats : IE archive .mht, Mozilla archive .maff and regular .pdf ) : https://app.box.com/s/1yjsgml85xr1na3m7olkeqsncgsyb2h5 (https://app.box.com/s/1yjsgml85xr1na3m7olkeqsncgsyb2h5)
Title: Re: DCamProf - a new camera profiling tool
Post by: howardm on September 07, 2015, 09:16:11 am
version 0.9.7 statically compiled for MacOSX

https://app.box.com/s/mpf11cs0clolsyb87hjuzzzrjx0e5qv1
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on September 08, 2015, 03:04:01 am
I'll probably code a bit less on DCamProf for a period.

The next area to look into I think may be the rendering of very high saturation colors, found in for example flowers. I've noted that it's quite typical to get issues with clipping, the blue range often seems to be the most hurt. In theory the raw converter's built-in gamut mapping would handle this, but few raw converters have any gamut mapping at all, and if they have it it's much to simplistic to handle this type of issues well. Commercial bundled profiles generally have gamut mapping built into the profile itself.

Rendering say a deep purple flower with many shades is difficult. If we want the chromaticity to be as accurate as possible we need to pull it towards gamut clipping, and then we lose tonality. An alternative is to desaturate and lighten to "artificially" add tones which otherwise would just clip.

I personally don't shoot these type of subjects much so I need to get to the florist and make some shots so I can investigate this area more.
Title: Re: DCamProf - a new camera profiling tool
Post by: Bart_van_der_Wolf on September 08, 2015, 08:08:00 am
I'll probably code a bit less on DCamProf for a period.

The next area to look into I think may be the rendering of very high saturation colors, found in for example flowers. I've noted that it's quite typical to get issues with clipping, the blue range often seems to be the most hurt.

Hi Anders,

Yes, although the clipping will often occur after White Balancing, while the Raw data is not clipped. Blue can result in underexposure if some flower Reds and Yellows are very pure, and those Reds and can be pushed into clipping depending on the illuminant's color temperature. RawTherapee allows to scale the yet undemosaiced linear gamma Raw data to prevent such issues, but if instead the profile is supposed to adjust for that, then things will get difficult pretty fast.

Quote
In theory the raw converter's built-in gamut mapping would handle this, but few raw converters have any gamut mapping at all, and if they have it it's much to simplistic to handle this type of issues well. Commercial bundled profiles generally have gamut mapping built into the profile itself.

Not optimal, but such is life ...

Quote
I personally don't shoot these type of subjects much so I need to get to the florist and make some shots so I can investigate this area more.

I have a nice Yellow/Red Raw example for you, which is just 1/3rd stop below saturation clipping of the Raw data and with extremely deep blue channel data bordering on underexposure clipping. I also have some others, all very saturated flowers. I'll PM you a link, if you can use it exclusively for testing, just let me know. I do understand that you'd want to visually compare the original with its rendered version, but I can't arrange that.

Cheers,
Bart
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on September 08, 2015, 09:24:43 am
Hi Anders,

Yes, although the clipping will often occur after White Balancing, while the Raw data is not clipped. Blue can result in underexposure if some flower Reds and Yellows are very pure, and those Reds and can be pushed into clipping depending on the illuminant's color temperature. RawTherapee allows to scale the yet undemosaiced linear gamma Raw data to prevent such issues, but if instead the profile is supposed to adjust for that, then things will get difficult pretty fast.

Not optimal, but such is life ...

I have a nice Yellow/Red Raw example for you, which is just 1/3rd stop below saturation clipping of the Raw data and with extremely deep blue channel data bordering on underexposure clipping. I also have some others, all very saturated flowers. I'll PM you a link, if you can use it exclusively for testing, just let me know. I do understand that you'd want to visually compare the original with its rendered version, but I can't arrange that.

You mean you won't send me flowers? ;D

I don't think I need to compare to the originals in the first stage, I just need to get a sense on how the profile behaves compared to others. After that investigation I'll decide if I need to do more work in the area or not. So please do send me a PM with the link.
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on September 08, 2015, 10:48:40 am
You mean you won't send me flowers? ;D
does it make any sense to get a petal, make some (many) readings with spectrophotometer (I'd assume putting it on a black non reflective background underneath) and then make a shot to produce a raw ? that will eliminate he says/she says about the color.
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on September 08, 2015, 11:02:58 am
does it make any sense to get a petal, make some (many) readings with spectrophotometer (I'd assume putting it on a black non reflective background underneath) and then make a shot to produce a raw ? that will eliminate he says/she says about the color.

There's FReD, Floral Reflectance Database too: http://www.reflectance.co.uk/ where you can get spectra for many flowers. In this case I will be more interested in the subjective qualities then the objective match though. I'd like to investigate if there is value in implementing some sort of gamut mapping feature or not.
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on September 08, 2015, 11:18:16 am
There's FReD, Floral Reflectance Database too: http://www.reflectance.co.uk/ where you can get spectra for many flowers.
but not the associated raw files with the actual shots of the full petal which is /IMHO/ might better for subjective (with your eyes and converter with profile being tested) evaluation than using that database + spectral data for the camera sensor & illumination...
Title: Re: DCamProf - a new camera profiling tool
Post by: Bart_van_der_Wolf on September 08, 2015, 11:51:42 am
but not the associated raw files with the actual shots of the full petal which is /IMHO/ might better for subjective (with your eyes and converter with profile being tested) evaluation than using that database + spectral data for the camera sensor & illumination...

I've just sent a couple of file links to Anders, one of which is a WhiBal shot to at least have 'some idea' (although natural light changes between shots) of what neutral is supposed to look like. The angle of the target (as do leafs) will pick up more, or less, sky and ambient reflections. Flower petals, especially in nature are hard to get 'correct' due to light being filtered by overhead foliage, reflections from the soil and other leafy material, some transparency, and semi-specular reflection of the sky that varies by surface angle of the petals. So a large part will be about subjective rendering anyway. Besides, the colors can also not be displayed without display/print gamut limitations anyway, they can be too pure/saturated at certain angles.

I think Anders is looking at the effect that gamut limitations that are designed into the profile, will have on the overall look.

Cheers,
Bart
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on September 08, 2015, 03:31:43 pm
Thanks for the files. I've just made some brief observations. The camera is a 1DsIII, I'm assuming it's almost exactly the same as 5DmarkII so I used profiles for the latter as I have DCamProf profiles for it.

DCamProf makes no gamut mapping, Adobe Standard makes some small gamut mapping, but not very successful I think, just a quick compression towards the end which reduces the number of tones, then C1 gamut map quite heavily which is likely causing a bit of hue shift, red roses become more orange. The C1 rendering gives the impression of having most tones though.

I think what happens is that the red roses are so red that they glow, and in that red range the eye don't differ tones that well, so it looks less detailed. With the C1 gamut mapping the rose is less saturated and more orange and thus looks more detailed. C1's result looks quite similar to Trantor's SSF-based sRGB gamut-mapped profiles published in this forum a number of months ago.

I don't feel any urgent need to do anything with DCamProf in this area. A small gamut compression towards the edge like Adobe does seems to be a bad idea, and while the C1 more heavy gamut compression (Hasselblad's Phocus does similar things) can make better prints per default, it does distort color and it should be quite easy to do the same with post-processing techniques. But at some point I'll probably include gamut mapping features for those that want it.
Title: Re: DCamProf - a new camera profiling tool
Post by: Bart_van_der_Wolf on September 08, 2015, 05:10:46 pm
Thanks for the files. I've just made some brief observations. The camera is a 1DsIII, I'm assuming it's almost exactly the same as 5DmarkII so I used profiles for the latter as I have DCamProf profiles for it.

Yes, quite likely they are rather close.

Quote
DCamProf makes no gamut mapping, Adobe Standard makes some small gamut mapping, but not very successful I think, just a quick compression towards the end which reduces the number of tones, then C1 gamut map quite heavily which is likely causing a bit of hue shift, red roses become more orange. The C1 rendering gives the impression of having most tones though.

Some of the tulips are indeed extremely red (I also have some backlit ones that really glow as if lit from the inside), and C1 does have a tendency towards orange if the conversion is pushed in brightness in postprocessing.

Quote
I think what happens is that the red roses are so red that they glow, and in that red range the eye don't differ tones that well, so it looks less detailed. With the C1 gamut mapping the rose is less saturated and more orange and thus looks more detailed.

Possible, but the details can also be achieved by boosting the higher spatial frequency (luminance) amplitude (I use "Topaz Labs Detail" for that), but that's a different beast than achieving it through a profile look. These are samples of very saturated tulips, not the 'regular' colorful ones one will commonly encounter.

Quote
I don't feel any urgent need to do anything with DCamProf in this area. A small gamut compression towards the edge like Adobe does seems to be a bad idea, and while the C1 more heavy gamut compression (Hasselblad's Phocus does similar things) can make better prints per default, it does distort color and it should be quite easy to do the same with post-processing techniques. But at some point I'll probably include gamut mapping features for those that want it.

Yes, I prefer the control that postprocessing offers, but if a profile adjustment can get us further in the right direction, then that might require less additional work and maybe less noise amplification. At least you have some more images to test your ideas on.

Cheers,
Bart
Title: Re: DCamProf - a new camera profiling tool
Post by: Frederic_H on September 12, 2015, 04:30:45 am
Ok, here the first results using a CC24 only. I'll work with the other charts later.

Except for the over-saturation and slightly reddish tint it's pretty close to what my eyes saw, much closer than the default P1 daylight profile.
Now I've to figure out what's going on with the reds, looking at the curve in Colorsync there's clearly an issue.
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on September 12, 2015, 06:49:18 am
Is the oversaturated look only in the reds? I'm on the mobile phone now (bad screen), but I get the sense that the DCamProf result is globally oversaturated. Could be some kind of problem in the patch reading process rather than make-profile make-icc step. If you PM me all data (including the cc24 raw shot) I can test on my side.

If it's a Leaf back there is a risk I need to make something different.
Title: Re: DCamProf - a new camera profiling tool
Post by: Frederic_H on September 12, 2015, 07:15:15 am
It's globally oversaturated, needs -35 saturation or so in C1, regarding reds it's only some slight global cast.

The back is the IQ260, the phenomenon is the same with the aptus though. I've tried to use your CC24-ref.cie instead of my CC24 averaged spectro readings, and the red cast is a bit lower but still present.

I'm uploading data for you to have a look, will PM you the link.
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on September 12, 2015, 07:20:21 am
I will look when I get a gap. I have some social stuff I need to attend, in order to maintain my popularity among my nearest.
Title: Re: DCamProf - a new camera profiling tool
Post by: Frederic_H on September 12, 2015, 07:22:23 am
Please do, nothing urgent on my side ;)

I will play with the IT8 and DT targets this afternoon
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on September 12, 2015, 10:34:34 am
Got the files in PM, and I think I found the problem, the data does not get linearized as the transfer function is lost along the way (you got detailed solution in PM). DCamProf is not user-friendly so it's not a strange mistake to do. To the next release I'll make sure DCamProf aborts instead of just prints a cryptic warning in this situation.
Title: Re: DCamProf - a new camera profiling tool
Post by: Frederic_H on September 12, 2015, 11:25:07 am
Good catch, that was it. I admit I didn't notice the warning.

So far the profile behaves quite well, thanks to it I've managed to get rid of that weird yellowish mess I had in a lot of pics.
Time to play with looks a bit, before some work with the IT8 chart.

Thanks for your work !
Title: Re: DCamProf - a new camera profiling tool
Post by: Tim Lookingbill on September 12, 2015, 03:58:58 pm
Don't know if it's due to the use of the CC24 target but I notice its familiar gradual roll off of shadow detail down to black in the foreground tree trunk on the right and the background trees as I get on DNG profiles with my camera using that chart. I do much prefer DCamProf's rendering mainly for its overall uniformity of color saturation and definition over the P1 Daylight version. And I can see the overly yellow cast in the brick wall on the left and in the tree foliage but it's a nasty dirty yellow.

But I don't know how that profile was constructed.

I'm impressed with the results. Good work, Torger.
Title: Re: DCamProf - a new camera profiling tool
Post by: Frederic_H on September 12, 2015, 04:11:13 pm
The dcamprof default neutral profile has been corrected since, here it is.
It clearly fixes that nasty yellow in the shadows on the left, and brings back the chroma of the rose building.

So yes, great job indeed  :)
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on September 12, 2015, 04:36:40 pm
If the difference is very large between the bundled profile and the DCamProf result then one should suspect some error in the profile making process. As seen in the corrected example above the result is different but not huge, as Capture One generally makes decently neutral color, more neutral than Adobe does for sure.

As usual I'm not at my best screen (I'm moving this autumn so my work places are more chaotic than usual), but I think I can see that the main difference between P1 and DCamProf's result is that P1 is a fair bit warmer, ie more yellow. I've noted this in many bundled profiles, also Phocus for my Hassy warms up the colors a bit, but less global. I think a slight warmup in the green to yellow range can be an advantage for many landscape scenes (sunlit foliage) so I have that in the "look" example I provide in the tutorial, but I avoid applying it to blues, reds and shadows.
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on September 13, 2015, 08:58:06 am
Here's an high saturation flower example. Thanks to Bart for the test file.

With high saturation colors the differences between different profiles start to become quite large. The attached JPEG has a Prophoto ICC embedded and you most likely need a wide gamut AdobeRGB-capable calibrated screen to get good display result. The embedded thumbnail has no ICC, so you need to click to enlarge to get the correct image.

I'm focusing on the red roses here.

I noted that I got quite large difference between an older version of DCamProf 0.8.2, and the current one. This is due to some difference in the neutral tone reproduction operator. I'm going to look into this, because I think the older result is better. The older does clip quite a lot, but lowering the exposure (final image) the tones are nicely rendered. I think this is better than the sharper rolloff of the current version.

In terms of tone separation I think ACR and current DCamProf 0.9.7 is quite similar, but ACR is a bit less saturated and a more orange hue. C1 render has good tone separation, but is quite desaturated and very orange hue. DcamProf 0.8.2 dark has same tone separation as C1 but I assume more correct hue.
Title: Re: DCamProf - a new camera profiling tool
Post by: Bip on September 13, 2015, 09:13:48 am

I agree with your analysis. Dcamprof 8.2 is more natural, Version 9.7 gives the idea of artificial color (Tulip) with too much saturation.
After, it is difficult to say which interpretation is faithful,
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on September 13, 2015, 11:38:23 am
Heh oh yes, red tulips of course, not red roses... ;D

Anyway, concerning what is most faithful I don't know the original, but both DCamProf versions are in agreement of what the colorimetric color is, the magic happens when the curve and neutral tone reproduction operator is applied, it works a little bit different between 0.8.2 and 0.9.7.

When the color is so saturated that we're dealing with clipping it's much a matter taste what is most "faithful", do we want to compromise saturation (and possibly hue) to gain better tone separation or not? It's a bit like comparing "perceptual rendering intent" vs "relative colorimetric" when making prints. My guess is that DCamProf 0.9.7 is more similar to "relative colorimetric" (be as correct as possible as far as possible and then just clip, with a short rolloff to avoid the ugliest artifacts), while DCamProf 0.8.2 is more "perceptual intent" with more desaturation to improve tonality. There's no intentional "gamut mapping" in 0.8.2 though, so there's something else going on.

At the moment I don't know what causes the difference, but I'm about to dive into it.
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on September 13, 2015, 12:18:10 pm
I now know what the difference between 0.8.2 and 0.9.7 is that causes the difference in look of the red tulips. Back in 0.8.2 the curve was applied in the luminance channel, but this was later changed to instead be "ghost-applied" in RGB-HSV and luminance taken from that (to get a more predictable/comparable result with a standard curve). Perhaps that was not a good idea, it does cause luminance separation to be reduced in this context.

I need to re-run tests on a bunch of images before I know if I can just revert it.
Title: Re: DCamProf - a new camera profiling tool
Post by: Frederic_H on September 13, 2015, 03:06:17 pm
Maybe we could have make-icc to automatically populate the ICC creator tag with dcamprof version number ?
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on September 13, 2015, 03:29:20 pm
Just released 0.9.8.

Tested through my images and adjusted the tone reproduction operator accordingly so it renders saturated colors with better tone separation, like older version did. The look is not 100% the same as the old version, as there has been some other improvements on the way too.

I've attached a comparison, here all darkened to about the same level to make it easier to compare. Still a prophoto jpeg so calibrated wide gamut screen is recommended.
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on September 13, 2015, 03:32:26 pm
Maybe we could have make-icc to automatically populate the ICC creator tag with dcamprof version number ?

It didn't get into 0.9.8... I'll look into it. I'm not sure it's okay by the standard, the creator signature should be registered I think. There's the copyright string and (set with -c "my copyright string") and description string (-n "my description") which you can set if you want to tag them yourself.
Title: Re: DCamProf - a new camera profiling tool
Post by: Tim Lookingbill on September 13, 2015, 05:57:51 pm
Just to test whether a wide gamut screen is necessary to view your posted sample I've uploaded a screengrab of the first attempt to confirm whether the red tulips of the DCamProf versions are suppose to have less definition and be on the slightly magenta side of red.

I'm viewing this on a Colormunki Display calibrated LG IPS 27" LED display with its profile embedded in the screengrab. I find the C1 the more realistic rendering and seeing its color issues described by torger.

Just a suggestion but I'ld find a test image having a wide range of saturated objects but exposed to look as real as the scene and not have over cranked color. I don't find this image as looking anything close to reality when it comes to flower rendering.
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on September 13, 2015, 10:28:50 pm
It didn't get into 0.9.8...
will it be a nice idea to have a command line parameter (for relevant dcamprof commands), that if used explicitly will embed some comments into JSONs and pass also then to icc/dcp in appropriate tags as to what were command line parameters of dcamprof used to build JSONs and dcps/iccs (so the final dcp or icc output can have somewhat complete history how it was built if the you so desire, for debugging purposes for example)...  $0.02
Title: Re: DCamProf - a new camera profiling tool
Post by: AlterEgo on September 14, 2015, 12:50:04 am
0.9.8 build for Windows ( mingw = dcamprof.exe + libgomp_64-1.dll + both manual & tutorial / = copies of Torger's web pages / in 3 formats : IE archive .mht, Mozilla archive .maff and regular .pdf ) : https://app.box.com/s/aj53xxbw2jy77uw4vdilk6ubathikb8p (https://app.box.com/s/aj53xxbw2jy77uw4vdilk6ubathikb8p)
Title: Re: DCamProf - a new camera profiling tool
Post by: torger on September 14, 2015, 01:46:40 am
Just to test whether a wide gamut screen is necessary to view your posted sample I've uploaded a screengrab of the first attempt to confirm whether the red tulips of the DCamProf versions are suppose to have less definition and be on the slightly magenta side of red.

I'm viewing this on a Colormunki Display calibrated LG IPS 27" LED display with its profile embedded in the screengrab. I find the C1 the more realistic rendering and seeing its color issues described by torger.

Just a suggestion but I'ld find a test image having a wide range of saturated objects but exposed to look as real as the scene and not have over cranked color. I don't find this image as looking anything close to reality when it comes to flower rendering.

It does not seem like your LG is a wide gamut monitor because the attached image is severely clipped. It's probably more close to an sRGB monitor? The image won't work on that, that's why I pasted a warning over the whole image. C1 rendering will work as the profile has built-in gamut mapping. I will probably introduce a gamut mapping option to DCamProf at some point, but currently you need to manually adjust post-processing to fit into your gamut.

Concerning if color is over-cranked or not it may be that, but it's not intentionally so. If it is then it's due to precision er