DCamProf - a new camera profiling tool

[AlterEgo]

Here is a link to the archive of files I used in this example:

https://www.dropbox.com/s/xq5w8whoi8sq1av/dcamprofDebugArchive.zip?dl=0

it is uploading now (1400 hr eastern daylight time) so give it a few minutes if the download fails.

The archive includes the source raw, the dcraw conversion, a crop of the conversion, the scanin target chart in file, etc.  

thanks in advance!

kirk

something is wrong with CM/FM inside DCP... if you replace a pair of CM/FM with the one from Adobe OEM - then ACR see the profile and can use it... so the way CM/FM were calculated put them against something that ACR code expects (it does some checks of course, you can't just feed any matrices)

[torger]

Seems like I've messed up with something, I'll report back when I have a fix.

[AlterEgo]

Seems like I've messed up with something, I'll report back when I have a fix.

check how you do FM - if you leave everything but use FM from Adobe OEM then ACR code can accept the profile... replace FM with yours and ACR does not accept it

[torger]

check how you do FM - if you leave everything but use FM from Adobe OEM then ACR code can accept the profile... replace FM with yours and ACR does not accept it

Thanks for the help, I'll have a look.

[AlterEgo]

Thanks for the help, I'll have a look.

and CM seems to operate properly in terms of how ACR works with WB - visual impression, so CM is calculated OK I guess...

[torger]

Got it working now, I'll soon make a new release. Just need to really make sure what the error was.

Edit: there where two errors:

1) Lightroom cannot handle FM with too high precision in the rational numbers, they use 1/10000, I have 1/1000000000 but then it goes haywire
2) Lightroom cannot handle other observer than 1931_2, DCamProf uses the more realistic 2006_10. By some (stupid) reason Lightroom makes a check that FM x 1,1,1 = D50, and D50 varies slightly depending on observer, and thus if you have not used 1931_2 it breaks.

I'll add a workaround for this and publish in an hour or so, but I can't say I think Lightroom does it the right way. There's no reason to "sanity-check" the D50 whitepoint this way, and why they can handle high precision rational numbers is a bit strange too.

[AlterEgo]

Got it working now, I'll soon make a new release. Just need to really make sure what the error was.

Edit: there where two errors:

1) Lightroom cannot handle FM with too high precision in the rational numbers, they use 1/10000, I have 1/1000000000 but then it goes haywire
2) Lightroom cannot handle other observer than 1931_2, DCamProf uses the more realistic 2006_10. By some (stupid) reason Lightroom makes a check that FM x 1,1,1 = D50, and D50 varies slightly depending on observer, and thus if you have not used 1931_2 it breaks.

I'll add a workaround for this and publish in an hour or so, but I can't say I think Lightroom does it the right way. There's no reason to "sanity-check" the D50 whitepoint this way, and why they can handle high precision rational numbers is a bit strange too.

the first issue is kind of funny... too much precision :-)

[torger]

The observer thing is a tricky one... in a way I understand why they check it, as their XYZ to prophoto matrix is hardcoded and expect 1931_2 D50 whitepoint. The easy way out is just that I change the default from 2006_10 to 1931_2, but I think that is worse design, I'd like it to be possible to use more realistic/accurate observers.

Maybe I could just do some whitepoint remapping trick... I need to think.

[TRANTOR]

10-degrees standard observer is not more accurate, it's a just different. Simply do not use it.

[torger]

10-degrees standard observer is not more accurate, it's a just different. Simply do not use it.

I'm quite sure the 2006 versions, http://www.cvrl.org/cmfs.htm, are more accurate. When it comes to 2 vs 10 degree I agree, it's more a matter of taste, but AFAIK the 2006_2 is more accurate than 1931_2 and 2006_10 is more accurate than 1964_10. The reason those new versions are made is mainly to improve accuracy.

If it matters is another aspect though, probably not I guess.

I think I will do an auto-wp-remapping in the make-dcp step and maybe change default to 1931_2.

[AlterEgo]

Maybe I could just do some whitepoint remapping trick... I need to think.

if you are so set on "10 degrees" then just supply a command line parameter for an end user to select and provide some explanantion in a manual and then may be it makes sense to default (if nothing is on the command line) to "2 degrees" - I 'd assume that all popular (for masses) software expects that and not "10 degrees" based on what I can read

[torger]

if you are so set on "10 degrees" then just supply a command line parameter for an end user to select and provide some explanantion in a manual and then may be it makes sense to default (if nothing is on the command line) to "2 degrees" - I 'd assume that all popular (for masses) software expects that and not "10 degrees" based on what I can read

Command line parameter is already there, you can do "-o 1931_2", so if it weren't for the 1/10000000 thing it would have worked with the current version. Well, no more talk, I need to code :-)

[torger]

I have now made a new release, and this time actually tested it on Lightroom :-)

http://www.ludd.ltu.se/~torger/files/dcamprof-0.5.1.tar.bz2
http://www.ludd.ltu.se/~torger/dcamprof.html

I added an auto-remapping of whitepoint in the make-dcp tool so regardless of observer used during profile creation it will be remapped to match what Lightroom and probably many other DCP-compatible converters are expecting.

I also changed the default observer to 1931_2 so the remapping is not really needed in the default case. Although I like the 2006_10 (or 2006_2) better I don't have the knowledge now to know if it's better to make profile with say 2006_2 and then remap the white, or if the supposedly slight improvements are nullified due to remapping and hard-coded prophoto matrices later in the pipeline which I can't control. So now by using 1931_2 it's done the way "everyone else" does it, which I suppose is a reasonable default anyway.

Kirk - now it should work for you too. Just compile the new version and run the make-profile and make-dcp steps again.

[torger]

Prophoto is defined against 1931_2, and as DNG works in that space it was natural that the pipeline was hardwired to that. sRGB, AdobeRGB etc are all related to 1931_2. I missed that aspect. Learning something new every day.

This means that it will be tricky to make use of alternate observers in a proper way, seems to me to do it right one would have to make a conversion LUT between the different observers too, quite messy. So using the 1931_2 as default for now is certainly the right choice.

[torger]

Done some reading on observers and did some further observer tests this morning, released a very minor update v0.5.2 which tunes the observer remapping a bit. This is however only relevant if experimenting with a different observer than the nowadays default 1931_2.

There's no great method to transform say 2006_10 XYZ to 1931_2 XYZ (found this paper while searching: http://www.rit.edu/cos/colorscience/re_WptNormalization.php), but as the difference is quite small I've tested with the Bradford transform and it seems to make a good result in practice, so good that any other inherent errors in profiling should be much larger.

I made some examples between 2006_10/2006_2 and 1931_2. The 2 and 10 degree observers render near zero visible difference, while it's quite large difference between 2006 and 1931. For the particular camera I tested this morning (Pentax K10D) I preferred the 1931_2 colors, they looked more on the spot to my eyes, so having to change the default to 1931_2 doesn't feel that bad

[kirkt]

Thanks!  I'll try recompiling again and report back.

Kirk

[Alexey.Danilchenko]

> By some (stupid) reason Lightroom makes a check that FM x 1,1,1 = D50, and D50 varies slightly depending on observer, and thus if you have not used 1931_2 it breaks.

That is not that stupid. FM is constructed in a way to map camera white balanced white point to XYZ D50 (that is according to DNG spec) and that implies this mapping exactly (white balanced camera space will have 1,1,1 as white point). FM is actually a very specific case of ICC profile matrix in this case.

[torger]

Yes I agree after some analysis that it's not that stupid, although unnecessary. The pipeline does not break if there's a small white balance shift, and the reference code doesn't do this type of check.

At some point I'm probably going to again review if one can do something useful with the more modern observers, for now it seems like 1931_2 is the way to go to get best results.

[Alexey.Danilchenko]

It is necessary at least to calculate those matrixes. The profiles (and correction matrices) are build on the that assumption. Nobody prevents to use the matrices on imperfectly white balanced camera space (this is what creative WB is for - I for one never completely white balance tungsten lighting and prefer warmer colours).

What is limiting in DNG is that it has to map to D50 - unlike ICC which can have any white point.

Great tool - thanks for sharing it. I was writing something custom for my own purpouses but will attempt to make use of DCamProf now. Have not managed to compile it on Windows yet - it does not seem to accept gcc short syntax for initialising structs.

[torger]

I just tried compiling with cygwin on windows, it worked but you have to remove long double stuff in blackbody code which probably is overkill anyway. I use c99 syntax, Microsoft compilers have been sloooow adopting that, but it's only a 16 year old standard