DCamProf - a new camera profiling tool

[Bip]

https://web.archive.org/web/20140107051347/http://www.cis.rit.edu/jwgu/research/camspec/camspec_database.txt
The original file is still alive in the previous location too, http://www.cis.rit.edu/~dxl5849/projects/camspec/database.txt
All the projects by Dengyu Liu, including camspec database - scroll to the bottom - http://www.cis.rit.edu/~dxl5849/projects/camspec/

Thank a lot for links

[torger]

Hi
After our exchanges, I tried to make a profile for Lr from a raw demosaiced with RT (Rawtherapee) by making a linear tif.
The dcp (or icc) profiles  give very good results with RT (very low delta E 2000)

With LR, dcp profile doesn't work well, there is a shift in the temperature color display and hue for the white balance, I tested dcp linear profile and with the TRC "ACR" without better results.

The command lines used:
scanin -v - G 1.0 -dipn target.tif target.ti3
dcamprof make-profile –i D55 (1) –I D50 (2) -b P11 (3) target.ti3 profil.json
dcamprof make-dcp -n "Nikon D700" -d "Profile name" –t acr profil.json profil.dcp

(1) The pattern is taken with flash
(2) The data of the test pattern are D50
(3) P11 is the most neutral patch (a and b values of L * a * b *) of the target

If you have any comments on the process ... thank you
Someone made profiles for Lr? (With what procedure)

When you use "-t acr" you will embed a tone curve which will activate the neutral tone reproduction operator which tries to make a "perceptual" color appearance match, it's then no longer a colorimetric profile whose accuracy can be measured with standard methods.

Without tone curve it should be colorimetric though, and it sounds strange that you've got different results from RT compared to Lr. The results should be the same. In earlier versions of DCamProf no linear curve was embedded per default which lead to that Lr added a default curve, destroying the colorimetric property. However, later version should add a linear curve per default, unless you disable it (or I have broken it, it was a while since I tested that aspect), and then you should get the same result as in say RT.

You could check using dcp2json command if there is a linear curve in there, or simply look at Lr if it renders with a contrast curve or not when applying the profile.

[torger]

https://web.archive.org/web/20140107051347/http://www.cis.rit.edu/jwgu/research/camspec/camspec_database.txt
The original file is still alive in the previous location too, http://www.cis.rit.edu/~dxl5849/projects/camspec/database.txt
All the projects by Dengyu Liu, including camspec database - scroll to the bottom - http://www.cis.rit.edu/~dxl5849/projects/camspec/

Thanks, will update the link on the web page to the next release.

[Bip]

When you use "-t acr" you will embed a tone curve which will activate the neutral tone reproduction operator which tries to make a "perceptual" color appearance match, it's then no longer a colorimetric profile whose accuracy can be measured with standard methods.

Without tone curve it should be colorimetric though, and it sounds strange that you've got different results from RT compared to Lr. The results should be the same. In earlier versions of DCamProf no linear curve was embedded per default which lead to that Lr added a default curve, destroying the colorimetric property. However, later version should add a linear curve per default, unless you disable it (or I have broken it, it was a while since I tested that aspect), and then you should get the same result as in say RT.

I tested with "t acr" and without in RT, although I found the addition of the tone curve (in fact it is possible to disable it with RT, and to see the difference).
With Lr, it is difficult to tell by looking whether there is or is not the curve, although I see difference, but it is not possible to tell the origin. With Lr, I see a shif of white balance, 5500k  Normally, and Lr displays 9900K and a hue to + 150, when I make a white balance with a pipette (I do not know how to say ...  :the measuring probe WB on the patch), the value turns to 7500K and always on the hue +150.

The profile is good with RT, not usable with LR.
I will try to redo a profile with shooting with white RGB values slightly lower. I found that LR has a management of the high lights different that RT.
I'll let you know

You could check using dcp2json command if there is a linear curve in there, or simply look at Lr if it renders with a contrast curve or not when applying the profile.

I will also try. Thank a lot

[torger]

For over-exposed images Lr will crunch the image to look more like over-exposed film, meaning applying some desaturation overall to make a smoother transition into the blown areas. RT doesn't do that automatically.

However for normally exposed images it should be the same. RT is much smoother to work with when it comes to DCP testing so I quite rarely fire up Lr and test there (I work almost exclusively on Linux boxes too which make using Lr a bit messy), which means that I may have introduced some bug that has passed undetected. I'm a bit busy with other stuff now so I don't have the opportunity to do much Lr testing.

Last time I checked I got the same result though.

However note that the white balance in RT is not using the DCP, it's always using the built-in matrices, and is having a different WB model than the one specified in DNG. This means that you cannot compare the temp/tint numbers in RT with what is shown in Lr. If you want to set the same WB in both either use the "as shot" balance as is, or use a WB color picker and click at some well-defined white/gray area.

That you get 7500k +150 tint in Lr does sound suspicious though, I'm starting to suspect some bug in DCamProf, unless there are some problem in your workflow.

[Bip]

Thank you for the information (difference between white balance with Lr and RT)

That you get 7500k +150 tint in Lr does sound suspicious though, I'm starting to suspect some bug in DCamProf, unless there are some problem in your workflow.

Bug? May be dcamprof, but it is possible that it is my workflow?

Here is the beginning of dcp profile file decoded with dcp2json:
==>
I find that the coefficients of the color matrix are odd (?)
Is that the forward matrix (D50) is correct?


Color matrix

{
  "UniqueCameraModel": "Nikon D700",
  "ProfileName": "Df-5500k_dt7-6-031",
  "ProfileCopyright": "Copyright, the creator of this profile",
  "ProfileEmbedPolicy": "No restrictions",
  "CalibrationIlluminant1": "D55",
  "ColorMatrix1": [
    [  1.430500, -0.307400, -0.122100 ],
    [ -0.598500,  1.364700,  0.204700 ],
    [ -0.100300,  0.190100,  1.008600 ]
   
  ],
  "ForwardMatrix1": [
    [  0.748500,  0.180100,  0.035600 ],
    [  0.324700,  0.810600, -0.135300 ],
    [  0.027200, -0.129500,  0.927400 ]
  ],
  "DefaultBlackRender": "None",
  "ProfileHueSatMapDims": [ 90, 30, 1 ],
  "ProfileHueSatMap1": [
    { "HueDiv":  0, "SatDiv":  0, "ValDiv":  0, "HueShift":   0.000000, "SatScale": 1.000000, "ValScale": 1.000000 },
    { "HueDiv":  0, "SatDiv":  1, "ValDiv":  0, "HueShift":  -3.063314, "SatScale": 0.971598, "ValScale": 0.997364 },
    { "HueDiv":  0, "SatDiv":  2, "ValDiv":  0, "HueShift":  -2.713044, "SatScale": 0.983901, "ValScale": 0.996794 },
  [...]

[AlterEgo]

Bug? May be dcamprof, but it is possible that it is my workflow?

may be you can also post a sequence of commands (dcamprof command line paramereters for each call) how do you build a profile too...

[Bip]

The command lines used for a dcp profile (mono illuminant):

scanin -v - G 1.0 -dipn target.tif target.ti3
dcamprof make-profile –i D55 (1) –I D50 (2) -b P11 (3) target.ti3 profil.json
dcamprof make-dcp -n "Nikon D700" -d "Profile name" profil.json profil.dcp

(1) The target is taken with flash
(2) The data of the test pattern are in D50
(3) P11 is the most neutral patch (a and b values of  L * a * b *) of the target

For an icc profile,the last line is:

dcamprof make-icc -n "Nikon D700" -p lablut (or xyzlut) profil.json profil.icc

[Bip]

With this command line :

dcamprof make-dcp -n "Nikon D700" -d "Profile name" -t linear -h 90,35,15 profil.json profil.dcp

The profile is linear, does not seem linear by "default" (I looked in the file "profile".json), but there are always a matter the white balance in LR et ACR (I verified for ACR), why, I dn't undestand (?)

I had made a dcp profile in June (dcampprof V7 or V8), I tested it and it has the same behavior.

A question that has no link: Why there are the -i and -I options with the command dcamprof make dcp?

[torger]

oops... the linear curve should be default, that's a bug, will fix that to next release. Anyway for now you can enable it with -t linear for now. Lr will apply a default curve to the profile if the profile doesn't have any, and RT doesn't do that. So to get a well-defined result one have to embed the curve.

The make-dcp has -i and -I options to make it possible to specify the EXIF illuminants for illuminant 1 and 2. Say if you have profiled with a custom light and recorded its spectrum, there is no EXIF name for that so the native format will say "Other" on the exif tag. When you make a dual-illuminant DCP you need to have defined light sources, so then you manually pick something close from the available EXIF illuminants.

I still can't figure out why you have a white balance difference between RT and Lightroom though. Since I got your report I haven't yet been able to run a Lr test so I haven't been able to verify myself. Maybe someone else in this thread can confirm if they've had it working in Lr recently?

[Bip]

oops... the linear curve should be default, that's a bug, will fix that to next release. Anyway for now you can enable it with -t linear for now. Lr will apply a default curve to the profile if the profile doesn't have any, and RT doesn't do that. So to get a well-defined result one have to embed the curve.

The make-dcp has -i and -I options to make it possible to specify the EXIF illuminants for illuminant 1 and 2. Say if you have profiled with a custom light and recorded its spectrum, there is no EXIF name for that so the native format will say "Other" on the exif tag. When you make a dual-illuminant DCP you need to have defined light sources, so then you manually pick something close from the available EXIF illuminants.

I still can't figure out why you have a white balance difference between RT and Lightroom though. Since I got your report I haven't yet been able to run a Lr test so I haven't been able to verify myself. Maybe someone else in this thread can confirm if they've had it working in Lr recently?

Hi Torger,

Thank you for the answers.
I have the idea that the dcp profile (made with dcamprof) does not run with lr (white balance problem) because Lr (or Acr) wants to have a bi-illumimant profile, this would cause the calculation error by Lr of white balance.
I opened (with dcp2json) dcp profiles mono illuminant made with DNG profile editor, and, inside, they have this structure: 2 illuminants (StdA and D65), 2 colors matrix, 2 forwards  matrix and 2 ProfileHueSatMap.

I tried to manually edit a dcp profile with a similar structure, but it does not work properly yet with Lr, I continue.

[torger]

FYI I just ran through a workflow making a lightroom profile and it works for me, single illuminant. I get the same result in RT as in Lightroom. The actual temperature number shown in the white balance widget is different though, but that's normal as explained a few posts back.

[AlterEgo]

I continue.

I created a simple single illuminant / linear curve dcp profile from available D700 SSF data, it works OK... beats me why you are refusing to use SSF data in D700 case to create profiles... why do you think that using a target makes it better - you really think that it, for example, will help you to account for a specific lens mounted on your camera ?

[Bip]

FYI I just ran through a workflow making a lightroom profile and it works for me, single illuminant. I get the same result in RT as in Lightroom. The actual temperature number shown in the white balance widget is different though, but that's normal as explained a few posts back.

Thank you. You refer to this:

However note that the white balance in RT is not using the DCP, it's always using the built-in matrices, and is having a different WB model than the one specified in DNG. This means that you cannot compare the temp/tint numbers in RT with what is shown in Lr. If you want to set the same WB in both either use the "as shot" balance as is, or use a WB color picker and click at some well-defined white/gray area.

That you get 7500k +150 tint in Lr does sound suspicious though, I'm starting to suspect some bug in DCamProf, unless there are some problem in your workflow.

I agree about the difference calculation between RT and LR, but but it doesn't seem normal (for me) to have a white balance to 7000k and 150 of hue (witjh color picker on neutral patch, white balance is about 7300k and always 150 of hue).
One more, with profil made with DNG adobe editor, this shift is not present.

I continued my tests this afternoon, the white balance shift is not a bi-illuminant profile issue, I found that this shift came from the color matrix, why, I don't know?

[Bip]

I created a simple single illuminant / linear curve dcp profile from available D700 SSF data, it works OK... beats me why you are refusing to use SSF data in D700 case to create profiles... why do you think that using a target makes it better - you really think that it, for example, will help you to account for a specific lens mounted on your camera ?

I created a simple single illuminant / linear curve dcp profile from available D700 SSF data, it works OK... beats me why you are refusing to use SSF data in D700 case to create profiles...

This will be the next step (profile with SSF), when I found a matter, I try to understand (and solve if possible).

why do you think that using a target makes it better -

No, with target,  it is difficult to find the right target with the good "inks" (reflectance matter), without to speak about the shot...

- you really think that it, for example, will help you to account for a specific lens mounted on your camera ?

Neither, I am not at this level of detail 

[torger]

I did find a suspiciously large deviation in white balance temp/tint calculation (which comes from the color matrix, the actual colors comes from the forward matrix and LUTs), the Adobe bundled profiles said ~4850, my profile said ~5500, so maybe there's some problem with the color matrix. I'll investigate that further when I get some time. It's a bit slower period now though so it may take a while.

[Bip]

I did find a suspiciously large deviation in white balance temp/tint calculation (which comes from the color matrix, the actual colors comes from the forward matrix and LUTs), the Adobe bundled profiles said ~4850, my profile said ~5500, so maybe there's some problem with the color matrix. I'll investigate that further when I get some time. It's a bit slower period now though so it may take a while.

Ok thank you.

A slight shift of the white balance is normal with a process DNG, from about 200 to 500K, because Adobe made another calculation of raw demosaicing.

What is amazing is that this does not affect RT. I think Adobe to do a special treatment, I am trying to read back the DNG specification to try to understand, but there are not all the information, including limit values for ColorMatrix and ForwardMatrix.
I continue my trials and if I found something, I tell you.

[Bip]

beats me why you are refusing to use SSF data in D700 case to create profiles...

What kind of virtual target  (CC24, CC digital SG* or other) you use?  (when you do profile with the SSF data from the camera)
Do you take generic or measured spectral datas to simulate virtual target?

For CC Digital SG, I dn't find the spectral datas. (It seems that Xrite did not given them)

[torger]

Ok thank you.

A slight shift of the white balance is normal with a process DNG, from about 200 to 500K, because Adobe made another calculation of raw demosaicing.

What is amazing is that this does not affect RT. I think Adobe to do a special treatment, I am trying to read back the DNG specification to try to understand, but there are not all the information, including limit values for ColorMatrix and ForwardMatrix.
I continue my trials and if I found something, I tell you.

For historical reasons RT always use its hard-coded color matrices for the white balance calculations, never the DCP color matrix (the RT white balance model needs some re-work, but noone's had time to do it yet). Adobe uses the DCP of course. Still if you use "as shot" you should see the same result in RT as in Lr, even if the stated color/temp is different, and this is the result I get. Have you tried "As Shot" white balance? Do you get the same result in RT as in Lr, as I do? If not you may have some other issue.

Only when you have set a manual temp/tint Lr will use the color matrix "in reverse" to figure out the multipliers and then the color matrix will matter, that is a different color matrix will result in a different tint of the image.

By using json2dcp/dcp2json and a text editor you can experiment with extracting the color matrix from Adobe's bundled profile and put that into the DCamProf profile. Then you should get the exact same white balance as the Adobe profile, but the color rendition is still completely DCamProf, as that is only affected by forward matrix and LUTs.

Really old DCPs had only a color matrix (no forward matrix), and in that case the color matrix also affected the color rendition, but as soon as you have a forward matrix, the color matrix is only used for calculating white balance multpliers from temp/tint or calculating temp/tint from the raw-embedded as-shot multipliers.

[parhelic14]

I am new to DCAMPROF.

I just follow the instructions below, X100.DNG is from DNG converter.

dcraw -v -r 1 1 1 1 -o 0 -H 0 -T -W -g 1 1 X100.DNG
scanin -v -G 1.0 -dipn X100.tif ColorChecker.cht cc24_ref.cie
dcamprof make-profile -w all 1.5,1,8,2,1 -l 0.1,0.1 X100.ti3 profile.json
dcamprof make-dcp -n "Fujifilm Finepix X100" -d "DCAMPROF" -t linear profile.json DCAMPROF_X100.dcp

Finally,I import DCAMPROF_X100.dcp to LR, and export to tif.
I make crop to show CC24 area. Is this ok for the calibration results? White balance looks fine here.
But it looked somewhat low contrast, dark and flat. Anything I am doing wrong here??
Thanks for help.