DCamProf - a new camera profiling tool

[torger]

If you want neutral/"accurate" profiles DCamProf should indeed do the job. I may need to make some contrast/curve/desaturation support though to work better with film curves to give you full satisfaction, but you can start testing and playing around today. DCamProf is today very "colorimetric", accurate results with linear gamma, which may lead to a bit too saturated look when a typical film curve is applied. I shall look into that later on.

Currently I'm experimenting with inkjet printed targets; I've made some quick comparisons with real natural spectra and real skintone spectra and a homemade printed target seem to do as good or slightly better than a CC24, a Colorchecker Digital SG is slightly better on skintones (it has many patches for that) but not on nature, the skintone matching might be compensated if I actally increase density in the skintone range. The spectral shapes looks a bit better from the X-rite colorcheckers if you plot them, but in practice it doesn't seem to make that big a difference.

Grayscale patches are no good from the inkjet, black patch is good and white (=unprinted) is good, but grays don't have completely flat spectra so no good for curve calibration, my current assumption is that lens does not alter the linearity of the camera that much that curve calibration would be necessary.

With the printer on semigloss baryta paper (make sure to use OBA-free) you can make supersaturated targets, but these seem difficult to match, high saturation colors are difficult. A matte target is much more user-friendly. Matte targets can't make colors in the dark violet range though.

[jwlimages]

what prevents you from using dcamprof right now this moment ?

--  sad to admit it, one phrase: "command line" (same thing that has kept me away from Argyll). With all the apps I need to stay current with, my brain is too full already. Just too hard to keep adding more & more things to learn & master, I guess.

John

[AlterEgo]

--  sad to admit it, one phrase: "command line" (same thing that has kept me away from Argyll). With all the apps I need to stay current with, my brain is too full already. Just too hard to keep adding more & more things to learn & master, I guess.

John

you can prepare .ti3 files with rawdigger profile edition - it is basically GUI to handle the target shot for your purposes... then it is something that you can just write once in a command file - 3 lines along the following lines, for example only, you can hardcode there all directories too  :

txt2ti3 -v -i <cgats file from rawdigger> <target .cie file, with spectral data for example> <output goes to .ti3 filename>
dcamprof make-profile -i <illuminant> <use the file from the line above .ti3 filename> <output goes to profile.json>
dcamprof make-dcp -c "your camera model" -t <curve of choice> <from the line above - profile.json> <output.dcp>

not much in a very simple case !

PS: I also modified (not in published .exe build of course !) the code for myself (.exe that I am using privately) to be able to put the baseline exposure offset for my camera and camera profile name through command line to save the time.

[torger]

I have now released v0.7.0, includes test chart generator and a flatfield correction command. Patch names are now kept so it's easier to track them in the reports and plots.

I've changed the default such that targets are auto-rebalanced so the whitest patch becomes perfect reference whitepont, this is because most users expect the profile's optimal white balance be exactly the same as when you color-pick the whitest patch on the target (which in reality is a tiny bit off-white). You can disable that behavior with the -B flag.

I've also updated the documentation on white balance stuff, test target shooting and design. I've noted that semi-gloss targets are tricky to shoot, but if you do it right good results is had. Doing it right means shooting indoors with controlled (halogen) light, it's virtually impossible to shoot in daylight without some glare destroying the linearity of the photo. I haven't checked but I suspect that matte target do have some glare issues too, so for the best results shooting indoors seems to be the way to go.

In the usual place: http://www.ludd.ltu.se/~torger/dcamprof.html

[AlterEgo]

I have a question about .sp (illumination) files - it seems that you 10nm steps are not enough ? I am getting an error message when I tested one...  and why 5nm steps in your "spectrum.json" example ? argyll spotread with -H option outputs 3.x nm steps, something like = 380.00000 383.333333 386.666667 390.000000

[AlterEgo]

0.7.0 build for Windows (mingw = dcamprof.exe + libgomp_64-1.dll) : https://app.box.com/s/0dk90n0h4k171f0jr6ftxsmuupzpzl4b

[torger]

I have a question about .sp (illumination) files - it seems that you 10nm steps are not enough ? I am getting an error message when I tested one...  and why 5nm steps in your "spectrum.json" example ? argyll spotread with -H option outputs 3.x nm steps, something like = 380.00000 383.333333 386.666667 390.000000

DCamProf currently always resamples to 5nm, this is due to laziness and that I do a lot of merging of targets when working with SSFs, and you can only have one sample interval in a single .ti3. I'll probably fix that in an update so resampling is only made when necessary.

I use .sp files now and then and it works for me, but so far I've always used with smaller bands than 10nm, maybe there's a bug, I'll check.

[torger]

A few things I've learnt about camera profiling so far:

* Modern cameras make quite good matching matrix-only in D50, I can certainly see that some will satisfy with matrix-only profiles
* In Tungsten light cameras sweat more, matching is hard.
* Glossy targets are very difficult to shoot due to patch glare, dark room with directed halogen spots seems to be the way to go.
* You can't really do magic to improve matrix matching, if it's a bad match it will be bad even if you remove 50% of the highest delta E patches, in other words you don't need that many patches to get the matrix behavior.
* Targets with many closely spaced patches, like my generated one, is at risk of making conflicting adjustments. Plotting, LUT weighting/relaxation is your friend.
* If you have SSF curves, like I have for my 5Dmk2, you will see that profiles generated from targets and those from SSF look highly similar -- a good sanity check to see that it works.
* One should start with a D50/D65 profile as the camera is better at matching there, if you get a poor match it's most likely due to a bad measurement. Tungsten profiles require more tradeoffs and will likely behave badly unless you apply some LUT relaxation etc.

[AlterEgo]

A few things I've learnt about camera profiling so far:

a question :

for example I am building a simple matrix profile for ACR/LR -single illuminant, CM+FM+curve, nothing else... I have a target, measured (spectral data) + a light (H3200K) & filter (80A), measured (spectral data) = .sp

now my exercise ends with (a simple case) :

dcamprof make-profile -i <.sp file> <.ti3 file> <profile.json>

dcamprof make-dcp -t linear -L -c "SONY ILCE-7" <profile.json> <profile.dcp>

does  -i <.sp file> in this case bring any benefit for ForwardMatrix really in addition to having a "better" ColorMatrix which provides more logically sane temperature/tint values in ACR/LR UI (slider values) - but ColorMatrix does not do anything with color transform so to say (only ForwardMatrix does, after WB of course), it certainly helps somewhat to a user with a "proper/right/you name it" starting point for WB corrections in UI ? <.ti3 file> has raw RGB data from a target shot and spectral data from target measurments... it seems to me that ForwardMatrix is the same no matter what illuminant I use, right ?... but shall it be so ?

[torger]

a question :

for example I am building a simple matrix profile for ACR/LR -single illuminant, CM+FM+curve, nothing else... I have a target, measured (spectral data) + a light (H3200K) & filter (80A), measured (spectral data) = .sp

now my exercise ends with (a simple case) :

dcamprof make-profile -i <.sp file> <.ti3 file> <profile.json>

dcamprof make-dcp -t linear -L -c "SONY ILCE-7" <profile.json> <profile.dcp>

does  -i <.sp file> in this case bring any benefit for ForwardMatrix really in addition to having a "better" ColorMatrix which provides more logically sane temperature/tint values in ACR/LR UI (slider values) - but ColorMatrix does not do anything with color transform so to say (only ForwardMatrix does, after WB of course), it certainly helps somewhat to a user with a "proper/right/you name it" starting point for WB corrections in UI ? <.ti3 file> has raw RGB data from a target shot and spectral data from target measurments... it seems to me that ForwardMatrix is the same no matter what illuminant I use, right ?... but shall it be so ?

Calibration illuminant has no effect on the ForwardMatrix. It will only affect the ColorMatrix and thus it's ability to predict the correlated color temperature, which is a task specific to the DNG color pipeline (ICC will only use the forward matrix and the calibration illuminant is 100% informational, it has no effect whatsoever).

If you have a single illuminant profile the DNG pipeline does not need to know which temperature the CM was designed for, as it has only one to choose from. When making dual illuminant you need to specify EXIF lightsource so the CM is tagged with a designed temperature. You can't choose any temperature, as the temp is just a table lookup on the EXIF lightsource. Run DCamProf without args to see a list of them.

Currently it's a bit messy, your workflow would be to make the profile with the spectrum (which causes the XYZ reference values to be regenerated from spectra), but the resulting DCP will then have "Other" as calibration illuminant which will work for a single illuminant profile but not for dual, so then you need to use dcp2json/json2dcp to replace that with whatever EXIF illuminant with the closest temperature. It doesn't matter if the light is called "fluorescent" or whatever, DNG only translates it to a CCT value, no spectral shape is taken into consideration.

If the measurement data is the same, the forward matrix should not change if you specify a different calibration illuminant, yes that is correct. The reason for this is that the forward matrix specifies conversion from white-balanced camera RGB to XYZ D50, that is always D50 regardless of calibration illuminant, that is the same way as ICC profiles do. The reason one always goes to D50 is that it's more well-defined to convert further, eg to RGB for your screen and printer.

That is, colormatrix is only about the profile's ability to estimate the CCT the chosen white balance corresponds to. How much of a value that is in the Lightroom user interface I guess is a personal thing. I don't use Lightroom much, but I guess you typically start off with "As shot" white balance, and the profile won't affect the WB multipliers provided by the camera, if the color matrix is off an incorrect temperature translation will show though. If you would apply a processing preset (or whatever it's called in lightroom) which has a temperature and a tint in it the profile will affect the result as Lightroom will calculated the multipliers by using the colormatrix.

When you make a dual-illuminant profile the derived CCT will be used to mix your two forward matrices, so if you have StdA 2850K and D65 6500K, and the color matrices says the white balance is 4675K (exactly inbetween) the average of both forward matrices will be used. So in that case the color matrix has some effect on the actual color conversion, but only then.

Well there's also those ancient DCP profiles which did not have any forward matrix, in that case the color matrix has a direct effect on color rendition of course, you can make such profiles with DCamProf but I don't recommend it. Forward matrices was introduced by Adobe, maybe because they realized that the ICC way was actually better... binding to D50 gives more robust color conversions.

EDIT: corrected some statements on illuminant temperatures to not confuse future readers.

[AlterEgo]

the resulting DCP will then have "Other" has calibration illuminant which I don't think Lightroom will like (as it can't look up a temp from that), so then you need to use dcp2json/json2dcp to replace that with whatever EXIF illuminant with the closest temperature.

in fact ACR does just fine with "255" value in illumination tag... on one other forum I posted some simple experiment

Adobe's CMs (2 from from Adobe Standard dual illuminant) vs DCamProf CM/"255", single illuminant, generated with .sp for H3200K+80A filter :

Adobe + styrofoam + no 80A = 3050К/+2
Adobe + whibal + no 80A = 3050К/+2

DCamProf + styrofoam + no 80A = 3100К/-2
DCamProf + whibal + no 80A = 3100К/-2

AND

Adobe + styrofoam + 80A = 4950К/+20
Adobe + whibal + 80A = 4900K/+19

DCamProf + styrofoam + 80A = 5200/+21
DCamProf + whibal + 80A = 5150K/+19

quite close, not completely insane at least

[AlterEgo]

If the measurement data is the same, the forward matrix should not change if you specify a different calibration illuminant, yes that is correct. The reason for this is that the forward matrix specifies conversion from white-balanced camera RGB to XYZ D50, that is always D50 regardless of calibration illuminant, that is the same way as ICC profiles do.

yes, _but_ ... I have raw rgb data from the actual target shot (from raw) and the actual target measurements, sprectral, from i1pro2 (which brings its own illumination) combined in .ti3 file ... now that I have the actual illuminant in .sp then I 'd assume that the data in .ti3 file shall be adjusted somewhat to account for that, no ? raw RGB data stays the same, because it is what it is - but now that we have the spectral data of the actual illumination during the shot that shall change how target measurements now are... unless I am missing something very simple... I mean why only CM change shall be enough to account for that ?

[AlterEgo]

How much of a value that is in the Lightroom user interface I guess is a personal thing. I don't use Lightroom much, but I guess you typically start off with "As shot" white balance, and the profile won't affect the WB multipliers provided by the camera, if the color matrix is off an incorrect temperature translation will show though.

offtopic, but - I use ACR, my "as shot" WB is always UniWB, so I start with "auto" WB or I subjectively type something between 4500-6500 with zero tint based on whether I want it colder/warmer in general...

[torger]

in fact ACR does just fine with "255" value in illumination tag... on one other forum I posted some simple experiment

Adobe's CMs (2 from from Adobe Standard dual illuminant) vs DCamProf CM/"255", single illuminant, generated with .sp for H3200K+80A filter :

Adobe + styrofoam + no 80A = 3050К/+2
Adobe + whibal + no 80A = 3050К/+2

DCamProf + styrofoam + no 80A = 3100К/-2
DCamProf + whibal + no 80A = 3100К/-2

AND

Adobe + styrofoam + 80A = 4950К/+20
Adobe + whibal + 80A = 4900K/+19

DCamProf + styrofoam + 80A = 5200/+21
DCamProf + whibal + 80A = 5150K/+19

quite close, not completely insane at least

Thanks for this, I had forgot, I needed to look into dngref.c to remember.

lsOther is fine as long as it's not dual-illuminant.

ACR will simply feed the white balance to the (inverted) color matrix so it gets XYZ and converts to a xy chromaticity coordinate which then is through a standard algorithm converted to a blackbody temp and tint. It doesn't need to know which temperature the color matrix was designed at, but it will make more accurate temperatures if the tested white balance is close to the designed.

When dual illuminant ACR needs to know which temperature each matrix was designed for, so it knows how much to mix in from each.

[torger]

yes, _but_ ... I have raw rgb data from the actual target shot (from raw) and the actual target measurements, sprectral, from i1pro2 (which brings its own illumination) combined in .ti3 file ... now that I have the actual illuminant in .sp then I 'd assume that the data in .ti3 file shall be adjusted somewhat to account for that, no ? raw RGB data stays the same, because it is what it is - but now that we have the spectral data of the actual illumination during the shot that shall change how target measurements now are... unless I am missing something very simple... I mean why only CM change shall be enough to account for that ?

The CM will need XYZ reference values for your calibration illuminant, so those will be recalculated from the reflectance spectra and your provided illuminant spectrum.

The FM will need XYZ reference values for D50 so those will not change, they're always calculated for D50, so the calibration illuminant won't matter. This may seems strange like it would be always the same FM regardless light, but don't forget that the camera RGB values are affected by the light you use during test target shooting, and that affects the FM. The task for the FM is convert RGB values that have been shot under the calibration illuminant and then white balanced. But DCamProf does not need to know which light you used for that, the RGB values are already there. And the target space is always XYZ D50 for the FM.

Only if you provide camera SSF curves so the RGB values can be recalculated from scratch the provided calibration illuminant will change the FM result.

Mathematically speaking it can be seen as a problem that FM always makes XYZD50 values, if you shoot under Tungsten wouldn't we want the tristimulus values for Tungsten? Well, the problem is that to convert to RGB or any other space we need to go through D50 (profile connection space) so there would still be a conversion to D50, using some sort of CAT. Then it's better if we make the conversion when designing the profile with full knowledge of spectra.

So a profile will correct the colors to make them look as if it was shot under D50, regardless of actual light used during shooting. For extreme light temperatures the eye will not fully adapt and colors will not appear the same as in D50, so the profile won't do the "correct" job for those temperatures. Is there a way to convert say 2000K color appearance but into D50 XYZ coordinates? I don't really know, maybe CIECAM02 can do that. It can be worth investigating. I suspect that it won't be feasible though, on cannot put on a slight red tint for example as the DNG pipeline requires the FM to map to neutral white. So for now it's up to the photographer's eye and raw converter sliders to simulate any changes in look due to extreme color temperatures.

[AlterEgo]

OK, after reading the notes here and there I am coming back to peace of mind then... at least I have lesser work with WB  , so that my exercise with spectrum measurments were not totally wasted

[torger]

OK, after reading the notes here and there I am coming back to peace of mind then... at least I have lesser work with WB  , so that my exercise with spectrum measurments were not totally wasted

Yes providing the spectrum gives the best XYZ reference values when the CM is derived, so it will be as good as it can be at estimating wb temperatures. Of course it will be best at estimating temperatures close to the actual temperature used when profiling, but any reasonable modern camera should have similar matrices over at least some temperature range so it should work for quite some range. I haven't made any specific investigation on how well it works though.

If one shoot under say tungsten but provide XYZ reference values for D50 the CM will certainly be bad at wb temperature guessing, so if one doesn't have a spectrometer so one cannot provide the spectra it's preferable to be reasonable good at guessing the temperature and provide a EXIF illuminant for that when making the profile. Using Adobe's own dual illuminant profiles and make a test shot can be used as an alternative to get some idea if unsure.

[AlterEgo]

and some general question, I did not even look into that area, so if it sounds stupid never mind - but given the set of real world patches (through the supplied spectral measurements) data, can software produce a target that shall be a close to "ideal" (what is the "ideal" though ? the next question) one with the given (as a parameter) number of patches ? for example I give a task to software - find me the best target with 24 patches only from the given data for patches (for example I have 200 different patches @ my disposal) that I can use and software then gives me which 24 out of those 200 I shall use to achieve the best result (may be defined as some dE over a big virtual set of patches in your database like Munsell set, etc) ? some kind of automated finding of the best target withing given limitations (# of patches in the target, set of patches that I can only use, illuminant may be and virtual set of patches that I want to aim at in terms of good conversion with that target that software suggests)

[torger]

In order to solve that problem the profiler needs to know the camera's SSF (as the solution will depend on that), but if you already know the camera's SSF you can profile directly against the real spectral data.

On a related subject, I have thought a little about estimating SSFs from a large set of printed patches, but I don't think meaningful precision can be had. There has been some academic work on SSF estimation, but as far as I know no method that reach the level of precision we would be interested of has been done. There is the method of using emissive targets with narrow-band filters, Image Engineering has such systems commercially, but I don't think those are feasible for enthusiast budgets.

[AlterEgo]

In order to solve that problem the profiler needs to know the camera's SSF (as the solution will depend on that), but if you already know the camera's SSF you can profile directly against the real spectral data.

that's when I do not... but why it is not possible w/o camera's SSF ? we can imagine some average, idealized SSF at least ?

On a related subject, I have thought a little about estimating SSFs from a large set of printed patches, but I don't think meaningful precision can be had. There has been some academic work on SSF estimation, but as far as I know no method that reach the level of precision we would be interested of has been done.

I modified that matlab script for my camera and IT8 target, just as an exercise



I will be redoing it with some additional measurements though, to see if it becomes better or not