DCamProf - a new camera profiling tool

[AlterEgo]

In what way it is not going to be worse?

because you always can do around the same transform as matrix profile does around those areas, no ? matrix is LUT of sort too, just the one that does purely linear transform (multiplication).

[Alexey.Danilchenko]

because you always can do around the same transform as matrix profile does around those areas, no ? matrix is LUT of sort too, just the one that does purely linear transform (multiplication).

You can use matrix to generate the same nodes as LUT yes. But whether the results of applying such profile for colours that located between those nodes will match the matrix one remains debatable.

[AlterEgo]

You can use matrix to generate the same nodes as LUT yes. But whether the results of applying such profile for colours that located between those nodes will match the matrix one remains debatable.

so Alex, what is going to happen if you (hopefully) get your monochromaror + sphere + ...  working ? you will get SSF and then you will build matrix (plus possible gamma/s/ or shaper/s/) profile or you will go for LUT ?

[Alexey.Danilchenko]

so Alex, what is going to happen if you (hopefully) get your monochromaror + sphere + ...  working ? you will get SSF and then you will build matrix (plus possible gamma/s/ or shaper/s/) profile or you will go for LUT ?

I will build  my perfect matrix profile (not relying on CC24 or other targets) 

I was not even interested in LUTs. If you need specifics I am interested in finding the better fitting matrix for (a) different illuminants and (b) for better matching the digital back with specific IR filter (the one I have namely). LUT would be interesting for me if I had nonlinearities (perhaps like those in D800) to correct for but as it stands I don't need it.

DCamProf is a perfect tool for me to experiment with these.

[AlterEgo]

I will build  my perfect matrix profile

waiting to see the results then !

[torger]

Just to show an example, this is how a LUT stretching can look like, the test target is an IT8 which has been cleaned up a bit to exclude most darker patches, the camera is a P45+.

Colored squares show the target positions, arrows show the LUT stretch from matrix position to end position (generally close to the target). The grid shows the LUT thin plate spline in action. There's some stretch in the lightness direction too (not seen in this view) but it's very small.

There are indeed quite large stretches, but no bad bends as there are no contradicting stretches.

It shows very typical behavior in that for low saturation colors there are small stretches (ie the matrix comes close to target), and for high saturation colors there are larger stretches. The u'v' diagram is not perfectly uniform though so it exaggerates that effect a bit.

This is natural, more saturation means more narrow band, which means that the difference between the camera's SSFs and the observer's CMFs are enlarged and becomes harder to correct with a linear equation.

A general comment on the IT8 target, the many repeats of the darker shades does not provide that much value, but the target does reach considerable higher saturation than a cc24, some patches even outside Pointer's gamut, so it looks like it's quite a good target.

[AlterEgo]

A general comment on the IT8 target, the many repeats of the darker shades does not provide that much value

I'd love to see some further discussion, becuse just recently somebody was stating even that intentional bracketing is worthwhile - see http://forum.luminous-landscape.com/index.php?topic=22471.msg818606#msg818606

[AlterEgo]

so it looks like it's quite a good target.

I had this one from WFaust for like 7 years and it was somewhere far away in the attic... but I extracted it back couple of days ago and liked (to my subjective taste) some results, I am certainly not an expert to render a scientific grade judgement - just my personal impression (so far)... for further experimenting I am assembling some additional gear - like it was suggested 80A filters for halogen light, waiting to get them... easel to mount targets on a proper surface (metal with magnets) - again as for example IBorg suggested

[Alexey.Danilchenko]

waiting to see the results then !

Sure - if I make the monochromator work...

[torger]

I'd love to see some further discussion, becuse just recently somebody was stating even that intentional bracketing is worthwhile - see http://forum.luminous-landscape.com/index.php?topic=22471.msg818606#msg818606

I think Iliah is referring to bracketing as a method to keep down noise. I don't think that is worthwhile either though. If by principle there will be relatively large errors in the results, say "5%" it doesn't provide much value to reduce measurement error from 0.5% to 0.1%. There is such a thing called overkill.

Darker patches provide some value in that they are probably printed using a different colorant mix so you get a slightly different spectrum shape than a lighter color of the same hue and chroma, that way you can make some more average correction. For very dark patches I would guess they contribute more noise than meaningful correction though. And yes I guess in that case bracketing would help, but why not have a test target with brighter colors instead I think.

Anyway, if you try to make exact correction for each shade, you are in 3D LUT space and pretending that the camera will be used as a scanner, with fixed light, fixed exposure *and* fixed media. The IT8 target steps is nice for a slide film target for a slide film scanner, I actually have one and have made Argyll ICC profiles for a slide scanner using it. As long as the slides you're scanning are made on the same film as the slide target you're getting very accurate results.

With a camera used in general conditions it won't be like that. Let's pretend for a while that the light is fixed and the media is fixed, but not the exposure. Then the 3D LUT correction still breaks, since when a color is exposed brighter than it was for the LUT it will be corrected wrong.

That is to DCamProf with a 2.5D LUT the only thing the darker shades provide is slight variation, but then I think a neighboring chromaticity of similar lightness would be better use of target space.

[torger]

Sure - if I make the monochromator work...

I'm thinking about getting a monochromator too. You can get one new for about $1200. Most use a spectroradiometer ($10000) to measure the output, and many an integrating sphere to smoothen the light ($2000 I suppose). So naturally one would like to skip the spectroradiometer and integrating sphere.

I'm thinking that one could use a halogen lamp, measure the spectra with a consumer spectrometer, then hope it will stay stable enough while filtering it through the monochromator and doing measurements, and you shoot the slit directly, without any diffuser. I think it could work, but I don't know, may be issues with reflections. If one uses a simple white card to reflect the light shooting becomes simpler but measuring the reference becomes harder, then one needs a spectroradiometer rather than a consumer spectrometer.

What's your plan?

Here below a high end setup:
http://www.cis.rit.edu/jwgu/research/camspec/img/camspec_database.png

[ErikKaffehr]

Hi Anders,

I have checked back on this and I now believe it is Lightroom that applies some kind of shoulder on the highlights. So it is not a profile issue.

Best regards
Erik

Hi,

....

There is an issue, that some of the light grey samples are to dark.

Excellent job from Anders. I hope he will continue to develop the tool.

Best regards
Erik

[AlterEgo]

With a camera used in general conditions it won't be like that. Let's pretend for a while that the light is fixed and the media is fixed, but not the exposure. Then the 3D LUT correction still breaks, since when a color is exposed brighter than it was for the LUT it will be corrected wrong.

what is the difference between more exposure time and more light (intensity wise = brighter patches) in this example ?  you will not see the difference in the raw file (let us ignore sensor heating)

[Alexey.Danilchenko]

I'm thinking that one could use a halogen lamp, measure the spectra with a consumer spectrometer, then hope it will stay stable enough while filtering it through the monochromator and doing measurements, and you shoot the slit directly, without any diffuser. I think it could work, but I don't know, may be issues with reflections. If one uses a simple white card to reflect the light shooting becomes simpler but measuring the reference becomes harder, then one needs a spectroradiometer rather than a consumer spectrometer.

What's your plan?

Much more modest - I got the full spectrum monochromator (manual one with light source) but no power supply (and a weird connectors for PSU). Not sure if it will work and it will need some calibrating/checking. I also will have an integrating sphere (albeit a small one) though I am half way though building my own (not so high spec of course as spectralon coated ones). Will update with progress if I make it work.

I am not sure how accurate the measurement without the sphere will be since you need to measure the irradiation (intensity) to adjust the camera responses and without sphere those readings will be hit or miss.

These discussions should perhaps go to a different thread - this one is about DCamProf after all.

[AlterEgo]

Hi Anders,

I have checked back on this and I now believe it is Lightroom that applies some kind of shoulder on the highlights. So it is not a profile issue.

Best regards
Erik

generally it shall be process 2010 in ACR/LR and also just in case infuse a linear tone curve (applied post exposure adjustments) in profile as
 
  <ToneCurve Size="2">
    <Element N="0" h="0.000000" v="0.000000"/>
    <Element N="1" h="1.000000" v="1.000000"/>
  </ToneCurve>


in dcptool speak

[AlterEgo]

What's your plan?

have to rob nearby university... or enroll in MSc  (repeat for each new camera)

[torger]

generally it shall be process 2010 in ACR/LR and also just in case infuse a linear tone curve (applied post exposure adjustments) in profile as
 
  <ToneCurve Size="2">
    <Element N="0" h="0.000000" v="0.000000"/>
    <Element N="1" h="1.000000" v="1.000000"/>
  </ToneCurve>


in dcptool speak

I should probably add a function to add a curve to the DCP. If there is no curve, like DCamProf DCPs are, some raw converters seem to add a default curve which may not be what you want. Meanwhile one can add one manually with dcp2json / json2dcp (or dcptool I suppose)

[torger]

what is the difference between more exposure time and more light (intensity wise = brighter patches) in this example ?  you will not see the difference in the raw file (let us ignore sensor heating)

No difference. But I was describing an example when exposure changes for a fixed light condition.

The difference then is that the LUT was designed with the patches at a different lightness. With a 2.5D LUT as DCamProf uses it does not matter, as the index into the LUT is only chromaticity, not lightness. If there's a 3D LUT index is hue, saturation (chromaticity) plus lightness. To match the correct patch the exposure must be the same as when the LUT was designed.

When lightness is not a parameter into the LUT there is much less value in making steps, the only value is in adding slight variation in terms of chromaticity and spectral shapes.

[Alexey.Danilchenko]

I should probably add a function to add a curve to the DCP. If there is no curve, like DCamProf DCPs are, some raw converters seem to add a default curve which may not be what you want. Meanwhile one can add one manually with dcp2json / json2dcp (or dcptool I suppose)

Lightroom/ACR add default one which is mix between gamma and contrast curve. So if linear profile is needed, then curve should be explicitly stated.

[kirkt]

Just a bug (maybe) in the "txt2ti3" command of dcamprof.

I assume that the input is a text file, whitespace delimited, that contains the spectral data from, for example, PatchReader.  Nothing else.  My PatchReader data is in 10nm increments, in rows, from 380nm to 730nm.  I cut and paste the spectral data from the PatchReader output text file into a text document called "spectral_data.txt" and issue:

$ dcamprof txt2ti3 -f 380,730,10 -l rows spectral_data.txt spectral_data.ti3

The range of the spectral elements is correct (380-730) but the increments are still in 5nm (the default) not 10nm (my command line argument to the -f flag).  Attached is a test file - spectraltest.txt - that is a copy of the whitespace delimited input file.

Still, it beats editing the PatchReader output by hand to conform to the reference CIE file you include with your distribution!  

kirk

EDIT -Is this command supposed to reissue/recalculate/interpolate my input spectra into 5nm increments, regardless of what the specified input spacing is?  If so, sorry, I misunderstood what the output was supposed to be!

EDIT2 - The scaling seems to be buggy though - I noticed in the reference CIE file that you include, the range of spectral values is 0-100 (not 0.0-1.0).  Given this reference range,  I output my values from PatchReader as 0-100.  When I run txt2ti3, I do not specify a scaling value (I assume the default, -s 1.0 according to the help message).  However, my spectral values get scaled by 100 in the output file.  If I use "-s 0.01" then the scaling is not applied and the output values are identical to the input values.  Is there a multiplication of 100 going on somewhere?  Or is the default value for the -s flag 100 and not 1.0?