Pages: 1 ... 32 33 [34] 35 36 ... 78   Go Down

Author Topic: DCamProf - a new camera profiling tool  (Read 767350 times)

torger

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 3267
Re: DCamProf - a new camera profiling tool
« Reply #660 on: July 08, 2015, 05:53:47 am »

Thought I'd post an example. I haven't actually made that many finished profiles as I've always been busy fixing something or adding some new feature.

Attached is an example dual-illuminant DNG profile for Canon 5D mark II, made using DCamProf v0.8.2 with the workflow outlined in the code block. It uses a CC24 for base colors combined with a homemade semi-glossy inkjet target for extreme colors. Light sources are halogen with voltage control, a normal lamp for StdA and a Solux on overdrive for D55. The uneven light (only one point light) has been corrected with DCamProf's flatfield correction function.

A curve was designed "by eye" in RawTherapee to reasonably match the camera's JPEG brightness/contrast, so we get predictable result when using the camera's auto exposure. The Canon does quite some black subtraction to blacken shadows in its JPEGs, I've chosen to have lower contrast shadows and keep detail. Then this curve is applied when generating the profile using DCamProf's neutral tone reproduction operator.

CAT02 was used for StdA apperance rendering, while Solux "D55" was deemed to be close enough to D50 so no CAT was used there (-C).

The DCP contains two pairs of matrices (StdA and D55) which aim to make best match for the normal range of colors (represented by the CC24), the glossy target has been excluded from matrix correction. Then there are two 2.5D HueSatMaps (StdA/D55) for further colorimetric correction, and which also stretches the borders to decently match the glossy colors. Weighting and relaxation has been used to (my) taste to make a tradeoff between smoothness and accuracy.

Then we have the Looktable+Tone Curve which implements the subjective contrast increase and with maintained neutral reproduction. The Looktable has its value dimension gamma-encoded for better perceptual resolution. The looktable is what makes the profile fairly big, as it must be a 3D LUT. Finally there's a baseline exposure offset of -0.3, which is supposed to be added on top of Adobe DNG Converter's baseline exposure of 0.4 stops it adds for 5Dmk2 files. If you don't use Adobe-generated DNGs (ie have 0 baseline exposure) that should be changed to +0.1 stops.

If you have "toggable" DCP handling like a RawTherapee built after July 6, you can turn off the Looktable+Curve to get a colormetric profile, and then turn off the HueSatMap to get a pure matrix profile. The elements are sort of layered, first matrix, then huesatmap on top and then looktable and then curve. I like that aspect of DNG profiles, but unfortunately this aspect is rarely used in profile design, and I think only RT supports toggling the elements in it.

   # 1. Make a custom target to top of the CC24 with some super-saturated colors
  dcamprof make-testchart -l 15 -d 14.46,12.26 -O -p 210 custom-target.ti1
  printtarg -v -S -iCM -r -h -T300 -p A4 custom-target
    # 2. Print custom target to a semi-glossy OBA-free paper
    # 3. Scan the target with a spectrometer
  chartread -v -H -T0.4 custom-target
    # 4. Create reference file
  spec2cie -v -i D50 custom-target.ti3 glossy.cie
    # 5. Setup light and measure spectrum, save to light.sp for later use
  spotread -a -H -x
    # 6. Shoot CC24 target, glossy target, and white card.
    # 7. Crop export and convert to cc24.tif, glossy.tif and ff.tif
    # 8. Apply flatfield
  dcamprof testchart-ff cc24.tif ff.tif cc24-ff.tif
  dcamprof testchart-ff glossy.tif ff.tif glossy-ff.tif
    # 9. Scan values
  scanin -v -dipn cc24-ff.tif ColorChecker.cht cc24.cie
  scanin -v -dipn glossy-ff.tif custom-target.cht glossy.cie
    # 10. Merge targets into one, letting CC24 have priority
  dcamprof make-target -p cc24-ff.ti3 -a cc24 -p glossy-ff.ti3 -a glossy -d 0.03 combo.ti3
    # 11. Make a preliminary profile, dumping plots. Exclude glossy
    #     from the matrix optimizer to get as good base match as
    #     possible for the important normal colors represented by cc24
    #     Add -C (no CAT) to all make-profile below if your light is
    #     close to D50.
  dcamprof make-profile -r dump1 -w cc24 0,1 -w glossy 0,0 -i light.sp combo.ti3 preliminary.json
    # 12. Start gnuplot (cd dump1; gnuplot -background gray) and plot
    #     target and LUT, plus LUT stretch vectors and DE vectors.
    #     Use 'set view equal xyz' and 'set view equal xy' to turn
    #     on/off scaling of lightness axis, must be turned off if
    #     error vectors are viewed in 3D.
  gnuplot> splot 'nve-lut.dat' w l lc "beige", 'gmt-locus.dat' w l lw 4 lc rgb var, \
    'gmt-adobergb.dat' w l lc "red", 'gmt-pointer.dat' w l lw 2 lc rgb var, \
    'target-nve-lutve2.dat' w vec lc "black", 'target-nve-lutvm.dat' w vec lw 2 lc "olive", \
    'targetd50-xyz.dat' pt 4 lc rgb var, 'targetd50-xyz.dat' using 1:2:3:5 w labels offset 3
    # 13. Look in the plot for patches that pull in opposite directions and cause a
    #     bad bend in the LUT. Add those (typically one or two) to an
    #     exclude.txt and render
  dcamprof make-profile -r dump1 -x exclude.txt -w cc24 0,1 -w glossy 0,0 -i light.sp combo.ti3 preliminary.json
    # 14. Make matrix-only and full correction profiles for sanity
    #     check comparisons later
  dcamprof make-dcp -n "Canon EOS 5D Mark II" preliminary.json no-relax.dcp
  dcamprof make-dcp -n "Canon EOS 5D Mark II" -L preliminary.json matrix.dcp
    # 15. Relax the LUT (primarily for the glossy class) to improve
    #     smoothness. Re-render and replot for each change.
    #  a) It may be worthwhile to lock the matrix before changing DE k
    #     weights: save profile to separate file matrix.json and
    #     provide -m and -f parameters.
    #  a) Try relaxing DE weight even for the important CC24, setting
    #     it to at least 1 leads often to some relax without much loss
    #     in accuracy.
    #  b) Try changing CIEDE2000 k weights, 4,1,1 good start (less
    #     weight on lightness)
    # Example result after iterating:
  dcamprof make-profile -r dump1 -x exclude.txt -f matrix.json -m matrix.json \
    -w cc24 1,1,4,1,1 -w glossy 2,0,4,4,1 -i light.sp combo.ti3 final-1.json
    # 16. Make final DCP, sanity check it by comparing it with
    #     no-relax.dcp and matrix.dcp. It should have better high
    #     saturation correction than matrix.dcp, and not lose too much
    #     accuracy compared to no-relax.dcp
  dcamprof make-dcp -n "Canon EOS 5D Mark II" -d "Final 1" final-1.json final-1.dcp
    # 17. Repeat steps 5 - 16 for the second illuminant
    # 18. Decide which tone curve to use, which baseline exposure
    #     offset if any, and if black subtraction should be
    #     automatically applied or not. Often the default "acr" curve
    #     with no baseline exposure offset provides what you want,
    #     unless you more closely want to match camera's JPEGs. If you
    #     prefer to make visual accuracy comparisons with tone curve
    #     applied, you can apply the curve earlier in the process.
    # 19. Merge to a dual-illuminant profile and apply your curve with
    #     DCamProf's neutral tone reproduction operator (enabled per
    #     default).
  dcamprof make-dcp -n "Canon EOS 5D Mark II" -d "My Profile" -i StdA -I D50 -b 0.1 -t curve.rtc final-1.json final-2.json final-dual.dcp
« Last Edit: July 08, 2015, 05:58:37 am by torger »
Logged

Iliah

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 770
Re: DCamProf - a new camera profiling tool
« Reply #661 on: July 08, 2015, 10:36:07 am »

>> power supply control to stabilize light is another one).

> I am quite interested in that (assuming you are talking about halogen sources) - would you point me in a right direction?

Much depends on the power supply you are using currently. If it is not a stabilizing one, you can use a dimmer with a sine wave output in front of it and control the dimmer using the feedback from the diode. If it is stabilizing, you may want to replace it with a regulated power supply, or introduce a dimmer right after it.
Logged

Alexey.Danilchenko

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 257
    • Spectron
Re: DCamProf - a new camera profiling tool
« Reply #662 on: July 08, 2015, 11:03:16 am »

> I am quite interested in that (assuming you are talking about halogen sources) - would you point me in a right direction?

Much depends on the power supply you are using currently. If it is not a stabilizing one, you can use a dimmer with a sine wave output in front of it and control the dimmer using the feedback from the diode. If it is stabilizing, you may want to replace it with a regulated power supply, or introduce a dimmer right after it.

No idea really, the one that is in the light source and it seems a rather primitive one: transformer with small board and regulator which provides 0..+17.6V on lamp connector.
Logged

Iliah

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 770
Re: DCamProf - a new camera profiling tool
« Reply #663 on: July 08, 2015, 11:07:33 am »

No idea really, the one that is in the light source and it seems a rather primitive one: transformer with small board and regulator which provides 0..+17.6V on lamp connector.

Than you may want to see if you can control the regulator, if it is a variable resistor you may be able to replace it with either analog output of Arduino, or with a digital potentiometer like https://www.sparkfun.com/products/10613
Logged

Alexey.Danilchenko

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 257
    • Spectron
Re: DCamProf - a new camera profiling tool
« Reply #664 on: July 08, 2015, 11:22:45 am »

Than you may want to see if you can control the regulator, if it is a variable resistor you may be able to replace it with either analog output of Arduino, or with a digital potentiometer like https://www.sparkfun.com/products/10613

Yep its a variable resistor (old and large). At first attempt I was trying not to change the light intensity - but set it to max output at 17.6V (I have Philips halogen bulb - this one), measure all SPDs and photodiode output with 5nm steps, then take a series of raws and correct them following the measured SPDs (to max SPD and adjusting by photodiode measurement deviation). The light stayed the same throughout the experiment.

If I understand you correctly you are saying that I need to build a calibration table and then attempt to control the light output with it using photodiode as a feedback adjusting to the same level (presumably blue one as it is the weakest)? If this is correct then I'd need that rotary encoder to know what wavelength it is at since photodiode output although linear is different in different parts of the spectrum.
Logged

Iliah

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 770
Re: DCamProf - a new camera profiling tool
« Reply #665 on: July 08, 2015, 11:44:10 am »

First, I would inspect the resistor (may take de-soldering), read the markings, see how it is used (potentiometer, rheostat with two pins connected together, or one pin free). It is also important to measure the current through the resistor. Next, I would figure out how to replace it with a processor-controllable element.

The goal is to maintain the light flux. This is usually achieved through a simple PID regulator (many implementations are available for microprocessor, it is bread-and-butter code - an example http://playground.arduino.cc/Code/PIDLibrary ). The input to the regulator is the measurement of the light (diode reading), the output is the "variable resistor" control signal.
Logged

Alexey.Danilchenko

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 257
    • Spectron
Re: DCamProf - a new camera profiling tool
« Reply #666 on: July 08, 2015, 11:59:48 am »

Thanks Iliah will do that
The goal is to maintain the light flux.

Do you mean luminous flux or radiant flux? The photodiode does not allow the radiance hence the tables and SPD or am missing something and the simple alignement  of light output to have the same photodiode readings will do?
Logged

Iliah

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 770
Re: DCamProf - a new camera profiling tool
« Reply #667 on: July 08, 2015, 12:10:32 pm »

> the simple alignement  of light output to have the same photodiode readings will do

Yes, that will do, works from the times of early "colour stabilizers" in enlargers. If one wants to go overkill, TCS3471x / TCS3472x and its variations are just what the doctor ordered :)
« Last Edit: July 08, 2015, 12:17:16 pm by Iliah »
Logged

Alexey.Danilchenko

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 257
    • Spectron
Re: DCamProf - a new camera profiling tool
« Reply #668 on: July 08, 2015, 12:24:01 pm »

> the simple alignement  of light output to have the same photodiode readings will do

Yes, that will do, works from the times of early "colour stabilizers" in enlargers. If one wants to go overkill, TCS3471x / TCS3472x and its variations are just what the doctor ordered :)
So with this then there is no need to scale raw values to adjust for intensity difference (provided all exposures were the same)? If so great - that is easier for me than taking 66 i2pro measurements and processing them all. Thanks Iliah for the help
Logged

Iliah

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 770
Re: DCamProf - a new camera profiling tool
« Reply #669 on: July 08, 2015, 12:37:42 pm »

> So with this then there is no need to scale raw values to adjust for intensity difference (provided all exposures were the same)?
Generally, yes; subject to periodic testing, taking SPD, and calculating CCT to log the aging of the lamp (I do it after each 10 hours of use approximately).
Logged

AlterEgo

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1995
Re: DCamProf - a new camera profiling tool
« Reply #670 on: July 09, 2015, 09:58:23 am »

at least it seems I have a chance to get a "proper" target, will see how it measures then... albeit if they state that the formulation was changed it still leaves the question about some /data/ basis vs which to compare your own measurements to see if something is out of the ordinary (unless they have a stock of legacy targets from the old days somewhere), like with this one from Jan 2014 - I am curious to see how all the devices I can use will measure the black patches if I /hopefully/ receive a replacement...

so after some ping-pong with X-Rite's customer service about RMA & proof of purchase I received today a replacement for my ColorChecker SG target... the one that was sent by X-Rite is "November 2014" edition (as printed on the back of the target) now, so this evening I will try to measure it and see how the patches are there (or rather how my spectrophotometers will be metering them).
Logged

AlterEgo

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1995
Re: DCamProf - a new camera profiling tool
« Reply #671 on: July 10, 2015, 09:50:31 am »

So, they knew it is Jan 2014 beforehand. Does not add much credibility to their "only".

I did one measurement (did not have time to do several to average yet) with a newer out of 2 spectrolinos that I have = attached

Results looks "better", at least black patches are darker in November 2014 edition of CC SG.
Logged

torger

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 3267
Re: DCamProf - a new camera profiling tool
« Reply #672 on: July 26, 2015, 02:55:47 pm »

I've just released version 0.8.3: http://www.ludd.ltu.se/~torger/dcamprof.html

The main thing is that I've reworked the tone reproduction operator as a part of the work of providing a realtime version for RawTherapee (which I just committed too). It works along the same principles and the result of it is almost the same as before, but there are some slight improvements especially concerning highlight rolloff. I've had some extra expert eyes to help me fine-tune weights. It has a bit better skin tone performance now.
Logged

AlterEgo

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1995
Re: DCamProf - a new camera profiling tool
« Reply #673 on: July 26, 2015, 09:28:20 pm »

0.8.3 build for Windows (mingw = dcamprof.exe + libgomp_64-1.dll + HTML & PDF manual / = copy of Torger's web page and the same converted to PDF /) : https://app.box.com/s/6dbmzwrvucjz6tprdhvit2cnjpdn3mo1
Logged

torger

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 3267
Re: DCamProf - a new camera profiling tool
« Reply #674 on: July 27, 2015, 03:31:09 am »

Seems like tone reproduction can keep one busy for a while.

With DCamProf the intention has from the start to make neutral profiles, but as soon as contrast should be applied it's impossible not to bring some taste to the table.

To keep color appearence with increased contrast I've noted that non-linear 3D tables are required. So far I keep hue stable, but saturation is changed depending on contrast, and also depending on lightness (more saturation increase of dark colors) and source saturation. This is not to create a "look", but to keep color appearance as constant as possible.

With the latest release I've made it possible for the user to tune all these weighting parameters with a JSON configuration file.

Another big thing when it comes to tone reproduction is highlight rolloff. It's often possible to retain the correct color close to clipping, but I've noted that there will then be problems with the look. We can't change that the brightest spot on the media is white, so making a nice transition to the whitepoint is an important aspect, even if it hurts color precision.

As said DCamProf doesn't attempt to do any subjective looks so far. I might add features for that at some point, but it will be difficult to use without a GUI, I would have to provide pre-made recipes that the user can adjust to taste. A typical base recipe is to warm up highlights/cool down shadows, limit saturation of high saturation colors, increase saturation of mid-saturated colors (being careful with skintones). I think it's better to control the look in the raw converter, but unfortunately few raw converters have tools that allow these types of fine adjustments.

The attached image shows a comparison between a DCamProf profile as used in RT and Hasselblad's native Phocus rendering and illustrates a neutral vs subjective look, the camera is a H4D-50. At first glance the images are very similar, and one reason for that is that I have matched the contrast curve quite well (I like my profiles to provide the same contrast as the native camera images so default rendering becomes predictable). Another reason is that Hasselblad's color is quite sane, not crazy pop as you can see in some consumer cameras, and is thus fairly close to the neutral result.

However looking closer we see some differences, the main one that it seems like the Hassy image has a warmer white balance. This is not the case though, the white balance multipliers are exactly the same, and if we look closer at neutrals and shadows the white balance seems the same, what the Hassy look does is to warm up some of the saturated tones. The Hassy look has a little bit more saturation overall, but the difference is certainly not huge. Warming up of tones does cause some notable hue errors, here seen in the red pants that look more yellow-red rather than the actual wine-red.

The example image is not good for comparing skin tones, but anyway I've looked at other more suitable pictures for that and quite big differences can seen there, much more than in the attached example. Hassy's subjective look make caucasian skin quite a lot more yellow/golden, while a neutral look has more red as there is in reality. Golden skin looks smoother though so I'd expect many prefer that look, and that's why Hassy have hard-coded it into their profile.

Since I've published DCamProf I've got quite many requests on making profiles for camera XYZ as the manufacturer provided profile "is no good". I've been a bit reluctant to help as I know there is a big chunk of subjectivity involved. Making a neutral realistic profile is now quite easy with the tone operator in place, but I'm not so sure that is what most people want especially when it comes to skin tones.
« Last Edit: July 27, 2015, 03:41:46 am by torger »
Logged

torger

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 3267
Re: DCamProf - a new camera profiling tool
« Reply #675 on: July 30, 2015, 11:11:28 am »

I got a request to be able to disable the lightness correction all-together, and with the new 0.8.4 release this is now possible, just specify "-l -1,0"  to make-profile and the LUT will only work with chromaticity.

As a sidenote this is how Adobe's DNG Profile Editor does, the LUT only corrects hue and saturation and leave lightness (actually RGB-HSV value) unchanged, or rather leave it at the result provided by the matrix. There is some robustness advantages to this, measurement errors in lightness is larger (I think) than in chroma and hue and there seems to be some robustness advantages in handling of clipped highlights.
Logged

AlterEgo

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1995
Re: DCamProf - a new camera profiling tool
« Reply #676 on: July 30, 2015, 12:15:21 pm »

0.8.4 build for Windows (mingw = dcamprof.exe + libgomp_64-1.dll + HTML & PDF manual / = copy of Torger's web page and the same converted to PDF /) : https://app.box.com/s/if132xhpl94u32jb4kg9h43oaz197xc9
Logged

Bart_van_der_Wolf

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 8913
Re: DCamProf - a new camera profiling tool
« Reply #677 on: July 30, 2015, 01:05:33 pm »

Thanks guys, for the ongoing improvements, and Windows binaries.

Cheers,
Bart
Logged
== If you do what you did, you'll get what you got. ==

torger

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 3267
Re: DCamProf - a new camera profiling tool
« Reply #678 on: August 04, 2015, 06:10:30 pm »

Here's a guess game for you;

I'm working with subjective tuning features in DCamProf so you can make the profile render "flattering" skin tones rather than accurate (and any other subjective tunings you may want to do). It's not exactly easy tuning without a GUI, but it can be done. You write a specification of the subjective adjustments in a JSON config file.

The attached image shows three images, one is a neutral DCamProf profile with a tone curve using the neutral tone reproduction operator, one is my work in progress concerning hand-tuned skin tones using DCamProf, and one is a Hasselblad Phocus original rendering. The camera is a H4D-50.

So which one is which? And which skin tone rendering do you prefer?
Logged

Tim Lookingbill

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2436
Re: DCamProf - a new camera profiling tool
« Reply #679 on: August 04, 2015, 09:06:46 pm »

Can't tell you which one is which but I prefer 'A''s skin tone.

Based on 40 years of observation I can say all skin has some yellow in it at any given time of day, shade or sunshine unless the subject is an albino.

'B' & 'C' are a bit too pinky where I see the tuft of hair just above the forehead has a magenta tinge. Being able to get rid of any Rosacea or other skin conditions that show pinkish hues in a profile is a big plus. I have to make hue adjustments constantly using ACR/LR's HSL.

Good work, torger. Thanks for posting samples. It helps me make sense of this thread.
Logged
Pages: 1 ... 32 33 [34] 35 36 ... 78   Go Up