Pages: 1 ... 47 48 [49] 50 51 ... 78   Go Down

Author Topic: DCamProf - a new camera profiling tool  (Read 767288 times)

AlterEgo

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1995
Re: DCamProf - a new camera profiling tool
« Reply #960 on: September 30, 2015, 12:51:15 pm »

3 stops down from saturation is midtones, that would be the lowest I go.
and rawdigger still can be used to flat field (we do not naturally clip anything in flat field raw) the target shot with some patches clipped in order to extract the data for that "blue" one so that we can replace (after normalization for the differences in exposure between 2 sets) the data in the shot where "blue" patch is "underexposed", right ? just want to check if no hidden issues w/ such merge of shots and flat fielding.
Logged

Iliah

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 770
Re: DCamProf - a new camera profiling tool
« Reply #961 on: September 30, 2015, 12:53:08 pm »

and rawdigger still can be used to flat field (we do not naturally clip anything in flat field raw) the target shot with some patches clipped in order to extract the data for that "blue" one so that we can replace (after normalization for the differences in exposure between 2 sets) the data in the shot where "blue" patch is "underexposed", right ? just want to check if no hidden issues w/ such merge of shots and flat fielding.

Yes, straightforward.
Logged

torger

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 3267
Re: DCamProf - a new camera profiling tool
« Reply #962 on: September 30, 2015, 01:18:37 pm »

I think the source of the blue problem

Oh well, I think I was full of sh*t... forget that about negative matrix values. But anyway, I'm on to it.
Logged

AreBee

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 638
Re: DCamProf - a new camera profiling tool
« Reply #963 on: September 30, 2015, 04:34:58 pm »

Iliah,

Quote from: torger
...my current assumption is that two reasonably modern cameras with reasonably large sensor in good light can be profiled to look almost exactly the same...

Quote from: Iliah
Two reasonably low noise cameras, yes; if they are designed without CFA shortcuts...But the richness of gradation is a different matter.

Which component(s) of camera hardware contributes to the richness of gradation?
Logged

hk1020

  • Newbie
  • *
  • Offline Offline
  • Posts: 19
Re: DCamProf - a new camera profiling tool
« Reply #964 on: September 30, 2015, 06:55:52 pm »

As you're making with SSF, try using a much larger target, like this for example:

dcamprof make-target -c nex5.json -p cc24 -p munsell -p adobergb-grid target.ti3

I don't think there is a bug going on, but a CC24 profile will have limited precision in high saturation colors, and that coupled with unrelaxed LUT can pull high saturation colors into poor directions. The pink plastic bags seems to suffer from that, get too dark and too saturated. With a larger target and/or relax this is fixed.

The difference in dark blues I think can be higher accuracy in DCamProf. I'd suggest to make a fixed reference comparison.

It is a problem today that if you make a LUT of a CC24 without relax it the spline effect may continue far outside (towards higher saturation colors) and may mess up a bit. I may look into that, but all profiles should have some relax anyway.

Though this discussion has progressed a lot by now I'd like to add a little more to my problem. Sorry I didn't properly refer to my posts in the dummy thread in the first place. Actually, I should have stayed there as this discussion is now way over my head. Anyway, I also made a profile from a target, namely the one on imaging-resource..com you cited in the dummies thread. There is almost no difference to using the ssf. It is marginally better, though, but I wouldn't bother.

Trying the munsell thing however, did have a visible effect. Its blues are better but still not right. I have some pictures with several very sensitive blues which in reality are very similar but always come up as quite different blues for the individual hues. The separation is way too large, the blues are almost always too dark and some might turn almost purple. Unfortunately, I don't like to post the pictures publicly but I would give Anders a link through PM if that would be acceptable.
Logged

Iliah

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 770
Re: DCamProf - a new camera profiling tool
« Reply #965 on: September 30, 2015, 07:18:01 pm »

> Which component(s) of camera hardware contributes to the richness of gradation?

Proper separation in CFA (colour filter array, the thing that forms Bayer mosaic), CDS (correlated double sampling, https://en.wikipedia.org/wiki/Correlated_double_sampling), overall noise (read, thermal, reset, etc). Interesting that proper CFA and low noise are somewhat cross-purpose. Some design issues, too: if during a 30-sec exposure with the lens cap on you can see a "cloud" in raw - it is improper heat dissipation, it hurts. Generally, all uniformity issues cause strange artifacts if one starts to employ 10+ stops of dynamic range.
Logged

GWGill

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 608
  • Author of ArgyllCMS & ArgyllPRO ColorMeter
    • ArgyllCMS
Re: DCamProf - a new camera profiling tool
« Reply #966 on: September 30, 2015, 09:07:30 pm »

Adobe doesn't seem to put negative values there, but instead let blue render much lighter than realistic, and in return get no negative colors in extreme ranges, ie a more robust profile.
It may not be deliberate on Adobe's part, but rendering highly saturated colors as lighter, in-gamut colors is desirable if you model the Helmholtz-Kohlrausch effect.
Logged

torger

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 3267
Re: DCamProf - a new camera profiling tool
« Reply #967 on: October 01, 2015, 01:37:19 am »

It may not be deliberate on Adobe's part, but rendering highly saturated colors as lighter, in-gamut colors is desirable if you model the Helmholtz-Kohlrausch effect.

Some of their newer profiles have only positive values in their matrices and produce a desaturated light look, not very realistic or accurate, but it has some advantages. With DNG you first make a matrix conversion to get into linear prophoto, clip if necessary, and then make a lookup in the LUT.

With a "normal" matrix optimized to match normal saturation colors you will get negative value and clipping etc in the extreme range, and then the matrix value is first clipped and the put into the LUT.

So I think their matrices in this case is a purely mathematical thing. Their blues also in the normal range are rendered much too light, but it's a part of the "Adobe look".

It may turn out that I will eventually be forced to make these type of "dumb" matrices that has nothing to do with color, but just as an intermediate step to get into the LUT. I'm not yet sure it's required though. I would very much prefer to have a "layered" DNG profile as before, ie run just the matrix and get as good as possible linear colorimetric color, further improve that with the HueSatMap LUT, and then run the LookTable and tone curve to get the tone reproduction operator. In RawTherapee you can toggle all these individual elements which make the profile more versatile.
Logged

torger

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 3267
Re: DCamProf - a new camera profiling tool
« Reply #968 on: October 01, 2015, 02:01:53 am »

Though this discussion has progressed a lot by now I'd like to add a little more to my problem. Sorry I didn't properly refer to my posts in the dummy thread in the first place. Actually, I should have stayed there as this discussion is now way over my head. Anyway, I also made a profile from a target, namely the one on imaging-resource..com you cited in the dummies thread. There is almost no difference to using the ssf. It is marginally better, though, but I wouldn't bother.

Trying the munsell thing however, did have a visible effect. Its blues are better but still not right. I have some pictures with several very sensitive blues which in reality are very similar but always come up as quite different blues for the individual hues. The separation is way too large, the blues are almost always too dark and some might turn almost purple. Unfortunately, I don't like to post the pictures publicly but I would give Anders a link through PM if that would be acceptable.

A PM would be nice. I'm a bit choked now so progress is probably going to be slow. Have you tried X-Rite's software or Adobe's DNG profile editor? If you can it would be nice to hear what result you get from that. It would be very useful to get some indication if it's a camera problem (ie you get blue issues from the other softwares too), or it's a problem with DCamProf's blues in some way. Please include those profiles in the PM too if you can. If it's a camera problem it's still an interesting case as I then can test the feasibility to manually adjust the profile, or add some feature that makes it easier to do so.

Please note that you can't really use the bundled profiles as reference, they are generally majorly wrong. So the image you send should have colors you know what they really are, preferably of objects you have near you and can look at again as I'm probably going to ask about their colors.

(I have seen on the A7r-ii and artificial blues that there might be some purple tainting there too, but it could be unrelated (I'm working on a DNG LUT clipping fix there).)

Did you test the file I posted based on Adobe's matrix? Did it work better? If not here's the message where it's attached:
http://forum.luminous-landscape.com/index.php?topic=100015.msg856054#msg856054
Logged

hk1020

  • Newbie
  • *
  • Offline Offline
  • Posts: 19
Re: DCamProf - a new camera profiling tool
« Reply #969 on: October 01, 2015, 04:06:52 am »

A PM would be nice. I'm a bit choked now so progress is probably going to be slow.

Don't worry, you are much faster than what I could come up with. I'll prepare something, maybe tonight. If not then maybe Saturday.
Logged

Torbjörn Tapani

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 319
Re: DCamProf - a new camera profiling tool
« Reply #970 on: October 01, 2015, 04:14:34 am »

I haven't tried dcamprof yet but would it be possible to keep a bunch of reference images from common cameras and whenever a new release is out run a script that generates a new set of profiles. Then people could try the new version quickly. You would only need linear and "torger look" versions.
Logged

torger

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 3267
Re: DCamProf - a new camera profiling tool
« Reply #971 on: October 01, 2015, 04:34:07 am »

I haven't tried dcamprof yet but would it be possible to keep a bunch of reference images from common cameras and whenever a new release is out run a script that generates a new set of profiles. Then people could try the new version quickly. You would only need linear and "torger look" versions.

Good idea. I'll probably do something like that.

It will be quite boring though, because in most circumstances the new version will look exactly like the old :) the gamut compression stuff has changed a bit though, so it's still a good idea.

What seems to be a quite big chunk of work left is extreme colors and clipping handling. I've already spent a lot of time on that, but it's difficult to get right, there's still issues. There will probably be a few incremental releases before all those things are sorted out. Clipping issues are only visible in certain type of images, night scapes with lots of colored artificial lights is something DCamProf currently doesn't handle that well.
Logged

torger

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 3267
Re: DCamProf - a new camera profiling tool
« Reply #972 on: October 01, 2015, 03:44:22 pm »

This blue clipping issue is driving me nuts.

I've attached a pixel peep screen shot of a corner detail of a night scape shot of an artificial narrow band light, made with an A7r-II. It's one of the shots I've got in a PM, I took the freedom of posting this size crop without asking, hope it's okay. I normally always ask, but this size from a high MP shot I thought was ok.

The first is without profile (eg identity matrix after WB) that shows the actual tonality in the file. Then to the right is adobe's (before looktable+curve) which is quite close to the identity matrix, that is tonality is mostly kept (they mess up a bit when adding looktable+curve but not too bad).

And in the middle is DCamProf (before looktable+curve). Here we see it's much bluer, so blue in fact that much of the tonality is lost, and a sharp transition into cyan. Why is this? When the matrix is optimized to match the CC24 a very good match can be had, but that requires a great deal of blue subtraction, otherwise the dark blue patch becomes much to light, and other colors as well. In theory the LUT could compensate that, but that leads to strong bends, and when relaxing you end up with quite large errors.

This can be seen with Adobe's profile, which doesn't match CC24 colors that well, and thus is not a very accurate profile. But in return very robust with extreme colors.

It seems like some subtraction is okay, say if the Yb value is -0.10 or larger you can still have good tonality (which it is on my Canon for example), but for the A7r-ii and some other Sony sensors it's like -0.28, and then blue is pushed down to the ground in the extreme range. I don't know why these sensors have this type of blue sensitivity, someone that dare to speculate?

By limiting the matrix optimizer to not have (large) negative blue value I can get about the same result as Adobe, but then accuracy in the normal range is lost (can't compensate with LUT without too strong bending) and I really don't like the idea to mess up the normal case to be robust in an extreme case.

I'm thinking of the idea to blend between two profiles/matrices, one for the normal range and gradually blending into another for the extreme range. It's a quite big amount of work to test if that even works though, so I'm probably going to add an "known limitations" section and put this issue there.

However, I still have some messy tone operator clipping issues in this range too, that needs separate fixing. So it's certainly going to be better than the current 0.9.13 when finished, but I think Adobe is going to win on "extreme blues" even after completion.
Logged

AlterEgo

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1995
Re: DCamProf - a new camera profiling tool
« Reply #973 on: October 01, 2015, 04:01:18 pm »

So it's certainly going to be better than the current 0.9.13 when finished, but I think Adobe is going to win on "extreme blues" even after completion.


a simple idea will be to provide some command line option (along with a writeup in manual with couple of illustrations) to build profiles just for those extreme cases... I think it is not reasonable to expect one profile to cover everything... if somebody is after extreme colors - let them build profiles that sacrifice regular colors (doubtfully they will be there anyways or play any center role if so much attention is towards the extremes in the frame).
Logged

torger

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 3267
Re: DCamProf - a new camera profiling tool
« Reply #974 on: October 01, 2015, 04:06:50 pm »


a simple idea will be to provide some command line option (along with a writeup in manual with couple of illustrations) to build profiles just for those extreme cases... I think it is not reasonable to expect one profile to cover everything... if somebody is after extreme colors - let them build profiles that sacrifice regular colors (doubtfully they will be there anyways or play any center role if so much attention is towards the extremes in the frame).

Yes I will probably provide the command line option to limit the "blue compensation". I've tested and it seems to work although weighting/relaxing becomes messier. So yes you could then make a "nightscape/narrow band light" specific robust profile which is more about keeping tonality in the extreme ranges than being accurate in normal range.
Logged

torger

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 3267
Re: DCamProf - a new camera profiling tool
« Reply #975 on: October 02, 2015, 05:55:02 pm »

Before LUT in a DNG profile you do a matrix conversion to linear prophoto space using the forward matrix. This conversion is then clipped to 0-1 range before going into the LUT.

A matrix optimized to match normal colors well will cause say 10% of the total space to be lost in clipping. These 10% is what appears when you shoot strange narrow band lights. In order to make a DNG profile that does not render these clipped flat you cannot make a normal matrix, instead of optimizing it to match colors you need to optimize it to not clip, which is what Adobe is doing in many of their newer profiles. The LUT then (hopefully) moves colors into sane positions in the normal range, and stays fairly passive in the extreme range, so the native camera tonality is kept rather than clipped away.

(A pure matrix profile of course must be optimized to match normal colors, and that simply won't work for the extreme range. Tonality is irreversibly lost.)

DCamProf's native format also uses a matrix for base conversion, on which it adds a LUT, just like a DNG profile. This is problematic for the extreme range. Although DCamProf doesn't clip like in the DNG pipeline, still when the matrix enters negative lightness it cannot really compensate without an extreme bend. This makes the format sort of flawed for making profiles that are robust in the extreme range. However, I don't think a major rewrite is necessary just yet, I'll play around with extended optimization techniques.

Controlling the blue Y value like discussed previously is the major aspect of this, if I'm not mistaken it's all DCamProf's native format needs (since then you can make sure lightness never gets negative, negative X and Z are okay since DCamProf makes addition on those axes), but it's not enough for a DNG profile as the matrix can still clip in the Prophoto conversion.

While working on this and other issues I've also discovered clipping issues in the CIECAM02 blue range which causes some headache as I'm using that here and there. LCMS2 CIECAM02 implementation clips uglier than another CIECAM02 implementation I've looked into, so maybe I'll switch, at least for some critical calculations.

Handling the "extreme range" is the largest can of worms I've opened lately. It's not just one place in the software that is affected, it's like 5-10 places and none of them has an easy fix.
« Last Edit: October 02, 2015, 05:58:06 pm by torger »
Logged

torger

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 3267
Re: DCamProf - a new camera profiling tool
« Reply #976 on: October 03, 2015, 06:22:07 pm »

Experiments are ongoing, very much "back to the drawing board" kind of thing. Looking a bit at alternate matrix optimizations.

For DNG profiles, one obvious method is to simply set forward matrix equal to ProPhotoRGB matrix, which means that what the LUT sees is the "raw" samples (just white balanced), and then gets to stretch that into place. This is quite robust for the extreme range as you don't get any clipping of data before entering the LUT, but to not clip when inside the LUT it must make sure that it reduces scaling towards 1.0 for the gamut border. You will also get some twists in the LUT which may be problematic.

Adobe has changed their profile design technique over the years, and probably vary between models. Looking at Adobe's own A7r-II and Pentax 645z profiles both seem to be designed with the same method. The forward matrix in their case seems to be a remapped AdobeRGB matrix, which fits inside ProPhotoRGB border, and also the spectral locus border. My guess is that the size of the gamut triangle is pre-defined (to something similar to AdobeRGB), then their matrix optimizer rotates it to match hues as well has possible, ignoring saturation. The matrix result is thus a heavily desaturated look, but with good hue match. Then their LUT simply stretches saturation straight outwards from the whitepoint. The gain from doing this instead of just picking the ProPhotoRGB matrix as is, I guess is that you don't get much hue rotation in the LUT, which is a bit messy with DNG LUTs (hue discontinuity problem discussed some time back).

DCamProf is not a DNG-only software so I'm thinking about how to tackle this. For DNG profiles there are indeed many advantages of this type of optimization technique. As DCamProf currently has clipping issues in its pipeline as well, it's actually an advantage also for DCamProf. However, it's painful to ditch the layered elegance - FM matrix gives you as accurate color as possible with a matrix and the LUT just refines that. Since matrix optimization is fast I'm thinking of having two matrices in the native format, one "real" matrix stored for using when a matrix-only profile is generated, and one matrix optimized to be used together with a LUT.

It's quite likely that the next release will introduce some significant behavior change...
« Last Edit: October 03, 2015, 06:24:33 pm by torger »
Logged

Alexey.Danilchenko

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 257
    • Spectron
Re: DCamProf - a new camera profiling tool
« Reply #977 on: October 04, 2015, 04:11:33 am »

Before LUT in a DNG profile you do a matrix conversion to linear prophoto space using the forward matrix. This conversion is then clipped to 0-1 range before going into the LUT.

A matrix optimized to match normal colors well will cause say 10% of the total space to be lost in clipping. These 10% is what appears when you shoot strange narrow band lights. In order to make a DNG profile that does not render these clipped flat you cannot make a normal matrix, instead of optimizing it to match colors you need to optimize it to not clip, which is what Adobe is doing in many of their newer profiles.

All Kodak SLR cameras and Pro Backs used the same approach - the ERIMM (input referenced ProPhoto) as interconnecting space. The code also did some clipping and standard matrix (for ideal camera from Photodesk profiles) was mapping it to 0..1 range perfectly. However each camera came with its own calibrated set of matrices for that specific sensor which were almost always not fitting the 0..1 range in ERIMM space (those were pacjed into every raw file). What Kodak software did if those were used was to also bundle exposure correction factor - this was really used to bring the output within the range of 0..1. Just  thought that might be useful.
Logged

Iliah

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 770
Re: DCamProf - a new camera profiling tool
« Reply #978 on: October 04, 2015, 09:12:40 am »

> The forward matrix in their case seems to be a remapped AdobeRGB matrix

As some here know ;) I map to BetaRGB http://brucelindbloom.com/BetaRGB.html
Logged

GWGill

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 608
  • Author of ArgyllCMS & ArgyllPRO ColorMeter
    • ArgyllCMS
Re: DCamProf - a new camera profiling tool
« Reply #979 on: October 04, 2015, 09:53:42 pm »

While working on this and other issues I've also discovered clipping issues in the CIECAM02 blue range which causes some headache as I'm using that here and there.
CIECAM02 has significant practical issues with colors near to or beyond the spectrum locus, particularly near blue. I think that the basic nature of CIECAM02 makes any elegant solution impossible. (I get passable results with a number of hacks to it). A "nice" solution would involve re-formulating the equations used to accommodate extrapolation, and then re-fitting them to the original data CIECAM02 data sets.

Timo Kunkel and Erik Reinhard's simplified and improved model inspired by CIECAM02 might be a place to start.
Logged
Pages: 1 ... 47 48 [49] 50 51 ... 78   Go Up