Luminous Landscape Forum

Equipment & Techniques => Beginner's Questions => Topic started by: kpz on November 27, 2018, 01:06:52 am

Title: Why do raw files require tone mapping to look perceptually correct?
Post by: kpz on November 27, 2018, 01:06:52 am
Why do raw files opened in Lightroom seem to almost always require massaging with the highlight and shadow sliders to look perceptually correct (that is, to reproduce what my eyes see in reality)?

I notice that virtually everyone I have seen edit photos in LR (e.g. on YouTube) lowers the highlight slider and increases the shadow slider to "recover detail" in order to mitigate this effect. Why is this necessary to create a realistic image? The camera sensor sees the same photons I see, and while there's a gamma curve correction to account for the fact that the human eye doesn't perceive light in a linear fashion, that still doesn't explain the need for these strange local tone mapping adjustments.

Alternatively, am I wrong, and can one get a nearly "realistic" image without touching these sliders at all? Obviously, if overdone it creates the "bad HDR" effect. I am trying to understand why their use is so common.

edit: Actually, I suppose the same question applies also to global contrast adjustments via the tone curve, since this is just a global tone mapping. And I guess "behind the scenes" what LR does with the shadow/highlight sliders can be roughly understood as some kind of tone curve manipulation.

So, more broadly, why do raw files require significant tone curving to look "right"? In Camera Neutral, for instance, they seem flat and lacking contrast. But again -- I should be seeing raw sensor data with basically just a gamma correction, and the sensor and I are seeing the same photons, so I expect roughly the same result. What explains the discrepancy? Surely some lack of microcontrast can be attributed to Bayer interpolation and the AA filter (if present), but that's not really what I'm talking about.
Title: Re: Why do raw files require tone mapping to look perceptually correct?
Post by: sandymc on November 27, 2018, 04:34:26 am
"General-Purpose Gamut-Mapping Algorithms: Evaluation of Contrast-Preserving Rescaling Functions for Color Gamut Mapping", Gustav J. Braun and Mark D. Fairchild
Title: Re: Why do raw files require tone mapping to look perceptually correct?
Post by: digitaldog on November 27, 2018, 12:25:09 pm
There is no such thing as 'perceptually correct'. There is scene referred and output referred and the later is subjective and differs based on a massive number of factors:
http://www.color.org/ICC_white_paper_20_Digital_photography_color_management_basics.pdf
Take transparency film as an example unlike color neg (even farther away from output referred): which is 'perpetually accurate', Agfachrome, Kodachrome, Velvia, Ektachrome? The manufacturers produce a rendering they believe you will visually prefer and buy. Just like default raw processing conversions which may or may not be to your liking or better for a starting point in editing the image. If you don't like the defaults, change them. Simple to do in many converters including of course, Adobe's.
Title: Re: Why do raw files require tone mapping to look perceptually correct?
Post by: Bart_van_der_Wolf on November 27, 2018, 01:04:10 pm
Why do raw files opened in Lightroom seem to almost always require massaging with the highlight and shadow sliders to look perceptually correct (that is, to reproduce what my eyes see in reality)?

I notice that virtually everyone I have seen edit photos in LR (e.g. on YouTube) lowers the highlight slider and increases the shadow slider to "recover detail" in order to mitigate this effect. Why is this necessary to create a realistic image?

Hi,

Because Lightroom (over)compresses the highlights (attempting to recover clipped highlights, even when they are unclipped).

Cheers,
Bart
Title: Re: Why do raw files require tone mapping to look perceptually correct?
Post by: kpz on November 27, 2018, 01:50:16 pm
Thanks for the answers. I wonder, can anyone recommend a readable but complete technical reference for color and raw conversion? I realize this a large question and a great deal of ink has been spilled, but surely there must be a more efficient way to learn this material than just picking up information online piecemeal.

Bart, if Lightroom over-compresses highlights by default, wouldn't lowering the highlight slider compress them even *more*? (Here I understand "highlight compression" to mean tone mapping the highlights to a lower, wider range of values, but perhaps I am getting the definition wrong.)
Title: Re: Why do raw files require tone mapping to look perceptually correct?
Post by: Bart_van_der_Wolf on November 27, 2018, 02:36:07 pm
Bart, if Lightroom over-compresses highlights by default, wouldn't lowering the highlight slider compress them even *more*? (Here I understand "highlight compression" to mean tone mapping the highlights to a lower, wider range of values, but perhaps I am getting the definition wrong.)

LR started compressing most tonality in highlights with then called Process Version 2012 (if I recall correctly). That indeed requires to apply a negative Highlights control setting to restore some of the original highlight tonality, e.g. white clouds to regain some of the original structure/definition. I don't know if/how the latest process version mangles tonality, since I've moved on to Capture One when Adobe introduced a subscription software model. I'm glad I did.

Cheers,
Bart
Title: Re: Why do raw files require tone mapping to look perceptually correct?
Post by: digitaldog on November 27, 2018, 02:40:32 pm
LR started compressing most tonality in highlights with then called Process Version 2012 (if I recall correctly).
Long before that but rather poorly. PV2012 was the version they got it 'right' (not ugly coloration upon recovery). And it doesn't do this all the time per se; if one or two channels are clipped, it can rebuild them from the one unclipped channel. But really, don't clip any channels you want highlight detail within and then there's no reason to go there.
Title: Re: Why do raw files require tone mapping to look perceptually correct?
Post by: Bart_van_der_Wolf on November 27, 2018, 02:48:42 pm
Long before that but rather poorly. PV2012 was the version they got it 'right' (not ugly coloration upon recovery). And it doesn't do this all the time per se; if one or two channels are clipped, it can rebuild them from the one unclipped channel. But really, don't clip any channels you want highlight detail within and then there's no reason to go there.

Hi Andrew,

I'm not talking about highlight 'recovery', which PV2012 indeed did a good job of even if one or two channels were clipped. I'm instead referring to the loss of tonality in unclipped bright to highlight regions. They look lifeless until the Highlight control gets a large negative setting.

Cheers,
Bart
Title: Re: Why do raw files require tone mapping to look perceptually correct?
Post by: kpz on November 27, 2018, 02:49:52 pm
LR started compressing most tonality in highlights with then called Process Version 2012 (if I recall correctly). That indeed requires to apply a negative Highlights control setting to restore some of the original highlight tonality, e.g. white clouds to regain some of the original structure/definition. I don't know if/how the latest process version mangles tonality, since I've moved on to Capture One when Adobe introduced a subscription software model. I'm glad I did.

So, by "highlight compression" you mean mapping the highlights to a narrower range than indicated by the raw file, and perhaps lowering the white point to compensate? And then the negative Highlights setting undos this "narrowing"?
Title: Re: Why do raw files require tone mapping to look perceptually correct?
Post by: digitaldog on November 27, 2018, 02:52:29 pm
Hi Andrew,

I'm not talking about highlight 'recovery', which PV2012 indeed did a good job of even if one or two channels were clipped. I'm instead referring to the loss of tonality in unclipped bright to highlight regions. They look lifeless until the Highlight control gets a large negative setting.

Cheers,
Bart
Then I honestly don't know what you're referring to then. Perhaps with some default settings and profiles, with some cameras, highlights are getting 'compressed' but it certainly isn't anything forced upon the user.
Title: Re: Why do raw files require tone mapping to look perceptually correct?
Post by: digitaldog on November 27, 2018, 03:50:26 pm
Then I honestly don't know what you're referring to then. Perhaps with some default settings and profiles, with some cameras, highlights are getting 'compressed' but it certainly isn't anything forced upon the user.
From what I see, PV 1 is buggy. PV2 onward isn't. PV1 shows that the white tile (BableColor target) is Lstar 100. It's not. And RawDigger confirms that this raw isn't over exposed what so ever (fig 1.).
Fig 2 shows PV1 where the tile (with default settings) shows Lstar 100. Yet PV2 onward with no additional settings in LR show it as Lstar 98 which is pretty much exactly what it should be according to actual measurements of the target.
So I don't see PV 2 onward compressing anything and in fact it appears to be reporting what it should while PV1 indicates the raw is over exposed but RawDigger claims otherwise.
Title: Re: Why do raw files require tone mapping to look perceptually correct?
Post by: kpz on November 27, 2018, 05:25:23 pm
Take transparency film as an example unlike color neg (even farther away from output referred): which is 'perpetually accurate', Agfachrome, Kodachrome, Velvia, Ektachrome? The manufacturers produce a rendering they believe you will visually prefer and buy. Just like default raw processing conversions which may or may not be to your liking or better for a starting point in editing the image. If you don't like the defaults, change them. Simple to do in many converters including of course, Adobe's.

Is your claim then that Lightroom is by default tone mapping the raw data to produce an image they think is more pleasing to me by messing with the highlights and shadows? I realize there are some adjustments baked in, which can be removed by going to an earlier process version and zeroing the sliders, but those are for things like the black point and global contrast (if I recall correctly), not the kind of local shadows/highlights manipulations I was thinking of.

What then explains why the "scene referred" image in that link, and other raw files, are noticeably different from what people remember seeing? Is it somehow related to the fact that the human eye can see vastly more colors than a typical sRGB monitor is capable of producing, and so some tone mapping is required to fit everything in (a sort of "HDR for color")?
Title: Re: Why do raw files require tone mapping to look perceptually correct?
Post by: digitaldog on November 27, 2018, 05:28:42 pm
Is your claim then that Lightroom is by default tone-mapping the raw data to produce an image they think is more pleasing to me by messing with the highlights and shadows?
That and more. Again depending on a massive number of factors.
The URL shows two output referred renderings that differ and a scene referred one that's far from pleasing. As one should expect from a scene referred rendering (it's not output referred as explained).
Number of colors; want to really go there?  ;)


http://digitaldog.net/files/ColorNumbersColorGamut.pdf
Title: Re: Why do raw files require tone mapping to look perceptually correct?
Post by: kpz on November 27, 2018, 05:39:12 pm
Okay, thank you. I think I am slowly beginning to understand.

I am still wondering, what exactly is going on perceptually that makes the scene referred renderings so "flat"? Obviously it is the case that they are much less pleasing than what we perceive with our eyes. But *why*?

Does the need for this tone mapping come (at least partially) from the need to deal with out of gamut colors? In other words, is it that monitors just can't display all the colors necessary for something perceptually correct, so some sort of HDR-type procedure is necessary as a substitute?

Also, if the scene referred renderings are colorimetrically accurate, and the "artistically correct" output referred ones usually aren't, then what is the purpose of a colorchecker? If accurate color is displeasing, why attempt to get perfectly accurate color anyway? And -- if LR messes with highlights and shadows, how come your test image gave the right LAB value? Wouldn't we expect the LR defaults to change it slightly?
Title: Re: Why do raw files require tone mapping to look perceptually correct?
Post by: digitaldog on November 27, 2018, 06:41:29 pm
Okay, thank you. I think I am slowly beginning to understand.

I am still wondering, what exactly is going on perceptually that makes the scene referred renderings so "flat"? Obviously it is the case that they are much less pleasing than what we perceive with our eyes. But *why*?

Does the need for this tone mapping come (at least partially) from the need to deal with out of gamut colors? In other words, is it that monitors just can't display all the colors necessary for something perceptually correct, so some sort of HDR-type procedure is necessary as a substitute?

Also, if the scene referred renderings are colorimetrically accurate, and the "artistically correct" output referred ones usually aren't, then what is the purpose of a colorchecker? If accurate color is displeasing, why attempt to get perfectly accurate color anyway? And -- if LR messes with highlights and shadows, how come your test image gave the right LAB value? Wouldn't we expect the LR defaults to change it slightly?
Answers:
1. Flat? Just not output referred (rendered for some output). Explained in the article.
2. Nothing per se about out of gamut colors. There are 'colors' a camera can capture we cannot see and thus not colors. There are colors we can see it cannot capture.
3. The ColorChecker is nothing more than a color reference, designed way back in 1976, long before any of us were using digital cameras: http://www.babelcolor.com/colorchecker.htm
4. I have no idea what some are speaking of here about LR 'messing' with highlights and shadows. It is producing output referred color based on all kinds of sliders and other 'edits' affecting the raw to rendered previews under the hood. My test image gave the correct Lstar values but that only tells us about the Lstar values of that highlight. And further, by moving a few sliders, I could alter those Lstar values a lot.
Title: Re: Why do raw files require tone mapping to look perceptually correct?
Post by: Bart_van_der_Wolf on November 27, 2018, 06:43:43 pm
Then I honestly don't know what you're referring to then. Perhaps with some default settings and profiles, with some cameras, highlights are getting 'compressed' but it certainly isn't anything forced upon the user.


I'm specifically referring to:
https://luminous-landscape.com/tonal-adjustments-in-the-age-of-lightroom-4/ (https://luminous-landscape.com/tonal-adjustments-in-the-age-of-lightroom-4/), and especially the part about Highlight Detail.

Quote
Highlight Detail

The Highlights slider, moved to the left, increases the highlight recovery as seen above.  It works especially well when just one or two of the channels are clipped, as with saturated colors.  But the Highlights slider also increases separation and detail in washed-out highlights.

Followed by two examples of a cloud, not clipped (!):
(https://luminous-landscape.com/wp-content/uploads/2012/04/clouds_compressed.jpg)  (https://luminous-landscape.com/wp-content/uploads/2012/04/clouds_LR4_fix.jpg)
The image on the left with Highlights at 0, and the one on the right at -100.

Cheers,
Bart
Title: Re: Why do raw files require tone mapping to look perceptually correct?
Post by: digitaldog on November 27, 2018, 06:46:24 pm

I'm specifically referring to:
https://luminous-landscape.com/tonal-adjustments-in-the-age-of-lightroom-4/ (https://luminous-landscape.com/tonal-adjustments-in-the-age-of-lightroom-4/), and especially the part about Highlight Detail.

Followed by two examples of a cloud, not clipped (!):
(https://luminous-landscape.com/wp-content/uploads/2012/04/clouds_compressed.jpg)  (https://luminous-landscape.com/wp-content/uploads/2012/04/clouds_LR4_fix.jpg)
The image on the left with Highlights at 0, and the one on the right at -100.

Cheers,
Bart
I still don't see your point other than sliders for affecting highlights can affect highlights. I only see that as a plus in how we might wish to render our highlights (and images).
Pull the sider to plus 100, I suspect you'll blow out the highlights. Highlights which of course are not blown out in the actual raw.
The same is true for other sliders or curves that allow us to affect anything on the tone curve.
What am I missing?  ;)
Title: Re: Why do raw files require tone mapping to look perceptually correct?
Post by: kpz on November 27, 2018, 06:51:29 pm
1. Flat? Just not output referred (rendered for some output). Explained in the article.

Could you suggest a reference for the details of what must happen to take a raw file and output refer it to make it visually pleasing? I think, after reading your replies, what this is what I really want to know (the approximate steps one takes to go from "scene referred" to output and why). I'm interested in both the technical side and the science of human perception. Any suggestions in this direction would be greatly appreciated...
Title: Re: Why do raw files require tone mapping to look perceptually correct?
Post by: digitaldog on November 27, 2018, 06:55:01 pm
Could you suggest a reference for the details of what must happen to take a raw file and output refer it to make it visually pleasing?
No. It's subjective. What I can recommend with the ACR/LR products is to consider working top down, left to right (for ACR) as the controls have been provided in an 'order' the designers believe should be used first. But you can break the rules. All edits are applied when rendering the raw in the best order, not the user order.
Title: Re: Why do raw files require tone mapping to look perceptually correct?
Post by: Bart_van_der_Wolf on November 27, 2018, 06:59:43 pm
I still don't see your point other than sliders for affecting highlights can affect highlights. I only see that as a plus in how we might wish to render our highlights (and images).
Pull the sider to plus 100, I suspect you'll blow out the highlights. Highlights which of course are not blown out in the actual raw.
The same is true for other sliders or curves that allow us to affect anything on the tone curve.
What am I missing?  ;)

Read the rest of the above-linked article, it has more examples of how lifeless the highlights are rendered, UNLESS one uses a -100 correction.

(https://luminous-landscape.com/wp-content/uploads/2012/04/Cramer-009395-5_2.jpg)   (https://luminous-landscape.com/wp-content/uploads/2012/04/Cramer-009395-7_2.jpg)
Increasing 'Exposure' even more heavily compresses the highlight tonality to avoid clipping or washing out, which can only be corrected by a negative Highlights correction (-100 in the example).

EDIT: Also read the section about "Brightening Highlights". Even the unadjusted histogram of the stepwedge shows tonal compression in the highlights, and it only gets worse with brightening the image.

(https://luminous-landscape.com/wp-content/uploads/2012/04/SW_default.jpg)

Cheers,
Bart

Title: Re: Why do raw files require tone mapping to look perceptually correct?
Post by: faberryman on November 27, 2018, 07:04:48 pm
If anything is inaccurate, it is probably my memory of the original scene rather than the digital capture itself.
Title: Re: Why do raw files require tone mapping to look perceptually correct?
Post by: digitaldog on November 27, 2018, 07:08:23 pm
Read the rest of the above-linked article, it has more examples of how lifeless the highlights are rendered, UNLESS one uses a -100 correction.

(https://luminous-landscape.com/wp-content/uploads/2012/04/Cramer-009395-5_2.jpg)   (https://luminous-landscape.com/wp-content/uploads/2012/04/Cramer-009395-7_2.jpg)
Increasing 'Exposure' even more heavily compresses the highlight tonality to avoid clipping or washing out, which can only be corrected by a negative Highlights correction (-100 in the example).

EDIT: Also read the section about "Brightening Highlights". Even the unadjusted histogram of the stepwedge shows tonal compression in the highlights.

Cheers,
Bart
That's just silly Bart. The two examples are utterly subjective and I suspect you can ask 100 people which the prefer and you'll get different answers from the group. Either is acceptable visually to me. But I'm NOT the image creator and I defer to the one rendering the image as they prefer. That's why we have the sliders in the first place; to render the image as we prefer.
You want to state that Zero isn't an ideal setting for your images? Fine. Don't use Zero. It's why we have the ability to produce our own defaults based on how we capture the image and wish to render it.
Meanwhile, unlike my example, we have no idea how the image was captured in terms of the raw data, a raw Histogram or info from RawDigger as to what really does reside somewhere within that range of highlights.
Again, I have no idea what point you're attempting to make other than LR/ACR sliders that affect highlights can affect highlights. To that I say "duh".  ;)
Title: Re: Why do raw files require tone mapping to look perceptually correct?
Post by: kpz on November 27, 2018, 07:09:07 pm
No. It's subjective. What I can recommend with the ACR/LR products is to consider working top down, left to right (for ACR) as the controls have been provided in an 'order' the designers believe should be used first. But you can break the rules. All edits are applied when rendering the raw in the best order, not the user order.

It doesn't seem entirely subjective to me. For example, it seems that most JPEG conversions do in general the same sorts of things, like adding contrast, and people usually find this more pleasing than the unmolested raw file. (That is, unless some special effect is desired. But for the "normal" photo look we are accustomed to, this seems roughly true.) This suggests some underlying technical or perceptual principles are at work.
Title: Re: Why do raw files require tone mapping to look perceptually correct?
Post by: digitaldog on November 27, 2018, 07:09:17 pm
If anything is inaccurate, it is probably my memory of the original scene rather than the digital capture itself.
Indeed! And the possible and massive difference in the dynamic range of what we can see and what we can capture as so well illustrated in this must read:
http://www.lumita.com/site_media/work/whitepapers/files/pscs3_rendering_image.pdf (http://www.lumita.com/site_media/work/whitepapers/files/pscs3_rendering_image.pdf)
Title: Re: Why do raw files require tone mapping to look perceptually correct?
Post by: digitaldog on November 27, 2018, 07:09:49 pm
It doesn't seem entirely subjective to me. For example, it seems that most JPEG conversions do in general the same sorts of things, like adding contrast, and people usually find this more pleasing than the unmolested raw file. (That is, unless some special effect is desired. But for the "normal" photo look we are accustomed to, this seems roughly true.) This suggests some underlying technical or perceptual principles are at work.
IF you think the JPEG is an ideal rendering, shoot JPEGs.  ;)
Title: Re: Why do raw files require tone mapping to look perceptually correct?
Post by: kpz on November 27, 2018, 07:16:02 pm
IF you think the JPEG is an ideal rendering, shoot JPEGs.  ;)

I certainly don't think the JPEG is an ideal rendering! It just seems that it is not entirely a matter of chance that raw converter defaults and in-camera renderers all do roughly similar things (again excepting special effects). I wouldn't call it "ideal," but I would say there is (broadly) some common, widely-preferred rendering style people expect when they see a "photo," and that it differs from straight scene referred input. I'd like to understand what that is and why people like it. Maybe it is a hopeless task, however.

That link you just posted a goes a long way toward answering my question, by the way!
Title: Re: Why do raw files require tone mapping to look perceptually correct?
Post by: digitaldog on November 27, 2018, 07:20:37 pm
I certainly don't think the JPEG is an ideal rendering! It just seems that it is not entirely a matter of chance that raw converter defaults and in-camera renderers all do roughly similar things (again excepting special effects). I wouldn't call it "ideal," but I would say there is (broadly) some common, widely-preferred rendering style people expect when they see a "photo," and that it differs from straight scene referred input. I'd like to understand what that is and why people like it. Maybe it is a hopeless task, however.

That link you just posted a goes a long way toward answering my question, by the way!
What straight scene referred input?
Title: Re: Why do raw files require tone mapping to look perceptually correct?
Post by: kpz on November 27, 2018, 07:26:32 pm
What straight scene referred input?

For instance the picture marked "scene referred image" in the link you posted earlier:
http://www.color.org/ICC_white_paper_20_Digital_photography_color_management_basics.pdf
Title: Re: Why do raw files require tone mapping to look perceptually correct?
Post by: Bart_van_der_Wolf on November 27, 2018, 07:29:52 pm
Meanwhile, unlike my example, we have no idea how the image was captured in terms of the raw data, a raw Histogram or info from RawDigger as to what really does reside somewhere within that range of highlights.

Try shooting a stepwedge if your senses do not see the highlight compression that many have been complaining about. Has nothing to do with Raw, but with the compressed rendering of the highlight tonality. The PV2012 highlight tonality compression has been complained about since it's inception. The Highlight control's ability to restore some sense is not just nice to have, it's a must, for many.

But hey, if you like washed out looking lifeless highlights, good for you.

Cheers,
Bart
Title: Re: Why do raw files require tone mapping to look perceptually correct?
Post by: digitaldog on November 27, 2018, 07:30:50 pm
For instance the picture marked "scene referred image" in the link you posted earlier:
http://www.color.org/ICC_white_paper_20_Digital_photography_color_management_basics.pdf (http://www.color.org/ICC_white_paper_20_Digital_photography_color_management_basics.pdf)
OK, that image is indeed scene referred because my co-author of the piece, Jack Holms of HP supplied it.
But anything else you see, in LR/ACR or any other raw converter that isn't specifically configured, IF possible for scene referred output is indeed output referred. Including everything we've seen here from Bart. That's important to keep in mind in this discussion. Ditto when someone is sharing a Histogram; it's not unless specifically produced as such, a raw Histogram. It's a Histogram of the currently rendered (and most often output referred) image.
It's a bit like trying to talk about the specific recipe for key lime pie by eating it alone. That's kind of a waste of time.
Title: Re: Why do raw files require tone mapping to look perceptually correct?
Post by: digitaldog on November 27, 2018, 07:34:28 pm
Try shooting a stepwedge if your senses do not see the highlight compression that many have been complaining about. Has nothing to do with Raw, but with the compressed rendering of the highlight tonality. The PV2012 highlight tonality compression has been complained about since it's inception. The Highlight control's ability to restore some sense is not just nice to have, it's a must, for many.

But hey, if you like washed out looking lifelesss highlights, good for you.

Cheers,
Bart
Waste of time Bart. You seem to be suggesting some agenda against how one raw converter allows us to alter a rendering. It's moot. If you cannot produce a rendering from a well captured raw, then you have a soap box to stand on. I could care less about compression when I can use or avoid it in producing an image I desire. Either of the waterfall images you provided (and now clearly didn't capture) is acceptable to me and I suspect others. If the author isn't happy with either, he's got a soapbox to stand on. Otherwise this is a solution in search of a problem that doesn't exist. And is kind of OT as well.
I have zero issues not getting (whatever lifeless highlights mean) in images I optimally expose with the ACR engine. But I think I did show that the original PV was a bit on the buggy side by default.
Title: Re: Why do raw files require tone mapping to look perceptually correct?
Post by: digitaldog on November 27, 2018, 07:43:22 pm
The PV2012 highlight tonality compression has been complained about since it's inception.
Title: Re: Why do raw files require tone mapping to look perceptually correct?
Post by: Slobodan Blagojevic on November 27, 2018, 08:39:43 pm
... The two examples are utterly subjective and I suspect you can ask 100 people which the[y] prefer and you'll get different answers from the group. Either is acceptable visually to me...

Ok, "prefer" is indeed subjective. And "acceptable" isn't really a high praise.

But let's rephrase the question. Let's ask 100 people which of the two they are more likely to use the following words to describe: dull, lifeless, flat. I am the first one to attached them to the first waterfall image. The OP does as well. I bet the majority of the 100 would do so.

And that really is the question the OP is asking since post #1: why are raw files generally perceived as more dull and flat, compared to adjusted files or even the out-of-camera jpegs.

My uneducated guess would be that raw files do not know which way users would like to go in post, so they offer a middle-of-the-road starting point, neither here nor there. If I may use an analogy, it is like a supermodel's picture in the morning, right out of bed, compared with the one after hours of makeup and hair styling. Now, of course, various people might prefer the natural look, but if asked which look would be more likely described as Plain Jane and which of a supermodel, the answers would be more unified.
Title: Re: Why do raw files require tone mapping to look perceptually correct?
Post by: digitaldog on November 27, 2018, 08:48:26 pm
And that really is the question the OP is asking since post #1: why are raw files generally perceived as more dull and flat, compared to adjusted files or even the out-of-camera jpegs.
Because that's the default settings for one product, to produce an adjusted file that may match the out of camera JPEG. It's a starting point for the OP's product of use (Lightroom).
This isn't at all anything anyone has to accept. Anyone can easily change the default settings, as they can change the camera profile which alone will play huge role here.
Quote
My uneducated guess would be that raw files do not know which way users would like to go in post, so they offer a middle-of-the-road starting point, neither here nor there.
Raw files know nothing, they are 1's and zero's (like the JPEG). This is what raw looks like:
(http://www.digitaldog.net/files/raw.jpg)


The raw needs to be rendered of course. How that's done is the name of the game here.
Someone may want to take Bart's bait and ask "well if ACR/LR can't render the step which (whatever why he thinks it should), what should I use instead. But the OP is using Lightroom, the ACR engine. And there's absolutely nothing that stops him, like perhaps 10's of thousands if not more knowledge uses, to render highlights and other areas of the tone curve as they desire. Case in point:


(http://digitaldog.net/files/Zia.jpg)
Title: Re: Why do raw files require tone mapping to look perceptually correct?
Post by: Bart_van_der_Wolf on November 27, 2018, 09:17:04 pm
Someone may want to take Bart's bait and ask "well if ACR/LR can't render the step which (whatever why he thinks it should), what should I use instead.

Huh?

All I'm saying is that the OP is correct, and that many people have a different preference than the, in this example, LR's default rendering.

I'm not arguing that it CAN be adjusted, but that it indeed SHOULD be adjusted in many cases, and that the reason is that LR's PV2012 (as it was called) attempts to compress highlight tonality separation as it attempts to avoid or reduce clipping. The latter (reducing/recovering) it does very well, the former (avoiding) leads to generally unwanted (by many) dullness in highlights, thus leading to an almost mandatory reduction in the Highlight control. Thus attempting to answer the OP's question.

Cheers,
Bart
Title: Re: Why do raw files require tone mapping to look perceptually correct?
Post by: digitaldog on November 27, 2018, 09:22:01 pm
All I'm saying is that the OP is correct, and that many people have a different preference than the, in this example, LR's default rendering.
What default rendering from what camera, with what camera profile? And yet, we're told from your keyboard, to our eyes: The PV2012 highlight tonality compression has been complained about since it's inception. And But hey, if you like washed out looking lifeless highlights, good for you.
IF you have a different preference than the out of the box default settings with (what camera and profile, they DO differ), alter the default settings. Or yank on a slider. That's why they exist Bart.
But that was discussed in Post #3 before the OT, PV2012 tonality compression rabbit hole (https://forum.luminous-landscape.com/index.php?topic=127736.msg1080223#msg1080223).
Title: Re: Why do raw files require tone mapping to look perceptually correct?
Post by: kpz on November 27, 2018, 11:05:38 pm
All I'm saying is that the OP is correct, and that many people have a different preference than the, in this example, LR's default rendering.

Thanks Bart.

While it may be entirely a personal artistic preference, as the Digital Dog notes, have you or others here noticed any other important "quirks" of the LR default rendering? And I suppose (based on some other posts of yours I have read) the ultimate "solution" is to try out Capture One? I'm just trying to understand the strengths and weaknesses of my tools, and what other options may be available to me.
Title: Re: Why do raw files require tone mapping to look perceptually correct?
Post by: digitaldog on November 27, 2018, 11:17:53 pm
There is a saying that a tool is only as good as the person using it.
Title: Re: Why do raw files require tone mapping to look perceptually correct?
Post by: kpz on November 27, 2018, 11:23:35 pm
There is a saying that a tool is only as good as the person using it.

Indeed, and I'll be the first to admit I have a ways to go in this direction. I'm just trying to understand how the thing works, so that I may use it better.
Title: Re: Why do raw files require tone mapping to look perceptually correct?
Post by: digitaldog on November 28, 2018, 09:36:02 am
Indeed, and I'll be the first to admit I have a ways to go in this direction. I'm just trying to understand how the thing works, so that I may use it better.
Then I recommend you learn how to make import presets (and other presets) to produce an initial rendering you prefer as a starting point than looking at/asking about Capture 1 or any other product you presumably don't own nor know how to use.  ;)
Title: Re: Why do raw files require tone mapping to look perceptually correct?
Post by: kpz on November 28, 2018, 01:07:26 pm
Then I recommend you learn how to make import presets (and other presets) to produce an initial rendering you prefer as a starting point than looking at/asking about Capture 1 or any other product you presumably don't own nor know how to use.  ;)

Oh, I've done such things. It's just that Capture 1 has a month-long free demo, so there's no harm in trying it out and seeing whether it gives better initial renderings that I prefer as a starting point.
Title: Re: Why do raw files require tone mapping to look perceptually correct?
Post by: digitaldog on November 28, 2018, 01:11:06 pm
Oh, I've done such things.
After writing this:

Why do raw files opened in Lightroom seem to almost always require massaging with the highlight and shadow sliders to look perceptually correct (that is, to reproduce what my eyes see in reality)?
If so, and you've found a preset that gives a better initial rendering you prefer as a starting point, my work is done here.  ;)
Title: Re: Why do raw files require tone mapping to look perceptually correct?
Post by: Bart_van_der_Wolf on November 28, 2018, 06:41:34 pm
Thanks Bart.

While it may be entirely a personal artistic preference, as the Digital Dog notes, have you or others here noticed any other important "quirks" of the LR default rendering?

All Raw converters have specific benefits and drawbacks, it's the balance that counts. LR is more than a Raw converter, it's also part editor, and has some useful DAM functionality. How the specific strengths and weaknesses are appreciated, also depend on specific workflow requirements one might have. Personally, I don't like subscription software, which is why I stopped upgrading since version 5.7.1. I've been using it since version 1 first saw light.

For me, and my kind of workflow, I prefer Capture One Pro as Raw converter because the Raw conversion quality is better, with fewer demosaicing artifacts than LR. That may be due to its origin which had to deal with sensors without anti-aliasing filters and with fat pixels which tend to exacerbate aliasing artifacts. Its clipping recovery is not as good as LR's but C1 assumes that a (professional) photographer knows (which is not always the case) how to expose correctly, especially when in a controlled studio setting. Features like Lens Cast Correction (LCC), which also allows removing sensor dust, and Diffraction Correction, and very good color control, and Adjustment Layers, and tethering capability, are all very valuable to me, and real timesavers. I do prefer using the Linear Response Curve instead of the Film Response Curve setting, again to preserve better highlight separation.

Cheers,
Bart
Title: Re: Why do raw files require tone mapping to look perceptually correct?
Post by: kpz on November 29, 2018, 03:44:30 pm
Indeed! And the possible and massive difference in the dynamic range of what we can see and what we can capture as so well illustrated in this must read:
http://www.lumita.com/site_media/work/whitepapers/files/pscs3_rendering_image.pdf (http://www.lumita.com/site_media/work/whitepapers/files/pscs3_rendering_image.pdf)

I finally got around to reading this paper. It is fantastic and answered virtually all of my questions. Thank you for posting it.

I do prefer using the Linear Response Curve instead of the Film Response Curve setting, again to preserve better highlight separation.

Cheers,
Bart

Do you know if Capture One "bakes in" any adjustments that cannot be removed? As pointed out earlier in the thread, "zero" on the ACR sliders is not actually "zero adjustment" in the more recent process versions. Some contrast, tone curve, and black point manipulation is being applied, and so on.

More precisely, suppose in Capture One that I set the tone curve to "scientific linear," the supplied camera profile is reasonably accurate, my monitor has been profiled, and I zero out all other sliders (e.g. sharpening). Then I take a shot of an X-Rite ColorChecker and process it in C1. Will the colors displayed on my screen by C1 be colorimetrically accurate to the physical ColorChecker (up to small errors in the profiling process)? (I would perform this experiment myself but I lack a ColorChecker -- with the questions I am asking it seems I should really get one...)

edit: I thought about this some more. We should ignore the bit about camera profiling for now since this can be fixed by a dedicated end user. The more pressing matter is the tone curve.

Consider "Blue Step Wedge example 1" in this link:
http://www.markshelley.co.uk/Astronomy/Processing/ACR_Critique/acr_critique.html

It shows conclusively that ACR/LR do not perform a linear tone curve, even in the early versions. So the real question is whether "scientific linear" properly renders the step wedge in C1. I am at work still but will try to perform this test later.
Title: Re: Why do raw files require tone mapping to look perceptually correct?
Post by: Bart_van_der_Wolf on November 29, 2018, 06:16:19 pm
Do you know if Capture One "bakes in" any adjustments that cannot be removed? As pointed out earlier in the thread, "zero" on the ACR sliders is not actually "zero adjustment" in the more recent process versions. Some contrast, tone curve, and black point manipulation is being applied, and so on.

I think all Raw converters do this to some extent. If one would use a truly linear Scientific profile, it would probably only be useful for reproductions under a specific light source. Capture One also offers two different Tone Curve Responses, one with a film-like shoulder roll-off in the highlights, and one that's more linear. I prefer the latter due to better highlight separation, and because the film roll-off pushes the brightness of optimally exposed Raw files into clipping (traditionally, PhaseOne cameras underexposed to preserve highlights).

Cheers,
Bart
Title: Re: Why do raw files require tone mapping to look perceptually correct?
Post by: digitaldog on November 29, 2018, 06:39:13 pm
Do you know if Capture One "bakes in" any adjustments that cannot be removed?
Nothing is 'baked in" that can't be removed per se. It is all read only, the raw remains raw. All converters produce an initial rendering just so you can see a full color image. And as we discussed, you can change this at any time.
Title: Re: Why do raw files require tone mapping to look perceptually correct?
Post by: Chris Kern on December 02, 2018, 05:35:52 pm
So, more broadly, why do raw files require significant tone curving to look "right"? In Camera Neutral, for instance, they seem flat and lacking contrast. But again -- I should be seeing raw sensor data with basically just a gamma correction, and the sensor and I are seeing the same photons, so I expect roughly the same result.

I'm arriving late to this party, but it seems to me one point missing from the responses to date—I think Andrew implied it, but I don't believe he stated it explicitly—is that until you change the defaults, Lightroom is configured to display colors and tones in a normalized fashion that renders them similarly for all camera manufacturers and models.

Quote
“If you fell in love with the way your image looked when you chimped the LCD on the back of the camera, this first look in Lightroom or Camera Raw might be disappointing. Neither Lightroom nor Camera Raw uses the camera-maker’s software development kit (SDK) for rendering the digital negative, so expecting the preview to look like the camera LCD is unreasonable. When he was designing the rendering engine, Thomas Knoll made a conscious decision not to try to match the camera companies’ ‘looks,’ but instead to present you with a reasonable and normalized preview of your image.”

Jeff Schewe: The Digital Negative: Raw Image Processing in Lightroom, Camera Raw, and Photoshop, second edition

Whenever you select a different profile (in the Basic panel of the Development module), you get different colors and tones.  Adobe provides its own set of standard profiles (these are the normalized ones), some monochromatic and "artistic" variants that more radically alter the default renderings, and a set that attempts to imitate the JPEG processing options offered by each camera; these last camera-specific ones vary in quality, in my experience—based, I suspect, on (1) how much assistance the manufacturer provided to Adobe and (2) how much personal experience the Adobe software developers had with a particular camera.  The initial position of the sliders doesn't change when you select a different profile, so from the starting point provided by each profile you have the full range of adjustments.

With respect solely to the tone curve, Lightroom offers an Auto setting, again in the Basic panel of the Development module, that works surprisingly well for many images.  This feature recently was updated by using machine learning (presumably "neural-network" technology) to mimic the way experts in post-processing adjusted the tones in various samples of photographs.  Like the profiles, this can be used as-is, or as a starting point for further adjustments.  Unlike the profiles, this tool does alter the sliders, so you can see precisely what it has done to achieve the effect it produces.  When I'm having difficulty getting the global tones I want from a particular image, I find that the Auto tool's magic incantation sometimes helps me figure out where I've gone wrong.  (Of course, in many cases difficult images will become more tractable if you apply local in addition to global adjustments.)

Long story short, if you don't like the Adobe profiles, don't use them.  If you don't like how Adobe has profiled your particular camera(s), roll your own: there are plenty of online tutorials on how to do so.  But with a little practice, it's always possible to get where you want to be no matter where you start.

With respect to the intellectual exercise of understanding the precise semantics of the Adobe profiles, or how its development staff arrived at the settings for specific cameras, I guess you'd have to ask the Lightroom and Camera Raw developers.  Good luck with that.
Title: Re: Why do raw files require tone mapping to look perceptually correct?
Post by: Rand47 on December 08, 2018, 12:34:52 pm
This has been a really good thread to follow along with.  Thanks to all who have contributed, and especially to Andrew for your expertise and the good resources. 

Good stuff ...

Rand