Preserving tonal ratios in Lightroom

[Jim Kasson]

I’ve been doing some macro photography, and I’m unsatisfied with the sharpness that I’m getting. I’ve been trying to develop a testing regime aimed at determining the amount of blurring caused by camera vibration, unflat field, aberrations, diffraction, and the like. To that end, I’ve created a target which has a fairly broad band of high-spatial-frequency energy. I am photographing the target, processing the images in Lightroom, exporting them as TIFFs, reading them into Matlab, converting them to a linear (gamma = 1) representation, and performing analysis, the critical part of which is measuring the standard deviation (or, if you prefer, root-mean-square noise) of the image.

In order to make the rms noise measurement meaningful with real-world images, I need to compensate for both global and local exposure differences. Since the value of the standard deviation of an image in a linear color space should, all else being equal and ignoring photon noise, be proportional to exposure, I perform the compensation by forming a correcting image by filtering the input image with a large (400x400 pixel) constant-value kernel, and then dividing the input image by the correcting image to form the corrected image.

There’s a problem: the technique overcorrects the images exported from Lightroom. I tried turning off everything I could find in the Develop module, including camera calibration. It made a difference, but didn’t fix things.

So I created two images that differed in exposure by a stop. I measured the ratio of the mean values of the G channels of the two raw images with Rawdigger: it was 2.015. In Lightroom, Using PV 2012, the ratio of the linearized Adobe RGB green channels was 1.688. With PV 2010 and PV 2003, it was 1.681. Using all three channels and converting to monochrome in Matlab produces similar result. There seems to be a tone curve applied by Lightroom that keeps ratios in the raw file from being preserved in linear representations of the converted image.

I tried Iridient Developer, and got different, but still incorrect (in the photogrammetric sense) ratios of about 1.65. The ratio varied with the processing options. I thought the raw channel mixer set to green only might produce the right ratio, but no joy.

Using DCRAW, with the command line incantation “dcraw -v -4 -a -w -j -T -o1 _D437349.NEF” produces sRGB files with the green channel mean ratio of 2.015, the same ratio as that of the raw files –- actually it’s not quite the same ratio (it differs in the fifth decimal place) but I attribute that to the change of color space from camera native to sRGB. DCRAW users will note that I’m white balancing to average; leaving this out makes little difference.

So, while I have a solution for raw conversion that I can use, it’s much less convenient for me than to do the conversions using Lightroom.

So, my question is: How do I set up Lightroom so that the tone ratios of the original raw file are preserved in a linear representation of the output file?

Thanks for any help on this.

Jim

PS. For a little background, my first fumbling steps in this project are covered in this and subsequent posts.

[Tim Lookingbill]

That's quite a quandary you've found yourself there, Jim.

Did you make sure to check the parabula statistus murado on the second go round on those linear compounder numbers you came up with? You may need to get Lightroom CSI, a version used by NASA and NORAD.

Is there a reason why you haven't posted image samples showing this sharpness issue with macro shots both here and on your blog?

I mean I use the crappiest lenses with extension tube and 2X teleconverter all (under $100) strapped to the cheapest DSLR I could find and get excellent sharpness editing all those foggy, crappy captures into tack sharp perfection in both Lightroom and ACR. No analysis needed.

Oh, and I can do all those edits faster than I could read and figure out what you were trying to accomplish in your post.

[Jim Kasson]

That's quite a quandary you've found yourself there, Jim.

Tim, I admit to having a analytical bent. I like doing this stuff, probably more than is good for my photography. Guilty as charged. I'm hoping maybe Eric can help me out, but I suspect that a lot of Lightroom gurus know how to turn off some processing I missed.

Is there a reason why you haven't posted image samples showing this sharpness issue with macro shots both here and on your blog?

I did. Here and here.

Oh, and I can do all those edits faster than I could read and figure out what you were trying to accomplish in your post.

I'm sure you can. I've found that I get better results if I spend most of my effort on the capture, and relatively little on the editing, but that's just me.

Thanks (I think).

BTW, I've figured out how to call DCRAW from Matlab and really don't need to know how to make Lightroom do the job anymore, but I'm still curious as to what it's doing with the mystery tone curve (if that's what it is) and why.

Jim

[Tim Lookingbill]

Jim, your firehouse images look better than what I can get straight out of my camera with regards to sharpness. I really don't see a sharpness problem with those posted images. They look real and follow the character of the light in how it's suppose to render sharpness appearance.

They do have a sort of natural looking tonal roll off "Low Clarity" for lack of a better descriptive term which brings me to mentioning something I noticed about display calibration variances between two displays I've been using in the past 3 years that have affected my perception of the appearance of clarity in combination with PV2003 vs PV2012 renderings.

Clarity works hand and glove with sharpness appearance and I'ld been using CS3's PV2003 Clarity rendering which behaves much different than PV2012's. But to compound things I also noticed when I went from the Dell 2209WA (subpar quality with regards to linearity/black point appearance), calibrated with old i1Match/i1Display package, and switching to my current LG 27ea63v-p (VERY LINEAR) with much deeper blacks and slightly punchier contrast calibrating with ColorMunki Display, PV2003's Clarity & Parametric Curve behavior noticeably improved working on the LG.

For some reason the LG induced me to re-edit quite a few low contrast scenes edited on the Dell in PV2003 that used to look over processed (HDCartooned?) where now the PV2003 Clarity slider can be cranked up even more than I could on the Dell now with the LG. When I switch to viewing these images edited on the Dell/PV2003 using Lightroom 4's PV2012 they look even less cartoonish but now the Clarity slider applies a much broader tonal roll off over PV2003's "HighPass" fat halo look to contrasted edges. This is most likely what you are seeing in Lightroom in how it renders low light, low contrast scenes in the firehouse images.

So my point is what I think you may be seeing is influenced by variances I've described in my situation that I think instead of applying precision math to sort it out is better mitigated by finding other ways to make your images look sharper using Lightroom's editing tools. I'm guessing your images are most likely over 12MP and with the low contrast/AA filter would most likely require a larger LR sharpening radius while maintaing low contrast. It's easier to make high contrast images look sharper.

I can't tell you how many times I've gone back and re-edited certain shots I've fallen in love with because I lingered on the image for too long to make them look 3D. That Parametric curve editor is addictive especially cranking up mids/highlights on low contrast, dim shots.

Below are two rendering attempts on two separate shots of the same scene shot through a window where one is darker/high contrast, the other is brighter and a bit lower contrast to show how contrast perception affects overall appearance of sharpness. Both have the Sharpness radius set at (2) instead of the normal (1) using PV2003 in CS3 ACR. Clarity is set at 65 on the darker/hi contrast version w/over-cranked parametric curve while the lighter less contrast version has Clarity at 100 w/less parametric midrange boost.

[ErikKaffehr]

Hi,

I think you can go to Process Version 2010 and set all relevant controls to zero. I don't think you can get straight rendition out of PV 2012.

Best regards
Erik

Tim, I admit to having a analytical bent. I like doing this stuff, probably more than is good for my photography. Guilty as charged. I'm hoping maybe Eric can help me out, but I suspect that a lot of Lightroom gurus know how to turn off some processing I missed.

I did. Here and here.

I'm sure you can. I've found that I get better results if I spend most of my effort on the capture, and relatively little on the editing, but that's just me.

Thanks (I think).

BTW, I've figured out how to call DCRAW from Matlab and really don't need to know how to make Lightroom do the job anymore, but I'm still curious as to what it's doing with the mystery tone curve (if that's what it is) and why.

Jim

[Tim Lookingbill]

Jim, have you examined the quality of pixel edge sawtoothed detail viewing your firehouse images in their current edited state at pixel level to see if the proper sharpen settings are adequate?

My shots posted above using subpar quality equipment and captured through a window are sharp in their current unedited Raw state that produce perfect halo-less sharp edges viewed at 400% after applying only 120 Amount in ACR PV2003.

What do the pixelated edges look like on your images compared to what I get shown below viewed at 400%?

[hjulenissen]

It is easy to process raw files in MATLAB using dcraw as a file-parser (decode in "document mode" into 16-bit TIFF, then use imread() in MATLAB).

I believe that Lightroom is very much about "offer users what we think they will be happy with" as opposed to "offer sliders that are mathematically/physically analyzable". And it seems to be a popular software package.

-h

[Jim Kasson]

It is easy to process raw files in MATLAB using dcraw as a file-parser (decode in "document mode" into 16-bit TIFF, then use imread() in MATLAB).

Good idea. Since yesterday, I've been constructing DCRAW commands as character strings in Matlab and using "system(commandString);" to execute them and then using  imread() to get the TIFF file. Your idea of using DCRAW's document mode rather than letting DCRAW do the demosaicing would give me access to each of the four raw planes, and I could analyze each individually, or use Matlab's demosaicing function, and stay in native camera color space.

Jim

[Jim Kasson]

Jim, your firehouse images look better than what I can get straight out of my camera with regards to sharpness. I really don't see a sharpness problem with those posted images. They look real and follow the character of the light in how it's suppose to render sharpness appearance.

Tim, here is an example image, with no clarity or contrast boosting in Lr, just using the default sharpening. I did tweak Exposure and Shadows a bit. The image was exposed at f/25 for 3 seconds with the 105mm f/2.8G Nikon Macro lens. I don't like to use openings that small because of diffraction, but I needed the DOF, and, as you'll see, I still don't have enough. I'm worried about camera motion, but don't know if is an issue with this image. The camera was a D4, which I used because I like the way it handles, but when I get this series dialed in, I'm going to switch to a D800E.

First, the overall image:



Next, the plane of sharpest focus, at 1:1:



It could be a lot sharper, IMHO. I know I can make improvements in post, but I'd rather start with a better image.

And, to boot, the DOF isn't adequate, as seen by the two crops from behind the plane of sharp focus:








So, I see my challenge as getting the main plane of focus sharper by opening up the lens and maybe (depending on what I find with my testing) doing something to mitigate vibration besides shutter delay and the self-timer. That will make the DOF problem worse, and I'll probably have to bite the bullet and do focus stacking to fix that.

Then there's my reflection in the shiny bits. I'm still thinking about what to do about that.

Thanks for your suggestions. I'm sure they'll help once I get a better capture.

Jim

[Bart_van_der_Wolf]

Tim, here is an example image, with no clarity or contrast boosting in Lr, just using the default sharpening. I did tweak Exposure and Shadows a bit. The image was exposed at f/25 for 3 seconds with the 105mm f/2.8G Nikon Macro lens.

Hi Jim,

That will not only give a lot of diffraction, but it will also lose effective resolution (and require long exposure times). An aperture of f/25 will limit resolution to 1/(N*wavelength) cycles/mm, or 1/(25*0.000555)=72 cy/mm for green light. The fire-truck red will be even worse. At 72 cy/mm and higher the MTF will be zero, i.e. no resolution, and all micro-detail will be lost without any chance of recovery. At the Nyquist frequency of the D4 sensor (68.5 cy/mm) the MTF is almost zero, so only the highest contrast (>100:1) detail has any chance to make it in the absence of noise. Add a tiny bit of lens aberration and there will be nothing to recover, and any added defocus will strip even the faint chance of resolving any high spatial frequency.

I don't like to use openings that small because of diffraction, but I needed the DOF, and, as you'll see, I still don't have enough.

Only focus-stacking can achieve what you want. Opening up the lens will significantly reduce the diffraction, and allow shorter exposure times.

I'm worried about camera motion, but don't know if is an issue with this image. The camera was a D4, which I used because I like the way it handles, but when I get this series dialed in, I'm going to switch to a D800E.

The smaller sensels of the D800 will be impacted by diffraction starting at f/5.6 (the D4 at f/8.0), if you want to gain resolution you will be limited to f/18 before total loss of high spatial frequency resolution.

It could be a lot sharper, IMHO. I know I can make improvements in post, but I'd rather start with a better image.

And, to boot, the DOF isn't adequate, as seen by the two crops from behind the plane of sharp focus:

Focus stacking at f/4 (maybe f/5.6 at most) is going to give you optimal results for macro shots at 1:1, but the DOF will be approx. 0.1 mm, so either a lot of focus brackets will be required, or you'll have to settle for a smaller (downsampled) output size,  and/or more diffraction.

Then there's my reflection in the shiny bits. I'm still thinking about what to do about that.

The wider aperture used with focus stacking may help, otherwise use some lens blur in post-processing.

Cheers,
Bart

[ErikKaffehr]

Hi,

This page shows the effects of diffraction vs. DoF quite well: http://echophoto.dnsalias.net/ekr/index.php/photoarticles/49-dof-in-digital-pictures?start=1

Personally, I would use a large radius (like 1.6 and deconvolution type sharpening).

Best regards
Erik

Hi Jim,

That will not only give a lot of diffraction, but it will also lose effective resolution (and long exposure times). An aperture of f/25 will limit resolution to 1/(N*wavelength) cycles/mm, or 1/(25*0.000555)=72 cy/mm for green light. The fire-truck red will be even worse. At 72 cy/mm and higher the MTF will be zero, i.e. no resolution, and all micro-detail will be lost without any chance of recovery. At the Nyquist frequency of the D4 sensor (68.5 cy/mm) the MTF is almost zero, so only the highest contrast (>100:1) detail has any chance to make it in the absence of noise. Add a tiny bit of lens aberration and there will be nothing to recover, and any added defocus will strip even the faint chance of resolving any high spatial frequency.

Only focus-stacking can achieve what you want. Opening up the lens will significantly reduce the diffraction, and allow shorter exposure times.

The smaller sensels of the D800 will be impacted by diffraction starting at f/5.6 (the D4 at f/8.0), if you want to gain resolution you will be limited to f/18 before total loss of high spatial frequency resolution.

Focus stacking at f/4 (maybe f/5.6 at most) is going to give you optimal results for macro shots at 1:1, but the DOF will be approx. 0.1 mm, so either a lot of focus brackets will be required, or you'll have to settle for a smaller (downsampled) output size,  and/or more diffraction.

The wider aperture may help, otherwise use some lens blur in post-processing.

Cheers,
Bart

[Rand47]

Tim, here is an example image, with no clarity or contrast boosting in Lr, just using the default sharpening. I did tweak Exposure and Shadows a bit. The image was exposed at f/25 for 3 seconds with the 105mm f/2.8G Nikon Macro lens. I don't like to use openings that small because of diffraction, but I needed the DOF, and, as you'll see, I still don't have enough. I'm worried about camera motion, but don't know if is an issue with this image. The camera was a D4, which I used because I like the way it handles, but when I get this series dialed in, I'm going to switch to a D800E.

First, the overall image:



Next, the plane of sharpest focus, at 1:1:



It could be a lot sharper, IMHO. I know I can make improvements in post, but I'd rather start with a better image.

And, to boot, the DOF isn't adequate, as seen by the two crops from behind the plane of sharp focus:








So, I see my challenge as getting the main plane of focus sharper by opening up the lens and maybe (depending on what I find with my testing) doing something to mitigate vibration besides shutter delay and the self-timer. That will make the DOF problem worse, and I'll probably have to bite the bullet and do focus stacking to fix that.

Then there's my reflection in the shiny bits. I'm still thinking about what to do about that.

Thanks for your suggestions. I'm sure they'll help once I get a better capture.

Jim

Change-over valve on a fire engine's pump panel, right? 

Rand

[Jim Kasson]

Change-over valve on a fire engine's pump panel, right? 

Sounds like you know a lot more about it than I do. Here's a view of most of the panel. The control is at the very bottom, just a little right of center. Engine is an early 70s American LaFrance.



Jim

[Jim Kasson]

This page shows the effects of diffraction vs. DoF quite well: http://echophoto.dnsalias.net/ekr/index.php/photoarticles/49-dof-in-digital-pictures?start=1

Thanks, Erik. And thanks for your post processing ideas. I'm not ready for them yet, but I'll get there.

Jim

[Jim Kasson]

Only focus-stacking can achieve what you want. Opening up the lens will significantly reduce the diffraction, and allow shorter exposure times.

The smaller sensels of the D800 will be impacted by diffraction starting at f/5.6 (the D4 at f/8.0), if you want to gain resolution you will be limited to f/18 before total loss of high spatial frequency resolution.

Thanks, Bart, for the suggestions and the theory. Now that I've got DCRAW and Matlab playing together, my first (strobe-lit) tests indicate that the sharpest aperture for the Nikon 105mm f/2.8G macro is f/8, and for the Zeiss 100mm f/2 ZF, it's f5.6, at least at the distance I'm using initially. That's very close to what you said independent of the particular lens. I still have more testing to do, with both lenses on both bodies, and I want to see if there is much variation in optimum aperture based on the spatial frequencies that I look at. Then I want to compare sharpness of the two lenses against each other. Then it'll be onto vibration-induced blur with various shutter and mirror release techniques. Then some other lenses: the Sigma 180 mm APO, and maybe the 60mm Micro Nikkor.

Jim

[ErikKaffehr]

Jim,

Please share your findings!

Best regards
Erik

Thanks, Bart, for the suggestions and the theory. Now that I've got DCRAW and Matlab playing together, my first (strobe-lit) tests indicate that the sharpest aperture for the Nikon 105mm f/2.8G macro is f/8, and for the Zeiss 100mm f/2 ZF, it's f5.6, at least at the distance I'm using initially. That's very close to what you said independent of the particular lens. I still have more testing to do, with both lenses on both bodies, and I want to see if there is much variation in optimum aperture based on the spatial frequencies that I look at. Then I want to compare sharpness of the two lenses against each other. Then it'll be onto vibration-induced blur with various shutter and mirror release techniques. Then some other lenses: the Sigma 180 mm APO, and maybe the 60mm Micro Nikkor.

Jim

[Rand47]

Sounds like you know a lot more about it than I do. Here's a view of most of the panel. The control is at the very bottom, just a little right of center. Engine is an early 70s American LaFrance.



Jim

Jim,
Sorry for the digression on the thread, but I can't resist.  Here's a MUCH younger version of me at the wheel of a '73 LaFrance engine (taken circa '82)



If you're interested, what that valve does is change the pump from pumping in series (for max pressure) to pumping in parallel (max volume/capacity).  The pump has two impellers and can be run both ways.
All this to prove, you can take the firefighter out of the firehouse... but....

Rand

[jjj]

Then there's my reflection in the shiny bits. I'm still thinking about what to do about that.

The trick to photographing reflective objects with curves and angles is camera position and lighting. Usually multiple exposures are made with different lighting set ups, which are then blended in post. Keeping light off camera will help too with reflections.
Jim Haefner has posted some behind the scenes of his car photography on LuLa which demonstrate exactly this method and he's done some stunning car photography. Couldn't any to link to though.

[nma]

I’ve been doing some macro photography, and I’m unsatisfied with the sharpness that I’m getting. I’ve been trying to develop a testing regime aimed at determining the amount of blurring caused by camera vibration, unflat field, aberrations, diffraction, and the like. To that end, I’ve created a target which has a fairly broad band of high-spatial-frequency energy. I am photographing the target, processing the images in Lightroom, exporting them as TIFFs, reading them into Matlab, converting them to a linear (gamma = 1) representation, and performing analysis, the critical part of which is measuring the standard deviation (or, if you prefer, root-mean-square noise) of the image.

In order to make the rms noise measurement meaningful with real-world images, I need to compensate for both global and local exposure differences. Since the value of the standard deviation of an image in a linear color space should, all else being equal and ignoring photon noise, be proportional to exposure, I perform the compensation by forming a correcting image by filtering the input image with a large (400x400 pixel) constant-value kernel, and then dividing the input image by the correcting image to form the corrected image.

There’s a problem: the technique overcorrects the images exported from Lightroom. I tried turning off everything I could find in the Develop module, including camera calibration. It made a difference, but didn’t fix things.

So I created two images that differed in exposure by a stop. I measured the ratio of the mean values of the G channels of the two raw images with Rawdigger: it was 2.015. In Lightroom, Using PV 2012, the ratio of the linearized Adobe RGB green channels was 1.688. With PV 2010 and PV 2003, it was 1.681. Using all three channels and converting to monochrome in Matlab produces similar result. There seems to be a tone curve applied by Lightroom that keeps ratios in the raw file from being preserved in linear representations of the converted image.

I tried Iridient Developer, and got different, but still incorrect (in the photogrammetric sense) ratios of about 1.65. The ratio varied with the processing options. I thought the raw channel mixer set to green only might produce the right ratio, but no joy.

Using DCRAW, with the command line incantation “dcraw -v -4 -a -w -j -T -o1 _D437349.NEF” produces sRGB files with the green channel mean ratio of 2.015, the same ratio as that of the raw files –- actually it’s not quite the same ratio (it differs in the fifth decimal place) but I attribute that to the change of color space from camera native to sRGB. DCRAW users will note that I’m white balancing to average; leaving this out makes little difference.

So, while I have a solution for raw conversion that I can use, it’s much less convenient for me than to do the conversions using Lightroom.

So, my question is: How do I set up Lightroom so that the tone ratios of the original raw file are preserved in a linear representation of the output file?

Thanks for any help on this.

Jim

PS. For a little background, my first fumbling steps in this project are covered in this and subsequent posts.

Hello,
I took a Johnny-come-lately perusal of this thread. You got some great info from Brat and Eric. I want to address your desire to compute the local noise in the image: Perhaps you can succeed in the technique you are trying but i think there is a more straightforward  analysis that will be helpful. First, the noise in the sensels is independent, from sensel to sensel, if you can work on the raw values. The key is not to convolve adjacent sensel values. Simply record about 20 identical images and compute the noise at each voxel as the sample standard deviation using Matlab. Hope this helps.

[MarkH2]

Jim,
Sorry for the digression on the thread, but I can't resist.  Here's a MUCH younger version of me at the wheel of a '73 LaFrance engine (taken circa '82)

Rand

So that was You doubling for Burt!