MergeRAW 1.0 Released - Lightroom Classic Plugin for Computational Photography

[Ryan Mack]

I’m happy to announce that, 6 years after I started, I’ve released a 1.0 version of my free MergeRAW Lightroom Classic plugin that adds all the computational photography features your phone has but your DSLR or mirrorless doesn’t…

• Lower Base ISO for long exposures - If you want blurry waterfalls in daylight without an ND or cleaner shadows for a huge print, do frame averaging directly in Lightroom!
• Night Mode - Smart photo stacking… align and stack discarding any blurry frames… why should your Z8 need a tripod if your phone doesn’t?
• Pixel Shift - NX Studio is slow and has serious quality-impacting bugs… merge pixel shift sets directly in Lightroom and get the best quality possible.
• Multiple Exposures - In camera multi-exp records NEFs/CR3s but the output is only a JPEG… blend the raw files in LR and get the DNG you really wanted.
• Astro Landscape in LR - Automatic sky alignment and foreground masking with satellite and hot pixel removal, with or without dark frames.

And more…

Let me know what you think!

https://mackman.net/mergeraw/

[Chris Kern]

Very interesting concept.  Can it ingest bursts from Fuji X-Trans cameras?

[Ryan Mack]

Very interesting concept.  Can it ingest bursts from Fuji X-Trans cameras?

Heh jumped right to my Achilles heel :-). No, right now it only works with RGGB sensors ;-). I wrote my own demosaic algorithms for alignment at sub pixel offsets and it definitely won’t work with x-trans. For the modes with alignment disabled it *might* work because it never demosaics the data. If you want to try let me know what happens. If there’s easy fixes for the unaligned path I’m happy to add it to a future release.

[Ryan Mack]

Actually come to think of it, I can probably have LR export a linear DNG for X-trans files and then operate on those but it’ll be slower. If you are interested in testing it, I’ll make a new release. I would need you to send me at least one x-trans file so I can figure out how to identify it. My email address is <first>.<last>@gmail.com. Thanks.

[Ryan Mack]

Lightroom can't generate a linear DNG for some reason (API looks like it should but <shrug>). I can use Adobe DNG Converter to do it though. Quick POC went well though. What mode are you interested in? The sky tracking right now doesn't work with demosaic data because it expect bayer filtered data when determining the sky rotation. If that's your main use case I'd need to spend some time figuring out how to get star tracking to work with linear DNGs.

[Ryan Mack]

I've uploaded a v1.2.0 release. If you have Adobe DNG Converter installed then my plugin will use Adobe DNG Converter to demosaic the file so it will work for most modes the plugin offers (not star tracking which aligns single pixels bayer data).

[Chris Kern]

I've uploaded a v1.2.0 release. If you have Adobe DNG Converter installed then my plugin will use Adobe DNG Converter to demosaic the file so it will work for most modes the plugin offers (not star tracking which aligns single pixels bayer data).

This approach works fine, Ryan, and the Adobe DNG Converter zips through a burst of fifteen 40Mpx X-Trans raw files in ~3.5 seconds on an Apple M5 MacBook Air.  So on a modern platform the time to convert to the linear DNG format and assemble the composite really isn't a concern.  Since the composite linear DNG (like the intermediate ones) is still scene-referred, most of Lightroom's raw controls remain available.

I tested the frame-averaging and night modes, and they work as advertised on the X-Trans files.  I didn't try the remove/embrace motion or sensor clean-up/art repro modes, but I presume they would also function as intended.  You mentioned that star-tracking needs to operate on a Bayer pattern and that also appears to be true of the pixel shift mode, which triggers a diagnostic error message when run against the linear DNGs.

It occurs to me that it might be useful to have a parameter to throttle the aggressiveness of the blurred-image removal in the night mode (i.e., to adjust the threshold to detect blur) or simply modify it automatically in relation to the resolution of the source files.

I presume "real cameras" eventually will acquire support for the types of computational photography techniques offered by your application.  (My Fuji X-T5 already offers a pixel shift option.  The newer X-E5 does not, despite having the same image sensor and processor.)

By the way, I wish Adobe could emit new revs of its applications as quickly as you did—if not to add features, at least to fix bugs and regressions.

[fdisilvestro]

I’m happy to announce that, 6 years after I started, I’ve released a 1.0 version of my free MergeRAW Lightroom Classic plugin that adds all the computational photography features your phone has but your DSLR or mirrorless doesn’t…

• Lower Base ISO for long exposures - If you want blurry waterfalls in daylight without an ND or cleaner shadows for a huge print, do frame averaging directly in Lightroom!
• Night Mode - Smart photo stacking… align and stack discarding any blurry frames… why should your Z8 need a tripod if your phone doesn’t?
• Pixel Shift - NX Studio is slow and has serious quality-impacting bugs… merge pixel shift sets directly in Lightroom and get the best quality possible.
• Multiple Exposures - In camera multi-exp records NEFs/CR3s but the output is only a JPEG… blend the raw files in LR and get the DNG you really wanted.
• Astro Landscape in LR - Automatic sky alignment and foreground masking with satellite and hot pixel removal, with or without dark frames.

And more…

Let me know what you think!

https://mackman.net/mergeraw/

Hi Ryan,

Thank you so much for this! I will be testing and using this plugin for sure!

[Ryan Mack]

Thanks for looking at it and sharing your feedback!

It occurs to me that it might be useful to have a parameter to throttle the aggressiveness of the blurred-image removal in the night mode (i.e., to adjust the threshold to detect blur) or simply modify it automatically in relation to the resolution of the source files.

It would be really helpful if you could upload some example files to dropbox/google drive/etc to show what you're trying to accomplish. You can send the link to my name at gmail.com.

My Fuji X-T5 already offers a pixel shift option.

I didn't know Fuji had pixel shift, my X100T sure didn't! If pixel shift is important to you and you can provide a set of pixel shift image files I can probably implement that in an hour or two, I just need to record the shifting and bayer pattern for the camera.

By the way, I wish Adobe could emit new revs of its applications as quickly as you did—if not to add features, at least to fix bugs and regressions.

Well, Adobe has to make million of photographers happy and test with tens of millions of photos. It's a lot easier to do free projects mostly for my own benefit :-)

[digitaldog]

Well, Adobe has to make million of photographers happy and test with tens of millions of photos. It's a lot easier to do free projects mostly for my own benefit :-)

Indeed, on top of millions of lines of code.

[Ryan Mack]

Here you go... announcing version 1.3.0:

• Night mode catches subtle blur — "Skip Blurry Frames" now detects the small motion smear image stabilization leaves behind and drops those frames: composites come out as sharp as the best shots in the burst
• Night mode picks the best alignment anchor — a shaken "most-selected" photo  (the active / highlighted frame in the LR grid) no longer anchors the alignment; a sharp, centrally-positioned frame does, while your photo still names the output and keeps its capture time
• Automatic hot pixel maps — merges of 8+ frames map and remove the sensor's persistently warm pixels from the shoot itself: dark-frame-style cleanup without shooting dark frames
• Honest exposure metadata — merged DNGs report exposure time and ISO from the frames actually stacked

[Chris Kern]

• Night mode catches subtle blur — "Skip Blurry Frames" now detects the small motion smear image stabilization leaves behind and drops those frames: composites come out as sharp as the best shots in the burst
• Night mode picks the best alignment anchor — a shaken "most-selected" photo  (the active / highlighted frame in the LR grid) no longer anchors the alignment; a sharp, centrally-positioned frame does, while your photo still names the output and keeps its capture time

The two-pass blur detection did the trick, Ryan.  Thanks for the quick turnaround.

(By the way, I never even noticed the Lightroom selection alignment issue, even though I changed it several times during my own testing.)

[Ryan Mack]

My Fuji X-T5 already offers a pixel shift option.  The newer X-E5 does not, despite having the same image sensor and processor.

I uploaded version 1.5.0 which supports 5 and 20 shot X-Trans pixel shift. I'm not sure if you camera lets you shoot 5 shot mode but that gives you R G and B at all photo sites without the 2x linear resolution bump. In LR you can select the first 5 shots or all 20 and it will "do the right thing." I was able to add exposure matching (thankfully Fuji seems to have fairly precise aperture control so not much difference in exposure between frames). I don't support deghosting with X-Trans because I want to avoid using Adobe DNG Converter for pixelshift to demosaic the image. Adobe intentionally blurs the image which rather undoes the point of pixel shift in my opinion. I would need to write my own x-trans demosaic algorihm which is definitely more than I'm looking to do wight now.

In my tests with my Z8 I get significantly better fine detail using pixel shift than I get with Adobe AI-based super-resolution but it takes a perfectly still tripod and room. I've confirmed this for a bunch of watercolor art reproduction as well as some Christmas tree photos where the pine tree needles on red ornaments (strong color shifts, similar luminance) really becomes too mushy for AI to cover as well as pixel shift. Would love to hear your feedback.

I got into a big debate with one of the core LR/ACR devs I met on a diving trip about whether it's better to have 4 images for pixel shift or a million images to train an AI. We agreed to disagree at the time. Years later all evidence indicates that if you have perfect technique, pixel shift wins. Since most folks don't have perfect technique most of the time, AI wins the common case :-)

[Chris Kern]

I uploaded version 1.5.0 which supports 5 and 20 shot X-Trans pixel shift. I'm not sure if you camera lets you shoot 5 shot mode but that gives you R G and B at all photo sites without the 2x linear resolution bump. In LR you can select the first 5 shots or all 20 and it will "do the right thing."

Fujifilm only supports a 20-shot sequence in the two X-Trans cameras that offer pixel shift, so being able to create a composite with only five images does indeed provide new functionality.

From my perspective, the most significant advantage of your approach over Fuji's is that the merge takes place within Lightroom.  Fuji's implementation requires that the merge be performed separately by a dedicated (free to download) program.

In my tests with my Z8 I get significantly better fine detail using pixel shift than I get with Adobe AI-based super-resolution but it takes a perfectly still tripod and room. I've confirmed this for a bunch of watercolor art reproduction as well as some Christmas tree photos where the pine tree needles on red ornaments (strong color shifts, similar luminance) really becomes too mushy for AI to cover as well as pixel shift. Would love to hear your feedback.

As I mentioned in our private message exchange, in my admittedly limited testing I have not so far been able to detect any significant visual difference in the merge of X-Trans files with either Fuji's or your 20-shot pixel shift and Lightroom's Super Resolution.

[Ryan Mack]

As I mentioned in our private message exchange, in my admittedly limited testing I have not so far been able to detect any significant visual difference in the merge of X-Trans files with either Fuji's or your 20-shot pixel shift and Lightroom's Super Resolution.

That very would well could be the effect of the x-trans sensor working better than a Bayer filter already, I suppose that was the point of the design. You are also likely approaching or exceeding the lens's resolving power with a 40mpix APS-C sensor even without the 20-shot trying to use half pixel offsets. I only get benefit with specific lenses stopped down 2 stops from max. More than that an diffraction makes superresolution pixel shift pointless. The 4 shot mode on my Z8 still reduces color artifacts though.

[kers]

I see that there are two posts about this... maybe delete the other...

Just had time to have a quick look at the plug-in 1.93 and my first reaction is : Many Thanks!
Installation was a breeze on a mac and
I have a 10 core machine and Nikon-NX only used one to merge- so yes way faster! and seems to work really well.

frame averaging works well and fast!

pixelshift
The quality of the pixelshift merge seems about the same with NX, there is a small difference in sharpness. MergeRAW seems somewhat sharper in the detail.
I like the list of faults you get when something went wrong...
NX has one small advantage: it detects which photos belong to a pixelshift sequence.