Pages: 1 2 [3] 4   Go Down

Author Topic: RAW Conversion Snake Oil  (Read 29794 times)

Fine_Art

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1172
Re: RAW Conversion Snake Oil
« Reply #40 on: January 16, 2013, 07:16:24 pm »

Like others, I can't discern the point of the OP's post. It appears to be more inflammatory than illuminating. I would not recommend ImagesPlus for general photographic use, but it is very useful for scientific work and sensor analysis using raw files. See my post in another thread for more details

Regards,

Bill

The topic was leaving in high iso chroma noise if you can't properly remove it. Eric showed a side by side of the improvement in LR2 vs 3 for this. There was still color banding created by the software in the file.

Since you have access to IPlus and the latest versions of probably every top Raw converter you can show the side by side of Iplus and LR4 on an ISO 3200 or 6400 shot. With NR and without.

I cannot. I did provide shots from Sony raw converter, Picture Ninja and a straight conversion from IPlus. All the raw converters were leaving color splotches from smoothing in the file. They were artificial drops of color from being unable to smooth the noise properly. If you do smooth the noise you wipe out all detail in the shot. You have no edges, even from a very sharp lens.

Its very simple to end the controversy which is "are raw converters good enough on high ISO?" A high ISO file with noise removed and NO sharpening can be attached cropped. If the file is as sharp as the same file from Iplus without NR then the raw converter has not damaged your picture, it has only removed noise. If you have to use extensive sharpening routines to reverse what the software did in NR there is room for improvement.
Logged

Fine_Art

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1172
Re: RAW Conversion Snake Oil
« Reply #41 on: January 16, 2013, 10:19:44 pm »

Here is a way for independent verification. Anyone can download the RAW of the best DSLR around the D800 from Imaging Resource. They are good enough to provide RAWs for your review.

Here is the raw of their ISO3200 shot NR set to 0 in camera.
http://www.imaging-resource.com/PRODS/nikon-d800/D800hSLI03200NR0.NEF.HTM

Here is my jpg of the conversion. I am going out on a limb to say the jpg of a conversion to 32bit fit is probably sharper than what your RAW converter provides. Nothing has been done to the file besides RAW to FIT and save as JPG. I tried to provide a rar'd tiff but it was too big for the upload site. Bob and anyone else can do a difference if they follow the same steps. No NR, no sharpening or anything else is applied, it is as your sensor sees it after de-bayering. You will have to do your own gamma.
http://www.sendspace.com/file/nrkgdg

Test your converter against the detail. Default settings, with NR, with NR and sharpening. I bet you will have to "capture sharpen", not because the file was not sharp, but because your software blurred it.
Logged

Fine_Art

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1172
Re: RAW Conversion Snake Oil
« Reply #42 on: January 16, 2013, 11:12:15 pm »

Here is the same file with white bal and brightening. I fealt absolutely no need to do NR or sharpening. I left it alone. Even the chroma noise looks fine if its individual pixels rather than clumps created by your converter.

http://www.sendspace.com/file/w1tq12

This is a very nice image. For ISO 3200 its incredible.
Logged

jeremypayne

  • Guest
Re: RAW Conversion Snake Oil
« Reply #43 on: January 16, 2013, 11:37:04 pm »

Tilting at windmills ...

Logged

Fine_Art

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1172
Logged

32BT

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 3095
    • Pictures
Re: RAW Conversion Snake Oil
« Reply #45 on: January 17, 2013, 03:12:24 am »

[X] clueless

besides, these type of posts without real names, doesn't that equate to trolling…

« Last Edit: January 17, 2013, 03:23:03 am by opgr »
Logged
Regards,
~ O ~
If you can stomach it: pictures

32BT

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 3095
    • Pictures
Re: RAW Conversion Snake Oil
« Reply #46 on: January 17, 2013, 03:29:52 am »

The whole point of RAW conversion is nicely summed up by this example. Notice the checkerboard patterns? Those are in a favourable area of the image, that is: an area which one would render to neutral gray. In neutral gray, demosaicing shouldn't be necessary.

Now the question is: how do you propose to remove to checkerboard patterns while maintaining your preferred noise pattern?

How do you sharpen this image for output?

Or do you also like the checkerboard pattern?
Logged
Regards,
~ O ~
If you can stomach it: pictures

Bart_van_der_Wolf

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 8913
Re: RAW Conversion Snake Oil
« Reply #47 on: January 17, 2013, 07:45:57 am »

The whole point of RAW conversion is nicely summed up by this example. Notice the checkerboard patterns? Those are in a favourable area of the image, that is: an area which one would render to neutral gray. In neutral gray, demosaicing shouldn't be necessary.

Hi Oscar,

To achieve a neutral grey, demosaicing is also necessary, because otherwise it would not be possible to know that it's neutral grey, or a subtle/pastel color, or detail afterall.

Quote
Now the question is: how do you propose to remove to checkerboard patterns while maintaining your preferred noise pattern?


By using a different Rawconverter? These mazing and zipper artifacts are a by-product of a demosaicing algorithm that tries to extract more detail than is available, yet is attempting to keep the speed of conversion high. I've attached an unsharpened CaptureOne Pro V7.02 conversion (one of many possible conversions), at 400 percent (Nearest Neighbor interpolated) zoom.

Cheers,
Bart
Logged
== If you do what you did, you'll get what you got. ==

32BT

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 3095
    • Pictures
Re: RAW Conversion Snake Oil
« Reply #48 on: January 17, 2013, 11:01:06 am »

Hi Oscar,

To achieve a neutral grey, demosaicing is also necessary, because otherwise it would not be possible to know that it's neutral grey, or a subtle/pastel color, or detail afterall.
 
By using a different Rawconverter? These mazing and zipper artifacts are a by-product of a demosaicing algorithm that tries to extract more detail than is available, yet is attempting to keep the speed of conversion high. I've attached an unsharpened CaptureOne Pro V7.02 conversion (one of many possible conversions), at 400 percent (Nearest Neighbor interpolated) zoom.

Cheers,
Bart

LOL, you preaching to the choir now?

OP was proposing that awful looking colornoise is the result of demosaicing, and while he is right, it is actually exactly that subtle compromise between detail and noise that needs to be solved by the demosaic algorithm. Simply focussing on the colornoise and then pointing out "failures" on the side of the converter is not exactly useful. So, to counter the proposition of the conversion he offered, i simply point out where the opposite fails.


In order to make this discussion at least somewhat useful to the readers, I will answer your suggestions with some ridiculous counter-suggestions, just to get the obvious stated:

To achieve a neutral grey, demosaicing is also necessary, because otherwise it would not be possible to know that it's neutral grey, or a subtle/pastel color, or detail afterall.

To achieve neutral gray one simply has to equalize the RGB values for each pixel. Certainly that does not require demosaicing.


By using a different Rawconverter? These mazing and zipper artifacts are a by-product of a demosaicing algorithm that tries to extract more detail than is available, yet is attempting to keep the speed of conversion high.

Seems more like a byproduct of not doing any conversion. Which would indeed provide more natural noise characteristics, but doesn't particularly help the look of normal detail in a file.

For reference a conversion with a different converter attached, without any filtering whatsoever. Notice that the text and logo are quite eligible and sharp, but the colornoise in the bottle does look somewhat spiky. Not anything that a reasonable RAW converter can not tackle, even while maintaining sharpness overall, but it does look unnatural.



Logged
Regards,
~ O ~
If you can stomach it: pictures

Fine_Art

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1172
Re: RAW Conversion Snake Oil
« Reply #49 on: January 17, 2013, 12:13:19 pm »

Thanks for pointing out the checker-boarding at 300%, I never noticed it.

Your attachment at 200% has significantly more noise than the jpg I provided. See side by side attached.

I was able to get the best detail from Raw Therapee with it's amaze de-bayer. It looked a hair crisper than the output from IPlus. Lines in the color wheel we a hair thinner. The noise when everything was turned off was basically the same as IPlus. The default looked quite a bit worse with lots of black peppering on the dark areas. With NR and sharpening it could look like Iplus without the noise. Its a fairly impressive program. Yes, RT is color managed.

Logged

Fine_Art

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1172
Re: RAW Conversion Snake Oil
« Reply #50 on: January 17, 2013, 12:18:37 pm »

Wake up and smell the irrelevance of this discussion.

You remind me of the guy in High School who couldn't stop talking about the "rotary" engine in his Mazda RX7. 

Why do you keep jumping into this thread that you have no interest in just to denigrate it?
Provide a comparison, show I am mistaken, do something useful or stay out.
Logged

Fine_Art

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1172
Re: RAW Conversion Snake Oil
« Reply #51 on: January 17, 2013, 12:51:47 pm »

Here is RT with everything turned off. It seems a bit crisper, a bit more artificial. Of the 2 for non-pixel peeping, I prefer Images Plus.

So the point of the thread, ISO 3200 noise, what everyone tries to avoid like plague, is not the shot noise everyone thinks it is. It is RAW converter noise.
Logged

jeremypayne

  • Guest
Re: RAW Conversion Snake Oil
« Reply #52 on: January 17, 2013, 01:39:27 pm »

Why do you keep jumping into this thread that you have no interest in just to denigrate it?
Provide a comparison, show I am mistaken, do something useful or stay out.

Because you make silly and provocative remarks that caught my attention.

Poke and get poked ... law of the jungle ...
Logged

32BT

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 3095
    • Pictures
Re: RAW Conversion Snake Oil
« Reply #53 on: January 17, 2013, 01:51:54 pm »

Here is RT with everything turned off. It seems a bit crisper, a bit more artificial. Of the 2 for non-pixel peeping, I prefer Images Plus.

So the point of the thread, ISO 3200 noise, what everyone tries to avoid like plague, is not the shot noise everyone thinks it is. It is RAW converter noise.

No, the color noise as depicted in my previous picture is the color noise that is actually captured in the file. The fact that your raw converters do not show the noise merely means they might be applying some base-line noise mitigation. The noise is real, you should generally want to avoid it as much as possible. The luminance noise that is also part of the problem is much finer grained, and generally not as objectionable as the color splotches. So, a conversion might look like attached image, were colornoise is removed, but luminance noise isn't.

But this really is very basic knowledge, and you should possibly try to read up on the information available all over the internuts before trying to make provocative and false statements in a forum like this in order to learn about the issues involved.





 
Logged
Regards,
~ O ~
If you can stomach it: pictures

Fine_Art

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1172
Re: RAW Conversion Snake Oil
« Reply #54 on: January 17, 2013, 02:09:49 pm »

No, the color noise as depicted in my previous picture is the color noise that is actually captured in the file. The fact that your raw converters do not show the noise merely means they might be applying some base-line noise mitigation. The noise is real, you should generally want to avoid it as much as possible. The luminance noise that is also part of the problem is much finer grained, and generally not as objectionable as the color splotches. So, a conversion might look like attached image, were colornoise is removed, but luminance noise isn't.

But this really is very basic knowledge, and you should possibly try to read up on the information available all over the internuts before trying to make provocative and false statements in a forum like this in order to learn about the issues involved.

 

It's amazing to me you would claim a scientific program is doing "baseline noise reduction". The color noise you presented is not captured in the file. It is an artificial pattern. It is not random at all. By providing a cleaned copy with NR and sharpening all you show is difference from the JPG I provided.

The very strong red and green color noise you showed is only there to make the NR algorithm look good. That noise is not in the picture. Real noise would be random or lines or an amp glow spot from heat in the camera.

That pattern is very artificial. I can create something similar by applying smoothing to channels. Even then I don't think I can make it look that strong from that file.

I learned from a very young age that the truth always wins in the end. There are people on this board with access to other scientific imaging programs that can verify your claims and mine.

One thing is for sure, any wedding photographers who see the ability to shoot at high ISO with minimal noise and strong detail on whites and clean blacks are going to go for that software. If a software puts noise in at baseline just to take it out with other sliders its a game.
Logged

Fine_Art

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1172
Re: RAW Conversion Snake Oil
« Reply #55 on: January 17, 2013, 07:31:03 pm »

I added noise reduction to my own file. I don't believe any of the commercial raw converters make a file look this natural, this sharp and this noise free from that ISO3200 file.

http://www.sendspace.com/file/csicku
That is the end of it as far as I am concerned. I have never seen ISO3200 look like ISO200 before.

I normally shoot at 100 or 200, 400 in a pinch, 800 If I have to. Now that I understand how to get the noise out I feel free to go Up to 3200 as needed.

Added an oversharpened screenshot. This was taken to the point artifacts are building up.
« Last Edit: January 17, 2013, 08:55:34 pm by Fine_Art »
Logged

32BT

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 3095
    • Pictures
Re: RAW Conversion Snake Oil
« Reply #56 on: January 18, 2013, 05:36:13 am »

Here are 2 interesting examples.

Example 1 compares:
1. RAW data samples in perceptual grayscale as provided by file
2. Nearest Neighbor duplication of the samples in color

Example 2 compares:
2. Nearest Neighbor duplication of the samples in color
3. Demosaic samples without any filtering

All files are processed using the same tone response which has auto-adjusted the source samples for exposure and brightness, that is: samples are distributed over the entire scale and the average perceptual brightness is at 50%. This may brighten the darktones more than you are used to seeing in your other RAW converters, which then makes colornoise more visible than what you are used to. Either way, the samples are what they are, if you look closely at the RT samples you posted, they also show the exact same colornoise, just buried in darkness, but it is there.

RAW data was read from file using DCRaw. The engines used to do conversion are proprietary, they do not use DCRaw. Since DCRaw is one of the most dense and most badly written code ever, in the entire history of software, I will not vouch for its accuracy, but since most of the academic world and scientific programs out there use it to read RAW files, I do not believe the data would be much different from those software options. (Note that I highly regard Dave's initiative as a monumental achievement for all kinds of reasons, but the coding style is simply not one of those reasons.)

Make of it what you will. If you're happy with your own way of processing, by all means use it. Other people use other methods, and use other RAW converters, not only for the processing, but for the entire production chain those converters may offer.
Logged
Regards,
~ O ~
If you can stomach it: pictures

Bart_van_der_Wolf

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 8913
Re: RAW Conversion Snake Oil
« Reply #57 on: January 18, 2013, 11:07:08 am »

To achieve neutral gray one simply has to equalize the RGB values for each pixel. Certainly that does not require demosaicing.

Hi Oscar,

And therein lies part of the misunderstanding. In an ideal noisefree CGI image, it is possible to locally neutralize to gray, but that is utterly impossible in real life. Photon shot noise, with some added read noise, is going to add color to sensels, even if the subject being photographed is perfectly gray. The noise added to a Red filtered sensel is independent from the noise added to the neighboring Green or Blue sensel, therefore the 'color' (signal level) will be different even from the other nearby Red filtered sensels.

Since neither the sensor array nor the demosaicing software recognize 'neutral', the demosaiced (and even the non-demosaiced) color balancing will result in colored noise around the average gray level. What's more, because Red and Blue are sampled at a lower density than Green, the interpolation result for Red and Blue will have a lower spatial frequency than the original photon noise (and a lower amplitude). How much lower depends on how the full color reconstruction is tuned between Luminance and Chrominance.

It is that inevitable lower spatial frequency (coming from interpolation) that is mistaken by the OP for smearing of color as if there is a targeted attempt to suppress noise. Of course, the separate operation of noise reduction does attempt to reduce the noise at specific spatial frequencies, but it usually also masks edges and other high spatial frequency areas from such noise reduction.

Cheers,
Bart
Logged
== If you do what you did, you'll get what you got. ==

Fine_Art

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1172
Re: RAW Conversion Snake Oil
« Reply #58 on: January 18, 2013, 11:57:54 am »

That sounds like a reasonable explanation Bart.

Keep in mind what put me on this was not the converter Oscar is using (whatever that is) it was see attached. When I moved the noise sliders all the way left sections of red/green color noise showed up that was >than 10 pixels across. That was in all dark areas of the file. See attached. Now, you could argue  that all the way left on the sliders is not 0 NR but noise enhancement for artistic use. I don't think anyone would buy that.

If the color noise is close to being at the pixel level, I would call it random to the best ability of the raw converter. When the red green starts to get bigger, especially taking on a directional component (repeating up/down, left/right), I would say it is not noise, it is an artificial construct of the raw converter. If you look at the sample provided further up the thread with strong color noise in shadows zoomed in it takes on that pattern.

Logged

Fine_Art

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1172
Re: RAW Conversion Snake Oil
« Reply #59 on: January 18, 2013, 12:09:17 pm »

This is a zoom of the noise form the other poster above. There is a very clear up/down, left/right component to it.
Logged
Pages: 1 2 [3] 4   Go Up