RAW Conversion Snake Oil

[Fine_Art]

Yesterday I stumbled upon a new raw converter. The default conversion looked pretty good.

I abused it relative to what the manufacturer expected by moving all the NR sliders to zero. I expected this to show a 'raw' RAW. I ended up with a mess of color blotches. This made me realize how much of the raw conversion is interpretation. I went for a 'real raw' RAW with the purest software I know. It was developed by a mathematician at a US university for amateur astronomy. What I found is the noise in a high ISO shot is nowhere near as objectionable as it normally appears in raw conversions. The raw converters themselves actually create the ugliness by trying to smooth away the noise! Yes, even chrominance noise looks fine when it is even at the pixel level. It is when it is grouped into small areas that it looks like hell. I'm not saying you dont want to remove it, I am saying if you go to print, pixel level color noise will vanish. Even the softness of the file is largely CREATED by trying to smooth colors away.

Attached is a real raw RAW.

I have added gamma 2.2, White balance yellow +2000 vs blue (it was a sun setting so a more golden hr look is correct)

I have also used the software to split each color from luminance. Note the heavy noise in red and green. Of course I then had the bright idea to do various types of smoothing on those channels so I could recombine with noise wiped out!

Of course I end up exactly where the RAW converters do with a soft looking image! Not only that but the smoothing of color noise per channel did not evenly line up. Surprise surprise I end up with the small color blotches we see on our raw conversions before further NR.

What am I doing here? I found I was pulling back the curtain on the Wizard of Oz. At the pixel level the RAW tif (yes RAWS are compressed tifs) can only have a luminance value and a color value. All the countless sliders you see in software con only manipulate those things. Keep that in mind.

[Fine_Art]

Here is the separated color channels.

[Fine_Art]

Then a recombined smoothed file. Does the mush of the detail look like what drives everyone to buy more expensive lenses? It is created by the software. Look at the detail in the original RAW above.

The multiple posts are from the attachment size limit.

[jeremypayne]

What am I doing here?

Chasing shadows of your tail?

[Fine_Art]

Chasing shadows of your tail?

You work for a RAW software provider?

[jeremypayne]

You work for a RAW software provider?

Nope.

[woos]

It's rather true lol.  Open up raw therapee and zero everything out, turn all NR off, etc.  Note what the noise looks like.  Then do the same in camera raw or lightroom.  Hmmmm...^_^  Now, floating point may be overkill, but on the other hand, maybe not. lol

[Fine_Art]

Nope.

In this other post you seem to be trying to make a mythology for a product.

Lightroom is on v4.x

Lightroom is beyond impressive to me ... it the single most impressive pieces of "consumer" software I have ever seen.  The ambition and "game-changing" nature of the project is inspiring to me as someone who lives in "legacy" world trying to innovate.

The story of Lightroom is a an amazing story.

[jeremypayne]

In this other post you seem to be trying to make a mythology for a product.

You are accusing me of what?  Pretending to admire the company as some kind of paid shill?

If so, you've lost your mind.

[Fine_Art]

You are accusing me of what?  Pretending to admire the company as some kind of paid shill?

If so, you've lost your mind.

I asked if you work for a related company. Your post did seem to gush excessively. Keyword asked.

[Fine_Art]

After using various noise and sharpening techniques I have a reasonable 50% crop out of the high ISO file. It has less sharpness and less noise that the RAW. If you have a very pure raw converter and plenty of pixels the file will look as good as a heavily processed file. "You can't turn a sow's ear into a silk purse." I could process this file with "Capture sharpening" techniques. It seems a joke knowing the sharpness of the original RAW.

Of course the real picture is a 5 wide x 2 high pano at ISO 100 not this camera test file.

[Tim Lookingbill]

What am I doing here? I found I was pulling back the curtain on the Wizard of Oz. At the pixel level the RAW tif (yes RAWS are compressed tifs) can only have a luminance value and a color value. All the countless sliders you see in software con only manipulate those things. Keep that in mind.

Keep this in mind. It's called a converter for a reason. It converts data from one state to another. It's all raster data. You want to call it a tiff? Call it tiff. It's pointless because a tiff is nothing but a raster mapping of data. Your OS video system's frame buffer is a raster of data as well. Do you want to call your video card a raster tiff generator? What's the point?

You know what raster means, right? You have an X/Y table mapping within a 2:3/3:4 ratio rectangle of every tiny square pixel of various luminance, the variation determined by the RGGB filtering of photons collected to create an electrical charge. That charge is assigned a voltage measurement. IT'S NOT EVEN A SQUARE PIXEL OF COLOR at this stage. A/D converter takes those charges that represent a luminance level (some of those are clumps of noise) and converts it to 1's and 0's where IT'S STILL NOT A PIXEL OR TIFF.

It's all interpreted by software! You are looking at a concoction of software that happens to generate an image that pleases you. You've just decided to pick what stage of interpretation is the most true according to what you see on your display.

That texture of noise I see in your first sample close up crop I've seen duplicated in ACR's Grain panel just by twiddling sliders long enough to find it.

[Tim Lookingbill]

There's another thing about ACR compared to your sample image in that ACR/LR's color engine and tools that act on the gradation of the interpreted raster data offer better color grading and variance that optically delivers more pleasing color.


I've seen this nuance in color gradation which to pull this off I'm assuming requires very sophisticated algorithms engineered not only in the color engine and the tools that act on each pixel but also in how it integrates with my OS video system's rasterized frame buffer and color managed previews that maps those color edits very smoothly.

What I see in your sample image is a lot of monotone/monochrome color gradation that effectively squelches the amount of color detail I know is in that type of scene. It looks very sharp but I know sawgrass has far more variation of beige's, browns and yellows I don't see in that sample. I've seen other Raw converters render color similarly to get that very sharp look but at the sacrifice of color detail which is inextricably tied to color noise especially in under exposed shots or shot with very little light like your sample.

[Vladimirovich]

What am I doing here?

repeating many posting on the same subject here from the days past...

[Fine_Art]

repeating many posting on the same subject here from the days past...

That would not surprise me.

[Fine_Art]

There's another thing about ACR compared to your sample image in that ACR/LR's color engine and tools that act on the gradation of the interpreted raster data offer better color grading and variance that optically delivers more pleasing color.


I've seen this nuance in color gradation which to pull this off I'm assuming requires very sophisticated algorithms engineered not only in the color engine and the tools that act on each pixel but also in how it integrates with my OS video system's rasterized frame buffer and color managed previews that maps those color edits very smoothly.

What I see in your sample image is a lot of monotone/monochrome color gradation that effectively squelches the amount of color detail I know is in that type of scene. It looks very sharp but I know sawgrass has far more variation of beige's, browns and yellows I don't see in that sample. I've seen other Raw converters render color similarly to get that very sharp look but at the sacrifice of color detail which is inextricably tied to color noise especially in under exposed shots or shot with very little light like your sample.

I did not mention a particular brand of software. I have noticed the issue I am talking about with several packages not the ACR/LR some want to champion.

There are several features in a raw converter that a straight conversion will fail on. Color fringing on highlights. These are well controlled in LR, RT and several other packages. Distortion correction. Vignetting. Any algorithms that manipulate groups of pixels for whatever reason.

Back to what I was talking about - clumps of colored pixels are purely a failure of the de-bayering. You either leave an area showing colors at individual pixels which zoomed out blend to the proper color or you have a color at the pixel level that is really in the scene. Clumps of red and green are not in the scene.

Doing a straight conversion of a low ISO image yields worse results than the various converter packages. On that they are fine. This is a problem specific to handling high ISO noise by blending areas.

[Fine_Art]

I looked up your sawgrass, it is not here.

That is either hay stalks or wheat stalks. And by stalks I mean it has been combine harvested.

Here is a wiki picture of hay.


Any variation you can see up close will not show up at 1/2 a km. So no magic software is going to find it.

[Tim Lookingbill]

The cropped close up image you posted has a ton of clumped color noise as well as limited color detail. It's not a pleasing looking image viewing at 100%. The lack of color detail gradations even shows in the separate RGB channel samples.

I'ld have to say that is a poor sample image to be demonstrating a converter's handling of noise. Or I'm at a loss on what you're trying to show us with that image. I don't see anything revealing in that image that is useable information.

I used the term "sawgrass" to describe the short grass I see covering the rolling plains. It's all one monochrome color.

I can tell you I've gotten better looking images with regard to noise and detail appearance shooting at 1/1250 sec., f/8, ISO 800 (the exposure of your posted image). That exposure indicates there was quite a bit of light that should've given less noise at ISO 800 than what's seen in your image sample.

[Tim Lookingbill]

Just found out it's a Sony SLT A55v using a 16MP APS-C sensor. That explains the image quality. ImagesPlus is the converter used.

Their website indicates its use in astronomy.

 

[jeremypayne]

At the pixel level the RAW tif (yes RAWS are compressed tifs) can only have a luminance value and a color value. All the countless sliders you see in software con only manipulate those things. Keep that in mind.

Can you please illuminate the consequence of this conclusion?

I'm not sure I understand what you are trying to share with us here ...