Pages: [1] 2   Go Down

Author Topic: What info can be lost once you have completed raw conversion?  (Read 22892 times)

ashaughnessy

  • Jr. Member
  • **
  • Offline Offline
  • Posts: 62
    • My wordpress blog
What info can be lost once you have completed raw conversion?
« on: December 04, 2012, 02:25:34 am »

I'm using the raw converter that came with my pentax k20d. I don't do any adjustments in there because it isn't very good. I do all my adjustments in Picture Window Pro once the raw file has been converted to tiff.

I recently read about highlight recovery. This makes it clear that there is information in the raw file that can only be recovered by the raw conversion process and that is then lost or not available once the file has been converted to tiff.

Apart from this highlight information, is there anything else that can only be done on the raw file and which would be lost/unavailable once you've moved to tiff? What can good raw converters do that a good photo editor simply cannot? I was happy with my toolset but now at least I realise I'm missing some of the highlight information that is in my raw files but not in the converted tiffs.

Thanks
Anthony
Logged
Anthony Shaughnessy
https://anthonyshaug

Tony Jay

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2965
Re: What info can be lost once you have completed raw conversion?
« Reply #1 on: December 04, 2012, 02:51:18 am »

Hi Anthony.
Really the answer to your question is: do as much as possible with the RAW file.
This pertains to tonal manipulation, colour manipulation, capture sharpening (really creative sharpening and output sharpening as well), noise reduction, lens profile corrections, softproofing, and printing.

Most images for example can have all these things done to them optimally as RAW images from within Lightroom. Occasionally for one reason or another rendering the image as TIFF or DNG may be required but this is rarely required.

I may be wrong but it appears that you do not own Lightroom - my simple advice is to buy it and use it. You will never look back.

Tony Jay
Logged

ashaughnessy

  • Jr. Member
  • **
  • Offline Offline
  • Posts: 62
    • My wordpress blog
Re: What info can be lost once you have completed raw conversion?
« Reply #2 on: December 04, 2012, 03:06:28 am »

Hi Tony, you're correct that I don't own Lightroom but my point is really about what is simply impossible to do once you've left raw. For example, I can't believe there is anything you can do with regard to printing from the raw file that couldn't just as well be done from a tiff file, and choice of printing application would then come down to personal preference. Whereas with highlight recovery, that can only be done to the raw file.
Thanks
Anthony
Logged
Anthony Shaughnessy
https://anthonyshaug

Petrus

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 952
Re: What info can be lost once you have completed raw conversion?
« Reply #3 on: December 04, 2012, 03:35:55 am »

When you make the RAW conversion you basically only take a slice of the data contained in the RAW file, and thus loose part of the color and luminance adjustment possibilities available there. Thus all further adjustments in PS are restricted to that information content and can be only subtractive. For this reason all basic adjustments should be done in RAW, before conversion. Lightroom 4.2 is an amazing tool for this, I really have stopped using Photoshop almost totally, only if I need local blurring etc. Better results with LR 4.2, faster.   
Logged

NikoJorj

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1082
    • http://nikojorj.free.fr/
Re: What info can be lost once you have completed raw conversion?
« Reply #4 on: December 04, 2012, 04:57:11 am »

my point is really about what is simply impossible to do once you've left raw.
Impossible : restoring crushed blacks or blown whites (this actually includes moderate-to-heavy white balance changes, as it changes tonal values of individual color channels)
Possible but with (sometimes much) lesser results : noise reduction, sharpening, correcting chromatic aberrations...

Do yourself a favor and try LR4! ;)
« Last Edit: December 04, 2012, 04:59:07 am by NikoJorj »
Logged
Nicolas from Grenoble
A small gallery

Tony Jay

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2965
Re: What info can be lost once you have completed raw conversion?
« Reply #5 on: December 04, 2012, 07:47:26 am »

...Do yourself a favor and try LR4!

Absolutely.
Other heavyweights will weigh in no doubt on the absolute advantages of doing noise reduction and sharpening with a RAW file.

Tony Jay
Logged

hjulenissen

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2051
Re: What info can be lost once you have completed raw conversion?
« Reply #6 on: December 04, 2012, 08:00:51 am »

On a more practical level, given limited time and resources, certain operations are fairly simple and predictable (such as white-balancing a linear raw image with an accurate camera profile). It can be as simple as clicking a "white" area in the image, and the result may look good to you.

On a jpeg, the process generating the image is far more unpredictable (although I have seen Lightroom bake in its conversion settings in jpeg files!). You may be able to tweak sliders until you end up with something "pleasing", but anything related to the true scene in a physical way may be hard.

-h
Logged

ErikKaffehr

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 11311
    • Echophoto
Re: What info can be lost once you have completed raw conversion?
« Reply #7 on: January 08, 2013, 05:03:44 pm »

Hi,

It depends. To begin with TIFF can either be 16 bits or 8 bits. 16 bit TIFFs may contain same information as RAW. In raw conversion you would typically set black and white points, effectively throwing away data below and above black point. Most cameras handle 12-14 bits of data, so going to 24-bit TIFF (8 bits per channel) looses 4-6 bits of data. That means that you are throwing away about 94% to 98% of the data for each channel. As long you choose what remains well, it matters little.

The raw data is linear but converted data is going to be logarithmic, making corrections in post much less predictible. Normally, an "S-curve" will also be applied to the image in raw conversion, enhancing mid tone contrast.

Best regards
Erik

I'm using the raw converter that came with my pentax k20d. I don't do any adjustments in there because it isn't very good. I do all my adjustments in Picture Window Pro once the raw file has been converted to tiff.

I recently read about highlight recovery. This makes it clear that there is information in the raw file that can only be recovered by the raw conversion process and that is then lost or not available once the file has been converted to tiff.

Apart from this highlight information, is there anything else that can only be done on the raw file and which would be lost/unavailable once you've moved to tiff? What can good raw converters do that a good photo editor simply cannot? I was happy with my toolset but now at least I realise I'm missing some of the highlight information that is in my raw files but not in the converted tiffs.

Thanks
Anthony
Logged
Erik Kaffehr
 

Fine_Art

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1172
Re: What info can be lost once you have completed raw conversion?
« Reply #8 on: January 13, 2013, 01:22:05 pm »

When you make the RAW conversion you basically only take a slice of the data contained in the RAW file, and thus loose part of the color and luminance adjustment possibilities available there. Thus all further adjustments in PS are restricted to that information content and can be only subtractive. For this reason all basic adjustments should be done in RAW, before conversion. Lightroom 4.2 is an amazing tool for this, I really have stopped using Photoshop almost totally, only if I need local blurring etc. Better results with LR 4.2, faster.   

That is what everyone says. I believed it too until I found almost every manufacturer uses ISO 12234-2 tiff as their RAW format. They add lossless compression. They add a wrapper with info such as shot parameters (EXIF).

I'll say it again, RAWs are tiffs.
http://en.wikipedia.org/wiki/TIFF/EP

People have mistaken what is lost taking a 12 or 14 bit tif to 8 bit jpg as the "baking" process that also affects tifs. As Erik said, when you do raw conversion you add a 2.2 gamma log factor so that can lose data if you push it past 255. Otherwise working in tif is safe.

Logged

jeremypayne

  • Guest
Re: What info can be lost once you have completed raw conversion?
« Reply #9 on: January 13, 2013, 05:18:07 pm »

I'll say it again, RAWs are tiffs ... Otherwise working in tif is safe.

I think you have confused yourself into believing that because many RAW formats are technically proprietary TIFs that there is no data lost in the RAW conversion ...

When you render the data from a proprietary RAW into an image and then save it - even as an uncompressed 16-bit image file - you have crystalized some choices that cannot be un-done.
Logged

Tim Lookingbill

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2436
Re: What info can be lost once you have completed raw conversion?
« Reply #10 on: January 13, 2013, 05:25:56 pm »

Quote
People have mistaken what is lost taking a 12 or 14 bit tif to 8 bit jpg as the "baking" process that also affects tifs. As Erik said, when you do raw conversion you add a 2.2 gamma log factor so that can lose data if you push it past 255. Otherwise working in tif is safe.

And to be more specific so as not to confuse this 12 or 14 bit terminology I would like to remind everyone what exactly these bits are referring according to this Cambridge In Colour quote...

Quote
Even if one's digital camera could capture a vast dynamic range, the precision at which light measurements are translated into digital values may limit usable dynamic range. The workhorse which translates these continuous measurements into discrete numerical values is called the analog to digital (A/D) converter. The accuracy of an A/D converter can be described in terms of bits of precision, similar to bit depth in digital images, although care should be taken that these concepts are not used interchangeably. The A/D converter is what creates values for the digital camera's RAW file format.

http://www.cambridgeincolour.com/tutorials/dynamic-range.htm

There's no such thing as a 12 or 14 bit tiff that is workable in an imaging editor after A/D conversion into 1's and 0's.
Logged

ashaughnessy

  • Jr. Member
  • **
  • Offline Offline
  • Posts: 62
    • My wordpress blog
Re: What info can be lost once you have completed raw conversion?
« Reply #11 on: January 15, 2013, 09:35:44 am »

The last four posts have confused things again.

Erik - you say data is lost if you set black and white points at raw conversion time. What if you don't? What if you simply demosaic without adjusting black and white points and move straight into an RGB TIFF image editor (e.g. Photoshop) ? Is any data lost in that case? Is there anything I can no longer do (i.e. impossible as opposed to just less convenient) ?

tlooknbill - I don't understand your last sentence in your last post at all.

Jeremy - when you go from RAW into an RGB TIFF, you may have crystallised some choices, but as long as they are the correct choices then what is the problem? If you make the wrong choice, you can just go back to the original raw file and start again. Being able to undo a change isn't usually something I want to do unless I've made a mistake, in which case I'll just try again. The point I'm asking about is - assuming I make the correct choices, what becomes impossible once I've moved from the raw converter into the RGB TIFF photo editor?

Thanks
Anthony
Logged
Anthony Shaughnessy
https://anthonyshaug

hjulenissen

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2051
Re: What info can be lost once you have completed raw conversion?
« Reply #12 on: January 15, 2013, 09:47:07 am »

Erik - you say data is lost if you set black and white points at raw conversion time. What if you don't? What if you simply demosaic without adjusting black and white points and move straight into an RGB TIFF image editor (e.g. Photoshop) ? Is any data lost in that case? Is there anything I can no longer do (i.e. impossible as opposed to just less convenient) ?
Most conversions from raw to "developed" would include at least:
1. Demosaic
2. Color correction and white balance
3. Choosing black point, white point
4. Sharpening, Denoising
5. Nonlinear tonecurve/gamma
6. Quantization to 8 bits
7. Matrixing to YCbCr and downsampling chroma channels
8. Lossy encoding to e.g. jpeg

Optionally highlight clipping recovery, lense corrections.

Most of these steps are (or can be) non-linear and difficult to reverse. Some are throwing away information and are therefore impossible to reverse.

I am sure that it would be _possible_ to disable most of those steps in order to make a jpeg (or tiff) as close as possible to a raw file. I am not sure that it makes much sense, though. The point of a raw file is to retain as much information possible about the signal coming from the camera sensor. The point of a developed file is usually to "look good". What is the point in trying to make a bastard that is neither?

The "colors" obtained if raw sensor channels (post demosaic) are routed directly to "r", "g", "b" channels looks unsaturated and dull. In order to make for something pleasing (and realistic), the raw devloper needs to apply some color transform that (among other things) tends to increase saturation. Unless you have precise knowledge of what it did, it can be hard to apply another transform (based on measurements of raw file behaviour) at a later point. The applied transform could also cause clipping in an image (even if the raw data were not clipped). In that case, it will be impossible to figure the original values.

The Demosaic process can be hard or impossible to invert. If an image scaling operation is "interpolatory", then original valules will be unchanged, only new ones will be inserted. If that is the case, you may be able to invert demosaic if you know the original sequence of "rggb" vs "bggr" etc, and the offset into the raw sensor data used for output image. If the process is non-interpolatory (i.e. changes pixels at both "known" and "unknown" data sites), it is going to be very hard to invert it.

In the case of closed-source and proprietary raw developers (such as internal camera jpeg generation, Adobe Lightroom,...) we dont even know exactly how the forward transformation works (i.e. what, mathematically, happens when I import a Canon 20D image and drag the exposure slider by "1.2 stops"), much less know how to best invert those processing steps.


And so on...

-h
« Last Edit: January 15, 2013, 09:59:07 am by hjulenissen »
Logged

ashaughnessy

  • Jr. Member
  • **
  • Offline Offline
  • Posts: 62
    • My wordpress blog
Re: What info can be lost once you have completed raw conversion?
« Reply #13 on: January 15, 2013, 10:11:39 am »

Thanks hjulenissen.
The point of my question is to work out if I am missing out on possibilities through my choice of raw converter. I've been using the Pentax raw conversion software as provided with my K20D followed  by Picture Window Pro for the picture editing. I've been doing the bare minimum in the Pentax software, just converting the raw DNG file into a TIFF that I can then process in PWP. Usually, the only thing I adjust in the Pentax software is the white balance (if it needs it). Or put more accurately, the only setting I adjust from the defaults is the white balance.

I can then do colour correction, contrast setting, black and white point setting, sharpening, noise removal, etc. in PWP, which has a far richer toolset than the pentax raw converter tool.

I can put up with inconveniences and I'm not so interested in discussions of which tool has the easier workflow, what I really want to know is whether I'm fundamentally missing out on processing possibilities by doing most of my work in PWP. The ability to recover highlights during demosaicing seems to be something that is fundamentally impossible after raw conversion but everything else seems to be a matter of choice as to whether I do it in the raw software or the picture editor.

Anthony
Logged
Anthony Shaughnessy
https://anthonyshaug

Petrus

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 952
Re: What info can be lost once you have completed raw conversion?
« Reply #14 on: January 15, 2013, 11:22:00 am »

I would say that do everything possible with the RAW converter, as all available adjustment leeway is still there. I.e. make the picture as good looking as possible, THEN do only those adjustments the RAW converter can not do with Photoshop or whatever. After starting to use Lightroom 4.3 I have almost stopped using Photoshop at all for most of my photos, and they look better now.
Logged

Fine_Art

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1172
Re: What info can be lost once you have completed raw conversion?
« Reply #15 on: January 15, 2013, 01:08:40 pm »

Hi Tony, you're correct that I don't own Lightroom but my point is really about what is simply impossible to do once you've left raw. For example, I can't believe there is anything you can do with regard to printing from the raw file that couldn't just as well be done from a tiff file, and choice of printing application would then come down to personal preference. Whereas with highlight recovery, that can only be done to the raw file.
Thanks
Anthony

According to Vlad recovery is created data from surrounding areas. It has nothing to do with recovering the data.

no it does not... in a typical raw converter "speak" recovery is not about "data is there" ( in a region with clipped raw channel(s) ), recovery is about postprocessing to paint (note that it is not a part of the raw conversion exactly) that part (where you have clipping) of an image using the data from unclipped raw channels and/or the data from surrounding areas in that image to make that area (where you have clipping) suitable/acceptable to your intended visual objective
Logged

ashaughnessy

  • Jr. Member
  • **
  • Offline Offline
  • Posts: 62
    • My wordpress blog
Re: What info can be lost once you have completed raw conversion?
« Reply #16 on: January 15, 2013, 01:26:25 pm »

Petrus - lots of people say as you say, but why? It sounds like the similar advice given about scanning in the "olden days" - do all your adjustments in the scanner software. I think this was mostly because photoshop in those days was mostly limited to 8-bits, but the myth hung around for much longer even when photo editors had better support for 16 bit files.

Fine-art - Vladimirovich is, I'm sure, correct in what he says, but it doesn't change the argument. I believe that the raw processor can recover/paint highlights to "unclip" them by using information that only exists in the raw file (by examing the raw luminance information in adjacent pixels). Once the raw data has been converted, the photo editor will not be able to do this "painting". Hence, highlight "recovery" as performed by Camera Raw, can only be done during the raw conversion phase and cannot be done afterwards (I think).

Anthony
Logged
Anthony Shaughnessy
https://anthonyshaug

Fine_Art

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1172
Re: What info can be lost once you have completed raw conversion?
« Reply #17 on: January 15, 2013, 01:44:14 pm »

Petrus - lots of people say as you say, but why? It sounds like the similar advice given about scanning in the "olden days" - do all your adjustments in the scanner software. I think this was mostly because photoshop in those days was mostly limited to 8-bits, but the myth hung around for much longer even when photo editors had better support for 16 bit files.

Fine-art - Vladimirovich is, I'm sure, correct in what he says, but it doesn't change the argument. I believe that the raw processor can recover/paint highlights to "unclip" them by using information that only exists in the raw file (by examing the raw luminance information in adjacent pixels). Once the raw data has been converted, the photo editor will not be able to do this "painting". Hence, highlight "recovery" as performed by Camera Raw, can only be done during the raw conversion phase and cannot be done afterwards (I think).

Anthony

I bet Vlad is right as well. The point is that if it is "filling" clipped areas from surrounding areas you can do the same. It is not something special about RAW conversion. Its just a matter of how good the content fill routine is.

There are 4 RAW converters that most people feel are the best. In alpha order Capture One, DxO, Lightroom, Raw Therapee. You should try demos of those programs. What you seem to be asking is how limited are you by the software Picture Window Pro that you are using. You will have to compare.

There are 2 steps in RAW conversion that are significant. De-Bayer and Gamma shift. The rest is mostly transforms that can be reversed. What you manipulate in a raw converter is probably before the gamma log is applied. If you use 16 bit tif you probably keep all your data. You have no way of reversing changes because you dont know the exact values of manipulations done. Math functions on the data would be reversible otherwise. The only thing you cant reverse is clipping the data or overwriting the data.
Logged

digitaldog

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 20614
  • Andrew Rodney
    • http://www.digitaldog.net/
Re: What info can be lost once you have completed raw conversion?
« Reply #18 on: January 15, 2013, 02:14:19 pm »

The point of a raw file is to retain as much information possible about the signal coming from the camera sensor. The point of a developed file is usually to "look good". What is the point in trying to make a bastard that is neither?

Beats me. You can always go back to the raw with differing converters. The rendered image is what counts. The question becomes, what is better (in terms of quality), faster and perhaps easier: to develop as much as possible with the raw converter or afterwards in say Photoshop? I know what I prefer...
Logged
http://www.digitaldog.net/
Author "Color Management for Photographers".

Tim Lookingbill

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2436
Re: What info can be lost once you have completed raw conversion?
« Reply #19 on: January 15, 2013, 02:35:12 pm »

Quote
tlooknbill - I don't understand your last sentence in your last post at all.

I thought the Cambridge In Colour link would've made it clear. That site is the best at explaining highly technical concepts broken down into concise but simplified levels so non-physist types can understand it.

The bit numbers 12 & 14 that are mentioned in this thread is in regard to precision of what data off the sensor in the form of voltage readings that define the value of each pixel site is kept and thrown away (as usable detail) and converted to 1's & 0's by way of the A/D converter. How many luminance steps of RGGB are devoted to making up detail from every full saturation pixel site across all levels of charged pixel sites down to no electrical charge/signal at all=(black) before noise overtakes/fogs up perceptible dynamic range.

IOW how finely does the electronics and software want to slice up usable detail in defining it? 12 bit precision?=sharp knife or 14 bit?=even sharper. And then throw out what's not usable detail within the A/D converter.

What's left in the form of a raster x/y coordinate plot for each RGGB pixel luminance reading has already been determined by the time it reaches the software that has to demosaic those raster plots into a viewable image using an 8 bit video preview.

In the case of ACR/LR the 16bit editing space interpolates up the existing 12 or 14 bit precise culling of demosaiced data so the ACR/LR tools used to edit that data has added precision in mapping a smoother response editing demosaiced pixel data in an 8 bit video preview environment.

This is what confuses people about bit depth and what is touched upon in that link I provided.
Logged
Pages: [1] 2   Go Up