Luminous Landscape Forum

Raw & Post Processing, Printing => Other Raw Converters => Topic started by: ashaughnessy on December 04, 2012, 02:25:34 AM

Title: What info can be lost once you have completed raw conversion?
Post by: ashaughnessy on December 04, 2012, 02:25:34 AM
I'm using the raw converter that came with my pentax k20d. I don't do any adjustments in there because it isn't very good. I do all my adjustments in Picture Window Pro once the raw file has been converted to tiff.

I recently read about highlight recovery. This makes it clear that there is information in the raw file that can only be recovered by the raw conversion process and that is then lost or not available once the file has been converted to tiff.

Apart from this highlight information, is there anything else that can only be done on the raw file and which would be lost/unavailable once you've moved to tiff? What can good raw converters do that a good photo editor simply cannot? I was happy with my toolset but now at least I realise I'm missing some of the highlight information that is in my raw files but not in the converted tiffs.

Thanks
Anthony
Title: Re: What info can be lost once you have completed raw conversion?
Post by: Tony Jay on December 04, 2012, 02:51:18 AM
Hi Anthony.
Really the answer to your question is: do as much as possible with the RAW file.
This pertains to tonal manipulation, colour manipulation, capture sharpening (really creative sharpening and output sharpening as well), noise reduction, lens profile corrections, softproofing, and printing.

Most images for example can have all these things done to them optimally as RAW images from within Lightroom. Occasionally for one reason or another rendering the image as TIFF or DNG may be required but this is rarely required.

I may be wrong but it appears that you do not own Lightroom - my simple advice is to buy it and use it. You will never look back.

Tony Jay
Title: Re: What info can be lost once you have completed raw conversion?
Post by: ashaughnessy on December 04, 2012, 03:06:28 AM
Hi Tony, you're correct that I don't own Lightroom but my point is really about what is simply impossible to do once you've left raw. For example, I can't believe there is anything you can do with regard to printing from the raw file that couldn't just as well be done from a tiff file, and choice of printing application would then come down to personal preference. Whereas with highlight recovery, that can only be done to the raw file.
Thanks
Anthony
Title: Re: What info can be lost once you have completed raw conversion?
Post by: Petrus on December 04, 2012, 03:35:55 AM
When you make the RAW conversion you basically only take a slice of the data contained in the RAW file, and thus loose part of the color and luminance adjustment possibilities available there. Thus all further adjustments in PS are restricted to that information content and can be only subtractive. For this reason all basic adjustments should be done in RAW, before conversion. Lightroom 4.2 is an amazing tool for this, I really have stopped using Photoshop almost totally, only if I need local blurring etc. Better results with LR 4.2, faster.   
Title: Re: What info can be lost once you have completed raw conversion?
Post by: NikoJorj on December 04, 2012, 04:57:11 AM
my point is really about what is simply impossible to do once you've left raw.
Impossible : restoring crushed blacks or blown whites (this actually includes moderate-to-heavy white balance changes, as it changes tonal values of individual color channels)
Possible but with (sometimes much) lesser results : noise reduction, sharpening, correcting chromatic aberrations...

Do yourself a favor and try LR4! ;)
Title: Re: What info can be lost once you have completed raw conversion?
Post by: Tony Jay on December 04, 2012, 07:47:26 AM
...Do yourself a favor and try LR4!

Absolutely.
Other heavyweights will weigh in no doubt on the absolute advantages of doing noise reduction and sharpening with a RAW file.

Tony Jay
Title: Re: What info can be lost once you have completed raw conversion?
Post by: hjulenissen on December 04, 2012, 08:00:51 AM
On a more practical level, given limited time and resources, certain operations are fairly simple and predictable (such as white-balancing a linear raw image with an accurate camera profile). It can be as simple as clicking a "white" area in the image, and the result may look good to you.

On a jpeg, the process generating the image is far more unpredictable (although I have seen Lightroom bake in its conversion settings in jpeg files!). You may be able to tweak sliders until you end up with something "pleasing", but anything related to the true scene in a physical way may be hard.

-h
Title: Re: What info can be lost once you have completed raw conversion?
Post by: ErikKaffehr on January 08, 2013, 05:03:44 PM
Hi,

It depends. To begin with TIFF can either be 16 bits or 8 bits. 16 bit TIFFs may contain same information as RAW. In raw conversion you would typically set black and white points, effectively throwing away data below and above black point. Most cameras handle 12-14 bits of data, so going to 24-bit TIFF (8 bits per channel) looses 4-6 bits of data. That means that you are throwing away about 94% to 98% of the data for each channel. As long you choose what remains well, it matters little.

The raw data is linear but converted data is going to be logarithmic, making corrections in post much less predictible. Normally, an "S-curve" will also be applied to the image in raw conversion, enhancing mid tone contrast.

Best regards
Erik

I'm using the raw converter that came with my pentax k20d. I don't do any adjustments in there because it isn't very good. I do all my adjustments in Picture Window Pro once the raw file has been converted to tiff.

I recently read about highlight recovery. This makes it clear that there is information in the raw file that can only be recovered by the raw conversion process and that is then lost or not available once the file has been converted to tiff.

Apart from this highlight information, is there anything else that can only be done on the raw file and which would be lost/unavailable once you've moved to tiff? What can good raw converters do that a good photo editor simply cannot? I was happy with my toolset but now at least I realise I'm missing some of the highlight information that is in my raw files but not in the converted tiffs.

Thanks
Anthony
Title: Re: What info can be lost once you have completed raw conversion?
Post by: Fine_Art on January 13, 2013, 01:22:05 PM
When you make the RAW conversion you basically only take a slice of the data contained in the RAW file, and thus loose part of the color and luminance adjustment possibilities available there. Thus all further adjustments in PS are restricted to that information content and can be only subtractive. For this reason all basic adjustments should be done in RAW, before conversion. Lightroom 4.2 is an amazing tool for this, I really have stopped using Photoshop almost totally, only if I need local blurring etc. Better results with LR 4.2, faster.   

That is what everyone says. I believed it too until I found almost every manufacturer uses ISO 12234-2 tiff as their RAW format. They add lossless compression. They add a wrapper with info such as shot parameters (EXIF).

I'll say it again, RAWs are tiffs.
http://en.wikipedia.org/wiki/TIFF/EP

People have mistaken what is lost taking a 12 or 14 bit tif to 8 bit jpg as the "baking" process that also affects tifs. As Erik said, when you do raw conversion you add a 2.2 gamma log factor so that can lose data if you push it past 255. Otherwise working in tif is safe.

Title: Re: What info can be lost once you have completed raw conversion?
Post by: jeremypayne on January 13, 2013, 05:18:07 PM
I'll say it again, RAWs are tiffs ... Otherwise working in tif is safe.

I think you have confused yourself into believing that because many RAW formats are technically proprietary TIFs that there is no data lost in the RAW conversion ...

When you render the data from a proprietary RAW into an image and then save it - even as an uncompressed 16-bit image file - you have crystalized some choices that cannot be un-done.
Title: Re: What info can be lost once you have completed raw conversion?
Post by: Tim Lookingbill on January 13, 2013, 05:25:56 PM
Quote
People have mistaken what is lost taking a 12 or 14 bit tif to 8 bit jpg as the "baking" process that also affects tifs. As Erik said, when you do raw conversion you add a 2.2 gamma log factor so that can lose data if you push it past 255. Otherwise working in tif is safe.

And to be more specific so as not to confuse this 12 or 14 bit terminology I would like to remind everyone what exactly these bits are referring according to this Cambridge In Colour quote...

Quote
Even if one's digital camera could capture a vast dynamic range, the precision at which light measurements are translated into digital values may limit usable dynamic range. The workhorse which translates these continuous measurements into discrete numerical values is called the analog to digital (A/D) converter. The accuracy of an A/D converter can be described in terms of bits of precision, similar to bit depth in digital images, although care should be taken that these concepts are not used interchangeably. The A/D converter is what creates values for the digital camera's RAW file format.

http://www.cambridgeincolour.com/tutorials/dynamic-range.htm

There's no such thing as a 12 or 14 bit tiff that is workable in an imaging editor after A/D conversion into 1's and 0's.
Title: Re: What info can be lost once you have completed raw conversion?
Post by: ashaughnessy on January 15, 2013, 09:35:44 AM
The last four posts have confused things again.

Erik - you say data is lost if you set black and white points at raw conversion time. What if you don't? What if you simply demosaic without adjusting black and white points and move straight into an RGB TIFF image editor (e.g. Photoshop) ? Is any data lost in that case? Is there anything I can no longer do (i.e. impossible as opposed to just less convenient) ?

tlooknbill - I don't understand your last sentence in your last post at all.

Jeremy - when you go from RAW into an RGB TIFF, you may have crystallised some choices, but as long as they are the correct choices then what is the problem? If you make the wrong choice, you can just go back to the original raw file and start again. Being able to undo a change isn't usually something I want to do unless I've made a mistake, in which case I'll just try again. The point I'm asking about is - assuming I make the correct choices, what becomes impossible once I've moved from the raw converter into the RGB TIFF photo editor?

Thanks
Anthony
Title: Re: What info can be lost once you have completed raw conversion?
Post by: hjulenissen on January 15, 2013, 09:47:07 AM
Erik - you say data is lost if you set black and white points at raw conversion time. What if you don't? What if you simply demosaic without adjusting black and white points and move straight into an RGB TIFF image editor (e.g. Photoshop) ? Is any data lost in that case? Is there anything I can no longer do (i.e. impossible as opposed to just less convenient) ?
Most conversions from raw to "developed" would include at least:
1. Demosaic
2. Color correction and white balance
3. Choosing black point, white point
4. Sharpening, Denoising
5. Nonlinear tonecurve/gamma
6. Quantization to 8 bits
7. Matrixing to YCbCr and downsampling chroma channels
8. Lossy encoding to e.g. jpeg

Optionally highlight clipping recovery, lense corrections.

Most of these steps are (or can be) non-linear and difficult to reverse. Some are throwing away information and are therefore impossible to reverse.

I am sure that it would be _possible_ to disable most of those steps in order to make a jpeg (or tiff) as close as possible to a raw file. I am not sure that it makes much sense, though. The point of a raw file is to retain as much information possible about the signal coming from the camera sensor. The point of a developed file is usually to "look good". What is the point in trying to make a bastard that is neither?

The "colors" obtained if raw sensor channels (post demosaic) are routed directly to "r", "g", "b" channels looks unsaturated and dull. In order to make for something pleasing (and realistic), the raw devloper needs to apply some color transform that (among other things) tends to increase saturation. Unless you have precise knowledge of what it did, it can be hard to apply another transform (based on measurements of raw file behaviour) at a later point. The applied transform could also cause clipping in an image (even if the raw data were not clipped). In that case, it will be impossible to figure the original values.

The Demosaic process can be hard or impossible to invert. If an image scaling operation is "interpolatory", then original valules will be unchanged, only new ones will be inserted. If that is the case, you may be able to invert demosaic if you know the original sequence of "rggb" vs "bggr" etc, and the offset into the raw sensor data used for output image. If the process is non-interpolatory (i.e. changes pixels at both "known" and "unknown" data sites), it is going to be very hard to invert it.

In the case of closed-source and proprietary raw developers (such as internal camera jpeg generation, Adobe Lightroom,...) we dont even know exactly how the forward transformation works (i.e. what, mathematically, happens when I import a Canon 20D image and drag the exposure slider by "1.2 stops"), much less know how to best invert those processing steps.


And so on...

-h
Title: Re: What info can be lost once you have completed raw conversion?
Post by: ashaughnessy on January 15, 2013, 10:11:39 AM
Thanks hjulenissen.
The point of my question is to work out if I am missing out on possibilities through my choice of raw converter. I've been using the Pentax raw conversion software as provided with my K20D followed  by Picture Window Pro for the picture editing. I've been doing the bare minimum in the Pentax software, just converting the raw DNG file into a TIFF that I can then process in PWP. Usually, the only thing I adjust in the Pentax software is the white balance (if it needs it). Or put more accurately, the only setting I adjust from the defaults is the white balance.

I can then do colour correction, contrast setting, black and white point setting, sharpening, noise removal, etc. in PWP, which has a far richer toolset than the pentax raw converter tool.

I can put up with inconveniences and I'm not so interested in discussions of which tool has the easier workflow, what I really want to know is whether I'm fundamentally missing out on processing possibilities by doing most of my work in PWP. The ability to recover highlights during demosaicing seems to be something that is fundamentally impossible after raw conversion but everything else seems to be a matter of choice as to whether I do it in the raw software or the picture editor.

Anthony
Title: Re: What info can be lost once you have completed raw conversion?
Post by: Petrus on January 15, 2013, 11:22:00 AM
I would say that do everything possible with the RAW converter, as all available adjustment leeway is still there. I.e. make the picture as good looking as possible, THEN do only those adjustments the RAW converter can not do with Photoshop or whatever. After starting to use Lightroom 4.3 I have almost stopped using Photoshop at all for most of my photos, and they look better now.
Title: Re: What info can be lost once you have completed raw conversion?
Post by: Fine_Art on January 15, 2013, 01:08:40 PM
Hi Tony, you're correct that I don't own Lightroom but my point is really about what is simply impossible to do once you've left raw. For example, I can't believe there is anything you can do with regard to printing from the raw file that couldn't just as well be done from a tiff file, and choice of printing application would then come down to personal preference. Whereas with highlight recovery, that can only be done to the raw file.
Thanks
Anthony

According to Vlad recovery is created data from surrounding areas. It has nothing to do with recovering the data.

no it does not... in a typical raw converter "speak" recovery is not about "data is there" ( in a region with clipped raw channel(s) ), recovery is about postprocessing to paint (note that it is not a part of the raw conversion exactly) that part (where you have clipping) of an image using the data from unclipped raw channels and/or the data from surrounding areas in that image to make that area (where you have clipping) suitable/acceptable to your intended visual objective
Title: Re: What info can be lost once you have completed raw conversion?
Post by: ashaughnessy on January 15, 2013, 01:26:25 PM
Petrus - lots of people say as you say, but why? It sounds like the similar advice given about scanning in the "olden days" - do all your adjustments in the scanner software. I think this was mostly because photoshop in those days was mostly limited to 8-bits, but the myth hung around for much longer even when photo editors had better support for 16 bit files.

Fine-art - Vladimirovich is, I'm sure, correct in what he says, but it doesn't change the argument. I believe that the raw processor can recover/paint highlights to "unclip" them by using information that only exists in the raw file (by examing the raw luminance information in adjacent pixels). Once the raw data has been converted, the photo editor will not be able to do this "painting". Hence, highlight "recovery" as performed by Camera Raw, can only be done during the raw conversion phase and cannot be done afterwards (I think).

Anthony
Title: Re: What info can be lost once you have completed raw conversion?
Post by: Fine_Art on January 15, 2013, 01:44:14 PM
Petrus - lots of people say as you say, but why? It sounds like the similar advice given about scanning in the "olden days" - do all your adjustments in the scanner software. I think this was mostly because photoshop in those days was mostly limited to 8-bits, but the myth hung around for much longer even when photo editors had better support for 16 bit files.

Fine-art - Vladimirovich is, I'm sure, correct in what he says, but it doesn't change the argument. I believe that the raw processor can recover/paint highlights to "unclip" them by using information that only exists in the raw file (by examing the raw luminance information in adjacent pixels). Once the raw data has been converted, the photo editor will not be able to do this "painting". Hence, highlight "recovery" as performed by Camera Raw, can only be done during the raw conversion phase and cannot be done afterwards (I think).

Anthony

I bet Vlad is right as well. The point is that if it is "filling" clipped areas from surrounding areas you can do the same. It is not something special about RAW conversion. Its just a matter of how good the content fill routine is.

There are 4 RAW converters that most people feel are the best. In alpha order Capture One, DxO, Lightroom, Raw Therapee. You should try demos of those programs. What you seem to be asking is how limited are you by the software Picture Window Pro that you are using. You will have to compare.

There are 2 steps in RAW conversion that are significant. De-Bayer and Gamma shift. The rest is mostly transforms that can be reversed. What you manipulate in a raw converter is probably before the gamma log is applied. If you use 16 bit tif you probably keep all your data. You have no way of reversing changes because you dont know the exact values of manipulations done. Math functions on the data would be reversible otherwise. The only thing you cant reverse is clipping the data or overwriting the data.
Title: Re: What info can be lost once you have completed raw conversion?
Post by: digitaldog on January 15, 2013, 02:14:19 PM
The point of a raw file is to retain as much information possible about the signal coming from the camera sensor. The point of a developed file is usually to "look good". What is the point in trying to make a bastard that is neither?

Beats me. You can always go back to the raw with differing converters. The rendered image is what counts. The question becomes, what is better (in terms of quality), faster and perhaps easier: to develop as much as possible with the raw converter or afterwards in say Photoshop? I know what I prefer...
Title: Re: What info can be lost once you have completed raw conversion?
Post by: Tim Lookingbill on January 15, 2013, 02:35:12 PM
Quote
tlooknbill - I don't understand your last sentence in your last post at all.

I thought the Cambridge In Colour link would've made it clear. That site is the best at explaining highly technical concepts broken down into concise but simplified levels so non-physist types can understand it.

The bit numbers 12 & 14 that are mentioned in this thread is in regard to precision of what data off the sensor in the form of voltage readings that define the value of each pixel site is kept and thrown away (as usable detail) and converted to 1's & 0's by way of the A/D converter. How many luminance steps of RGGB are devoted to making up detail from every full saturation pixel site across all levels of charged pixel sites down to no electrical charge/signal at all=(black) before noise overtakes/fogs up perceptible dynamic range.

IOW how finely does the electronics and software want to slice up usable detail in defining it? 12 bit precision?=sharp knife or 14 bit?=even sharper. And then throw out what's not usable detail within the A/D converter.

What's left in the form of a raster x/y coordinate plot for each RGGB pixel luminance reading has already been determined by the time it reaches the software that has to demosaic those raster plots into a viewable image using an 8 bit video preview.

In the case of ACR/LR the 16bit editing space interpolates up the existing 12 or 14 bit precise culling of demosaiced data so the ACR/LR tools used to edit that data has added precision in mapping a smoother response editing demosaiced pixel data in an 8 bit video preview environment.

This is what confuses people about bit depth and what is touched upon in that link I provided.
Title: Re: What info can be lost once you have completed raw conversion?
Post by: Fine_Art on January 15, 2013, 02:39:27 PM
If you really want to be particular about data loss you need a program that uses scientific image formats. This will avoid losing rounding errors in image manipulations making operations truly reversible.

I use an astronomy program Images Plus which can do RAw conversion to 32 bit floating point fit format. This is widely used in scientific imaging.

http://fits.gsfc.nasa.gov/fits_viewer.html (http://fits.gsfc.nasa.gov/fits_viewer.html)
Title: Re: What info can be lost once you have completed raw conversion?
Post by: Tim Lookingbill on January 15, 2013, 02:54:57 PM
If you really want to be particular about data loss you need a program that uses scientific image formats. This will avoid losing rounding errors in image manipulations making operations truly reversible.

I use an astronomy program Images Plus which can do RAw conversion to 32 bit floating point fit format. This is widely used in scientific imaging.

http://fits.gsfc.nasa.gov/fits_viewer.html (http://fits.gsfc.nasa.gov/fits_viewer.html)

Yes, but how good is it at producing a pleasing image for photographers.

An electron microscope (according to what I saw on the Discovery Science channel) has now gotten so powerful they can show us what an atom looks like as perfectly lined up fuzzy balls in a grid pattern, but I wouldn't want to frame it and put it on my wall.
Title: Re: What info can be lost once you have completed raw conversion?
Post by: Fine_Art on January 15, 2013, 03:17:23 PM
Its really limited by the ability of the person. All image editing functions can be done in 32 bit floating point space. You will never get combing.

No, it does not have a "secret sauce" function that makes your image wonderful. You have to do it, as in know what you are doing with luminance and color curves, etc.

Maybe people should look at http://clarkvision.com/ (http://clarkvision.com/), his site is what got me into using the program. He uses it.
Title: Re: What info can be lost once you have completed raw conversion?
Post by: hjulenissen on January 15, 2013, 03:32:21 PM
If you really want to be particular about data loss you need a program that uses scientific image formats. This will avoid losing rounding errors in image manipulations making operations truly reversible.

I use an astronomy program Images Plus which can do RAw conversion to 32 bit floating point fit format. This is widely used in scientific imaging.

http://fits.gsfc.nasa.gov/fits_viewer.html (http://fits.gsfc.nasa.gov/fits_viewer.html)
Floating-point implementation does not avoid round-off errors, consequently arithmetic implemented in float32 is generally not "truly reversible".

-h
Title: Re: What info can be lost once you have completed raw conversion?
Post by: Fine_Art on January 15, 2013, 04:10:01 PM
Well, Im no expert on the implementation, but it seems to me any rounding error on a 32 bit floating point file is going to be so small you will never see it on a histogram, a display, or out of a printer made in the next 10 years.

(http://upload.wikimedia.org/wikipedia/commons/thumb/d/d2/Float_example.svg/590px-Float_example.svg.png)

http://en.wikipedia.org/wiki/Single_precision (http://en.wikipedia.org/wiki/Single_precision)
Title: Re: What info can be lost once you have completed raw conversion?
Post by: Fine_Art on January 15, 2013, 11:48:31 PM
[Gollum clips his data]

"It's lost! It's lost! my precious is lost!
We could use recovery.
We don'ts have recovery!
We could stretch it, what's left.
No. No. That would make very nasty banding. And combing. We don't wants that do we Precious.
In 32bit FP we won'ts see it.
Won'ts see it?
Look, the histogram is zoomed to 14 bit view on a 12 bit RAW after very strong curves. There should be huge holes!
That's impossible! Gollum. Gollum. The edge of the sun is so smooth! How does it do its?!
Its the working space my Love, the working space. We do the maths in a large working space!

Title: Re: What info can be lost once you have completed raw conversion?
Post by: hjulenissen on January 16, 2013, 03:27:34 AM
I dont know where you get your kicks from, I merely pointed out that your statement was clearly wrong:
Quote
If you really want to be particular about data loss you need a program that uses scientific image formats. This will avoid losing rounding errors in image manipulations making operations truly reversible.

If you want to argue that _you_ are unable to _see_ visible artifacts running _your algorithms of choice_ on _your images_ employing float32, then that is fine.

32-bit float can be insufficient for certain algorithms and applications. 8-bit integer can be more than enough for other algorithms and applications. As always, single marketing numbers does not tell the whole story.

-h
Title: Re: What info can be lost once you have completed raw conversion?
Post by: jeremypayne on January 16, 2013, 06:39:59 AM
That would make very nasty banding.

Can you show us some practical examples where using a 32bit FP implementation you were able to achieve a visibly superior results?
Title: Re: What info can be lost once you have completed raw conversion?
Post by: Fine_Art on January 16, 2013, 12:41:38 PM
Can you show us some practical examples where using a 32bit FP implementation you were able to achieve a visibly superior results?

I could do a one shot raw conversion in 16 bit tif vs 32 bit fp, then display the difference. I will do that later. Look at the edge of the sun in the screenshot. I duplicated a portion of the shot to test a curve that would be called a strong hyperbolic stretch of the data. Do you see banding?

Another example that comes to mind immediately is a several shot bracket merged to one HDR. I can take several shots then combine them to one. They all fit in a 32 bit fit. I can save the file to fit then save again to jpg or tif with tone mapping.

Title: Re: What info can be lost once you have completed raw conversion?
Post by: Fine_Art on January 16, 2013, 02:33:51 PM
Ok, Here is my first HDR attempt in this program. I had to spend a fair amount of 1/2 hr figuring out how to do it.

This was a 3 shot bracket on a bobbing ferry hand held to the window. The image is nothing special, it was just an opportune shot to pass time. You can see a black edge around 2 sides where the shots had to be aligned. I could have done translate and rotate but it was fast enough I don't think there was much rotate.

The whites were badly clipped in one shot. The whole lower tree area was black or near black in the other. You get the idea. It was done in part of the time since the prior post.

Now, we have all seen many gross HDRs put on the web for display as final images. This looks pretty realistic considering it was a fast cobble together with a lot of time spent trying to align the shots.
Title: Re: What info can be lost once you have completed raw conversion?
Post by: bjanes on January 16, 2013, 04:08:07 PM
Its really limited by the ability of the person. All image editing functions can be done in 32 bit floating point space. You will never get combing.

No, it does not have a "secret sauce" function that makes your image wonderful. You have to do it, as in know what you are doing with luminance and color curves, etc.

Maybe people should look at http://clarkvision.com/ (http://clarkvision.com/), his site is what got me into using the program. He uses it.

ImagesPlus is an excellent but specialized program. It is more suited for processing of astronomical images than for general photography and is often used for sensor analysis using raw files. However, it is less suitable for general photography. Among other considerations, it does not appear to be color managed. It converts my D800e images into an untagged color space. ACR/LR (among others) is better suited for general photography. What about printing with profiles and softproofing?

Iris (http://www.astrosurf.com/buil/us/iris/iris.htm) is another astronomical program favored by techies for analysis of raw files. It can split the CFA image into its channels (RGGB), perform mathematical operations, and perform demosaicing, WB and gamma corrections. It also does Richardson-Lucy deconvoluiton and VanClittret filtering. It also is not color managed. For general photography, I will stick to ACR/LR.

Regards,

Bill
Title: Re: What info can be lost once you have completed raw conversion?
Post by: Tim Lookingbill on January 16, 2013, 04:29:23 PM
Ok, Here is my first HDR attempt in this program. I had to spend a fair amount of 1/2 hr figuring out how to do it.

This was a 3 shot bracket on a bobbing ferry hand held to the window. The image is nothing special, it was just an opportune shot to pass time. You can see a black edge around 2 sides where the shots had to be aligned. I could have done translate and rotate but it was fast enough I don't think there was much rotate.

The whites were badly clipped in one shot. The whole lower tree area was black or near black in the other. You get the idea. It was done in part of the time since the prior post.

Now, we have all seen many gross HDRs put on the web for display as final images. This looks pretty realistic considering it was a fast cobble together with a lot of time spent trying to align the shots.

Now show how the 32 bit floating point benefited the image which was the reason for asking you to post examples.
Title: Re: What info can be lost once you have completed raw conversion?
Post by: Fine_Art on January 16, 2013, 06:26:38 PM
ImagesPlus is an excellent but specialized program. It is more suited for processing of astronomical images than for general photography and is often used for sensor analysis using raw files. However, it is less suitable for general photography. Among other considerations, it does not appear to be color managed. It converts my D800e images into an untagged color space. ACR/LR (among others) is better suited for general photography. What about printing with profiles and softproofing?

Iris (http://www.astrosurf.com/buil/us/iris/iris.htm) is another astronomical program favored by techies for analysis of raw files. It can split the CFA image into its channels (RGGB), perform mathematical operations, and perform demosaicing, WB and gamma corrections. It also does Richardson-Lucy deconvoluiton and VanClittret filtering. It also is not color managed. For general photography, I will stick to ACR/LR.

Regards,

Bill

It converts the file as straight linear data. Gamma is under the color menu, you have to do it yourself.

You are absolutely right it is not good for going to print. It will not understand a color calibrated screen. It will not provide a soft-proof for printing. You will have to send the document to another program for that. I have always used PS for that.
Title: Re: What info can be lost once you have completed raw conversion?
Post by: Fine_Art on January 16, 2013, 06:39:27 PM
Now show how the 32 bit floating point benefited the image which was the reason for asking you to post examples.

The benefit of having your data manipulated in a 32bit fp should be clear. I believe the latest version of PS uses the same for it's HDR for the same reason.

One thing about that file is the light tones in the snow were not gamma adjusted. Quite often tones near clipping look so similar they cannot be distinguished. By giving the user control of gamma after a straight linear conversion the program allowed me to show the whites as the camera sees them. Tone mapped,if you will over a large luminance range. I was able to combine that with the same at the low end from another shot which I used a very strong curves on instead of gamma. The curve was hyperbolic, almost a right angle. 2 shots with 12 bit data were put together over 32 bit fp and output to a full 16 bit tif. Maybe I could have output to a 32 bit fit, I'm not sure, I was struggling to learn the features.

Title: Re: What info can be lost once you have completed raw conversion?
Post by: bjanes on January 16, 2013, 08:35:39 PM
It converts the file as straight linear data. Gamma is under the color menu, you have to do it yourself.

You are absolutely right it is not good for going to print. It will not understand a color calibrated screen. It will not provide a soft-proof for printing. You will have to send the document to another program for that. I have always used PS for that.


Gamma is easy enough to do, but you seem to miss the point that ImagesPlus in not color managed. Do you render into ProPhotoRGB? And what camera profile do you use.

Bill
Title: Re: What info can be lost once you have completed raw conversion?
Post by: Fine_Art on January 16, 2013, 08:54:01 PM
Gamma is easy enough to do, but you seem to miss the point that ImagesPlus in not color managed. Do you render into ProPhotoRGB? And what camera profile do you use.

Bill

When I go to print I render to AdobeRGB. It is bigger than what my printer (Epson R1800) can handle. The camera profile is the same. That is in PS, I would not try to print from IPlus.
Title: Re: What info can be lost once you have completed raw conversion?
Post by: hjulenissen on January 17, 2013, 02:47:05 AM
ImagesPlus is an excellent but specialized program. It is more suited for processing of astronomical images than for general photography and is often used for sensor analysis using raw files. However, it is less suitable for general photography. Among other considerations, it does not appear to be color managed. It converts my D800e images into an untagged color space. ACR/LR (among others) is better suited for general photography. What about printing with profiles and softproofing?

Iris (http://www.astrosurf.com/buil/us/iris/iris.htm) is another astronomical program favored by techies for analysis of raw files. It can split the CFA image into its channels (RGGB), perform mathematical operations, and perform demosaicing, WB and gamma corrections. It also does Richardson-Lucy deconvoluiton and VanClittret filtering. It also is not color managed. For general photography, I will stick to ACR/LR.

Regards,

Bill
I suggest MATLAB. Most anything that can be done to a dataset in a usual programming language (such as c++, java and whatever photoshop is implemented in) can be done in MATLAB. The trade-off is that implementation effort is lower, while execution time is higher. There are specialized "toolboxes" for e.g. image processing. It even defaults to 64-bit float precision :-)

Calling dcraw (used as a proprietary raw format parser only, not for its development capabilities), I have setup simple development scripts using on the order of 50 lines of script code. Using only well-defined, mathematical vector-matrix operations.
fname = 'IMG_4311';
system(['./dcraw -v -i ', fname, '.CR2'])
system(['./dcraw -D -4 -T ', fname, '.CR2']);
im = double(imread([fname, '.tiff']));
im = im(1:2*floor(end/2), 1:2*floor(end/2)); %crop to even multiple of 2 for simplicity

%% normalization
im = im - 2^11;%remove bias
im = im ./ (2^14-2^11);%normalize

%% separate channels, naiive demosaicing
kernel1 = [1 2 1;...
           2 4 2;...
           1 2 1]./4;
kernel2 = [0 1 0;...
           1 4 1;...
           0 1 0]./4;     
r = zeros(size(im));
r(1:2:end-1,1:2:end-1)  = im(1:2:end-1,1:2:end-1);
r = conv2(r, kernel1, 'same');

g = zeros(size(im));
g(1:2:end-1,2:2:end  ) = im(1:2:end-1,2:2:end  );
g(2:2:end  ,1:2:end-1) = im(2:2:end  ,1:2:end-1);
g = conv2(g, kernel2, 'same');

b = zeros(size(im));
b(2:2:end  ,2:2:end  )  = im(2:2:end  ,2:2:end  );
b = conv2(b, kernel1, 'same');


%% white balance
white_patch = [1800 1200];%manually found "white patch"
r = r./r(white_patch(1), white_patch(2));
g = g./g(white_patch(1), white_patch(2));
b = b./b(white_patch(1), white_patch(2));

%% assembly and clipping
im2 = cat(3, r, g, b);
im2(im2<0) = 0;
im2(im2>1) = 1;

%% plot
figure
image(im2)
Results in the attached image. No part of my script is claimed to be "high quality" or even free of bugs. The sole purpose of my post is to show the relative ease of doing everything yourself if you are interested in the technical details.

-h