Pages: [1]   Go Down

Author Topic: What really are RAW files?  (Read 5464 times)

Jae_Moon

  • Full Member
  • ***
  • Offline Offline
  • Posts: 133
    • http://
What really are RAW files?
« on: September 21, 2006, 07:12:05 pm »

I want to make a QUALIFIER up front: I am technical just enough to be dangerous.

The points which I ASSUME are true.

1.   RAW file contains luminous data (0-4095 for 12 bits systems, for example) from pixels within the Image Sensor which has Bayer matrix pattern (or similar) overlay.
2.   Data has been processed for ‘noise reduction’ via built-in electronics.
3.   Data has been compressed.
4.   RAW file contains Meta Data.
5.   Phase One attaches a calibration data file for an individual image sensor according to LLVJ (approximately 1 meg file for P25 which corrects the responses of defective pixels; dead pixels and out-of-specs pixel responses, etc) to be used by its RAW converter, Capture One. I don't know how other MFDB manufacturers handle this issue, any one?
6.   Leaf’s allows the uploading of ‘special programs’ to its Aptus DB so RAW files can be modified (per email from Mr. Shachar Kedem of Leaf to Michael Kravit, http://luminous-landscape.com/forum/index....howtopic=12079).
It states ‘Gain functionality within Leaf Capture 8 enables creating and using gain corrections for specific lenses. We have developed a new version of Leaf Capture 8 which will produce User Gain files that can be loaded to the Aptus back and used during any shoot (including portable shooting).

Item 5 and 6 raise the questions about the RAW files and RAW Converters.

A special calibration file for Phase One DB makes senses as long as the RAW files are processed by its converter, Capture One. But, as far as I know, Phase One did not make their file protocol public, still certain 3rd party software supposedly able to convert the files even better than Capture One. Is it possible? Are they ignoring the calibration file?

Phase One can calibrate its DBs individually since they charge a lot for them, but what about Canon and Nikon? I am not talking about digicams but their high-end cameras, 1D’s and D2Xs’. Do they calibrate each image sensor? If they don’t, how are they dealing with the problems of non-uniform responses with pixels? I deduced LEAF do calibrate its sensors individually.

DPReview has been testing and comparing the performances of different cameras mostly with JPEG (how one is better to make red redder and lines look sharper than the others using its proprietary software). Are RAW files from Aptus DBs more than just RAW files?  I am interpreting Mr. Kedem’s email as ‘image enhancing programs can be uploaded to its DB so its files can be processed in specific manner before it would be downloaded to a computer as RAW files.’

PLEASE COMMENT, IF I AM READING HIS EMAIL WRONG.

If one can load a program to a DB to fix the image sensor problem that Mr. Kravit was reporting by means of an ‘image enhancing program,’ can one also load ‘style/profile’ program to give images certain looks? Does Aptus DB do ‘image enhancing’ on its RAW files?

I understand how a better image sensor can create better RAW files with wider Dynamic Range, with wider ISO range, with lower noise ratio, with better designed micro lens (off-centered lenses for Leica M8), with ability to have long exposure, etc. But will these improvements (or advantage of one vendor to others) create better looks?  A better design of Bayer matrix and decoding algorithms would bring better color rendition, less aliasing, and sharper images, but will it bring different looks?

For the record, I own P45, H2 and Arca-Swiss F6x9 View Camera with Schneider Digitar lenses. I also had Canon 1Ds until recently for three years.

Jae Moon
« Last Edit: September 22, 2006, 09:05:34 pm by Jae_Moon »
Logged

yaya

  • Guest
What really are RAW files?
« Reply #1 on: September 22, 2006, 03:58:39 am »

Just answering your question regarding the Aptus backs being able to apply different looks to the raw file, I will try to avoid getting too technical:-)

The Aptus has a Windows CE operating system installed that allows for an advanced set of factory settings: Colour/B&W, Input profile, Working colour space, Grey balance, sharpening, tone curve and even size/ crop and soon gain calibration.

Grey balance can be set on the back as a custom setting and in addition there is a facility to load your own custom settings to the storage media (CF card and/ or Digital Magazine). Each combination of these settings will give the file a different look.

Each back goes through a calibration prcess in the factory to map out faulty pixels and this information is stored in the back and is applied to each and every file by the on board image processor. The back is also being calibrated to create a factory gain file that is also stored in the back. This gain file is designed to even out the image in terms of luminance response.

As you know from use of your P45/ Arca system, when wide angle lenses with movements are being used, the file may show colour casts and in extreme cases (with the A75) also the "centrefold" effect...
Imagine that you can create an LCC reference file in C1 to a specific lens/ aperture/ shift combination (with some tolerance) and load a set of these to the back so that they can be used in the field. We also offer a way to conrol the ammout of lens falloff removal, with or without the use of a centre filter.

In addition, there's a facility to add metadata, such as copyrights, file description, notes etc. by use of a keypad or a QWERTY keyboard a' la' PDA.

All this information is carried in the header of the raw file, not embedded in it, and therefore can be read in fully by the Leaf Capture software. Other raw convertors may choose to read all or parts of this information.

The ONLY information that is embedded in the RAW file is the ISO, the creation date/ time and the EXIF data that comes from the camera body and the lens being used, such as aperture, speed and EV.

I hope this helps, Yair
« Last Edit: September 22, 2006, 04:00:06 am by yaya »
Logged

eronald

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 6642
    • My gallery on Instagram
What really are RAW files?
« Reply #2 on: September 22, 2006, 05:02:35 am »

Quote
Just answering your question regarding the Aptus backs being able to apply different looks to the raw file, I will try to avoid getting too technical:-)

The Aptus has a Windows CE operating system installed that allows for an advanced set of factory settings: Colour/B&W, Input profile, Working colour space, Grey balance, sharpening, tone curve and even size/ crop and soon gain calibration.

Grey balance can be set on the back as a custom setting and in addition there is a facility to load your own custom settings to the storage media (CF card and/ or Digital Magazine). Each combination of these settings will give the file a different look.

Each back goes through a calibration prcess in the factory to map out faulty pixels and this information is stored in the back and is applied to each and every file by the on board image processor. The back is also being calibrated to create a factory gain file that is also stored in the back. This gain file is designed to even out the image in terms of luminance response.

As you know from use of your P45/ Arca system, when wide angle lenses with movements are being used, the file may show colour casts and in extreme cases (with the A75) also the "centrefold" effect...
Imagine that you can create an LCC reference file in C1 to a specific lens/ aperture/ shift combination (with some tolerance) and load a set of these to the back so that they can be used in the field. We also offer a way to conrol the ammout of lens falloff removal, with or without the use of a centre filter.

In addition, there's a facility to add metadata, such as copyrights, file description, notes etc. by use of a keypad or a QWERTY keyboard a' la' PDA.

All this information is carried in the header of the raw file, not embedded in it, and therefore can be read in fully by the Leaf Capture software. Other raw convertors may choose to read all or parts of this information.

The ONLY information that is embedded in the RAW file is the ISO, the creation date/ time and the EXIF data that comes from the camera body and the lens being used, such as aperture, speed and EV.

I hope this helps, Yair
[a href=\"index.php?act=findpost&pid=77230\"][{POST_SNAPBACK}][/a]


CCDs can do analogical processing of the data before they do the conversion to digital. I believe ISO is one such analogical process, and and there may be advantages in doing an analogical white balance.  It would be interesting to know what the Leaf backs do, as far as they are willing to divulge this information - in fact it would be interesting to know what the "generic" processing steps for all backs are.

Yair - it might help your company to make a clear statement here, since Canon has a white paper on CMOS technology  - The CCDs like the ones you use doubtless have certain advantages compared to CMOS and there are certain difficulties (eg. stitched sensor, multiple readouts) to be overcome. Writing a white paper on the use of CCDs would help with marketing.

Let me put it this way: people are asking for more information, when they get it  they feel more comfortable with the product, that makes the buy decision easier, so you sell more ...

Edmund
« Last Edit: September 22, 2006, 05:05:49 am by eronald »
Logged
If you appreciate my blog posts help me by following on https://instagram.com/edmundronald

psorantin

  • Jr. Member
  • **
  • Offline Offline
  • Posts: 55
What really are RAW files?
« Reply #4 on: September 22, 2006, 08:19:37 am »

Quote
CCDs can do analogical processing of the data before they do the conversion to digital. I believe ISO is one such analogical process, and and there may be advantages in doing an analogical white balance.  It would be interesting to know what the Leaf backs do, as far as they are willing to divulge this information - in fact it would be interesting to know what the "generic" processing steps for all backs are.



... This touches on a question of WhiteBalance I am really interested in:

If you do a whitebalance versus a grey card shot to set whitebalance in-camera for all following shots, versus setting whitebalance in software during RAW-conversion:

Will the results vary?
I am particularly interested in that question for my Aptus back...

I have not done rigorous testing around this, but for my Canon's it seems to make a difference in favor of the in-camera setting versus during RAW conversion.

Greetings,
Peter
Logged
Peter Sorantin
New Jersey

yaya

  • Guest
What really are RAW files?
« Reply #5 on: September 22, 2006, 08:28:17 am »

Quote
... This touches on a question of WhiteBalance I am really interested in:

If you do a whitebalance versus a grey card shot to set whitebalance in-camera for all following shots, versus setting whitebalance in software during RAW-conversion:

Will the results vary?
I am particularly interested in that question for my Aptus back...

I have not done rigorous testing around this, but for my Canon's it seems to make a difference in favor of the in-camera setting versus during RAW conversion.

Greetings,
Peter
[a href=\"index.php?act=findpost&pid=77253\"][{POST_SNAPBACK}][/a]

Peter for the Aptus it does not make any difference at all, the White balance does not affect the raw data, it is just a tag that is attached to it.

Yair
Logged

John Sheehy

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 838
What really are RAW files?
« Reply #6 on: September 22, 2006, 10:02:38 am »

Quote
CCDs can do analogical processing of the data before they do the conversion to digital. I believe ISO is one such analogical process, and and there may be advantages in doing an analogical white balance.[a href=\"index.php?act=findpost&pid=77238\"][{POST_SNAPBACK}][/a]

No doubt, there would be an advantage to have different levels of analog amplification to do white balance before digitization, but the practical problem is that a line of pixels in a CFA matrix alternate between two colors, so the amplifier, reading them in order, would have to switch amplification between each pixel, creating a new source of noise, which may be worse than what you get in return.
.
Logged

Jae_Moon

  • Full Member
  • ***
  • Offline Offline
  • Posts: 133
    • http://
What really are RAW files?
« Reply #7 on: September 22, 2006, 12:39:28 pm »

Thank you, Yair, for your response.

I want to clarify points you made regarding Aptus.

1.   RAW files are created with individual sensor calibration data files for faulty pixels and for the normalization of luminous responses of pixels. Therefore, these calibration data files are not part of the ‘header’ of the RAW file.
2.   Advanced set of factory settings are stored in the header of the RAW files.
3.   A user can create his/her own ‘setting’ and can upload them to the DB.

I am saying item # 1, in spite of your last sentence, ‘the ONLY information that is embedded in the RAW file is the ISO, the creation date/ time and the EXIF data that comes from the camera body and the lens being used, such as aperture, speed and EV,’ since you said in earlier sentence that (calibration data) is applied to each and every file by the on board image processor.

Three questions, 1) Can users override the advanced set of factory settings and set all to null?, 2)  Did Leaf make public how to interprete the header files to 3rd party software developers?,  3) How do two images, with and without factory settings on, differ in ‘looks?’

What I am trying to understand is the statements made by many loyal Leaf users concerning its ‘film like looks.’

Is it from better chip designs by Dalsa over Kodak?
Is it from better image processing software of Leaf?
Are we comparing an apple to an apple, or an apple to an orange?

Jae Moon
Logged

yaya

  • Guest
What really are RAW files?
« Reply #8 on: September 22, 2006, 01:17:53 pm »

Thank you, Yair, for your response.

I want to clarify points you made regarding Aptus.

1.   RAW files are created with individual sensor calibration data files for faulty pixels and for the normalization of luminous responses of pixels. Therefore, these calibration data files are not part of the ‘header’ of the RAW file.
2.   Advanced set of factory settings are stored in the header of the RAW files.
3.   A user can create his/her own ‘setting’ and can upload them to the DB.

I am saying item # 1, in spite of your last sentence, ‘the ONLY information that is embedded in the RAW file is the ISO, the creation date/ time and the EXIF data that comes from the camera body and the lens being used, such as aperture, speed and EV,’ since you said in earlier sentence that (calibration data) is applied to each and every file by the on board image processor.
The "normalization" file is a large raw file of the same size as a n image file, it is not stored in the header but is used as a reference while writing the file into the storage media. The raw file can be "re-written" using another gain file in post
Correction of faulty pixels is done in the back and cannot be altered or "removed"


Three questions:
1) Can users override the advanced set of factory settings and set all to null?,
You can switch some of them off (colour management, sharpening)
2) Did Leaf make public how to interprete the header files to 3rd party software developers?
Yes, but each software has its own algorithm and matrix for analising and presenting colour and toning information. We even went further and supplied our backs to Adobe and Apple so they can fine tune their own settings. Iridient were able to do so as well with good results  
3) How do two images, with and without factory settings on, differ in ‘looks?’
Mostly in contrast, saturation, sharpness and certain colour tones. The "naked" file will look flat and desaturated

What I am trying to understand is the statements made by many loyal Leaf users concerning its ‘film like looks.’

Is it from better chip designs by Dalsa over Kodak? probably, but not only
Is it from better image processing software of Leaf? probably, but there is more than just the software there's the electronic bits in the back that construct the raw file in a certain way
Are we comparing an apple to an apple, or an apple to an orange? Apples...different look, different taste but still grow on an apple (silicon) tree...maybe the Leaf file is the organic one i.e. grows with no chemicals or genetic manipulation. The contribution of the Dalsa sensor to it is a "meatier" and "healthier" bare data

Jae Moon
[a href=\"index.php?act=findpost&pid=77276\"][{POST_SNAPBACK}][/a]
[/quote]
Logged

Jae_Moon

  • Full Member
  • ***
  • Offline Offline
  • Posts: 133
    • http://
What really are RAW files?
« Reply #9 on: September 22, 2006, 03:25:56 pm »

Yair:

Well, I got my answers for Leaf, thank you. Now I have a reasonable understanding of how two DBs are similar and different in their designs.

I still would like know how Phase One's calibration data are utilized by 3rd party RAW converter software, and how Canon and Nikon handle their image sensor calibration.

Jae Moon
Logged

damien

  • Full Member
  • ***
  • Offline Offline
  • Posts: 111
    • http://www.lovegroveportraits.com
What really are RAW files?
« Reply #10 on: September 22, 2006, 07:01:27 pm »

Sorry to be a bit late on this thread. Back to Peter's question of white balance in camera or converter: I have found that on my P25 if I get the white balance as good as I can in camera my histogram on the back better reflects the true nature of the exposure. It's easy to clobber a single chanel when doing big changes later. With the right exposure the end results are identical its a bit of a guessing game when assesing a shot with excessive cast and a histogram to match.
Damien.
Pages: [1]   Go Up