A700 -greater than 12bit RAW files?

[dunmunro]

Hi all. My first post.

I have examined several A700 RAW files with IRIS, an astronomy oriented imaging/data processing program:

http://www.astrosurf.com/buil/us/iris/iris.htm

It can read the non colourized RAW file data units directly, for most DSLRs. In every camera RAW file I have examined, it will show values of 0-4095 for 12bit RAW and 0-16383 for 14bit RAW files, but the A700 files show values of 0 - 8190 when examining the RAW BFA (non colourized images), values for example:

Model: SONY DSLR-A700
Date/Hour: 21/09/2007 18:19:12
Exposure time:  0.040 s
Aperture: f/8.0
Focal length: 60.0 mm
ISO: 200 ASA
Format: 4288x2856
 
Mean: 1050.9         Median: 1028
Sigma: 641.6
Maxi.: 8190.0       Mini.: 238.0

An very overexposed example from a D50:

Model: NIKON D50
Date/Hour: 24/05/2006 19:47:04
Exposure time:  0.002 s
ISO: 200 ASA
Format: 3039x2014
 
Mean: 3995.1         Median: 4070
Sigma: 88.5
Maxi.: 4095.0       Mini.: 3879.0

A 40D:

Model: Canon EOS 40D
Date/Hour: 11/09/2007 21:53:33
Exposure time:  0.005 s
Aperture: f/8.0
Focal length: 100.0 mm
ISO: 1600 ASA
Format: 3908x2602
 
Mean: 2708.9         Median: 2953
Sigma: 1323.7
Maxi.: 16224.0       Mini.: 689.0
 
and a very overexposed K10D image:

Model: PENTAX K10D
Date/Hour: 19/11/2007 06:45:16
Exposure time:  1.000 s
Aperture: f/5.6
Focal length: 50.0 mm
ISO: 800 ASA
Format: 3896x2616
 
Mean: 4078.4         Median: 4080
Sigma: 15.5
Maxi.: 4095.0       Mini.: 6.0

When I convert the BFA to colour, the images retain the same range of values, except the A700, which shows, typically, a data range from 0 to 10,000 to 12,000.

I conversed briefly, with the author of IRIS and he confirms the data values for the A700 are actually encoded in the file. What I am wondering, is if someone else, with access to a program that can read the actual RAW data values, per pixel can confirm this? The file sizes seem to suggest a 12bit file, but the values suggest otherwise, and it is possible that some sort of compression scheme is being used to reduce the file size, or Sony is  using a compression algorithm only on values above 4095, to retain data in the highlights.

thanks

Duncan

[Panopeeper]

What about posting such files? (The simplest is via yousendit.)

[dunmunro]

What about posting such files? (The simplest is via yousendit.)
[{POST_SNAPBACK}][/a]

I got the A700 raw files here:

[a href=\"http://www.imaging-resource.com/PRODS/AA700/AA700RAW.HTM]http://www.imaging-resource.com/PRODS/AA700/AA700RAW.HTM[/url]

and here:

http://www.sendspace.com/file/zitq6o

Imaging resources has sample D300, 40D and D3 raw files in the same format.

[Panopeeper]

Which A700 file? Can't you be somewhat specific? There are about 35 A700 raw files on the Imageing Resource site. I downloaded two of them and they don't have anything over 2000.

[dunmunro]

Which A700 file? Can't you be somewhat specific? There are about 35 A700 raw files on the Imageing Resource site. I downloaded two of them and they don't have anything over 2000.
[a href=\"index.php?act=findpost&pid=166032\"][{POST_SNAPBACK}][/a]

What are you viewing them with?

I looked at aa700hslio800.arw and others, also DCS00273.arw  and 277 from the other site.

[Panopeeper]

What are you viewing them with?

Rawnalyze (converted in DNG format)

I looked at aa700hslio800.arw and others, also DCS00273.arw  and 277 from the other site

All three are 12-bit. There are no values over 2000 in any of the mentioned files. The SR2-files are 14-bit, and they do go up to 16383.

[dunmunro]

Rawnalyze (converted in DNG format)
All three are 12-bit. There are no values over 2000 in any of the mentioned files. The SR2-files are 14-bit, and they do go up to 16383.
[{POST_SNAPBACK}][/a]

OK, I'll look that program up:

here's an IRIS histogram of dsc000273.arw, of the decoded file, green channel:

[a href=\"http://picasaweb.google.com/Dunmunro1/TechStuff/photo#5152521623966470434]http://picasaweb.google.com/Dunmunro1/Tech...521623966470434[/url]

all my other 12bit RAW files will show a 0-4095 data range.

[dunmunro]

BTW, how did you convert the file to DNG? perhaps the conversion program rescaled the data into a 12bit format?

[Panopeeper]

I can't say anything re the result displayed by Iris.

I am using Adoba's free DNG converter, which converts every raw file in 16bit form. The input processing of the DNG converter is the same as the one in ACR, I highly doubt, that the conversion is erroneous. I compared the SR2 and the DNG files in ACR, they look the same.

You can download the DNG converter from the Adobe site together with ACR 4.3.1 (if you don't need ACR 4.3.1, delete it, but DNG comes only in that package).

[dunmunro]

Rawnalyze (converted in DNG format)
All three are 12-bit. There are no values over 2000 in any of the mentioned files. 
[a href=\"index.php?act=findpost&pid=166038\"][{POST_SNAPBACK}][/a]

That must be a mistake because dsc00273.arw has lots of blown highlights, for example the lights hanging on the wall on either side of the doorway. BTW, I cannot get DNG to convert the files, it just gives me a "parsing error", message.

[Panopeeper]

That must be a mistake

That was a mistake. The histogram "ends" at about 2000 - BUT at the very end there are thin lines, visible only when viewed up-close.

However, it ends at 4095; there is nothing higher.

because dsc00273.arw has lots of blown highlights

0.5% of the pixels clipped.

BTW, I cannot get DNG to convert the files, it just gives me a "parsing error", message

You have to rename them from TIFF to ARW. If it still does not work, then you got an older version; click on About. Can you process the files with ACR directly?

Here is the histogram of DCS00273:

[dunmunro]

Got the DNG converter working and IRIS returns the similar results, using the DNG files, only now the data values have been halved:

http://picasaweb.google.com/Dunmunro1/TechStuff

note the 3 dsc00237 histograms. The data in the A700 RAW files is extending past 4095, and is probably being mapped somehow into a 14bit data space to preserve highlights. Note how the astroart histogram is clipped at the values equivalent to 4095. Astroart is remapping the RAW file into a 16bit space but is clipping the values above 4095 in the "12bit" RAW file.

[John Sheehy]

OK, I'll look that program up:

here's an IRIS histogram of dsc000273.arw, of the decoded file, green channel:

http://picasaweb.google.com/Dunmunro1/Tech...521623966470434

all my other 12bit RAW files will show a 0-4095 data range.
[a href=\"index.php?act=findpost&pid=166039\"][{POST_SNAPBACK}][/a]

IRIS is not up-to-date for new cameras.  IRIS uses the RAW decoding section from DCRAW, and the current version of IRIS uses a very old version of DCRAW, probably at least 8 months old, before the A700 existed.  IRIS will often attempt to open a file it really doesn't support, and misunderstand the data.

For the Canon mk3 cameras, for example, it includes parts of the file that aren't even in the image, so the image needs to be cropped.

Your histogram looks like half of your data is at normal scale, and half of it is doubled.  That's why the "line" in the histogram is so thick; it is alternating between two population levels every other value.  The even numbers are probably the higher populations.  This may be because the format that IRIS/DCRAW thinks your RAW is in may have needed to have some values doubled.

[Guillermo Luijk]

When calculating the undemosaiced histogram with dcraw v8.80 over the 'aa700hslio800.arw' file:
C:\DOCUME~1\gluijk\ESCRIT~1>dcraw -v -D -T -4 test.arw

This is what you get:



Levels range up to 8192 (double as expected), and in 1:1 zoom not all levels are filled and the ones that are, are not equally spaced.

Any explanation for this? Is my histogram routine missinterpreting the integer coding? is a DCRAW v8.80 bug?

[dunmunro]

IRIS is not up-to-date for new cameras.   d.
[a href=\"index.php?act=findpost&pid=166107\"][{POST_SNAPBACK}][/a]

IRIS 5.52 is based upon the newest version of Dcraw, afaik, as previous versions would not even open A700 ARW files.

[Panopeeper]

I have no experience with DCRaw. Isn't it possible, that what you see is WBd?

[Guillermo Luijk]

I have no experience with DCRaw. Isn't it possible, that what you see is WBd?
[a href=\"index.php?act=findpost&pid=166205\"][{POST_SNAPBACK}][/a]

DCRAW's -D is pure RAW, no balancing at all. In fact if opened in PS (and bright corrected to see anything out of that dark) you can even guess the Bayern pattern.

[John Sheehy]

IRIS 5.52 is based upon the newest version of Dcraw, afaik, as previous versions would not even open A700 ARW files.
[a href=\"index.php?act=findpost&pid=166163\"][{POST_SNAPBACK}][/a]

Yes, I just downloaded 5.52.

5.52 still just closes itself on me unexpectedly when the file picker is open, like 5.50 and 5.51 did.

It looks like only 21 out of every 50 values are populated, at least in a couple ISO 1600 files I am looking at.  That means that there are about 8192*21/50 or 3440 unique values in the ISO 1600 RAW data; less than 4095, but more than some Canons at certain ISOs.

[dunmunro]

Yes, I just downloaded 5.52.

5.52 still just closes itself on me unexpectedly when the file picker is open, like 5.50 and 5.51 did.

It looks like only 21 out of every 50 values are populated, at least in a couple ISO 1600 files I am looking at.  That means that there are about 8192*21/50 or 3440 unique values in the ISO 1600 RAW data; less than 4095, but more than some Canons at certain ISOs.
[a href=\"index.php?act=findpost&pid=166237\"][{POST_SNAPBACK}][/a]

It does, occasionally, close on me as well. Do you see the values above 8190 for the colourized images?

[Panopeeper]

ISO1600 is apparently fake, like the between-the-stops ISOs of the Canon cameras. Every fourth/fifth level, alternating, is a gap.