Pages: [1]   Go Down

Author Topic: D2X raw compressed OK?  (Read 5002 times)

DiaAzul

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 777
    • http://photo.tanzo.org/
D2X raw compressed OK?
« on: April 22, 2005, 09:54:07 am »

NEF files, DNG, CR2 and lossless JPEG compression all use Huffman encoding to reduce the size of the file. This is a very common encoding technique and uses the following algorithm (simplified form):

1/Take the difference between the value of this pixel and the previous pixel (if at the first pixel use the difference between this pixel and the median value).
2/This will then give you a string of values for the row of differences between pixels. Now, for a typical image most of these values are going to be at or close to zero (i.e the distribution of values is strongly skewed towards smaller values unless the image contains a lot of high frequency detail - see note below). The reasoning for this is that most pixels are similar in value to the preceding one (flat or slowly changing tones - NB each channel is processed separately).
3/Using the statistical distribution of data we can create a code table which outputs numbers of various length, such that common input values are represented by shorter code lengths. By encoding fixed 16-bit values using a variable length code with shorter codes being assigned to more common input values then the average number of bits required to represent a pixel is reduced from say 16-bits to an average of 8-10bits per pixel. Hence, there is a 40-50% reduction in file size.

4/Once on the computer the process is reversed and the original data is extracted.

The above gives the gist of the encoding process, though in reality it is a little (though not much more) complex. The key element is in choosing or calculating the encoding table. For in camera applications this is likely to be a fixed table (i.e. the same for each image) based upon a statistical sample of images. However, more complex algorithms can calculate an optimal table per image (provided you have processing power and time in order to undertake the statistical analysis).

The note is that Huffman encoding is the second part of the lossy-JPEG encoding process, the first part strips out high frequency information from the image which makes the Huffman encoding very efficient and produces much more compact files.

To re-iterate, in case it isn't clear, Huffman encoding is a LOSSLESS encoding process, so no information is lost by way of using this algorithm.

It sounds as if Nikon marketing really doesn't have a clue as there is NO effect on image quality the 'almost' is misleading and totally redundant.

(NB The reason I know all this is that I have written my own raw convertor for research purposes and had to determine the above decoding process).
Logged
David Plummer    http://photo.tanzo.org/

Robert Spoecker

  • Full Member
  • ***
  • Offline Offline
  • Posts: 162
D2X raw compressed OK?
« Reply #1 on: April 22, 2005, 05:20:26 pm »

Quote
It sounds as if Nikon marketing really doesn't have a clue as there is NO effect on image quality
This is in camera compression we're talking about.  Are you seriously suggesting that the Nikon folks don't understand what their own software is doing?
Until I get some very very sure news that there's absolutely NO compromise or risk in using in camera raw compression, I'll go with the big files.  If the compression were absolutely lossless, I think Nikon would know this and not make uncompressed the default option.  From everything I've read about the D2X (being at least as good as 1dsMKII, etc.), I also don't feel all that bad about the bigger files, since the quality is perhaps almost proportionately bigger and since CF cards are really cheap enough and also small enough to carry any number anywhere.
Logged

didger

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2030
D2X raw compressed OK?
« Reply #2 on: April 22, 2005, 07:18:28 pm »

Oh, well, I can live with 20MB files.  I back raw to DVD and that's dirt cheap and I have enough CF cards.
Logged

didger

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2030
D2X raw compressed OK?
« Reply #3 on: April 24, 2005, 10:46:19 am »

I've also been very skeptical and apprehensive about optical media.  However, I'm double backing all raw to DVD and I'm using Tayo Yuden, which is vigorously (but perhaps still falsely) claimed to be extremely robust archival quality.  Redoing everything every 6 months is out of the question with the amount of stuff I produce.  It would be a lot cheaper just to back to hard disks, which I do anyway (double backup) for all my converted files.  I've encountered almost no situations so far where I need to go back to raw after conversion, but just in case...  the DVD's because it's so cheap and not much hassle.
Logged

didger

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2030
D2X raw compressed OK?
« Reply #4 on: April 25, 2005, 07:24:02 am »

Well, I consider the issue settle; no compressed raw.  I have 2 4GB cards and 8 1 GB, and even my most profligate shooting won't outstrip that.  CF cards are so light that even carrying up to 10 of them is insignificant weight and bulk even to the most extreme backpacking weight saver.
Logged

didger

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2030
D2X raw compressed OK?
« Reply #5 on: April 22, 2005, 09:37:55 am »

I recently learned about the compressed raw option for D2X.  The manual says 40-50% reduction in file size and "almost no effect on image quality".  Anyone know just what this means?  "Almost" is a very stretchy ambiguous word.  What are the effects on image quality?  Are colors not as exact, is effective resolution compromised, are there artifacts introduced?  In view of the fact that D2X raw files are nearly double the size of 1ds, getting that cut back down to about the same size as 1ds would be nice, but not if there's risks of any sort of visible image deterioration.  Exact color values would concern me the least, since I always diddle with that in PS anyway, but resolution and artifact issues are another matter.

Anyone have any definite knowledge?
Logged

ausoleil

  • Newbie
  • *
  • Offline Offline
  • Posts: 38
D2X raw compressed OK?
« Reply #6 on: April 22, 2005, 09:57:29 am »

didger, I have played with it a bit, and have noticed little if any artifacting.

I agree with the file size, as now I suddenly need to double my CF cards in order to be able to shoot the same number of shots before  I need to visit the computer.  Ah well, it's a good problem to have, really, and CF cards are getting cheaper all the time.

I tend to shy away from it though because of something I noticed with the D100 -- if I use compressed NEF files it seemed that sharpening later lead to artifacting a lot quicker.  I cannot back that assertation with anything more than seat of the pants observation, but it really did seem to occur.

Perhaps someone more knowledgeable about NEF formats could illuminate further.
Logged

DiaAzul

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 777
    • http://photo.tanzo.org/
D2X raw compressed OK?
« Reply #7 on: April 22, 2005, 06:44:21 pm »

I did some more research on the NEF file format. I was unaware at the time of writing the first post that Nikon perform quantisation of the image prior to compression. You can read the details here:

NEF Compression

This gives a better picture of what is actually happening. It seems a strange thing to do to chuck away valuable information when saving a compressed RAW file, Canon manages to save a compressed RAW file with no loss of image information and achieves a good level of compression.
Logged
David Plummer    http://photo.tanzo.org/

ausoleil

  • Newbie
  • *
  • Offline Offline
  • Posts: 38
D2X raw compressed OK?
« Reply #8 on: April 24, 2005, 10:40:07 am »

Didger, just make sure that you back up the backups on a bi-annual basis.  CD Rot is well understood, and I would not be the least bit surprised to hear of DVDR rot as well.
Logged

ausoleil

  • Newbie
  • *
  • Offline Offline
  • Posts: 38
D2X raw compressed OK?
« Reply #9 on: April 24, 2005, 10:56:49 am »

Ooops, I said bi-annual whjen I meant every other year.
Logged

BernardLanguillier

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 13983
    • http://www.flickr.com/photos/bernardlanguillier/sets/
D2X raw compressed OK?
« Reply #10 on: April 25, 2005, 06:55:10 am »

Hi Didger,

Checking other discussions on this topic, it seems that some people are seeing a loss of information in the highlights when using compressed RAW.

I used to use it, but have kind of stopped. The transition from blown highlights to non blown areas tends to be a bit sharp with the D2X, and I thought loosing details in that part of the curve was not such a great idea. Back to non compressed RAW and using the Epson P2000.

200 images on a 4GB micro-drive is not much though, and I would typically out shoot that during a busy day of shooting without too much moving around. Obvisouly I wouldn't reach those levels when trekking over even averagely long distances.

cheers,
Bernard
Pages: [1]   Go Up