Pages: [1]   Go Down

Author Topic: Nikon RAW format  (Read 9596 times)

soslund

  • Jr. Member
  • **
  • Offline Offline
  • Posts: 76
Nikon RAW format
« on: February 03, 2006, 01:31:00 am »

Has Nikon de-encrypted their RAW files, or does one need to use their proprietary NEF software?  If the data is non-encrypted, what are some of the helpful RAW software programs?  I am familiar with Adobe Camera Raw and Capture One, but am curious about other programs.  Thanks
Logged

BernardLanguillier

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 13983
    • http://www.flickr.com/photos/bernardlanguillier/sets/
Nikon RAW format
« Reply #1 on: February 03, 2006, 02:46:41 am »

The only part of the D50, D200 and D2x .nef file that is said to be encrypted is the Auto White Balance (AWB) value.

Nikon does offer a set of API through a light SDK that "de-encrypt" this AWB value for downstream RAW conversion applications to use. Adobe ACR 3.3 appears to be using this method to decode the AWB value of D200 and D2x .nef files.

Other RAW converters have decided to reverse engineer the AWB "encryption" without using the Nikon light SDK, these include DCRAW, Bibble and RSP at least.

I am not sure about Capture One and DxO, but DxO is clearly able to get the information one way or another.

From a practical standpoint, the "encryption" of the .nef AWB data is a non issue.

Regards,
Bernard

Ronny Nilsen

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 361
    • The Quiet Landscape
Nikon RAW format
« Reply #2 on: February 03, 2006, 05:02:43 am »

Quote
The only part of the D50, D200 and D2x .nef file that is said to be encrypted is the Auto White Balance (AWB) value.
......
From a practical standpoint, the "encryption" of the .nef AWB data is a non issue.

But still, the intent of the encryption is to limit their customers in using the camera and to make more money by hindering customers from using other RAW converters. Maybe they do a better job of the encryption in the next firmware release.

It's one thing to not document and publish the RAW format as that cost money, but to use time and money to encrypt part of the RAW file to rip off your own customers is something else. The D200 will not work with third party batteries either. More rip off. Nikon is showing a trend here....

The best solution at this time seems to be if all manufacturers adoptet DNG, Adobe have documentet that format for free so the camera manufacturers don't have to spend money on it, and third party RAW converter sw can then consentrate their efforts on smart algorithms, not decoding RAW formats.
« Last Edit: February 03, 2006, 05:03:35 am by ronnynil »
Logged
Ronny A. Nilsen
www.ronnynilsen.com

BryanHansel

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 369
    • www.bryanhansel.com
Nikon RAW format
« Reply #3 on: February 03, 2006, 12:57:19 pm »

This Nikon thing was argued to death not to long ago on this forum.  Since, Nikon has stated a different intent than you suggest, I'm not sure that they really want to hinder their customers.  But hey, I was on the losing side of the argument anyway last time.

I remember when the D70 first came out.  There was no third party battery, at least, in the store where I bought mine.  Given time, a third party battery manufacture will produce a battery.

I don't know enough about Adobe's DNG to understand if it limits RAW file development in anyway.  My opinion, changed from the above mentioned arguments, is that RAW formats should be documented and published at some point.

And, hey, at least, Nikon and Adobe worked something out, and, at least, Nikon doesn't encrypt the whole file like some other camera manufactures.

Quote
what are some of the helpful RAW software programs?

Capture does an excellent job of converting Nikon Raw files.  It's a little slow compared to some others out there.  If you're using Photoshop, you should give that a try.  I just finished producing a book of lighthouse pictures for a fund raiser and did all the conversions in photoshop instead of Capture.  It went much much faster, but it seemed to me, maybe because I'm used to Capture, a little harder to get the RAW file to look the way I wanted it.  Both worked, but if Capture was faster, it would be the winner for me.

Bryan
Logged
Bryan Hansel
[url=http://www.paddling

BernardLanguillier

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 13983
    • http://www.flickr.com/photos/bernardlanguillier/sets/
Nikon RAW format
« Reply #4 on: February 04, 2006, 08:22:05 am »

Quote
But still, the intent of the encryption is to limit their customers in using the camera and to make more money by hindering customers from using other RAW converters. Maybe they do a better job of the encryption in the next firmware release.

It's one thing to not document and publish the RAW format as that cost money, but to use time and money to encrypt part of the RAW file to rip off your own customers is something else. The D200 will not work with third party batteries either. More rip off. Nikon is showing a trend here....
[a href=\"index.php?act=findpost&pid=57355\"][{POST_SNAPBACK}][/a]

Frankly speaking, BS.

If Nikon wanted to prevent us to use another RAW converter they:

- would have sued those companies that reversed engineered the "encryption" of their .nef,
- would not have developped and made available a special light version of their SDK to help other RAW converters like Adobe ACR to get the right AWB value,
- would have used a real strong encryption method, like a public key method, that is simply unbreakable,
- would have encrypted the whole of the .nef file, and not just that one AWB value.

I am in favour of the Open RAW initiative, but I am really tired of the cheap bashing done by some on Nikon against all evidences.

As far as the batteries are concerned, I am not sure about the status on this issue, but Canon and Nikon have been trying to lock their users in their system for years. Had they not, we would have a standard lens mount by now.

You might as well complain that the sun will rise tomorrow...

Regards,
Bernard

61Dynamic

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1442
    • http://
Nikon RAW format
« Reply #5 on: February 04, 2006, 04:18:27 pm »

Any thoughts that encryption is not intended to control the use of something is not quite in alignment with reality.

Nikon encrypted the WB tag to limit the customers use of the raw file in a certain way. if you wanted use of auto WB, you had to use Nikon's converter. That is an act of restriction no bones about it.

Quote
- would have sued those companies that reversed engineered the "encryption" of their .nef,
- would not have developped and made available a special light version of their SDK to help other RAW converters like Adobe ACR to get the right AWB value,
These points overlook some history. The first item being the public outcry over the issue. Nikon was being bombarded with a considerable amount of bad press. What do you honestly think would have happened if they started suing people?

Nikon was a bit stubborn-headded on the issue and the mini-SDK did was not released until some time after the issue first came about (5 months). Their first response was for companies to use their full SDK which is to be frank a POS only letting edits be done to a jpeg or tiff version of the raw file. If you wanted to use the actual raw data and the auto WB, you needed to use Nikon's software.

Quote
- would have used a real strong encryption method, like a public key method, that is simply unbreakable,
- would have encrypted the whole of the .nef file, and not just that one AWB value.
Ever hear the story about the frog in hot water? If you drop a frog into boiling water, the frog will take notice and jump out. However, if you put the frog into cold water and then gradually turn up the heat, the frog won't notice until it is too late.

One could easily speculate due to the events regarding this issue that Nikon was hoping their customers were like frogs. They introduced the encryption into the auto WB tag to gradually lock-down their format hoping people wouldn't notice or make a huge fuss about it. If they had introduced full-blown encryption, then people would have taken immediate notice.

Unfortunately their timing was bad. If they had tried this a couple or three years ago, they might have gotten away with it. However they did this at a time when people were starting to push for open formats and the issue of raw file longevity started to become a big concern. All it took was for a public complaint from a software developer to break the camels back and lead this to be the large issue that it was.

ronnynil was dead-on in his first point. The encryption was intend to limit their customers. Nikon's own statements verify that but it blew up in their face. They eventually caved in to public pressure and then released the light SDK giving developers full access to the raw data.

BernardLanguillier is right that at this point the encryption is not an issue in a practical standpoint. That however, ignores the greater issue of longevity as that encryption will make those files more difficult to read in the future than they would have been sans-encryption. The solution is to convert the raw files into DNG.
Logged

BernardLanguillier

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 13983
    • http://www.flickr.com/photos/bernardlanguillier/sets/
Nikon RAW format
« Reply #6 on: February 05, 2006, 02:40:04 am »

Various articles on the topic of encryption have explained us that it is enough to be willing to restrict access to data by encrypting them in order to have a form of legal protection of the content.

In other words, the intend to protect is key.

How to measure the intend of a company? There is no way to do it, the only solution is to watch their actions, and to theoritized the intend based on the actions.

I see no actions from Nikon after the release of the D2x that point at an intent to prevent access to their RAW data. I am therefore personnally unable to reach a conclusion on the reason why they have seemingly encrypted part of the RAW file, but preventing their customers from accressing the data does not seem to be it based on facts.

Michael himself reported after speaking with Phase that there were valid technical reasons for not including explicetely various information in the RAW file. This being related to the calibration of the sensors... etc... Whether this is what is happening with the D2x .nef or not, I don't know.

As I pointed above, I am a supported of the Open RAW initiative, that requests camera vendors to make the spec of their RAW format public.

I am currently not in favour of the adoption of the DNG format because:

- today, it is the property of Adobe,
- today, it is unclear whether or not imposing a format to the manufactuers could or could not end up slowing down their ability to innovate.

Regards,
Bernard

61Dynamic

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1442
    • http://
Nikon RAW format
« Reply #7 on: February 05, 2006, 03:19:10 am »

Sure there are legit reasons for encrypting data but the encrypted data in the NEF is nothing but a WB tag so the comparison to Phase One holds no water.

Quote
I see no actions from Nikon after the release of the D2x that point at an intent to prevent access to their RAW data. I am therefore personnally unable to reach a conclusion on the reason why they have seemingly encrypted part of the RAW file, but preventing their customers from accressing the data does not seem to be it based on facts
Emphasis added. Nikon themselves stated that was the case in their press release which I linked to previously:

Quote
After a developer’s software is created using the Nikon SDK, a NEF file can be opened, edited in either TIFF or JPEG format, and then saved in formats available in the developers’ software.  This process has been available since the first Nikon SDK for NEF.
Emphasis added. At the time, if you wanted to access the raw data and the auto WB tag (sans converters that hacked it) you had to use Nikon's software. That is factual evidence of restricting the use of the file.

Quote
- today, it is unclear whether or not imposing a format to the manufactuers could or could not end up slowing down their ability to innovate.
One only has to look at other examples of open standards to see that that is not an issue. In fact, the very technologies that allow us to post to these boards is one such example of how open standards has not negatively effected innovation. In fact, it has had an enormous positive benefit to innovation. Another example is open source software. IBM, Linksys, Sysco, Sun, Dell and Apple are just some examples of companies that have thrived from openness.
« Last Edit: February 05, 2006, 03:21:03 am by 61Dynamic »
Logged

BernardLanguillier

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 13983
    • http://www.flickr.com/photos/bernardlanguillier/sets/
Nikon RAW format
« Reply #8 on: February 05, 2006, 04:44:05 am »

OK, one last reply on this fascinating topic.

Quote
Sure there are legit reasons for encrypting data but the encrypted data in the NEF is nothing but a WB tag so the comparison to Phase One holds no water.
[a href=\"index.php?act=findpost&pid=57444\"][{POST_SNAPBACK}][/a]

Really? It appears that the "encrypted" AWB value in the .nef file can be "decrypted" thanks to the serial number of the body... which does obviously point to a given sensor batch.

The reason mentioned by Phase for not adopting DNG was that their RAW file contains references to the actual characterisitcs of the sensor used in the back as measured at the end of the assembly line. Something similar to a serial number.

In other words, both formats are based on the usage of non explicit values to determine the actual colors... I see at least some ground for comparison, even if I confess that I don't know the whole story.

Quote
Emphasis added. Nikon themselves stated that was the case in their press release which I linked to previously:
[a href=\"index.php?act=findpost&pid=57444\"][{POST_SNAPBACK}][/a]

You'll notice the usage of quotes arount the word "encryption". They are basically saying that only them can garantee the quality of the conversions of the .nef, through the usage of their SDK.

Most Westerns would say to this "go to hell, I am responsible enough to do my own conversions myself", but it isn't very surprising for someone used to doing business with Japanese companies that they see the scope of their responsibilities as going rather far. To my eyes, Nikon just didn't want to take the risk of having bad press because of files converted in a non Nikon certified way.

They probably went too far in doing so, but my interpretation differs from yours as far as their motives is concerned.

Quote
Emphasis added. At the time, if you wanted to access the raw data and the auto WB tag (sans converters that hacked it) you had to use Nikon's software. That is factual evidence of restricting the use of the file.
[a href=\"index.php?act=findpost&pid=57444\"][{POST_SNAPBACK}][/a]

How is "hacking" difference from "reverse engineering" which is done for all RAW files? The difference is only in the object you are working on, which takes us back to the question whether it was encryption or not...

Nobody sued the hackers, which can only mean that they were not hacking anything since there was nothing to be hacked.

Quote
One only has to look at other examples of open standards to see that that is not an issue. In fact, the very technologies that allow us to post to these boards is one such example of how open standards has not negatively effected innovation. In fact, it has had an enormous positive benefit to innovation. Another example is open source software. IBM, Linksys, Sysco, Sun, Dell and Apple are just some examples of companies that have thrived from openness.
[a href=\"index.php?act=findpost&pid=57444\"][{POST_SNAPBACK}][/a]

I happen to have a bit of knowledge on this topic... :-) Open standards, which is different from open source, has indeed contributed to helping some companies do business... BUT...

- the open standards were always defined by groups of companies that AGREED on the standards, which is obviously not the case with DNG,
- most of these open standards were geared at enabling the communication between systems and the companies that benefited from them were in the business of making these devices that had to communicate. The middelware was not where the battle was taking place then,
- in most fields, the developement of such standards took years of iterations, and the intermediate steps were not always succesful (ever heard of CORBA?),
- there are many examples, and databases are a good one, where the players have always resisted with success the usage of standardized objects because it DID slow their innovation and ability to add functions and capabilities, and to differentiate themselves from their competitors. The base is a standard, but the proprietary extensions are where each vendors adds real value, and this prevents actual interoperability.

My point with DNG not being a clearcut option right now is that there is no point in having a standard format if each camera manufacturer has to add proprietary extensions to be able to do what they want to do for our good...

Regards,
Bernard

61Dynamic

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1442
    • http://
Nikon RAW format
« Reply #9 on: February 05, 2006, 01:01:30 pm »

Quote
Nobody sued the hackers, which can only mean that they were not hacking anything since there was nothing to be hacked.[a href=\"index.php?act=findpost&pid=57446\"][{POST_SNAPBACK}][/a]
Pardon me while I smack my forehead with the palm of my hand. Hacking does not exist unless someone gets sued for it? Sorry, but there is not an ounce of logic in that statement.

Which seems to be par for the course in the rest of your argument. Clearly this conversation is not going to go anywhere but in circles.
Logged

BernardLanguillier

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 13983
    • http://www.flickr.com/photos/bernardlanguillier/sets/
Nikon RAW format
« Reply #10 on: February 05, 2006, 04:34:49 pm »

Quote
Pardon me while I smack my forehead with the palm of my hand. Hacking does not exist unless someone gets sued for it? Sorry, but there is not an ounce of logic in that statement.

Which seems to be par for the course in the rest of your argument. Clearly this conversation is not going to go anywhere but in circles.
[a href=\"index.php?act=findpost&pid=57472\"][{POST_SNAPBACK}][/a]

You are correct, the above statement taken out of its context can be a bit hard to understand, but I think that you must have gotten my point. This is a typical problem with these discussions, they take time, and you typically stop considering the full depth of the arguments of the interlocutor...  I won't throw the first stone at you, I have done that many times in the past as well...

For the sake of clarity, allow me to re-state it one last time.

My point on this given item was that there is basically no technical difference between reverse engineering and hacking, the main difference lies in that you are legally allowed to do one, and not the other. Besides, since "encryption" is a function of the intend of the right owner to protect or not, and since this intend can only be measured by deeds, whether the AWB was hacked or reversed engineered can only be measured by the existence of legal actions engaged by Nikon on the "hackers". Hence, since there has been none so far, I see no ground for calling the "reverese engineer" a "hacker". Is that logical enough for you?

I am of course willing to re-assess the relevance of this logical conclusion if someone more knowledgeable on the legal side of IP can show that some of the logical premisses are not correct.

Anyway, I understand fully what you are saying, but my view is that you are basically just re-stating the official party line, and that your whole argument is sustained only by the "belief" that Nikon wanted to prevent competition in the RAW converters area.

All I am saying is that this could be the case, but it could also not be the case, and the latter seems more likely when analyzing facts.

At least, you'll have to agree with me that IF their intend was to prevent competition (which we all do one way or another by the way), then they were <I>extremely</I> bad at doing it.

Cheers,
Bernard

BryanHansel

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 369
    • www.bryanhansel.com
Nikon RAW format
« Reply #11 on: February 05, 2006, 07:28:13 pm »

Quote
At least, you'll have to agree with me that IF their intend was to prevent competition (which we all do one way or another by the way), then they were <I>extremely</I> bad at doing it.

Bernard, I totally agree with you on every point you listed.  I think it's very hard to believe that Nikon, the powerhouse, a profitable company, a huge player, with very smart managers, business leaders, and a massive R&D department with massive resources which is able to build ground breaking cameras like the D2x and the D200, would be so incompetent at accomplishing to preventing competition if that was their goal.  I also find it so hard to believe that the big guys sat in their offices working out some conspiracy to start just with AWB and then end up encrypting the whole file by pulling a frog in the cold water stunt.  It's so hard to believe in these conspiracy theories and the logic jumps you have to make above and over the press releases, the copy on website, that it really points to a different intent than the conspiracies suggest.  
It takes a lot less logic to believe that Nikon's intent was good instead of bad, but anything more that I write probably won't change a single opinion.  This issue has turned out to be similar to the abortion issue, the gay marriage issue, and the death penalty issue in American politics.  The sides will never agree.

Bryan
Logged
Bryan Hansel
[url=http://www.paddling

Mark D Segal

  • Contributor
  • Sr. Member
  • *
  • Offline Offline
  • Posts: 12512
    • http://www.markdsegal.com
Nikon RAW format
« Reply #12 on: February 05, 2006, 08:26:01 pm »

There is no clear-cut answer to any of this. I'm no lawyer, and worse yet I don't specialise in matters of intellectual property, but let me put an economics spin on it, and spare us the economist jokes, I've probably heard all of them over the years.

Consumers have an interest in competition for reasons we all know. You maximize competition inter-alia by minimizing restricted access and maximizing public knowledge. Producers have an interest in creating monopolies at least long enough to get their investment back with a rate of return, and they do that inter-alia by restricting access and hiding knowledge.  

The rate of technological progress from which we all benefit would be slower without both competition and some legalized protection preventing competition from exposing investors to unacceptable risk. So how long is long enough, and in what respects............back to the beginning - it isn't clear-cut, so perhaps the ultimate winners are the patent attorneys!
Logged
Mark D Segal (formerly MarkDS)
Author: "Scanning Workflows with SilverFast 8....."

Ronny Nilsen

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 361
    • The Quiet Landscape
Nikon RAW format
« Reply #13 on: February 06, 2006, 03:00:16 am »

Quote
Anyway, I understand fully what you are saying, but my view is that you are basically just re-stating the official party line, and that your whole argument is sustained only by the "belief" that Nikon wanted to prevent competition in the RAW converters area.

What the intension of Nikon was is not known to us, and is also irrelevant. It's a piece of equipment with a set of features. All that is interesting to a user of that equpiment is the  consequence of the feature.

Nikon may have added encryption with the best of all intensions, but the consequence for the phographer is that it potentially restricts the use of the raw files. Intension is irrelevant to the assessment of the consequence for each user.
« Last Edit: February 06, 2006, 03:02:03 am by ronnynil »
Logged
Ronny A. Nilsen
www.ronnynilsen.com

gryffyn

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 323
    • http://www.tarafrost.com
Nikon RAW format
« Reply #14 on: February 07, 2006, 10:54:34 am »

Quote
- would have used a real strong encryption method, like a public key method, that is simply unbreakable,

Just had to correct this erroneous statement.  Public key encryption in the Nikon case would be easily breakable.  Think about it...the decryption key would have to be embedded in Nikon's own Capture software, so it would be pretty straightforward for someone to reverse engineer Capture, get the key and write their own decryption algorithm.

Unless Nikon tried to make the decryption server-side based, which would never fly with the customer community.
Logged
.....Andrzej
Pages: [1]   Go Up