Michael's DNG comment

[deejjjaaaa]

If the camera makers were to have DNG as a native output what would be the difference between that and what you get from converting NEF/CR2/etc --> DNG in batch post processing? If there's no difference why keep the raw file? is there any information in the RAW file that would not be in the DNG?

if camera maker has DNG as it is native in camera raw - that is OK, it is no different than non DNG - however the question is = how to innovate when you need Adobe to adjust the standard... which means you have to disclose what you are going to do to competition and then wait for Adobe to allow you to do that... or just start using a BLOB field (like DNGPrivateData) in DNG file to write all your data there... how good is that ?

[deejjjaaaa]

There is nothing, and I mean NOTHING that a camera company can come up with that Thomas the the ACR engineers can't deal with..

Schewe - can you personally ask Adobe people to support Sigma raws in a proper manner  ?

[deejjjaaaa]

If you want to do some reading, read this...Digital Preservation.

I prefer (and do myself) the following :

"...for archival purposes I recommend (and practice) safely storing the original raw files in the form that they came off the camera, whether they be DNG or non-DNG..." (ц) Eric Chan

there were enough bugfixes and silent changes in how Adobe DNG converter operate to say that it is not wise to discard the original raw file...

PS: I wonder whether Library was made aware about those and if Library discarded the originals (of non DNG raw files) after those were converted by prev. versions of Adobe DNG converter...

[deejjjaaaa]

puts our cultural heritage at risk

+

there are no "secrets" in the raw file–all of that can be decoded.

so where is the risk then  ?

[deejjjaaaa]

and deejjjaaaa, if you don't have something substantive to contribute regarding the technical limitations of DNG (so it can be advanced) I suggest you quit trying to take the adversarial position

thank you for suggestion, but I will ignore it - I still did not hear what Panasonic was supposed to do about optics correction... delay release of their cameras to market (and suffer financially) to wait for Adobe to accomodate that in DNG ? huh ?

[Schewe]

I still did not hear what Panasonic was supposed to do about optics correction... delay release of their cameras to market (and suffer financially) to wait for Adobe to accomodate that in DNG ? huh ?

Well, the opcode is now in DNG so it's currently a moot point. As far as the timing of when Panasonic developed the cameras/lens that needed so much optical lens correction because of really bad barrel distortion and the inclusion of opcodes into DNG what what sort of time period? Do you know? I don't...all I know is that the ACR/DNG engineers worked with several companies (not Nikon/Canon) to get lens corrections in place for ACR/LR and DNG. I would call that a good result, wouldn't you?

[Schewe]

Schewe - can you personally ask Adobe people to support Sigma raws in a proper manner  ?

deejjjaaaa, can you ask Sigma to work with Thomas and Eric to do so? Their raw file format really and truly sucks.

[digitaldog]

if camera maker has DNG as it is native in camera raw - that is OK, it is no different than non DNG - however the question is = how to innovate when you need Adobe to adjust the standard... which means you have to disclose what you are going to do to competition and then wait for Adobe to allow you to do that... or just start using a BLOB field (like DNGPrivateData) in DNG file to write all your data there

Not so! There is a switch. You want the proprietary goodies, select Native raw. You want DNG and don’t give a crap about that proprietary stuff, switch on DNG. Or just give us a Raw+JPEG, Raw+DNG option. Simple. Nothing stops all these wonderful proprietary processes the manufacturers promise the unsuspecting customer....

[BJL]

Jeff,
    To one of my questions

Does anyone know if there is any effort to have something like an ISO or JPEG comittee mechanism for developing a raw file format? Either starting with DNG released by Adobe, or from another starting point?

you eventually gave the reply I was hoping for

<expletives etc. deleted> ... Adobe has already offered DNG to the ISO for their next TIFF-EP update (last I heard "they're working on it"–the ISO is pretty slow to change standards).

That could be a good route: a new format initially developed by one competent company or research group (faster than waiting for a standards body to do it all) and then handed over to a standards body for maintenance, so that other companies do not have to worry about the creator keeping too much control or a competitive advantage. Consider the fights over Java, first Sun vs Microsoft and now Oracle vs Google, for a hint as to why companies are naturally uncomfortable with committing to a standard that is to a large extent controlled by another company --- no matter how talented and nice Thomas Knoll and Eric Chan are.

So maybe you can answer my other main question: I read your article Digital Preservation published in Adobe's PhotoshopNews, and my remaining question is:

how much of what you dislike and how much archivists worry about different camera companies using different raw formats rather than Adobe's DNG would be solved by "Disclosure" and "Transparency", with or without standardization on a single format?
By this I mean the publication of full specifications for each raw format, which could then either be stored with any archives using that format, or used to convert to an approved archival format.

[ErikKaffehr]

Hi Jeff,

Thanks for your well articulated arguments. Actually, I am a strong proponent of DNG. On the other hand I have found that some programs Bibble Pro and DxO would not open my DNG files. This was before LR3, with it's much improved processing pipeline.

At that time I used NoiseNinja to reduce noise, and BibblePro implemented NoiseNinja in the processing stage, so I really wanted to try out BibblePro. (Un)fortunately BibblePro 5 sucked and LR3 was introduced with it's new processing pipeline. I was a very early adopter of that beta and even had some contacts with Eric Chan, who seems to be a most helpful character at Adobe. I event sent a sample file with impulse noise to Eric.

Processing pipeline 2010 turned out great, as published here: http://echophoto.dnsalias.net/ekr/index.php/photoarticles/32-new-raw-processing-pipeline-in-lightroom-3-beta-2 , so I lost interest in BibblePro.

I mailed Bibble pro about the need to support DNG, and got their view, valid or not.

I'm very satisfied with Lightroom. I was actually thinking about writing something like Lightroom but realised that it would take me 300 years of work. So, I'm much thankful that Adobe did it for me. Saved me 300 years of work.

Best regards
Erik

Ps. So, Adobe give us a 4.2 that handles Sony RX100 raws with geometry correction, please!

Again, it would be useful if you could stick to the friggin' facts...Eric Hyman didn't "like" DNG (for some reasons I could mention not in a public forum). Bibble didn't offer full DNG support but only offered DNG support for cameras they already decoded and supported natively–which is not true DNG support. DxO support DNG but must demosiac the file to do the lens correction (duh) and writes out linear DNG.

There are a few cameras out of the 275+ cameras supported whose private maker note metadata is not  moved and stored in a DNG...Sigma/Foveon raw files are an example because their raw files are, well goofy (and again, totally undocumented)...there are some early Canon raw files whose private maker notes were, in essence, encrypted (more by accident vs. intentionally) so that metadata isn't written (from what I remember with discussions from Thomas-but I may be wrong). Metadata that DNG doesn't understand but it properly formed and stored safely is move into the DNG even if ACR/DNG doesn't know what the metadata is or means.

Fact is, if Thomas wanted to, I'm pretty darn sure he could take a proprietary raw file form most any current camera (except Sigma), convert the file to DMNG and the decode the DNG and write back a proprietary raw file that would fool even the camera companies' software and be able to be processed in the camera software.

Again, anybody reading this that doesn't work for Nikon and Canon are are not pissed as hell at the current state of the industry, you are part of the problem, not part of the solution...there is simply no legitimate reason to let Nikon and Canon get away with this crap–which is the point Mike was trying to make.

[deejjjaaaa]

deejjjaaaa, can you ask Sigma to work with Thomas and Eric to do so? Their raw file format really and truly sucks.

it is not about the format - Adobe knows how to read it... it is about how Adobe process the data that was read... which is not process 2012 and not even process 2010 but some prehistoric processing w/ forced NR which is more like 2003 or so.

PS: plus - I do not have Sigma's ear - but you have Adobe's 

[deejjjaaaa]

Well, the opcode is now in DNG so it's currently a moot point.

it speaks about MO - certainly Adobe can support anything - but the question is why manufacturer has to ask Adobe's permission, moreover - see my point about Google, MS, others - tomorrow Adobe might start making cameras and then what ? and even if decisions about DNG changes will be done by some form of industry committee where Adobe is just one vote w/o veto power - that means disclosing some things to your competition, sitting there, ahead of time... 

[ErikKaffehr]

Hi,

The question I may ask is who is going to support a given raw format in 20 years. I guess Adobe will stay around, but even without Adobe, DNG may survive because it is open and well documented.

Best regards
Erik

it speaks about MO - certainly Adobe can support anything - but the question is why manufacturer has to ask Adobe's permission, moreover - see my point about Google, MS, others - tomorrow Adobe might start making cameras and then what ? and even if decisions about DNG changes will be done by some form of industry committee where Adobe is just one vote w/o veto power - that means disclosing some things to your competition, sitting there, ahead of time... 

[Alan Goldhammer]

it speaks about MO - certainly Adobe can support anything - but the question is why manufacturer has to ask Adobe's permission, moreover - see my point about Google, MS, others - tomorrow Adobe might start making cameras and then what ? and even if decisions about DNG changes will be done by some form of industry committee where Adobe is just one vote w/o veto power - that means disclosing some things to your competition, sitting there, ahead of time... 

This is not correct.  When I was still working full time, I was the part of two standards organizations and the whole point is to bring all the stakeholders together to work on an open standard that is published and available for all to adopt.  Adobe was the original developer of the DNG but has placed it out there for all to use and there are some camera manufacturers (Pentax, Leica and maybe a couple of others) that use it for capture.  Ideally, the standards organization would bring all the camera manufacturers and software companies together along with interested participants from the public sector, such as those who have responded to this thread, to work out all the key issues and refine the standard.  This can be a multi-year process if there are complicated issues to be resolved (I remember tracking the C++ programing language standard a while back and that one certainly took a while).  At the end of the day you get a standard that benefits everyone and helps advance the technology.  I cannot believe that Canon and Nikon are making much money from their RAW processing software and it was my experience with Nikon's Capture NX2 that it really didn't offer me much at all relative to LR and PS (I don't have any familiarity with any of the other software out there).  It's time for Canon, Nikon, and other camera makers to quit thinking they are software providers and focus on what they do best, making cameras and lenses.  Settling on a DNG standard will certainly free up resources at those and other camera companies.

[Schewe]

how much of what you dislike and how much archivists worry about different camera companies using different raw formats rather than Adobe's DNG would be solved by "Disclosure" and "Transparency", with or without standardization on a single format?
By this I mean the publication of full specifications for each raw format, which could then either be stored with any archives using that format, or used to convert to an approved archival format.

Disclosure and transparency would be useful but that's really only 2 of the seven sustainability factors. With almost 300 raw file formats out there right now, one of the biggest current issues is one of Adoption. Going back to the early 1990's and the Kodak 400 series cameras (which Kodak has ceased any support for) till now, we have about averaged about 15 new undocumented, proprietary raw file formats a year in the 20 years or so of digital photography. The moving average is much higher because very few cameras started coming out till after 2002 or so. So, take that 15 new file formats forward, what's gonna happen in another 10 years, 25 years, 50 years? Will anybody even support all the new and old file formats forever? It's far less likely with so many file formats out there now and new ones every 6 months of so.

Unchecked by adopting some sort of standard, it's only getting worse, not better. Unless and until somebody stands up to the Nikons & Canons and says, enough...and holds their toes to the fire and forces them to adopt standards, this stuff isn't gonna get better.

That's why it's so critical that photographers know this stuff and don't get hoodwinked by a few doubting Toms who spread FUD and keep letting the camera companies off the hook for the mess they have created.

That's why it's important that every photographer that buys a new camera and then complains that Adobe is taking too long to support their new baby, get their butts kicked so they know who to really blame...the camera companies! We just went through this again with the Nikon D800 & Canon 5D MIII...every time some idiot stands up and screams about Adobe dragging their feet, read them the friggin' riot act and tell them THEY are the problem...but a new camera before the rest of the industry can adopt the new raw file formats is completely the camera companies' fault–and their fault for enabling the camera companies to pull this crap repeatedly each and every time they foist a new camera on the industry.

Look, the camera companies have fallen into this trap. For the vast majority of the history of photography, it was never the camera companies in charge of conservation and preservation. It was the film and paper companies who have the longer history (and even they weren't too good at it...got many unfaded Kodacolor prints from the 1970's left).The camera companies' traditional role has been to make a light tight camera that could hold the film (without scratches) and form a light onto the film. Nikon and Canon know diddly squat about conservation and preservation...

You really want the foxes in charge of the hen house? I don't...

So each time some yoyo says, "but DNG will retard the camera companies' freedom to develop new cameras" (read make new file formats) tell darn tooting and maybe that's a good idea for the industry for the long term. Short term camera churn is how we got into this mess in the first place.

[Schewe]

but the question is why manufacturer has to ask Adobe's permission

That's a complete distortion of the facts...they don't have to ask Adobe's permission...all they have to do is license for free, the DNG spec and adhere to the minimum standards imposed. No problem...camera companies have already done so. It hasn't seemed to negatively impact Leica one iota (and in fact has been very good for Leica–who is selling every camera they can make).

Again, what you are saying is bullshyte...

Heck, CR2 and NEF files are so close to DNG (now, after DNG taught them how to write well formed file formats alas still undocumented) that the odds are all of the currently shipping cameras could have a firmware update to enable writing DNGs as an option. Why don't the camera companies want to? Because there's no pressure for them to do so...because photographers continue to let them off the hook.

Yes, there are technical challenges to adopting DNG...and yes, Adobe has to keep updating the spec to keep on track but that's stuff that has to be done anyway. Every new camera Canon and Nikon introduce means they are forced to upgrade their own software. If they adopted DNG their life could actually be easier. But they are too arrogant and ignorant to do so.

[Bryan Conner]

I agree that DNG is not a bad idea: it is a good idea overall, much better than the Tower of Babble scenario we are in now. 

Are there any steps that we (consumers) can take to encourage the camera manufacturers to adopt DNG?  The only things that I have thought of so far is that we could not purchase a new camera until the raw file is supported by your favorite raw processor (Adobe, C1, Bibble etc.)  The second thought is one that someone suggested in a forum a while back (I can not remember the forum nor the person). We could make a statement by sending the software cd that came with your camera back to the manufacturer along with a note stating that the software does not support dng, therefore it is not needed.

Any other ideas or suggestions? 

[Schewe]

Any other ideas or suggestions? 

I think the single biggest thing we can do is make absolutely sure that when a photographer bitches about the fact Adobe isn't supporting the new camera de jure, that we all make sure the blowback goes directly and firmly into the camera company's face and not the rest of the software industry. The more blowback that Nikon and Canon get, the more closely they will evaluate the benefits of adopting some standards. I don't think it's realistic to expect people NOT to buy the new cameras...I do think it's critical that the blame is correctly channeled...don't let the camera companies off the hook, it's their friggin' fault.

Back when Nikon released the the D2X camera, the highest Google search results about the camera were NOT reviews of the camera but the controversy of Nikon encrypting the white balance data. To be precise, Thomas was a major protagonist in that drama, but the net result was Nikon contacted Adobe and said "how do we fix this situation?". The solution was Nikon biting the bullet and releasing a mini-SDK that enabled Thomas to decode the D2X white balance.

The camera company does indeed respond to pressure...but if there's some yoyo that is backing the camera company play, it seriously dilutes the message and lets the camera makers off the hook. Nikon and Canon are run by bean counters...if the bean counter deem sticking with proprietary raw file formats is hurting the bottom line, guess what, their perspective will change. If photographers raise their legs and take it, then they are not predisposed to change.

Make no mistake, regarding the current situation, DNG or some other standard is not a technical issue, it's a political issue.

[Bryan Conner]

I think the single biggest thing we can do is make absolutely sure that when a photographer bitches about the fact Adobe isn't supporting the new camera de jure, that we all make sure the blowback goes directly and firmly into the camera company's face and not the rest of the software industry. The more blowback that Nikon and Canon get, the more closely they will evaluate the benefits of adopting some standards. I don't think it's realistic to expect people NOT to buy the new cameras...I do think it's critical that the blame is correctly channeled...don't let the camera companies off the hook, it's their friggin' fault.

Back when Nikon released the the D2X camera, the highest Google search results about the camera were NOT reviews of the camera but the controversy of Nikon encrypting the white balance data. To be precise, Thomas was a major protagonist in that drama, but the net result was Nikon contacted Adobe and said "how do we fix this situation?". The solution was Nikon biting the bullet and releasing a mini-SDK that enabled Thomas to decode the D2X white balance.

The camera company does indeed respond to pressure...but if there's some yoyo that is backing the camera company play, it seriously dilutes the message and lets the camera makers off the hook. Nikon and Canon are run by bean counters...if the bean counter deem sticking with proprietary raw file formats is hurting the bottom line, guess what, their perspective will change. If photographers raise their legs and take it, then they are not predisposed to change.

Make no mistake, regarding the current situation, DNG or some other standard is not a technical issue, it's a political issue.

Interesting info.  I do not remember the Nikon situation as I am a Canon shooter.  I have thought about your statement "I don't think it's realistic to expect people NOT to buy the new cameras."   I can not see myself buying a new camera before the raw format is supported by Adobe.  Lightroom and/or ACR are just as critical to my photography as a camera.  Sure, it would be nice to have the new body in order to learn all of the new features etc, but I can do that these days in a few hours at the most.  After I finished with the ooooohs and aaaaaaahs over my new toy, I want to dive into a raw file in Lightroom or ACR.  If I can not do that, the fun is over.  Looking at jpegs, to me, is the same as almost having sex. 

Maybe the next time I am considering buying a new camera and the raw format is not yet supported by Adobe, I will sent an email to the manufacturer, or a phone call, and complain because I have to wait to be able to fully use the camera.

Thank you Jeff for using your position to give us a prominent voice that may fall upon the correct ears enough times that they are either wise and follow the advice, or they are sick of hearing about it.....LOL.

[dreed]

Thanks for your well articulated arguments. Actually, I am a strong proponent of DNG. On the other hand I have found that some programs Bibble Pro and DxO would not open my DNG files.

Was there a reason for this, such as the DNG files being of a newer version than your software would support?

Maybe the next time I am considering buying a new camera and the raw format is not yet supported by Adobe, I will sent an email to the manufacturer, or a phone call, and complain because I have to wait to be able to fully use the camera.

Nobody takes email very seriously.

Ring up Canon/Nikon support, mention that you are not satisfied with the product that you have bought and wish to contact the local CEO (or whatever) to convey your displeasure. Ask for the postal address of the local CEO and write them a letter or ask for their Fax number and fax them something.

Anyone that you can reach by phone is unlikely to be anyone of consequence (just someone in a call centre).

Bits of paper with words written and signatures on them tend to garner more attention.

I think the single biggest thing we can do is make absolutely sure that when a photographer bitches about the fact Adobe isn't supporting the new camera de jure, that we all make sure the blowback goes directly and firmly into the camera company's face and not the rest of the software industry. The more blowback that Nikon and Canon get, the more closely they will evaluate the benefits of adopting some standards. I don't think it's realistic to expect people NOT to buy the new cameras...I do think it's critical that the blame is correctly channeled...don't let the camera companies off the hook, it's their friggin' fault.

And in that, I think it is time reviewers everywhere started mentioning this and marking cameras down explicitly for it.
"Lovely new camera from Canikon but the raw file format is not recognised by Lightroom/Photoshop/whatever. If you're interested in buying it, understand that you won't get the best from it with the software that is provided."

Yes, there are technical challenges to adopting DNG...and yes, Adobe has to keep updating the spec to keep on track but that's stuff that has to be done anyway. Every new camera Canon and Nikon introduce means they are forced to upgrade their own software. If they adopted DNG their life could actually be easier. But they are too arrogant and ignorant to do so.

The only reason I've been able to come up with myself is that adopting DNG might impose limitations on how they can introduce new innovations with their cameras and/or that they might be too obvious to the competition. i.e. the undocumented raw file format somehow gives them a tiny bit more secrecy about any additions/changes that they've made when compared to DNG.