Should Camera Companies adopt non-proprietary raw?

[jrsforums]

BTW as a relative newbie to this forum

You registered in Sept 2011

I registered in Feb 2005

[Rhossydd]

Just to get back to the OP.

Yes, I'd like to see a ratified and standardised raw format that every camera maker supported and offered. It really is the 'missing link' in digital photography at the moment.

[Tony Jay]

Just to get back to the OP.

Yes, I'd like to see a ratified and standardised raw format that every camera maker supported and offered. It really is the 'missing link' in digital photography at the moment.

Thank you.

Tony Jay

[stamper]

I'm starting this to ask:
If we, as a photographic community,
believe it is our best interests to support the adoption of a non-proprietary raw format, by the camera manufactures or not. If so why and if not why.

While reading a recent thread it was pointed out how large the LULA community is and since the camera companies just might (hopefully) be listening let's tell them what we think.
Personally I think that a standard format is necessary for archival reasons, for the long term preservation of our images, as well as being able to re-develop the image with newer processes, which, for me, is one of the main advantages that digital photography offers. Since the camera companies create the formats for their products, the ball seems to be clearly in their court.

I wonder why this thread was started considering the heat that the previous thread generated? This post doesn't have anything to say that wasn't said in the previous one.  What were the motives? 

[Tony Jay]

I wonder why this thread was started considering the heat that the previous thread generated? This post doesn't have anything to say that wasn't said in the previous one.  What were the motives? 

I doubt there was anything sinister here Stamper, the OP is fresh out of the box.

Tony Jay

[Bart_van_der_Wolf]

If you believe that the issue is not acute then you blithely disregarding obvious facts.
Andrew Rodney himself told you how he lost RAW images from first-generation digital cameras.

Hi Tony,

I'm not sure if/where Andrew, in another thread maybe, said that he lost Raw images.
Assuming he can read/copy the Raw files from disk, converting them to another format doesn't solve an issue.

Access to the code to interpret all the file data may be an issue when the original manufacturer ceases to exist, but there are plenty openly available solutions (DCRaw, RawTherapee, etc.), and a number of commercial ones, to decode the image data to a degree that an excellent image can be produced (sometimes with even better quality than the original solution). Without access to all of the original proprietary data (by means of an SDK, or with the source code), converting to another file format doesn't solve an issue.

To give you an idea, DCRaw apparently can convert (amongst others) the following Kodak Camera Raws (don't know which one Andrew Rodney is referring to):
•Kodak DCS200
•Kodak DCS315C
•Kodak DCS330C
•Kodak DCS420
•Kodak DCS460
•Kodak DCS460A
•Kodak DCS520C
•Kodak DCS560C
•Kodak DCS620C
•Kodak DCS620X
•Kodak DCS660C
•Kodak DCS660M
•Kodak DCS720X
•Kodak DCS760C
•Kodak DCS760M
•Kodak EOSDCS1
•Kodak EOSDCS3B
•Kodak NC2000F
•Kodak ProBack
•Kodak PB645C
•Kodak PB645H
•Kodak PB645M
•Kodak DCS Pro 14n
•Kodak DCS Pro 14nx
•Kodak DCS Pro SLR/c
•Kodak DCS Pro SLR/n

I have no idea what the proprietary data in those Raw files could add to the conversion quality, but I do know that the demosaicing quality has improved a lot since those cameras were current, so maybe todays conversion quality trumps the old, even without the missing info. I'm glad that we didn't convert the original Raws into something that is an interpretation/abstraction of the original data, and afterwards came to realize that we didn't convert it as good as we can today. Still, it would be nice to also understand the unknown bits.

You believe that your RAW images are safe - for how long: five years, ten, fifty years?

I don't think it's an issue about being safe. Safe from what? It can't be floods or Global warming, because converting to another format would not make any difference in that respect.

The only potential issue from a user perspective seems to be the interpretation of some of the proprietary bits, the non-imaging sensel data, and the undisclosed maker notes in the EXIF metadata (as far as they are related to image-data decoding). Again, without access to an SDK or original source code that utilizes that data, converting to another file format doesn't change anything for the better.

We also should realize that much (most?) of the turmoil is caused by companies like Adobe who, for commercial reasons, force people to update the image editing software, because newer camera models are deliberately locked out. It is therefore no surprise that Adobe also offers a 'DNG solution' (for an issue they partially create themselves), which would benefit them by not having to interpret multiple Raw formats.

Frankly, I'm increasingly amazed by the high conversion quality that programs like RawTherapee can produce, even without intimate knowledge of the unknown bits ..., and some of the issues I see with the ACR/LR Raw conversion quality. It would seem wiser if they spent more resources on that conversion quality before others (also lacking the proprietary info) take too much of a lead.

Cheers,
Bart

[Tony Jay]

Bart:

I cannot speak for the details of what Andrew was referring to in another thread dealing with the same subject matter as this one.
Maybe the information that you have shared may benefit Andrew - who knows.

Safe in the context that I referred to the concept includes everything that would be important in digital asset management, partly but certainly not exclusively, it includes being able to use that RAW image that one may have so assiduously otherwise preserved.
If the format of that RAW file is orphaned (no software that recognizes it) in the future then what is the point of preserving it?

Actually writing that paragraph has stimulated my memory about another issue that Andrew raised with regards to his RAW files that became orphaned. It is also an issue that I have raised independently.
Just because an image format is technically accessible does not necessarily mean that for most of us it is practically accessible.
To use an extreme example: If the FBI requires an image from a file format that is practically orphaned, if it is technically possible, then the considerable resources of the FBI can be put to the task. If national security is at risk the FBI will find a solution.
If accessing RAW images requires us to find and run long out of date software on obselete computers running superceded operating systems for practical purposes this may not be possible even if it is technically possible.
There is no doubt that is a current reality and not just theoretical posturing.

Currently our hope is based on the fact that no more proprietary RAW formats are lost and third-party RAW converters continue to support formats for cameras that are no longer made by companies that no longer exist and do not go out of business themselves, and, if they do, that new players will reverse engineer those formats all supposing that they can access the software development kits in that situation.
I deliberately made that last sentence an absolute mouthful to demonstrate what I think is a dodgy proposition.
Again this is not a theoretical situation - many formats are already gone, dead and buried, along with the companies that engineered them.

How much easier just to have a universal open RAW format so even if camera-makers insist on continuing to spawn new RAW formats for every camera model that they produce at least we can convert those image timeously to the open format secure in the knowledge that at least the open RAW format WILL BE SUPPORTED by RAW converters into perpetuity.
(Of course there are lots of other ways that images need to be protected into the future but this is a fundamental step.)

The ideal situation is that each and every camera-maker actually adopts that open RAW format as the RAW format for every camera model that they produce.
Amongst other benefits this will remove that annoying situation of buying newly released camera models and having to wait several months for the third-party RAW converters to reverse engineer the RAW format and update their software.
It will also remove any need for us to convert our files from format to format.

The bottom line is that no RAW file format is safe.
The brutal Darwinian process that is especially prevalent in the electronics industry means that not even companies that currently hold iconic status are necessarily safe for the long-term.
Perhaps our own demise will precede that of Canon or Nikon or Phase One or whatever in which case the issue may not concern us personally but on a society-wide basis gazillions of digital images may be lost to the future.
Maybe one day we will have digital archeologists cum anthropologists whose entire job description is to try and figure out details of our existance from digitalized images and documents in formats where the initial challenge is just to decode the format itself.
Maybe the previous statement is a poor attempt at levity but the implications are real enough.

Tony Jay

[PeterAit]

Peter, just to enlarge on Jeff's point.
This is NOT a DNG crusade.
This is all about the adoption of a standardized open RAW format.
DNG is the only candidate in play currently but it may not be the format eventually adopted.
What if many people converted all their RAW files to DNG and DNG was not adopted as the standardized RAW format.
If Adobe were to subsequently go out of business and DNG lost its support then all those DNG files would effectively be orphaned.
BACK TO SQUARE ONE!

So, the only way forward is the adoption of a standardized RAW format.

Tony Jay

Tony,

I understand, and I never thought it was a "DNG crusade!" Still, I just don't see any practical advantage to have a single accepted RAW file standard, therefore I question the wisdom of putting effort into that goal. Given that DNG is documented, even if Adobe withdraws support, the files will always be readable. Any competent programmer could write an application to translate DNG into whatever new "file format of the month" is being used.

[Tony Jay]

I understand, and I never thought it was a "DNG crusade!" Still, I just don't see any practical advantage to have a single accepted RAW file standard, therefore I question the wisdom of putting effort into that goal. Given that DNG is documented, even if Adobe withdraws support, the files will always be readable. Any competent programmer could write an application to translate DNG into whatever new "file format of the month" is being used.

I agree that it is possible for anyone with the knowledge to write software that recognizes DNG but that in no way guarantees that it will actually happen.

Also, It would be much preferable if none of us had to deal with this issue of porting our images from format to format to keep them current with software that can read them.
In the bigger picture considering the billions of RAW images that are being shot daily it doesn't take Einstein to figure out that not a high percentage of them are going survive the impost of being ported from format to format purely because the magnitude of the task is just overwhelming and it takes a lot of effort and motivation from a lot of people to keep it up.
You personally may be motivated enough to do it and perhaps large organizations with vested interests in the images that they own will do it but the bottom line is that the losses will be staggering.

If we had a universal RAW format produced out of the camera then none of these issues come up.
Any RAW converter/developer will recognize the image.
No reverse engineering formats required, no delays waiting for formats to be supported.
Images can be archived in the knowledge that in decades to come they will probably be recognized by software currently in production.
None of this cancels out the rest of the links of the chain required for robust digital asset management into the future but it is an essential first step.

The issue is much much bigger than one's personal image collection.

Tony Jay

[PeterAit]

Well, Tony, I think that you and I just have different outlooks on the future, with you tending to fear the worst. Why fret about a possible future situation that almost surely will never come to pass, and if it does will be easily rectified? I have more important issues to fret about - like whether taking the lens off is a clockwise or counter-clockwise twist (don't laugh, my Nikon and Panasonic are opposite and I am forever getting it wrong!).

[jrsforums]

You personally may be motivated enough to do it and perhaps large organizations with vested interests in the images that they own will do it but the bottom line is that the losses will be staggering.

Yes they will....and it has nothing to do with RAW format.

Of all the analog negatives ever made....what % of them survive.  How many of them wound up in landfills when "Aunt Sally" died?  Assuming she had not already thrown them out to make room in her closet.

Even in the film industry, where great effort was taken to preserve the original works, much was lost...for many reasons....none of which was format related.

In digital...how many SD cards have been lost...which were never copied to a computer.  When the family photographer, "Uncle Harry" passes,  who is going take on the role to find all his digital negatives and preserve the bits someplace....either for memories of future family members, historians wanting a picture of a past era....or, maybe, a museum, when Uncle Harry is posthumously discovered as a great artist.

It is a much bigger issue...too big to be narrowly focus on an open RAW format.

[Tony Jay]

Peter, sadly the issue is not just a theoretical future one.
Many RAW formats are already dead and buried and who know how many millions (?billions) of RAW images already shot in those formats are just taking up useless space on hard drives, servers and other storage locations.
Those images are already dead men walking because there is no practical way of opening them right now.

The biggest fallacy appears to be that this issue is an airy fairy theoretical issue that may only happen in the future.
It is already part of our reality now, recognized, or not.

You may be confident that your .NEF files will be supported ad infinitum - I am not - and recent history is on my side.

Tony Jay

[jrsforums]

Peter, sadly the issue is not just a theoretical future one.
Many RAW formats are already dead and buried and who know how many millions (?billions) of RAW images already shot in those formats are just taking up useless space on hard drives, servers and other storage locations.
Those images are already dead men walking because there is no practical way of opening them right now.

The biggest fallacy appears to be that this issue is an airy fairy theoretical issue that may only happen in the future.
It is already part of our reality now, recognized, or not.

You may be confident that your .NEF files will be supported ad infinitum - I am not - and recent history is on my side.

Tony Jay

Please name one widely used (millions of RAW images) RAW format that cannot be opened.

The only ones that was put forward in Jeff's arguments were Kodak DCS and Photo CD.  Subsequent, reliable, information showed that all these images were accessible and usable.

[Rhossydd]

I just don't see any practical advantage to have a single accepted RAW file standard

Really ?

A bigger question is why do camera manufacturers keep on changing formats and making it harder for software vendors to support their products ?
Does each new iteration of CR2/NEF/etc on every new model really help anyone ?

[Tony Jay]

jrsforums: as per usual your selectivity is astounding.

No-one, and certainly not I, has suggested that the question of an open RAW format, by itself, will solve every issue related to the safe and perpetual archiving of digital images.

Your are just trumpeting your ignorance from the rooftops.
There was a reason why it I stressed the term 'Digital Asset Management' in many of my posts. It was to emphasize that the issue is much broader than just adopting an open RAW format.
I invited you in one of my posts, if memory serves, to go and read up about the topic.
Again, if memory serves, you derisively dismissed me.
I also seem to recall earlier that you admitted that digital asset management was not an area of strength for you.
Do you really suppose that your latest comments are groundbreaking news to me, or anyone else for that matter?

The whole point of the exercise is to limit the attritional loss of RAW files into the future that could otherwise be saved and accessible into the future.
As I have just mentioned to Peter Ait the issue is much bigger than one's personal image collection.

You have shown conclusively that facts will not sway you.
You have complained, bitterly, that this topic, is depriving you of the opportunity to do other things that you would rather be doing.
Yet you keep coming back to layer inanity on inanity.
My medical training suggests that you have an interesting five-axis diagnosis.

This is the last post of yours to which I will respond.

Tony Jay

[jrsforums]

Really ?

A bigger question is why do camera manufacturers keep on changing formats and making it harder for software vendors to support their products ?
Does each new iteration of CR2/NEF/etc on every new model really help anyone ?

That is a good question....why do they do it?  In fact, why do the ALL (?) do it?

Do you think they do it to disrupt, get even, whatever with the software vendors?

Maybe they want to disrupt and delay the early adoption (purchasing) of their new camera until the software makers support the new RAW?

Some have suggest the manufacturers are just arrogant, stupid, prideful, sloppy, etc. when creating their new RAW format.

I have no fact...only suspicions....if someone really knows, we would all be made wiser by it.

However, I do know how many development operations work.  They are going to do things that give them the best product (features, operation, speed, etc.) and do it in the most efficient manner.  Expense is a key determinant to the actions taken.  If a standard format reduced, rather than increased expense, it would be used (unless it effected other parameters, such as operational speed or components required).

Forgetting, for the moment, the internals to the camera.  Canon provides RAW conversion software, which they make no money on.  However, even though they know the internals of RAW changes way before the rest of the world, they need to incorporate those changes into their code...which drives an expense, which will effect the overall cost, therefore the price, of the camera.  If this cost could be saved without causing increased cost or other issues you can be sure the Canon financial people would on this like flies on stink.

[jrsforums]

jrsforums: as per usual your selectivity is astounding.

No-one, and certainly not I, has suggested that the question of an open RAW format, by itself, will solve every issue related to the safe and perpetual archiving of digital images.

Your are just trumpeting your ignorance from the rooftops.
There was a reason why it I stressed the term 'Digital Asset Management' in many of my posts. It was to emphasize that the issue is much broader than just adopting an open RAW format.
I invited you in one of my posts, if memory serves, to go and read up about the topic.
Again, if memory serves, you derisively dismissed me.
I also seem to recall earlier that you admitted that digital asset management was not an area of strength for you.
Do you really suppose that your latest comments are groundbreaking news to me, or anyone else for that matter?

The whole point of the exercise is to limit the attritional loss of RAW files into the future that could otherwise be saved and accessible into the future.
As I have just mentioned to Peter Ait the issue is much bigger than one's personal image collection.

You have shown conclusively that facts will not sway you.
You have complained, bitterly, that this topic, is depriving you of the opportunity to do other things that you would rather be doing.
Yet you keep coming back to layer inanity on inanity.
My medical training suggests that you have an interesting five-axis diagnosis.

This is the last post of yours to which I will respond.

Tony Jay

I have only responded to statements you have made.  If you have expressed them poorly, I cannot help it.

I have done some reading, mostly without your suggestion.
From: http://dpbestflow.org/node/634
Long-term accessibility
In the relatively short history of raw photography, there have been some formats that have become obsolete, and therefore largely inaccessible. The early Kodak cameras used a patented method to record color, and the software to read it has not been updated to work with modern computers. This has put the images at risk of inaccessibility.

For modern DSLR cameras (generally those produced after the year 2000), the risk of inaccessibility is significantly less. Even if commercial software packages eventually drop support for very old cameras, open source software will probably continue to provide access. The program dcraw, written by Dave Coffin, includes support for a very large number of cameras. Because it is open source, it is unlikely to simply go away. The codebase can be picked up and modified in the future by any number of independent developers.

You don't agree with my position, you call me stupid.  Then that I "complain, bitterly"...when one I responded "Good grief" to one of your arm waving posts.  Then that my responses are "inanity", mainly because you don't agree with them, not because they are not well thought out.

Doctor, if you are one, rather than playing remote, biased, diagnostician in an attempt to wn your arguement or to diminish your opponent....HEAL THYSELF

BTW....I do not see any FACTS from you.  I am waiting for the name of the one lost RAW format, with millions of images, which cannot be recovered.

[Bryan Conner]

My momma used to tell me that I would argue with a fence post.  I wish she were here to read this thread now. 

[michael]

OK guys. End of personal attacks.

On topic please, or it's closed.

BTW, one major university with a substantial collection of contemporary photography told me about a year ago that they were not accepting raw files from photographers work that they were collecting unless they were in DNG. Reason? They feared the inability for future generations to be able read the data.

They have a program in place to migrate over time to new storage formats (that's a given) but they believed (rightly or wrongly – and I think rightly) that they didn't believe and trust in proprietary formats for archival and historical storage purposes. They also told me that a national consortium of museum conservators were planning on making DNG their defacto standard across institutions.

So, my personal observation is... where's the beaf?

[Bart_van_der_Wolf]

BTW, one major university with a substantial collection of contemporary photography told me about a year ago that they were not accepting raw files from photographers work that they were collecting unless they were in DNG. Reason? They feared the inability for future generations to be able read the data.

Hi Michael,

That's interesting, and very strange at the same time. A bit similar to storing an Ansel Adams negative (or a scan of it) to recreate one of his prints ... And in these times, a digitally enhanced output file may even be impossible to create from a single Raw file unless one also has the different adjustment layers and or Raw conversion versions that led to the final image. Very strange, unless they teach journalism.

They have a program in place to migrate over time to new storage formats (that's a given) but they believed (rightly or wrongly – and I think rightly) that they didn't believe and trust in proprietary formats for archival and historical storage purposes. They also told me that a national consortium of museum conservators were planning on making DNG their defacto standard across institutions.

One can only hope they don't throw the original Raws away, and embed the full Raw in the DNG. They'll have to spend some extra money on storage capacity as well then.

Cheers,
Bart