Luminous Landscape Forum
Raw & Post Processing, Printing => Digital Image Processing => Topic started by: rgs on December 29, 2013, 08:57:05 am
-
This question is about DNG or RAW for original camera files. Which format is your choice and why. I'm really more interested in your answers than in the poll but thought it might be interesting o see numbers. I usually don't save JPEG (unless it's the only thing available) and scanner files are TIFF. Files edited in PS are either TIFF or PSD.
-
Opening up a big can of worms here ;D
Here are a few reasons I use DNG:
http://www.ppmag.com/reviews/200709_adobedng.pdf
The format since this article has evolved with more functionality (ability to embed DNG profiles in container, Lossy DNG, Fast Load Previews etc). But biggest reason, it's not a proprietary file format.
-
Please not again....it is a can of "worms"
Please search for prior, quite vociferous discussions....
-
This question is about DNG or RAW for original camera files. Which format is your choice and why.
just don't mix the archival with the workflow (and both with the argument about whether manufactureres shall use DNG or not for in camera raws)... for archival purposes even Adobe's people were on record that they save the original raws /which might be already DNGs for some minority camera makers/ in the form that camera's firmware produced them... but we have some people here trying to be more popish than the pope ;)
-
Hi,
Difficult to foresee the future is…
- It is possible to embed the original raw file in DNG, so you can have it both ways.
- Some raw converters support DNG not at all or half heartedly.
- Some raw converters support DNG but not all raw files.
Personally, I feel that all these raw formats are a bad idea and make the community a disservice.
Best regards
Erik
This question is about DNG or RAW for original camera files. Which format is your choice and why. I'm really more interested in your answers than in the poll but thought it might be interesting o see numbers. I usually don't save JPEG (unless it's the only thing available) and scanner files are TIFF. Files edited in PS are either TIFF or PSD.
-
Sadly, this is a scenario where philosophy and practicality are in opposition.
Tony Jay
-
Well, you really cannot save your "originals" as DNG, can you? AFAIK, no camera produces DNG files - it's always the camera maker's RAW format. Yes, Lightroom and other software can convert to DNG, but then it's been converted and IS NO LONGER THE ORIGINAL.
-
Well, you really cannot save your "originals" as DNG, can you?
Leica 'native' RAW format is DNG. There may be others.
-
Pentax too. And Hasselblad.
-
Yes Peter, there are now some other manufacturers that propose to save to DNG format !
Another example is Nokia Lumia 1520.
But that's sadly a minority.
-
Leica 'native' RAW format is DNG. There may be others.
Here's a start in terms of a list:
http://www.barrypearson.co.uk/articles/dng/products_cameras.htm
-
Yes, Lightroom and other software can convert to DNG, but then it's been converted and IS NO LONGER THE ORIGINAL.
I am aware that a DNG contains metadata that may not be present in the original, but in what ways does the per pixel content differ from the original?
-
I am aware that a DNG contains metadata that may not be present in the original, but in what ways does the per pixel content differ from the original?
No difference!
Tony Jay
-
No difference!
Tony Jay
I maybe wrong. However, unless you include the actual RAW data, I do not believe you can claim to have the original data.
If you are going to include the the space for the original RAW data or if you are going to save the original RAW image, I am at a loss to understand the benefit to converting to DNG.
John
-
I maybe wrong. However, unless you include the actual RAW data, I do not believe you can claim to have the original data.
You have all the original raw image data. You care about proprietary metadata and such?
-
I maybe wrong. However, unless you include the actual RAW data, I do not believe you can claim to have the original data.
If you are going to include the the space for the original RAW data or if you are going to save the original RAW image, I am at a loss to understand the benefit to converting to DNG.
The truly daft thing about all of this is that most modern RAW formats are actually based on the DNG standard, most of the "differences" relate to the metadata headers and the encoding of some of the metadata.
Wouldn't it be helpful just to have, at least as an option, the ability to capture RAW images as DNG.
Some manufacturers do use DNG as their native RAW format but it is not yet the norm with more mainstream manufacturers.
Tony Jay
-
Hi,
Difficult to foresee the future is…
- It is possible to embed the original raw file in DNG, so you can have it both ways.
- Some raw converters support DNG not at all or half heartedly.
- Some raw converters support DNG but not all raw files.
Personally, I feel that all these raw formats are a bad idea and make the community a disservice.
Best regards
Erik
I agree 100%. I began converting all of my raw files to dng upon import into Lightroom beginning with Lightroom 4.0. But, since the release of DXO 9 and specifically it's PRIME noise reduction, I have stopped converting to dng upon import as DXO 9 will not open the Lightroom converted dng files.
How much of this non acceptance between companies is simply out of spite and how much has an actual technical/workflow reason?
-
I agree 100%. I began converting all of my raw files to dng upon import into Lightroom beginning with Lightroom 4.0. But, since the release of DXO 9 and specifically it's PRIME noise reduction, I have stopped converting to dng upon import as DXO 9 will not open the Lightroom converted dng files.
Good point
How much of this non acceptance between companies is simply out of spite and how much has an actual technical/workflow reason?
The not-invented-here syndrome is a very human thing.
-h
-
Hi,
I do it similar, but embedding the raw image in the DNG file. I essentially ignore DxO for incompatibility with non-linear DNG. I am a base ISO guy, so I don't need noise reduction.
Best regards
Erik
I agree 100%. I began converting all of my raw files to dng upon import into Lightroom beginning with Lightroom 4.0. But, since the release of DXO 9 and specifically it's PRIME noise reduction, I have stopped converting to dng upon import as DXO 9 will not open the Lightroom converted dng files.
-
Hi,
I do it similar, but embedding the raw image in the DNG file. I essentially ignore DxO for incompatibility with non-linear DNG. I am a base ISO guy, so I don't need noise reduction.
Best regards
Erik
Erik
You understand that converting to dng effectively locked you in the adobe products? IMHO not a wise long term strategy.
-
This question is about DNG or RAW for original camera files. Which format is your choice and why. I'm really more interested in your answers than in the poll but thought it might be interesting o see numbers. I usually don't save JPEG (unless it's the only thing available) and scanner files are TIFF. Files edited in PS are either TIFF or PSD.
I use RAW for :
No vendor lock in.
DNG stores extra metadata in the "large" RAW file while doing some -non destructive- changes, which makes backups and "backup snapshots" a lot bigger.
For good raw conversion the camera specifics need to be known to the RAW-convertor, this is a lot more work than decoding a RAW file (although it's probably less boring work).
-
Erik
You understand that converting to dng effectively locked you in the adobe products? IMHO not a wise long term strategy.
You understand you can happily use DNGs in Aperture? For example.
OK, you do have a point, but it's not black and white. Adobe's recent shenanigans should make us all more aware of the need to have exit strategies, but you're not locked in.
John
-
DNG stores extra metadata in the "large" RAW file while doing some -non destructive- changes, which makes backups and "backup snapshots" a lot bigger.
With DNG you do not need to keep backing up the files simply because you've saved metadata to them! For one thing, the metadata doesn't contain all your work, so the backup value would be questionable. Secondly you simply fine tune your backup strategy to backup DNGs when they are new, not thereafter. Together with backups of your Lightroom catalogue, this provides have 100% coverage of your work. An inability to fine tune one's backup strategy isn't a reason for avoiding DNG.
-
You understand you can happily use DNGs in Aperture? For example.
OK, you do have a point, but it's not black and white. Adobe's recent shenanigans should make us all more aware of the need to have exit strategies, but you're not locked in.
John
I use a PC, Aperture is, as far as I know, Apple only.
-
With DNG you do not need to keep backing up the files simply because you've saved metadata to them! For one thing, the metadata doesn't contain all your work, so the backup value would be questionable. Secondly you simply fine tune your backup strategy to backup DNGs when they are new, not thereafter. Together with backups of your Lightroom catalogue, this provides have 100% coverage of your work. An inability to fine tune one's backup strategy isn't a reason for avoiding DNG.
I don't use Lightroom, oh but then there's no use for DNG anyway. It's an Adobe only format.
I like my backups to be as generic as possible, those are to important to mess around with ("finetune").
-
Fine tuning isn't messing around, just clear thinking.
-
An inability to fine tune one's backup strategy isn't a reason for avoiding DNG.
Hi John,
To be fair, the same could be said about keeping one's Raw files Raw, and also save the XMPs with the same name. In fact, when only the small XMP file changes, it's much faster to back-up (think network or off-site storage) than a entire container DNG. It's also much easier to copy entire folders for others to work on.
What I fail to see is how one can assume that the conversion to DNG isn't built on Adobe's assumptions how the file should be rendered. Unless Adobe have unraveled all proprietary Raw formats, they cannot possibly utilize all the Maker-note data and masked sensel area information that the original Raw has to offer. There is a reason why e.g. Canon users get much more reliable color (notably Reds) with Canon's Digital Photo Professional (DPP) than with Adobe's profiles.
IMHO, all DNG does, is make it easier to process images in (predominantly) Adobe applications.
Cheers,
Bart
-
But, Bart, those xmp files fail to include a load of my Lightroom work, so their backup value is only second-rate. Backing up DNGs when only they are new, then routinely backing up the catalogue, provides 100% coverage of all images and work, and it isn't any more onerous in a network/offline environment. But this line of "you've got to backup your DNGs when you write metadata to them" keeps being wheeled out by the anti-DNG crowd even though it has always been bogus.
If you want to attack DNG with any credibility, it is far better to point to its patchy support by non-Adobe apps. That's where I would have doubts, but it's a case of inconsistency or limited choice rather than lack of options.
Is any of that maker note information really that useful? Really? Adobe do preserve it, just don't use or expose it (IIRC there's an unparsed additional metadata field in Lightroom's SQL).
John
-
But, Bart, those xmp files fail to include a load of my Lightroom work, so their backup value is only second-rate. Backing up DNGs when only they are new, then routinely backing up the catalogue, provides 100% coverage of all images and work, and it isn't any more onerous in a network/offline environment. But this line of "you've got to backup your DNGs when you write metadata to them" keeps being wheeled out by the anti-DNG crowd even though it has always been bogus.
If you want to attack DNG with any credibility, it is far better to point to its patchy support by non-Adobe apps. That's where I would have doubts, but it's a case of inconsistency or limited choice rather than lack of options.
Is any of that maker note information really that useful? Really? Adobe do preserve it, just don't use or expose it (IIRC there's an unparsed additional metadata field in Lightroom's SQL).
John
Like said, backup is very important and most backup solution work on the file level. Operational it should be working without thinking.
What you suppose is to backup only new files and NOT changed files. This isn't something most backup programs support and for good reason. Also things like snapshots, delorean backups (clever using hard links to create snapshots), time machine (for MAC users) will store a DNG every time it's changed.
If a upload of images from a memory card gives a problem with a file, it's hopeful that you notice it before the backup is made.
I agree that it's also important that it's an adobe format without much (if any) support outside adobe. It's not a nice message to someone that he/she has to keep renting Lightroom just to be able to open it's old photo's. But this is still better than having to say he/she screwed the backups.
-
But, Bart, those xmp files fail to include a load of my Lightroom work, so their backup value is only second-rate.
But, John, that also applies to my Photoshop layers, DNG files do not store those. Frankly, that has not much to do with DNG as a future proofing vehicle, but more with the application functionality (and databases, and warts, and such).
If you want to attack DNG with any credibility, ...
Huh? I just do not see the benefits for my workflow, YMMV.
it is far better to point to its patchy support by non-Adobe apps.
Now that would be the day, blame the others for not complying with Adobe's way of seeing things?
Is any of that maker note information really that useful? Really? Adobe do preserve it, just don't use or expose it (IIRC there's an unparsed additional metadata field in Lightroom's SQL).
After initially leaving out some data that was required to successfully reduce pattern noise, Adobe have come to realize that they'd better store everything(?) in their container, even if they do not (know how to) use it. Next someone will blame other software makers if they do not use the DNG as intended, while the Original Raw Converter will be able to utilize the original Raw data file. This will only get more complicated over time.
Currently, as another example, DPP will allow to retroactively correct for lens distortions and vignetting quite effectively, with the data stored in the original Canon Raws from my previous models as well. It even allows to 'de-fish' my fisheye wide angle shots. The same goes for dust removal if stored in the Raw file data, and there are no doubt more benefits we do not know about (I'm sure there could be a benefit for noise reduction when the camera's internal temperature field is intelligently combined with the exposure level data).
I'm not saying DPP, or LR, or PS, or you name it, is the best tool for everybody and their specific workflow requirements, that's not what this thread is about. What it is about, is that using DNG is mostly beneficial for working with certain Adobe applications (e.g. Lightroom), and may be a drawback for working with other software.
Cheers,
Bart
-
To be honest, I'm not going to get into addressing every point, Bart. These discussions always disappear down rabbit holes with all sorts of micro-exceptions and nuances. People either get the advantages of DNG and use it, or they don't / won't. And yes, those advantages are most with Adobe applications. And yes, others - most of all, the camera makers - deserve blame for not supporting a non-proprietary standard.
"What you suppose is to backup only new files and NOT changed files. This isn't something most backup programs support and for good reason."
Sure. So you just physically separate new or "virgin" DNGs from the "working" DNGs to which LR writes metadata. Once new DNGs are backed up, they can be moved over to the drive/folder for working DNGs which isn't targeted by the backup program. It takes a few seconds.
John
-
DPP will allow to retroactively correct for lens distortions and vignetting quite effectively
IMHO DPP can do an astonishing job optimizing files from a lens like the Canon 16-35 2.8 Mk I. I guess Canon put their optical scientists to good work there! I don't know why DPP needs to modify the original CR2 file but that is what it does. It would be so nice if Adobe could license this technology from Canon, because it's a real shame it's locked away in a RAW convertor that struggles in (many) other areas.
Bart I didn't realize that ACR also does a poorer job with reds compared to DPP. I had always thought occasional poor reds from my CR2s were purely because of the PhotoRGB -> SRGB workflow, but now I see there might be more to it. Something to keep in mind when I next process a CR2 with plenty of reds (assuming I don't need operations that Adobe excels at like highlight recovery).
-
...
"What you suppose is to backup only new files and NOT changed files. This isn't something most backup programs support and for good reason."
Sure. So you just physically separate new or "virgin" DNGs from the "working" DNGs to which LR writes metadata. Once new DNGs are backed up, they can be moved over to the drive/folder for working DNGs which isn't targeted by the backup program. It takes a few seconds.
John
This is not so "simple" with several offside-backups, I have a rotation schedule that takes up to about one month (but only a few days after an important shoot). I also don't need to think carefully when making backups. Are you suggesting that DNG's should not to be worked on until all backups are done?
A simple suggestion : keep the original RAW's in a separate location and backup those. A lot safer and easier. Storage has become cheap, a 3 TB drive is less than 100 euro's. Using 3 for offside-backups will get quite far.
BTW. I hope that you do have offside-backups.
-
People always find ways to make things difficult, don't they?
-
Hi,
Having 75000 images in Lightroom is a mighty lock in…
DNG is an open standard, so anyone can use it. All programs I use support DNG fully, with Capture One and DxO being the exceptions, but I use neither.
An open and publicly specified standard developed by people with a long tradition in image handling seems like decent option to me, compared with formats invented by camera manufacturers. I don't think Nikon is more credible than Adobe.
Best regards
Erik
Erik
You understand that converting to dng effectively locked you in the adobe products? IMHO not a wise long term strategy.
-
But, Bart, those xmp files fail to include a load of my Lightroom work, so their backup value is only second-rate. Backing up DNGs when only they are new, then routinely backing up the catalogue, provides 100% coverage of all images and work, and it isn't any more onerous in a network/offline environment. But this line of "you've got to backup your DNGs when you write metadata to them" keeps being wheeled out by the anti-DNG crowd even though it has always been bogus.
I'm confused....
How is backing up DNGs plus the catalog provide a better Lightroom backup than having backups of my RAWS, XMPs, and catalog? Or for that matter, just my original RAWs and the catalog? (though I prefer backing up the XMPs also)
-
You care about proprietary metadata and such?
Andrew.....shame :-)
You know that is a fox....which should not be followed down the hole. :-)
-
How is backing up DNGs plus the catalog provide a better Lightroom backup than having backups of my RAWS, XMPs, and catalog? Or for that matter, just my original RAWs and the catalog? (though I prefer backing up the XMPs also)
It's not better but it is complete - 100% coverage of my images and any work on them. The XMPs are superfluous but might provide the warm feeling of doing something.
-
It's not better but it is complete - 100% coverage of my images and any work on them. The XMPs are superfluous but might provide the warm feeling of doing something.
just to be clear....both are "complete", correct?
-
Thanks for the responses, I was not aware that some cameras now use DNG as their RAW format.
-
Hi,
It has been used for a long time.
It is really no magic. DNG is a TIFF format for handling raw data. It is well specified and the specifications are published. You can download a PDF from Adobe describing it in detail. It is proposed for standardisation.
Propriatory RAW formats are also TIFF containers but the layout of the information and the fields within are not published. A DNG file can encapsulate any RAW file fully.
I did a little test, converted a P45+.raw file embedding the RAW in the DNG, renamed the raw file end extracted the original from the DNG and run an md5 checksum. Every bit was retrieved correctly.
bigmacpro:DNGDemo ekr$ md5 20131231-CF044462*
MD5 (20131231-CF044462.IIQ) = 091651d5856c30e882a3369d44c0c98f
MD5 (20131231-CF044462_DNGConverted.dng) = 397ffc1d9ae1b93998735e6d063bf2e1
MD5 (20131231-CF044462_original.IIQ) = 091651d5856c30e882a3369d44c0c98f
bigmacpro:DNGDemo ekr$
Best regards
Erik
Thanks for the responses, I was not aware that some cameras now use DNG as their RAW format.
-
DNG here because it fits my workflow best. Neither are perfect but DNG has more pros for me right now. I used to be RAW. #1 working speed is important to me, and having the smaller file size along with more importantly no cache files, makes editing MUCH faster especially on SSD. Yes it takes longer to import but I simply do other things while that happens, there's no more waiting during editing itself. #2 file management is easier because of the lack of cache files - the SSD stays cleaner longer. Backups are a non-issue, the speed difference is a matter of seconds or minutes.
-
Andrew.....shame :-)
You know that is a fox....which should not be followed down the hole. :-)
Says you! I asked a very legitimate question which you should answer. Is proprietary metadata useful to you? And I assume you understand what the term proprietary means and what kind of data we are talking about. The implication of that data upon a possible workflow. It doesn't matter if the raw data is native camera raw or DNG, that proprietary data is only accessible and useful to those who wrote that proprietary data for use only in their converter. IOW, this has nothing to do with DNG. It has to do with proprietary data only one manufacturer can use. Will you use it? If not, it's useless data IMHO.
If I reply to your question in a code or language you can't decipher or understand, WTF good is it to you?
-
I don't use Lightroom, oh but then there's no use for DNG anyway. It's an Adobe only format.
Just like TIFF is (and PSD if anyone is still serious about that format).
-
I'm keeping the Raw files Raw,
typically using Bridge + ACR for editing, but not exclusively:
hence we don't want to discard the camera manufacturer's proprietary metadata – either for use in the corresponding converter, or, for a potential future use in ACR (as we can not know if "proprietary" is really carved in stone (may be just the principle of hope)). Aside from the metadata, I find it hard to have 100% trust in the Raw-to-DNG conversion "per pixel".
With Bridge + ACR the metadata edits are saved in XMP sidecar files,
however, we had several cases where the connection between a Raw file and the corresponding XMP sidecar got broken, for unknown reason. Means that the Raw file opens unedited even though it was edited before and the XMP sidecar is still present.
That's why we backup everything, Raw + XMP to DNG.
Also, the DNG file embeds the Camera Profile - unlike the XMP sidecar which just includes a note which profile was used (which is important to keep in mind when 'happily' deleting old profiles, or, when migrating to a new computer).
Peter
--
-
Hi,
The reason for ETTR is that noise is reduced. If you increase exposure one stop, noise will be reduced by 41%. If you take the P45+ that I have, it has a sensor surface of 48x37 mm, twice the area of a full frame sensor. So the P45+ would have something like a 41% advantage in shot noise. Shot noise is a property of light, or photon statistics. For that reason, ideal exposure is one that collects as much light as possible.
At the dark end, DR is limited by the noise floor, called readout noise. Increasing exposure increases the signal above readout noise. You can make an experiment. Take a ColorChecker (or grey card),
expose near saturation, and reduce exposure until the black patch clips. I am pretty sure you will see an increase in noise for each step exposure is reduced.
Notching up ISO to achieve ETTR is a different can of worms and depends on how ISO is handled, but in general, least noise will be achieved when as much light is captured as possible.
Best regards
Erik
Hi,
It has been used for a long time.
It is really no magic. DNG is a TIFF format for handling raw data. It is well specified and the specifications are published. You can download a PDF from Adobe describing it in detail. It is proposed for standardisation.
Propriatory RAW formats are also TIFF containers but the layout of the information and the fields within are not published. A DNG file can encapsulate any RAW file fully.
I did a little test, converted a P45+.raw file embedding the RAW in the DNG, renamed the raw file end extracted the original from the DNG and run an md5 checksum. Every bit was retrieved correctly.
bigmacpro:DNGDemo ekr$ md5 20131231-CF044462*
MD5 (20131231-CF044462.IIQ) = 091651d5856c30e882a3369d44c0c98f
MD5 (20131231-CF044462_DNGConverted.dng) = 397ffc1d9ae1b93998735e6d063bf2e1
MD5 (20131231-CF044462_original.IIQ) = 091651d5856c30e882a3369d44c0c98f
bigmacpro:DNGDemo ekr$
Best regards
Erik
-
It is really no magic. DNG is a TIFF format for handling raw data. It is well specified and the specifications are published. You can download a PDF from Adobe describing it in detail. It is proposed for standardisation.
Technically, DNG is fully compatible with TIFF-EP which is a specialized version of TIFF 6. TIFF-EP is also loosely used by other raw camera formats but the adoption of TIFF-EP by the camera makes is spotty at best. In point of fact, both Canon and Nikon initially took Kodak's .TIF file format and reformulated it for use in their camera, NEF by Nikon and .TIF and .CRW by Canon. Those early file formats were very crude and caused a lot of problems for early adopters. For example, Canon's .TIF file format could be opened in Photoshop because of the .tif extension but it would only open the preview file of the raw image. If you made the mistake of saving the file it would overwrite your original raw file.
In point of fact (and I know this from personal knowledge) all of the major camera makers (except Phase One and Leaf) really paid a great deal of attention to DNG and it's spec when Thomas first released it. The camera makers downloaded the DNG SDK and pored over the spec and as a direct result of the DNG format, subsequent raw file formats including NEF and CR2 were heavily influenced by what the camera makers learned from DNG.
Even if DNG never becomes a "standard", it's already had a huge positive impact on digital photography and for that, the industry owes Thomas Knoll a debt of gratitude...
Adobe has offered DNG to the ISO for consideration in the next revision of TIFF-EP...we're waiting to see what the ISO decides and if the ISO does incorporate DNG into TIFF-EP a lot of barriers to the use of DNG will fall.
Personally, I keep all my raw files as original raw files in both LR & ACR so my backups only need to backup the .XMP files. I do use DNG copies for final files that will not be further edited and for those rare occasions where I deliver a raw file (which is rare) because my renderings and metadata are embedded and not left to the sidecar file which can be lost or separated from the raw file.
In any event, DNG has been good for the industry and has been adopted successfully by some camera makers. DNG is not an Adobe conspiracy to try to control the industry, just to advance the art and science. There have been technical reasons in the past that gave certain issues as an excuse to not adopt DNG...in each case, Adobe (read Thomas and Eric Chan) have created revisions to address the technical issues.
The lack of adoption of DNG as a standard isn't really because of technical issues but political issues. Adopting DNG would not stifle creativity or block progress. Adopting DNG does not require camera companies to lift the kimono and expose proprietary technology. Private Maker Notes can still live in DNG files (except for certain encrypted files such as early Canon CRWs and Sigma files) and hide the camera companies' "secrets". I'm pretty sure that Thomas and Eric can take a file that has been converted from proprietary raw to DNG and take that DNG and reverse encode it back into a camera makers' raw file format that would pass the test of being processed in the camera company's proprietary software. I know the reasons that it's a function that is unlikely to ever show up in DNG Converter but I'm sure Thomas could do this if he felt like it.
I'm disinclined to get into a pissing match with anybody who thinks DNG is a bad thing...if you feel like that, you are welcome to wallow in your ignorance. What I'm primarily interested is making sure facts remain facts and that myths don't get turned into "facts".
Edit: Erik the above post was in response to yours but I'm trying to imply you are ignorant of the benefits of DNG...I started out simply explaining the TIFF-EP thingie and got a bit carried away :~)
-
Thanks for the valuable clarifications, Jeff.
-
Good post, Jeff
-
The lack of adoption of DNG as a standard isn't really because of technical issues but political issues.
Hi Jeff,
Could you elaborate? Or is this an assumption based on a mix-up of correlation and causation again, only too common in (photography) fora.
When other companies than Adobe do not anticipate a benefit to their bottom-line by adopting DNG (but a draw-back by demanding backward compatibility instead of a brandnew Raw format), then who are we to blame them for not adopting.
Cheers,
Bart
-
When other companies than Adobe do not anticipate a benefit to their bottom-line by adopting DNG (but a draw-back by demanding backward compatibility instead of a brandnew Raw format), then who are we to blame them for not adopting.
In some [large?] part, photographers are to blame. Consumers. Having options does us no harm. There is no heavy engineering and a low cost to provide a DNG option. It doesn't harm those that wish to continue to use camera raw files as is. If the buying public doesn't push, the big companies have no reason to do anything different. It's political because there's no other excuse that makes sense. It isn't technological. Why just a rendered JPEG? Why not a switch for a TIFF because you don't want/need raw but you want better than JPEG?
-
When other companies than Adobe do not anticipate a benefit to their bottom-line by adopting DNG (but a draw-back by demanding backward compatibility instead of a brand new Raw format), then who are we to blame them for not adopting.
Their customers....
-
Their customers....
Hi John,
So you are suggesting that the majority of customers are willing to pay more for their cameras for the addition of DNG Raws, and get larger Raw files and/or slower capture rates and more battery drain, rather than use a 'free' converter (in case they would like such support)? Don't think so.
Cheers,
Bart
-
So you are suggesting that the majority of customers are willing to pay more for their cameras for the addition of DNG Raws, and get larger Raw files and/or slower capture rates and more battery drain, rather than use a 'free' converter (in case they would like such support)? Don't think so.
So we don't even get the choice even if for most, it isn't their choice? If most customers shot JPEG, just providing that single option would be OK with you?
I owned a 5D and now 5DMII. The MIII had nothing that compelling to warrant an upgrade for me. If Canon added native DNG support, GPS, I'd upgrade in a minute.
-
So we don't even get the choice even if for most, it isn't their choice? I
What data supports your use of "most"?
My guess would be "most" don't even care.
-
So you are suggesting that the majority of customers are willing to pay more for their cameras for the addition of DNG Raws, and get larger Raw files and/or slower capture rates and more battery drain, rather than use a 'free' converter (in case they would like such support)? Don't think so.
No, Bart, that is merely your FUD (eg DNGs are typically smaller than proprietary raws). We are the camera makers' customers and have every right to blame them for not offering a non-proprietary format as an option.
John
-
My guess would be "most" don't even care.
So the ignorance is bliss excuse. Along with: those that don't care (while outnumbering those who do) means those that do care don’t count. Interesting perspective.
-
So the ignorance is bliss excuse. Along with: those that don't care (while outnumbering those who do) means those that do care don’t count. Interesting perspective.
More interesting are those who think that they know better than any others....but are unable to provide a convincing benefit which might make those who's "ignorance is bliss" to even give any thought to the idea.....much less any direction to what they should do IF they believed.
PS....I noticed you totally ignored my question....which I expected from past experience.
-
More interesting are those who think that they know better than any others....but are unable to provide a convincing benefit which might make those who's "ignorance is bliss" to even give any thought to the idea.....much less any direction to what they should do IF they believed.
From 2007
http://www.ppmag.com/reviews/200709_adobedng.pdf
-
What's not convincing about support from day 1 of their newly-released camera?
-
From 2007
http://www.ppmag.com/reviews/200709_adobedng.pdf
Nice discussion....but obviously it has not been view as a compelling arguement.
-
What data supports your use of "most"?
How specific is this?: all users not currently converting to DNG.
-
Nice discussion....but obviously it has not been view as a compelling arguement.
Argue for what other than a user option. How does that affect you or anyone else who will never use it. I will NEVER set my 5DMII for JPEG. I'm cool if any or everyone else does.
I don't care if this fix doesn't even cost the manufacturer's a new dial on their camera. Let me use some Firmware update or user setting to give me a DNG. You set it for whatever you want. How is siding with the manufacturer and against the request of fellow photographers hurting you?
-
What's not convincing about support from day 1 of their newly-released camera?
Not every one buys on day one. Last time I did, Canon 5d3, the RAWs were supported by LR by the time I had the camera in hand.
Even if it hadn't, it usually is not a long time for support of mass market cameras. Plus, if it was really important I could use Canon's software.
I understand Adobe's business case for DNG.
-
Not every one buys on day one.
So that this affects them and not you doesn't count. I would submit that those early buyers who wanted raw support outside the manufacturer's converter were not pleased by this behavior. There was no reason for this disappointment had there been another raw option.
-
Argue for what other than a user option. How does that affect you or anyone else who will never use it. I will NEVER set my 5DMII for JPEG. I'm cool if any or everyone else does.
I don't care if this fix doesn't even cost the manufacturer's a new dial on their camera. Let me use some Firmware update or user setting to give me a DNG. You set it for whatever you want. How is siding with the manufacturer and against the request of fellow photographers hurting you?
Frankly, I don't understand your "logic" above, but...
Andrew, I asked that this not be discussed because of the vociferous and vicious rhetoric of the DNG proponents. I am not against your or others desire for having DNG. I get tired of the, "if you are not for it, you are part of the problem" arguments that get put forward.
We all have the right to our opinions....I respect yours...why can't you respect mine?
-
Not every one buys on day one. Last time I did, Canon 5d3, the RAWs were supported by LR by the time I had the camera in hand.
Even if it hadn't, it usually is not a long time for support of mass market cameras. Plus, if it was really important I could use Canon's software.
But some do, and do you never hear them moaning about how they can't use their raw editing software? If you're unlucky, it can be 3 months while you're screwing around with DPP or NX2. Not convincing? Or won't you accept any convincing case, whatever it is?
I understand Adobe's business case for DNG.
Go on then, you explain it.
-
I get tired of the, "if you are not for it, you are part of the problem" agruements that get put forward.
The question asked was why isn't this format adopted and I put the blame where it belongs. Your opinions about DNG are clear and please don't use them. I find it interesting that it's OK to have thousands of proprietary raw files but one more in the mix, one that isn't proprietary is somehow a problem. Don't use it, I'm fine with that. I want an option to get that kind of data from the camera natively and that's super easy to do. So unless this request somehow hurts you, can you request it for us fellow shooters who want it and in no way wish to force that option onto you. Or is there some burden upon you in aiding some of us in our feature requests?
-
But some do, and do you never hear them moaning about how they can't use their raw editing software? If you're unlucky, it can be 3 months while you're screwing around with DPP or NX2. Not convincing? Or won't you accept any convincing case, whatever it is?
I agree it is an annoyance and/or short term problem. However, if this is the only "compelling" reason, it is pretty easy to see why DNG has not been accepted universally.
Go on then, you explain it.
I am shocked that you would need to ask this....isn't it obvious...??
-
The question asked was why isn't this format adopted and I put the blame where it belongs. Your opinions about DNG are clear and please don't use them. I find it interesting that it's OK to have thousands of proprietary raw files but one more in the mix, one that isn't proprietary is somehow a problem. Don't use it, I'm fine with that. I want an option to get that kind of data from the camera natively and that's super easy to do. So unless this request somehow hurts you, can you request it for us fellow shooters who want it and in no way wish to force that option onto you. Or is there some burden upon you in aiding some of us in our feature requests?
So....you admit you can repeatedly express your opinion, but you do not want me to.... Just to be clear, I am NOT against DNG, I just do not see a compelling case....for the manufacturers nor the users....some minor advantages, but...
I do not know, nor, I suspect, do you know, what the true reasons or costs for the manufacturers to convert to DNG. Nor do I know what would be involved with offering it as an option, though in my experience it would impact not only developments, but, most likely the firmware space and timing within the camera.
It is really easy to sit back and conceptualize a "simple" fix. I know you have some development experience, but it suspect it did not involve major computer systems hardware, software, and firmware design, development, and testing....if it did, you would know that NOTHING is "simple".
-
I am shocked that you would need to ask this....isn't it obvious...??
Why don't you enlighten us then?
-
I noticed you totally ignored my question....which I expected from past experience.
-
I noticed you totally ignored my question....which I expected from past experience.
Are you quoting me for a reason....without attribution BTW.
If you are making an attempt to be humorous...or vicious...I did answer your question....which if you were not so blinded by "my way or the highway" would be obvious to any thoughtful person.
-
Why don't you enlighten us then?
It is pretty simple....translation of RAw to a form Adobe can use falls totally on the manufacturer. Adobe....and the other RAW converts...now must expend no effort.
So...added cost for the camera manufacturer and only cost savings for Adobe. No apparent benefit to end user, other than early adoptors. Possible diminished code efficiency to Manfacturers and also lack of ability to easily incorporate need ideas due to the need to adher to a standard...and as we know, adoption of new ideas to "standards" is a long, winding road.
If one were a conspiracy theorist, I am sure that cases could be made that could probably hold water, but that goes beyond what is needed right now.
-
It is pretty simple....translation of RAw to a form Adobe can use falls totally on the manufacturer. Adobe....and the other RAW converts...now must expend no effort.
So...added cost for the camera manufacturer and only cost savings for Adobe. No apparent benefit to end user, other than early adoptors. Possible diminished code efficiency to Manfacturers and also lack of ability to easily incorporate need ideas due to the need to adher to a standard...and as we know, adoption of new ideas to "standards" is a long, winding road.
If one were a conspiracy theorist, I am sure that cases could be made that could probably hold water, but that goes beyond what is needed right now.
Sorry this does not hold any water - almost all new RAW formats are really DNG in drag!
Read Schewe's post for a bit more detail.
A firmware update would be all that was required in most cases to allow the addition of DNG along with JPEG and the proprietary RAW as image options out-of-camera.
As for the benefits - if you don't get that part, perhaps you never will...
Tony Jay
-
Are you quoting me for a reason....without attribution BTW.
No. I asked a question, you didn't answer it and that's happened in the past. Is there anything wrong with the language I used?
-
It is pretty simple....translation of RAw to a form Adobe can use falls totally on the manufacturer.
Right, just like it does with those JPEGs.
So...added cost for the camera manufacturer and only cost savings for Adobe.
And every raw converter software developer that can read a DNG (easy).
-
This is, once again, a very interesting discussion. I decided to send a customer support ticket to DXO expressing my opinion on the fact that DXO Pro 9 does not support dng files. I received a very nice email response explaining that basically the DXO developer team is discussing adding dng support in the future. No decision has been made and dng support may, or may not be included in a future version. I responded by giving my personal reasons for using dng format as I was told in their email response that any opinions/views would be passed onto the developers.
I suggest that voicing your opinion directly to the manufacturers and software developers is one of, if not the best, way to bring about change. I also provided the rep at DXO with a link to this thread in case anyone there is interested in reading your opinions that have been shared here.
-
So...added cost for the camera manufacturer and only cost savings for Adobe. No apparent benefit to end user, other than early adoptors. Possible diminished code efficiency to Manfacturers and also lack of ability to easily incorporate need ideas due to the need to adher to a standard...and as we know, adoption of new ideas to "standards" is a long, winding road.
Yeah, ya know...I'm not going down that rabbit hole with you again. What you wrote above is FUD (Fear, Uncertainty and Doubt). The added cost to the manufacturers? Just how much would a firmware update cost? If you don't know then you are merely speculating that adding DNG would be costly (hint, it would be easy and cheap for Nikon and Canon to do so if they wanted to–they just don't want to).
Benefit to the end user? The beginning of the end of the wild west comes to mind. It took DNG to teach the camera makers how to formulate a sturdy, documented raw file format (Nikon and Canon had zero clues other that what they "borrowed" from Kodak). If the camera makers adopted DNG (even as an option) it would provide end users with options to use a non-proprietary format. If the camera makers' software also supported DNG (which is actually a different issue) it would enable users to use their own choice of file format while still retaining raw image editing. To date, the only camera maker who supports DNG is Capture One (sadly the DNG support is at least 2 DNG SDK versions behind).
"Diminished code efficiency"? Sorry, that is a non-sequitor...those terms are meaningless. What are you trying to say? That Nikon and Canon are incompetent code developers? (if that's what you mean, I tend to agree, they can barely code their way out of a paper bag). But hey, DNG provides a full fledged SDK as a guide to how to do what you seem to think is a difficult task (hint, it would be easy for Nikon and Canon to add DNG) Fact is, current NEF and CR2 files are so close to DNG that all they would have to do is change how and where they write metadata and do so in the properly documented manner (which is spelled out in the SDK in case Nikon or Canon are having a tough time).
So, you think adhering to a standard is an unbearable burden? They already adhere to several standards such as TIFF-EP and EXIF (well, more or less). Would that mean that Nikon and Canon had to play by the rules? Yep...and that's their primary reason for rejecting DNG at the moment. They don't want to play by any rules (and this is good for the industry how?).
In terms of stifling creativity...bullshit (and you know it-you are talking of the top of your head without any real knowledge if the technical implications that are actually involved). Yes, there have been a couple of times that new cameras required changes to the DNG SDK that would have impacted adoption of DNG with a new camera. When Sony first released a camera whose captures could be stored as raw files, Sony bragged about the fact that they had encrypted the raw file data and that it would take Adobe months to decode the file. Fact is, Dave Coffin had already decoded the encryption...imagine Sony's chagrin when they realized how wrong they were (and point of fact, the appreciated the 3rd party raw file format support when it came to selling cameras).
I'm ok if you don't want DNG-your choice...but it would be useful to stay away from stuff you don't really understand and resist spreading misinformation and making uneducated speculations regarding DNG's viability, usefulness and difficulty in adoption.
The reasons why Nikon and Canon refuse to adopt DNG are primarily political, not technical. The cost of DNG adoption would be minimal, the difficulty adopting minimal. Adopting DNG would not stifle creativity nor prevent new sensors and cameras from coming to market. Yes, adopting DNG would require the camera makers to adopt some standards (which I personally think would be a good thing for the industry–for a change). Adopting DNG is free and Adobe has already offered DNG to the ISO for free (Adobe allowed the ISO to adopt and adapt TIFF-6 for TIFF-EP for free, which of course the camera makers are already using).
Look, DNG is simply a well formed, intelligent and fully documented guide for making raw file formats. Yes, it would require the burden of adopting some standards (finally). While Nikon and Canon cling to their undocumented, proprietary raw file formats, the industry continues to suffer from the impact of their rejection of standards. The real secret about all these proprietary raw file formats is the fact that in reality, there are no secrets in the files. The proprietary secrets are in the chips and DSPs that process a raw capture and convert from analog to digital conversion. By the time the capture data is written to media, it's just data. And it's data that Dave Coffin, Thomas Knoll and Eric Chan can "easily" decode (when I say "easily" what I mean is it's time wasted decoding new cameras simply because the camera companies can get away with it–and it's time Thomas and Eric don't have to spend on new processing algorithms).
Personally, I would prefer Thomas and Eric spend their time making my raw images look the best they can possible be rather than wasting time decoding yet another new sensor/camera.
Just so ya know, I'm not interested in getting down into the mud again...let me know if there's anything I wrote that you don't understand but I'm not particularly interested to hearing what you disagree with–again.
-
Why is it that mobile phone manufacturers chose to implement a gazillion of essentially similar chargers/plugs? But when the EU/China demanded that they had to use USB charging, they were quickly able to switch to micro-USB (except Apple, that is).
This seems like a win-win. Manufacturers get to concentrate on building great phones, instead of carrying an inventory of umpteen chargers. Customers gets to have one charger to rule them all. The environment gets to… So why did not this happen before the regulators forced it to happen? This puzzles me. Was it the "not-invented-here" syndrome? Or did the manufacturers actually hope to earn on user lock-in, car-chargers and such?
What does this little story tell us about in-camera DNG?
-h
-
What does this little story tell us about in-camera DNG?
It tells us that undocumented, proprietary raw file formats are bad for the industry...and when I say "industry" I'm talking the entire eccosphere of digital photography including; the original camera makers, 3rd party software developers (both big & small) 3rd party hardware developers (as in accessories, cards, readers etc) professional photographers, amateur photographers (and soccer moms), photo labs, ad agencies, prepro houses, graphic arts, printing, media such as magazines and electronic media and social media and humanity at large.
The only people who are currently benefiting from proprietary raw file formats are the camera makers. For everybody else, it a pain in the ass. As Spock used to say, "the needs of the many outweigh the needs of the few".
-
The only people who are currently benefiting from proprietary raw file formats are the camera makers. For everybody else, it a pain in the ass. As Spock used to say, "the needs of the many outweigh the needs of the few".
What I don't get is how the proprietary formats can even benefit the camera makers. Personally I convert everything to DNG on import and the CR2 files simply serve as a temporary backup on a second hard drive. I honestly don't understand their thinking here.
-
What I don't get is how the proprietary formats can even benefit the camera makers.
…
I honestly don't understand their thinking here.
This was the line of thinking that I wanted to discuss. I honestly don't see how proprietary cell phone chargers could benefit the cell phone makers (well, I see a few, but it seems to me that the disadvantages are just as big, or bigger). So why do the camera makers insist on doing proprietary?
1) One obvious possible reason is the not-invented-here syndrome. They have clever people on their payroll. They have invested time into their existing way of doing things. Falling in line with the industry may be hard. Especially in "proud" cultures.
2) Another reason may be that their (typically free) in-house raw development software people feels that proprietary file formats gives them an upper-hand vs competitors.
Both 1 and 2 are "irrational" as seen from the company as a whole. Canon does not exist to boost their R&D people egos, or to make DPP at any cost. They (I would assume) exist for their owners to maximize profits.
The only people who are currently benefiting from proprietary raw file formats are the camera makers. For everybody else, it a pain in the ass. As Spock used to say, "the needs of the many outweigh the needs of the few".
If this is true, I would expect them to continue. Canon & friends have no other responsibility than earning money (abiding the law etc). The question is if the pain this decision is inflicting on their users can be translated into a disadvantage for them (in which case the sum may be a disadvantage to camera manufacturers).
-h
-
Both 1 and 2 are "irrational" as seen from the company as a whole. Canon does not exist to boost their R&D people egos, or to make DPP at any cost. They (I would assume) exist for their owners to maximize profits.If this is true, I would expect them to continue.
The reality is it's both 1 & 2...Not Invented Here is a really big motivation in Japan (not racist, just reality). As far as #2, they (Nikon and Canon both) "think" that having an undocumented, proprietary raw file format "might" give them a leg up and it's that "might" that is the main motivation...
If the industry (meaning everybody else other than Nikon and Canon) were to come out really strongly AGAINST undocumented, proprietary raw file formats, they would be far more likely to blink.
That's why it's sad that any photographers would ever consider supporting Nikon and Canon's current misbehavior...
Unfortunately, the more photographers that drink their Kool-Aid and give them any possible wiggle room (meaning reject the call for standards) the longer it will take to make them behave correctly...
I know there are a few people here on LuLa who equate DNG with Evil Adobe Greed...but I can't express just how ignorant and wrong this is...DNG is a gift to the industry by Thomas Knoll. Adobe supported Thomas' desire to develop a raw file format standard...Adobe created DNG for free, revised the DNG SDK multiple times to address industry critiques, offered DNG to the ISO and has bent over backwards to try to help the industry. Sadly, some people still reject the value of DNG for a multitude of motivations (most of which are either ignorant or coming from petty agendas).
Adobe has done a lot for the industry that they don't get credit for...initiatives such as DNG, XMP and other technologies that Adobe has developed and given away for industry use.
The real problem is Nikon and Canon can get away with their misbehavior because way too many photographers are willing to accuse Adobe of being an evil monopoly while giving Nikon and Canon a pass on being petulant brats that refuse to play well with others.
-
Hi,
Add Phase One to the bad guys, too...
Best regards
Erik
The reality is it's both 1 & 2...Not Invented Here is a really big motivation in Japan (not racist, just reality). As far as #2, they (Nikon and Canon both) "think" that having an undocumented, proprietary raw file format "might" give them a leg up and it's that "might" that is the main motivation...
If the industry (meaning everybody else other than Nikon and Canon) were to come out really strongly AGAINST undocumented, proprietary raw file formats, they would be far more likely to blink.
That's why it's sad that any photographers would ever consider supporting Nikon and Canon's current misbehavior...
Unfortunately, the more photographers that drink their Kool-Aid and give them any possible wiggle room (meaning reject the call for standards) the longer it will take to make them behave correctly...
I know there are a few people here on LuLa who equate DNG with Evil Adobe Greed...but I can't express just how ignorant and wrong this is...DNG is a gift to the industry by Thomas Knoll. Adobe supported Thomas' desire to develop a raw file format standard...Adobe created DNG for free, revised the DNG SDK multiple times to address industry critiques, offered DNG to the ISO and has bent over backwards to try to help the industry. Sadly, some people still reject the value of DNG for a multitude of motivations (most of which are either ignorant or coming from petty agendas).
Adobe has done a lot for the industry that they don't get credit for...initiatives such as DNG, XMP and other technologies that Adobe has developed and given away for industry use.
The real problem is Nikon and Canon can get away with their misbehavior because way too many photographers are willing to accuse Adobe of being an evil monopoly while giving Nikon and Canon a pass on being petulant brats that refuse to play well with others.
-
The reality is it's both 1 & 2...Not Invented Here is a really big motivation in Japan (not racist, just reality). As far as #2, they (Nikon and Canon both) "think" that having an undocumented, proprietary raw file format "might" give them a leg up and it's that "might" that is the main motivation...
My own theory is that the "proprietary" mentality is completely hard wired into Canon and Nikon's corporate world view. To be fair, while in the case of file formats this is completely irrational, I can understand the wider mentality because it has served them so well in the past. For example, I have always bought Canon. Why is this? Because I started with Canon and now own over $5000 worth of EF mount lenses which I really like. So I'm going to continue buying Canon bodies until they discontinue the EF mount system. I am sure that other manufacturers make better bodies but the fact is that Canon have locked me in, and I am fine with that.
In the case of their refusal to use DNG the mentality is completely irrational because it does not make their system any more or less sticky to the consumer, but I guess it is just a kneejerk thing for them. After all, the strategy has served them well in the past.
It would be totally cool if some geek out there were to start making firmware hacks for all the major Canon and Nikon bodies to allow DNG output. While they are about it they could add a custom function to map a button on the Canon bodies to Mirror Lock Up! I don't know if this is possible though. :-\
-
The reality is it's both 1 & 2...Not Invented Here is a really big motivation in Japan (not racist, just reality). As far as #2, they (Nikon and Canon both) "think" that having an undocumented, proprietary raw file format "might" give them a leg up and it's that "might" that is the main motivation...
If the industry (meaning everybody else other than Nikon and Canon) were to come out really strongly AGAINST undocumented, proprietary raw file formats, they would be far more likely to blink.
That's why it's sad that any photographers would ever consider supporting Nikon and Canon's current misbehavior...
Unfortunately, the more photographers that drink their Kool-Aid and give them any possible wiggle room (meaning reject the call for standards) the longer it will take to make them behave correctly...
I know there are a few people here on LuLa who equate DNG with Evil Adobe Greed...but I can't express just how ignorant and wrong this is...DNG is a gift to the industry by Thomas Knoll. Adobe supported Thomas' desire to develop a raw file format standard...Adobe created DNG for free, revised the DNG SDK multiple times to address industry critiques, offered DNG to the ISO and has bent over backwards to try to help the industry. Sadly, some people still reject the value of DNG for a multitude of motivations (most of which are either ignorant or coming from petty agendas).
Adobe has done a lot for the industry that they don't get credit for...initiatives such as DNG, XMP and other technologies that Adobe has developed and given away for industry use.
The real problem is Nikon and Canon can get away with their misbehavior because way too many photographers are willing to accuse Adobe of being an evil monopoly while giving Nikon and Canon a pass on being petulant brats that refuse to play well with others.
Well companies that want to use a "American" format must be 100% sure that there are no "patent" issues (remember gif, jpeg (!),...), with adobe, but also with all other possible patents. This can be costly and if not done 100% right could be still be a lot more problems.
I don't expect a big camera maker to gamble on that.
-
The reality is it's both 1 & 2...Not Invented Here is a really big motivation in Japan (not racist, just reality).
I'm all in favor of an open, evolving, documented standard for RAW, and if DNG were more popular that would be great. But if someone can tell us how we got to this state of affairs then I'm all eyes and ears:
- Canon's raw software developer DPP can modify CR2 files in post processing. In so doing it can greatly increase the file size. In certain aspects through this process DPP can produce noticeably superior results to Adobe ACR / Lightroom. Martin Bailey, who is a much better photographer than me, explains it here (http://www.martinbaileyphotography.com/2012/07/14/quick-look-canon-digital-photo-professional-digital-lens-optimizer/).
- DNG convertor will produce absolutely identical output from a CR2 regardless of whether DPP has modified the CR2 or not (it's obviously easy to confirm this). The obvious conclusion is that it simply ignores the modifications (extra data) that Canon introduces into the CR2. This is a huge pity. It's a raw file and Canon's optical scientists clearly know their stuff!
I guess their could be many reasons for this. Maybe Canon doesn't want to share their cool technology with Adobe (why not?). Maybe Adobe would have to pay Canon and they don't want to? Maybe it's too much work for Adobe and they don't want to make ACR more complex? Maybe it's coming in the next major revision of ACR? Or maybe this situation is an example of why Canon et al. like to retain absolute control of the file format their cameras produce?
Speaking as a user, it seems to me the major villain in this whole state of affairs is the proprietary nature of most photography software itself. The incentives are too great for companies to act in a too-narrow self-interest. I applaud Adobe for trying to do the right thing regarding DNG.
Finally, as an aside, I guess someone somewhere has written a good academic paper contrasting why some standards become real standards and others fall by the wayside.
-
- Canon's raw software developer DPP can modify CR2 files in post processing. In so doing it can greatly increase the file size. In certain aspects through this process DPP can produce noticeably superior results to Adobe ACR / Lightroom. Martin Bailey, who is a much better photographer than me, explains it here (http://www.martinbaileyphotography.com/2012/07/14/quick-look-canon-digital-photo-professional-digital-lens-optimizer/).
That's really interesting, I didn't know that. Thanks for the link and the knowledge.
However, I still don't understand Canon's strategy. Canon make superb camera bodies and lenses. This is how they make their money. They certainly don't make any money from their free DPP software - possibly the least usable image editing programme I have ever seen, notwithstanding any clever under the hood stuff. Seriously, how many photographers do you know of who actually use DPP on a regular basis? No? Me neither. Canon have to give the software away and still nobody uses it.
Given that Canon's business is to sell cameras and lenses, surely it would be in their interest to make all their clever under-the-hood stuff freely available to the developers at Adobe, Apple, DxO and whoever else wants it. If Canon's lens correction algorithms work well and could be incorporated into Lightroom et al, then this would (a) be good for those Canon users who use third party RAW editing software, and (b) possibly become a source of competitive advantage to Canon. Leaving these algorithms stuck in a piece of software which basically nobody ever uses just seems such a waste.
-
I think its about time Adobe made a camera!
Ho Ho Ho
happy new year everbody
-
Given that Canon's business is to sell cameras and lenses, surely it would be in their interest to make all their clever under-the-hood stuff freely available to the developers at Adobe, Apple, DxO and whoever else wants it.
Hi Ed,
What some of the DNG advocates conveniently forget to mention is that e.g. Canon is one of the more prolific producers of intellectual property / recipients of Patents. For example in the USA, they are ranking in the very top (http://en.wikipedia.org/wiki/List_of_top_United_States_patent_recipients), often in 3rd or 4th position.
Now imagine that Canon would want to commercialize one of those patents, and use it in their own equipment. Even to allow usage by others without them paying royalties (which is unlikely for a commercial enterprise), they would probably have to make sure that it can be encoded in the DNG so that it can be used. That would alarm their competitors and give them an early warning about what is to come and which direction their competitor is going, long before it materializes.
It would be safer and easier to not disclose the proprietary data, but why then hide it in a DNG. What's the benefit for Canon? They might as well keep it proprietary from the start and avoid having to deal with changing a standard by committee, and waste time and effort.
Of course the benefit of standardization for Adobe is much larger, not having to adapt to all those camera makers of those dreaded proprietary Raw formats (think about the quality of Fuji Xtrans conversions, or Foveon files, or the earlier encrypted Nikon white-balance data), so they try to enforce some sort of common flexible system that saves themselves cost and effort in the long run. That also explains their benevolent altruism to take the lead in creating some open standards, it's for their own benefit (more than for others if any) and reduces the risk of having to conform to someone else's standard, or even worse having to pay for it, at a later stage.
It's nothing to do with not-invented-here syndromes or politics or being altruistic, it's all about money (and Adobe is no different than others, as the move to cloud subscriptions proves).
Cheers,
Bart
-
I decided to send a customer support ticket to DXO expressing my opinion on the fact that DXO Pro 9 does not support dng files.
Thank you for that effort. IF those who spend countless hours arguing against a simple option for fellow photographers would spend a the fraction of the time it takes to do what you did, we might see a change within our industry. I applaud you and all other's who make your preferences known to those who you've spent your money on.
DNG or Proprietary raw, it is my data. Should I be forced to use one lab to process my film?
Given that Canon's business is to sell cameras and lenses, surely it would be in their interest to make all their clever under-the-hood stuff freely available to the developers at Adobe, Apple, DxO and whoever else wants it.
Or not, that's fine with me. IF they build a better raw processor solely using their proprietary stuff, it is up to us to decide if we want that or not. This is all about options, nothing more. IF given the option's today of proprietary and better processing in DPP or use the workflow I've developed using LR, I'd stick with LR but that's my choice. How other photographers can argue that providing options in the data that comes off the camera, be it DNG or a TIFF instead of a JPEG is totally unwarranted and unnecessary. Uncool too!
-
Hi Andrew,
As I understand your argument, it boils down to this:
- proprietary data formats are a social evil because you give up control of something you own
- proprietary algorithms to process that data are socially benevolent because you pragmatically believe our current ability to control those processes is adequate
Am I missing something?
Damon
-
It would be safer and easier to not disclose the proprietary data, but why then hide it in a DNG. What's the benefit for Canon? They might as well keep it proprietary from the start and avoid having to deal with changing a standard by committee, and waste time and effort.
my beloved example is the story of how Panasonic introduced software optics correction back then and how much time it took Adobe to support it in .DNG... so what DNG camp 'd suggest to Panasonic then ? disclose the move to competition in advance ? go through the hassle to hide optics correction data in DNG maker notes first and then to support what Adobe invented (rewrite their firmware code twice) ? etc... with their own format Panasonic was free to do it how they see it fit and when they see it fit... that goes to as to why camera manufacturers (except low single digit % of the market represented by Ricoh, Leica, few others) do not want to have their hands tied (not about 2 other issues : DNG as a workflow option and converted DNG as a sole archival format w/ discarding original raw files)
-
I think its about time Adobe made a camera!
indeed... remember how M$ ventured into tablets... Google into actual phones (and not just prototypes for developers to use) w/ their purchase of relevant parts of Motorola... so tomorrow Adobe will start making imaging devices and that adds to the reasons why you don't want to be in bed w/ Adobe about "in-camera" formats, not if you are not an irrelevant player marketwise (like Ricoh and other adopters).
-
Hi,
Is there any code involved, not just a couple of tags describing the errors to be corrected?
Best regards
Erik
my beloved example is the story of how Panasonic introduced software optics correction back then and how much time it took Adobe to support it in .DNG... so what DNG camp 'd suggest to Panasonic then ? disclose the move to competition in advance ? go through the hassle to hide optics correction data in DNG maker notes first and then to support what Adobe invented (rewrite their firmware code twice) ? etc... with their own format Panasonic was free to do it how they see it fit and when they see it fit... that goes to as to why camera manufacturers (except low single digit % of the market represented by Ricoh, Leica, few others) do not want to have their hands tied (not about 2 other issues : DNG as a workflow option and converted DNG as a sole archival format w/ discarding original raw files)
-
Is there any code involved, not just a couple of tags describing the errors to be corrected?
code, no matter how small LOC number actually is, is always involved...
-
- proprietary data formats are a social evil because you give up control of something you own
- proprietary algorithms to process that data are socially benevolent because you pragmatically believe our current ability to control those processes is adequate
Am I missing something?
I believe you are, a lot. Let me try again.
I have no issue with proprietary data. As long as you understand where and when it's useful. Nearly all the processing in Adobe products are proprietary. Adjustment layers, the text that tells ACR what +14 Vibrance should do. I'm fine with that.
What I'm not fine with is the creation of a new proprietary raw format for each camera who's affect is on users is keeping them from processing that data as they desire until the unnecessary proprietary data is hacked by all the other converters so they can actually ignore the use of that proprietary data. There is no reason for this. There is a reason why +14 Vibrance exists and is only understood by the product that build that set of instructions. Those instructions are gobbledegook in any other raw converter but that's not a problem and is expected.
That proprietary data could be inserted within a tag within the DNG such it's still proprietary AND it doesn't keep users from accessing their data the day the camera ships is the point I believe you are missing.
The camera manufacturer's under discussion here provide two formats and only two. One is an open format that is accessible the day the camera ships. But it's a JPEG. The other isn't an open format and isn't accessible the day the camera ships expect in the camera vendor's software few of us wish to use. AND a couple months later, thanks to the efforts (time and money) of all other raw converter manufactures, that limitation which never needed to exist in the first place is gone! So how do you justify this on-going practice as useful to anyone? It's perhaps not beneficial to the camera manufacturers. Especially if their customers make their desires known.
-
Hi Andrew,
So to put it another way you believe proprietary formats for data (including data you create) are socially benevolent when the algorithms that can be expected to process them proprietary too. Your expectation is that data that comes from a camera ought not to be proprietary because there already exist generic proprietary processes to manipulate the data, e.g. ACR. If there weren't presumably you wouldn't have problem because you're not against proprietary data formats per se.
From my perspective your argument is exceedingly narrow -- much more narrow for instance than the debates that emerged around the establishment of ISO standards for word processing and spreadsheet data formats, for instance.
Damon
-
Your expectation is that data that comes from a camera ought not to be proprietary because there already exist generic proprietary processes to manipulate the data, e.g. ACR.
I simply want access from day one to the unrendered raw image data. I don't care about any of the other data. Proprietary or otherwise.
-
Hi,
The image data is very simple, it just RGB values. To interpret the RGB values you need white balance (two numbers) and color conversion matrix (9 numbers). Additional data can be helpful. DNG can contain proprietary data. In fact you can embed a raw file as a whole in a DNG file.
Best regards
Erik
Hi Andrew,
So to put it another way you believe proprietary formats for data (including data you create) are socially benevolent when the algorithms that can be expected to process them proprietary too. Your expectation is that data that comes from a camera ought not to be proprietary because there already exist generic proprietary processes to manipulate the data, e.g. ACR. If there weren't presumably you wouldn't have problem because you're not against proprietary data formats per se.
From my perspective your argument is exceedingly narrow -- much more narrow for instance than the debates that emerged around the establishment of ISO standards for word processing and spreadsheet data formats, for instance.
Damon
-
I simply want access from day one to the unrendered raw image data. I don't care about any of the other data. Proprietary or otherwise.
It seems to me that what you are arguing for is far narrower than aspects of the DNG spec itself. One of the the points of a metadata standard is to include information about how the data is to be transformed. That's why XMP files contain that data in an open format. That metadata about files we produce today is useful now, and will be useful to some people after all of us are long dead. It is still useful even though we don't know the precise implementation of the algorithms that actually transform the data because they are proprietary. In other words, you may not care about how a change in vibrancy is recorded, but there is already a standard for that.
In the U.S. before there was a national standard for time, every town set their own time according to their local norms and regulations. That made making railway timetables an absolute nightmare. That spurred people on to standardize time nationally. Now we take it for granted, forgetting how different life once was. I hope one day in our lifetimes we will take it for granted that the data that comes from our cameras is not merely accessible, but that our degree of control of it allows the true freedom to do what we want with it, now and in the future. I fear however that the factors that Bart has identified are all too real, and unlike office document formats, not enough people care.
-
It seems to me that what you are arguing for is far narrower than aspects of the DNG spec itself.
As the bare minimum yes but simply saving out the spec'ed DNG is possible today and provides even MORE benifits to my workflow. How and why should any of this be different with the raw as with the JPEG? Treat them equally.
-
Hi,
The image data is very simple, it just RGB values. To interpret the RGB values you need white balance (two numbers) and color conversion matrix (9 numbers). Additional data can be helpful. DNG can contain proprietary data. In fact you can embed a raw file as a whole in a DNG file.
Best regards
Erik
I believe that the RGB data may be lossless or lossy compressed, meaning that interpreting the bits representing "raw rgb data" may be non-trivial.
Of course, for manufacturers to not disclose the encoding of those rgb data, instead having people like Dave Coffin reverse-engineering it is bad for customers.
-h
-
As the bare minimum yes but simply saving out the spec'ed DNG is possible today and provides even MORE benifits to my workflow. How and why should any of this be different with the raw as with the JPEG? Treat them equally.
What you're advocating is analogous to a word processing document format in which the textual content is saved in an open format but all the formatting (e.g. text formatting, page size etc.) is proprietary. It's not a vision I share.
-
What you're advocating is analogous to a word processing document format in which the textual content is saved in an open format but all the formatting (e.g. text formatting, page size etc.) is proprietary. It's not a vision I share.
That's your analogy, I made clear what I desire, a DNG from the camera as an option. If you prefer JPEG or the camera proprietary raw, I'm AOK with that.
-
Pentax too. And Hasselblad.
Some Pentax cameras have raw DNG output, but the higher end models have PEF.
Rob
-
I believe that the RGB data may be lossless or lossy compressed, meaning that interpreting the bits representing "raw rgb data" may be non-trivial.
Of course, for manufacturers to not disclose the encoding of those rgb data, instead having people like Dave Coffin reverse-engineering it is bad for customers.
-h
It's a complete other thing to make the encoding available, even with specific contracts (including NDA's) or using a format that is published from another (American) company.
-
Hi,
Customers who buy a system should have ownership of their data. To have to sign an NDA so you can process your images with the tool of your choice is a bit obscene.
Best regards
Erik
It's a complete other thing to make the encoding available, even with specific contracts (including NDA's) or using a format that is published from another (American) company.
-
Hi,
Customers who buy a system should have ownership of their data. To have to sign an NDA so you can process your images with the tool of your choice is a bit obscene.
Best regards
Erik
Erik
Can you point me to the location of the PS/Lightroom/dxo/C1 information how to get to the data, aka all parameters I have set?
-
http://wwwimages.adobe.com/www.adobe.com/content/dam/Adobe/en/products/photoshop/pdfs/dng_spec_1.4.0.0.pdf
http://www.adobe.com/support/downloads/detail.jsp?ftpID=5474
Best regards
Erik
Erik
Can you point me to the location of the PS/Lightroom/dxo/C1 information how to get to the data, aka all parameters I have set?
-
http://wwwimages.adobe.com/www.adobe.com/content/dam/Adobe/en/products/photoshop/pdfs/dng_spec_1.4.0.0.pdf
http://www.adobe.com/support/downloads/detail.jsp?ftpID=5474
Best regards
Erik
Erik
That's not what I was asking. I'm talking about complex changes to images on layers, like dodging and burning, spot healing, cloning, sharpning etc. All data that's being made while PP's an image.
-
It's a complete other thing to make the encoding available, even with specific contracts (including NDA's) or using a format that is published from another (American) company.
I don't see your point.
If I use a trivial 1970s compression scheme to compress my rgb data by 2:1, my users should not have to reverse-engineer actual files in order to be able to access their data. If I am a billion dollar camera company, I should allocate 3 days of engineering time to write a mock-up code (using some high-level inefficient language) example decoding the current scheme and release it as-is, e.g. open-source.
I am not asking Canon to tell us how they do noise reduction or demosaic or lense correction, as that is something that can fairly be considered trade-secrets.
-h
-
Erik
That's not what I was asking. I'm talking about complex changes to images on layers, like dodging and burning, spot healing, cloning, sharpning etc. All data that's being made while PP's an image.
In the case of Lightroom, a surprising amount of verbal description of editing history is included as meta-data.
Of course, Adobe cannot tell you exactly how to translate that description into image manipulation, as that would effectively be documenting their proprietary image processing engine.
I believe strongly in opening up data, APIs, while I accept that your core intellectual property should be hidden. Not only because this is good for society, but because I think that it improves your business.
-h
-
Hi,
Working on an answer, but not there yet.
When saving a DNG file in say TIFF a 'side car' file is exported, like the one below:
http://echophoto.dnsalias.net/ekr/Articles/MFDJourney/RawImages/MFDB_VS_DSLR2/Femore2_20131124-CF044335.xmp
According to the DNG documentation I posted this is stored with tag=700 in a DNG file.
That file contains among other data settings for the gradient tool:
<crs:GradientBasedCorrections>
<rdf:Seq>
<rdf:li>
<rdf:Description
crs:What="Correction"
crs:CorrectionAmount="1.000000"
crs:CorrectionActive="true"
crs:LocalExposure="-0.124183"
crs:LocalSaturation="0.294118"
crs:LocalContrast="0.000000"
crs:LocalClarity="0.000000"
crs:LocalSharpness="0.000000"
crs:LocalBrightness="-0.098039"
crs:LocalToningHue="0.000000"
crs:LocalToningSaturation="0.000000"
crs:LocalExposure2012="-0.156453"
crs:LocalContrast2012="0.000000"
crs:LocalHighlights2012="-0.411765"
crs:LocalShadows2012="0.000000"
crs:LocalClarity2012="0.202614"
crs:LocalLuminanceNoise="0.000000"
crs:LocalMoire="0.000000"
crs:LocalDefringe="0.000000"
crs:LocalTemperature="0.000000"
crs:LocalTint="0.000000">
<crs:CorrectionMasks>
<rdf:Seq>
<rdf:li
crs:What="Mask/Gradient"
crs:MaskValue="1.000000"
crs:ZeroX="0.596873"
crs:ZeroY="0.631574"
crs:FullX="0.595705"
crs:FullY="0.375582"/>
</rdf:Seq>
</crs:CorrectionMasks>
</rdf:Description>
</rdf:li>
</rdf:Seq>
</crs:GradientBasedCorrections>
As you can see this information is readable. On the other hand it just lists parameters which are used, for instance:
crs:LocalHighlights2012="-0.411765"
Tells that LocalHiglights is at -0.411765, to implement it you need emulate higlight compression method in Lightroom.
You can easily transfer non "well know" parameters, like exposure, ranking and cropping. Indeed, many tools handling DNG files transfer exposure, cropping etc, but not proprietary methods.
Best regards
Erik
Erik
That's not what I was asking. I'm talking about complex changes to images on layers, like dodging and burning, spot healing, cloning, sharpning etc. All data that's being made while PP's an image.
-
Erik, is there anything in those xmp instructions that exposes how Adobe's color engine translates those instructions to the video card in order to evoke a response from the user editing and creating the instructions?
There's got to be some level of proprietary "secret sauce" even with Adobe that makes what looks like non-proprietary instructions in the form of xmp code create magic in one Raw converter but crap in another using the same instructions.
Converting to a DNG format doesn't show how it controls how that works in order to clearly indicate it's not considered proprietary. DNG is just a container of instructions and data no matter what Raw converter creates it and so if all camera manufacturers and software makers adopted DNG then the instructions written and embedded should deliver the same results across all converters regardless of the originator of the DNG.
IOW is the proprietary portion of a Raw workflow still most prominent in the source file or the converter that first creates and embeds the instructions that deliver the initial look of the image that can't be considered proprietary since everyone is now using the same DNG format (when and if that ever happens)?
-
DNG is just a container of instructions and data no matter what Raw converter creates it and so if all camera manufacturers and software makers adopted DNG then the instructions written and embedded should deliver the same results across all converters regardless of the originator of the DNG.
No. There should always be a fundamental difference between data and it's transformation. Transformation varies; the original data is the same. Metadata specifies how the data was transformed and with which version of the process.
-
No. There should always be a fundamental difference between data and it's transformation. Transformation varies; the original data is the same. Metadata specifies how the data was transformed and with which version of the process.
I think it's important to put the various bits and pieces, in terms of what is and isn't (or will be) proprietary.
The container may be proprietary for a few months. Canon releases a 32DII Jan 1 2014. The raw data container is proprietary such only those with Canon's software can process their raw data. This is unacceptable for a lot of reasons. Imagine if Canon produced a raw file that only their raw converter could process. All the DNG naysayers would be shitting bricks (I'd love to see that).
Months after any or all raw converter manufacturers spend time and money to understand a container, with or without some other proprietary bits unnecessary for them to process that data, the container is understood by said raw converters. We waited months to unlock something that should not have been locked in the first place and cannot be locked such others can't have access to that data. WHY? The DNG container isn't proprietary. If on Jan 1 2014 the Canon 32DII spit out a DNG as the spec defines it, we'd be processing that raw in LR or C1 or Iridient Developer that very day.
There is proprietary processing. Nothing unusual, nothing unexpected. I don't expect C1 to handle the same raw data within it's processing engine, with proprietary tags or not as LR or Iridient Developer. No one (as yet) is asking for a universal raw converter that understands everyone else’s processing instructions (proprietary) and can handle that data the same way. Unnecessary. Even if we took 4 different raw converters, all having the same slider called Saturation, all having the same scale (0-100), expecting instructions for Saturation +7 to be applied in C1 from LR's instructions isn't going to happen and isn't expected. Even if it understood +7 Saturation, the entire proprietary raw processing engine is different and will produce a different result. Not an issue! I no more expect LR's proprietary Saturation to be the same as C1's as I expect the E6 lab in Santa Fe to produce identical processing as A&I in LA on all film types.
DNG allows the camera raw data, and other data that may be almost as important to be accessible to all the converters the day the camea ships. Not 3 months later when all the people writing code have to stop doing so in terms of making a better product, but solely to access my data, data I should have been able to feed to the processor of my choice on day one. I bring up JPEG and will do so again. Why isn't the raw data treated just like the JPEG in terms of my ability to access that data on day one? Not months later when the enviable happens as it has for years and years: I have access to my raw data because the engineers of my raw converter have hacked what was proprietary (mostly container data) that never needed to be in that state in the first place. Why?
-
Erik, is there anything in those xmp instructions that exposes how Adobe's color engine translates those instructions to the video card in order to evoke a response from the user editing and creating the instructions?
There's got to be some level of proprietary "secret sauce" even with Adobe that makes what looks like non-proprietary instructions in the form of xmp code create magic in one Raw converter but crap in another using the same instructions.
The entire processing is proprietary, there's lots of it. See my last post. You can produce XMP that says +18 Vibrance. You could have two products that actuallty have a silder called Vibrance but you'll get different results even if (big if) product A understood product B's XMP that has XMP with +18 Vibrance in it.
In a way, this is lovely, your raw image data can be molded as you desire based on a product you prefer for it's controls and rendering. Otherwise, raw is raw and the DNG part of all this is simply having a camera spit out a non proprietary document we can use.
-
Imagine if Canon produced a raw file that only their raw converter could process.
Hi Andrew,
That's a Red Herring statement. Even the Foveon format can be decoded by e.g. DCraw, but it's not as good as the dedicated converter can achieve. Any Raw format can (and will) be decodable. It's a non-argument.
We waited months to unlock something that should not have been locked in the first place and cannot be locked such others can't have access to that data. WHY?
Frankly, non-sense, because the camera came with a Raw converter. Maybe not you favorite one at that time, but your files would be accessible. Besides, putting the data in a DNG container makes no difference, the structure would still need to be understood. Raw or DNG, which is the topic of this thread, makes no difference. Remember the Fuji Xtrans 'quality' in LR/ACR, compared to Capture One which was pretty good from the get go?
The DNG container isn't proprietary.
But that's not what the thread is about. The data in the container can still be proprietary, so that container makes no difference whatsoever.
If on Jan 1 2014 the Canon 32DII spit out a DNG as the spec defines it, we'd be processing that raw in LR or C1 or Iridient Developer that very day.
Nope, it takes time to reverse engineer a proprietary set of 'data' before it becomes an image. The container makes no difference in that respect. The only thing that the DNG container allows is to work smoothly with e.g. Lightroom, in the sense of storing proprietary processing steps in the same container, and a few other nice features for some, but irrelevant for others.
I understand you do not like to be obstructed in access to your original image content, but that is a different discussion. Even film required a conversion process. Kodachrome also could not be processed in chemicals intended for another conversion, but a conversion will always be part of photography.
Cheers,
Bart
-
Bart.....very well said.....
-
Hi,
I would say that there is a need to make a distinction of raw data and the processing applied to it. Data that has been processed is not raw any longer, it is cooked.
DNG gives a well defined structure to store raw data and it defines means to describe the data. Having a standardised description doesn't mean that the data can be used, for that the application process needs to know how to interpret the data. Data from Bayer sensors can be interpreted at ease, but non bayer data may need special processing. Packing data in a DNG container doesn't mean the data can be interpreted immediately, but makes it probably that it can be handled correctly.
DNG can also contain proprietary maker data.
On the other hand, DNG can also contain processing instructions for the data. These are in a well defined format and are humanely readable, which reduces the risk they get inreadable with time. An application can just read the information, and parse it. On parsing and processing a tag it can invoke whatever method it wants or just skip the tag.
Best regards
Erik
Erik, is there anything in those xmp instructions that exposes how Adobe's color engine translates those instructions to the video card in order to evoke a response from the user editing and creating the instructions?
There's got to be some level of proprietary "secret sauce" even with Adobe that makes what looks like non-proprietary instructions in the form of xmp code create magic in one Raw converter but crap in another using the same instructions.
Converting to a DNG format doesn't show how it controls how that works in order to clearly indicate it's not considered proprietary. DNG is just a container of instructions and data no matter what Raw converter creates it and so if all camera manufacturers and software makers adopted DNG then the instructions written and embedded should deliver the same results across all converters regardless of the originator of the DNG.
IOW is the proprietary portion of a Raw workflow still most prominent in the source file or the converter that first creates and embeds the instructions that deliver the initial look of the image that can't be considered proprietary since everyone is now using the same DNG format (when and if that ever happens)?
-
Maybe a kind soul could write an article for Lula going into the problems DNG tries to solve, the barriers it faces, and just as importantly put it in historic context of what we've learned in the last few generations regarding best practices regarding digital data and its manipulation. It would be great if such an article could cover not only immediate benefits to a whole range of people, e.g. using shared metadata to facilitate inter-program processing (a trivial example is Photo Mechanic reading crop data from ACR XMP files), but also the benefits to specialists like archivists. I know Jeff and Michael talked about DNG in one of the video series that came out a few years ago, but they didn't go into this material. Why, for instance, do we have two de facto industry standards for electronic office documents, but in comparison DNG is struggling? There might be very good reasons. I for one would be interested to hear an expert with inside knowledge discuss such reasons.
-
Even film required a conversion process. Kodachrome also could not be processed in chemicals intended for another conversion, but a conversion will always be part of photography.
Agreed. But the film, processing and the labs that could do this were an option the day I purchased the film. That's not the case today and it should be. This isn't about a conversion process. It is about the choice of that process which to some of us is rather important. The rational that I can process that data using the supplied manufacturer's converter doesn't wash, the entire force of process is the issue.
-
Agreed. But the film, processing and the labs that could do this were an option the day I purchased the film. That's not the case today and it should be. This isn't about a conversion process. It is about the choice of that process which to some of us is rather important. The rational that I can process that data using the supplied manufacturer's converter doesn't wash, the entire force of process is the issue.
Hi Andrew,
I also like full control of my images/data. But I also don't mind switching to another application/process if that gives me better results. It's the result that counts for me. It also makes the workflow more complicated for me, but I don't believe there is a single universal best solution either.
This means that the file format, or the container of my image data, does not constitute the biggest weight in my decision process. For you that may be different, no problem with that.
However, DNG does not solve these issues, it's just a convenient container mostly when used in combination with Adobe software (less so with other software, currently). And Adobe, with their "perpetual pay or be locked out" subscriptions, is not exactly the first company I think of when it comes to unrestricted access to my data. So maybe that's another reason I'm not overly enthusiastic about that proposition. The DNG container does not solve my issues, especially when it 'forces' me in the arms of Adobe it tends to do the exact opposite.
Raw data is not just a set of RGB values that needs to be stored in a predefined space in the container, it also requires info about black-points, saturation point, color-balance between CFA filter transmissions and spectral range/overlap, CFA layout, color temperature of the dominant illuminant if Auto WB fails due to subject colors, sensor response curve (usually linear but can also be gamma encoded like with Leica raws), has sensor calibration data to adjust for PRNU noise and/or amp glow, maps out hot/dead sensels, may incorporate lens parameters, just to name a few obvious ones.
The container it comes in is not the problem as such, nor is it the solution, but it's the decoding of the data that makes a difference. And that decoding is more complex than deciding if we're dealing with a Color negative or Reversal film process. Such is the nature of the beast we call digital photography. We may or may not like the complexity of the process, but we do like the results when good image quality is the result. The container is not the issue.
Cheers,
Bart
-
On the other hand, DNG can also contain processing instructions for the data. These are in a well defined format and are humanely readable, which reduces the risk they get inreadable with time. An application can just read the information, and parse it. On parsing and processing a tag it can invoke whatever method it wants or just skip the tag.
Erik, that concept makes sense now as long as it assumes those humanely readable instructions will be an adopted standard and readable by more advanced software decades from now. Archivists should be the most concerned. This also applies to DNG as an adopted standard.
I'ld REALLY like to have actual engineers or similarly technically knowledgable personnel from Nikon, Canon and other camera manufacturer non-DNG adopters tell the rest of the industry (including us) why they shun DNG format as Andrew has been pointing out. That's what's annoying.
Nikon and Canon aren't stupid and I don't see them as greedy opportunists guarding their "secret sauce" proprietary Raw format, so I'm guessing they must have thought this through and come up with good reasons. Just put us all out of our misery and give us a logical answer will suffice.
-
However, DNG does not solve these issues, it's just a convenient container mostly when used in combination with Adobe software (less so with other software, currently).
The same could be said of the JPEG. I just want the DNG so I have no more restriction than I'd get selecting JPEG. DNG solves that.
The container it comes in is not the problem as such, nor is it the solution, but it's the decoding of the data that makes a difference.
It is a problem if I want to decode that data and the decoder I wish to use doesn't understand what it will understand a few weeks or months later. That cycle is unnecessary.
-
Nikon and Canon aren't stupid and I don't see them as greedy opportunists guarding their "secret sauce" proprietary Raw format, so I'm guessing they must have thought this through and come up with good reasons. Just put us all out of our misery and give us a logical answer will suffice.
Sadly, I think you are wrong about Nikon and Canon guarding their "secret sauce" proprietary Raw format...
In terms of their "objections", over the years, Thomas has revised the SDK based upon the camera companies technical objections. Their objections regarding the Adobe devised (and owned) standard has also been addressed by Adobe by Adobe offering DNG for TIFF-EP. Unfortunately, standards bodies tend to go slow and those bodies include the companies who currently don't use DNG.
The camera company's objections to adopting DNG keep getting less and less defendable...the bottom line is they don't adopt DNG because they don't want to. When will they want to? When they have to because of industry pressure which starts with photographers.
-
The camera company's objections to adopting DNG keep getting less and less defendable...the bottom line is they don't adopt DNG because they don't want to. When will they want to? When they have to because of industry pressure which starts with photographers.
With the polled ratio of RAW to DNG users being a bit over 2:1, it wouldn't seem that they (camera makers) need worry too much. That is if the process is democratic. :)
-
With the polled ratio of RAW to DNG users being a bit over 2:1, it wouldn't seem that they (camera makers) need worry too much. That is if the process is democratic. :)
...And I am sure many, such as I, did not bother to vote (RAW).
Even if I was a confirmed DNG advocate. I would not stop buying cameras over the issue...and I do not understand what other lever would be effective...certainly not this forum.
-
Sadly, I think you are wrong about Nikon and Canon guarding their "secret sauce" proprietary Raw format...
In terms of their "objections", over the years, Thomas has revised the SDK based upon the camera companies technical objections. Their objections regarding the Adobe devised (and owned) standard has also been addressed by Adobe by Adobe offering DNG for TIFF-EP. Unfortunately, standards bodies tend to go slow and those bodies include the companies who currently don't use DNG.
The camera company's objections to adopting DNG keep getting less and less defendable...the bottom line is they don't adopt DNG because they don't want to. When will they want to? When they have to because of industry pressure which starts with photographers.
Canon and Nikon have as much right to protect their "secret sauce" as Adobe does.
-
Canon and Nikon have as much right to protect their "secret sauce" as Adobe does.
No one nor the DNG container denies them of this. No one's suggested that they don't have that right. So it's pointless.
-
No one nor the DNG container denies them of this. No one's suggested that they don't have that right...
Absolutely correct - how all the 0's and 1's get into the file, proprietary or DNG, is proprietary, the file itself should not be.
Tony Jay
-
Hi Andrew,
I also like full control of my images/data. But I also don't mind switching to another application/process if that gives me better results. It's the result that counts for me. It also makes the workflow more complicated for me, but I don't believe there is a single universal best solution either.
This means that the file format, or the container of my image data, does not constitute the biggest weight in my decision process. For you that may be different, no problem with that.
However, DNG does not solve these issues, it's just a convenient container mostly when used in combination with Adobe software (less so with other software, currently). And Adobe, with their "perpetual pay or be locked out" subscriptions, is not exactly the first company I think of when it comes to unrestricted access to my data. So maybe that's another reason I'm not overly enthusiastic about that proposition. The DNG container does not solve my issues, especially when it 'forces' me in the arms of Adobe it tends to do the exact opposite.
Raw data is not just a set of RGB values that needs to be stored in a predefined space in the container, it also requires info about black-points, saturation point, color-balance between CFA filter transmissions and spectral range/overlap, CFA layout, color temperature of the dominant illuminant if Auto WB fails due to subject colors, sensor response curve (usually linear but can also be gamma encoded like with Leica raws), has sensor calibration data to adjust for PRNU noise and/or amp glow, maps out hot/dead sensels, may incorporate lens parameters, just to name a few obvious ones.
The container it comes in is not the problem as such, nor is it the solution, but it's the decoding of the data that makes a difference. And that decoding is more complex than deciding if we're dealing with a Color negative or Reversal film process. Such is the nature of the beast we call digital photography. We may or may not like the complexity of the process, but we do like the results when good image quality is the result. The container is not the issue.
Cheers,
Bart
Bart, unfortunately, it don't think the DNG advocates want just the info in the container....including the proprietary info.
If all they did was put it in the container, Adobe would need to do what they do today and write the "translator" to make use of it....which Adobe would then use with their proprietary code.
They want the info from the camera manufacturers to be translated into the Adobe defined form, with no hidden proprietary info standing in the way, so that Adobe does not have to do it.
At least, that is what I am getting from all their responses.
-
Hi,
I guess I would! I bought a Sony 100 RX after reading Michaels article of glowing price, than there was almost three months of wait before I could use raw, I will never buy a camera again that is not supported by Lightroom! Film cameras excepted. That's a promise.
Best regards
Erik
...And I am sure many, such as I, did not bother to vote (RAW).
Even if I was a confirmed DNG advocate. I would not stop buying cameras over the issue...and I do not understand what other lever would be effective...certainly not this forum.
-
The camera company's objections to adopting DNG keep getting less and less defendable...the bottom line is they don't adopt DNG because they don't want to. When will they want to? When they have to because of industry pressure which starts with photographers.
Jeff, did you hear this or read it from authoritative personnel from these camera companies that they just don't want to adopt DNG? Or is this your assumption based on their actions?
As for photographers pressuring these companies to go DNG there's not much to do but write them and keep bugging them about it with the implied threat to buy from their competitors who do offer DNG support which there are few.
I don't see a lot of folks dropping Nikon and Canon just over a data file format option any time soon.
Any other ways to pursued, Jeff?
-
Hi,
I guess I would! I bought a Sony 100 RX after reading Michaels article of glowing price, than there was almost three months of wait before I could use raw, I will never buy a camera again that is not supported by Lightroom! Film cameras excepted. That's a promise.
Best regards
Erik
So you would delay you purchase.
-
Yes,
Until the camera is supported by LR.
Best regards
Erik
So you would delay you purchase.
-
They want the info from the camera manufacturers to be translated into the Adobe defined form, with no hidden proprietary info standing in the way, so that Adobe does not have to do it.
No, that isn't what we want. But at this point, it is clear you're not following along. If you were, you wouldn't have made the comment about Canon and Nikon have as much right to protect their "secret sauce" as Adobe does. Pointless, obvious, not part of the problem or solution, no one has ever suggested otherwise. One of Jeff's described rabbit holes.
-
No, that isn't what we want. But at this point, it is clear you're not following along. If you were, you wouldn't have made the comment about Canon and Nikon have as much right to protect their "secret sauce" as Adobe does. Pointless, obvious, not part of the problem or solution, no one has ever suggested otherwise. One of Jeff's described rabbit holes.
Interesting....you said the above, in response to my statement, "...They want the info from the camera manufacturers to be translated into the Adobe defined form, with no hidden proprietary info standing in the way, so that Adobe does not have to do it..."
If you are going to quote Jeff's rabbit hole statement, quote the applicable part of his post, where he says, "...By the time the capture data is written to media, it's just data. And it's data that Dave Coffin, Thomas Knoll and Eric Chan can "easily" decode (when I say "easily" what I mean is it's time wasted decoding new cameras simply because the camera companies can get away with it–and it's time Thomas and Eric don't have to spend on new processing algorithms)..."
I think it is pretty clear. You guys want the camera manufacturers to change what they do to make it easier and less costly for Adobe. This is irrespective of how it changes what they do, impacts any of their future development, and at the manufacturer's cost....therefore their customer's costs....because, as I sure you are aware....all cost must eventually be passed on to the customer.
Am I reading this wrong? Are you guys in agreement or not? Or, as seems obvious to me, you keep changing your response to poke at any disagreement that comes up.
-
Yes,
Until the camera is supported by LR.
Best regards
Erik
I cannot say that I would not do the same. :-)
-
Jeff, did you hear this or read it from authoritative personnel from these camera companies that they just don't want to adopt DNG? Or is this your assumption based on their actions?
Yes, I have heard this from multiple, reliable sources from within the companies involved. And, no, I can not name names–sorry.
-
A few years ago, I made a suggestion on a forum (can't remember where) that everyone send the software cd that came with their camera back to the manufacturer(to the world headquarters address) with a note stating that the software was not needed because it did not fully support dng. The note should also state the wish for the camera to have the option to save as dng. I know, the software is not the problem here. But, I think that the mass influx of cds coming in would get someone's attention. Huge numbers of cd's arriving at a single address would get the attention of the postal service and probably the media.
We have to do something to get their attention and to demonstrate the power of the user. If our demonstration is not powerful, then we have no power. If we have no power, the dng issue is of no significance to the manufacturers. Is that where we are at the moment?
-
We have to do something to get their attention and to demonstrate the power of the user. If our demonstration is not powerful, then we have no power. If we have no power, the dng issue is of no significance to the manufacturers. Is that where we are at the moment?
Yep...that's we're we've been since DNG was released...
But don't get down on the effort. It takes a long time to bring about grassroots changes. I'm in this for the long hall (so are Thomas, Eric and all the Adobe guys). Adobe isn't going to stop supporting undocumented, proprietary raw files any time soon. Each quarter, Thomas and Eric (and a couple of others) gather up the latest crop of cameras and decode the raw files so ACR/LR customers get their new camera support. They also update the free DNG Converter so previous users of older software can get backwards compatibility.
The latest camera, once supported by DNG Converter, can be processed in older versions of Camera Raw all the way back to ACR 2.4 hosted in Photoshop CS (that's the first version of CS mind you). And, they do this for free. That says something about Thomas, Adobe and their commitment to the industry.
Also, I know for a fact that all of the camera makers have dissected DNG and the DNG SDK and have learned a lot about raw file format "best practices".
Remember, neither Nikon nor Canon have any real background, history or tradition in digital imaging...prior to digital cameras, their only responsibility was to provide a light tight environment to expose film in. Yes, they have a long history of developing cameras and lenses...digital imaging apps or file formats? Not so much. Kodak was the company that brought Nikon and Canon into the digital age...the early pro digital cameras were offered by Kodak with their sensors built into either a Nikon or Canon mount. Those were really, expensive cameras that early adopters paid dearly for. ($40K or so in some cases). Nikon was the first to jump the Kodak ship followed a couple of years later by Canon.
In both cases, Nikon and Canon took what they learned from Kodak, redesigned and adopted the tech and made their own cameras. Nikon's first pro digital camera (the D1) was released in 1999. The first Canon digital camera for pro use was the Canon D30 (I had one). So, Nikon and Canon have been directly involved in making digital cameras for 14/13 years respectively. In terms of history and tradition, this is only yesterday. Seriously, Nikon and Canon had no friggin' clue how to do this stuff when they started...it was all jump started by Kodak who ended up screwing the pooch and getting outmaneuvered repeatedly by Japanese companies. Think about it...Kodak had the tech, worked with Nikon and Canon to help them go digital...Nikon and Canon jump ship and leave Kodak eating dust. Make no mistake about it...Nikon and Canon are great at taking existing technology and advancing it–that is a massive Japanese capability. Take what somebody else develops and push it forward...kinda like what they've done with DNG :~)
Sadly, as long as photographers let the camera companies off the hook, change won't happen quickly. Look at the results of the posted poll...29.6% save originals as DNG, 70.4% saves as original raw (I actually fall into the raw camp until I finish raw editing when I do convert to DNG–while also keeping a raw copy).
Look, if you are a photographer I encourage people to educate themselves on the relative upside/downside of using DNG. It's a matter of workflow and how it impacts yours. DNG solves a lot of problems (and creates a few others). If you are a photographer, I encourage you to also have a longer view on what's in the long term best interest of the industry. If you think that Nikon and Canon should alter their behavior, speak out. If you honestly think that the behavior of Nikon and Canon regarding proprietary raw files is good for the industry I suggest you look into the mirror and ask yourself why you are so misinformed. Seriously, unless you under contract to advocate Nikon or Canon software, I simply can't understand how you can possibly think this is ok.
And, for those of you who hate Adobe and want to see them fail, understand that Adobe doesn't expect to benefit by DNG (or some other format) being adopted by the industry in general. DNG isn't something Adobe is trying to use to create vender lock-in. It's free for anybody to use. Software besides ACR/LR can use and benefit from DNG. DNG is a gift by Thomas Knoll to the industry. You wanna look a gift horse in the mouth? Go ahead...horses mouths are very pretty to look at :~)
-
A few years ago, I made a suggestion on a forum (can't remember where) that everyone send the software cd that came with their camera back to the manufacturer(to the world headquarters address) with a note stating that the software was not needed because it did not fully support dng. The note should also state the wish for the camera to have the option to save as dng. I know, the software is not the problem here. But, I think that the mass influx of cds coming in would get someone's attention. Huge numbers of cd's arriving at a single address would get the attention of the postal service and probably the media.
We have to do something to get their attention and to demonstrate the power of the user. If our demonstration is not powerful, then we have no power. If we have no power, the dng issue is of no significance to the manufacturers. Is that where we are at the moment?
What a good idea. This might get someone's attention.
An alternative (and cheaper / lower effort) option could be this: each time Canon or Nikon release a new model camera, send them an email congratulating them on their new machine but say that sadly you will postpone buying for a few months until the guys at Adobe / Apple / Capture 1 (insert your RAW editor of choice) get around to updating their software to be compatible with the new machine, which could take several months. Then say how frustrated you are not to be able to buy their excellent new camera sooner, and finally point out that this inconvenience could be completely avoided if they, the camera maker, would include a DNG option in future models.
Perhaps we could get a campaign going on Lula: someone could post an alert each time Canon or Nikon bring out a new camera, and I am sure someone like Jeff would know the of the most appropriate people to direct the emails to.
-
Hi
Once upon the time I was interested in something called DxO. They supported my camera that time, I had a Konica-Minolta A2, than I upgraded camera firmware, and DxO didn't work with my camera any longer. Goodbuy DxO…
The interesting thing was just a firmware update broke DxO support for my camera.
Best regards
Erik
-
Bart, unfortunately, it don't think the DNG advocates want just the info in the container....including the proprietary info.
If all they did was put it in the container, Adobe would need to do what they do today and write the "translator" to make use of it....which Adobe would then use with their proprietary code.
They want the info from the camera manufacturers to be translated into the Adobe defined form, with no hidden proprietary info standing in the way, so that Adobe does not have to do it.
At least, that is what I am getting from all their responses.
We need to clarify this or else the discussion is going to keep going around in circles! Is the DNG format designed from the outset by Thomas Knoll to:
- Ensure that the RAW data from the digital camera sensor of all different manufacturers who use it is written in a uniform manner, regardless of sensor design, analogous to the way the world has settled on Unicode as the standard representation of text? or
- Ensure that the RAW data is in an industry standard container?
I am putting aside other important considerations like the purpose behind an open standard for metadata, including metadata for transforming the RAW data, because it seems impossible to get everyone on the same page here.
-
Each quarter, Thomas and Eric (and a couple of others) gather up the latest crop of cameras and decode the raw files so ACR/LR customers get their new camera support. They also update the free DNG Converter so previous users of older software can get backwards compatibility.
Hi Jeff,
You make it sound as if they are doing this pro-bono, in their spare time. Fortunately that's not the case, they get paid by Adobe, because it helps the company's bottom line ... (they would not likely get paid to waste company resources). And since Adobe is not in this game for charity, they must believe it will benefit their investors to do the effort.
The latest camera, once supported by DNG Converter, can be processed in older versions of Camera Raw all the way back to ACR 2.4 hosted in Photoshop CS (that's the first version of CS mind you). And, they do this for free. That says something about Thomas, Adobe and their commitment to the industry.
As in they do not charge customers directly for the application, but indirectly. Or are you suggesting the are doing it in their spare time, without getting paid to do it, and without benefit for Adobe?
Also, I know for a fact that all of the camera makers have dissected DNG and the DNG SDK and have learned a lot about raw file format "best practices".
You've got to be kidding, that's reversing history as it evolved. The DNG format was based on the earlier TIFF/EP (http://en.wikipedia.org/wiki/Tag_Image_File_Format_/_Electronic_Photography) standards and a number of other open standards which have been around quite a bit longer than DNG. DNG is compatible with TIFF/EP.
Remember, neither Nikon nor Canon have any real background, history or tradition in digital imaging...prior to digital cameras, their only responsibility was to provide a light tight environment to expose film in.
Ludicrous. Canon e.g. has been involved in electronic imaging since the 1960s. And of course, duh, before digital cameras there were no digital cameras. However, they have a long history of involvement in various aspects of imaging, scanning, and electro-photographic imaging with toners in copiers and printers, and the production of equipment for medical applications, and scientific research, and cameras for broadcasting, and photo-lithography, part of which is used to produce wafer steppers for the production of integrated circuits. EDIT Here (http://www.canon.com/technology/approach/history/digital_tech.html) is some additional info about digital imaging sensor research.
Enough said about your attempt to discredit the real innovators, just have a look at this nice summary (http://www.canon.com/technology/pdf/Canon_Technology_Highlights_2013_e.pdf) of what a company like Canon really knows about imaging, and compare it to the interesting but humble contributions of Adobe.
We need to clarify this or else the discussion is going to keep going around in circles! Is the DNG format designed from the outset by Thomas Knoll to:
- Ensure that the RAW data from the digital camera sensor of all different manufacturers who use it is written in a uniform manner, regardless of sensor design, analogous to the way the world has settled on Unicode as the standard representation of text? or
- Ensure that the RAW data is in an industry standard container?
I am putting aside other important considerations like the purpose behind an open standard for metadata, including metadata for transforming the RAW data, because it seems impossible to get everyone on the same page here.
I agree, the DNG fanboy-ism is not helpful. However, the purpose behind an open standard is potentially also partly responsible for the success or failure of such an attempt. It is an important consideration indeed.
Cheers,
Bart
-
The anti-DNG fanboyism is far less helpful. It even denies the possibility of enlightened self-interest.
John
-
The anti-DNG fanboyism is far less helpful. It even denies the possibility of enlightened self-interest.
Hi, John,
Then by all means, do share your vision on how DNG is going to (or is designed to) benefit non-Adobe centric digital imaging, or image processing.
Cheers,
Bart
-
Hi Bart,
I find it boring that there is so much Japan bashing in the world. It is often said that Japanese don't know how to develop software/firmware and so on. I have much respekt for Japanese innovation.
On the other hand Adobe has created much of standard tools in the photographic industry. If we mention TIFF it is a set of formats originating from Aldus/Adobe, so when you refer to TIFF/EP as an open standards predating DNG you also refer to Adobe technology.
Personally, I object to the state of the industry, where not only has each vendor an own format but that format is nut compatible between different products from the same firm. A company like Adobe or Phase One or anyone else needs to support not just CR2 or NEF but different versions of those programs. The differences are trivial.
Another question, if your raw vendor goes out of business, who will keep your raw format alive? Yes, Nikon and Canon may live forever, but there is a lot of speculation that other companies may go belly up.
It is very hard to demonstrate what benefit proprietary formats offer for either vendors or customers. What is raw data really? An array binary data, supposedly representing the sampled sensor signal. A couple of matrices describing the conversion from sensor RGB to CIE XYZ. A few numbers describing white balance. A list of bad pixels. What is secret about it, once Nikon even encrypted WB information.
Who owns my images, I the photographer or my camera vendor?
A standard format, intelligently used, would help us to use any camera data we wish. A program can use well known and well defined tags and ignore vendor specific tags, which are possible in DNG as well in proprietary format.
Best regards
Erik
You've got to be kidding, that's reversing history as it evolved. The DNG format was based on the earlier TIFF/EP (http://en.wikipedia.org/wiki/Tag_Image_File_Format_/_Electronic_Photography) standards and a number of other open standards which have been around quite a bit longer than DNG. DNG is compatible with TIFF/EP.
-
Yep...that's we're we've been since DNG was released...
But don't get down on the effort. It takes a long time to bring about grassroots changes. I'm in this for the long hall (so are Thomas, Eric and all the Adobe guys). Adobe isn't going to stop supporting undocumented, proprietary raw files any time soon. Each quarter, Thomas and Eric (and a couple of others) gather up the latest crop of cameras and decode the raw files so ACR/LR customers get their new camera support. They also update the free DNG Converter so previous users of older software can get backwards compatibility.
And, for those of you who hate Adobe and want to see them fail, understand that Adobe doesn't expect to benefit by DNG (or some other format) being adopted by the industry in general. DNG isn't something Adobe is trying to use to create vender lock-in. It's free for anybody to use. Software besides ACR/LR can use and benefit from DNG. DNG is a gift by Thomas Knoll to the industry. You wanna look a gift horse in the mouth? Go ahead...horses mouths are very pretty to look at :~)
As always Jeff, I appreciate what you say and the directness with which you say it. I also had a Canon D30 as my first pro dslr. My first digital camera was a Kodak dc 120 in 1998. I remember being amazed at the first image that I looked at on my computer. It only produced raw files. I believe it was KDC format, which no longer can be read by any software that I possess. Not that I want to view any of these extremely noisy low res images!
But, we have come a long way in digital imaging and I agree that some type of standardization will be key to us progressing forward in an organized way. I appreciate the voice that you give to reason in the digital photographic world. Thanks for sharing. Keep up the good work.
-
Hi, John,
Then by all means, do share your vision on how DNG is going to (or is designed to) benefit non-Adobe centric digital imaging, or image processing.
Cheers,
Bart
Haven't you read the thread? Why don't you share your vision of how it doesn't? Why don't you explain how proprietary raw data achieves that goal?
-
Hi,
Lets for instance assume that I want to process a Phase One image in Raw Therapee. Or lets assume that a guy named Anders Torger wants to develop an application to blend HDR images.
Just two examples.
Can you give some example of benefits for any customer of using CR2 over a CR2 encapsulated in DNG?
But, I don't see DNG as a solution to all problems, just as a well defined and well documented container format developed and controlled by the same folks who developed and control TIFF and TIFF/EP. If you don't trust Adobe why would you trust TIFF?
Best regards
Erik
Hi, John,
Then by all means, do share your vision on how DNG is going to (or is designed to) benefit non-Adobe centric digital imaging, or image processing.
Cheers,
Bart
-
Hi Bart,
I find it boring that there is so much Japan bashing in the world. It is often said that Japanese don't know how to develop software/firmware and so on. I have much respekt for Japanese innovation.
Hi Erik,
I agree, and that's why I occasionally feel the need to also give a voice to the other side of the situation, for a more balanced view of what we're actually dealing with. Neither the anti Japanese companies attitude, nor Adobe idolization will lead to progress.
On the other hand Adobe has created much of standard tools in the photographic industry. If we mention TIFF it is a set of formats originating from Aldus/Adobe, so when you refer to TIFF/EP as an open standards predating DNG you also refer to Adobe technology.
Indeed, Aldus was bought by Adobe, but not much progress has been made since:
Adobe Systems, which acquired Aldus, now holds the copyright to the TIFF specification. TIFF has not had a major update since 1992, though several Aldus/Adobe technical notes have been published with minor extensions to the format, and several specifications, ...
One could ask oneself, what image quality improvement do we get from converting a Proprietary Raw format to DNG? If one is honest, the answer is none. It is just another container for the same data. It may benefit those who choose to work with Adobe programs in convenience because of some features that were added for that purpose.
Personally, I object to the state of the industry, where not only has each vendor an own format but that format is nut compatible between different products from the same firm. A company like Adobe or Phase One or anyone else needs to support not just CR2 or NEF but different versions of those programs. The differences are trivial.
I understand, but without getting too philosophical, one could also ask what are the benefits of patents or copyrights? There are those who say that patents slow down progress, and there are also those who claim that without patents there would not be enough incentive to do these investments in costly research without such a mechanism to recover the cost. I believe that issues around intellectual property are what is actually holding back the standardization of Raw formats. To be clear, DNG is not a Raw format, it is a container and as such not a solution.
I also agree that the differences between subsequent versions of Raw files from the same company may be small (when observed casually), and the differences between different manufacturers also appear to be not always that huge. However, that also means that there has to be no backwards compatibility to other Raw formats, maybe even technologies once used but by now surpassed by something superior. It allows to drop certain pieces of irrelevant information, and add new (even before they are unlocked by firmware for specific models). Maintaining backwards compatibility is very expensive, and leads to mistakes.
Another question, if your raw vendor goes out of business, who will keep your raw format alive? Yes, Nikon and Canon may live forever, but there is a lot of speculation that other companies may go belly up.
If any vendor goes out of business, and there is enough financial benefit, there will be others to step in. In fact, I think most Raw formats are readable by some converter, even if they may not provide better conversion quality than was available at the time. Sometimes there are even improvements, like for the Photo CD format where some official libraries produced issues that were later (after Kodak stopped supporting the format) solved by enthousiasts.
Let's also be clear that DNG does not solve this, because it is just a container. It does not provide decoding, it is the application that uses the container that adds that functionality.
It is very hard to demonstrate what benefit proprietary formats offer for either vendors or customers. What is raw data really? An array binary data, supposedly representing the sampled sensor signal. A couple of matrices describing the conversion from sensor RGB to CIE XYZ. A few numbers describing white balance. A list of bad pixels. What is secret about it, once Nikon even encrypted WB information.
Indeed, but as I've said earlier, it almost certainly has to do with Intellectual Property and Patents. Nikon probably thought that they could make/recuperate more money by selling royalties, and leave their competition guessing a bit longer about their secret developments and plans for future innovations.
Who owns my images, I the photographer or my camera vendor?
The photographer owns his images, but nobody is locked out because a converter will be supplied with the camera (and a little while later a reasonable alternative will be available for those who prefer to use other software).
A standard format, intelligently used, would help us to use any camera data we wish. A program can use well known and well defined tags and ignore vendor specific tags, which are possible in DNG as well in proprietary format.
Again, and Damon Lynch asked the right question, what is it that the DNG format intends to do or offer? Clearly converting from one format to another will not improve the quality, and there is a 'free' converter available for those who prefer DNGs for smoother interaction with/between Adobe applications.
DNG currently is nothing else than a container that mostly offers users of Adobe software some benefits. It does not decode anything, because that is the task of the image processing done by the software. I also know that e.g. Capture One V7 is able to extract higher quality conversions out of my legacy Raw files than it could before, so I'm glad that nothing was pre-cooked yet, and the original data was available. Converting my Canon Raws to DNG would not have made any difference, it might have even made it more risky had I done that with one of the earlier DNG versions.
The future proofing argument so far is pretty theoretical. New technologies will continue to replace old technologies, but we can usually convert from one to the next before that window of opportunity passes (think video to DVD, or grammophone record to one of the many alternative digital versions, or film images to scanned versions).
So the question remains, since we do not gain image quality gain by converting, or what else would be a general benefit (also for the Camera makers) from such an activity.
Cheers,
Bart
-
The anti-DNG fanboyism is far less helpful. It even denies the possibility of enlightened self-interest.
John
John...
Not supporting DNG is not being an anti-DNG fanboy.....even if the self appointed industry gurus viciously attack any other views and accuse those who don't support DNG as part of the problem.
They throw out anti-Japanese, totally biased statements demeaning how Canon and Nikon became successful.
Thet relate purported conversations with Canon and/or Nikon where the camera manufacturers reps (high level executives or engineering wonks???) stated they did not want to do DNG....trying to insinuate that this is based on anti industry or anti Adobe reasons.
They allow no credence to the economic and DE advantage the Adobe would gain with a DNG standard, nor the costs and other factors that would burden the camera manufacturers.
From a user standpoint, I, personally, do not see any difference in the world with or without DNG as a standard. I have little fear that CR2 will have support...someplace...during my lifetime and probably far beyond it. DNG, by itself, today, does not buy me any advantage...and no one has shown a future advantage over the CR2 file.
-
So the question remains, since we do not gain image quality gain by converting, or what else would be a general benefit (also for the Camera makers) from such an activity.
Here you go again. Apart from storing image data in a publicly-documented format, apart from day 1 support for shiny new cameras, apart from storing adjustment and descriptive metadata within the file, apart from adjusted previews readable in other apps, apart from validation hash, apart from.... what have the Romans ever done for us?
-
Here you go again. Apart from storing image data in a publicly-documented format, apart from day 1 support for shiny new cameras, apart from storing adjustment and descriptive metadata within the file, apart from adjusted previews readable in other apps, apart from validation hash, apart from.... what have the Romans ever done for us?
I thought Bart covered all of these....at least the ones which are important. He even covered the delay of support for a given converter.
-
Hi,
Lets for instance assume that I want to process a Phase One image in Raw Therapee. Or lets assume that a guy named Anders Torger wants to develop an application to blend HDR images.
Just two examples.
Hi Erik,
The Phase One file example is a particularly interesting one. Phase One does a lot of behind the scenes processing on its files, both in camera and when one uses Capture One. There is lots of calibration taking place to make sure that the stitched sensor array data doesn't show the seams, and intermediate frames may be stored to improve noise performance. After the fact LCCs can be used to reduce color cast issues and or improve the light fall-off characteristics, and lens data to correct for distortions and sharpness fall-off. Lots of proprietary data is involved, and that's how they can make a difference in image quality.
Without that benefit of distinction between them and their competitors, they would probably be in a much poorer situation than they are now (although there are also other factors that play a role).
As for HDR blending, I've been using/defending/promoting tonemapping of 16-bit and HDR data for quite a while (at least more than a decade). I remember discussing the need for Adobe to step up their effort in the field of 16-bit and 32-bit processing (http://www.velocityreviews.com/forums/t240195-p2-re-any-good-16-bit-photo-editors-out-there.html) with one of their programmers on Usenet. It was not seen by Chris to be in enough demand by their users at the time. Which demonstrates that companies do not move until there is a perceived benefit. The efforts to promote DNG must be seen in that light as well.
Can you give some example of benefits for any customer of using CR2 over a CR2 encapsulated in DNG?
Well there is not much technical difference (same data(?) in another container), other than that I can convert my CR2s in pretty much any raw converter, including DPP from Canon (with very decent color quality). DNG containers are generally supported by Adobe applications and a few others, but I get better results with Capture One with CR2s in most cases and the Prime noise reduction in DxO is said to be very good, but currently that software only supports original CR2s not DNG encapsulated versions.
But, I don't see DNG as a solution to all problems, just as a well defined and well documented container format developed and controlled by the same folks who developed and control TIFF and TIFF/EP. If you don't trust Adobe why would you trust TIFF?
True, but Raws are not that poorly documented either, they conform to lots of standards, but not all Makernote parameters are disclosed. It seems that Adobe proponents would like that changed, which seems to be not in the interest of the camera makers, as discussed before.
The TIFF libraries are mostly maintained, including the Big-TIFF version, by independent parties (http://www.remotesensing.org/libtiff/) (maybe Adobe sponsors them but their involvement seems pretty slim).
Cheers,
Bart
-
...and no one has shown a future advantage over the CR2 file.
See previous comments about the Romans.
People were making the same comments as you back then I first started using DNGs around the time of CS3 (it may have been CS2 but it's before parametric editors like LR and Aperture). Some of the potential advantages of DNG were already apparent back then, and over that period I have been able to cash in some "future advantages" that I wouldn't have gained from CR2 or NEF files.
For instance, the embedded previews and thumbnails meant I could see images' adjusted appearance in Extensis Portfolio which I used to manage pictures. I could also output the adjusted images to web / email etc at high speed and without needing to reconvert the files. Only for NEFs, and only if I had used Capture NX, would I have cashed in on that future advantage without DNG.
Similarly consider how DNG has helped me migrate IPTC metadata through various apps over the years. I moved on to iView and PhotoMechanic, and DNG's embedded metadata meant all my previous metadata entry effort wasn't wasted - or I didn't have to figure out ways to exchange data where one or more apps couldn't understand xmp files. I recall using other best-of-breed apps such as Houdeh Geo to add GPS data (remember we're still before LR and Aperture) directly into the DNG file. The data was read by other apps such as my asset manager or Bridge, but I wouldn't have reaped that future advantage if I'd used raw files - instead I would have been trying to figure out whether the metadata-creating app would write directly to the raw file (eek), or battling conflicts between metadata embedded in the file and metadata in xmp sidecars. Ever had to figure out such things? What fun! So again, embedded metadata is a "future advantage" that I have banked.
So over the 8-9 years that I've used DNGs I feel I have crystallised "future advantages" which you would claim didn't exist. I'll cash in on the same advantages again in the future, in the same and in different ways, and I'll benefit from others too. I'll buy / borrow / steal the hottest new Leica and I'll be able to convert its DNGs on day 1. OK, that last future advantage is a bit less probable!
And yes, I do think that not supporting DNG is being anti-DNG (it wasn't me who introduced the f word). You're either on its side or you're in the FUD Front, the Front for DFU, the UFD Organisation....
John
Edited for clarity
-
Haven't you read the thread? Why don't you share your vision of how it doesn't? Why don't you explain how proprietary raw data achieves that goal?
John, see post #2 (Opening up a big can of worms here). This is typical of the 'arugments' we see here ever few months on this subject.
It boils down to two sides of course, often talking over the other and not listening as you point out.
One side asks for an option, points out a problem that actually exists for some, and as was posted in that #2 entry, expresses some of the workflow benefits of an open format container.
The other side, for whatever reason wishes to deny the first group the option. It suggests no problem exists with solutions like "just use the manufacturer's product" and attempts to find any flaw that exists with the said format. Eventually this side has to come up with the Adobe fan boy reference as what they believe to be their ultimate insult and imply that the problem and solution discussed is just an excuse to better Adobe's bottom line, not ours. And yet it appears they continue to use Adobe solutions and Adobe file formats like TIFF.
IF we waved a magic wand and all cameras would produce an open raw file (DNG or otherwise), the nay-sayer’s life would not be affected a lick. Even if the option now helped 5% of users, that benefit to them should not exist and isn't worthwhile for what reason? Never answered. (jrsforums and others) The poll is a perfect example. Unless 100% want a DNG solution or 100% say never build one, we're supposed to believe that both sides cannot live harmoniously. Bullcrap.
Again, if the anti-DNG posters spent a fraction of the time looking at other's workflow requests and ignored them, as it doesn't affect them, we'd all be better off. Instead we have to go into minutia about what is or isn't proprietary and for how long, how DNG does or doesn't provide any benefits etc. One side requests a choice. The other side says it's better if we don't have it. It is typical of the system we see here in the US and why nothing can get done. If progress to aid one group is suggested, it's shot down even if it has no effect on the other party. It's like that silly suggestion about LR6 having modules removed because the installer is too big. Screw anyone who wants to use said modules, I don't so what you do doesn't affect me and go to hell. What a great attitude from what is presumably an industry of image makers and supporters of image makes.
IF you don't like DNG or see the need for an open raw format, by all means go that route, doesn't affect me and I'm happy you are happy with the current system. Some of us are not happy with it, we've been burned in the past, expect to see others burned in the future (for a limited amount of time but that's no justification for the pain). How and why denying those people an option is never expressed, clearly or otherwise. They don't want you to have an option, that has to be good enough now shut up and stop saying anything about Adobe that might be taken as a useful step to aid their customers.
As for anti-Japanese, totally biased statements, I missed them. They don’t belong in the discussion any more than the anti-Adobe totally biased statements.
Agree to disagree. You don't like or use DNG, those of us that do are fine with that. Let those of us that do see a benefit in workflow, the ability to archive our data to do so. IF DNG then falls flat on it's face, no downside for you folks and you can now point out how stupid we were to ask for it. That hasn't happened and can't happen without the option which is all we are asking for.
-
...and no one has shown a future advantage over the CR2 file.
It was in the 2nd post here but it's understandable how and why you'd miss it. It was shown in subsequent posts but it's understandable how you missed it. You don't use DNG so again, it is understandable how you missed it and consistent.
-
John which part of your workflow benefits apart from the embedded previews could not have been realized if we had only the XMP spec? Please don't misread me. I'm not against the DNG format at all. I hope it succeeds. But it seems to me 90% of what you mentioned as having been helpful to you concerns XMP, which works with many different file types.
-
It was in the 2nd post here but it's understandable how and why you'd miss it. It was shown in subsequent posts but it's understandable how you missed it. You don't use DNG so again, it is understandable how you missed it and consistent.
I read you simplistic (but informative to many) article, which was mainly an intro to DNG. I thnk I commented on this article in one of your other post, but am not gonna bother to search.
Your later points...ability to embed DNG profiles in container, Lossy DNG, Fast Load Previews...were not worth commenting about at the time, but will just to prove to you that I am listening...if not believing.
Embed in DNG container....nice in concept, but XMP works just fine for me, and many. Even Jeff says he mainly uses this only if sending RAW data to someone...which we all could use.
Lossy DNG...I don't use, but, if I am correct, I can use it without converting my RAWs to DNG.
Fast Load Previews...I am not sure if you are talking to the Lossy DNG or the embedded data, but eventually, to really work on the info you need the entire RAW (DNG or otherwise) data and the ACR commands. Reading the actual RAW plus cached RAW data from SSD drive will be just as fast as just reading the DNG file....maybe somewhat faster if the DNG file is bloated with the original RAW.
I am consistent....I have no seen nor heard a convincing benefit to me to use DNG....not that I would not use it if it were what the camera gave me. Convince or benefit the camera manufacturers to convert...stop yelling at, based on the poll, the satisfied users.
PS....from your other post....providing a choice Providing a choice, by which I assume you mean the camera mfg. provide a choice of RAW or DNG will effect me and any other buyer of the camera. It will increase DE as code will be needed to provide the choice and testing will be needed for both outputs. This is not trivial. In addition, add'l firmware will be needed and space needed to store it. This will add to component cost. It could also possibly impact timing and throughput. All these factors you seem to trivialize or ignore.
-
I am consistent....I have no seen nor heard a convincing benefit to me to use DNG
Please don't! Is it OK if I and other's do and if we believe it would aid us in having this data output directly from the camera?
-
As for anti-Japanese, totally biased statements, I missed them. They don’t belong in the discussion any more than the anti-Adobe totally biased statements.
Have you not read into the wording and meaning of some of our guru's posts...??? Quite maybe they were not meant...but, to many, the feelings oozed out.
Commenting that Adobe gets a economic and workflow benefit by adoption of DNG is not anti-Adobe.
-
Have you not read into the wording and meaning of some of our guru's posts...??? Quite maybe they were not meant...but, to many, the feelings oozed out.
You appear to read into comments what you desire so what you make of his language or the language I use (even when it is a direct quote of yours) never surprises me!
-
John which part of your workflow benefits apart from the embedded previews could not have been realized if we had only the XMP spec? Please don't misread me. I'm not against the DNG format at all. I hope it succeeds. But it seems to me 90% of what you mentioned as having been helpful to you concerns XMP, which works with many different file types.
I'm not sure I understand your questions, Damon, but it's nothing to do with xmp which exists for both DNGs and proprietary raw files. I was describing how I've benefited in the past from DNGs having adjusted embedded previews (nothing to do with xmp) and from DNG having embedded metadata rather than sidecar-based metadata. The former is still an advantage (I can see LR adjustments in Aperture or PhotoMechanic), the latter less so nowadays since more apps read sidecars.
-
So, how about listing your reasons why you think that camera makers should not offer dng file format as an option in the same manner as they offer jpeg? Why is it a bad idea?
-
You appear to read into comments what you desire so what you make of his language or the language I use (even when it is a direct quote of yours) never surprises me!
I was not the first to make that observation.
-
So, how about listing your reasons why you think that camera makers should not offer dng file format as an option in the same manner as they offer jpeg? Why is it a bad idea?
I believe you will find them throughout this thread.
In particular, concentrate on Bart's posts as I think he has quite clearly stated reasons.
Also, my response to Andrew on "providing a choice".
-
The former is still an advantage (I can see LR adjustments in Aperture or PhotoMechanic), the latter less so nowadays since more apps read sidecars.
Another possible (for some) advantage is one can contain a pretty good JPEG of the current rendering. If so set one can output an impressive 11x17, at least the samples I saw from Peter Krogh, that DAM guy. Yes that takes up more space, but talk about belt and suspenders backup of image data. For those of us working with custom DNG camera profiles, having that data embedded in the container is akin to having rendered images with embedded ICC profiles. Imagine arguing that ICC profiles shouldn't be embedded in TIFFs. Imagine how that would affect your workflow. There are some workflows where lossy DNG is super useful. If you are a wedding photographer and you wish to archive the raws, one large group can be lossy rather than camera JPEGs, the hero's sold can remain DNG OR proprietary raw.
-
Hi,
The fact that DNG is documented container is a great advantage.
Just a theoretical example, let's assume that a vendor called Sonican adds a new model 9x with 54 MP resolution. So they add a new tag to the raw saying '123' meaning the internal project name at Sonican. Sonican software knows it means 54 MP of data at 0X0eff ofset and biased by 0x100 in fake fifteen bits mode encoded into 12 bits space. They just go into the specifications. DNG has a well defined tag for those parameters, as long as well defined tags are properly used any vendor can write raw processing software that handle the file.
Now, Sonican releases a new firmware, and call the camera 123a. The other vendors perhaps even did not realize that 123 was hex? Now they have to figure out what 123a means. With standard and well defined tags a file can be interpreted reasonably well, as long as you can find the necessary tags.
DNG doesn't make the file better, just defines how to interpret it in a consistent way.
Best regards
Erik
DNG currently is nothing else than a container that mostly offers users of Adobe software some benefits. It does not decode anything, because that is the task of the image processing done by the software. I also know that e.g. Capture One V7 is able to extract higher quality conversions out of my legacy Raw files than it could before, so I'm glad that nothing was pre-cooked yet, and the original data was available. Converting my Canon Raws to DNG would not have made any difference, it might have even made it more risky had I done that with one of the earlier DNG versions.
Cheers,
Bart
-
Another possible (for some) advantage is one can contain a pretty good JPEG of the current rendering. If so set one can output an impressive 11x17, at least the samples I saw from Peter Krogh, that DAM guy. Yes that takes up more space, but talk about belt and suspenders backup of image data.
That's what I mean when I refer to adjusted embedded previews, and Peter tells me very few people distinguish the print from the embedded preview from one made directly from the original file. It's partly why I periodically backup "working" DNGs which have been updated (despite my backed-up "virgins" and not-backed-up "working" copies distinction!)
John
-
I'm not sure I understand your questions, Damon, but it's nothing to do with xmp which exists for both DNGs and proprietary raw files. I was describing how I've benefited in the past from DNGs having adjusted embedded previews (nothing to do with xmp) and from DNG having embedded metadata rather than sidecar-based metadata. The former is still an advantage (I can see LR adjustments in Aperture or PhotoMechanic), the latter less so nowadays since more apps read sidecars.
I'm pretty sure I understood your original example sufficiently well. We both already understand what XMP and embedded previews are about. In my case I've written code that manipulates XMP data and I guess you may have done so too. I've read the DNG spec and so I'm aware that you can write the XMP data into the file. I also use Photo Mechanic and Lightroom.
As I've also said, it is not at all clear to me if the main point of DNG really is to encode RAW data from a variety of camera sensors in a uniform, predictable way, like Unicode does for the world's scripts, or it's just a standard container for storing RAW data and metadata. I hope someone can clear that up once and for all. Heck I've read the spec and I can't tally what it says with the simple fact that Canon can add data to their CR2 in post, in ways that appreciably improve image quality, and which are then totally ignored in the conversion of that CR2 to DNG. Why is that? I have no doubt the Adobe and Canon engineers could say so in a second if they were not constrained by their institutional responsibilities.
Finally, if Dave Coffin were to sit in on this conversation I do wonder what he'd have to say!
-
Hi,
I think you need to separate data from processing. If canon does something with the CR2 file that Adobe cannot handle it is probably just a tag. The tag tells DPP (?) what to do with the bits in the mage to achieve a an effect. Such manipulation is made on rendition, it is not really a part of the dng file.
For instance, when I process an image in LR I would probably:
1) Crop
2) Adjust exposure
3) Add a gradient filter to handle sky
4) Adjust shadow clipping
5) Enhance shadow detail
6) Adjust exposure
7) Adjust higlight
8) Adjust contrast
Highlight and shadow adjustment are locally adaptive proprietary algorithms. The instructions say which parameters to use, but nothing about the algorithm. Each software developer is free to emulate the Adobe action, or just do something similar.
Best regards
Erik
I'm pretty sure I understood your original example sufficiently well. We both already understand what XMP and embedded previews are about. In my case I've written code that manipulates XMP data and I guess you may have done so too. I've read the DNG spec and so I'm aware that you can write the XMP data into the file. I also use Photo Mechanic and Lightroom.
As I've also said, it is not at all clear to me if the main point of DNG really is to encode RAW data from a variety of camera sensors in a uniform, predictable way, like Unicode does for the world's scripts, or it's just a standard container for storing RAW data and metadata. I hope someone can clear that up once and for all. Heck I've read the spec and I can't tally what it says with the simple fact that Canon can add data to their CR2 in post, in ways that appreciably improve image quality, and which are then totally ignored in the conversion of that CR2 to DNG. Why is that? I have no doubt the Adobe and Canon engineers could say so in a second if they were not constrained by their institutional responsibilities.
Finally, if Dave Coffin were to sit in on this conversation I do wonder what he'd have to say!
-
Hi,
The fact that DNG is documented container is a great advantage.
Hi Erik,
It's similar to how the current non-DNG Raws are encoded (see here (http://www.sno.phy.queensu.ca/~phil/exiftool/) for extensive documentation of virtually all Raws of some significance), with tags according to the TIFF specifications (both Raw and DNG are variations of TIFF, they both systematically use Tags).
In your example, there will probably be no Tag for '123' be cause that would mean all camera makers would have to agree to make a tag for such a functionality, whatever it does. It's more likely to be a parameter that's hidden in the Maker note block of data. The start and end position of the Maker note data block is described in tags (both in DNG and Raw), but not what its contents means. The content of the data block is only meaningful for the Camera maker, or those who have reverse engineered its meaning.
So any new Camera model will be able to come with new parameters that are not common between all cameras. Utilizing that new functionality will be only possible for the original camera maker, or those who pay royalties to unlock that functionality should they wish to.
DNG doesn't make the file better, just defines how to interpret it in a consistent way.
Again, it's really no different in that respect from all Raws that follow the TIFF specifications. Common info is known and shared, e.g. where the top left sensel within the image data block is located, and what pattern the Bayer CFA has and thus what color that top left pixel represents. However, it does not say exactly what spectral transmission that filter color had. So it may be a very pure Red or one with a trace of Green transmission as well to increase its transparency and make the camera more sensitive to light.
That info, required for very good color conversions may be tied to a patent for a very special Red dye that that camera maker can produce for a fraction of the cost of its competitors, but it bleaches faster than the Red from competitors. They do not want to tip of their competitors, because they might want to also change to a cheaper Red dye or switch source, which would reduce the competitive advantage. Only the original manufacturer will know that they will have to modify the Raw conversion for Red a bit faster as time goes by, to compensate for the rate of bleaching, by e.g. using the number of exposures that's being recorded in the maker notes, combined with the average exposure level that's encoded in common tags in the EXIF section.
Now when Madmanchan does a very good job, he may be able to quickly find an assumption for the Red characteristics to use during conversion for new cameras that's pretty close to what is optimal for the Adobe conversion engine, and within a month or two the Camera will be supported, with the DNG converter than can be used during the first ingestation/conversion into Lightroom. But he won't know of the fading characteristics of that dye because Adobe wouldn't pay royalties for that invention. So that same file will convert the same over time, but later Raw Captures will deviate in color.
Now this is just a silly example, but it might also be something like unique image verification data that's tagged to the file for use in law enforcement situations, or a piece of code that uniquely identifies the retina of the person who looked at the EVF to make that kiddy porn, or a feature that unlocks 3D image data that was encoded by splitting sensels into a left image and right image, which also allows to adjust depth of field after taking the image based on depth clues.
Intellectual Property is big business, and companies will protect that. They will even make life a bit less convenient for people that insist on using a certain brand of converter because those people feel that it suits their workflow better. Those people may perhaps not be able to change DOF after the fact, but just get the left or right image, just like they took it.
Cheers,
Bart
-
I think you need to separate data from processing. If canon does something with the CR2 file that Adobe cannot handle it is probably just a tag.
You might well be right, but if it is a tag (or a series of many of them), it's an extremely big one, because the CR2 can almost double in size as I recall. And although it's definitely done in post, I suppose someone could make the argument that it's legitimate to conceptualize it as "RAW" data because the extra data the CR2 contains is the result of algorithms directly derived from the highly complex optical properties of the lens used to make the image. In other words maybe if the optical engineers had their way, all new cameras could conceivably apply these transformations when creating the RAW image in camera, so it's still "RAW" data. Does that make sense? My point is not to argue either way, just merely that I don't see the value of being a purist about it. (Not that I'm saying you are, mind you!!). I hope I'm making sense. It's the end of a long day for me ;-)
-
Sadly, as long as photographers let the camera companies off the hook, change won't happen quickly. Look at the results of the posted poll...29.6% save originals as DNG, 70.4% saves as original raw (I actually fall into the raw camp until I finish raw editing when I do convert to DNG–while also keeping a raw copy).
Allow me to interrupt the heated debate. I feel like I am walking into no man's land with shells going off left and right.
Nevertheless, among the many posts expressing strong opinions. both pro and con, one can occasionally find bits of information that may be truly useful. The above remark caused me to wonder about this workflow, and what advantages it provides. Jeff, I assume you refer to parametric editing the RAW file in LR/ACR, which places the edits in a sidecar file. To me, one of the major advantages of DNG was having the edits embedded in the DNG file and not having to worry about the location of the xmp. I understand your method eventually accomplishes the same thing when you save as DNG after completing the raw editing. Is there a reason for this approach, in particular for keeping the raw copy as well as the DNG?
-
Hi Jeff,
You make it sound as if they are doing this pro-bono, in their spare time. Fortunately that's not the case, they get paid by Adobe, because it helps the company's bottom line ... (they would not likely get paid to waste company resources). And since Adobe is not in this game for charity, they must believe it will benefit their investors to do the effort.
In point of fact, both Camera Raw and DNG started as unpaid sandbox projects that Thomas wanted to do. Back when Thomas started what Camera Raw would end up becoming, Thomas started working on decoding Canon raw files working by himself while on vacation. He did this because he was exasperated by how poor Canon's software was at the time–pre DPP).
At the time Thomas was an independent contractor with Adobe, not an actual employee. So, he created Camera Raw on his own before Adobe assigned it to him. It turns out that when Thomas returned from vacation, the big hot button for Photoshop was how to deal with raw digital captures and get them open in Photoshop. It turns out that the timing of Thomas' project fit in perfectly. When Adobe asked Thomas about decoding and processing raw captures, Thomas basically said "don't worry, I got that :~)" (seriously, Thomas developing ACR was a fluke (lucky for us).
In terms of DNG, Thomas was getting pretty pissed off having to decode each and every camera's raw files. In developing Camera Raw, he came up with a process of reading the camera metadata and converting the metadata in order to apply normalized default settings. It was the process of taking the raw image data and metadata and normalizing the adjustments that led to developing DNG. Thomas reasoned that the basic raw image data and the image metadata should not be hidden from easy access.
DNG was another personal project Thomas did for himself prior to pitching it to Adobe to release as a free SDK. In point of fact, any raw file is now automatically converted to DNG on the fly so that Camera Raw and Lightroom are getting normalized raw data when opened. So, DNG was a natural offshoot and Thomas convinced Adobe to release DNG as a free SDK and offer it to the industry. And believe me when I tell you that DNG has been VERY informative to both Nikon and Canon (I know this from personal experience while sitting in meetings).
Also note that it was Thomas that talked Adobe into doing the free DNG Converter. Thomas thought it would be a useful tool to allow older applications the ability to read new camera files. I happen to know that Adobe was, uh, a bit reluctant to release a free application that would make new raw files work in old versions of Camera Raw. If you think about it, why would Adobe want to provide a free tool for backwards compatibility to new cameras. Adobe could have simply said no to Thomas and require new cameras to have the user update to the most recent version of Photoshop.
So, yes, ACR and DNG came from Thomas working in his free time. Yes, Adobe did listen to Thomas (Adobe wisely listens really carefully to Thomas).
Recently Thomas was added as an Adobe employee (after all these years being a contractor). And yes, Eric is an employee (thanks goodness–which we could clone him a dozen times).
Look, I was around in the very early days and helped Thomas work with raw files. I rode my motorcycle over to Ann Arbor so he could shoot test files from my Canon D30 to decode. Myself and a few others worked as alpha testers for Camera Raw well before the release of ACR 1 in Feb 2003. I also had multiple meetings with Canon during that time–I was a Canon Explorer of Light back then and met both Canon USA and Canon Japan executives and engineers. During a trip to Japan for Canon, I met the then lead engineer of Digital Photo Pro and got permission to demo Camera Raw to him.
I've been to several conferences where representatives from Adobe, Apple, MSFT, Kodak, Leaf, Nikon and Canon met and talked about digital photography–some of those meetings were, uh, pretty nasty...Nikon has traditionally (back then) been very, very anti-Adobe. Canon a bit less so.
There are some interesting stories I could tell you about but I can't do so in public...once story was a major magazine was just about ready to ink a deal to become the exclusive raw processing application and would require all their photographers to shoot a singe camera brand. The photographers that didn't shoot with that brand (as you might expect) was terrified at the prospect of being forced to change cameras systems. But, in comes Adobe with a camera agnostic raw processing engine (Camera Raw) that made the magazine re-evaltuate this exclusive deal. As you might expect, that camera company was really pissed off that Adobe came up with a camera agnostic method of raw processing–but the photographers were really, really happy that they could keep shooting with their chosen cameras and not be forced to change.
I also know that another camera makers was just about convinced to offer DNG as an additional option (they were convinced it could be done with a firmware update) but something happened (and I personally have no idea what occurred) and all of a sudden, that DNG option was taken off the table.
I do know that Hasselblad DID ad DNG to their digital cameras as an option–I introduced Thomas to Christian Poulson from Hasselblad and got them talking...however, Christian decided he wanted to lock the Hasselblad system, killed DNG in a firmware update and went on to sue Phase One and basically attempt to vender lock users into an all Hasselblad solution. How did that work for Hasselblad–and yes, if you sense a certain anti-Hasselblad bent from me, that's why–I was a Hasselblad shooter way before I was a Phase One shooter.
When you look at DNG, it would be wrong to presume it is some attempt by Adobe to leverage their position in the industry. In point of fact, it's really a situation where Adobe said "yes" to Thomas Knoll (yet again). And Bart, if you don't believe that, you don't know Thomas and Adobe–I do...what I'm telling you is the truth.
Give Adobe shit for all the mistakes they have made, but don't give them shit for the things they have done on our behalf.
-
I understand your method eventually accomplishes the same thing when you save as DNG after completing the raw editing. Is there a reason for this approach, in particular for keeping the raw copy as well as the DNG?
Well, I shoot with cameras and back that produce large raw files...I have a lot of 1Ds MIII, Phase One P65+ and IQ 180 files. For example, an IIQ raw file from the IQ 180 is about 79MBs (the back is an 80 MP capture). If I convert that to a non-lossey DNG the file size grows to about 96MBs on disk. But here's the kicker...if I modify the original raw file and save the edits to a .xmp file, that XMP file is about 8-9 KBs on disk. If I change the modifications to the DNG, the edits are stored in the DNG (which can be a good thing).
But, if you do your backups based on file modification dates (which is standard), if I modify a raw file, 8-9 KBs must be backed up. To back up the DNG, the whole 96 MBs file must be backed up. Multiply that file size difference by how many captures to shoot and you'll see that editing the raw files–not DNG files saves a lot of disk space.
Yes, when I'm "done" editing a raw file, I'll often convert that file to DNG (usually inside of Lightroom) so I can bake the edits & snapshots and metadata into the image. Prior to that, I edit the original raws...
That fact that I've modified my workflow for my own personal use does not in the least mitigate that current situation where camera makers are spawning off new, undocumented raw files for every new camera released (expect for the several camera companies that use DNG).
Note: one of the reasons I segregate my raw editing from edited files is that I often work in both Camera Raw and Lightroom. In the case of Lightroom, I'm not so worried about writting the .xmp metadata out to the file because the LR catalog is backed up separately. With ACR, that's not the case and any/all changes must be written to .xmp either in the side car (smaller) or the DNG (larger).
-
John, I have read your responses as well as the other posts in this thread. I have seen opinions on why dng could be a bad idea. It has been said that offering dng file format as an option could add to the price of a camera or to the cost of producing the camera. It has been said that offering dng file format as an option could.... I have not read anything in this thread or in any other source that factually states anything negative that offering dng format as an option will do. I have read lots of reasons that offering dng format as an option will make a positive change in the digital imaging world to the end user.
I am looking for facts. Does anyone have any facts to present that clarify why it is a bad idea to offer dng file format as an option for the user to choose, or not to choose at his own discretion?
-
John, I have read your responses as well as the other posts in this thread. I have seen opinions on why dng could be a bad idea. It has been said that offering dng file format as an option could add to the price of a camera or to the cost of producing the camera. It has been said that offering dng file format as an option could.... I have not read anything in this thread or in any other source that factually states anything negative that offering dng format as an option will do. I have read lots of reasons that offering dng format as an option will make a positive change in the digital imaging world to the end user.
I am looking for facts. Does anyone have any facts to present that clarify why it is a bad idea to offer dng file format as an option for the user to choose, or not to choose at his own discretion?
Bryan, all those coulds are based on experience of working with complex computer system's hardware/software/firmware...which is what today's cameras are. Translate could as would.
Canon and Nikon are not going to open their books to give you facts....and I am not going to relate facts about the company I worked for. Anyone who has worked in this or related industries, particularly major firms, would understand.
-
In terms of DNG, Thomas was getting pretty pissed off having to decode each and every camera's raw files. In developing Camera Raw, he came up with a process of reading the camera metadata and converting the metadata in order to apply normalized default settings. It was the process of taking the raw image data and metadata and normalizing the adjustments that led to developing DNG. Thomas reasoned that the basic raw image data and the image metadata should not be hidden from easy access.
.....
Give Adobe shit for all the mistakes they have made, but don't give them shit for the things they have done on our behalf.
Interesting, but a bit irreverent stories. Pretty much common to any view of inter-industry company interaction.
Adobe has been good for us. I like an use there software....even with the "rental" (as long as they don't make LR rental only). I, for one, do not relate the rental action to any benefits or DNG activty.
However, as you relate, one of the primary reasons for Thomas creating DNG was to avoid decoding all the multiple camera raw files. I understand the benefit to Adobe....to date, other benefits to users are minimal.
-
John, I have read your responses as well as the other posts in this thread. I have seen opinions on why dng could be a bad idea. It has been said that offering dng file format as an option could add to the price of a camera or to the cost of producing the camera. It has been said that offering dng file format as an option could.... I have not read anything in this thread or in any other source that factually states anything negative that offering dng format as an option will do. I have read lots of reasons that offering dng format as an option will make a positive change in the digital imaging world to the end user.
I am looking for facts. Does anyone have any facts to present that clarify why it is a bad idea to offer dng file format as an option for the user to choose, or not to choose at his own discretion?
Bryan, the real issue has nothing to do with DNG per se.
It is all about the volatility of your images.
Perhaps you don't care whether your RAW images are still accessible in ten, twenty, or fifty years time, or a century from now.
(Rhetorical observation only.)
But on an industry level this is a massive problem.
There are already plenty of orphaned RAW formats out there.
The software business itself is very volatile and one should not make any assumptions that one's favourite RAW converter will still be available in years to come to make sense of your legacy RAW formats.
The fact that, with some effort, it will be possible to reverse engineer the format, as is done now by Adobe and others, in the future does not mean that it will happen.
Also, software that might still convert early RAW formats may not run with current hardware/software configurations.
That situation will continue to get worse as computer hardware and OS software evolve.
This forum is replete with whizz kids and geeks (affectionate term) who can continue to run old software/hardware combinations and maintain them.
This might work on a hobby level but it isn't sustainable at an industry level.
Currently we wait up to six months for someone to reverse engineer the avalanche of new formats that are released every year so that one can do RAW conversions with one's converter of choice. This is not an issue if one chooses to work with the manufacturer's RAW converter, but for most in the industry this option appears not be the one of choice.
What is really required is a universal RAW format that is generated by all cameras that will make the process of RAW conversion trivial. Trivial in the sense that any RAW converter can process it without any reverse-engineering being required.
Why DNG is being punted currently is that it is the only candidate on the horizon.
DNG is in the public domain with its entire specification available as a SDK and has been offered to the ISO for standardisation.
The standardisation thing does not mean that the format cannot evolve just that the ISO will control how it changes.
All the hot air about how this will stifle camera sensor development in the future is just that - hot air.
The current DNG standard is heavily influenced by the TIFF-EP standard. Most (nearly (all)) current RAW formats are also essentially the TIFF-EP with proprietary metadata influencing how the manufacturer's RAW converter will handle the RAW file.
So, strangely a lot of proprietary RAW formats look a lot like DNG, just with proprietary metadata thrown in.
(This is the basis for Schewe's regular comments about secret RAW formats not really been secret anymore - they are just a pain in the ass to reverse-engineer.)
From an industry perspective photographers are only just coming to terms with digital asset management with most of the current emphasis being entirely focused (pun intended) on the survival of 0's and 1's. The problem is that it will be all for nothing if in a hundred years time there are half a million legacy RAW formats that need something to convert them with.
What do you think the chances are that your format(s) will still be supported?
From an individual perspective, as I began this post, it may not matter to you personally. After all in a hundred years YOU will be dead. So too will your images be dead.
But maybe, just maybe your images are worth something to somebody, perhaps only your family, but, maybe your images are worthy of being hung in the Smithsonian or the British Museum, and, maybe, that is not currently recognised right now.
Even if the 0's and 1's survive (still unlikely at this stage) is any software going to be available to convert your assiduously conserved digital images.
From an industry perspective the sad thing is that slides from twenty years ago stand a much better chance of surviving a hundred years than current digital image files.
Are the current issues solvable? Yes, I believe so.
Does DNG absolutely have to be a part of the solution? No!
But currently there is no universal RAW format and the bottom line is that we do need a universal RAW for archival purposes.
A universal RAW format, by itself is not enough but it is an essential part of the solution.
If someone can come up with a better universal RAW format I am all for it - bring it on.
Until then, if we have any sense, lets not throw the baby out with the bathwater.
Tony Jay
-
If someone can come up with a better universal RAW format I am all for it - bring it on.
Until then, if we have any sense, lets not throw the baby out with the bathwater.
Well, on the Planet Earth, the number of people who "could" possibly come up with an alternative to DNG could be counted on one hand and the initials TK head the lists...Other than Dave Coffin (dcraw (http://en.wikipedia.org/wiki/Dcraw)) and maybe a couple of others, I can't think of another individual or group that could beat out the guys above...DNG has been offered to the ISO, let's see where it ends up.
-
Tony, thanks for your well thought out response. You and I are on the same side of the fence. I understand everything that you said and I have agreed with everything for the past few years. All of my images for 2012 and most of 2013 are all dng files. The fact that all raw processors do not read dng files is the only reason why some of 2013's images are not dng.
I see the benefits of dng format in digital photography. I agree that it is not so much about dng, or it should not be about dng and/or Adobe. I can not think of any logical (to me) reasons that offering dng would be a bad idea.
I have not heard anyone give any reasons that show that making dng an option is definitely a bad idea. No one has even shown evidence that making dng an option probably would be a bad idea. I am not interested in hearing or reading about someones predictions that they apparently got from their magic 8 ball. I want facts.
-
...You and I are on the same side of the fence...
I did actually figure Bryan, nonetheless I thought it important to lay out a few key facts and put the whole issue in context for all.
Sadly this road was travelled earlier last year and the result was polarisation of positions.
But you never know...
Tony Jay
-
Hi,
It is still nothing that hinders proprietary tags to be include in DNG files. The vendor can add any data in a DNG as well as in a proprietary file format. Any program that can read a file can ignore tags it doesn't recognize.
Regarding spectral response data, it is normally handled by colour transformation matrices as far as I understand. DNG offers two one for 2800K and one for 6500K and a recommended way to interpolate between the two. As far as I believe to know, Adobe actually measures the spectral response of the sensors they develop DNG profiles for. (It is not terribly hard, you need a monochromator for that).
As a side note, I made a limited test of colour accuracy on my P45+ and Alpha 99 using both Capture One and Adobe Lightroom. Lightroom consequently outperformed C1, even if I would say that the winner was the Alpha 99 by a wide margin. That is of course not the whole truth. We can see millions of colours and the ColorChecker has just 16 colour patches, that happen to be industry standard. But correctly reproducing the 16 ColorChecker colours is not a bad start.
In honesty, I started looking at colour calibration as I hated colours coming out of the P45+ with Adobe standard profile, so just matching ColorChecker decently is not enough.
My results are here: http://echophoto.dnsalias.net/ekr/index.php/photoarticles/79-p45-colour-rendition
As a side comment, as a long time Minolta/Sony user I am accustomed to "Sony color", I also feel that Sony has a AWB that is biased towards a bit cold (bluish rendition), when I shoot a grey card I almost always prefer the auto white balance by Sony compared to grey card exposure.
Best regards
Erik
Again, it's really no different in that respect from all Raws that follow the TIFF specifications. Common info is known and shared, e.g. where the top left sensel within the image data block is located, and what pattern the Bayer CFA has and thus what color that top left pixel represents. However, it does not say exactly what spectral transmission that filter color had. So it may be a very pure Red or one with a trace of Green transmission as well to increase its transparency and make the camera more sensitive to light.
That info, required for very good color conversions may be tied to a patent for a very special Red dye that that camera maker can produce for a fraction of the cost of its competitors, but it bleaches faster than the Red from competitors. They do not want to tip of their competitors, because they might want to also change to a cheaper Red dye or switch source, which would reduce the competitive advantage. Only the original manufacturer will know that they will have to modify the Raw conversion for Red a bit faster as time goes by, to compensate for the rate of bleaching, by e.g. using the number of exposures that's being recorded in the maker notes, combined with the average exposure level that's encoded in common tags in the EXIF section.
Now when Madmanchan does a very good job, he may be able to quickly find an assumption for the Red characteristics to use during conversion for new cameras that's pretty close to what is optimal for the Adobe conversion engine, and within a month or two the Camera will be supported, with the DNG converter than can be used during the first ingestation/conversion into Lightroom. But he won't know of the fading characteristics of that dye because Adobe wouldn't pay royalties for that invention. So that same file will convert the same over time, but later Raw Captures will deviate in color.
Now this is just a silly example, but it might also be something like unique image verification data that's tagged to the file for use in law enforcement situations, or a piece of code that uniquely identifies the retina of the person who looked at the EVF to make that kiddy porn, or a feature that unlocks 3D image data that was encoded by splitting sensels into a left image and right image, which also allows to adjust depth of field after taking the image based on depth clues.
Intellectual Property is big business, and companies will protect that. They will even make life a bit less convenient for people that insist on using a certain brand of converter because those people feel that it suits their workflow better. Those people may perhaps not be able to change DOF after the fact, but just get the left or right image, just like they took it.
Cheers,
Bart
-
What is really required is a universal RAW format that is generated by all cameras that will make the process of RAW conversion trivial.
Hi Tony,
I'm wondering how that could be possible when the hardware that generates the image data differs? Those hardware differences will need to be converted into human viewable images, and the Raw images will always be different because the input/hardware/firmware is different.
Again, we're not talking about the image data that the ADC produces and gets recorded in the image data section of Raw files. That data is tagged in all formats, including DNG, and all converters read out the same numbers (one hopes). It's what happens with that Raw data that differs based on how intimately one knows what is additionally recorded in the proprietary Raw Maker notes, and how to interpret that image data.
That proprietary data will not be surrendered by the IP holders, because it represents incurred cost and future value. It can also make a competitive difference, even if the current differences between final results may seem small from the casual observer's position (and clever software can reduce those differences even after the fact). That's why there cannot be some universally agreed encoding of future features, because that would hurt competitive advantages. Competition is what fuels innovation and hopefully reduces end-user cost.
There are still many innovations ahead in sensor technology, so timing the introduction to market of those innovations is also critical for companies. Not spilling the beans too soon can also mean a lot of additional sales of more mature technology, but early warnings will halt most sales with the current sales models and alert competition.
Cheers,
Bart
-
Hi,
I have no issue with proprietary information. What I want is:
- A well publicly documented standard container format, universally supported
- A set of well defined essential tags, including white balance, colour conversion matrices and other essential data
- Bad pixel mapiing, bad pixel columns, dark frame data
- Proprietary data needed for proprietary processing is fine with me, I can use a proprietary processor if I need proprietary functions.
I am a bit concerned about having dozens to hundreds of different data formats. I don't know how many of those will be readable in say 10 or twenty years.
I would also say that standards are good things, and I don't think standards limit evolution. Many standards are extensible.
Best regards
Erik Kaffehr
Hi Tony,
That proprietary data will not be surrendered by the IP holders, because it represents incurred cost and future value. It can also make a competitive difference, even if the current differences between final results may seem small from the casual observer's position (and clever software can reduce those differences even after the fact). That's why there cannot be some universally agreed encoding of future features, because that would hurt competitive advantages. Competition is what fuels innovation and hopefully reduces end-user cost.
There are still many innovations ahead in sensor technology, so timing the introduction to market of those innovations is also critical for companies. Not spilling the beans too soon can also mean a lot of additional sales of more mature technology, but early warnings will halt most sales with the current sales models and alert competition.
Cheers,
Bart
-
Hi,
It is still nothing that hinders proprietary tags to be include in DNG files. The vendor can add any data in a DNG as well as in a proprietary file format. Any program that can read a file can ignore tags it doesn't recognize.
Correct, and that's why there is no benefit for Camera makers to drop their proprietary Raws. It would add the burden to make sure that all tags that are not used by their cameras (even their own legacy ones) are set to values that might not inadvertently screw up the conversion of their Raw data in whatever application is used. They have no control over the negative side effect that others may create in their Raw converter, based on some interpretation that should have no use for the camera.
When a Raw converter produces substandard conversions, then that's up to the converter manufacturer to explain or fix. When a Camera maker screws up a parameter that has no use for them, the blame will fall on them. Therefore it will add effort/cost to avoid such mistakes.
Cheers,
Bart
-
I understand the benefit to Adobe....to date, other benefits to users are minimal.
Minimal FOR YOU jrsforums (whoever that may be). For you. There are others here who find the benefits far, far more than miniumal. But you obviously don't care about anyone but yourself based on your writings. You prefer to speak for everyone too (kind of dumb). You can't muster up any reason why those of us that desire a DNG should be kept from using that format. And you're the inventor of very deep rabbit holes based on pretty shallow thinking. Please don't use DNG. Please let the rest of us decide what's a useful or non useful workflow (based on facts).
Please make an attempt to separate your loathing of Adobe and think about your fellow photographers who intend to use Adobe solutions even if you have scrubbed your computers of all their products and file formats out of (what protest?).
-
Quote from: BartvanderWolf on January 04, 2014, 06:50:59 AM
Hi Jeff,
You make it sound as if they are doing this pro-bono, in their spare time. Fortunately that's not the case, they get paid by Adobe, because it helps the company's bottom line ... (they would not likely get paid to waste company resources). And since Adobe is not in this game for charity, they must believe it will benefit their investors to do the effort.
In point of fact, both Camera Raw and DNG started as unpaid sandbox projects that Thomas wanted to do
Well that effectively kills that silly anti DNG argument! The format started pro-bono (which isn't salient in terms of issues with DNG), but now Thomas and every other raw converter engineer outside the camera makers DO get paid to hack these formats so that's a waste of money and engineering resource. Either way you slice it, bringing up the economies here shows the silly nature of a very time limited proprietary raw format.
Cost or no cost to build a DNG from a new proprietary raw, the need is simply unnecessary and has a negative affect on a group of people. All this shows the need for an open raw format, not the opposite. Doesn't matter of there are two highly qualified people who could hack any proprietary raw format, I can't and few of you can either. Some of the so called arguments in favor of proprietary raw formats are based on some of the silliest ideas I've heard outside the US political arena in a very long time!
-
Well that effectively kills that silly anti DNG argument!
Hi Andrew,
Apple also started from a garage as a hobby project to build some circuit boards for fellow hobbyists. That's as irrelevant to this discussion as the suggestion that Thomas is more charitable than mother Theresa (which she wasn't really) is phony. Thomas is an enhousiast with great skills that were used for the benefit of Adobe. Nothing wrong with that. But it has nothing to do with the discussion, it's totally irrelevant.
Besides, for that DNG conversion the proprietary Raw still needs to be reverse engineered, because hardware changes. DNG changes nothing about that, except that the process is a bit more efficient for Thomas/Adobe. Nothing wrong with that either.
Cost or no cost to build a DNG from a new proprietary raw, the need is simply unnecessary and has a negative affect on a group of people.
No, YOU find it unnecessary (despite having access to DNGs), and would prefer others to pick up the cost (time/expense/competitive disadvantage). Adobe, understandably from their perspective, also wouldn't mind if others took the burden on their necks. You and they are entitled to that opinion, but you at the same time find it difficult to accept that others (those camera makers) at the receiving end disagree. Seems a bit shortsighted.
I can also think of a few things that would fit my personal preferences better than the current situation, but I tend to dream with my eyes closed. During the day I try to be realistic, which is not the same as being in favor of proprietary formats.
Cheers,
Bart
-
Apple also started from a garage as a hobby project to build some circuit boards for fellow hobbyists. That's as irrelevant to this discussion as the suggestion that Thomas is more charitable than mother Theresa (which she wasn't really) is phony. Thomas is an enhousiast with great skills that were used for the benefit of Adobe. Nothing wrong with that. But it has nothing to do with the discussion, it's totally irrelevant.
Come on, really? First, no one stated Thomas was more or less charitable than anyone else. Thomas did go about trying to fix an issue. Whether he gets paid or not, there's a cost to this problem we're discussing, someone, somewhere pays, it's a problem that has zero reason to exist! That's the bottom line.
Besides, for that DNG conversion the proprietary Raw still needs to be reverse engineered, because hardware changes. DNG changes nothing about that, except that the process is a bit more efficient for Thomas/Adobe. Nothing wrong with that either.
Which is exactly the justification for the camera makers providing this for us like they do the JPEG. Or are you suggesting that when a new camera is produced, there is zero cost to the manufacture’s to provide that JPEG but there always is with raw? If so, evidence please (along with the cost which I'd be happy to pay for and likely am somewhere).
No, YOU find it unnecessary (despite having access to DNGs), and would prefer others to pick up the cost (time/expense/competitive disadvantage).
I'm happy to pay and already do. So do you or any other user who's NOT using the camera manufacturer's raw converter.
-
Minimal FOR YOU jrsforums (whoever that may be). For you. There are others here who find the benefits far, far more than miniumal. But you obviously don't care about anyone but yourself based on your writings. You prefer to speak for everyone too (kind of dumb). You can't muster up any reason why those of us that desire a DNG should be kept from using that format. And you're the inventor of very deep rabbit holes based on pretty shallow thinking. Please don't use DNG. Please let the rest of us decide what's a useful or non useful workflow (based on facts).
Please make an attempt to separate your loathing of Adobe and think about your fellow photographers who intend to use Adobe solutions even if you have scrubbed your computers of all their products and file formats out of (what protest?).
As usual you and your friends turn to vicious personal attacks....
You can use DNG today...or not...your choice. You just can't get it directly from most camera manufacturers, but you do not want to listen to any reasons why it may not be of advantage to them.
"loathing of Adobe"....what "hole" did you pull that from...??? Just another way you dismiss logical discussion into personal attacks.
-
But you obviously don't care about anyone but yourself based on your writings.
Hi Andrew, I admire your passion in favor of DNG. Good for you that you feel strongly about it and that you want to encourage others too. But to be honest I find the moral side of your advocacy wholly unconvincing. In saying that I don't mean to undermine your argument -- on the contrary, I encourage you to strengthen the moral aspects of your advocacy. Consider Dave Coffin's stance, for example: he states he does his work not for financial gain, but to win the respect of his peers. He undoubtedly has the track record to back up his stance. He makes his valuable work and genuine expertise on decoding RAW formats available freely, in every sense of the word free. He clearly does care about others. I assume you admire his track record for more than just his technical expertise.
However based on what you wrote earlier, at the same time you're apparently happy to lock up in a proprietary format the metadata on how we process our RAW files. From my perspective this is a contradiction. I'm glad that this metadata is part of the XMP spec (and therefore DNG spec) and that an increasing number of tools we rely on to do our work utilize it. We need much more of this, not less, for both pragmatic and moral reasons.
And finally absolutely nothing I read in jrsforums' arguments suggested to me that he or she is any less moral than any of the rest of us! I feel characterizing him or her otherwise is unfair.
-
He clearly does care about others. I assume you admire his track record for more than just his technical expertise.
Absolutely!
However based on what you wrote earlier, at the same time you're apparently happy to lock up in a proprietary format the metadata on how we process our RAW files. From my perspective this is a contradiction. I'm glad that this metadata is part of the XMP spec (and therefore DNG spec) and that an increasing number of tools we rely on to do our work utilize it. We need much more of this, not less, for both pragmatic and moral reasons.
I'm not happy locking up anything that I could use. However I understand there's all kinds of functionality that is and should be proprietary. I gave several examples including Photoshop Layers (timely when some Adobe customers are thinking of using other image processing soltuions and some assuming all that data will transfer).
Is there some proprietary data, data that can only be used by the camera manufacture's converters which becomes non proprietary months after the new format ships due to the labors of other's (Dave included)? Does that data change by virtue of it being understood? It is still not used, but deciphered such every other converter can now access the meat and potatoes (the raw sensor data) such they can render the image? I believe not but I'm open to learning more. If this temporary proprietary data, data that I'll never use is only proprietary for a few months, causing others an inability to access all the data as they please, what good is it being proprietary in the first place?
-
I'm not happy locking up anything that I could use. However I understand there's all kinds of functionality that is and should be proprietary.
So you're against proprietary file formats except when you're in favor of them? More specifically, you're against them when you see that they detract from your workflow, but in favor of them when you perceive they are not. It seems you don't pay much attention to the workflow of others when it comes to these aspects of post-processing. Some people do want that data to be non-proprietary, for all kinds of reasons. It may not affect you directly but it surely affects them. I don't mention this to "score points" or something similarly juvenile, but only to say that reality is not as black and white as seems.
Is there some proprietary data, data that can only be used by the camera manufacture's converters which becomes non proprietary months after the new format ships due to the labors of other's (Dave included)?
We are using the same terminology to describe different phenomena here. When Dave Coffin figures out how camera manufacturers have encoded data in a file format he has decoded it. The file format remains proprietary. For example, MS Word .doc files are in proprietary format, even though the format has been reverse engineered by programmers who never worked for Microsoft. By contrast, MS Word .docx have never been proprietary, are fully documented, and are an ISO standard.
-
So you're against proprietary file formats except when you're in favor of them?
I'm against temporary unnecessary proprietary data that doesn't need to be proprietary for any reason other than to limit my use of my data. It is as simple as that.
When Dave Coffin figures out how camera manufacturers have encoded data in a file format he has decoded it.
How long after this non necessary proprietary data is this deciphered, how does that assist me and others who have to wait on Dave and others? Again, for what purpose is this temporary proprietary data useful other than to keep Dave and other's with busy work and me and other's from accessing our data?
The file format remains proprietary. For example, MS Word .doc files are in proprietary format, even though the format has been reverse engineered by programmers who never worked for Microsoft. By contrast, MS Word .docx have never been proprietary, are fully documented, and are an ISO standard.
Correct, agreed. The difference is, an MS user who wishes to use the useful and proprietary MS word doc should understand this, the proprietary MS doc doesn't become non proprietary for use outside MS word months later (again begging the question, why make it proprietary in the first place)? No more than a Photoshop user should expect that his proprietary layers or Smart Objects will transfer exactly outside that app. The difference is there are two formats the camera provide and one is open and accessible the day the unit ships, the other isn't but will be and once that happens, what darn good was it that the data was proprietary or that many of us had to wait to use it like the JPEG? Is that such a difficult question?
It seems you don't pay much attention to the workflow of others when it comes to these aspects of post-processing.
How's that? My suggestion is a choice in the raw data provided. It in no way affects those that don't want the DNG. It does affect those that do. Seems like ample choice for workflows, not the opposite so what I'm I missing in asking for options?
-
Hi guys.
Bit of a ding dong going on here.
Heres a few thought from sunny Scotland.
This goes out to everyone following this thread.
As a matter of interest, whom amongst us use the camera manufacturers own Raw software of if indeed, have ever done so?
If yes which is better colour rendering, Propriety of 3rd party?
I bought my new nikon d7100 and as Andrew says could not use it in LR or ACR or even in my version of DCraw. I had to use Nikon View NX2 or my jpegs.
But buy using View NX2 and Photoshop (sometimes luminance hdr) there was really no need to use LR or ACR or any other 3rd party converter, well at least for my type of photography any way. I doubt that would go for every one.
But the point is a you can use your RAW files 'straight out of the box' BUT just not with any other software but NIKON'S for example and that is frustrating if you need LR workflow, understandable.
BUT as NIKON charge for their upgrades to Capture NX2 they will probably want photographers to go down this route as opposed to say ADOBE. Its all business. Keeping you in their 'loop'.
So why would NIKON use an open format? It wouldn't help their sales of Capture NX2 or would it? And then one must ask why are NIKON developing colour software?
The other point is re the DNG spec itself. I think herein lies the main problem or sticking block.
Ill focus (pun alert) in on two main areas that (amongst others) would bother me if I were a camera manufacturer. (I'm not by the way, i'm just a colour scientist)
The spec asks for the colour matrix and white balance information.
Spectral characterisation of a digital camera to produce a high quality matrix is not trivial (as was stated somewhere in a previous post) and refining the process takes time and money and knowhow. Why should I then give it away?
Same for white point estimation. A very complex subject, not least it is under-constrained, and years of research have gone into this problem. If then, I have developed a new white balance algorithm, again why should I give it away?
Without these bits the DNG spec fall flat on its face really and we are back to Thomas and Madmanchan and their camera profiling. Or are we.
So just a final thought or two. Maybe if ADOBE redesigned ARC /LR (and to all raw developers) so that I could 'slot in' my matrixes from my profiling method instead, might that not be the way to go. Then that would open up the door for 3rd party spectral profiling services etc. That might take the pressure off of ABODE.
ADOBE will just have to buy out NIKON, CANON .........
I love what ADOBE do, great innovators, but im afraid the DNG idea is a dead duck for now.
Iain
-
Read this thread with interest and was undecided until now if I should join in ;D. So FWIW my take on the subject:
I have played with DNG conversions but have decided for now to stick with Nef and process via LR or ACR (Do not like Nikons own now outdated NX2).
I do however think that the number of raw proprietary formats is crazy – not sure how many there are but Adobe must have support for at least 400 cameras?
To have a standard to comply to seems to me a good idea and if that standard can also incorporate manufacturers proprietary information (that is probably encrypted anyway) then we can have some compatibility across multiple platforms and still be able to access the proprietary stuff in the manufacturers own applications if required. This does seem to be what DNG offers and I think we should thank Adobe or probably more particularly Thomas Knoll and others that worked on this and provided a free of charge converter and the fact that this is open source therefore effectively in the public domain.
I am sure that there must be many other non Adobe applications that support DNG and also sure that numbers must grow once more manufacturers opt in to the format either by their own choice or in response to public demand.
One problem is that manufacturers do feel that their own proprietary format gives them the edge over their competitors and just maybe complying to a standard they perceive would lose them that edge– I speak from firsthand experience working for such a manufacturer!
The main problem is, I think that until there is this huge public demand we will be stuck in the new camera yet another new proprietary solution without a choice – other than to look elsewhere. I doubt that Joe public is particularly interested in anything other than getting a nice pic and suggest that the volume of users wanting adoption of a non proprietary standard is relatively low even for professional grade cameras.
So how can the manufacturers be lobbied effectively to adopt new and foreign standards?
I draw some parallels here to my own experience in digital imaging in medicine. I was quite heavily involved in the first introductions of X ray digital imaging in the UK. At that time all manufacturers’ equipment had to use proprietary formats and needless to say individual components were very expensive. Typically a single clinician’s workstation cost around £45,000, comprising of a PC box running under Unix and a greyscale CRT monitor.
Once an image had been acquired it would be archived available for viewing and manipulation but only via the manufacturer’s equipment using the proprietary algorithms. Need to view the image elsewhere without the manufacturers workstation an proprietary processing bad luck for you even worse maybe for the patient!
The fact of so many proprietary formats for so many imaging modalities and lack of standards was recognised by ACR (American College of Radiologists) and NEMA (National Electrical Manufacturers Association) and a joint committee was formed to expand standards for digital imaging in medicine. The net result being the development of the DICOM standards in use today.
To sell or have any hope of selling equipment to any health provider required the manufacturer to conform to the DICOM standards and explicitly confirm such within their documentation.
Conformance to standards did not mean that the manufacturer lost anything as proprietary stuff could be included that only their systems could interpret and this was accommodated with the provision of private tags and fields. What it did mean is that clinical users worldwide could view images from any modality from any manufacturer without the need for specific manufacturer’s software.
Seems to me that this is pretty much what the DNG format is trying to accomplish and I do feel that it would benefit the industry as a whole.
The point though is ACR NEMA had the clout to push standardisation forward and I do not at this time see how a relatively small body of independent photographers can accomplish this – if there is a way point me to it and I will sign up.
Sorry for a rather long and rambling post but wanted to get it off my chest
-
Hi Andrew,
Can we please clarify something important before continuing our discussion? It's important that you understand that a proprietary format like CR2 remains proprietary regardless of whether Dave Coffin or Thomas Knoll decodes it for use in a raw convertor. When and only when the data in the CR2 file is converted to an open format like DNG or jpeg or TIFF only then is it no longer proprietary (allowing for the proviso that DNG allows some proprietary metadata and even embedding the original raw file). Maybe you mean to convey this meaning too, but your present terminology is quite ambiguous!
Damon
-
Can we please clarify something important before continuing our discussion? It's important that you understand that a proprietary format like CR2 remains proprietary regardless of whether Dave Coffin or Thomas Knoll decodes it for use in a raw convertor. When and only when the data in the CR2 file is converted to an open format like DNG or jpeg or TIFF only then is it no longer proprietary (allowing for the proviso that DNG allows some proprietary metadata and even embedding the original raw file). Maybe you mean to convey this meaning too, but your present terminology is quite ambiguous!
Understood, agree 100% of the above, that's exactly what I expect. And yes, there is a proviso that DNG allows (some/all?) proprietary metadata and even embedding the original raw file if one desires either or both.
What Dave and Thomas do is allow me access to that data outside of the manufacturer's converter of course, and should I decude that perhpas some of that proprietary data could be useful and I might visit said manufacturer's converter, I'd make sure I had that data archived somewhere. If it's not in the DNG, I'm to blame for not archiving the camrea raw original. Or if I had the choice (which I don't), for setting the camera to give me a DNG instead of the proprietary raw (and convert, as I'm doing now, after import).
-
Hi Andrew,
Can we please clarify something important before continuing our discussion? It's important that you understand that a proprietary format like CR2 remains proprietary regardless of whether Dave Coffin or Thomas Knoll decodes it for use in a raw convertor.
Hi Damon,
I do get the distinct impression that some others do not grasp the notion yet, that the image data in a regular DNG and in a proprietary Raw file format are exactly the same set of numbers (ADUs or DNs). They are equally proprietary and abstract, they are just a set of numbers.
They are just Digital Numbers (DN) or Analog Digital Units (ADU), and the application that reads them will need to decide which number represents a Red filtered sensel, a Green filtered sensel, a Blue filtered sensel, or whatever filter color was used. The arrangement of the Bayer CFA filters is encoded in a different spot in the file.
Both the location of those filtered digital numbers, and the pattern arrangement, are stored at a different tag location. That tag location and the offset position into the file they hold may be different, but they are essentially pointing to the same abstract data. Only the container is different, and therefore the physical locations of those numbers in the file may be different.
To make photographic sense of those abstract numbers and to allow demosaicing, they are decoded by an application according to it's own recipe. Lightroom uses ACR for decoding, Capture One uses C1 decoding, DxO uses yet another decoding of those same numbers, as does Photo Ninja. The set of data numbers they extract for processing is always the same, regardless of the file container, whether it's DNG or Camera Raw.
Several other pieces of information are used to improve the interpretation of the Digital Numbers, the more intimate the knowledge of the meaning of those other pieces of information, the better the demosaicing result can be (although the quality of the demosaicing can also make a difference). Even something as trivial as what's Up and what's Down and Left or Right needs to be defined, and it is, regardless of the container.
When and only when the data in the CR2 file is converted to an open format like DNG or jpeg or TIFF only then is it no longer proprietary (allowing for the proviso that DNG allows some proprietary metadata and even embedding the original raw file).
But the conversion to DNG doesn't really convert to an image (it just rearranges the same data into an openly documented order), while a TIFF or a JPEG do contain visible image data (encoded in RGB(A), or CMYK, or Lab, or YCrCb, coordinates). The data in a Raw DNG format is as colorless as any Camera Raw, it yet needs to be demosaiced into an RGB image.
Because DNG is just an openly documented container, it can instead also store (additional) other types of data than Raw uncooked Bayer CFA values. It can e.g. store already demosaiced image data in a proprietary Adobe format, and processing parameters as used by Adobe software, but that is just processed/cooked (no longer Raw) data with proprietary Adobe application tags added, which is useful if one uses that particular image processing application, but may be useless for other applications.
This additional cooked data is not what we are talking about when Raw data in a DNG versus Raw data in a Camera Raw format is concerned. Those two are identical sets of abstract numbers, rearranged (=converted) in a different order, that still need to be interpreted and processed before they mean something useful that resembles a visual image instead of mere data numbers.
Cheers,
Bart
-
I do however think that the number of raw proprietary formats is crazy – not sure how many there are but Adobe must have support for at least 400 cameras?
typical FUD is to claim that there are 400 different formats - no, there are much less... bother yourself to compare 2 sequential releases of dcraw code to see the differences introduced... true that once in a while there is a major feature being introduced and that is when more work is done to support that... having said that the issue w/ in camera raw is not whether it is DNG or non DNG - it is a matter whether manufacturers have reasons to document or not to document the content of the raw files... the real question is not why they are not using DNG, but why they are not providing the description of their own formats.
-
and who we leave the original question - that is about archiving (save the originals or not)... and not about using DNG in workflow (which is a non question, for as long as it does not involve non archiving of originals) or merits for manufacturers to document or not the format/content of their raw files (which they have... not to)
-
Hi
The DNG falls flat on its face as it requires the camera matrix in order to function properly as an open standard.
Without the colour matrix as supplied by say NIKON etc we are back to the situation we are in at the moment, which is WAIT for someone to profile the camera.
NIKON etc wont give up their camera matrix, why should they, it cost them lots of investment in time resources skills to produce a camera matrix.
Like I said before WHY should they give it away for free?
This is why DNG doesnt work.
What you might get at best from NIKON etc is an non-encrypted RAW file to an ISO standard perhaps, but again we would require someone to profile the camera.
Iain
-
typical FUD is to claim that there are 400 different formats - no, there are much less...
Tony wrote about 400 cameras that were supported not 400 formats, you popped that in there. I agree with you, 400 isn't correct (it's hardly FUD IMHO), Adobe states more than 350:
The Adobe DNG Converter is a free utility that enables you to easily convert camera-specific raw files from more than 350 cameras to the more universal DNG raw format.
bother yourself to compare 2 sequential releases of dcraw code to see the differences introduced
So there are differences. And those differences do or do not allow other raw converters to understand that difference the day that new camera is released or not?
It doesn't matter if the difference is one byte. If it's different such it can't be read until that byte is updated/understood in all the other converters, your points are not helpful in solving the problem (access of the data).
it is a matter whether manufacturers have reasons to document or not to document the content of the raw files...
Fair enough. Let's stick with big two, Canon and Nikon. Do they document prior to release? Seems not as we wait weeks and months for updates to our preferred raw converter.
the real question is not why they are not using DNG, but why they are not providing the description of their own formats.
Again, fair enough! So now the question becomes, is the industry pressure better served by demanding they release this info OR to give us that DNG switch on the camera? I'm fine with either but would prefer DNG onto my PC card.
-
The DNG falls flat on its face as it requires the camera matrix in order to function properly as an open standard.
Without the colour matrix as supplied by say NIKON etc we are back to the situation we are in at the moment, which is WAIT for someone to profile the camera.
NIKON etc wont give up their camera matrix, why should they, it cost them lots of investment in time resources skills to produce a camera matrix.
What stops them from encrypting this into their DNG (we're talking about native DNG from said camera)? Or just using that data not shared to internally provide a DNG?
-
and who we leave the original question - that is about archiving (save the originals or not)... and not about using DNG in workflow (which is a non question, for as long as it does not involve non archiving of originals) or merits for manufacturers to document or not the format/content of their raw files (which they have... not to)
Absolutely agree.
For archiving it makes no significant difference, conversion to DNG only rearranges the same Camera Raw data to openly documented tagged positions. No real benefits, just a slowdown before industry wide agreement on new tags is reached.
As a workflow format, there can be some benefits for image editing software developers (mostly Adobe, given its dominant position). No benefits for camera makers to be achieved, only an increased workload burden and competitive risks.
Cheers,
Bart
-
Hi Andrew.
Im not quite understanding what you mean by "What stops them from encrypting this into their DNG".
Well if they did anyone could use their own RAW Processor to develop the files (which ideally would be great). Who would need NIKON software, they would be cutting off a revenue stream for one. But I think what stops them is for the reason I outlined previous post:-
Why should NIKON give away the matrix which has cost them to produce?
Would you if you invested a lot into camera characterisation. Again this is not a trivial process.
Iain
-
Hi Andrew.
Im not quite understanding what you mean by "What stops them from encrypting this into their DNG".
There's a place in the DNG for them to put this data no? IF I understand you correctly, there's some necessary data. It's proprietary. Certainly on day one of the camera release.
At some point in the future, Adobe and others are able to decifer what they need for us to render that data right? So what I don't understand is if this data you refer to is always proprietary (it's some secret sauce) can't it be placed into a private tag within the DNG? If the secret sauce is at some point understood (such other converteres have access), what's the point of it being proprietary in the first place? IOW, DNG as a container allows this secret sauce to remain secret.
Who would need NIKON software, they would be cutting off a revenue stream for one.
It's needed to be sure on day one! That's the problem. It isn't needed on day 30+ once everyone who needs to support that format in their product gets their hands on the original raw. So I don't see how this is an issue in terms of provding a camera DNG as an option. What am I missing? I really DO want to know (and FWIW huge thanks to Bart for his last two post. Made all this a worthwhile investment in time).
Why should NIKON give away the matrix which has cost them to produce?
If that data is going to be understood anyway, they should give it away because it will presumably aid some users who wish to use the new format the day the camera ships in a product they wish to use (like the JPEG). The proposal (one proposal) is to let Nikon build the DNG so it may be possible to do so without providing anything Nikon feels is secret sauce. If someone is going to eventually undrestand this data anyway, what's the point of doing such a poor job of keeping that data proprietary?
-
Hi
Ah I see where you are going wrong. The matrix is not included in the NEF file at present.
It never was there at any time.
Its in their software.
To render a RAW file after encryption we need to produce a matrix ie Thomas et al.
Cant see why NIKON would want to just give it away. It probably cost them lots to produce.
-
Ah I see where you are going wrong. The matrix is not included in the NEF file at present.
It never was there at any time.
Its in their software.
OK understood now. It really truly is proprietary and only useful/accessible in their software. If so, it's not an issue for us DNG users, we don't expect to use that product.
-
Yes, thats right, but without the matrix in the DNG, the DNG cant be rendered so is no good unless a matrix is supplied, as is at the moment , by ADOBE on a 30day lead time.
DNG/ACR/LR and any other RAW converter (which use DCRAW and Thomas's matrixes) is fine if you want your colour matrix supplied by ADOBE.
If you want to use NIKON colour matrix use their software.
I would suggest that ADOBE update their products to allow other profilers to insert their colour matrixes for a particular camera. Now that would be beneficial to the industry.
iain
-
Yes, thats right, but without the matrix in the DNG, the DNG cant be rendered so is no good unless a matrix is supplied, as is at the moment , by ADOBE on a 30day lead time.
Could that data be stored in a camera generated DNG? Or maybe it's moot if we're using someone else’s converter...
DNG/ACR/LR and any other RAW converter (which use DCRAW and Thomas's matrixes) is fine if you want your colour matrix supplied by ADOBE.
Not an issue (kind of expected). And not just Adobe (as an example of another product that supports DNG, Iridient Developer)
If you want to use NIKON colour matrix use their software.
Got it. If this provides significantly superior images, then let the photographer decide and use a solution they prefer. I'm all for choices!
-
Tony wrote about 400 cameras that were supported not 400 formats, you popped that in there. I agree with you, 400 isn't correct (it's hardly FUD IMHO), Adobe states more than 350...
Thanks Andrew, camera support is what I was referring to and I had read some time ago about number of cameras supported and now remember the figure as 350+ .
What is not clear to me is the format of the container that is DNG and what limitations would be imposed, and without looking at any documentation my guess is that it is constructed in such a way that manufacturers secret squirrel stuff and secret source algorithms do not have to be revealed to any other than the manufacturers own software.
Put simply the actual processing algorithms that allow a camera manufacturer to give a Super High Inverse Tone or a Vivid Burn You Eyes Out setting or in medicine a view of both soft tissue and bone detail is proprietary and guarded by the manufacturer and only interpreted exactly by their own software.
The rendering of the raw file to get an acceptable first image on our screens will probably also be proprietary and hence the need for Thomas and co to work on each camera to get an acceptable starting point image (Adobe Standard) which may be different to the manufacturers ‘look ‘. As a bonus for some cameras certain presets are created that get fairly close to the presets for ‘the look’.
The manufacturer does not need to actually give us their interpretation (‘the Nikon/Canon Look’) but a normalised image within the DNG container – this may be as simple as the JPEG representation we see on the camera LCS. If you shoot JPEG only (shame on you ;D) you can take this file into any photo editing program without issue and there must be hundreds to choose from (there I go exaggerating again!) so why should raw files be so much different?
As far as Nikon go my impression is that they are not really that interested in improving the software and in fact they do not produce this in house, at least Capture NX2. This is/was from Nik software which has now been absorbed into Google. AFAIK there have been no major changes to the software in a number of years probably going back to the 64 bit change. The only major changes I think have been updates to the software to offer support for their new cameras – and the odd bug fix. Similarly they have had to do this with the free Nikon View NX with each new camera release.
To any of the old school that actually remember that wonderful stuff called film and have some nostalgia for the non automatic everything cameras that did not use batteries a question.
Would you have purchased a particular camera if you had to use the manufacturers own film to get an acceptable result?
-
The manufacturer does not need to actually give us their interpretation (‘the Nikon/Canon Look’) but a normalised image within the DNG container –
They don't even have to do that. Read Bart's last two posts. The data we're talking about in DNG is raw. It's yet to be rendered.
DNG does allow minimal processing (Lossy DNG) and fully processed data. But that's a different story. All the manufacturer needs to do inside the camera is functionally equivalent of what we do when we convert to DNG in LR or with the standalone DNG converter.
-
They don't even have to do that. Read Bart's last two posts. The data we're talking about in DNG is raw. It's yet to be rendered.
DNG does allow minimal processing (Lossy DNG) and fully processed data. But that's a different story. All the manufacturer needs to do inside the camera is functionally equivalent of what we do when we convert to DNG in LR or with the standalone DNG converter.
Andrew thanks another brain fart for me - I did know that we are discussing raw DNG data. Bart, my apologies for not reading fully what you posted.
-
Hi Andrew.
Yes the matrix can be stored in the DNG there is a tag for it.
DNG is ok if you have a matrix. At the moment the matrix is added by ADOBE at the 'convert to DNG' stage.
Iridient Developer. Do they make their own camera matrix for each camera or use ADOBE'S. ADOBE'S I think (correct me if im wrong).
Again the BIG PROBLEM is getting the matrix from NIKON etc.
I think ADOBE jumped the gun in the DNG spec. I said this when it first came out. MATRIX tags! Thats asking a lot.
What they should have pushed for was a open un-encrypted, matrix free, tiff file. Matrix to be added later by 3rd parties at rendering stage. There would have probably been more of an uptake by manufacturers.
Now ADOBE have to provide colour matrix for all new cameras if they want people to use their DNG spec.
Do you see the problem.
-
Iridient Developer. Do they make their own camera matrix for each camera or use ADOBE'S. ADOBE'S I think (correct me if im wrong).
Both it appears. I know I can load my custom DNG camera profiles built for LR/ACR there and use them. Not sure if that helps any. You can use ICC camera profiles too if you wish.
Do you see the problem
Not really, not yet, sorry. Not that your suggestions about DNG having access to the color matrix shouldn’t be considered. We'd need Eric Chan or someone else to comment on that aspect.
-
I am really at a loss here....and this is a real question....not an attempt to poke at anyone.
If the camera manufacturers put the raw data, as Bart describes, not in their RAW container, but in a DNG container.....and all the proprietary info, wb, filters, matrix, etc. were still kept proprietary....what would be gained?
-
If the camera manufacturers put the raw data, as Bart describes, not in their RAW container, but in a DNG container.....and all the proprietary info, wb, filters, matrix, etc. were still kept proprietary....what would be gained?
The data spit out on day one would be accessible to any raw converter that understands DNG. That's been the issue/problem from post 1 here.
Again, think JPEG accessibility on day one and what a JPEG represents (yes it's rendered but more importantly, the way the container & data is described and defined is such it is accessible in lots of software products the day that camera ships). Further, no one has to continuously update their software. Imagine how Photoshop, Elements, Preview on Mac etc behaved if every JPEG was different such, it wasn't understood. Chaos!
-
I am really at a loss here....and this is a real question....not an attempt to poke at anyone.
If the camera manufacturers put the raw data, as Bart describes, not in their RAW container, but in a DNG container.....and all the proprietary info, wb, filters, matrix, etc. were still kept proprietary....what would be gained?
What would be gained is that the DNG file is not encrypted like a NEF and as such could be read AND RENDERED by any application that supports DNG.
Proprietary data would have to become open data, no encrptyion
The problem is NIKON wont give all the information needed ie the MATRIX, to be able to RENDER from DNG from the camera.
DNG would be an open format so any 3rd party software could open and render directly from a DNG produce by the camera.
At present the DNG can can only be used after the NEF has been converted by ADOBE DNG to a DNG and the missing MATRIX inserted.
It is then and only then, open format available to everyone who supports DNG.
iain
-
Hi,
The "MATRIX" is just 9 numbers, describing the mapping of RGB values to CIE XYZ coordinates. DNG specifies several matrices, one for 2850 K and another for 6500K and suggests how to interpolate in between. As I have mentioned before, measuring the spectral response of a sensor is not rocket science, check this: http://www.maxmax.com/spectral_response.htm
A monochromator is needed and some other tools, a monochromator is not necessarily very expensive: http://www.edmundoptics.com/testing-targets/spectrometers/manual-mini-chrom-monochromators/1379?#products
Best regards
Erik
What would be gained is that the DNG file is not encrypted like a NEF and as such could be read AND RENDERED by any application that supports DNG.
Proprietary data would have to become open data, no encrptyion
The problem is NIKON wont give all the information needed ie the MATRIX, to be able to RENDER from DNG from the camera.
DNG would be an open format so any 3rd party software could open and render directly from a DNG produce by the camera.
At present the DNG can can only be used after the NEF has been converted by ADOBE DNG to a DNG and the missing MATRIX inserted.
It is then and only then, open format available to everyone who supports DNG.
iain
-
Hi,
The "MATRIX" is just 9 numbers, describing the mapping of RGB values to CIE XYZ coordinates. DNG specifies several matrices, one for 2850 K and another for 6500K and suggests how to interpolate in between. As I have mentioned before, measuring the spectral response of a sensor is not rocket science, check this: http://www.maxmax.com/spectral_response.htm
A monochromator is needed and some other tools, a monochromator is not necessarily very expensive: http://www.edmundoptics.com/testing-targets/spectrometers/manual-mini-chrom-monochromators/1379?#products
Best regards
Erik
Hi Eric.
Its not rocket science its Colour Science! Boom Boom.
Please refer to a paper I published on Colour Appearance. It contains some extra information on producing a matrix. Not as straight forward as you may think. Response curves are one thing, producing a high quality matrix is another. But dont forget a matrix is only part of the characterisation data needed to produce CIEXYZ coordinates from device RGB.
Iain
-
Hi Iain,
Thanks for the paper! Very interesting! I admit it is not so straight forward as I may have thought.
Best regards
Erik
Hi Eric.
Its not rocket science its Colour Science! Boom Boom.
Please refer to a paper I published on Colour Appearance. It contains some extra information on producing a matrix. Not as straight forward as you may think. Response curves are one thing, producing a high quality matrix is another. But dont forget a matrix is only part of the characterisation data needed to produce CIEXYZ coordinates from device RGB.
Iain
-
Hi Eric.
Its not rocket science its Colour Science! Boom Boom.
Please refer to a paper I published on Colour Appearance. It contains some extra information on producing a matrix. Not as straight forward as you may think. Response curves are one thing, producing a high quality matrix is another. But dont forget a matrix is only part of the characterisation data needed to produce CIEXYZ coordinates from device RGB.
Iain
Iain, could you post sample color images that show errors created by not getting the color appearance matrix just right and tie it in with their respective associated mathematical formula and/or graphs?
Your PDF is quite a bit over my head but I do get the gist of what you're trying to pin down with regard to matrix driven WB and HSL adjusts in order to achieve predictability in processing data by characterizing a camera sensor's performance. The image sample of the woman at a picnic doesn't make this more understandable nor does it make a direct correlation to the formula's and calculation's influences on results a photographer can use.
Appreciate your hard work on this and posting the pdf here.
-
Iain, could you post sample color images that show errors created by not getting the color appearance matrix just right and tie it in with their respective associated mathematical formula and/or graphs?
Your PDF is quite a bit over my head but I do get the gist of what you're trying to pin down with regard to matrix driven WB and HSL adjusts in order to achieve predictability in processing data by characterizing a camera sensor's performance. The image sample of the woman at a picnic doesn't make this more understandable nor does it make a direct correlation to the formula's and calculation's influences on results a photographer can use.
Appreciate your hard work on this and posting the pdf here.
Hi
I don't think i have pictures produced with bad matrixes. I dont think i produced any real bad ones. Most were pretty spot on due the the spectral characterisation method employed was quite precise.
The pic of mary is to really do with the local adaptation part of the project and the use of CIECAM02
Again a matrix may be optimised for accurate blues and loose it in the reds or vice versa. it down to the manufactures preference.
iain
-
Hi,
I played a lot with DCP profiles when I started using my P45+ back as Adobe Standard rendered some colours bad. I have posted an article with my results so far, it's a bit messy and needs to be clarified/updated but may give some insight.
I used a ColorChecker Passport with Adobe DNG Editor and also ColoChecker Passport software. I also tried QPCard and it's software.
http://echophoto.dnsalias.net/ekr/index.php/photoarticles/79-p45-colour-rendition
The article begins with some results from using Imatest ColorCheck and ends with a lot of sample.
Best regards
Erik
Iain, could you post sample color images that show errors created by not getting the color appearance matrix just right and tie it in with their respective associated mathematical formula and/or graphs?
Your PDF is quite a bit over my head but I do get the gist of what you're trying to pin down with regard to matrix driven WB and HSL adjusts in order to achieve predictability in processing data by characterizing a camera sensor's performance. The image sample of the woman at a picnic doesn't make this more understandable nor does it make a direct correlation to the formula's and calculation's influences on results a photographer can use.
Appreciate your hard work on this and posting the pdf here.
-
Again the BIG PROBLEM is getting the matrix from NIKON etc.
people know that Nikon OEM converter will leave temp files available to grab with their ICC camera profiles during the work, so how that might be a big problem beats me... surely not documented, but it is available and DNG container can have ICC profiles in it... now the question is as to why ACR/LR will not use ICC camera profiles that are in DNG according to the spec ? what if camera manufacturer does not want to go DCP route ?
-
Thanks Andrew, camera support is what I was referring to and I had read some time ago about number of cameras supported and now remember the figure as 350+ .
come on... see how he is making FUD by equating the number of really different __FORMATS__ with a number of different camera models, which totally incorrect approach.
-
Absolutely agree.
For archiving it makes no significant difference, conversion to DNG only rearranges the same Camera Raw data to openly documented tagged positions. No real benefits, just a slowdown before industry wide agreement on new tags is reached.
it makes a difference... first of all Adobe along the way changed their approach as to what they transfer to DNG from the data read off sensor and written by camera's firmware in the original raw files (and so there were times when some Camera Raw data was discarded, yes)... that alone shall be enough once and for all to stop discarding original raw files... where I can make DNG file on demand (to use in my workflow), why shall I discard the original data... then why take a risk that there is a bug in Adobe software on top of everything else...
-
Tony wrote about 400 cameras that were supported not 400 formats, you popped that in there. I agree with you, 400 isn't correct (it's hardly FUD IMHO), Adobe states more than 350:So there are differences.
like camera model names, yes... a great difference indeed.
-
Seems not as we wait weeks and months for updates to our preferred raw converter.
that was supposed to be fixed with that new subscription model... oh wait... we are still getting 4 ACR releases a year and not JIT ;D ...
-
come on... see how he is making FUD by equating the number of really different __FORMATS__ with a number of different camera models, which totally incorrect approach.
Again, if the formats are not identical, they are different no? If all 350 cameras produced identical files, I'd be kind of shocked. As I said, any difference that results in file A being accessible outside the manufacturer's converter but file B not, it doesn't matter how many cameras or formats you wish to define, the file B can't be accessed until someone alters some code.
-
that was supposed to be fixed with that new subscription model... oh wait... we are still getting 4 ACR releases a year and not JIT ;D ...
Can you point out where Adobe said that thanks to a subscription model, they will more quickly provide camera support that I submit they shouldn't have to provide in the first place?
So Nikon or Canon comes out with a new camera today. Assuming (big assumption) Adobe and everyone else do NOT have to purchase the camera, all they need is some kind customer to send them a raw, even with a subscription model, support isn't going to be less than weeks! Software has to be updated. Tested. Documented. Installers have to be built. Uploaded. It's still work albet it, thanks to subscription, we might see the support a bit faster. That's not an excuse for the current conditions anyway. No argument you or anyone else has made so far lets the manufacturer off the hook. And the subscriptions model doesn't help our friends at PhaseOne, Dx0, and all the other raw converter software companies.
-
Hi
I don't think i have pictures produced with bad matrixes. I dont think i produced any real bad ones. Most were pretty spot on due the the spectral characterisation method employed was quite precise.
The pic of mary is to really do with the local adaptation part of the project and the use of CIECAM02
Again a matrix may be optimised for accurate blues and loose it in the reds or vice versa. it down to the manufactures preference.
iain
Guess I should've worded it differently, so I'll ask it this way... Could you induce an error in an image to show how and where in the calculations and formulas all this works?
It's a demo similar to assigning the wrong profile to an image encoded in another space that causes color appearance distortions according to human perception. Maybe show a scene with memory colors that go wacky on account of these distortions.
Or maybe post images you created using this specific knowledge you've acquired represented in your pdf that you define as correct looking color appearance versus incorrect.
-
come on... see how he is making FUD by equating the number of really different __FORMATS__ with a number of different camera models, which totally incorrect approach.
like camera model names, yes... a great difference indeed.
Come on get your facts correct before you start trying to sling mud or insults assuming that you are accusing me personally of spreading FUD.
There are differences between cameras raw files from the same manufacturer and therefore the formats are not the same and the data still remains proprietary.
The fact that very little may have changed between camera models may be true but it seems that often the case is that even seemingly identical cameras have to wait for support in ACR and LR.
A typical recent case being the Nikon D610 a virtually identical model to the D600 with minor changes to combat a dirt/oil sensor issue – all should have remained the same but...
Nikon D600 NEF minimum ACR version 7.3 minimum LR version 4.3
Nikon D610 NEF minimum ACR version 8.3 minimum LR version 5.3
In this case these changes may have been no more than adding the camera model number to the ACR/LR database but could have involved much more work than that – I do not know and unless you are employed by Adobe neither do you
-
Tim
Hi Here is a link to a talk i gave to local camera club.
Its a cut down version but illustrates the point.
Nikon left My matrix right. both sRGB
https://dl.dropboxusercontent.com/u/90361226/CIECAM02.pdf
Iain
-
Tim
Hi Here is a link to a talk i gave to local camera club.
Its a cut down version but illustrates the point.
Nikon left My matrix right. both sRGB
https://dl.dropboxusercontent.com/u/90361226/CIECAM02.pdf
Iain
Thanks, Iain.
Interesting how each image is made to look different with some looking like a warm, yellowish ivory white balance was applied while others just have their hue/sat tables tweaked for more richer blue skies and slightly more magenta saturated reds.
It looks similar to what can be done in ACR's HSL but without kicking up a lot of noise and a lot less work. I'm assuming your matrix profile produces less noise or at least keeps it the same, right? And do the profiles produce consistent results across a wide range of image dynamics and color gamut?
-
Hi Tim
I made a matrix for each image based on the estimated white point and scene luminance at time of capture.
The project was really about a new and better method of optimising a matrix. the resulting dE's of training data sets was enough to evaluate the method.
I produced about 20 to 30 images to illustrate the method. We never got as far as putting through psychophysical testing. That was the next stage when I get round to it.
Basically the effect I was looking for was that as luminance increases our perception of colourfulness increases. This was achieved not just by the matrix but by using CIECAM02 and a CIECAM02 optimised matrix.
Its still work in progress when I get time.
iain
-
What would be gained is that the DNG file is not encrypted like a NEF and as such could be read AND RENDERED by any application that supports DNG.
Proprietary data would have to become open data, no encrptyion
The first step should be describing the format of the raw data (sensel values) in such a way that they can be translated into 16-bit linear light numbers. I.e. revealing any encryption, data compression etc format.
Revealing the color plane subsampling and means of setting black/white points would be sufficient to render an off-colored output.
The problem is NIKON wont give all the information needed ie the MATRIX, to be able to RENDER from DNG from the camera.
DNG would be an open format so any 3rd party software could open and render directly from a DNG produce by the camera.
At present the DNG can can only be used after the NEF has been converted by ADOBE DNG to a DNG and the missing MATRIX inserted.
It is then and only then, open format available to everyone who supports DNG.
The second step might be embedding a camera description into the raw file. I.e. spectral response of the camera sensor. As this is relatively easy to reverse-engineer, and camera manufacturers surely have this info readily available, I cannot understand why they would make this hard for their customers. Describing the OLPF filtering (if any) etc might also be sensible.
Including information about _how the data should be rendered_ might be more difficult, as that might include development practices that the manufacturers guard closely?
-h
-
The three main problems with the proprietary formats are that they are 1) not publically documented and 2) incomplete (ie you need information about the camera that's not stored in the format itself), 3) there are many of them causing a substantial effort to implement support for them, making it harder for small software developers to get into the game.
When camera companies go out of business, like Kodak for example, you may end up with incomplete information so you cannot make full use of your raw files. As a medium format user I don't trust that the companies will stay around forever, in fact my own digital back has been discontinued and the format is no longer used by the company so it's just a matter of time before it disappears from software too (fortunately it's been reverse-engineered to a decent level in open source software). I only use cameras whose formats have been reverse-engineered to a satisfying level, ie good results can be had in open-source software.
The problem with DNG from a camera manufacturer is that it's substantially different from their own established format and it would cost a lot to change format, and for raw converter makers like Phase One's Capture One the color model of DNG differs substantially from their own making it hard to integrate. So DNG adoption is no easy task.
The main challenge with a standardized raw format like DNG is color rendition. Color science is no exact science and you can convert the raw data into colors in lots of different ways, and current raw converters have much different models, only Adobe has the one that is specified in the DNG standard. It's also something that vendors use for differentiation. Some like the color rendition in Lightroom, some prefer Capture One etc. Worth noting is that most/all proprietary formats have only little information in the format itself how colors should be rendered, you need information on the side (integrated into the proprietary raw converters) to be able to render colors.
Anyway, it's not hard to understand that Phase One, Nikon or Canon are not particularly fond of the idea to implement Adobe's color model in their raw converter, as they already have their own which they and many users like better than Adobe's found in Lightroom. This lack of standard however means that you as a user one cannot expect to be able to recreate the same color in 20 years from now with the software that exists then.
Instead of having a rigid color rendition model in the format (like DNG has today) I think it would be better to skip that and just concentrate on describing what has been captured, ie describing the spectral response of the sensor color filters so converters today and in the future can relate to that in the current best way. You'd lose the possibility to exactly recreate the color as you did with a 20 year old software, but I think that's doomed anyway since you just need to change a contrast curve or similar in the raw converter to alter color, if you want to be able to recreate a print far into the future you should save the readily processed TIFF.
Instead the principle of a standard raw format should be to save all information needed to adapt to any color model the raw converter chooses to implement.
-
Including information about _how the data should be rendered_ might be more difficult, as that might include development practices that the manufacturers guard closely?
Yes, and that's a mistake of DNG. Too little description of what has been captured and too much stuff about how color should be rendered. This mistake can be repaired though, you just need to extend the format with spectral information and soften the standard such that you're allowed to used own color models.
In addition I think it's good to have a reference color model though, and the current one with (matrix, DCP, white point stuff) is okay for that I suppose. It's mostly the tonecurve stuff I don't like about the current.
-
To actually answer the question this threads was originally about; I save my originals in raw format, but only use cameras that have been satisfactory reverse engineered. To me that means that it has good coverage in both proprietary raw converters and the open-source world, there must be more than just one software/company/entity supporting it. If Adobe's closed source DNG converter is the only way to get me into DNG it's not enough for me, and I won't buy the camera.
While I store originals in the camera's native format I do use DNG here and there in my workflow, I use my own Lumariver HDR for tonemapping tasks and then I often export to a "cooked" DNG file which then can be processed in many raw converters. If I make a print I also store the TIFF and sometimes other intermediate files, but the final TIFF is the only format that can guarantee that the print will be possible to recreate many years from now. I don't do a huge number of prints though so saving a few hundreds of megabytes of data per print is not a problem.
At some point I may convert originals to DNG, if/when the native raw format is losing support. I don't see a hurry to do it though. If I had been a world class photographer I'd probably have DNG copies of originals in the safe though, just for future proof safety, even if I would not use them in the workflow.
-
Hi Torger
I think the way the DNG spec should have been presented to the camera manufacturers was for an open source file with scene referred image data ie where all the device RGB to CIEXYZ transforms, interpolation have been done. The rendering for output then could be by third party. This would be as well as their RAW file , as an option on the camera menu.
This might have been more appealing. I think ADOBE misjudged what their response would be and asked for far too much.
NIKON would not have to reveal their camera matrix, spectral curves. etc.
User the could process in LR etc or any other 3rd part Rendering Software without having to wait for someone to produce a camer matrix.
might not have the same degree of flexibility but would be better that this situation we have now.
-
I think the way the DNG spec should have been presented to the camera manufacturers was for an open source file with scene referred image data ie where all the device RGB to CIEXYZ transforms, interpolation have been done.
Interesting idea. I'm afraid users would not consider the format to be "raw enough" though, as color conversions, possibly non-linear, then takes place inside the camera and you get a "cooked" raw.
Spectral response of the sensor is easy to measure if you have the gear, so it's not something that can or need to be a trade secret. In addition to this the format must support a way to store white balance setting, ie the scene illuminant as it affects color rendition. From a color model perspective it would ideally be the spectral response of the illuminant (which would work for presets), but for practical reasons one would probably want to have various ways to describe it, RGB multipliers being the most common.
If you have the spectral response of the sensor channels and some information about the white balance I think it should be possible to devise a color model that can generate a reference color matrix, which could have further non-linear refinements. Ie such a format would make it possible to generate sane color out of the box without any further knowledge of the camera. I'm not 100% sure if it would work in practice though. Maybe it's impossible to make a color model that is robust and reliable enough based on this input, and that you still would have to make a color checker type of calibration and attach a LUT. Of course most raw converters would have camera-specific color fine-tunings for important models anyway.
Assuming it would work, you would with this type of format gain that there would be *zero* things specified of how color should be rendered, the raw converter simply gets all information needed to be able to make their own decisions about color. This would make DNG look more like an open raw container and less like a native Adobe Lightroom format (as it looks today).
Concerning camera manufacturer trade secrets, they could still have their own proprietary color models in their proprietary converters, the above format would not expose any of that information.
In fact I'm a bit surprised that Adobe exposes so much of their Lightroom color engine through the current DNG format. But it's just too much expecting that competitors would rewrite their converters to adopt Adobe's engine, and then naturally always be one step behind Adobe themselves. I think Adobe could have done a better job making DNG an attractive format for established manufacturers and software vendors.
-
''Interesting idea. I'm afraid users would not consider the format to be "raw enough" though, as color conversions, possibly non-linear, then takes place inside the camera and you get a "cooked" raw.''
Hi
Spectral response is perhaps not a trade secret but its not a particular simple thing to do as you have expressed. It can get to a fine art. Hence time and money and hence reluctance to give away spectral data. etc.
Iam only surmising. I have on idea how NIKON etc think.
Of course it would be fantastic to have all the spectral data etc provided by NIKON etc. so i could bypass their software from day 1. It would suit me but perhaps not NIKON.
Iain
-
Of course it would be fantastic to have all the spectral data etc provided by NIKON etc. so i could bypass their software from day 1. It would suit me but perhaps not NIKON.
Is Nikon software free like Canon? It beats me why Canon seems to bother if I use their free tools or not, as long as I lay out the cash for the camera that they actually charge money for.
-h
-
I don't know if it's specifically Japanese or not, but I do get the sense that companies like canon, Nikon and Sony per default consider everything a trade secret. There's no rational discussion if something can be open or not, it seems to be in the culture that everything should be kept proprietary. So I don't have high hopes that those big companies ever will adopt a standard format :-\
-
Is Nikon software free like Canon? It beats me why Canon seems to bother if I use their free tools or not, as long as I lay out the cash for the camera that they actually charge money for.
-h
There are two applications available from Nikon first the free version Nikon ViewNX and the second a paid for version Nikon Capture NX2.
ViewNX is supplied when you purchase the camera and is also available as free download when updates are made. It was designed really to be a browser for NEF, TIFF and JPEG and also quick editing of NEF files including file conersion and basic WB and exposure changes. Also designed to link to Capture NX
The paid for Capture NX is more of a full blown raw processor with most of the features you would expect in any decent raw editor. AFAIK both produced for Nikon by Nik software and now seemingly only supported by Nikon in terms of updates for new cameras.
-
I don't know if it's specifically Japanese or not, but I do get the sense that companies like canon, Nikon and Sony per default consider everything a trade secret. There's no rational discussion if something can be open or not, it seems to be in the culture that everything should be kept proprietary. So I don't have high hopes that those big companies ever will adopt a standard format :-\
Based on my own experiences these companies all consider trade secrets anything to do with their own product, not specifically Japanese companies but between all companies regardless of being viewed as a competitor and therefore a threat or not. To get specific information relating to making different systems work together can result in hitting a blank wall, misinformation and finally complex NDA's to be signed.
I share the view about realistic hopes for a future standard format and genuinely feel that this is only likely to happen if enough pressure can be applied by a strong body representing photography industry as a whole
-
I have contacted Dave Coffin (http://www.cybercom.net/~dcoffin/) and let him know we were having a discussion here about DNG. I asked if he would be willing to take any questions from us. He has graciously agreed. If you do have a question or two for one of the world's foremost experts on RAW formats then share it here or if need be send me a personal message. I will collect them and forward them on.
I guess another person we could ask would be Eric Hyman (http://www.linkedin.com/pub/eric-hyman/1/22/a06) - I recall he or one of his team members had some definite ideas about DNG several years ago. However I'm not sure what he is doing these days, and I don't know his contact information. His LinkedIn profile indicates he is no longer working at Microsoft.
-
But it's just too much expecting that competitors would rewrite their converters to adopt Adobe's engine
several raw converters started to support (more or less, may be not all DNG tags related to color transforms) Adobe's DNG camera profiles - Iridient Rawdeveloper and RT (RT you shall be able to comment with the developer's knowledge)
and then naturally always be one step behind Adobe themselves.
changes in color transform models do not happen every year, so while there is a delay when Adobe introduces new tags (like with DNG 1.4 spec) until they published it is tolerable... but then if you want DNG to be adopted by camera manufacturer's with an argument that it will solve the delay between camera release and support by raw converters you have to disclose how color engine shall work to the full extent (all tags)... otherwise there is no use... I don't see our proponents of DNG here running to use pure matrix .dcp profiles ;D... so they 'd like to use in camera DNG raw files, but they'd like to use camera profiles for example with LUTs to clamp scene referred colors like Adobe does... otherwise when Adobe will release their support for a camera, you will have quite different colors rendering with saturated colors (and have to adjust your conversion parameters to use Adobe's profiles) or not use Adobe's profiles at all...
-
At some point I may convert originals to DNG, if/when the native raw format is losing support. I don't see a hurry to do it though.
but I bet you 'd not completely destroy your archived native raw originals at that same moment... and that is the real answer to the topic's question.
-
The three main problems with the proprietary formats are that they are 1) not publically documented and 2) incomplete (ie you need information about the camera that's not stored in the format itself), 3) there are many of them causing a substantial effort to implement support for them, making it harder for small software developers to get into the game.
When camera companies go out of business, like Kodak for example, you may end up with incomplete information so you cannot make full use of your raw files. As a medium format user I don't trust that the companies will stay around forever, in fact my own digital back has been discontinued and the format is no longer used by the company so it's just a matter of time before it disappears from software too (fortunately it's been reverse-engineered to a decent level in open source software). I only use cameras whose formats have been reverse-engineered to a satisfying level, ie good results can be had in open-source software.
The problem with DNG from a camera manufacturer is that it's substantially different from their own established format and it would cost a lot to change format, and for raw converter makers like Phase One's Capture One the color model of DNG differs substantially from their own making it hard to integrate. So DNG adoption is no easy task.
The main challenge with a standardized raw format like DNG is color rendition. Color science is no exact science and you can convert the raw data into colors in lots of different ways, and current raw converters have much different models, only Adobe has the one that is specified in the DNG standard. It's also something that vendors use for differentiation. Some like the color rendition in Lightroom, some prefer Capture One etc. Worth noting is that most/all proprietary formats have only little information in the format itself how colors should be rendered, you need information on the side (integrated into the proprietary raw converters) to be able to render colors.
Anyway, it's not hard to understand that Phase One, Nikon or Canon are not particularly fond of the idea to implement Adobe's color model in their raw converter, as they already have their own which they and many users like better than Adobe's found in Lightroom. This lack of standard however means that you as a user one cannot expect to be able to recreate the same color in 20 years from now with the software that exists then.
Instead of having a rigid color rendition model in the format (like DNG has today) I think it would be better to skip that and just concentrate on describing what has been captured, ie describing the spectral response of the sensor color filters so converters today and in the future can relate to that in the current best way. You'd lose the possibility to exactly recreate the color as you did with a 20 year old software, but I think that's doomed anyway since you just need to change a contrast curve or similar in the raw converter to alter color, if you want to be able to recreate a print far into the future you should save the readily processed TIFF.
Instead the principle of a standard raw format should be to save all information needed to adapt to any color model the raw converter chooses to implement.
DNG container can store ICC profiles and so there is no need for P1 to use dcp color model... even P1 is probably not using color transforms as ICC (the organization) prescribes, it still can use icc (container) to store whatever data they are storing there currently but inside DNG container.
-
Come on get your facts correct before you start trying to sling mud or insults assuming that you are accusing me personally of spreading FUD.
There are differences between cameras raw files from the same manufacturer and therefore the formats are not the same and the data still remains proprietary.
The fact that very little may have changed between camera models may be true but it seems that often the case is that even seemingly identical cameras have to wait for support in ACR and LR.
A typical recent case being the Nikon D610 a virtually identical model to the D600 with minor changes to combat a dirt/oil sensor issue – all should have remained the same but...
Nikon D600 NEF minimum ACR version 7.3 minimum LR version 4.3
Nikon D610 NEF minimum ACR version 8.3 minimum LR version 5.3
In this case these changes may have been no more than adding the camera model number to the ACR/LR database but could have involved much more work than that – I do not know and unless you are employed by Adobe neither do you
I simply suggest you just change the model in .NEF with hex editor and see for yourself ;D...
-
Again, if the formats are not identical, they are different no?
you mix the format with the content... you know the difference, do you ;D ? ... if I store the different value in a tag that contains camera model name, that will be a different content , but the same format...
-
I simply suggest you just change the model in .NEF with hex editor and see for yourself ;D...
That is not what you were suggesting in the reply to my post!
Ignoring the percieved intent of your reply, the suggestion of using a hex editor to change camera details has been mentioned in many posts. I am very familiar with hex editors but for this example I would use something like Exiftool as a quicker and simpler way to change data.
While changing the camera model name (and even the pixel count) may work to allow you to view a raw image currently unsuported it does not necessarily mean that you will get a comparable view of a normalised raw file i.e. as seen in any other raw converter that supports the new format. Many minor things can and do change when the manufacturer updates or changes a camera. Makernotes and other data may be moved rearranged etc. So unless you are prepared to do a byte for byte comparison of two cameras raw files to establish and understand what is happening your data changes are at the best suspect.
The D600, D610 changes are both a good and bad example. Good in the fact that technically the cameras are very similar including pixel count and (AFAIK) the demosaicing and processing algorithms. But nevertheless under the hood other changes have been made notably Nikon has made changes to the auto white balance system and also extended the ISO range - what else may there be? It is also a bad example due to the fact that the cameras are so close technically and to expect to make large changes to a raw from a very different current model from a manufacturers range is folly.
I would suggest that the vast majority of users would not want to go down this route of low level editing, therefore we are once again back at the beginning
-
That is not what you were suggesting in the reply to my post!
I suggesting that you try to FUD people by mixing the format and the content...
-
But nevertheless under the hood other changes have been made notably Nikon has made changes to the auto white balance system and also extended the ISO range - what else may there be?
and the values of WB and ISO are still recorded in the same tags in NEFs... data was changed, format was not... see the difference... it is like recoding different WB/ISO values in the same tags in DNG files... does Nikon introduce once in a while some different tags that Adobe's ACR/LR needs to use ? yes, but very rarely... ACR/LR are using quite limited set of data from NEFs as you know... just those that you see in DNG spec and a good chunk of those tags are filled by Adobe itself w/o extracting any data from NEFs for them becuase they suit how Adobe's raw converters do their work... again comparing sequential releases of dcraw (whose author is cooperating with Adobe, as it was on record said by, for example, Eric Chan) gives a very good idea about how often __FORMAT__ changes in terms of what Adobe needs to account for vs data (like camera model names, etc) in the same tags.
-
Hi,
My take is that there are minor changes, but enough changes so a new camera file won't work in existing raw converters.
Best regards
Erik
and the values of WB and ISO are still recorded in the same tags in NEFs... data was changed, format was not... see the difference... it is like recoding different WB/ISO values in the same tags in DNG files... does Nikon introduce once in a while some different tags that Adobe's ACR/LR needs to use ? yes, but very rarely... ACR/LR are using quite limited set of data from NEFs as you know... just those that you see in DNG spec and a good chunk of those tags are filled by Adobe itself w/o extracting any data from NEFs for them becuase they suit how Adobe's raw converters do their work... again comparing sequential releases of dcraw (whose author is cooperating with Adobe, as it was on record said by, for example, Eric Chan) gives a very good idea about how often __FORMAT__ changes in terms of what Adobe needs to account for vs data (like camera model names, etc) in the same tags.
-
Hi,
Phase One is a Danish company and Leaf is Israelian, are they better at sharing information? With MFD-s you can download spec sheets from CCD manufacturers site.
Best regards
Erik
I don't know if it's specifically Japanese or not, but I do get the sense that companies like canon, Nikon and Sony per default consider everything a trade secret. There's no rational discussion if something can be open or not, it seems to be in the culture that everything should be kept proprietary. So I don't have high hopes that those big companies ever will adopt a standard format :-\
-
I have read this entire thread and, despite the disagreement, I gained some knowledge and valuable insights. Still I am left scratching my head over two questions (thoughts) left unanswered; perhaps because the answers are too obvious to those better informed than I. I venture to enquire now that the fireworks have stopped.
On this and every other photography forum on the net we have all seen the following question and answer:
"I just bought a new camera and my ACR / LR (or other raw converter, other than the one provided by the camera maker) will not open the raw files." And every time the following answer is offered: "Download Adobe's DNG Converter (free), convert your raw files and they will open in your converter software."
So apparently it is not necessary for Thomas or Eric to "hack" the new raw format first, before making it possible for the DNG converter to read the file and convert it to a form that is at least readable (and quite serviceable, if not optimally so) by conversion software not yet "updated" for the newer raw format. So the question that occurs to me is: If the DNG converter can "decipher" these files (on the fly, so to speak) why has Adobe not built in this same capability into ACR / LR, thus short circuiting the intermediate conversion to DNG?
OK, next slide. Several weeks have passed since the new camera's raw format was introduced. Adobe has done their homework and decoded as much of the secret sauce as necessary to update ACR / LR so that the software can directly read the new format. At the same time they (Adobe) usually provide one or more new dcp profiles to be used with the new camera's raw files. And for their efforts they get more complaints than thanks: "I can't get the same colors with your software that I obtain using the camera makers converter." So apparently, despite their efforts, Thomas and Eric can't dig deep enough to truly duplicate the secret sauce in the camera makers raw format. That being the case, what is the downside of the camera makers adopting a DNG raw format wherein they could still conceal their magic sauce? My answer to the complainers would be: "So sh*t man; use the camera makers software if you prefer it, or create your own camera (dcp) profile." That works fine whether you routinely convert your raw files to DNG or not. Having a choice is nice.
One last question: I have read that conversion to DNG embeds a dcp profile into the file. I think this is, by default, the Adobe Standard profile (for the camera used). If I have created a custom dcp profile for my camera, is it possible to configure the DNG converter to embed my chosen profile while converting.
Thanks for your attention.
-
So apparently it is not necessary for Thomas or Eric to "hack" the new raw format first, before making it possible for the DNG converter to read the file and convert it to a form that is at least readable (and quite serviceable, if not optimally so) by conversion software not yet "updated" for the newer raw format. So the question that occurs to me is: If the DNG converter can "decipher" these files (on the fly, so to speak) why has Adobe not built in this same capability into ACR / LR, thus short circuiting the intermediate conversion to DNG?
Actually they do!
Tony Jay
-
So apparently it is not necessary for Thomas or Eric to "hack" the new raw format first, before making it possible for the DNG converter to read the file and convert it to a form that is at least readable (and quite serviceable, if not optimally so) by conversion software not yet "updated" for the newer raw format.
Actually as Tony wrote, no. Let me explain. Let's say that the latest DNG converter is 6.0, released January 1st 2014 (that's NOT the case but this is an analogy). Let's say that version supports 350 cameras. Now let's say yesterday, January 15th, 2014, Canon comes out with a new camera. Version 6.0 will not understand that new raw and cannot convert it to DNG.
Thomas has to get his hands on the camera, at the very least a raw file. He has to hack it. The DNG converter will be updated to 6.0.1. That takes time (Thomas has to get the camera. Adobe Q&E has to test it, an installer has to be built. The web page has to be updated to describe all this). Now that version understands 351 cameras. And then this unnecessary cycle happens again. Is that clear?
-
So apparently it is not necessary for Thomas or Eric to "hack" the new raw format first, before making it possible for the DNG converter to read the file and convert it to a form that is at least readable (and quite serviceable, if not optimally so) by conversion software not yet "updated" for the newer raw format.
Sorry, wrong...Thomas and Eric have to decode new raw files in order to have DNG Converter read them. And ACR/LR/DNG Converter all get updated at the same time. DNG Converter is for people with OLDER software whose ACR plug-in will never get upgraded..
So apparently, despite their efforts, Thomas and Eric can't dig deep enough to truly duplicate the secret sauce in the camera makers raw format.
Oh, they could, but to do so they would beed to adopt the camera maker's block box raw processing SDK. The vender match profiles are intended to simulate camera color. The Adobe Standard profile they build for each camera is designed to produce more accurate, normalized color.
One last question: I have read that conversion to DNG embeds a dcp profile into the file. I think this is, by default, the Adobe Standard profile (for the camera used). If I have created a custom dcp profile for my camera, is it possible to configure the DNG converter to embed my chosen profile while converting.
If you take a raw file, apply a new custom camera profile (other than the default Adobe Standard) and then save out the DNG, yes, the different custom profile is embedded in the DNG. But no, you can't have DNG Converter embed a custom profile...
-
If you take a raw file, apply a new custom camera profile (other than the default Adobe Standard) and then save out the DNG, yes, the different custom profile is embedded in the DNG. But no, you can't have DNG Converter embed a custom profile...
Not sure I understand that;
Firstly you say that a different profile is embedded in the DNG when it's converted from RAW, but then you say you can't embed a custom profile into a DNG.
-
Not sure I understand that;
Firstly you say that a different profile is embedded in the DNG when it's converted from RAW, but then you say you can't embed a custom profile into a DNG.
What Jeff wrote seems clear to me.
You can embed a DNG profile in a DNG. But NOT with the DNG converter (that's not it's role or job).
-
Many thanks to all who replied and corrected my misconceptions. :-[
-
What Jeff wrote seems clear to me.
You can embed a DNG profile in a DNG. But NOT with the DNG converter (that's not it's role or job).
actually you can, but certainly it is not like you can select a profile to embed there from within Adobe DNG converter itself - Adobe DNG converter will for example embed the camera profile that you use in ACR to work with original raw image... it will pickup which one from a sidecar xmp for example... as it was said by prev. poster... so it is the same effect (converter embedding the profile), albeit not so straighforward
-
Version 6.0 will not understand that new raw and cannot convert it to DNG.
just because DNG converter is written in such a manner that it will not work even there are no changes at all...
-
actually you can, but certainly it is not like you can select a profile to embed there from within Adobe DNG converter itself - Adobe DNG converter will for example embed the camera profile that you use in ACR to work with original raw image... it will pickup which one from a sidecar xmp for example... as it was said by prev. poster... so it is the same effect (converter embedding the profile), albeit not so straighforward
I'm not entirely clear in what you're writing ;D
It sounds like you're saying, the DNG converter embeds a DNG profile, not a custom DNG Profile.
-
just because DNG converter is written in such a manner that it will not work even there are no changes at all...
Sorry, I'm not understanding that either. No changes at all in the new DNG? I suppose that's very possible. Something has to be different, otherwise why would the DNG converter not accept and convert that newer raw? I understand that some have suggested we can ourselves alter something in a new camera file but why should we and what is being altered that would prevent this and why?
-
One last question: I have read that conversion to DNG embeds a dcp profile into the file. I think this is, by default, the Adobe Standard profile (for the camera used). If I have created a custom dcp profile for my camera, is it possible to configure the DNG converter to embed my chosen profile while converting.
you can open your raw file in ACR, select your custom profile, make sure that conversion setting are saved in sidecar .xmp and then Adobe DNG converter will embed that profile into converted DNG - but Adobe DNG converter itself does not allow you to select an arbitrary profile in its UI... now if you can't use ACR w/ the original raw file you can always embed profile later in DNG with ACR after raw -> DNG conversion.
-
Sorry, I'm not understanding that either. No changes at all in the new DNG?
in camera raw (DNG or not)... no changes at all except the camera model name inside... wait... was it Canon with different names for cameras in Japan and the rest of the world ::) - what a perfect example
-
I'm not entirely clear in what you're writing ;D
It sounds like you're saying, the DNG converter embeds a DNG profile, not a custom DNG Profile.
it embeds custom DNG profile if that profile is referenced in a sidecar xmp file at the moment of conversion
-
in camera raw (DNG or not)... no changes at all except the camera model name inside... wait... was it Canon with different names for cameras in Japan and the rest of the world ::) - what a perfect example
OK, got it. But there IS a change and that change prevents the conversion to DNG. Even if it takes Thomas 30 seconds to change this, the issues remain, users can't access that raw outside the manufacturer's converter. And Adobe and everyone else has to spend 30 seconds and all the time updating their software while we wait. And the cycle continues.
-
Hi,
My take is that there are minor changes, but enough changes so a new camera file won't work in existing raw converters.
Best regards
Erik
actually it works... just take hex editor and play w/ raws files :)
-
it embeds custom DNG profile if that profile is referenced in a sidecar xmp file at the moment of conversion
OK so if I have this correct:
You open the camera raw (which of course that converter understands).
You set say ACR for a custom DNG camera profile (since DNG profiles work with non DNG raws thankfully).
THEN you convert to DNG, that newer DNG camera profile is embedded. Of course that makes perfect sense, any other behavior would be a bad idea.
And of course lots of other data that was previously stored in a sidecar is now embedded in the DNG.