DNG or RAW

DNG

22 (26.8%)

RAW

60 (73.2%)

Total Members Voted: 81

[TonyW]

come on... see how he is making FUD by equating the number of really different __FORMATS__ with a number of different camera models, which totally incorrect approach.

like camera model names, yes... a great difference indeed.

Come on get your facts correct before you start trying to sling mud or insults assuming that you are accusing me personally of spreading FUD.

There are differences between cameras raw files from the same manufacturer and therefore the formats are not the same and the data still remains proprietary.
 
The fact that very little may have changed between camera models may be true but it seems that often the case is that even seemingly identical cameras have to wait for support in ACR and LR.  

A typical recent case being the Nikon D610 a virtually identical model to the D600 with minor changes to combat a dirt/oil sensor issue – all should have remained the same but...
Nikon D600 NEF minimum ACR version 7.3 minimum LR version 4.3
Nikon D610 NEF minimum ACR version 8.3 minimum LR version 5.3

In this case these changes may have been no more than adding the camera model number to the ACR/LR database but could have involved much more work than that – I do not know and unless you are employed by Adobe neither do you

[papa v2.0]

Tim
Hi Here is a link to a talk i gave to local camera club.
Its a cut down version but illustrates the point.
Nikon left My matrix right. both sRGB

https://dl.dropboxusercontent.com/u/90361226/CIECAM02.pdf

Iain

[Tim Lookingbill]

Tim
Hi Here is a link to a talk i gave to local camera club.
Its a cut down version but illustrates the point.
Nikon left My matrix right. both sRGB

https://dl.dropboxusercontent.com/u/90361226/CIECAM02.pdf

Iain

Thanks, Iain.

Interesting how each image is made to look different with some looking like a warm, yellowish ivory white balance was applied while others just have their hue/sat tables tweaked for more richer blue skies and slightly more magenta saturated reds.

It looks similar to what can be done in ACR's HSL but without kicking up a lot of noise and a lot less work. I'm assuming your matrix profile produces less noise or at least keeps it the same, right? And do the profiles produce consistent results across a wide range of image dynamics and color gamut?

[papa v2.0]

Hi Tim

I made a matrix for each image based on the estimated white point and scene luminance at time of capture.

The project was really about a new and better method of optimising a matrix. the resulting dE's of training data sets was enough to evaluate the method.

I produced about 20 to 30 images to illustrate  the method. We never got as far as putting through psychophysical testing. That was the next stage when I get round to it.

Basically  the effect I was looking for was that as luminance increases our perception of colourfulness increases. This was achieved not just by the matrix but by using CIECAM02 and a CIECAM02 optimised matrix.
Its still work in progress when I get time.

iain

[hjulenissen]

What would be gained is that the DNG file is not encrypted like a NEF and as such could be read AND RENDERED by any application that supports DNG.

Proprietary data would have to become open data, no encrptyion

The first step should be describing the format of the raw data (sensel values) in such a way that they can be translated into 16-bit linear light numbers. I.e. revealing any encryption, data compression etc format.

Revealing the color plane subsampling and means of setting black/white points would be sufficient to render an off-colored output.

The problem is NIKON wont give all the information needed ie the MATRIX, to be able to RENDER from DNG from the camera.

DNG would be an open format so any 3rd party software could open and render directly from a DNG produce by the camera.
At present the DNG can can only be used after the NEF has been converted by ADOBE DNG to a DNG and the missing MATRIX inserted.
It is then and only then, open format available to everyone who supports DNG.

The second step might be embedding a camera description into the raw file. I.e. spectral response of the camera sensor. As this is relatively easy to reverse-engineer, and camera manufacturers surely have this info readily available, I cannot understand why they would make this hard for their customers. Describing the OLPF filtering (if any) etc might also be sensible.

Including information about _how the data should be rendered_ might be more difficult, as that might include development practices that the manufacturers guard closely?

-h

[torger]

The three main problems with the proprietary formats are that they are 1) not publically documented and 2) incomplete (ie you need information about the camera that's not stored in the format itself), 3) there are many of them causing a substantial effort to implement support for them, making it harder for small software developers to get into the game.

When camera companies go out of business, like Kodak for example, you may end up with incomplete information so you cannot make full use of your raw files. As a medium format user I don't trust that the companies will stay around forever, in fact my own digital back has been discontinued and the format is no longer used by the company so it's just a matter of time before it disappears from software too (fortunately it's been reverse-engineered to a decent level in open source software). I only use cameras whose formats have been reverse-engineered to a satisfying level, ie good results can be had in open-source software.

The problem with DNG from a camera manufacturer is that it's substantially different from their own established format and it would cost a lot to change format, and for raw converter makers like Phase One's Capture One the color model of DNG differs substantially from their own making it hard to integrate. So DNG adoption is no easy task.

The main challenge with a standardized raw format like DNG is color rendition. Color science is no exact science and you can convert the raw data into colors in lots of different ways, and current raw converters have much different models, only Adobe has the one that is specified in the DNG standard. It's also something that vendors use for differentiation. Some like the color rendition in Lightroom, some prefer Capture One etc. Worth noting is that most/all proprietary formats have only little information in the format itself how colors should be rendered, you need information on the side (integrated into the proprietary raw converters) to be able to render colors.

Anyway, it's not hard to understand that Phase One, Nikon or Canon are not particularly fond of the idea to implement Adobe's color model in their raw converter, as they already have their own which they and many users like better than Adobe's found in Lightroom. This lack of standard however means that you as a user one cannot expect to be able to recreate the same color in 20 years from now with the software that exists then.

Instead of having a rigid color rendition model in the format (like DNG has today) I think it would be better to skip that and just concentrate on describing what has been captured, ie describing the spectral response of the sensor color filters so converters today and in the future can relate to that in the current best way. You'd lose the possibility to exactly recreate the color as you did with a 20 year old software, but I think that's doomed anyway since you just need to change a contrast curve or similar in the raw converter to alter color, if you want to be able to recreate a print far into the future you should save the readily processed TIFF.

Instead the principle of a standard raw format should be to save all information needed to adapt to any color model the raw converter chooses to implement.

[torger]

Including information about _how the data should be rendered_ might be more difficult, as that might include development practices that the manufacturers guard closely?

Yes, and that's a mistake of DNG. Too little description of what has been captured and too much stuff about how color should be rendered. This mistake can be repaired though, you just need to extend the format with spectral information and soften the standard such that you're allowed to used own color models.

In addition I think it's good to have a reference color model though, and the current one with (matrix, DCP, white point stuff) is okay for that I suppose. It's mostly the tonecurve stuff I don't like about the current.

[torger]

To actually answer the question this threads was originally about; I save my originals in raw format, but only use cameras that have been satisfactory reverse engineered. To me that means that it has good coverage in both proprietary raw converters and the open-source world, there must be more than just one software/company/entity supporting it. If Adobe's closed source DNG converter is the only way to get me into DNG it's not enough for me, and I won't buy the camera.

While I store originals in the camera's native format I do use DNG here and there in my workflow, I use my own Lumariver HDR for tonemapping tasks and then I often export to a "cooked" DNG file which then can be processed in many raw converters. If I make a print I also store the TIFF and sometimes other intermediate files, but the final TIFF is the only format that can guarantee that the print will be possible to recreate many years from now. I don't do a huge number of prints though so saving a few hundreds of megabytes of data per print is not a problem.

At some point I may convert originals to DNG, if/when the native raw format is losing support. I don't see a hurry to do it though. If I had been a world class photographer I'd probably have DNG copies of originals in the safe though, just for future proof safety, even if I would not use them in the workflow.

[papa v2.0]

Hi Torger

I think the way the DNG spec should have been presented to the camera manufacturers was for an open source file with scene referred image data  ie where all the device RGB to CIEXYZ transforms, interpolation have been done. The rendering for output then could be by third party. This would be as well as their RAW file , as an option on the camera menu.

This might have been more appealing. I think ADOBE misjudged what their response would be and asked for far too much.

NIKON would not have to reveal their camera matrix, spectral curves. etc. 
User the could process in LR etc or any other 3rd part Rendering Software without having to wait for someone to produce a camer matrix.
might not have the same degree of flexibility but would be better that this situation we have now.

[torger]

I think the way the DNG spec should have been presented to the camera manufacturers was for an open source file with scene referred image data   ie where all the device RGB to CIEXYZ transforms, interpolation have been done.

Interesting idea. I'm afraid users would not consider the format to be "raw enough" though, as color conversions, possibly non-linear, then takes place inside the camera and you get a "cooked" raw.

Spectral response of the sensor is easy to measure if you have the gear, so it's not something that can or need to be a trade secret. In addition to this the format must support a way to store white balance setting, ie the scene illuminant as it affects color rendition. From a color model perspective it would ideally be the spectral response of the illuminant (which would work for presets), but for practical reasons one would probably want to have various ways to describe it, RGB multipliers being the most common.

If you have the spectral response of the sensor channels and some information about the white balance I think it should be possible to devise a color model that can generate a reference color matrix, which could have further non-linear refinements. Ie such a format would make it possible to generate sane color out of the box without any further knowledge of the camera. I'm not 100% sure if it would work in practice though. Maybe it's impossible to make a color model that is robust and reliable enough based on this input, and that you still would have to make a color checker type of calibration and attach a LUT. Of course most raw converters would have camera-specific color fine-tunings for important models anyway.

Assuming it would work, you would with this type of format gain that there would be *zero* things specified of how color should be rendered, the raw converter simply gets all information needed to be able to make their own decisions about color. This would make DNG look more like an open raw container and less like a native Adobe Lightroom format (as it looks today).

Concerning camera manufacturer trade secrets, they could still have their own proprietary color models in their proprietary converters, the above format would not expose any of that information.

In fact I'm a bit surprised that Adobe exposes so much of their Lightroom color engine through the current DNG format. But it's just too much expecting that competitors would rewrite their converters to adopt Adobe's engine, and then naturally always be one step behind Adobe themselves. I think Adobe could have done a better job making DNG an attractive format for established manufacturers and software vendors.

[papa v2.0]

''Interesting idea. I'm afraid users would not consider the format to be "raw enough" though, as color conversions, possibly non-linear, then takes place inside the camera and you get a "cooked" raw.''

Hi
Spectral response is perhaps not a trade secret but its not a particular simple thing to do as you have expressed. It can get to a fine art. Hence time and money and  hence reluctance to give away spectral data. etc.

 Iam only surmising. I have on idea how NIKON etc think.

Of course it would be fantastic to have all the spectral data etc provided by NIKON etc. so i could bypass their software from day 1. It would suit me but perhaps not NIKON.

Iain

[hjulenissen]

Of course it would be fantastic to have all the spectral data etc provided by NIKON etc. so i could bypass their software from day 1. It would suit me but perhaps not NIKON.

Is Nikon software free like Canon? It beats me why Canon seems to bother if I use their free tools or not, as long as I lay out the cash for the camera that they actually charge money for.

-h

[torger]

I don't know if it's specifically Japanese or not, but I do get the sense that companies like canon, Nikon and Sony per default consider everything a trade secret. There's no rational discussion if something can be open or not, it seems to be in the culture that everything should be kept proprietary. So I don't have high hopes that those big companies ever will adopt a standard format 

[TonyW]

Is Nikon software free like Canon? It beats me why Canon seems to bother if I use their free tools or not, as long as I lay out the cash for the camera that they actually charge money for.

-h

There are two applications available from Nikon first the free version Nikon ViewNX and the second a paid for version Nikon Capture NX2.  

ViewNX is supplied when you purchase the camera and is also available as free download when updates are made.  It was designed really to be a browser for NEF, TIFF and JPEG and also quick editing of NEF files including file conersion and basic WB and exposure changes.  Also designed to link to Capture NX

The paid for Capture NX is more of a full blown raw processor with most of the features you would expect in any decent raw editor.  AFAIK both produced for Nikon by Nik software and now seemingly only supported by Nikon in terms of updates for new cameras.  

[TonyW]

I don't know if it's specifically Japanese or not, but I do get the sense that companies like canon, Nikon and Sony per default consider everything a trade secret. There's no rational discussion if something can be open or not, it seems to be in the culture that everything should be kept proprietary. So I don't have high hopes that those big companies ever will adopt a standard format 

Based on my own experiences these companies all consider trade secrets anything to do with their own product, not specifically Japanese companies but between all companies regardless of being viewed as a competitor and therefore a threat or not.  To get specific information relating to making different systems work together can result in hitting a blank wall, misinformation and finally complex NDA's to be signed. 

I share the view about realistic hopes for a future standard format and genuinely feel that this is only likely to happen if enough pressure can be applied by a strong body representing photography industry as a whole

[Damon Lynch]

I have contacted Dave Coffin and let him know we were having a discussion here about DNG. I asked if he would be willing to take any questions from us. He has graciously  agreed. If you do have a question or two for one of the world's foremost experts on RAW formats then share it here or if need be send me a personal message. I will collect them and forward them on.

I guess another person we could ask would be Eric Hyman - I recall he or one of his team members had some definite ideas about DNG several years ago. However I'm not sure what he is doing these days, and I don't know his contact information. His LinkedIn profile indicates he is no longer working at Microsoft.

[Vladimirovich]

But it's just too much expecting that competitors would rewrite their converters to adopt Adobe's engine

several raw converters started to support (more or less, may be not all DNG tags related to color transforms) Adobe's DNG camera profiles - Iridient Rawdeveloper and RT (RT you shall be able to comment with the developer's knowledge)

and then naturally always be one step behind Adobe themselves.

changes in color transform models do not happen every year, so while there is a delay when Adobe introduces new tags (like with DNG 1.4 spec) until they published it is tolerable... but then if you want DNG to be adopted by camera manufacturer's with an argument that it will solve the delay between camera release and support by raw converters you have to disclose how color engine shall work to the full extent (all tags)... otherwise there is no use... I don't see our proponents of DNG here running to use pure matrix .dcp profiles  ... so they 'd like to use in camera DNG raw files, but they'd like to use camera profiles for example with LUTs to clamp scene referred colors like Adobe does... otherwise when Adobe will release their support for a camera, you will have quite different colors rendering with saturated colors (and have to adjust your conversion parameters to use Adobe's profiles) or not use Adobe's profiles at all...

[Vladimirovich]

At some point I may convert originals to DNG, if/when the native raw format is losing support. I don't see a hurry to do it though.

but I bet you 'd not completely destroy your archived native raw originals at that same moment... and that is the real answer to the topic's question.

[Vladimirovich]

The three main problems with the proprietary formats are that they are 1) not publically documented and 2) incomplete (ie you need information about the camera that's not stored in the format itself), 3) there are many of them causing a substantial effort to implement support for them, making it harder for small software developers to get into the game.

When camera companies go out of business, like Kodak for example, you may end up with incomplete information so you cannot make full use of your raw files. As a medium format user I don't trust that the companies will stay around forever, in fact my own digital back has been discontinued and the format is no longer used by the company so it's just a matter of time before it disappears from software too (fortunately it's been reverse-engineered to a decent level in open source software). I only use cameras whose formats have been reverse-engineered to a satisfying level, ie good results can be had in open-source software.

The problem with DNG from a camera manufacturer is that it's substantially different from their own established format and it would cost a lot to change format, and for raw converter makers like Phase One's Capture One the color model of DNG differs substantially from their own making it hard to integrate. So DNG adoption is no easy task.

The main challenge with a standardized raw format like DNG is color rendition. Color science is no exact science and you can convert the raw data into colors in lots of different ways, and current raw converters have much different models, only Adobe has the one that is specified in the DNG standard. It's also something that vendors use for differentiation. Some like the color rendition in Lightroom, some prefer Capture One etc. Worth noting is that most/all proprietary formats have only little information in the format itself how colors should be rendered, you need information on the side (integrated into the proprietary raw converters) to be able to render colors.

Anyway, it's not hard to understand that Phase One, Nikon or Canon are not particularly fond of the idea to implement Adobe's color model in their raw converter, as they already have their own which they and many users like better than Adobe's found in Lightroom. This lack of standard however means that you as a user one cannot expect to be able to recreate the same color in 20 years from now with the software that exists then.

Instead of having a rigid color rendition model in the format (like DNG has today) I think it would be better to skip that and just concentrate on describing what has been captured, ie describing the spectral response of the sensor color filters so converters today and in the future can relate to that in the current best way. You'd lose the possibility to exactly recreate the color as you did with a 20 year old software, but I think that's doomed anyway since you just need to change a contrast curve or similar in the raw converter to alter color, if you want to be able to recreate a print far into the future you should save the readily processed TIFF.

Instead the principle of a standard raw format should be to save all information needed to adapt to any color model the raw converter chooses to implement.

DNG container can store ICC profiles and so there is no need for P1 to use dcp color model... even P1 is probably not using color transforms as ICC (the organization) prescribes, it still can use icc (container) to store whatever data they are storing there currently but inside DNG container.

[Vladimirovich]

Come on get your facts correct before you start trying to sling mud or insults assuming that you are accusing me personally of spreading FUD.

There are differences between cameras raw files from the same manufacturer and therefore the formats are not the same and the data still remains proprietary.
 
The fact that very little may have changed between camera models may be true but it seems that often the case is that even seemingly identical cameras have to wait for support in ACR and LR.  

A typical recent case being the Nikon D610 a virtually identical model to the D600 with minor changes to combat a dirt/oil sensor issue – all should have remained the same but...
Nikon D600 NEF minimum ACR version 7.3 minimum LR version 4.3
Nikon D610 NEF minimum ACR version 8.3 minimum LR version 5.3

In this case these changes may have been no more than adding the camera model number to the ACR/LR database but could have involved much more work than that – I do not know and unless you are employed by Adobe neither do you

I simply suggest you just change the model in .NEF with hex editor and see for yourself  ...