DNG or RAW

DNG

22 (26.8%)

RAW

60 (73.2%)

Total Members Voted: 81

[digitaldog]

So you're against proprietary file formats except when you're in favor of them?

I'm against temporary unnecessary proprietary data that doesn't need to be proprietary for any reason other than to limit my use of my data. It is as simple as that.

When Dave Coffin figures out how camera manufacturers have encoded data in a file format he has decoded it.

How long after this non necessary proprietary data is this deciphered, how does that assist me and others who have to wait on Dave and others? Again, for what purpose is this temporary proprietary data useful other than to keep Dave and other's with busy work and me and other's from accessing our data?

The file format remains proprietary.  For example, MS Word .doc files are in proprietary format, even though the format has been reverse engineered by programmers who never worked for Microsoft. By contrast, MS Word .docx have never been proprietary, are fully documented, and are an ISO standard.

Correct, agreed. The difference is, an MS user who wishes to use the useful and proprietary MS word doc should understand this, the proprietary MS doc doesn't become non proprietary for use outside MS word months later (again begging the question, why make it proprietary in the first place)? No more than a Photoshop user should expect that his proprietary layers or Smart Objects will transfer exactly outside that app. The difference is there are two formats the camera provide and one is open and accessible the day the unit ships, the other isn't but will be and once that happens, what darn good was it that the data was proprietary or that many of us had to wait to use it like the JPEG? Is that such a difficult question?

It seems you don't pay much attention to the workflow of others when it comes to these aspects of post-processing.

How's that? My suggestion is a choice in the raw data provided. It in no way affects those that don't want the DNG. It does affect those that do. Seems like ample choice for workflows, not the opposite so what I'm I missing in asking for options?

[papa v2.0]

Hi guys.
Bit of a ding dong going on here.
Heres a few thought from sunny Scotland.

This goes out to everyone following this thread.

As a matter of interest, whom amongst us use the camera manufacturers own Raw software of if indeed, have ever done so?
If yes which is better colour rendering, Propriety of 3rd party?


I bought my new nikon d7100 and as Andrew says could not use it in LR or ACR or even in my version of DCraw. I had to use Nikon View NX2 or my jpegs.
But buy using View NX2 and Photoshop (sometimes luminance hdr) there was really no need to use LR or ACR or any other 3rd party converter, well at least for my type of photography any way. I doubt that would go for every one.

But the point is a you can use your RAW files 'straight out of the box' BUT just not with any other software but NIKON'S for example and that is frustrating if you need LR workflow, understandable.

BUT as NIKON charge for their upgrades to Capture NX2 they will probably want photographers to go down this route as opposed to say ADOBE. Its all business. Keeping you in their 'loop'.

So why would NIKON use an open format? It wouldn't help their sales of Capture NX2 or would it? And then one must ask why are NIKON developing colour software?

The other point is re the DNG spec itself. I think herein lies the main problem or sticking block.

Ill focus (pun alert) in on two main areas that (amongst others) would bother me if I were a camera manufacturer. (I'm not by the way, i'm just a colour scientist)
The spec asks for the colour matrix and white balance information.

Spectral  characterisation of a digital camera to produce a high quality matrix is not trivial (as was stated somewhere in a previous post) and refining the process takes time and money and knowhow. Why should I then give it away?

Same for white point estimation. A very complex subject, not least it is under-constrained, and years of research have gone into this problem. If then, I have developed a new white balance algorithm, again why should I give it away?

Without these bits the DNG spec fall flat on its face really and  we are back to Thomas and Madmanchan and their camera profiling. Or are we.

So just a final thought or two. Maybe if ADOBE redesigned  ARC /LR  (and to all raw developers) so that I could 'slot in' my matrixes from my profiling method instead, might that not be the way to go. Then that would open up the door for 3rd party spectral profiling services etc. That might take the pressure off of ABODE.

ADOBE will just have to buy out NIKON, CANON .........

I love what ADOBE do, great innovators, but im afraid the DNG idea is a dead duck for now.

Iain

[TonyW]

Read this thread with interest and was undecided until now if I should join in  .  So FWIW my take on the subject:

I have played with DNG conversions but have decided for now to stick with Nef and process via LR or ACR (Do not like Nikons own now outdated NX2).
 
I do however think that the number of raw proprietary formats is crazy – not sure how many there are but Adobe must have support for at least 400 cameras?  

To have a standard to comply to seems to me a good idea and if that standard can also incorporate manufacturers proprietary information (that is probably encrypted anyway) then we can have some compatibility across multiple platforms and still be able to access the proprietary stuff in the manufacturers own applications if required.  This does seem to be what DNG offers and I think we should thank Adobe or probably more particularly Thomas Knoll and others that worked on this and provided a free of charge converter and the fact that this is open source therefore effectively in the public domain.

I am sure that there must be many other non Adobe applications that support DNG and also sure that numbers must grow once more manufacturers opt in to the format either by their own choice or in response to public demand.  

One problem is that manufacturers do feel that their own proprietary format gives them the edge over their competitors and just maybe complying to a standard they perceive would lose them that edge– I speak from firsthand experience working for such a manufacturer!

The main problem is, I think that until there is this huge public demand we will be stuck in the new camera yet another new proprietary solution without a choice – other than to look elsewhere.  I doubt that Joe public is particularly interested in anything other than getting a nice pic and suggest that the volume of users wanting adoption of a non proprietary standard is relatively low even for professional grade cameras.  

So how can the manufacturers be lobbied effectively to adopt new and foreign standards?

I draw some parallels here to my own experience in digital imaging in medicine.  I was quite heavily involved in the first introductions of X ray digital imaging in the UK.  At that time all manufacturers’ equipment had to use proprietary formats and needless to say individual components were very expensive.  Typically a single clinician’s workstation cost around £45,000, comprising of a PC box running under Unix and a greyscale CRT monitor.  

Once an image had been acquired it would be archived available for viewing and manipulation but only via the manufacturer’s equipment using the proprietary algorithms.  Need to view the image elsewhere without the manufacturers workstation an proprietary processing bad luck for you even worse maybe for the patient!

The fact of so many proprietary formats for so many imaging modalities and lack of standards was recognised by ACR (American College of Radiologists) and NEMA (National Electrical Manufacturers Association) and a joint committee was formed to expand standards for digital imaging in medicine.  The net result  being the development of the DICOM standards in use today.  

To sell or have any hope of selling equipment to any health provider required the manufacturer to conform  to the DICOM standards and explicitly confirm such within their documentation.  

Conformance to standards did not mean that the manufacturer lost anything as proprietary stuff could be included that only their systems could interpret and this was accommodated with the provision of private tags and fields.  What it did mean is that clinical users worldwide could view images from any modality from any manufacturer without the need for specific manufacturer’s software.
 
Seems to me that this is pretty much what the DNG format is trying to accomplish and I do feel that it would benefit the industry as a whole.

The point though is ACR NEMA had the clout to push standardisation forward and I do not at this time see how a relatively small body of independent photographers can accomplish this – if there is a way point me to it and I will sign up.

Sorry for a rather long and rambling post but wanted to get it off my chest

[Damon Lynch]

Hi Andrew,

Can we please clarify something important before continuing our discussion? It's important that you understand that a proprietary format like CR2 remains proprietary regardless of whether Dave Coffin or Thomas Knoll decodes it for use in a raw convertor.  When and only when the data in the CR2 file is converted to an open format like DNG or jpeg or TIFF only then is it no longer proprietary (allowing for the proviso that DNG allows some proprietary metadata and even embedding the original raw file).  Maybe you mean to convey this meaning too, but your present terminology is quite ambiguous!

Damon

[digitaldog]

Can we please clarify something important before continuing our discussion? It's important that you understand that a proprietary format like CR2 remains proprietary regardless of whether Dave Coffin or Thomas Knoll decodes it for use in a raw convertor.  When and only when the data in the CR2 file is converted to an open format like DNG or jpeg or TIFF only then is it no longer proprietary (allowing for the proviso that DNG allows some proprietary metadata and even embedding the original raw file).  Maybe you mean to convey this meaning too, but your present terminology is quite ambiguous!

Understood, agree 100% of the above, that's exactly what I expect. And yes, there is a proviso that DNG allows (some/all?) proprietary metadata and even embedding the original raw file if one desires either or both.

What Dave and Thomas do is allow me access to that data outside of the manufacturer's converter of course, and should I decude that perhpas some of that proprietary data could be useful and I might visit said manufacturer's converter, I'd make sure I had that data archived somewhere. If it's not in the DNG, I'm to blame for not archiving the camrea raw original. Or if I had the choice (which I don't), for setting the camera to give me a DNG instead of the proprietary raw (and convert, as I'm doing now, after import).

[Bart_van_der_Wolf]

Hi Andrew,

Can we please clarify something important before continuing our discussion? It's important that you understand that a proprietary format like CR2 remains proprietary regardless of whether Dave Coffin or Thomas Knoll decodes it for use in a raw convertor.

Hi Damon,

I do get the distinct impression that some others do not grasp the notion yet, that the image data in a regular DNG and in a proprietary Raw file format are exactly the same set of numbers (ADUs or DNs). They are equally proprietary and abstract, they are just a set of numbers.

They are just Digital Numbers (DN) or Analog Digital Units (ADU), and the application that reads them will need to decide which number represents a Red filtered sensel, a Green filtered sensel, a Blue filtered sensel, or whatever filter color was used. The arrangement of the Bayer CFA filters is encoded in a different spot in the file.

Both the location of those filtered digital numbers, and the pattern arrangement, are stored at a different tag location. That tag location and the offset position into the file they hold may be different, but they are essentially pointing to the same abstract data. Only the container is different, and therefore the physical locations of those numbers in the file may be different.

To make photographic sense of those abstract numbers and to allow demosaicing, they are decoded by an application according to it's own recipe. Lightroom uses ACR for decoding, Capture One uses C1 decoding, DxO uses yet another decoding of those same numbers, as does Photo Ninja. The set of data numbers they extract for processing is always the same, regardless of the file container, whether it's DNG or Camera Raw.

Several other pieces of information are used to improve the interpretation of the Digital Numbers, the more intimate the knowledge of the meaning of those other pieces of information, the better the demosaicing result can be (although the quality of the demosaicing can also make a difference). Even something as trivial as what's Up and what's Down and Left or Right needs to be defined, and it is, regardless of the container.

When and only when the data in the CR2 file is converted to an open format like DNG or jpeg or TIFF only then is it no longer proprietary (allowing for the proviso that DNG allows some proprietary metadata and even embedding the original raw file).

But the conversion to DNG doesn't really convert to an image (it just rearranges the same data into an openly documented order), while a TIFF or a JPEG do contain visible image data (encoded in RGB(A), or CMYK, or Lab, or YCrCb, coordinates). The data in a Raw DNG format is as colorless as any Camera Raw, it yet needs to be demosaiced into an RGB image.

Because DNG is just an openly documented container, it can instead also store (additional) other types of data than Raw uncooked Bayer CFA values. It can e.g. store already demosaiced image data in a proprietary Adobe format, and processing parameters as used by Adobe software, but that is just processed/cooked (no longer Raw) data with proprietary Adobe application tags added, which is useful if one uses that particular image processing application, but may be useless for other applications.

This additional cooked data is not what we are talking about when Raw data in a DNG versus Raw data in a Camera Raw format is concerned. Those two are identical sets of abstract numbers, rearranged (=converted) in a different order, that still need to be interpreted and processed before they mean something useful that resembles a visual image instead of mere data numbers.

Cheers,
Bart

[Vladimirovich]

I do however think that the number of raw proprietary formats is crazy – not sure how many there are but Adobe must have support for at least 400 cameras?  

typical FUD is to claim that there are 400 different formats - no, there are much less... bother yourself to compare 2 sequential releases of dcraw code to see the differences introduced... true that once in a while there is a major feature being introduced and that is when more work is done to support that... having said that the issue w/ in camera raw is not whether it is DNG or non DNG - it is a matter whether manufacturers have reasons to document or not to document the content of the raw files... the real question is not why they are not using DNG, but why they are not providing the description of their own formats.

[Vladimirovich]

and who we leave the original question - that is about archiving (save the originals or not)... and not about using DNG in workflow (which is a non question, for as long as it does not involve non archiving of originals) or merits for manufacturers to document or not the format/content of their raw files (which they have... not to)

[papa v2.0]

Hi
The DNG falls flat on its face as it requires the camera matrix in order to function properly as an open standard.

Without the colour matrix as supplied by say NIKON etc we are back to the situation we are in at the moment, which is WAIT for someone to profile the camera.

NIKON etc wont give up their camera matrix, why should they, it cost them lots of investment in time resources skills to produce a camera matrix.

Like I said before WHY should they give it away for free?

This is why DNG doesnt work.

What you might get at best from NIKON etc is an non-encrypted RAW file  to an  ISO standard perhaps, but again we would require someone to profile the camera.

Iain

[digitaldog]

typical FUD is to claim that there are 400 different formats - no, there are much less...

Tony wrote about 400 cameras that were supported not 400 formats, you popped that in there. I agree with you, 400 isn't correct (it's hardly FUD IMHO), Adobe states more than 350:

The Adobe DNG Converter is a free utility that enables you to easily convert camera-specific raw files from more than 350 cameras to the more universal DNG raw format.

bother yourself to compare 2 sequential releases of dcraw code to see the differences introduced

So there are differences. And those differences do or do not allow other raw converters to understand that difference the day that new camera is released or not?

It doesn't matter if the difference is one byte. If it's different such it can't be read until that byte is updated/understood in all the other converters, your points are not helpful in solving the problem (access of the data).

it is a matter whether manufacturers have reasons to document or not to document the content of the raw files...

Fair enough. Let's stick with big two, Canon and Nikon. Do they document prior to release? Seems not as we wait weeks and months for updates to our preferred raw converter.

the real question is not why they are not using DNG, but why they are not providing the description of their own formats.

Again, fair enough! So now the question becomes, is the industry pressure better served by demanding they release this info OR to give us that DNG switch on the camera? I'm fine with either but would prefer DNG onto my PC card.

[digitaldog]

The DNG falls flat on its face as it requires the camera matrix in order to function properly as an open standard.
Without the colour matrix as supplied by say NIKON etc we are back to the situation we are in at the moment, which is WAIT for someone to profile the camera.
NIKON etc wont give up their camera matrix, why should they, it cost them lots of investment in time resources skills to produce a camera matrix.

What stops them from encrypting this into their DNG (we're talking about native DNG from said camera)? Or just using that data not shared to internally provide a DNG?

[Bart_van_der_Wolf]

and who we leave the original question - that is about archiving (save the originals or not)... and not about using DNG in workflow (which is a non question, for as long as it does not involve non archiving of originals) or merits for manufacturers to document or not the format/content of their raw files (which they have... not to)

Absolutely agree.

For archiving it makes no significant difference, conversion to DNG only rearranges the same Camera Raw data to openly documented tagged positions. No real benefits, just a slowdown before industry wide agreement on new tags is reached.

As a workflow format, there can be some benefits for image editing software developers (mostly Adobe, given its dominant position). No benefits for camera makers to be achieved, only an increased workload burden and competitive risks.

Cheers,
Bart

[papa v2.0]

Hi Andrew.
Im not quite understanding what you mean by "What stops them from encrypting this into their DNG".

Well if they did anyone could use their own RAW Processor to develop the files (which ideally would be great). Who would need NIKON software, they would be cutting off a revenue stream for one. But I think what stops them is for the reason I outlined previous post:-

Why should NIKON give away the matrix which has cost them to produce?

Would you if you invested a lot into camera characterisation. Again this is not a trivial process.

Iain

[digitaldog]

Hi Andrew.
Im not quite understanding what you mean by "What stops them from encrypting this into their DNG".

There's a place in the DNG for them to put this data no? IF I understand you correctly, there's some necessary data. It's proprietary. Certainly on day one of the camera release.
At some point in the future, Adobe and others are able to decifer what they need for us to render that data right? So what I don't understand is if this data you refer to is always proprietary (it's some secret sauce) can't it be placed into a private tag within the DNG? If the secret sauce is at some point understood (such other converteres have access), what's the point of it being proprietary in the first place? IOW, DNG as a container allows this secret sauce to remain secret.

Who would need NIKON software, they would be cutting off a revenue stream for one.

It's needed to be sure on day one! That's the problem. It isn't needed on day 30+ once everyone who needs to support that format in their product gets their hands on the original raw. So I don't see how this is an issue in terms of provding a camera DNG as an option. What am I missing? I really DO want to know (and FWIW huge thanks to Bart for his last two post. Made all this a worthwhile investment in time).

Why should NIKON give away the matrix which has cost them to produce?

If that data is going to be understood anyway, they should give it away because it will presumably aid some users who wish to use the new format the day the camera ships in a product they wish to use (like the JPEG). The proposal (one proposal) is to let Nikon build the DNG so it may be possible to do so without providing anything Nikon feels is secret sauce. If someone is going to eventually undrestand this data anyway, what's the point of doing such a poor job of keeping that data proprietary?

[papa v2.0]

Hi

Ah I see where you are going wrong.  The matrix is not included in the NEF file at present.
It never was there at any time.
Its in their software.  

To render a RAW file after encryption we need to produce a matrix ie Thomas et al.

Cant see why NIKON would want to just give it away. It probably cost them lots to produce.

[digitaldog]

Ah I see where you are going wrong.  The matrix is not included in the NEF file at present.
It never was there at any time.
Its in their software.

OK understood now. It really truly is proprietary and only useful/accessible in their software. If so, it's not an issue for us DNG users, we don't expect to use that product.

[papa v2.0]

Yes, thats right,  but without the matrix in the  DNG, the DNG cant be rendered so is no good unless a matrix is supplied, as is at the moment , by ADOBE on a 30day lead time.

DNG/ACR/LR and any other RAW converter (which use DCRAW and Thomas's matrixes) is fine if you want your colour matrix supplied by ADOBE.

If you want to use NIKON colour matrix use their software.

I would suggest that ADOBE update their products to allow other profilers to insert their colour matrixes for a particular camera. Now that would be beneficial to the industry.

iain

[digitaldog]

Yes, thats right,  but without the matrix in the  DNG, the DNG cant be rendered so is no good unless a matrix is supplied, as is at the moment , by ADOBE on a 30day lead time.

Could that data be stored in a camera generated DNG? Or maybe it's moot if we're using someone else’s converter...

DNG/ACR/LR and any other RAW converter (which use DCRAW and Thomas's matrixes) is fine if you want your colour matrix supplied by ADOBE.

Not an issue (kind of expected). And not just Adobe (as an example of another product that supports DNG, Iridient Developer)

If you want to use NIKON colour matrix use their software.

Got it. If this provides significantly superior images, then let the photographer decide and use a solution they prefer. I'm all for choices!

[TonyW]

Tony wrote about 400 cameras that were supported not 400 formats, you popped that in there. I agree with you, 400 isn't correct (it's hardly FUD IMHO), Adobe states more than 350...

Thanks Andrew, camera support is what I was referring to and I had read some time ago about number of cameras supported and now remember the figure as 350+ .

What is not clear to me is the format of the container that is DNG and what limitations would be imposed, and without looking at any documentation my guess is that it is constructed in such a way that manufacturers secret squirrel stuff and secret source algorithms do not have to be revealed to any other than the manufacturers own software.
 
Put simply the actual processing algorithms that allow a camera manufacturer to give a Super High Inverse Tone or a Vivid Burn You Eyes Out setting or in medicine a view of both soft tissue and bone detail is proprietary and guarded by the manufacturer and only interpreted exactly by their own software. 

The rendering of the raw file to get an acceptable first image on our screens will probably also be proprietary and hence the need for Thomas and co to work on each camera to get an acceptable starting point image (Adobe Standard) which may be different to the manufacturers ‘look ‘.  As a bonus for some cameras certain presets are created that get fairly close to the presets for ‘the look’. 

The manufacturer does not need to actually give us their interpretation (‘the Nikon/Canon Look’) but a normalised image within the DNG container – this may be as simple as the JPEG representation we see on the camera LCS.  If you shoot JPEG only (shame on you  ) you can take this file into any photo editing program without issue and there must be hundreds to choose from (there I go exaggerating again!) so why should raw files be so much different?

As far as Nikon go my impression is that they are not really that interested in improving the software and in fact they do not produce this in house,  at least Capture NX2.  This is/was from Nik software which has now been absorbed into Google.  AFAIK there have been no major changes to the software in a number of years probably going back to the 64 bit change.  The only major changes I think have been updates to the software to offer support for their new cameras – and the odd bug fix.  Similarly they have had to do this with the free Nikon View NX with each new camera release.

To any of the old school that actually remember that wonderful stuff called film and have some nostalgia for the non automatic everything cameras that did not use batteries a question. 
Would you have purchased a particular camera if you had to use the manufacturers own film to get an acceptable result? 

[digitaldog]

The manufacturer does not need to actually give us their interpretation (‘the Nikon/Canon Look’) but a normalised image within the DNG container –

They don't even have to do that. Read Bart's last two posts. The data we're talking about in DNG is raw. It's yet to be rendered.
DNG does allow minimal processing (Lossy DNG) and fully processed data. But that's a different story. All the manufacturer needs to do inside the camera is functionally equivalent of what we do when we convert to DNG in LR or with the standalone DNG converter.