Yet some DNG comments (from a raw software developer)

[torger]

I'd like to make some comments on the DNG format, from a raw software development perspective. I've been involved in writing Lumariver HDR, a DNG-supporting HDR software, longtime contributor to RawTherapee, an open-source raw conversion software which supports DNG, and now my own personal project DCamProf, a camera profiling software that can do both DNG profiles and ICC profiles for Capture One and others. So I know quite a bit of the internals of the format.

I've also reverse-engineered parts of the Phase One IIQ format, Leaf .mos format, Hasselblad 3FR format, and Kodak DCR format. Kodak was one of those companies that actually went out of business without ever releasing any documentation for their format by the way.

It's a popular view on the internet that it's easy to reverse-engineer raw formats and that we therefore DNG is unnecessary, just wait for some guy on the internet (someone like me) to reverse engineer. Well, reverse-engineering is an utmost waste of programming talent, and you can rarely be fully sure that you got it exactly right. In many cases you leave some aspects of the format unsupported. I certainly don't enjoy reverse-engineering, but as a MFD user and open-source user I've had to reverse-engineer just to be able to use my own gear.

Then we have the DNG format. I'm actually a little bit disappointed with it, from a technical aspect. It's not ideally composed for wide adoption in the industry, many aspects of it make it look like an Adobe-specific format rather than something anyone can adopt.

One rather provoking aspect is that many tags in the DNG spec are only weakly described in the documentation and are clearly just parameter settings for Adobe Camera Raw, such as the green split tag, which says: "This tag specifies, in arbitrary units, how closely the values of the green pixels in the blue/green rows track the values of the green pixels in the red/green rows." You can't have "arbitrary units" if you expect standard support, every tag must be precisely defined! There are many more examples of this.

The DNG format does not support much camera calibration data, frequently used in medium format camera formats such as IIQ and 3FR, things like flatfield correction. Adobe's own DNG converter solves this by cooking the file -- irreversibly applying the calibration data in the conversion process -- which is incompatible with some scientific applications where you indeed want the calibration data separate.

With the DNG format comes DNG camera profiles (DCP). These too have some issues:

- Can't alter hue/saturation of neutral axis (ICC can)
- LUT can't alter value of neutral axis (not really a problem of the format, but a problem of the ACR pipeline)
- Can't handle hue discontinuity (not really a problem of the format, but a problem of the ACR pipeline)
- Baseline exposure split between DNG file and DCP. This means that the camera profile can't specify the exposure offset on its own, but must relate to what tag the DNG converter has set. This is not good design.
- Blackrender not clearly defined how it affects rendering. A camera profile format that has tags of undefined meaning, and this is a central one -- not acceptable for a standard format.
- Matrices must be white-point preserving (generally what you want, but there are other ways to optimize a matrix)
- Color space cropped to ProPhoto primaries, which clips some of the human gamut (can be issue for scientific applications)
- White balance model built into the profile format, it's not a bad model but it's not so wise if you want others to adopt the format (which of course already have their own white balance models)

The whole DNG profile format is designed to work in an Adobe Camera Raw conversion pipeline with a ProPhoto primary RGB working space, and of course the given white balance model. I think Adobe should have excluded this from the format, or at least separate it more clearly so you could use DNG but instead use a different color conversion pipeline. You can of course by ignoring parts of the document, but if Adobe had wanted this to be adopted by others they shouldn't have integrated their own color model so deeply into the format.

That DNG profiles can't alter the neutral color, like making a Sepia look, Adobe has explained with that DCPs are intended for colorimetric corrections, not "looks", but that's not really true. Adobe don't make colorimetric profiles themselves, as soon as the tone curve is applied (all Adobe's profiles have tone curves, embedded or implicitly) the colorimetric idea is broken. They have indeed also added a LookTable (good!) which is intended to apply subjective looks. They should update their format to fully adopt the flexibility of both colorimetric and subjective renders that ICC profiles already have.

These are quite many issues, and I think Adobe would have to work with these in order to make the format more widely useful, rather than just being a format for Adobe Lightroom.

So my conclusion is that yes, I love the attempt to make DNG a widely adopted standard and I really don't like reverse-engineering formats, but DNG is far from perfect and I really think it needs more work to make it easier to adopt by others than Adobe themselves, both in terms of format but also the documentation.

Now the DNG format is like "here's how we do it (we didn't document it all though), take it or leave it". By putting some effort into it I think you can make a standardized raw format that can work for all the variety of raw conversion pipelines out there, and it's a zillion times better than having lots of different proprietary digital formats. But it's not only the others fault. We can't really demand manufacturers to adopt an Adobe-specific format, and today I think DNG is too much of that thing.

End of rant.

[AlterEgo]

With the DNG format comes DNG camera profiles (DCP).

I think that it is not correct to mix (as in put together) DCP with DNG in a rant... nobody forces a raw converter to use DCP model... you can use whatever you wish and DNG container can store ICC container(s).

[digitaldog]

No Rant that I see, just an opinion based on facts (unless other's who know than I do about this have a counter set of facts). Thanks.

I think the points about how DNG as it exists today isn't ideal is a better and more useful tactic to result in change then the anti-Adobe, anti-DNG real rants we here around the web on a regular basis. How can DNG be improved, not dismissed is far more useful to discuss. So thanks for that.

[torger]

I think that it is not correct to mix (as in put together) DCP with DNG in a rant... nobody forces a raw converter to use DCP model... you can use whatever you wish and DNG container can store ICC container(s).

Yes, it's a fair point. But I think the documentation is written in such a way that DCP and DNG is overlapping and the line is not at all clear.

I can agree that this aspect is more about fixing the documentation, rather than the format. Clearly separating what is providing the raw samples and calibration data etc, and then have DCP with their matrices, colorspaces and baseline offsets even in a separate document. Now all tags, belonging to the color model or not are just listed as "DNG tags".

[torger]

No Rant that I see, just an opinion based on facts (unless other's who know than I do about this have a counter set of facts). Thanks.

I think the points about how DNG as it exists today isn't ideal is a better and more useful tactic to result in change then the anti-Adobe, anti-DNG real rants we here around the web on a regular basis. How can DNG be improved, not dismissed is far more useful to discuss. So thanks for that.

I agree with that. I would certainly not say that because DNG has some issues it's great with proprietary raw formats, because it's not. Proprietary raw formats is just lazy, not fair to the customers, and not responsible in terms of archival.

When reverse-engineering the 3FR I as always asked for the spec, and I actually got a reply (there are some really nice folks at Hasselblad), but the reason they didn't want to share it was not because the format was really secret, but that they wanted to have the freedom to change their format with new camera models without having to spend resources on documenting and inform third parties (indeed if I want to support the new H5D-50 with the Sony sensor I have more reverse-engineering work to do...). Today they only share with Adobe so they can do the 3FR format in the DNG converter.

Indeed using a standard format can be more work for the manufacturer, at least for a starter. Still there are a few players that has already adopted the DNG format, Pentax, Sinar, Leica, probably some more. So I think DNG has potential, but I'd love to see some more work from Adobe to "shape it up" to be easier to adopt and look less Adobe-specific.

[sandymc]

Agree with Torger.

What I would add is that the problem, and I think the reason why we're unlikely to see it moving closer to a standard, is that DNG isn't really a raw format. It's closer to being Adobe's intermediate format. So when e.g., ACR loads a raw file, that raw file is converted to a representation that's consistent for all raw formats. That consistent representation, in a form that can be written to disk, is DNG.

The end result is that while DNG is very useful - a number of my products use it, both as an input and an output - in some situations it doesn't easily fit.

Sandy

[AlterEgo]

but the reason they didn't want to share it was not because the format was really secret, but that they wanted to have the freedom to change their format with new camera models without having to spend resources on documenting and inform third parties

exactly the point which "pro DNG" party prefers to ignore ... the issue is not the format, but that there is no reason to document or rather a reason not to document and have the ability to change things at will and w/o informing 3rd parties.

[AlterEgo]

Still there are a few players that has already adopted the DNG format, Pentax, Sinar, Leica, probably some more.

and some players in proper cameras realm decided to abandon DNG in favor of their own formats - for example Samsung... as for Pentax - there were 2 companies using DNG : Ricoh and Pentax (sold to Hoya, sold to Ricoh), now there is one - Ricoh... Pentax is just a brand...


however DNG has better chances in cell phones

[torger]

It's closer to being Adobe's intermediate format.

Very good point, I should have mentioned that. After a while working with the format it becomes quite clear that DNG is a sort of dump of what Lightroom uses internally as a common representation of all raw formats. All raw converters have this type of common representation format internally, but they can be quite different converter to converter.

It's not that hard to just ignore the Adobe-specific parts and just care about the features that is needed for getting the raw samples in. However I think it's almost a bit arrogant by Adobe to have such a strong Adobe-specific tone over it, as if they have not really asked themselves "what would we need to do to make this format acceptable by many?".

If I would be another strong player in the industry I would certainly be a bit provoked by that. For example if I would be Phase One I would of course think Capture One has the best color, and I would not like to get deep into a format which is tainted by Adobe's ideas of how colors should be rendered, which is drastically different from how Capture One does it.

[AlterEgo]

I think it's almost a bit arrogant by Adobe

or rather certain people there 

[Schewe]

It's not that hard to just ignore the Adobe-specific parts and just care about the features that is needed for getting the raw samples in. However I think it's almost a bit arrogant by Adobe to have such a strong Adobe-specific tone over it, as if they have not really asked themselves "what would we need to do to make this format acceptable by many?".

That's not Adobe's doing...it's Thomas Knoll's doing. He personally oversees the development of DNG (with a hands off approach by Adobe) and Thomas does always try to do the right thing (he really does) but Adobe releases DNG for free so it's totally understandable that DNG has a Camera Raw pipeline centric approach. I'll ping Eric Chan about this thread (he doesn't post here lately). But he can pass along your "rant" to Thomas. It's useful to get real feedback about technical DNG issues (rather than political issues). Thomas has, in the past, made changes based on useful feedback.

[Telecaster]

Proprietary raw formats is just lazy, not fair to the customers, and not responsible in terms of archival.

+ (somewhere near infinity on this) 1

-Dave-

[torger]

That's not Adobe's doing...it's Thomas Knoll's doing. He personally oversees the development of DNG (with a hands off approach by Adobe) and Thomas does always try to do the right thing (he really does) but Adobe releases DNG for free so it's totally understandable that DNG has a Camera Raw pipeline centric approach. I'll ping Eric Chan about this thread (he doesn't post here lately). But he can pass along your "rant" to Thomas. It's useful to get real feedback about technical DNG issues (rather than political issues). Thomas has, in the past, made changes based on useful feedback.

I've only done raw conversion work for a few years, so I haven't followed DNG from start and really don't know the politics behind it, why they released it etc. I just thought that since they released it, it would be with the intention to make it a standard, but maybe it was for some other reason. In any case, if one wants it to be a standard, having it Adobe-centric is a bit problematic. One could say it's sort of a political issue, or shall we say diplomatic issue.

There are some real technical issues too though, should I provide serious feedback directly to Thomas Knoll I would need to sort it out a bit more to really think what the most important are.

Documentation first, that some tags are left with incomplete descriptions so that only Adobe's software can interpret them is the most serious aspect. As said I also think one should clearly separate out the color rendering aspects, which is the Adobe-centric stuff.

When it comes to the technical issues there may be differing opinions on what is the right thing to do.

I've most recently worked with the DCP part, so there is where I have the deepest technical insight.

- I think the baseline exposure should only belong to the DCP, not be split between DNG and DCP. I can't figure out a good reason to have it the way it is now. Now you can'ẗ specify the baseline in the DCP without knowing what the DNG writer (which may be some other software than Adobe's!) has added as baseline exposure.
- I think the pipeline should be more accepting on LUTs, as it is now it requires neutral axis to have 1,1,0 values, actually ACR ignores those LUT entries and have it hard-coded to 1,1,0. I also think that the pipeline should be able to handle discontinuity in the hue rotation which would allow more flexible hue control. When designing subjective look tables one can quite easily run into the hue discontinuity problem.
- Blackrender behavior must be precisely defined so other can replicate it, otherwise only Adobe can interpret those DCPs.
- I would rather have seen an unbounded floating point Lab space for the LUT rather than Prophoto RGB-HSV, but that is a quite big change... maybe just adding a new table type which is a basic 3D LUT just like in ICC profiles. Anything in anything out.
- Although it doesn't really hurt, one should know that having a tone curve embedded in the profile doesn't make that much sense. To make a sane color appearance result you cannot rely on a simple RGB-style curve (and using Adobe's RGB-HSV curve makes your look rather Adobeish which few except Adobe wants). You need to apply the curve in the LUT so you can have a more advanced tone reproduction operator, but since you can't scale the neutral axis in the LUT you need to employ the curve anyway, and then correct for its "wrongdoings" in the LUT, it works, but profile design becomes kind of strange. If I remember correctly it's actually not specified in the docs how the tone curve should work, but you can find it in the DNG SDK source code.

I really like the dual LUT approach, one for colorimetric corrections done pre-exposure (HueSatMap) and one for subjective looks (LookTable) applied post exposure/fill light etc. I think that split should be more clearly defined in the docs.

[digitaldog]

There are some real technical issues too though, should I provide serious feedback directly to Thomas Knoll I would need to sort it out a bit more to really think what the most important are.

I suggest you post here, can't hurt and I'm pretty sure, certainly though Jeff, Thomas will get those data points.

[Schewe]

There are some real technical issues too though, should I provide serious feedback directly to Thomas Knoll I would need to sort it out a bit more to really think what the most important are.

Yep...that would be the most efficient method of bringing about any changes :~)

I've already pinged Eric so any additional thoughts could also be posted here...

[torger]

As DNG involves both storing the raw samples and actual rendering, a good test to see if it can work as a standard is if a third party can based on the docs implement a pipeline that renders the same result.

There are currently a few challenges concerning this:

• ACR adds a default tone curve if the DCP lacks one. This is not defined in the documentation. Personally I think it would be more logical that no curve in the DCP means no curve (=linear), but now it is the way it is.
• Adobe's own DNG converter adds a baseline exposure, which is an offset the DCP works on. So the DCP must thus be designed with the DNG-embedded baseline exposure in mind. If the raw source is not Adobe's DNG converter the baseline exposure may be different (or more likely missing), and then the DCP will render with the wrong offset. I think the baseline exposure+baseline exposure offset tags should be phased out and replaced with a fixed offset belonging to the DCP only, so the DCP becomes autonomous.
• The "auto black subtraction" of the DefaultBlackRender tag needs to be defined, otherwise a third party can't render correctly with a DCP which has black render auto
• How the ProfileToneCurve should be applied is not documented

When it comes to the DNG as a raw container I don't have much to complain about (except for the docs on some tags), except for that I think it should have better support for calibration data. Being able to store IIQ and 3FR calibration data should be a good tester. I don't think it's ideal like now that the DNG converter applies the calibration data on the raw samples instead of storing it on the side as the original formats.

The DNG format has ICC support with the AsShotPreProfileMatrix and AsShotICCProfile tags, but I'm not sure if anyone is using them? In any case it needs to be more clearer defined how the pipeline should work in this case. Unfortunately ICC for cameras is in a sad state, as raw converters apply them so differently, so it would be hard to provide a set of tags that would work for all currently available raw pipelines. Capture One has one of the more elaborate ways, and actually two different ways, their own, and Leaf's method they inherited. Pre-matrix, curve, then exposure, fill light etc, and finally the ICC profile in the very end, and the ICC profile applies a gamma too. I don't think Capture One is a good role model in any case, whose design comes from the integer math days.

[BrianVS]

http://www.digitalpreservation.gov/formats/content/tiff_tags.shtml

I used to do a lot of data acquisition and image processing work when this stuff was new. NATO standard image transfer format was around, but it was by-and-large unique formats for each sensor. Now, writing a custom demosaic program for the Leica is just for fun. All I needed to do this for the Leica M8, M9, and M Monochrom was found in the page linked to above. It seems that most "raw" formats are TIFF 6.0 with proprietary extensions. The definitions for these proprietary extensions are not as well documented as DNG.

Kodak was happy to describe their DCS data format over the phone in 1993. Did not take much code to implement it.

It took 30 minutes to look at the NEF files from the Df and write code to make LR4.4 think it was a D4. ASCII "f" to ASCII "4". Stupid that anyone had to wait for an upgrade to LR to process Df files. Meaningless changes to Raw files to force an upgrade.

[torger]

The proprietary formats are riddled with these tiny silly upgrades, and requiring meta data from the side. Most irritating for us software developers.

In the open source world Dave Coffin with his DCRaw has made is still doing a massive contribution when it comes to keeping the proprietary formats accessible. I don't know what's going to happen when he stops, it will be a major blow to open source and many of the smaller commercial companies that rely on DCRaw for import.

Still many formats supported by DCRaw are only partially supported. There may be special shooting modes or subtle calibration data which is not applied etc. With the Kodak DCS it was already supported by DCRaw when I had a look at it, but there was some aspects around white balance I think that was not working properly.

The base structure of a proprietary format is usually simple, indeed all I know of are TIFF-based, but to support all tags for all models in all shooting modes can be a massive job, and what we usually end up with is a partially supported format. With the IIQ the open source world can't do compressed mode, can't do sensor+ mode, with the 3FR there are a number of models whose calibration data has not been reverse-engineered -- and I can guarantee that those aspects are not a 30 minute job :-/.

So indeed it would be great with a wider adoption of DNG.

As an added feature to DNG for archival I'd like to see that the camera SSF (=color filter response) could be embedded in tags, that would mean that raw converters could generate profiles on the fly, so you could get support for a new camera directly without even having a profile made for it. There are many ways to render color, but if you have the spectral sensitivity functions you have the source to do in any way you want.

[AlterEgo]

There may be special shooting modes or subtle calibration data which is not applied etc.

which DNG does not solve either... solution is not DNG, but documenting of whatever format is being used, that's it... and it is already shown that (dSLR/dSLM) manufacturers do not want or do not have compelling reasons to document...

PS: Ricoh/Pentax - what about "DNGPrivateData" tag in their DNG ?

[sandymc]

In the open source world Dave Coffin with his DCRaw has made is still doing a massive contribution when it comes to keeping the proprietary formats accessible. I don't know what's going to happen when he stops, it will be a major blow to open source and many of the smaller commercial companies that rely on DCRaw for import.

Dave is amazing. Hopefully, if he ever gets tired of doing DCRaw single handed, DCRaw could be formed up into a proper open source project, or perhaps e.g., LibRaw could take a more active role in developing new camera support. I for one would be happy to be part of that.

Sandy