DNG (from original .tiff) and CO8

[lkuhlmann]

Hi Jeff,

I agree it would be nice but, although the release notes for Capture One version 8.3 already mentioned DNG 1.4 support, as far as I can see there are still issues in e.g. color rendering, which is a bit surprising, as we are now at version 8.3.2. So support may be quite basic like that the files can be opened without issues, but not that all features are equally usable, if at all, or there are other reasons..

In another recent LuLa thread, Doug Peterson confirmed that C1 "doesn't provide full support for DNG", but that could mean many things. Not supporting Adobe DCP and lens correction data e.g. would be obvious candidates, because Capture one uses ICC camera profiles and their own lens correction data (to accommodate Tilt/Shift lenses and extreme wide-angle lenses with resulting Color Cast effects, and optional noise reduction that is tuned per camera model.

The fact that color rendering from the same file (as original camera Raw and as DNG conversion) with the same controls settings is so different is puzzling. I think it would offer Capture One a larger chance of having Adobe users switch to Capture One for high quality conversions, than the other way around.

Cheers,
Bart

Capture One 8.3.2 is "raw data" compatible with DNG 1.4. For some cameras the color model used by Abobe is just not compatible with the "Capture One color model". Also the lens distortion model are not the same. In 8.3.2 Capture One internally converts the color model used in the DNG.

Will Capture One embrace DNG fully going forward, even for colors? Yes as much as makes sense - including "native" colors. 8.3.2 is just first step.

-Lionel 

[digitaldog]

In another recent LuLa thread, Doug Peterson confirmed that C1 "doesn't provide full support for DNG", but that could mean many things.

It could, but it's pretty clear, why this product, unlike many others, doesn't produce the results Adobe and Jeff and I have repeatedly told you should occur and  does occur when software correctly handles a DNG. As I said on that post, Doesn't support full support' is code word for bug! Oh on, it's not a bug, it just doesn't work as designed. You're in denial!

[Bart_van_der_Wolf]

well, I am a newbie with CO, a one month of trial :-), but have experience with Lightroom. My impression is that capture one does a bit better job with dng files in general, and when the conditions are hard it can make a significant difference.

Hi Hans,

It doesn't mean that C1 cannot process DNGs at all, it's just that they are rendered differently compared to the Camera Raw originals if the same development settings are used. I'm pretty sure that a DNG conversion can be made that looks good by itself, it just needs different settings than for the original camera Raw, which is puzzling, but not the end of the world.

See my previously posted file of a night shot in Argentine at which nobody answered... The fact is that the 5d2 raw file was in 2009 converted to dng (no embedded camera raw).

So that would have been encoded with a much older DNG version, if the file hasn't been rewritten after editing. Which shows that there is a certain level of support, but not of all the features that could be stored in the DNG container, because many would not make sense to a different Raw converter.

Cheers,
Bart

[Bart_van_der_Wolf]

Capture One 8.3.2 is "raw data" compatible with DNG 1.4. For some cameras the color model used by Abobe is just not compatible with the "Capture One color model". Also the lens distortion model are not the same. In 8.3.2 Capture One internally converts the color model used in the DNG.

Will Capture One embrace DNG fully going forward, even for colors? Yes as much as makes sense - including "native" colors. 8.3.2 is just first step.

-Lionel  

Thanks a lot for chiming in Lionel. Don't worry about a few folks who go ape, I don't, they are allowed to make fools of themselves, it's a free society.

The Original Camera Raws that are supported are generally pretty well converted (although probably there's always room for improvement). It's just the fact that the Raw data in the original Camera Raws, and the DNG container version of that same (or it it altered?) data, produces different results with the same settings which is puzzling for users.

Good to learn that you intend to reduce the differences in rendering of 'native' colors, going forward. I know that resources are always limited, and that supporting DNG is not as easy as some might think, although the differences in color rendering are hard to grasp for people on the outside, myself included.

I've compared the EXIFs of an original CR2 and a DNG conversion of it with a text difference tool in NotePad+, and I can see that as a result from  the conversion some original data gets stripped, some data is changed, and some data is added (some of the stripped data in another form, and newly created DNG specific data), but I do not immediately see changes that could explain (to me) where the color rendering difference could be created, but it's perhaps something deeper than the EXIF allows to see.

Cheers,
Bart

[digitaldog]

It doesn't mean that C1 cannot process DNGs at all, it's just that they are rendered differently compared to the Camera Raw originals if the same development settings are used.

Why is that and what benefit does that provide? You remain silent on that question, hopefully Doug will reply.

I'm pretty sure that a DNG conversion can be made that looks good by itself, it just needs different settings than for the original camera Raw, which is puzzling.

It's interesting that despite the facts and testing by more than one person as to the reasons, and explanation from someone inside the company, you find this puzzling. Yet you're not puzzled why it is justified?

Don't worry about a few folks who go ape, I don't, they are allowed to make fools of themselves, it's a free society.

Hilarious. Someone (Myself and Eric) using actual testing methodology to illustrate multiple raw converters operate as they should with a DNG, instead of making assumptions based on heresy of others, (you) equates to you as 'going ape' and make fools of themselves. Is anyone taking you seriously other than you Bart?

Good to learn that you intend to reduce the differences in rendering of 'native' colors, going forward.

He shouldn't have to! The product should either treat DNG's correctly or not at all.

[AlterEgo]

Capture One 8.3.2 is "raw data" compatible with DNG 1.4. For some cameras the color model used by Abobe is just not compatible with the "Capture One color model". Also the lens distortion model are not the same. In 8.3.2 Capture One internally converts the color model used in the DNG.

Will Capture One embrace DNG fully going forward, even for colors? Yes as much as makes sense - including "native" colors. 8.3.2 is just first step.

-Lionel  

nice to see somebody who is actually from PhaseOne itself... what was the logic for that - "8.3.2 Capture One internally converts the color model used in the DNG. " instead of just treating DNG just as any other raw file format and applying the color transforms (guided by the data in icc/icm containers) the same way as for other raw files ?

[AlterEgo]

Part of that is, I would think, that DNG is also includes a recipe for raw conversion

you do not need to use neither "dcp" nor "icc/icm" data that may or may not be embedded in DNG container for a raw conversion - you shall be able just select whatever profile /external to DNG container/ for example P1 shipped with C1 - just like for example RPP does (with DNG files)

[brandon]

Thanks all for the range of opinions and insights. It is less than ideal that C1 cant make use of the DNG format, but I have learnt to avoid the conversion anyway. (so my question was about historical conversion on import to LR)
A little off the original topic, but as i posted it Id be interested to hear from proponents of DNG about 1) with the different standards mentioned is DNG really forward compatible (eg to some future time when camera raw support might be lost), and 2) what is the actual or perceived future value in using it (including eg creating a DNG with embedded raw =increased size). Im really struggling to see the functional or potential archival value.

[Bart_van_der_Wolf]

A little off the original topic, but as i posted it Id be interested to hear from proponents of DNG about 1) with the different standards mentioned is DNG really forward compatible (eg to some future time when camera raw support might be lost), and 2) what is the actual or perceived future value in using it (including eg creating a DNG with embedded raw =increased size). Im really struggling to see the functional or potential archival value.

A DNG is basically a TIFF type of file, similar to Camera Raws. As far as we know, if it started as an original camera Raw file, the DNG contains the same original Bayer CFA data. During conversion to DNG some data is stripped from the EXIF metadata header (e.g. I saw some missing GPS data references), some minor changes are made to some EXIF data fields, a lot of header data is added to the DNG that is of specific benefit for Adobe products (but might also be used by others should they choose to). Some of the added data may be calculated from the data it stripped, other data (e.g. an exposure bias, and color tables) is created purely to accommodate the ACR processes.

As people like Dave Coffin (author of DCRaw) have shown, a reasonable Raw conversion is possible based on little more than the basic Bayer CFA (or other layouts) data, but there is a lot of additional data available in the Raw file's metadata headers that could be utilized to make even better conversions. If and how the various Raw converters use that data is unknown. Some data is hidden in proprietary data sections (e.g. Makernotes, but there may be more), but some clever folks have succeeded in partly decoded that.

So future access to the Raw Bayer CFA data, and a reasonable demosaiced conversion of that, was already possible (and will be) for the future, even if the original manufacturers of the cameras and the Raw file format cease to exist. In fact, some reverse engineered conversion code (e.g. for Foveon) is said to perform better than the manufacturer's .

Another argument that is sometimes used is that it would be faster to release new camera support. Well, of course the cameras already come with camera support from the manufacturers. But for those who would like to use it in their own ecosystem, it would be nice if e.g. Adobe didn't have to do the work, but rather the manufacturers for Adobe. So that would benefit Adobe more than the manufacturers. In fact it only increases the dependency of users on Adobe instead of alternative software developers, which is neither in the interest of Camera makers, nor of the makers of alternative Raw converters, nor for users of those cameras.

Still, it would also be nice if DNG support (processing of DNGs Raw data) is good in other (non-Adobe) products, because that would increase competition. Some already do support DNGs in the sense that they basically just use the Bayer CFA data (just like DCRaw does) and then add their own secret sauce to that. So they can just ignore a lot of the data that Adobe dictates as mandatory to others who want to create DNGs, it's read only. But for those it also matters little if the Bayer CFA data was coming from the original Camera Raw file, or from a DNG conversion, it's the same Raw data.

For those who want to write DNG, there are much stricter guidelines which take some effort (time and money) to implement correctly for the ever changing new camera features. Some features are dropped, but the now irrelevant fields in the DNG still need to be populated with correct (but ineffective for conversion) data (to avoid tripping up some other software that makes a mistake an tries to use it), and new fields may need to be sanctioned by Adobe (and competitors?) before they can be added to the DNG standard and populated, well ahead in time before a new product can be launched.

Cheers,
Bart

[AlterEgo]

what is the actual or perceived future value in using it

it is sime a good intermediate format for a workflow, that's it

there are several different questions that people usually forget to distinguish

1) why camera manufacturers do not switch to "DNG" - a lot of legit reasons why __AND__ the real question is why they do not want to document whatever format they are using in terms of what all their different tags mean and how they shall be used by raw converters... that is the core issue.

2) why converting to DNG and deleting the original raws is a bad idea - even Adobe employees like Eric Chan don't do this (and publicly state that they don't do) and they know better 

3) using DNG as an intermediate format for a specific workflow - no issues at all, if that suits you - use by all means

[digitaldog]

A little off the original topic, but as i posted it Id be interested to hear from proponents of DNG about 1) with the different standards mentioned is DNG really forward compatible (eg to some future time when camera raw support might be lost), and 2) what is the actual or perceived future value in using it (including eg creating a DNG with embedded raw =increased size). Im really struggling to see the functional or potential archival value.

1. AFAIK, there's no issue converting a converted DNG to update the spec. I know I've converted older DNG's that didn't have Fast Load previews to newer versions with that feature. Jeff might have better data to share about this.
2. The actual value of DNG for some of us is well documented. Often smaller file size, no sidecar files, ability to embed multiple converter's parametric instructions within the container, ability to embed DNG camera profiles, Fast Load previews, a rendered JPEG of the current instructions, etc. So there are workflow advantages for some of us and there's the issue of file compatibility moving into the future (which no one can predict).

For some of us, those who've been doing this awhile, the history of proprietary file formats is either unsettling or down right painful. I've got all kinds of proprietary data who's original applications and/or developers have gone the way of the dodo bird. Kodak PhotoYCC data, iView from Live Picture, Xrez, old Kodak DCS camera 'raw' data etc. Is it impossible to access this data? No, not yet. I have very old hardware running very old operating systems such if I needed to access say PhotoCD data, I could. As long as I can get that very old hardware to remain viable. The JPEG and TIFF's I created from Photoshop 1.0.7 back in 1990, I can open today in Photoshop CC 2015 among lots of other applications on the most current OS and hardware. As should be an option for our image data.

So there are two ends to this DNG debate. Workflow advantages some of us desire and having some open (more open?) file format based on our experiences with proprietary file formats over the years. And no, rendering all the data to a TIFF before the old hardware dies isn't a good solution albeit, better than having no access to the data.

Lastly, proprietary image data, raw or otherwise isn't in a photographers best interest. An open, non proprietary raw file that could be accessed as easily and as differently in terms of software options isn't a technological issue, it's a political issue. Why should photographers have to be involved with politics surrounding their image data? The folks who dismiss DNG or another openly documented raw format never explain that.

[AlterEgo]

Kodak PhotoYCC data, iView from Live Picture, Xrez, old Kodak DCS camera 'raw' data etc.

see how he is painfully trying to come up with any relevant examples... counted one example from DCS cameras (which ones exactly ?)... http://www.nikonweb.com/forum/viewtopic.php?t=327 or http://www.nikonweb.com/forum/viewtopic.php?t=328 or did your old PhotoDesk, the last one version that was supporting those cameras,  magically stopped working in the proper OS in VmWare/Fusion/whatever else you can use to keep the old OS running ?

[AlterEgo]

Lastly, proprietary image data, raw or otherwise isn't in a photographers best interest.

so are proprietary parametric adjustments that lock photographer into a specific raw converter like ACR/LR 

[ErikKaffehr]

Hi Andrew,

Well said! I share your opinion.

Best regards
Erik

Lastly, proprietary image data, raw or otherwise isn't in a photographers best interest. An open, non proprietary raw file that could be accessed as easily and as differently in terms of software options isn't a technological issue, it's a political issue. Why should photographers have to be involved with politics surrounding their image data? The folks who dismiss DNG or another openly documented raw format never explain that.

[Bart_van_der_Wolf]

I have very old hardware running very old operating systems such if I needed to access say PhotoCD data, I could. As long as I can get that very old hardware to remain viable.

FYI,

There are several alternatives, some for free, that run on modern hardware. And this one runs on Windows 10 (and the MAC version runs on Mac OS X 10.5 (Leopard) or later on either Intel or PowerPC based systems) ...

In addition to that, PhotoCDs are film scans (so already a second generation image, a bit like archiving a TIFF), and if one has taken care of the film then even higher quality scans can be made.

Cheers,
Bart

[Bart_van_der_Wolf]

see how he is painfully trying to come up with any relevant examples... counted one example from DCS cameras (which ones exactly ?)... http://www.nikonweb.com/forum/viewtopic.php?t=327 or http://www.nikonweb.com/forum/viewtopic.php?t=328 or did your old PhotoDesk, the last one version that was supporting those cameras,  magically stopped working in the proper OS in VmWare/Fusion/whatever else you can use to keep the old OS running ?

And under current Operating Systems, the free DCraw (and also all programs that use it to extract the Raw data) will support Kodak DSLR's: DCS200, DCS315C, DCS330C, DCS420, DCS460, DCS460A, DCS520C, DCS560C, DCS620C, DCS620X, DCS660C, DCS660M, DCS720X, DCS760C, DCS760M, EOSDCS1, EOSDCS3B, NC2000F, Kodak ProBack, PB645C, PB645H, PB645M, DCS Pro 14n, DCS Pro 14nx, DCS Pro SLR/c, DCS Pro SLR/n

There are several people who provide pre-compiled binaries (as a command line tool) for most popular desktop Operating Systems (e.g. here for Windows).

Cheers,
Bart

[Some Guy]

Aside, I find it interesting that most all the RAW capable cellphones are using DNG as their output file (Not a NEF, RAW, etc.) along with the JPG.  Sony makes some sensors for the phones and they use DNG.  Samsung S6 Edge is also using DNG for their RAW images as of June.  DNG might be the standard of the future, at least for the cellphone cameras.  Of course they are pretty much pre-set in that they have one fixed lens, set diaphragm, a pre-set vignetting, and noise routine so all that may be compensated in the DNG easier than trying to write all that into various methods buried in a RAW file too.

No doubt the camera makers can produce a better DNG than third parties too as they know their proprietary hidden RAW info.  I was surprised that Nikon included PF Coma Correction in Capture NX-D for their new Fresnel lenses.  Doubt if a third party converter is onto that one yet, and likely just trying to produce with what looks good.

SG

[AlterEgo]

Aside, I find it interesting that most all the RAW capable cellphones are using DNG as their output file (Not a NEF, RAW, etc.) along with the JPG. 

that simply means it makes sense for those companies to use DNG for that market, specifically if they have already code support from the core OS code vendor (Android, etc)... there is nothing wrong with DNG itself ___if__ the company itself wants to use it and sees benefits for itself in that (which is clearly not the case with most manufacturers in the area of still cameras)... but again the core problem is if the manufacturer wants to document what their firmware is writing, be it DNG or other tiff based format or non tiff based format.

Samsung S6 Edge is also using DNG for their RAW images as of June. 

and they actually dropped DNG for their dSLM cameras, even after experiencing it while collaborating with Pentax in dSLRs... go figure 

[digitaldog]

Aside, I find it interesting that most all the RAW capable cellphones are using DNG as their output file (Not a NEF, RAW, etc.) along with the JPG. 

Indeed! Probably far more to do with their disinterest in politics than those other companies who remain nameless. 

[AlterEgo]

Indeed! Probably far more to do with their disinterest in politics than those other companies who remain nameless. 

Samsung example was naturally ignored (dropping DNG for still cameras)... companies do use what makes sense for them, that's it...