To DNG or not to DNG?

[Bart_van_der_Wolf]

I don't understand the idea of not fully supporting a format as specified and supported by others (competitors). I have zero issues with C1 not supporting DNG at all. As a potential future customer, I have a big issue with any format not being correctly or fully supported; DNG or otherwise. And what makes you think this is 'big engineering'?
The one man band who writes Iridient Developer, a superb raw processor, doesn't have this issue. Why should a 'slightly' larger company be any different?

Hi Andrew,

With all due respect, you are obviously not a programmer (software engineer). I am a parttime programmer, but not by profession. In simple terms, what others who offer DNG 'compatibility' might do, is extract the data section (raw undemosaiced Bayerized pixels) from the DNG and ignore the rest, some just use DCRaw or the LibRAW libraries to also do demosaicing anf the standard color tables, other cook their own recipe, and either go postprocessing with standard color table matrix conversions for color in Linear gama space or not.

That is not the internal Capture One workflow. Capture One has its own conversions (which according to many leads to superior results) based on ICC input source profiles, not output referred DNG and ProPhotoRGB (partly linear gamma version) workingspace like Adobe does. Apparently they tried to follow the DNG specification (by using the SDK) but ran into inconsistencies, incompatibilities, and or other issues, so they cannot simply shoehorn the DNG output into their own processing pipeline without additional conversions. DNG data is also not always Raw, it can also be demosaiced RGB per pixel, so lots of things need to be verified/converted back if even possible.

As has been explained in earlier threads on this same topic, but not understood by most, Capture One uses its own different color models to connect the scene referred ICC input profiled Bayerized data with the output profile, and does most processing in a sort of camera space that is large enough to avoid gamut clipping from the image processing. This also allows to modify the ICC input profile and save a modified ICC profile version for future use or as a new default for a camera.

This means that a whole lot of data from the DNG needs to be verified and converted to make that possible. And since a DNG can be created outside of a camera, a lot can be wrong that needs verification before it is used. Its usually not wrong, but it still needs to be verified to make sure. Apparently, as Anders Torger discovered, not all metadata fields are unambiguously described in the specification, so there is more room for (wrong) interpretation than a truly open specification should provide.

This all takes a lot of software engineering to cover, debug, maintain (Adobe has changed parts of the specification several times, we're currently at the fifth major version 1.4), and verify. And all this while there is already a perfectly usable and high quality conversion from original Camera Raw implemented. It's all extra work to achieve nothing better than is already there. Capture One chose to first spend their money on improving the Raw conversion engine (improved with version 7, and again with version 9), while Adobe is still using their 2012 engine, so they apparently are focusing on other things.

As said, small one-man band application writers may cut some corners (like only using the Raw Bayerized data or DCRaw output, and ignore the rest of the DNG file) as explained above, and that may also work to a point (although rarely as good as what Capture One produces). It seems like Phase One got trapped when trying to use the DNG SDK but ran into conversion issues. Maybe they should have ignored the DNG data with the exception of the Raw Bayerized data like others, but we'll see what the future holds. Maybe they'll figure out a way to get better quality out of a DNG than Adobe ...

Cheers,
Bart

[digitaldog]

Hi Andrew,
With all due respect, you are obviously not a programmer (software engineer).

No, but I'm in the software business (Pixel Genius) and pay for engineering! So I have a very good idea what it can cost me and my partners to request anything from engineering.

In simple terms, what others who offer DNG 'compatibility' might do, is extract the data section (raw undemosaiced Bayerized pixels) from the DNG and ignore the rest, some just use DCRaw or the LibRAW libraries to also do demosaicing anf the standard color tables, other cook their own recipe, and either go postprocessing with standard color table matrix conversions for color in Linear gama space or not.

None of that changes the basic fact that a tiny company, among others, can support a format properly, fully. while another can't or doesn't. The solution is simple; stop "supporting", poorly,  file formats. Or do it correctly. It's as simple as that!
If the C1 gang had a clue, they would see how many Adobe customers are looking to jump ship. They could and should address their needs to get them into their customer base. Affinty (Affinty Photo) is doing this! It's a smart move, got my money as a 'backup' plan should I ever bail on Photoshop.

That is not the internal Capture One workflow.

Then they should not support the format at all (and they should not be dictating workflow to me, that's another topic).

Capture One has its own conversions (which according to many leads to superior results) based on ICC input source profiles, not output referred DNG and ProPhotoRGB (partly linear gamma version) workingspace like Adobe does. Apparently they tried to follow the DNG specification (by using the SDK) but ran into inconsistencies, incompatibilities, and or other issues, so they cannot simply shoehorn the DNG output into their own processing pipeline without additional conversions. DNG data is also not always Raw, it can also be demosaiced RGB per pixel, so lots of things need to be verified/converted back if even possible.

Which has zero to do with supporting the file format correctly. IF their ICC approach is better, great (prove it), but what does that have to be with the raw data and it's file format container? Lightroom/ACR doesn't support DNG camera profiles on rendered images, it does support ICC profiles. So what?
Here are the facts: there are companies bigger and smaller than Phase that properly support DNG. Phase has made a conscious and I think stupid business decision to play favorites with file formats. Now if that's what they want to do, they will lose potential customers; I'm one! And I'm not alone here. Again, they can remove all DNG support which is fine with me. What they shouldn’t do is do a half baked effort which is what they are doing. There's zero excuse for that!

[AlterEgo]

That is not the internal Capture One workflow.

you don't need to follow DNG model of raw conversion (including using dcp profiles and how/when they are applied) to use the data in DNG container just like you use the date from "non DNG" raw files... and as for DNG for example C1 does support Ricoh cameras which are DNG only w/o any issues.

[Bart_van_der_Wolf]

No, but I'm in the software business (Pixel Genius) and pay for engineering! So I have a very good idea what it can cost me and my partners to request anything from engineering.

So have you (paid) for implementing DNG conversions, or do you piggy back on the Adobe flavor of conversions? Quite a difference I can tell you.

None of that changes the basic fact that a tiny company, among others, can support a format properly, fully. while another can't or doesn't.

Define 'support fully'. Is that 'ignore almost everything in the DNG except for the Raw Bayerized data', or convert using the SDK and use its (Adobe flavored) output?

Phase has made a conscious and I think stupid business decision to play favorites with file formats.

Fact? Or assumption based on you observation from a distance?

' The solution is simple; stop "supporting", poorly,  file formats. Or do it correctly.

Maybe they will, maybe they will do better than that, Who knows?

It's as simple as that! If the C1 gang had a clue, they would see how many Adobe customers are looking to jump ship.

Why, aren't they happy? Most people I hear about it seem quite happy with e.g. Lightroom (great DAM features, some are even happy with the conversion quality, good enough for the intended use).

They could and should address their needs to get them into their customer base. Affinty (Affinty Photo) is doing this! It's a smart move, got my money as a 'backup' plan should I ever bail on Photoshop. Then they should not support the format at all (and they should not be dictating workflow to me, that's another topic).

What do you mean, by 'dictating the workflow''?  Don't all applications have their own workflow?

IF their ICC approach is better, great (prove it), but what does that have to be with the raw data and it's file format container?

It has to do with getting the required data out of that container, not Adobe precooked, because that then takes an almost impossible job to convert back to the required really Raw input (if at all possible).

Here are the facts: there are companies bigger and smaller than Phase that properly support DNG.

Again, specify 'support', ignore or use the SDK precooked data and build on that? With programming, details matter.

Cheers,
Bart

[Bart_van_der_Wolf]

you don't need to follow DNG model of raw conversion (including using dcp profiles and how/when they are applied) to use the data in DNG container just like you use the date from "non DNG" raw files...

I agree, but apparently (it seems to me anyway that) Phase One took an early decision to follow the SDK, but found out that the back conversion is an almost impossible exercise.

...  and as for DNG for example C1 does support Ricoh cameras which are DNG only w/o any issues.

Well, we don't really know, because we can't compare the output with the non-DNG camera Raw, or can we? If we can, we could see which produces better results. I can use the DNG converter to produce a DNG version of my camera original Raw, and see that the difference in comparison may sometimes differ significantly.

Cheers,
Bart

[digitaldog]

So have you (paid) for implementing DNG conversions, or do you piggy back on the Adobe flavor of conversions? Quite a difference I can tell you.

You can? How so?

Define 'support fully'

Well for one, not having to embed a raw into the container so the app doesn't crash! Or the need to embed more data to speed it up (silly).

Why, aren't they happy? Most people I hear about it seem quite happy with e.g. Lightroom (great DAM features, some are even happy with the conversion quality, good enough for the intended use).

Very happy, today. I'm always interested in a backup plan, don't care to paint myself into a workflow corner. But then I have at least two non Adobe raw converters that support DNG as I desire (they certainly don't crash or require I embed the raw). C1 might have been on the list, it's not.

What do you mean, by 'dictating the workflow''?  Don't all applications have their own workflow?

What do you mean by it (you raised the issue)? Workflows that demand I embed the proprietary raw without crashing isn't a workflow I'll consider.

[digitaldog]

I agree, but apparently (it seems to me anyway that) Phase One took an early decision to follow the SDK, but found out that the back conversion is an almost impossible exercise.

Clearly for them, the crux of some of our disappointment in the company. Doesn't matter. There ARE companies that don't find it an impossible exercise.

[AlterEgo]

Well, we don't really know, because we can't compare the output with the non-DNG camera Raw, or can we?

we can compare how C1 works with Ricoh in camera DNG vs DNG converted from some other raw files...

[AlterEgo]

Clearly for them

however C1 developers did improve how it is working with DNG recently...

[digitaldog]

however C1 developers did improve how it is working with DNG recently...

That's good news. Maybe the pressure is working 
Got nothing against the company per se, other than this DNG issue. I'm on the sidelines, looking at possible other raw solutions. As I may have stated here, I've used software from this fine company dating back to their scanning back, 1998 or so*. If indeed they improve and fix the issue reported (crashing or faster access without having to embed the original raw), I applaud the effort.

* http://forum.luminous-landscape.com/index.php?topic=107870.msg888172#msg888172

[Hoggy]

we can compare how C1 works with Ricoh in camera DNG vs DNG converted from some other raw files...

As one whose DSLR's are Ricoh-Pentax, the K-3/II is selectable between PEF or DNG.  I believe all their flagships are selectable that way.  More prosumer types like the K-30 are DNG-only.

...... (replying to an earlier post) .....
As per the option to write into the DNG, alone, necessitating the image-data verification hash..  Once created, a DNG never has to be written into ever again - and in fact is the default for LR.  Only if one chooses to do so does it ever have to be written into again.  I don't understand the failure by various C1 evangelists to realize this.
Besides, the backup issue is a red herring.  Everyone has their own preferences for backup strategies.

[AlterEgo]

As one whose DSLR's are Ricoh-Pentax

a note = I was talking about genuine Ricoh-Ricoh cameras... not about Pentax brand.

[AlterEgo]

and in fact is the default for LR. 

that was not always the case  ... there was a time when DNG were written into w/o a warning... a thing that DNG proponents prefer not to remember, just like other things like dropping data read off sensor written to raw by camera during raw to DNG conversion, etc, etc... Adobe silently fixed a lot of bugs during those years, but so much for calls to use DNG as an archival format for non DNG raw files  ...

[digitaldog]

that was not always the case  ... there was a time when DNG were written into w/o a warning...

DNG's were written to in Lightroom automatically at some point in it's lifetime? What version was that?

[AlterEgo]

DNG's were written to in Lightroom automatically at some point in it's lifetime? What version was that?

that was probably couple of years ago... for example embedded (in DNG) JPG thumbnail was overwritten (Adobe fixed the issue since then).

[digitaldog]

that was probably couple of years ago... for example embedded (in DNG) JPG thumbnail was overwritten (Adobe fixed the issue since then).

What version? I don't recall that behavior. AFAIK, to write back to the DNG, you have to (have always had to?) invoke that command (on Mac, Command S) or to write XMP to DNG or TIFF/PSD, have the Auto Save XMP on.


From LR FAQ:

UPDATE DNG PREVIEW & METADATA
When you’re manually writing metadata to DNG files, you’ll note that the Metadata menu offers two options:
• Save Metadata to File just updates the XMP metadata, as it would with any other kind of file.
• Update DNG Preview & Metadata does the same, but it also updates the embedded preview  any DNG files to include your Develop edits.

[digitaldog]

Recent piece from Martin Evening:


DNG Pros, Cons and Myths
http://blogs.adobe.com/photoshop/2015/12/dng-pros-cons-and-myths.html

[Bart_van_der_Wolf]

What version? I don't recall that behavior. AFAIK, to write back to the DNG, you have to (have always had to?) invoke that command (on Mac, Command S) or to write XMP to DNG or TIFF/PSD, have the Auto Save XMP on.

Or when using the recommended Fast Load Data option (see attachment). I know it's an option, but who wants a slower performance when loading files ...

Cheers,
Bart

[Denis de Gannes]

Just a comment on the subject "To DNG or not to DNG" ?. This is subject that has been in contention and ongoing for over 10 years since Adobe has been lobbying for the format to be adopted as a universal standard, without success.
Most of the major digital camera manufacturers, including Nikon, Canon, Sony, Olympus, Panasonic, have not indicated the slightest interest in adopting the format.
It is of interest that Adobe who is not involved in the actual manufacture or sale of digital cameras is the one who is trying direct and influence the actual players in the market.
I am pretty sure Adobe is convinced that the format is beneficial to them and the users of their products, somehow they have not been able to convince some of the major players in the market.

[Hoggy]

that was not always the case  ... there was a time when DNG were written into w/o a warning... a thing that DNG proponents prefer not to remember, just like other things like dropping data read off sensor written to raw by camera during raw to DNG conversion, etc, etc... Adobe silently fixed a lot of bugs during those years, but so much for calls to use DNG as an archival format for non DNG raw files  ...

I started with LR4, and I somewhat recall something about that..  So it might have been around LR3.  Possibly 4, but I just can't remember.
Either way it aught not auto-write to it, so it's a good thing that the default got changed.

Regarding the Pentax mention, I meant to elaborate..  That if anyone with better cognitive ability wanted me to take pictures with my K-3 of some sort of test scene, recorded as both native-PEF and native-DNG, and send them, I'd be willing to do so.  (Secondary conversions to DNG could be done by the recipient.)