Pages: [1] 2   Go Down

Author Topic: DNG (from original .tiff) and CO8  (Read 45178 times)

brandon

  • Jr. Member
  • **
  • Offline Offline
  • Posts: 80
DNG (from original .tiff) and CO8
« on: July 19, 2015, 03:53:18 am »

Sorry if this is an oldie , but I havent seen a full explanation (and maybe things have changed). Some years ago I converted my raw files to DNG (and didnt keep the raws , from 1DsMk1, G10 etc) and imported to LR. Later using CO (vers 6,7 and now 8) I cant seem to get the same functionality as if they were native files esp lens corrections etc. Proponents of DNG seem to say it contains the raw file data in a different container, and is open spec and therefore should be able to be used by all software manufacturers etc. Im not interested in the "politics" of it all, just what way I can utilise all the original file information within CO8 in developing these images. Do I need to "reconstruct" the original raw formats? if so how? can CO8 now do this? Thanks in advance for advice.
Logged

Oscar Avellaneda-Cruz

  • Newbie
  • *
  • Offline Offline
  • Posts: 5
    • Ave Photo
Re: DNG (from original .tiff) and CO8
« Reply #1 on: July 19, 2015, 04:13:21 am »

Each camera manufacturer can embed proprietary data into their RAW file.
They do this so there is an incentive to experience processing with their software.
For example, some manufactures will embed which AF sensor was active at the time
of exposure.

Adobe having a monopoly, is the decider of what stays and goes.
After all they want the experience to be best for their platform.

The only sure solution to your situation is embedding the original RAW.
Logged

Bart_van_der_Wolf

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 8914
Re: DNG (from original .tiff) and CO8
« Reply #2 on: July 19, 2015, 05:22:03 am »

Sorry if this is an oldie , but I havent seen a full explanation (and maybe things have changed). Some years ago I converted my raw files to DNG (and didnt keep the raws , from 1DsMk1, G10 etc) and imported to LR. Later using CO (vers 6,7 and now 8) I cant seem to get the same functionality as if they were native files esp lens corrections etc.

Hi Brandon,

Sad to hear that you tossed the camera original Raws (which usually is a TIFF type of format) after conversion to DNG. The DNG specs have changed over the years, and earlier versions didn't record all data that the Raws had available. More recent versions seem to be more complete, but you can never tell if everything that may prove to be beneficial later, was carried over. The only way to do that is by embedding the entire original Raw in the DNG, but that only increases file size for little benefit. If you had done that, then at least you'd be able and restore the original Raw.

In the case of Capture One, it possibly is another reason. C1 supports the original camera Raws of a huge number of cameras. They invested time and money to decode the files and maybe even figured out some of the secret stuff that's hidden in the file, to allow even better quality conversions. The support for DNGs, is not as extensive, and may yield lower quality conversion. There is little incentive for Phase One to recode the work they did years ago, and run the risk of screwing something up in the conversion process.

They also would have to redo some of that (like adding code to initialize the new fields, or changing the way the Raw conversion uses that data) if the DNG standard is changed in the future even though the original camera Raws didn't change since they were recorded. Since the DNG can also be created by others than the Camera maker (e.g. LR/ACR panoramas, or HDRs, or other sources), they cannot be certain either that the converter gets fed valid Raw data that can be used by Capture One (encoding errors or abnormal sizes), so they are suddenly forced to add validations to the converter code to avoid problems for which they would also need to ramp up support.

So it adds (future) cost to redo past work and to support a standard they do not own, and thus cannot plan resources for. And there is no real benefit for them to fully support DNG, because they still need to create camera profiles or specific decoding (e.g. Fuji X-trans) for new cameras that require support, and in addition supporting DNGs will only lower the threshold for users that for some specific feature want to switch to Adobe products (who try to lock in customers from switching back) and drop future C1 upgrades at any time they choose.

So there is more of a benefit for Adobe than for Phase One in fully supporting DNGs.

Quote
Proponents of DNG seem to say it contains the raw file data in a different container, and is open spec and therefore should be able to be used by all software manufacturers etc. Im not interested in the "politics" of it all, just what way I can utilise all the original file information within CO8 in developing these images. Do I need to "reconstruct" the original raw formats? if so how? can CO8 now do this? Thanks in advance for advice.

Only if you created DNGs that embedded the entire original Raws. Otherwise you got screwed by the propaganda, and you have to suffer the consequences, not the propagandists. Lesson for the future, never toss originals, ever.

Cheers,
Bart
Logged
== If you do what you did, you'll get what you got. ==

brandon

  • Jr. Member
  • **
  • Offline Offline
  • Posts: 80
Re: DNG (from original .tiff) and CO8
« Reply #3 on: July 19, 2015, 06:38:10 am »

Thanks Bart for a thorough and learned explanation as always. Ive stayed in native raw format since and learnt the lesson. Its not the end of the world of course, but photos from LR with lens correction (eg the G10 I used as a travel camera around Cameroun in 2010 and 2011) look very different (distortion wise) from the CO ones subsequently let alone other changes. I may very well have embedded the original in the DNG (how do I tell in retrospect?) as Ive tended to be very conservative (in the true sense), but as I was living in that AFrican nation with limited media cards and backup I never stored the proprietary raws before reusing in the field.
Logged

Bart_van_der_Wolf

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 8914
Re: DNG (from original .tiff) and CO8
« Reply #4 on: July 19, 2015, 11:00:52 am »

Thanks Bart for a thorough and learned explanation as always. Ive stayed in native raw format since and learnt the lesson. Its not the end of the world of course, but photos from LR with lens correction (eg the G10 I used as a travel camera around Cameroun in 2010 and 2011) look very different (distortion wise) from the CO ones subsequently let alone other changes. I may very well have embedded the original in the DNG (how do I tell in retrospect?) as Ive tended to be very conservative (in the true sense), but as I was living in that AFrican nation with limited media cards and backup I never stored the proprietary raws before reusing in the field.

It would probably take the Adobe DNGconverter (version 9.1 currently), to reverse the embedding of the original camera Raw in the DNG, assuming it was originally fully embedded.

Here are the Windows versions for download, and here for the Mac.

Cheers,
Bart
Logged
== If you do what you did, you'll get what you got. ==

ErikKaffehr

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 11311
    • Echophoto
Re: DNG (from original .tiff) and CO8
« Reply #5 on: July 19, 2015, 04:54:01 pm »

Hi,

I don't agree with Bart on this one.

To begin with, DNG is well documented, so Capture One doesn't need to do a lot of decoding effort to support DNG.

The other fact is that many other raw developers support DNG very well. The ones I have used are RawTherapy, AccuRaw, Iridient Raw Developer and of course LR/ACR.

I have used DNG since back in 2006. I decided to use the option to embed the original raw file in the DNG. Using this option I can always extract the original raw file from the DNG.

On the other hand, I must freely admit that three raw converters I tried never did properly support DNG, and these were Apples's Aperture, Capture One and DxO. Personally, I don't understand why some vendors refuse to support the only well documented raw format, but I guess this is more politics than an engineering choice.

All raw converters I have seen, except the three I mentioned do support DNG.

I would think that it is possible to write code that can generate raw format from DNG, but as there are something like 300 different raw formats it may be an excessive effort.

Best regards
Erik

Hi Brandon,

Sad to hear that you tossed the camera original Raws (which usually is a TIFF type of format) after conversion to DNG. The DNG specs have changed over the years, and earlier versions didn't record all data that the Raws had available. More recent versions seem to be more complete, but you can never tell if everything that may prove to be beneficial later, was carried over. The only way to do that is by embedding the entire original Raw in the DNG, but that only increases file size for little benefit. If you had done that, then at least you'd be able and restore the original Raw.

In the case of Capture One, it possibly is another reason. C1 supports the original camera Raws of a huge number of cameras. They invested time and money to decode the files and maybe even figured out some of the secret stuff that's hidden in the file, to allow even better quality conversions. The support for DNGs, is not as extensive, and may yield lower quality conversion. There is little incentive for Phase One to recode the work they did years ago, and run the risk of screwing something up in the conversion process.

They also would have to redo some of that (like adding code to initialize the new fields, or changing the way the Raw conversion uses that data) if the DNG standard is changed in the future even though the original camera Raws didn't change since they were recorded. Since the DNG can also be created by others than the Camera maker (e.g. LR/ACR panoramas, or HDRs, or other sources), they cannot be certain either that the converter gets fed valid Raw data that can be used by Capture One (encoding errors or abnormal sizes), so they are suddenly forced to add validations to the converter code to avoid problems for which they would also need to ramp up support.

So it adds (future) cost to redo past work and to support a standard they do not own, and thus cannot plan resources for. And there is no real benefit for them to fully support DNG, because they still need to create camera profiles or specific decoding (e.g. Fuji X-trans) for new cameras that require support, and in addition supporting DNGs will only lower the threshold for users that for some specific feature want to switch to Adobe products (who try to lock in customers from switching back) and drop future C1 upgrades at any time they choose.

So there is more of a benefit for Adobe than for Phase One in fully supporting DNGs.

Only if you created DNGs that embedded the entire original Raws. Otherwise you got screwed by the propaganda, and you have to suffer the consequences, not the propagandists. Lesson for the future, never toss originals, ever.

Cheers,
Bart
Logged
Erik Kaffehr
 

Doug Peterson

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 4210
    • http://www.doug-peterson.com
Re: DNG (from original .tiff) and CO8
« Reply #6 on: July 19, 2015, 05:03:58 pm »

In general:
- leave your raws as they come out of the camera
- process your very favorite raws to TIFF and keep the TIFF

IMO the above two simple recommendations represent the greatest chance of avoiding hassle/loss as a result of obsolescence, with minimal up front hassle.

ErikKaffehr

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 11311
    • Echophoto
Re: DNG (from original .tiff) and CO8
« Reply #7 on: July 19, 2015, 05:08:55 pm »

Hi,

Just keep in mind that TIFFS are not containing all info from raws. Tiffs are demosaiced. Also, TIFFs are three to six times the size of raws.

Best regards
Erik

In general:
- leave your raws as they come out of the camera
- process your very favorite raws to TIFF and keep the TIFF

IMO the above two simple recommendations represent the greatest chance of avoiding hassle/loss as a result of obsolescence, with minimal up front hassle.
Logged
Erik Kaffehr
 

ErikKaffehr

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 11311
    • Echophoto
Re: DNG (from original .tiff) and CO8
« Reply #8 on: July 20, 2015, 12:06:02 am »

Hi,

If the Raw images are embedded in the DNG-file they can be extracted. I use DNG normally, and whenever I want to use a DNG file with Capture One I extract it.

The raw files are fully recoverable by that method. The original files (in my case IIQ) have the same MD5 checksum as the extracted files. So if you checked the checkbox to embed original raw into the DNG, your problems are solved.

Best regards
Erik

It would probably take the Adobe DNGconverter (version 9.1 currently), to reverse the embedding of the original camera Raw in the DNG, assuming it was originally fully embedded.

Here are the Windows versions for download, and here for the Mac.

Cheers,
Bart
Logged
Erik Kaffehr
 

brandon

  • Jr. Member
  • **
  • Offline Offline
  • Posts: 80
Re: DNG (from original .tiff) and CO8
« Reply #9 on: July 20, 2015, 12:18:29 am »

Thanks for the advice (including trying most updated DNG converter to recover the embedded raw files) and opinions. I use both LR and CO and currently the native raws work best for both, and Im not about to drop CO because it wont work with all aspects of DNG (which is as Bart notes is complex to guarantee forward compatibility with all features). Maybe Im wrong in my logic here but keeping the native raws (of any camera) and copies of DNG converter programs, if and only if, the native raw loses support gowing forward and only then converting to DNG is a better path forward than DNG converting. (simpler to keep some DNG converter applications in the download folder if needed as an "insurance" policy). Esparanto was supposed to be an "open source" universal language that didnt really unify the world, but not all was lost.....
Logged

David Grover / Capture One

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1324
    • Capture One
Re: DNG (from original .tiff) and CO8
« Reply #10 on: July 31, 2015, 09:08:30 am »

Thanks for the advice (including trying most updated DNG converter to recover the embedded raw files) and opinions. I use both LR and CO and currently the native raws work best for both, and Im not about to drop CO because it wont work with all aspects of DNG (which is as Bart notes is complex to guarantee forward compatibility with all features). Maybe Im wrong in my logic here but keeping the native raws (of any camera) and copies of DNG converter programs, if and only if, the native raw loses support gowing forward and only then converting to DNG is a better path forward than DNG converting. (simpler to keep some DNG converter applications in the download folder if needed as an "insurance" policy). Esparanto was supposed to be an "open source" universal language that didnt really unify the world, but not all was lost.....

Just to mention, Capture One 8.3.2 supports DNG version 1.4 now.

However, I agree it is far simpler to stick with the manufacturers RAW file.
Logged
David Grover
Business Support and Development Manager

Bart_van_der_Wolf

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 8914
Re: DNG (from original .tiff) and CO8
« Reply #11 on: July 31, 2015, 10:17:39 am »

Just to mention, Capture One 8.3.2 supports DNG version 1.4 now.

However, I agree it is far simpler to stick with the manufacturers RAW file.

Hi David,

I just tried a conversion of an EOS-5DS R raw file, and compared it to a DNG converted version of it. With exactly the same controls settings, the Whitebalance is significantly different. The color after re-Whitebalancing on neutral grey, still shows color differences from the same source file, the only difference is that the files are the original Camera Raw and a DNG converted file version.

I do not know if that has anything to do with the 1.4 update, or that there are DNG interpretation issues (or something else?), but it's not what one would expect. When I look at the file metadata tool panel in C1 8.32, I do see that the "Vendor Specific, White balance" field is empty in the DNG version, but I have set a manual WB after importing the files so that should not be relevant, I think?

It's a bit puzzling, although I'm not affected because I do not convert to DNG, but still...
I know it is just one file, but maybe your service desk has more reports about differences?

I also read on the Capture One forum a reply by Paul Steunebrink where he confirmed that Capture One doesn't play nice on DNGs, and another post where he says that DNG (in that case linear DNGs produced by DxO, they may have made an error or its due to C1, dunno) is not as straightforward as Adobe wants us to believe.

So what's up with DNG support in Capture One (in case one made the mistake to toss the original Raws)? Anything specific to look out for? Or is there a general difficulty with C1 conversions from DNG. It's a general question, not a complaint from me (because I'm personally not affected), but I'd like to be able and give proper advice to clients (preferably without having to test all possible scenarios for them).

Cheers,
Bart
Logged
== If you do what you did, you'll get what you got. ==

AlterEgo

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1995
Re: DNG (from original .tiff) and CO8
« Reply #12 on: July 31, 2015, 11:58:03 am »

Hi David,

I just tried a conversion of an EOS-5DS R raw file, and compared it to a DNG converted version of it. With exactly the same controls settings, the Whitebalance is significantly different. The color after re-Whitebalancing on neutral grey, still shows color differences from the same source file, the only difference is that the files are the original Camera Raw and a DNG converted file version.

I do not know if that has anything to do with the 1.4 update, or that there are DNG interpretation issues (or something else?), but it's not what one would expect. When I look at the file metadata tool panel in C1 8.32, I do see that the "Vendor Specific, White balance" field is empty in the DNG version, but I have set a manual WB after importing the files so that should not be relevant, I think?

It's a bit puzzling, although I'm not affected because I do not convert to DNG, but still...
I know it is just one file, but maybe your service desk has more reports about differences?

I also read on the Capture One forum a reply by Paul Steunebrink where he confirmed that Capture One doesn't play nice on DNGs, and another post where he says that DNG (in that case linear DNGs produced by DxO, they may have made an error or its due to C1, dunno) is not as straightforward as Adobe wants us to believe.

So what's up with DNG support in Capture One (in case one made the mistake to toss the original Raws)? Anything specific to look out for? Or is there a general difficulty with C1 conversions from DNG. It's a general question, not a complaint from me (because I'm personally not affected), but I'd like to be able and give proper advice to clients (preferably without having to test all possible scenarios for them).

Cheers,
Bart

the same camera profiles used for both DNG and non DNG ?
Logged

Bart_van_der_Wolf

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 8914
Re: DNG (from original .tiff) and CO8
« Reply #13 on: July 31, 2015, 12:10:12 pm »

the same camera profiles used for both DNG and non DNG ?

Yes, I've manually duplicated all settings, one-by-one, because perhaps not everything is automatically copied over despite me specifying that it should. That of course is just the user interface, and I do not know if under the hood it is actually picked up. There might also be something that's related to GPU acceleration, which I have not yet switched off, or to last night's graphics drivers update of the nVidia controller/GPU.

I have also not tried an older DNG converter, or an older DNG standard <1.4, or maybe the 9.1.1 version developed a glitch. Many possibilities, but it's odd for sure.

Cheers,
Bart
Logged
== If you do what you did, you'll get what you got. ==

digitaldog

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 20646
  • Andrew Rodney
    • http://www.digitaldog.net/
Re: DNG (from original .tiff) and CO8
« Reply #14 on: July 31, 2015, 03:55:48 pm »

So what's up with DNG support in Capture One (in case one made the mistake to toss the original Raws)?

http://forum.luminous-landscape.com/index.php?topic=102466.msg841266#msg841266

Quote
is not as straightforward as Adobe wants us to believe.
As you know by now, it's straightforward for at least four different products, sadly for C1, it's not straightforward for their engineers.
A one man shop (Brian Griffith / Iridient Digital) had no such 'issues'.

Im not interested in the "politics" of it all, just what way I can utilise all the original file information within CO8 in developing these images.
Phase One's Capture One simply doesn't provide full (correct) support for DNG.
« Last Edit: July 31, 2015, 04:02:01 pm by digitaldog »
Logged
http://www.digitaldog.net/
Author "Color Management for Photographers".

digitaldog

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 20646
  • Andrew Rodney
    • http://www.digitaldog.net/
Re: DNG (from original .tiff) and CO8
« Reply #15 on: July 31, 2015, 04:42:23 pm »

The other fact is that many other raw developers support DNG very well. The ones I have used are RawTherapy, AccuRaw, Iridient Raw Developer and of course LR/ACR.
To add to that list (which we should continue to compile), add Raw Photo Processor 64 to those that correctly deal with DNG. I tested it, LR/ACR and Iridient Developer and like you, found identical pixel for pixel renderings from a CR2 and a DNG converted (in LR) from that original.
Logged
http://www.digitaldog.net/
Author "Color Management for Photographers".

ErikKaffehr

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 11311
    • Echophoto
Re: DNG (from original .tiff) and CO8
« Reply #16 on: August 01, 2015, 01:20:45 am »

Hi,

My understanding is that Capture One does not properly support DNG as a whole.

Part of that is, I would think, that DNG is also includes a recipe for raw conversion, like compromise matrices and colour space conversions. Capture One has a different pipeline, and I don't think they use the information in the DNG Colour Profiles.

Personally, I would say it is a great pity. I would say a standard for raw images is badly needed, not least for permanence, who nows which file formats will be supported in 20-30 years. Some file formats my have less permanence than the media they have been written on.

Best regards
Erik

http://forum.luminous-landscape.com/index.php?topic=102466.msg841266#msg841266
As you know by now, it's straightforward for at least four different products, sadly for C1, it's not straightforward for their engineers.
A one man shop (Brian Griffith / Iridient Digital) had no such 'issues'.
 Phase One's Capture One simply doesn't provide full (correct) support for DNG.
Logged
Erik Kaffehr
 

Schewe

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 6229
    • http:www.schewephoto.com
Re: DNG (from original .tiff) and CO8
« Reply #17 on: August 01, 2015, 05:19:24 am »

My understanding is that Capture One does not properly support DNG as a whole.

Correct...at best C1 attempts to simulate DNG compatibility but hasn't really embraced DNG. If the newest version on C1 claims to be completely compatible with DNG spec 1.4, that's a welcome change...in the past C1 has only supported an older 1.3 SDK which allowed them to skirt several issues. Would be nice is Phase One stated that C1 embraced DNG 1.4.
Logged

Bart_van_der_Wolf

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 8914
Re: DNG (from original .tiff) and CO8
« Reply #18 on: August 01, 2015, 07:56:55 am »

Correct...at best C1 attempts to simulate DNG compatibility but hasn't really embraced DNG. If the newest version on C1 claims to be completely compatible with DNG spec 1.4, that's a welcome change...in the past C1 has only supported an older 1.3 SDK which allowed them to skirt several issues. Would be nice is Phase One stated that C1 embraced DNG 1.4.

Hi Jeff,

I agree it would be nice but, although the release notes for Capture One version 8.3 already mentioned DNG 1.4 support, as far as I can see there are still issues in e.g. color rendering, which is a bit surprising, as we are now at version 8.3.2. So support may be quite basic like that the files can be opened without issues, but not that all features are equally usable, if at all, or there are other reasons..

In another recent LuLa thread, Doug Peterson confirmed that C1 "doesn't provide full support for DNG", but that could mean many things. Not supporting Adobe DCP and lens correction data e.g. would be obvious candidates, because Capture one uses ICC camera profiles and their own lens correction data (to accommodate Tilt/Shift lenses and extreme wide-angle lenses with resulting Color Cast effects, and optional noise reduction that is tuned per camera model.

The fact that color rendering from the same file (as original camera Raw and as DNG conversion) with the same controls settings is so different is puzzling. I think it would offer Capture One a larger chance of having Adobe users switch to Capture One for high quality conversions, than the other way around.

Cheers,
Bart
Logged
== If you do what you did, you'll get what you got. ==

HansKoot

  • Newbie
  • *
  • Offline Offline
  • Posts: 20
    • HKPHOTO
Re: DNG (from original .tiff) and CO8
« Reply #19 on: August 01, 2015, 08:32:22 am »

well, I am a newbie with CO, a one month of trial :-), but have experience with Lightroom. My impression is that capture one does a bit better job with dng files in general, and when the conditions are hard it can make a significant difference.
See my previously posted file of a night shot in Argentine at which nobody answered... The fact is that the 5d2 raw file was in 2009 converted to dng (no embedded camera raw).
Links in the insert lightroom.
Logged
"Its better to create something that others criticize than to create nothing and criticize others" (Ricky Gervais)
Pages: [1] 2   Go Up