Pages: [1] 2   Go Down

Author Topic: DNG for everyone  (Read 19910 times)

Martin Ocando

  • Full Member
  • ***
  • Offline Offline
  • Posts: 112
    • Lightcraftings
DNG for everyone
« on: February 09, 2012, 11:20:26 pm »

So, it appears DNG might be a better option after all. Will like to hear from Jeff, Andrew and Eric about this.

http://news.cnet.com/8301-17938_105-57371809-1/adobe-offering-new-reasons-to-get-dng-religion/
Logged
Martin Ocando
Olympus OM-D E-M1 -

Schewe

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 6229
    • http:www.schewephoto.com
Re: DNG for everyone
« Reply #1 on: February 09, 2012, 11:31:57 pm »

So, it appears DNG might be a better option after all. Will like to hear from Jeff, Andrew and Eric about this.

http://news.cnet.com/8301-17938_105-57371809-1/adobe-offering-new-reasons-to-get-dng-religion/


Mike and I cover the new DNG options in the LR4 Advanced vids...but yes, Adobe (read Thomas/Eric) have advanced DNG with the most recent SDK. You can add "Fast Load" when creating a DNG which speeds opening the raw file in LR & ACR. You can also create Lossey Compressed DNG files which are linear DNGs with compression and in LR4 you can downsample a DNG based on megapixel sizes or long edges...which means you can make a DNG that is TINY by comparison to the original and maintain the ability to make ACR/LR adjustments. An IQ 180 file that is 80MP (and about 80 MBs) can be lossey compressed and downsamped to 10 MP and end up just under 3 MBs.

So, DNG continues to advance...also note that Adobe has submitting DNG to the ISO for an upcoming TIFF-EP file format update...if/when that happens, there will be a lot less reason for NOT using DNG by the major camera makers like Nikon/Canon. At this point it's really a political question not a technical question...
Logged

Martin Ocando

  • Full Member
  • ***
  • Offline Offline
  • Posts: 112
    • Lightcraftings
Re: DNG for everyone
« Reply #2 on: February 09, 2012, 11:42:09 pm »

Mike and I cover the new DNG options in the LR4 Advanced vids...but yes, Adobe (read Thomas/Eric) have advanced DNG with the most recent SDK. You can add "Fast Load" when creating a DNG which speeds opening the raw file in LR & ACR. You can also create Lossey Compressed DNG files which are linear DNGs with compression and in LR4 you can downsample a DNG based on megapixel sizes or long edges...which means you can make a DNG that is TINY by comparison to the original and maintain the ability to make ACR/LR adjustments. An IQ 180 file that is 80MP (and about 80 MBs) can be lossey compressed and downsamped to 10 MP and end up just under 3 MBs.

So, DNG continues to advance...also note that Adobe has submitting DNG to the ISO for an upcoming TIFF-EP file format update...if/when that happens, there will be a lot less reason for NOT using DNG by the major camera makers like Nikon/Canon. At this point it's really a political question not a technical question...

Awesome. Just for the speed and size improvements I'll be an early adopter. I think camera makers have no real reason/advantage, economical or otherwise for not adopting DNG natively. I mean, Adobe is not making any money because of the format, is good for them, because they can streamline and optimize their software for it, is good for us, because it gives additional advantages proprietary RAW formats does not provide, and is good for the camera makers, because they won't have to dedicate any time or money on adjusting their own formats to fit newer sensors. The key, I think, is on the demosaicing, which I believe is very specific to the maker and sensor, unless I'm mistaken.
Logged
Martin Ocando
Olympus OM-D E-M1 -

Schewe

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 6229
    • http:www.schewephoto.com
Re: DNG for everyone
« Reply #3 on: February 10, 2012, 12:04:39 am »

The key, I think, is on the demosaicing, which I believe is very specific to the maker and sensor, unless I'm mistaken.

No...actually it's specific to the raw processor. ACR and LR at equal version are the same. DxO does a linear (demosiaced) DNG which works pretty well even though DxO's demosiacing isn't...various apps such as Capture One can save out DNG (but is limited to the DNG SDK they are using).

There are two base flavors of DNG; raw DNG which is un-demosiaced raw files converted to a DNG wrapper and linear DNG which are locked into a DNG demosiacing but still in a linear gamma (with ProPhoto RGB colors). Either DNG flavor works well in ACR/LR. There is another flavor of linear DNG, a file that started in a gamma encoded color space such as sRGB or Adobe RGB. So, a JPEG opened into ACR'LR will be limited because the file started life in a non-linear gamma linear. Still, editing a JPEG in ACR/LR is still better than editing in say Photoshop.

The new additions to DNG are the added Fast Load which can be applied to raw or linear DNGs and the lossey compressed or downsampled DNG which are both linear (demosaiced) DNGs.

I suspect we'll see some cameras come with lossey DNG built in as a replacement for JPEGs–the file size reduction can be important.

Personally, I still use the proprietary raw files in Lightroom because I find it easier to use sidecar .xmp files in LR. The .xmp files are tiny compared to the DNG files with the XMP built in. So if you change a DNG processing settings, the whole DNG has to be backed up based on modification date changes. A proprietary raw file with a sidecar only needs to write the tiny sidecar file on backup.

I use DNG for any file that might be an interchange file (handing off to another user). I can embed IPTC metadata as well as snapshots of various image settings. Wit raw files those live only in the .xmp files which can be lost.
Logged

Martin Ocando

  • Full Member
  • ***
  • Offline Offline
  • Posts: 112
    • Lightcraftings
Re: DNG for everyone
« Reply #4 on: February 10, 2012, 12:25:54 am »

No...actually it's specific to the raw processor. ACR and LR at equal version are the same. DxO does a linear (demosiaced) DNG which works pretty well even though DxO's demosiacing isn't...various apps such as Capture One can save out DNG (but is limited to the DNG SDK they are using).

There are two base flavors of DNG; raw DNG which is un-demosiaced raw files converted to a DNG wrapper and linear DNG which are locked into a DNG demosiacing but still in a linear gamma (with ProPhoto RGB colors). Either DNG flavor works well in ACR/LR. There is another flavor of linear DNG, a file that started in a gamma encoded color space such as sRGB or Adobe RGB. So, a JPEG opened into ACR'LR will be limited because the file started life in a non-linear gamma linear. Still, editing a JPEG in ACR/LR is still better than editing in say Photoshop.

The new additions to DNG are the added Fast Load which can be applied to raw or linear DNGs and the lossey compressed or downsampled DNG which are both linear (demosaiced) DNGs.

I suspect we'll see some cameras come with lossey DNG built in as a replacement for JPEGs–the file size reduction can be important.

Personally, I still use the proprietary raw files in Lightroom because I find it easier to use sidecar .xmp files in LR. The .xmp files are tiny compared to the DNG files with the XMP built in. So if you change a DNG processing settings, the whole DNG has to be backed up based on modification date changes. A proprietary raw file with a sidecar only needs to write the tiny sidecar file on backup.

I use DNG for any file that might be an interchange file (handing off to another user). I can embed IPTC metadata as well as snapshots of various image settings. Wit raw files those live only in the .xmp files which can be lost.

Go it. Gotta start using sidecar .xmp files for backup.

So it will still be better, for the time being, to keep our undemosaiced DNGs, or proprietary RAWs, since newer demosaicing algorithms might come out and we can reprocess our RAWs and take advantage of them. Am I right ?

What about when the camera supports DNG natively. It will still have to update the whole DNG when backing up, or you can attach an .xmp sidecar to a DNG ?
Logged
Martin Ocando
Olympus OM-D E-M1 -

Schewe

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 6229
    • http:www.schewephoto.com
Re: DNG for everyone
« Reply #5 on: February 10, 2012, 12:55:00 am »

So it will still be better, for the time being, to keep our undemosaiced DNGs, or proprietary RAWs, since newer demosaicing algorithms might come out and we can reprocess our RAWs and take advantage of them. Am I right ?

More or less...yes. The thing about raw processing is we still keep seeing improvements as time goes by. Which means newer processing is likely better than older processing. Fact of life...some things keep getting better.

Quote
What about when the camera supports DNG natively. It will still have to update the whole DNG when backing up, or you can attach an .xmp sidecar to a DNG ?

There's no easy way of setting a DNG NOT to update the XMP metadata internally. You can lock the DNG which should force LR/ACR to pop a .xmp file but it gets complicated when dealing with DNGs...there a preference in ACR (not sure about LR) to ignore .xmp files which you would need to set to off.
Logged

wolfnowl

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 5824
    • M&M's Musings
Re: DNG for everyone
« Reply #6 on: February 10, 2012, 02:17:58 am »

As I understand it, the default in Lightroom is to have 'autowrite changes to .xmp' turned off.  That means that any metadata added and any history settings are stored only in the catalog.  You can update the metadata in the .xmp file (or the .dng, .tif, .jpg, or .psd) by pressing Ctrl/Cmd-S.

Mike.
Logged
If your mind is attuned t

john beardsworth

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 4755
    • My photography site
Re: DNG for everyone
« Reply #7 on: February 10, 2012, 04:10:47 am »

You need to be very careful to distinguish between lossy DNGs and full-blooded ones. If not, it would be very easy to create lossy DNGs for some reason like taking on the road or saving disc space by compressing frames shot for timelapses, and then confuse them with the real negatives, even overwrite non-lossy DNGs.

A couple of features in Lightroom 4 are helpful. Library's filter panel's File Type now distinguishes between the different types of DNG, and you can examine individual images in the Metadata panel's DNG view.

But outside Lightroom, you may not be able to see them at all, and in Explorer / Finder the only indications that you've a group of lossy DNGs may be file size and that the thumbnail doesn't resolve (at least until an OS update).

I have no real reason to create lossy DNGs, so I'm not doing so. If I did so, then I would be taking extra steps to prevent confusion - probably adding a filename suffix to every lossy DNG. Whatever app I am using, I want to distinguish between the full blooded DNG "120210_0123 Blea Tarn.dng" and its lossy counterpart "120210_0123 Blea Tarn LSS.dng".

I'm not saying there isn't some progress, and I welcome fast load (even if I can't see any speed difference) and I like that the small number of JPEGs I shoot with my iPad can now be saved as a sensible-size DNG. But I do feel we shouldn't dive blindly into these waters. We've got the new capabilities, they do present their risks to the integrity of our picture collections, and have we got sufficient workflow benefits? For me, no, not yet. You?
Logged

Alan Goldhammer

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 4344
    • A Goldhammer Photography
Re: DNG for everyone
« Reply #8 on: February 10, 2012, 08:29:41 am »


I suspect we'll see some cameras come with lossey DNG built in as a replacement for JPEGs–the file size reduction can be important.

Personally, I still use the proprietary raw files in Lightroom because I find it easier to use sidecar .xmp files in LR. The .xmp files are tiny compared to the DNG files with the XMP built in. So if you change a DNG processing settings, the whole DNG has to be backed up based on modification date changes. A proprietary raw file with a sidecar only needs to write the tiny sidecar file on backup.

I wish Nikon would give us the option of using DNG vs NEF.  I really don't know any Nikon user that uses Nikon software to process NEF files.  On the backup issue, you raise an important point for those of use using cloud storage as a backup.  It's a lot faster to back up changed .xmp files than a DNG file that incorporates the process directions.
Logged

john beardsworth

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 4755
    • My photography site
Re: DNG for everyone
« Reply #9 on: February 10, 2012, 09:41:58 am »

The "need" to back up the whole DNG after processing changes is dubious. At best, it's a second rate backup that omits  a good chunk of your Lightroom work. Backup the DNG upon its creation, then backup your catalogue, and you've backed up all your work.
Logged

digitaldog

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 20630
  • Andrew Rodney
    • http://www.digitaldog.net/
Re: DNG for everyone
« Reply #10 on: February 10, 2012, 09:47:03 am »

You need to be very careful to distinguish between lossy DNGs and full-blooded ones. If not, it would be very easy to create lossy DNGs for some reason like taking on the road or saving disc space by compressing frames shot for timelapses, and then confuse them with the real negatives, even overwrite non-lossy DNGs.

Exactly, good example. I’m excited by the new possibilities with DNG and the fast load is huge. But I think it is very important end users understand what John is saying here and not assume that just because you have a DNG, all variants are equal.

As for the article, everything seems good but I don’t really like the title.
Logged
http://www.digitaldog.net/
Author "Color Management for Photographers".

john beardsworth

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 4755
    • My photography site
Re: DNG for everyone
« Reply #11 on: February 10, 2012, 09:54:16 am »

What, you don't think we should argue such things on rational grounds, do you? Sheesh.
Logged

Alan Goldhammer

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 4344
    • A Goldhammer Photography
Re: DNG for everyone
« Reply #12 on: February 10, 2012, 11:52:10 am »

The "need" to back up the whole DNG after processing changes is dubious. At best, it's a second rate backup that omits  a good chunk of your Lightroom work. Backup the DNG upon its creation, then backup your catalogue, and you've backed up all your work.
John, maybe I'm a bit confused then about the DNGs.  I thought that the process changes that are in .xmp files associated with the RAW are contained in the DNG if one is using that format (I don't do this right now, sticking with RAW + .xmp.  From what you are implying is that there is redundancy in the processing directions.

Alan
Logged

digitaldog

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 20630
  • Andrew Rodney
    • http://www.digitaldog.net/
Re: DNG for everyone
« Reply #13 on: February 10, 2012, 11:55:14 am »

John, maybe I'm a bit confused then about the DNGs.  I thought that the process changes that are in .xmp files associated with the RAW are contained in the DNG if one is using that format (I don't do this right now, sticking with RAW + .xmp.  From what you are implying is that there is redundancy in the processing directions.

John’s point is quite valid and about the only disadvantage I see to a DNG workflow (although IMHO not a big one).

Suppose you have a DNG that contains all this useful data. You simply make a tiny update like altering some metadata. The entire DNG has to be backed up. It takes a lot longer than updating a small XMP sidecar file. I don’t think there is any way around this. You have a single container of a document and if something changes and you want to back it up, you back up the whole thing.

I do this unattended at night using a script so it doesn’t impact my time. But it is something to consider.
Logged
http://www.digitaldog.net/
Author "Color Management for Photographers".

Schewe

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 6229
    • http:www.schewephoto.com
Re: DNG for everyone
« Reply #14 on: February 10, 2012, 12:01:56 pm »

You need to be very careful to distinguish between lossy DNGs and full-blooded ones.

Very true...there is a good thread on the LR4 beta forums talking about this issue. Anybody interested should check it out. Eric Chan has taken the time to give extensive answers to a lot of questions. Info on lossy DNG?

In addition to the JPEG compression, it turns out that lossey compressed DNGs are also mapped to a different tone curve and are not linear. So, it treats the lossey compressed DNG image data more like it treats a JPEG or TIFF that is turned into a DNG.

I think great care and experimenting should be done before people jump into the waters of mass conversion of raw files into lossey compressed files.
Logged

john beardsworth

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 4755
    • My photography site
Re: DNG for everyone
« Reply #15 on: February 10, 2012, 01:53:49 pm »

The entire DNG has to be backed up.
My point is that it does not, unless you want a level of redundancy. The DNG has only had some of your Lightroom work saved back to it, so if you do choose to make this backup you are only getting a second rate level of comfort anyway. On the other hand, if you backup the original DNG when it's created, and routinely backup the catalogue, then 100% of your work is covered.

Logged

digitaldog

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 20630
  • Andrew Rodney
    • http://www.digitaldog.net/
Re: DNG for everyone
« Reply #16 on: February 10, 2012, 02:02:15 pm »

My point is that it does not, unless you want a level of redundancy. The DNG has only had some of your Lightroom work saved back to it, so if you do choose to make this backup you are only getting a second rate level of comfort anyway. On the other hand, if you backup the original DNG when it's created, and routinely backup the catalogue, then 100% of your work is covered.

In my case, I just back up the entire drive that has the catalog, DNG etc. So what you are saying is, just backup the catalog. The only other issue is, as I add new DNG’s to this drive, I’m not sure how easy it would be to tell my simple backup app (SuperDuper), just back up DNG’s that didn’t exist the last time you backed up. Now if only new DNGs are added, no issue. But if 100 new DNG’s were added, plus 5 were edited, then SD is going to update 105 DNGs. Since as I said, this is done unattended, I guess it isn’t a big deal.

And I do love back up redundancy.
Logged
http://www.digitaldog.net/
Author "Color Management for Photographers".

Alan Goldhammer

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 4344
    • A Goldhammer Photography
Re: DNG for everyone
« Reply #17 on: February 10, 2012, 02:29:10 pm »

My point is that it does not, unless you want a level of redundancy. The DNG has only had some of your Lightroom work saved back to it, so if you do choose to make this backup you are only getting a second rate level of comfort anyway. On the other hand, if you backup the original DNG when it's created, and routinely backup the catalogue, then 100% of your work is covered.


This still does not answer my question above.  If I convert to DNG upon import and then do LR processing, where are these process changes being stored (this in the absence of .xmp files)?  Are they in the catalog or the DNG?  My understanding of the catalog is that its a database pointing to the location of files but maybe there is more to it.
Logged

john beardsworth

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 4755
    • My photography site
Re: DNG for everyone
« Reply #18 on: February 10, 2012, 02:32:31 pm »

Sorry, Alan, in the catalog. The writing to the DNG is optional, like saving xmp sidecars to your NEFs.
Logged

Alan Goldhammer

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 4344
    • A Goldhammer Photography
Re: DNG for everyone
« Reply #19 on: February 10, 2012, 02:39:47 pm »

Sorry, Alan, in the catalog. The writing to the DNG is optional, like saving xmp sidecars to your NEFs.
You beat me to it.  I found the discussion in Martin Evening's book.  Thanks!  I'm still on the fence with regard to DNGs.
Logged
Pages: [1] 2   Go Up