Luminous Landscape Forum

Raw & Post Processing, Printing => Digital Image Processing => Topic started by: PeterAit on October 07, 2018, 10:54:29 am

Title: DNG puzzle
Post by: PeterAit on October 07, 2018, 10:54:29 am
I have never used DNG files before today when I had the occasion to export some from LR. The original files were Sony RAWs which tend to be in the 80 MB range. Yet the exported DNGs were all 37-45 MB. It's my understanding that the DNG format preserves all the info in the RAW, so how can I get a 50% file size reduction? Thx.
Title: Re: DNG puzzle
Post by: digitaldog on October 07, 2018, 10:58:34 am
I have never used DNG files before today when I had the occasion to export some from LR. The original files were Sony RAWs which tend to be in the 80 MB range. Yet the exported DNGs were all 37-45 MB. It's my understanding that the DNG format preserves all the info in the RAW, so how can I get a 50% file size reduction? Thx.
A. There are options for lossy or lossless conversions so check your settings.
B. It's to be expected that the DNG when ideally converted is smaller in size than the original in many cases, due to better compression. It usually isn't as dramatic as you report so do check on A above and your settings for conversion,.
Title: Re: DNG puzzle
Post by: Frans Waterlander on October 07, 2018, 12:44:51 pm
Be aware that DNG discards proprietary camera information. Opinions vary on how much of an issue that is. DNG hasn't taken off like it was supposed to and now several folks recommend to leave it alone.
Title: Re: DNG puzzle
Post by: digitaldog on October 07, 2018, 12:48:19 pm
Be aware that DNG discards proprietary camera information. Opinions vary on how much of an issue that is. DNG hasn't taken off like it was supposed to and now several folks recommend to leave it alone.
Be aware it's proprietary and useless in any raw converter other than the manufacturers. If you understand what proprietary actually means.
Opinions are like ass*oles, everyone has them even when they are based on misunderstandings of technology.
IF the questions arise, what are the advantages of a DNG workflow, some here can provide that data point but the OP isn't asking that (yet) but rather about why his DNGs are smaller in size than his proprietary raws. He knows that's to be expected (one advantage of the format, a small but pertinent one).
As for DNG not taking off, please provide a lick of data to back up who and how many use it vs. those that do not. You may be correct but you may just be making stuff up.
You are of course entitled to your own opinions but not your own facts again.  ;)
Title: Re: DNG puzzle
Post by: Frans Waterlander on October 07, 2018, 01:55:37 pm
Yes, I actually know what proprietary actually means, actually.

DNG was supposed to become the de-facto standard format and it hasn't; that's a fact. Many people now recommend not to use it; that's a fact; I researched this issue quite extensively.
Title: Re: DNG puzzle
Post by: digitaldog on October 07, 2018, 01:59:57 pm
DNG was supposed to become the de-facto standard format and it hasn't; that's a fact. Many people now recommend not to use it; that's a fact; I researched this issue quite extensively.
Many recommend DNG too. I researched it!  :P
http://digitaldog.net/files/ThePowerofDNG.pdf
https://www.cnet.com/news/adobe-offering-new-reasons-to-get-dng-religion/
Attempt to stay OT! The question wasn’t about not using this file format but about file size of the converted DNG: read the question again! And the correct answer.
Title: Re: DNG puzzle
Post by: Frans Waterlander on October 07, 2018, 02:33:27 pm
Many recommend DNG too. I researched it!  :P
http://digitaldog.net/files/ThePowerofDNG.pdf
https://www.cnet.com/news/adobe-offering-new-reasons-to-get-dng-religion/
Attempt to stay OT! The question wasn’t about not using this file format but about file size of the converted DNG: read the question again! And the correct answer.

One undated article and one dated 2012. And these demonstrate DNG has become the de-facto standard and that many today not recommend using it?
Title: Re: DNG puzzle
Post by: digitaldog on October 07, 2018, 02:51:28 pm
One undated article and one dated 2012. And these demonstrate DNG has become the de-facto standard and that many today not recommend using it?
Why is your posting goals here to get threads locked down and post off topic rubbish?
The 2012 article (and the 2nd) is as pertinent today as it was when published. Your political need to infuse your political opinions san's facts about DNG is utterly off topic.
Why don't you start your own post here, or another duplicate on PhotoNet entitled "I'm an self described expert in file formats and why I don't recommend using DNG". Then you'll be on topic and those of us that do not desire to read political topics can ignore it.
The OP asked a question you've failed to address and the question was answered before you arrived. So I can only suspect once again, based on your posting history in multiple forums, that your goal is to get Jeremy to lock another thread you’ve posted in. I don't see the goal so please, PLEASE stop and if you must post here, write something that's about the topic of the DNG puzzle about size. Assuming you have any knowledge, after all your extensive research (when, published where?) that might provide a lick of useful data to answer the specific question asked here. 


Title: Re: DNG puzzle
Post by: Frans Waterlander on October 07, 2018, 03:30:20 pm
The OP used DNG for the first time today, so I give him a heads-up. Any problem with that, Andrew?
Title: Re: DNG puzzle
Post by: digitaldog on October 07, 2018, 03:32:37 pm
The OP used DNG for the first time today, so I give him a heads-up. Any problem with that, Andrew?
What part of off topic is a foreign concept to you? No need to attempt an answer....
Title: Re: DNG puzzle
Post by: digitaldog on October 07, 2018, 03:40:10 pm
In an attempt to provide actual data to actually further answer a question with visual specifics, this is for Peter. Notice the fly-out comments about the options for lossy/lossless:
Quote
"The difference between genius and stupidity is that genius has its limits." - Albert Einstein
So true, so very true.
Now another advantage of DNG in LR: Fast Load data!
Title: Re: DNG puzzle
Post by: Frans Waterlander on October 07, 2018, 04:54:43 pm
What part of off topic is a foreign concept to you? No need to attempt an answer....

To somebody just now starting to look into DNG, no, it's not off-topic to give a heads-up about its drawbacks. I don't want people to start using DNG without knowing the drawbacks. Switching to DNG looked like the right thing to do many year ago, but now I regret having thousands of files in that format. Better safe than sorry.
Title: Re: DNG puzzle
Post by: digitaldog on October 07, 2018, 05:10:03 pm
OT posts continue to be mysterious concept for you. Lack of attention. That’s a drawback.
Title: Re: DNG puzzle
Post by: john beardsworth on October 07, 2018, 05:33:16 pm
Be aware that DNG discards proprietary camera information.

Nope, it's preserved in an unparsed metadata block.
Title: Re: DNG puzzle
Post by: Jeremy Roussak on October 07, 2018, 05:38:26 pm
To somebody just now starting to look into DNG, no, it's not off-topic to give a heads-up about its drawbacks. I don't want people to start using DNG without knowing the drawbacks.

Yes, of course it is off topic. Your posts have contributed nothing. Peter wasn't asking for advice on whether to start using DNG; he had a very specific query about file sizes and you've not helped to answer it by creating an argument about the advisably of converting images to that format.

Try to stay relevant.

Jeremy
Title: Re: DNG puzzle
Post by: digitaldog on October 07, 2018, 05:40:35 pm
Nope, it's preserved in an unparsed metadata block.
Indeed there are provisions for private metadata, good call.
But keep in my some FUD that’s OT is from someone who this week alone didn’t know the difference between a bug and a feature in ACR found in Adobe Elements. That post had to be shut down too.
Title: Re: DNG puzzle
Post by: Telecaster on October 07, 2018, 05:47:36 pm
Here's an actual answer to Peter's question. Sony offers two Raw file options: uncompressed and loss-y (that is, some info is thrown away) compressed. The uncompressed Raws are very big while the compressed ones are much smaller but somewhat compromised. DNG, however, offers a third option: loss-less compressed. It's compression scheme is pretty efficient, thus the size reduction.

Geezus!

-Dave-
Title: Re: DNG puzzle
Post by: john beardsworth on October 07, 2018, 05:54:17 pm
Yes, there is a sense of "crying wolf" a bit too often.

As for the original question, I think it's worth saying that DNG sometimes produces a smaller file size, sometimes not, and it is mostly dependent on the efficiency of the camera's compression. Some people  see this as a reason for adopting DNG, but it's better to see it as only a by-product (and not pay much attention to it).
Title: Re: DNG puzzle
Post by: digitaldog on October 07, 2018, 05:54:51 pm
Here's an actual answer to Peter's question. Sony offers two Raw file options: uncompressed and loss-y (that is, some info is thrown away) compressed. The uncompressed Raws are very big while the compressed ones are much smaller but somewhat compromised. DNG, however, offers a third option: loss-less compressed. It's compression scheme is pretty efficient, thus the size reduction.

Geezus!

-Dave-
Sounds quite similar to post #2  :)
Title: Re: DNG puzzle
Post by: Frans Waterlander on October 07, 2018, 05:56:44 pm
Yes, of course it is off topic. Your posts have contribute nothing. Peter wasn't asking for advice on whether to start using DNG; he had a very specific query about file sizes and you've not helped to answer it by creating an argument about the advisably of converting images to that format.

Try to stay relevant.

Jeremy

Giving a newbie to the issue a heads-up about the possible drawbacks is not OK? My posts have contributed nothing? Are we now only allowed to strictly answer a question and not digress in any way, shape or form, even if it is intended to be helpful? Good grief!
Title: Re: DNG puzzle
Post by: Frans Waterlander on October 07, 2018, 05:59:17 pm
Indeed there are provisions for private metadata, good call.
But keep in my some FUD that’s OT is from someone who this week alone didn’t know the difference between a bug and a feature in ACR found in Adobe Elements. That post had to be shut down too.

And that from someone who this week alone didn't know the difference between using a database and using sidecars.
Title: Re: DNG puzzle
Post by: digitaldog on October 07, 2018, 06:01:17 pm
Yes, there is a sense of "crying wolf" a bit too often.

As for the original question, I think it's worth saying that DNG sometimes produces a smaller file size, sometimes not, and it is mostly dependent on the efficiency of the camera's compression. Some people  see this as a reason for adopting DNG, but it's better to see it as only a by-product (and not pay much attention to it).
And sometimes larger depending on what's stuffed into the container. My ARWs from a Sony RX10 are about 21MB each but with fast load data and a rendered JPEG, my DNGs of the same camera are about 26MB. And that's fine. My Canon 5DMII is the opposite however. So I suspect we need to hear back from Peter about what settings he might be using for the conversions and what the differences might be if Fast Load Previews are used; do you know how big they are? I suspect the preferences set for previews plays a role. My JPEG Previews in LR are set for Large as well.
Title: Re: DNG puzzle
Post by: digitaldog on October 07, 2018, 06:04:44 pm
My posts have contributed nothing?
Just remove the question mark. :P
Title: Re: DNG puzzle
Post by: DP on October 08, 2018, 12:45:13 am
I have never used DNG files before today when I had the occasion to export some from LR. The original files were Sony RAWs which tend to be in the 80 MB range. Yet the exported DNGs were all 37-45 MB. It's my understanding that the DNG format preserves all the info in the RAW, so how can I get a 50% file size reduction? Thx.

here is what Eric Chan of Adobe (Camera Raw team) said and that is the ultimate truth:

Eric Chan :: http://forums.adobe.com/message/1210133#1210133
"...for archival purposes I recommend (and practice) safely storing the original raw files in the form that they came off the camera , whether they be DNG or non-DNG..."

"...To be clear, Adobe did __not__ create DNG in the hopes that photographers shooting non-DNG raw files would suddenly convert them all to DNG files and then throw away their original non-DNG raw files. Instead, Adobe created DNG has an example of a documented format (a set of TIFF extensions) that would improve interoperability among hardware and software vendors, as well as be suitable for archiving..."
Title: Re: DNG puzzle
Post by: DP on October 08, 2018, 12:48:52 am
Nope, it's preserved in an unparsed metadata block.

we have a lot of examples when then current version of DNG Converter (or the same code in ACR/LR as they too can convert) discarded data during conversion, etc... Adobe makes enough errors in their code not to discard the original raw files (see above from Eric Chan).
Title: Re: DNG puzzle
Post by: Schewe on October 08, 2018, 12:59:46 am
Be aware that DNG discards proprietary camera information.

Not true...ANY metadata that DNG does not recognize is stored in the file and is not discarded. This is a typical misunderstanding perpetrated by people who don't understand what DNG is and why it was done. DNG is simply a container format to teach the camera makers how to make robust raw files. It was basically done because Thomas Knoll got tired of trying to teach the camera makers how to store raw data.

BTW, the only reason Adobe ended up releasing DNG as a proposed standardized raw file format is because somebody needed to do something to reduce the Tower of Babel that the camera makers were making...

As for DNG files being smaller, even without using the lossy DNG option, the raw files will often be smaller as DNGs because, well, the camera makers are bad at making raw file formats and don't know how to maximize lossless compression. So, many (if not most) raw files will get smaller as DNGs.
Title: Re: DNG puzzle
Post by: Jeremy Roussak on October 08, 2018, 03:50:43 am
BTW, the only reason Adobe ended up releasing DNG as a proposed standardized raw file format is because somebody needed to do something to reduce the Tower of Babel that the camera makers were making...

And their intentions were good, no doubt, but probably doomed.

(https://imgs.xkcd.com/comics/standards.png)

Jeremy
Title: Re: DNG puzzle
Post by: Schewe on October 08, 2018, 09:55:07 am
And their intentions were good, no doubt, but probably doomed.

Actually no...the results have been much better. Since DNG was released, NEF, CR2 (Nikon & Canon raw formats) are much better at doing the main job of acting as relabel containers. The other major manufactures likewise improved their formats and some manufactures actually adopted DNG as their camera format like Leica.

The main difference between the proprietary formats and DNG are that they are undocumented...the camera makers think they should keep their formats proprietary rather than fully document the format and metadata.

It's silly really because pretty much everything can be deduced with revers engineering. Unless the undocumented metadata is encrypted (remember when Nikon encrypted the white point metadata and Thomas refused to decode the encryption and therefor ACR wouldn't use the file's whitepoint metadata to correct).

The funny thing is that the first thing ACR and Lightroom do with proprietary raw files when it sees them is to convert to DNG anyway (even if you don't convert the raw and save as DNG) because that's the only way the proprietary files can be processed. In point of fact, a little birdie once told me that the major raw file formats are so close to being fully formed DNG containers that the cameras could prolly be converted to write to DNG file format as the camera file with a simple camera firmware update.

Personally, I don't use DNG because I like the tiny size of the side car file compared to the massive size of a DNG. So when I backup a changed file the backup only has to write the tiny .xmp file instead of the whole file that has been changed. I do use DNG as a handy delivery container for those rare times I must deliver a raw file format. That way the metadata settings are baked in the file and can't be lost if separated from the .xmp file.

Title: Re: DNG puzzle
Post by: Bart_van_der_Wolf on October 08, 2018, 10:14:54 am
Not true...ANY metadata that DNG does not recognize is stored in the file and is not discarded. This is a typical misunderstanding perpetrated by people who don't understand what DNG is and why it was done. DNG is simply a container format to teach the camera makers how to make robust raw files. It was basically done because Thomas Knoll got tired of trying to teach the camera makers how to store raw data.

BTW, the only reason Adobe ended up releasing DNG as a proposed standardized raw file format is because somebody needed to do something to reduce the Tower of Babel that the camera makers were making...

As for DNG files being smaller, even without using the lossy DNG option, the raw files will often be smaller as DNGs because, well, the camera makers are bad at making raw file formats and don't know how to maximize lossless compression. So, many (if not most) raw files will get smaller as DNGs.

While generally true, camera makers face other priorities, e.g. storing 5-10 raw images per second.
I don't see how that in any way compares to a separate compression step that can spend more time on it, while using much more powerful hardware.

In fact, even if one were to use lossy / irreversible compression, there is still a lot of size that can be saved without perceptual losses. A fine example is Google's Guetzli JPEG compression, which shrinks JPEGs to really small sizes with hardly noticeable visual losses. But it's very slow, and can take minutes per image, depending on imagesize and computer horsepower. But as a demonstration, and for Website resources, a great invention (https://github.com/google/guetzli).

Cheers,
Bart
Title: Re: DNG puzzle
Post by: RSL on October 08, 2018, 11:58:52 am
I’m with Jeff. I convert everything I’m gonna save to DNG. The only place we disagree is on .xmp files. I understand what he’s saying, and if I had the kind of volume Jeff probably works with I’d probably do the same thing. But I hate those little .xmps hanging around, so I convert to DNG. (I also keep the originals on DVDs in many cases).

And I hear what Bart’s saying. My only problems with it are these:

If you’re going to shoot movies you’re better off doing it with a movie camera. It really bugs me that my still cameras now have to be overloaded with movie bells and whistles I’ll never use.

My other beef is with “speed.” Why does anybody need to store 5 to 10 raw images a second with a still camera? That’s not what “still” means.

But, of course, people tell me all this is “progress.”
Title: Re: DNG puzzle
Post by: PeterAit on October 08, 2018, 12:35:00 pm
Whoo! As the OP, I feel like someone who has tossed a filleted pig into a piranha pool.
Title: Re: DNG puzzle
Post by: digitaldog on October 08, 2018, 12:57:41 pm
Whoo! As the OP, I feel like someone who has tossed a filleted pig into a piranha pool.
You shouldn’t. You didn't do anything wrong but ask a legitimate question. Hopefully you're satisfied with the answers about file size. As for piranha:

Title: Re: DNG puzzle
Post by: Jeremy Roussak on October 08, 2018, 02:07:23 pm
Personally, I don't use DNG because I like the tiny size of the side car file compared to the massive size of a DNG. So when I backup a changed file the backup only has to write the tiny .xmp file instead of the whole file that has been changed.

Exactly my reasons, too.

Jeremy
Title: Re: DNG puzzle
Post by: digitaldog on October 08, 2018, 04:13:29 pm
Exactly my reasons, too.

Jeremy
It's a very legitimate potential concern. That said, all my backup's take place in the middle of the night, every night, unattended (including backup to the cloud/CrashPlan) so it really has no effect on me personally. It would be kind of cool if there were a way not to force an update and backup if you did something as tiny as say embed a new keyword. But thus far, the backup hasn't been a concern but I can understand for those working with hundred of megs a day or more, it could be.
On the other hand, having my DNG's with my edits, a preview big enough to print from, my camera profiles and such all embedded in one file allows me to keep all that data in one place and easy to access; within the DNG.
And no sidecar's for 'rendered' images like JPEG, TIFF, PSD, the same issues for backup, even with a tiny edit somewhere in the document still applies.
Title: Re: DNG puzzle
Post by: DP on October 08, 2018, 06:39:08 pm
Not true...

true - Iliah Borg provided a lot of examples (due to constant errors in Adobe's code)
Title: Re: DNG puzzle
Post by: Schewe on October 08, 2018, 06:56:15 pm
true - Iliah Borg provided a lot of examples (due to constant errors in Adobe's code)

Iliah found some errors due to Adobe code and some errors because of the faulty method of encoding...and where found, Adobe fixed the errors.
Title: Re: DNG puzzle
Post by: Telecaster on October 09, 2018, 04:27:53 pm
I've never converted proprietary Raws to DNG, other than early on out of curiosity, but I do use two camera systems that write DNGs natively: Leica and Pentax (DNG is an option with the latter).

-Dave-
Title: Re: DNG puzzle
Post by: Ivophoto on October 10, 2018, 09:32:10 am
And their intentions were good, no doubt, but probably doomed.

(https://imgs.xkcd.com/comics/standards.png)

Jeremy

My father claimed to know a lot about humanism.
I asked him if he could explain what humanism exactly was.

I saw on his face he was uncomfortable with the question and he replied:
“Well, that depends what kind of humanism you mean. “

Title: Re: DNG puzzle
Post by: RSL on October 10, 2018, 09:52:25 am
And their intentions were good, no doubt, but probably doomed.

(https://imgs.xkcd.com/comics/standards.png)

Jeremy

The only problem with this cartoon is that the "standards" aren't "competitive." They're not even standards. How many times have you, as a Nikon shooter, decided to try Olympus's .ORF instead of Nikon's NEF? .ORF is only a "standard" for Olympus, and .NEF is only a standard for Nikon.

The only "competitive" standard is DNG.
Title: Re: DNG puzzle
Post by: Jeremy Roussak on October 10, 2018, 01:54:31 pm
The only problem with this cartoon is that the "standards" aren't "competitive." They're not even standards. How many times have you, as a Nikon shooter, decided to try Olympus's .ORF instead of Nikon's NEF? .ORF is only a "standard" for Olympus, and .NEF is only a standard for Nikon.

The only "competitive" standard is DNG.

True, perhaps. But it's funny anyway.

Jeremy
Title: Re: DNG puzzle
Post by: RSL on October 10, 2018, 02:03:14 pm
Yes, it is. I should have done this:  :) :D ;D 8)
Title: Re: DNG puzzle
Post by: DP on October 11, 2018, 08:21:18 pm
Iliah found some errors due to Adobe code and some errors because of the faulty method of encoding...and where found, Adobe fixed the errors.

yes, the point is if you converted and deleted the original raws then it is irreversible - who cares that Adobe fixed their errors, next time they will make a more serious error, and the next, etc... the best proof that DNG conversion is not reliable (for backup instead of the originals) is that Adobe never produced an utility that can reverse their allegedly "lossless" conversion back to the original source  ;) ... hence the point - DNG are a beautiful option for number of workflows (I myself use it for Fuji raws through Iridient XTransformer because of the very subpar demosaick implemented in ACR/LR for that X-Trans CFA arrangement), but the words of Eric Chan will stand forever - save/backup exactly what is written by your camera's firmware
Title: Re: DNG puzzle
Post by: digitaldog on October 11, 2018, 10:26:22 pm
yes, the point is if you converted and deleted the original raws then it is irreversible - ...
I wonder if that older buggy DNG could be converted again and if so??? Seems one can update older to newer version of DNG.
Title: Re: DNG puzzle
Post by: Schewe on October 12, 2018, 06:11:00 pm
yes, the point is if you converted and deleted the original raws then it is irreversible

So, show me were Adobe, Thomas Knoll or any of the Adobe evangelizers have advocated converting to DNG and deleting the original proprietary raw files? That's a straw man argument...if you understand long term preservation and conservation of digital objects you would want to store not only the original raw files but also the original software that can be used to open the original raw files. For long term sustainability the Library of Congress has outlined the main Sustainability Factors of Digital Formats. Sustainability of Digital Formats: Planning for Library of Congress Collections (https://www.loc.gov/preservation/digital/formats/sustain/sustain.shtml). All of those factors really advocate the adoption of a raw file format such as DNG which can be sustained over time. There are already file formats that have been orphaned by the original manufacturer (like Kodak). Fortunately, Thomas Knoll reversed engineered those formats and allow access to those formats in ACR and DNG Converter.

Quote
the best proof that DNG conversion is not reliable (for backup instead of the originals) is that Adobe never produced an utility that can reverse their allegedly "lossless" conversion back to the original source

Horseshyte bud...Adobe released DNG Converter for the benefit of Adobe customers whose cameras were not supported by certain versions of software. There's no charge, it's free...as such it's an freeware software for the purpose of doing conversions...so, you want a utility to convert back? Why? In case you change you mind about converting? If you are so concerned about undocumented, proprietary file format metadata, why would you convert and delete your originals in the first place? I know for a fact a reconversion utility is possible...but there's little need to build and release one.

As it stands, DNG has been a valuable contribution to the digital photo industry–even if you don't think so. It was never intended to force any file formats on the manufacturers but intended to teach how a raw file format could store metadata in a fully documented manner. The fact that camera makers refuse to do the right thing by their customers and the industry is because too many people spread FUD for a variety of reasons...

DNG has been a gift to the industry...by the same guy that gave us Photoshop in the first place. Maybe a bit of gratitude might be in order? DNG is not perfect–nothing is. But if you find a problem with anything, Thomas is pretty responsive responding to legitimate criticisms–I've seen that in person so I know it is true.

BTW, The Library of Congress has made the following Recommended Formats Statement (https://www.loc.gov/preservation/resources/rfs/stillimg.html). The Library of Congress Recommended Formats Statement (RFS) includes DNG as a preferred format for photographs in digital form and other graphic images in digital form. So, maybe people who do conservation and preservation for a living might have something useful to say on the subject?

So, that brings us to you...who are you? What is your name, what are your credentials and who are you to offer anything other than questionable opinions that may serve a dubious agenda? Care to enlighten us who you are and why we should care in the least what you think?
Title: Re: DNG puzzle
Post by: Dave Rosser on October 13, 2018, 08:25:13 am
So, show me were Adobe, Thomas Knoll or any of the Adobe evangelizers have advocated converting to DNG and deleting the original proprietary raw files?
You could read chapter 2 of The Adobe Photoshop Lightroom Classic CC Book 2018 edition by your friend Martin Evening. It persuaded me.  >:(

Dave
Title: Re: DNG puzzle
Post by: Schewe on October 13, 2018, 11:59:15 am
You could read chapter 2 of The Adobe Photoshop Lightroom Classic CC Book 2018 edition by your friend Martin Evening. It persuaded me.  >:(

He advocates converting to DNG anddeletingoriginals? Or is that merely one particular workflow?
Title: Re: DNG puzzle
Post by: Dave Rosser on October 13, 2018, 12:16:19 pm
He advocates converting to DNG anddeletingoriginals? Or is that merely one particular workflow?
He makes a strong case for DNG workflow and is vague about keeping originals. To quote from the book (my italics)
"Personally, I have no trouble converting everything I shoot to DNG and never bother to embed the original raw data. I do, however, sometimes keep backup copies of the original raw files as an extra insurance policy, but in practice I’ve never had cause to use these—or at least not yet!"

Title: Re: DNG puzzle
Post by: digitaldog on October 13, 2018, 12:24:49 pm
He makes a strong case for DNG workflow and is vague about keeping originals. To quote from the book (my italics)
"Personally, I have no trouble converting everything I shoot to DNG and never bother to embed the original raw data. I do, however, sometimes keep backup copies of the original raw files as an extra insurance policy, but in practice I’ve never had cause to use these—or at least not yet!"
Failure to provide text he recommends deleting the original. Like Martin, I see lots of reasons not to embed the proprietary raws into the DNG. And like Martin, I’ve never seen the need to use those proprietary raws. Nothing vague. If you feel the need to archive them, do so.
Title: Re: DNG puzzle
Post by: Alan Goldhammer on October 13, 2018, 01:30:37 pm
I don't have the most current version of Martin Evening's LR book but the LR3 book definitely discusses deleting the RAW files after converting to DNGs.  On page 59 there is a side note box that asks the question "...should you keep the original RAW files?"  His response, "...it depends on whether you feel comfortable discarding the originals and keeping just the DNGs."  He does note that for Canon files, you would not be able to use Canon proprietary software if you discarded the originals unless you chose to embed the original RAW file data.
Title: Re: DNG puzzle
Post by: Schewe on October 13, 2018, 04:14:16 pm
"Personally, I have no trouble converting everything I shoot to DNG and never bother to embed the original raw data. I do, however, sometimes keep backup copies of the original raw files as an extra insurance policy, but in practice I’ve never had cause to use these—or at least not yet!"

Pardon me but that sure doesn't come across as a strong advocacy for converting to DNG and deleting originals...it sounds more like an encouragement to the reader to make an informed decision.

I would never embed the original raw file inside the DNG...that seems like an overly complicated workflow. Pretty sure that's what Martin was primarily referring to there.

As for converting to DNG, you'll note that he "sometimes" keeps backup copies of the original. So do I for the primary reason of being able to reconstruct exactly what was written to a camera card in case I need to recreate the process of ingesting raw images into Lightroom for screenshots in books.

Again, what Martin is advocating is to make an informed decision...he's not telling you to convert to DNG and delete your originals.
Title: Re: DNG puzzle
Post by: Schewe on October 13, 2018, 04:16:04 pm
"...should you keep the original RAW files?"  His response, "...it depends on whether you feel comfortable discarding the originals and keeping just the DNGs."

Again, not advocating converting to DNG and deleting originals...he's advocating making an informed decision.
Title: Re: DNG puzzle
Post by: digitaldog on October 13, 2018, 04:18:52 pm
Again, not advocating converting to DNG and deleting originals...he's advocating making an informed decision.
The last bit is difficult for some  ;)
Title: Re: DNG puzzle
Post by: fdisilvestro on October 13, 2018, 04:42:42 pm
...he's advocating making an informed decision.

You would be surprised (maybe not) about how many people don't want to make decisions but follow what others have said and then having someone to blame when things go wrong.
Title: Re: DNG puzzle
Post by: Telecaster on October 13, 2018, 05:37:18 pm
I'm still amazed at the complacency of allowing one's own intellectual/creative property (photographs) to be held within proprietary containers (camera makers' Raw file formats). I allow it myself with most of my cameras. Canon, Nikon, Sony, etc. don't own our photos. So by what right are they encoding them via non-publicly-disclosed means?

-Dave-
Title: Re: DNG puzzle
Post by: 32BT on October 13, 2018, 05:47:13 pm
I'm still amazed at the complacency of allowing one's own intellectual/creative property (photographs) to be held within proprietary containers (camera makers' Raw file formats). I allow it myself with most of my cameras. Canon, Nikon, Sony, etc. don't own our photos. So by what right are they encoding them via non-publicly-disclosed means?

-Dave-

Because the end product is a jpeg which is a publicly readable image. Whatever intermediate dataformats are used, is none of your business even if you think it is. In reality your treshpassing on their ip rights, not the other way around.
Title: Re: DNG puzzle
Post by: DP on October 13, 2018, 06:00:29 pm
I'm still amazed at the complacency of allowing one's own intellectual/creative property (photographs) to be held within proprietary containers (camera makers' Raw file formats). I allow it myself with most of my cameras. Canon, Nikon, Sony, etc. don't own our photos. So by what right are they encoding them via non-publicly-disclosed means?

-Dave-

you claim that something "yours" is held within proprietary containers, but you forget that Adobe (for example only, same goes for C1, etc) are keeping your work (raw conversion) hostage even more by not disclosing how their parametric adjustments work and it is way more serious problem than the issue with raw "formats"
Title: Re: DNG puzzle
Post by: digitaldog on October 13, 2018, 06:26:14 pm
you claim that something "yours" is held within proprietary containers, but you forget that Adobe (for example only, same goes for C1, etc) are keeping your work (raw conversion) hostage even more by not disclosing how their parametric adjustments work and it is way more serious problem than the issue with raw "formats"
Only if knowing how the raw is rendered is necessary to producing a rendered image. I don't know how Canon and Sony produce a JPEG in camera from raw, yet I can end up with a JPEG and do what I please with it. Isn't that the bottom line? I don't know how Kodachrome is processed but it didn't stop me from shooting and producing transparencies. So how are we being held hostage by not knowing every proprietary set to produce an image?
Title: Re: DNG puzzle
Post by: fdisilvestro on October 13, 2018, 09:10:13 pm
Adobe, Phase one, DxO & others provide tools that let you work on your images, raw and rendered, and produce a final image for display or print. They don't have to disclose how they do it, and they never said they would. Most people would not understand the associated maths anyway and what would likely happen is having a bunch of developers trying to copy the software.
Title: Re: DNG puzzle
Post by: Schewe on October 14, 2018, 12:19:59 am
you claim that something "yours" is held within proprietary containers, but you forget that Adobe (for example only, same goes for C1, etc) are keeping your work (raw conversion) hostage even more by not disclosing how their parametric adjustments work and it is way more serious problem than the issue with raw "formats"

There is a fundamental difference between having access to your original image inside a proprietary file and undisclosed parametric adjustments which can always be redone using a different software–assuming you have access to the original image.

If you don't understand the fundamental difference then you really don't understand the real and serious risk of loosing access to a legacy of original digital images in the future.

I guess you didn't read about the sustainability of digital objects and the risk of not being able to access original content for future generations. See, we have photography from the birth of the medium that we can access–as long as the original negs or positives survive but we WON'T be able to access original digital content if steps are not taken to preserve them.

So, ya see, photographers get kinda uppity when you start dong things that put access to their original images at risk (assuming you are a photographer–we still don't know you from Adam).
Title: Re: DNG puzzle
Post by: Bart_van_der_Wolf on October 14, 2018, 06:07:24 am
There is a fundamental difference between having access to your original image inside a proprietary file and undisclosed parametric adjustments which can always be redone using a different software–assuming you have access to the original image.

If you don't understand the fundamental difference then you really don't understand the real and serious risk of loosing access to a legacy of original digital images in the future.

Hi Jeff,

I don't follow, but perhaps I'm misreading your statements.

Allow me to quote the mission statement by Dave Coffin, the author of DCRaw (https://www.cybercom.net/~dcoffin/dcraw/):

Quote from: Dave Coffin
So here is my mission: Write and maintain an ANSI C program that decodes any raw image from any digital camera on any computer running any operating system.

That program is called dcraw (pronounced "dee-see-raw"), and it's become a standard tool within and without the Open Source world. It's small (about 9000 lines), portable (standard C libraries only), free (both "gratis" and "libre"), and when used skillfully, produces better quality output than the tools provided by the camera vendor.

And building on to that, there is LibRAW (https://www.libraw.org/), an open source library project by Alex Tutubalin by Iliah Borg and it's actively being updated with new Raw file formats, and some of their own applications built on that, such as RawDigger and FastRawViewer.

You then say (bold emphasis is mine):
Quote
I guess you didn't read about the sustainability of digital objects and the risk of not being able to access original content for future generations. See, we have photography from the birth of the medium that we can access–as long as the original negs or positives survive but we WON'T be able to access original digital content if steps are not taken to preserve them.

So that's something I do not understand because, since DCRAW, there is no risk that we won't be able to access original digital content.

Maybe you are referring to something else, I don't know.

Cheers,
Bart
Title: Re: DNG puzzle
Post by: Jeremy Roussak on October 14, 2018, 07:32:11 am
... since DCRAW, there is no risk that we won't be able to access original digital content.

That may well be true, but only as long as DCRAW is kept up to date with any new file formats and any changes in OS which might cause it to break.

I confess there are issues which cause me to lose more sleep than this one, though.

Jeremy
Title: Re: DNG puzzle
Post by: Bart_van_der_Wolf on October 14, 2018, 07:50:01 am
That may well be true, but only as long as DCRAW is kept up to date with any new file formats and any changes in OS which might cause it to break.

Hi Jeremy,

Same for any other software product, including DNG converters.

The difference is that DCRaw and siblings are open source projects.
The source code is available to anyone who wants to use, or improve it.

Cheers,
Bart
Title: Re: DNG puzzle
Post by: Aram Hăvărneanu on October 14, 2018, 08:21:33 am
only as long as DCRAW is kept up to date with any new file formats

Well, we were discussing image preservation here, a.i. images you already have, in already supported formats.

any changes in OS which might cause it to break.

Dcraw and libraw are standard ANSI C code. I'm much more convinced I'll be able to compile C code in 50 years, than I am convinced Adobe will still be around.
Title: Re: DNG puzzle
Post by: KLaban on October 14, 2018, 08:40:01 am
Are there such things as proprietary DNG files? For example do the native Leica DNG files differ from other DNG files? And, if the answer is no then this is a good thing, right?

Merely the curiosity of a tog who is pig ignorant on such things.
Title: Re: DNG puzzle
Post by: Rob C on October 14, 2018, 02:48:01 pm
Are there such things as proprietary DNG files? For example do the native Leica DNG files differ from other DNG files? And, if the answer is no then this is a good thing, right?

Merely the curiosity of a tog who is pig ignorant on such things.

As I imagine many of us are. I just want things to remain simple, constant, and without challenging me to new departures and curved learning! Snaps are demanding enough without complications on top!

:-)
Title: Re: DNG puzzle
Post by: digitaldog on October 14, 2018, 03:24:49 pm
Are there such things as proprietary DNG files? For example do the native Leica DNG files differ from other DNG files? And, if the answer is no then this is a good thing, right?

Merely the curiosity of a tog who is pig ignorant on such things.
The answer should be no aside from proprietary metadata that can be stored within private tags. DNG is openly documented. That doesn't mean everyone follows the spec's or does so correctly (a recent post here (https://forum.luminous-landscape.com/index.php?topic=127023.msg1072408#msg1072408) about C1 is a case in point and the C1 folks can't answer if they've fixed such issues).
Title: Re: DNG puzzle
Post by: Chris Kern on October 14, 2018, 03:50:33 pm
I guess you didn't read about the sustainability of digital objects and the risk of not being able to access original content for future generations. See, we have photography from the birth of the medium that we can access–as long as the original negs or positives survive but we WON'T be able to access original digital content if steps are not taken to preserve them.

This is a very significant issue for the professional preservationists.  Some years ago, I had work-related discussions regarding the preservation of digital data with specialists at the U.S. National Archives and the Library of Congress.  These guys think in terms of hundreds of years; both used the analogy of analog photography (we were talking about other forms of digital data) and the timelessness of appropriately stored negatives and prints.

National Public Radio (a private company despite its name, for those of you outside the United States) tackled the issue some years ago in an April 1* segment you can listen to here (https://www.npr.org/templates/story/story.php?storyId=1216161).  In case that link doesn't work for you, I've made the MPEG-3 audio file available here (http://www.chriskern.net/temp/20030401_atc_18.mp3).

———
*Full disclosure: the part of "project curator" Alan Hovermale was played by my brother.
Title: Re: DNG puzzle
Post by: KLaban on October 14, 2018, 04:18:46 pm
The answer should be no aside from proprietary metadata that can be stored within private tags. DNG is openly documented. That doesn't mean everyone follows the spec's or does so correctly (a recent post here (https://forum.luminous-landscape.com/index.php?topic=127023.msg1072408#msg1072408) about C1 is a case in point and the C1 folks can't answer if they've fixed such issues).

Many thanks.
Title: Re: DNG puzzle
Post by: Telecaster on October 14, 2018, 05:45:34 pm
IMO Raw formats have no business being proprietary, period. They can certainly be different from maker to maker. But all these formats should be publicly documented.

-Dave-
Title: Re: DNG puzzle
Post by: faberryman on October 14, 2018, 05:54:28 pm
IMO Raw formats have no business being proprietary, period. They can certainly be different from maker to maker. But all these formats should be publicly documented.
The can't be all that secret. Every editing program seems to be able to read them.
Title: Re: DNG puzzle
Post by: digitaldog on October 14, 2018, 06:49:57 pm
The can't be all that secret. Every editing program seems to be able to read them.
After being reverse engineered. Which takes all the 3rd party software companies time and money to do which of course the consumer ends up paying for. Waiting at the very least for this unnecessary step to take place. Not what we see with the camera JPEGs. So what's the point? It's political, not a technical issue.
Title: Re: DNG puzzle
Post by: Aram Hăvărneanu on October 14, 2018, 06:55:47 pm
Not to mention that even if "every" program can parse the file, every program renders it differently, because generally programs do not reverse-engineer the subjective look and interpolation/sharpening strategies that are supplied by other manufacturer. In fact, every one applies its own subjective look.

DNG has the same problem too, as identified here: https://forum.luminous-landscape.com/index.php?topic=103292.0

The meaning of various DNG fields is not specified more concretely that "whatever ACR does".
Title: Re: DNG puzzle
Post by: Bart_van_der_Wolf on October 14, 2018, 07:31:30 pm
Not to mention that even if "every" program can parse the file, every program renders it differently, because generally programs do not reverse-engineer the subjective look and interpolation/sharpening strategies that are supplied by other manufacturer. In fact, every one applies its own subjective look.

DNG has the same problem too, as identified here: https://forum.luminous-landscape.com/index.php?topic=103292.0

The meaning of various DNG fields is not specified more concretely that "whatever ACR does".

What many are not saying/understanding is that Adobe created DNG to benefit Adobe.

If camera manufacturers were to adopt DNG as their Raw container, it would mean less work for Adobe, and any issues with the DNG content would be for the OEMs to solve. Now, Adobe still have to create a translation from camera-specific parameters to generic DNG ones, so there is a time lag before new camera models are supported, because the DNG converter has to be updated.

DNG was designed to make things easier for Adobe, and create a lower threshold for using Adobe applications. Plain and simple. That's why the DNG converter is free. Image quality has not improved, because the camera specific innovations are just stored in a generic undocumented area of the DNG, and the Raw converter has basically not been improved.

Cheers,
Bart
Title: Re: DNG puzzle
Post by: Schewe on October 15, 2018, 01:48:37 am
What many are not saying/understanding is that Adobe created DNG to benefit Adobe.

And your perception doesn't match my knowledge of the actual history...

Adobe didn't create DNG, Thomas Knoll did. At the time Thomas was tired of seeing how poorly the camera makers were at formulating both the metadata as well as the manner of packaging it inside a container.

Do you remember the first time Canon offered their pro digital camera, the 1D in late 2001? Remember what the file format extension for the raw was? It was a .tif. Now, Canon thought it was ok the use tiff because, well it was publicly documented and free to use but what Canon failed to grasp is that any software that could read tiffs would try to open the file and it would successfully open the embedded jpg thumbnail. The problem was if you saved the file as a tiff, it would save the jpeg thumbnail over writing the original raw file.

I know this was a problem because at the time I was a Canon Explorer of Light and was due to get an early camera to play with. Unfortunately (or fortunately for me), several other photographers got the camera first. A well known celeb photographer shot a bunch of celeb shots that were ruined because a digital tech opened the raw files into Photoshop (before Photoshop supported raw files) to check them and didn't realize by saving them, overwrote the originals. It required that photographer to do a reshoot...when the head of Canon USA found out about the problem, he was livid that the engineers were so stupid that they didn't anticipated using the .tif extension as a problem...it worked fine with the Canon software.

Ya see, when Canon left the Nikon/Canon/Kodak DCS camera club, Canon had to bootstrap a lot of digital engineering in a big hurry...seems that a little thing like raw file format extensions would cause any problems. They didn't really know what they were doing.

Canon then started using CR2 as their extension.

It was against this background that Thomas, who at the time worked with Dave Coffin and leveraged some of his work in Camera Raw, started to realize that somebody needed to at least attempt the creation of a standardized raw file format. He created DNG as an example of how a raw file format could be created and documented. Yes, Adobe allowed him to do it (and he got paid to do it) but what you need to understand about Adobe is if Thomas thinks it's important to do something, Adobe is pretty good and letting Thomas do what he thinks needs to be done.

You don't really think Adobe has gotten much of anything besides grief for creating DNG do you? DNG has not made Adobe's life (or Thomas' life) easier...it's actually added work to Thomas' workload because he has to oversee the DNG specs and SDK and DNG Converter. The job of reverse engineering still has to be done because only a few camera makers have adopted DNG. Those that have have received the benefits of not needing to spec a file format.

Adobe has already offered DNG to the ISO for use in an upcoming TIFF-EP file format update (things like ISO changes takes a long time so I don't know where that stands). I do know that all of the major camera makers use TIFF-EP (https://en.wikipedia.org/wiki/TIFF/EP) and that TIFF-EP was derived from TIFF-6 and Adobe inherited the file format from Aldus and was given over to the ISO as an open standard.

The funny thing is some people think raw file formats contain "secret" stuff...they really don't. Everything in a raw file format can be reverse engineered and decoded–even when things may be encrypted...the claim that camera makers would have to give up private and proprietary secrets to use DNG is a spurious claim. DNG, like TIFF-EP and TIFF-6 provide standardized methods of encoding private maker notes and data.

The main reason that I'm aware of that the camera makers DON'T want to use DNG is that they simply don't want to be told what to do. They don't want to adhere to a standard if they don't have to. And the fact that the industry at large allows the camera makers to get away with using undocumented, proprietary raw file formats give them little reason to adopt DNG.

The fact is, DNG has already done the job Thomas really wanted it to do...teach the camera makers by example, how to create and use a well formed raw file format.
Title: Re: DNG puzzle
Post by: 32BT on October 15, 2018, 06:47:08 am
And your perception doesn't match my knowledge of the actual history...

Adobe didn't create DNG, Thomas Knoll did. At the time Thomas was tired of seeing how poorly the camera makers were at formulating both the metadata as well as the manner of packaging it inside a container.

Do you remember the first time Canon offered their pro digital camera, the 1D in late 2001? Remember what the file format extension for the raw was? It was a .tif. Now, Canon thought it was ok the use tiff because, well it was publicly documented and free to use but what Canon failed to grasp is that any software that could read tiffs would try to open the file and it would successfully open the embedded jpg thumbnail. The problem was if you saved the file as a tiff, it would save the jpeg thumbnail over writing the original raw file.

I know this was a problem because at the time I was a Canon Explorer of Light and was due to get an early camera to play with. Unfortunately (or fortunately for me), several other photographers got the camera first. A well known celeb photographer shot a bunch of celeb shots that were ruined because a digital tech opened the raw files into Photoshop (before Photoshop supported raw files) to check them and didn't realize by saving them, overwrote the originals. It required that photographer to do a reshoot...when the head of Canon USA found out about the problem, he was livid that the engineers were so stupid that they didn't anticipated using the .tif extension as a problem...it worked fine with the Canon software.

Ya see, when Canon left the Nikon/Canon/Kodak DCS camera club, Canon had to bootstrap a lot of digital engineering in a big hurry...seems that a little thing like raw file format extensions would cause any problems. They didn't really know what they were doing.

Canon then started using CR2 as their extension.

It was against this background that Thomas, who at the time worked with Dave Coffin and leveraged some of his work in Camera Raw, started to realize that somebody needed to at least attempt the creation of a standardized raw file format. He created DNG as an example of how a raw file format could be created and documented. Yes, Adobe allowed him to do it (and he got paid to do it) but what you need to understand about Adobe is if Thomas thinks it's important to do something, Adobe is pretty good and letting Thomas do what he thinks needs to be done.

You don't really think Adobe has gotten much of anything besides grief for creating DNG do you? DNG has not made Adobe's life (or Thomas' life) easier...it's actually added work to Thomas' workload because he has to oversee the DNG specs and SDK and DNG Converter. The job of reverse engineering still has to be done because only a few camera makers have adopted DNG. Those that have have received the benefits of not needing to spec a file format.

Adobe has already offered DNG to the ISO for use in an upcoming TIFF-EP file format update (things like ISO changes takes a long time so I don't know where that stands). I do know that all of the major camera makers use TIFF-EP (https://en.wikipedia.org/wiki/TIFF/EP) and that TIFF-EP was derived from TIFF-6 and Adobe inherited the file format from Aldus and was given over to the ISO as an open standard.

The funny thing is some people think raw file formats contain "secret" stuff...they really don't. Everything in a raw file format can be reverse engineered and decoded–even when things may be encrypted...the claim that camera makers would have to give up private and proprietary secrets to use DNG is a spurious claim. DNG, like TIFF-EP and TIFF-6 provide standardized methods of encoding private maker notes and data.

The main reason that I'm aware of that the camera makers DON'T want to use DNG is that they simply don't want to be told what to do. They don't want to adhere to a standard if they don't have to. And the fact that the industry at large allows the camera makers to get away with using undocumented, proprietary raw file formats give them little reason to adopt DNG.

The fact is, DNG has already done the job Thomas really wanted it to do...teach the camera makers by example, how to create and use a well formed raw file format.

So basically DNG was a very round about way to have an extension changed?

Overwriting originals seems an operator error first and foremost, and a software error second. Trying to shove it into Canon's shoes is a real stretch of the imagination. Their tif was actual a proper tif, and, according to my memory, they changed the extension because it didn't play well when assigning apps to double-clicking the file. (Either ALL tifs opened in Canon's converter, or ALL tifs opened in Photoshop.)

Either way, DNG would have helped ALL rawconversion software, and thus it would have helped Adobe as well. The original assertion doesn't necessarily mean anything evil towards Adobe and therefore doesn't necessarily require a condescending history lesson or hail praise of saint thomas of adobe.

It will be funny though, once tiff-ep is accepted if ever, what extension would be proposed for files written in that format?
Title: Re: DNG puzzle
Post by: Schewe on October 15, 2018, 04:10:07 pm
So basically DNG was a very round about way to have an extension changed?

Was that intended to be snarky?

But no...it wasn't merely a fix for file format extensions, it was also a lesson on how to safely pack private maker notes in a raw file format without resorting to "accidentally" encrypting the white balance metadata like Nikon did...remember that? Seems like the Nikon engineers were having problems getting the then new Nikon D2X camera's white balance data to be recognized by the Nikon software so they encrypted it. They didn't realize the act of encrypting was deemed copy protection therefore something Thomas Knoll refused to decode due to Digital Millennium Copyright Act implications...Pixels and Protocol (https://www.nytimes.com/2005/05/05/technology/circuits/pixels-and-protocol.html)

Oooops...Nikon didn't realize that would be a problem...oh, and BTW, is there some logic to the fact Nikon used the file extension .nef on both their scanners and digital cameras? That caused a bit of problems too, until Nikon got out of the scanner market.

Ya see, it's my considered option that hardware companies, by and large, are not very good when it comes to software. The camera makers cling to their file formats like there's something of value in them...there isn't. Can anybody point out any actual value to the .nef or .cr2 file formats? Is there anything Nikon or Canon would be loosing by publicly documenting how and where metadata is stored–even if the nature of the metadata remained private and proprietary?

Quote
The original assertion doesn't necessarily mean anything evil towards Adobe and therefore doesn't necessarily require a condescending history lesson or hail praise of saint thomas of adobe.

Yeah, ok...pretty sure THAT was supposed to be snarky...the original assertion was clearly a swipe at Adobe and so is the above.

Do you know Thomas? Have you ever argued the merits of this or that approach to software development? Have you ever seen him use trial and error to laboriously decode and reverse engineer a new camera? I have...I've also seen a lot of people attribute nefarious motives to both Thomas and Adobe that are simply bullshit. Thomas ain't a saint...he's far from perfect, but even YOU must admit that Thomas has done a lot for the industry-yeah, and gotten kinda rich for it. But if you knew Thomas you would realize he does what he does not for money or power...he does what he does because he thinks it's the right thing to do. In point of fact, he really can't help himself...and while he ain't a saint, he is a friggin' boy scout when it comes to digital imaging...an industry he kinda helped make.

Snak all ya want butthead...but I'm pretty sure Thomas has done far more for the industry than you have, right?
Title: Re: DNG puzzle
Post by: digitaldog on October 15, 2018, 04:13:20 pm
Snak all ya want butthead...but I'm pretty sure Thomas has done far more for the industry than you have, right?
We can only assume as all we know about this poster (despite you asking) is he's a male and lives in the Netherlands

Title: Re: DNG puzzle
Post by: Rob C on October 15, 2018, 05:30:42 pm
And you should both know the danger in assumptions...

So much barely suppressed violence; quite disturbing to realise it abounds within the technical world.

At least he's capable of posting some pretty good photographs.

Rob
Title: Re: DNG puzzle
Post by: digitaldog on October 15, 2018, 05:33:17 pm
And you should both know the danger in assumptions...
Its about all you can do when the question asked are not answered (for whatever reasons).
Jeff tried.
His pretty good Photographs are where?
Title: Re: DNG puzzle
Post by: Telecaster on October 15, 2018, 05:55:46 pm
So much barely suppressed violence; quite disturbing to realise it abounds within the technical world.

My first exposure to online raging came courtesy of the OS/2>—<Windows (aka IBM>—<Microsoft) flame wars of the late 1980s. (As discussed a bit elsewhere in LuLa Land recently.) IMO it has more than a little to do with the ability to create clockwork software worlds where everything unfolds according to design. From experience I can say there's a seductive, even intoxicating, aspect to this. By contrast the messy real world of human behavior & interaction can appear even more infuriating than otherwise.

-Dave-
Title: Re: DNG puzzle
Post by: 32BT on October 15, 2018, 06:38:27 pm
Snak all ya want butthead...but I'm pretty sure Thomas has done far more for the industry than you have, right?

I have no problem with Thomas. I have a problem with pulling the thomasknollgospeltrumpcard as some kind of justification for anecdotal assertions.

Title: Re: DNG puzzle
Post by: Rob C on October 16, 2018, 04:48:11 am
Its about all you can do when the question asked are not answered (for whatever reasons).
Jeff tried.
His pretty good Photographs are where?

Right here, on LuLa.

I have heard that Schewe was a good commercial snapper and I have distant memory of seeing an impressive shot of a glass of something with great condensation on the surface. Having shot a good few beer company calendars I know just how difficult it can be to make it appear photographically convincing, especially out in the sunshine, and in competition with a girl. (This mental image of some old shot may or may not have been correctly attributed to Schewe; if correct, it is the only image of his I can recall seeing. Why so shy?)

On the other hand, I don't remember seeing a single one of yours.

;-)


Title: Re: DNG puzzle
Post by: Rob C on October 16, 2018, 04:57:27 am
My first exposure to online raging came courtesy of the OS/2>—<Windows (aka IBM>—<Microsoft) flame wars of the late 1980s. (As discussed a bit elsewhere in LuLa Land recently.) IMO it has more than a little to do with the ability to create clockwork software worlds where everything unfolds according to design. From experience I can say there's a seductive, even intoxicating, aspect to this. By contrast the messy real world of human behavior & interaction can appear even more infuriating than otherwise.

-Dave-

Good point! The difference, in a way, between the science/maths mindset and the artist's less then expectant of certainties one.

Rob
Title: Re: DNG puzzle
Post by: Jim Pascoe on October 16, 2018, 05:26:04 am
I've not visited the site for a few weeks and was surprised to find this topic in the Coffee Corner.....

Jim


Title: Re: DNG puzzle
Post by: KLaban on October 16, 2018, 05:37:35 am
It's an interesting discussion that is being undermined by egos and name calling.

Needs moving and moderating.
Title: Re: DNG puzzle
Post by: DP on October 16, 2018, 07:09:23 am
but I'm pretty sure Thomas has done far more for the industry than you have, right?

at some point old farts inevitably have to step down (or rather up to rest on laurels) ;)
Title: Re: DNG puzzle
Post by: DP on October 16, 2018, 07:14:18 am
IMO Raw formats have no business being proprietary, period.
I invest way more effort into working with raw in terms of dialing in raw conversion parameters and various adjustments and IMHO parametric adjustments interpretation have no business being proprietary ! I cam easily take my raws to almost any raw converter w/o issues - but I can't take my work in ACR outside of Adobe realm ;) ... now think who is the worst offender really !
Title: Re: DNG puzzle
Post by: DP on October 16, 2018, 07:14:49 am
libraw are standard ANSI C
C++
Title: Re: DNG puzzle
Post by: DP on October 16, 2018, 07:20:34 am
There is a fundamental difference between

there is no difference - just your vested paid interest in all things Adobe  ;D ... one more time - I can take my proprietary raws to almost any raw converter w/o any issues, but I can't take my proprietary parametric adjustments ... so who really keeps my work hostage, who does not really let me own MY OWN WORK  ? Adobe does (and other companies making raw converters - so if I am working with Canon's raws and DPP - my issue is not with Canon's raw format - my issue is with DPP)
Title: Re: DNG puzzle
Post by: digitaldog on October 16, 2018, 09:15:26 am
Right here, on LuLa.

I have heard that Schewe was a good commercial snapper and I have distant memory of seeing an impressive shot of a glass of something with great condensation on the surface. Having shot a good few beer company calendars I know just how difficult it can be to make it appear photographically convincing, especially out in the sunshine, and in competition with a girl. (This mental image of some old shot may or may not have been correctly attributed to Schewe; if correct, it is the only image of his I can recall seeing. Why so shy?)
Rob, I've know Jeff for over 20 years, been to his studio many times, stayed at his home many times, we are business partners and I know much of HIS work. It's that other guy (opgr) I was asking about. Tell me about his photography.

Quote
On the other hand, I don't remember seeing a single one of yours.

As for mine, some are here scattered in the forums (more recently in show your pets) but if you take the time to click on the web link to my site (http://digitaldog.net), there are some web galleries there if you want to look at say the Amazon shots I took while working with Michael/Chris and Jay Maisel (Jay and I were instructors on that trip).

As for my professional photography clients (alas, from the 1990's):
https://www.linkedin.com/in/digitaldog/?trk=hp-identity-name (https://www.linkedin.com/in/digitaldog/?trk=hp-identity-name)
Title: Re: DNG puzzle
Post by: Schewe on October 16, 2018, 12:42:59 pm
(This mental image of some old shot may or may not have been correctly attributed to Schewe; if correct, it is the only image of his I can recall seeing. Why so shy?)

My commercial work... (https://jeffschewe.myportfolio.com/commercial)

Not shy...just rarely show my old work.
Title: Re: DNG puzzle
Post by: Martin Kristiansen on October 16, 2018, 01:41:59 pm
My commercial work... (https://jeffschewe.myportfolio.com/commercial)

Not shy...just rarely show my old work.

Well that’s proper.
Title: Re: DNG puzzle
Post by: Jeremy Roussak on October 16, 2018, 02:10:09 pm
It's that other guy (opgr) I was asking about. Tell me about his photography.

He's put a few images up on this site, Andrew. Look around.

Jeremy
Title: Re: DNG puzzle
Post by: digitaldog on October 16, 2018, 02:13:44 pm
He's put a few images up on this site, Andrew. Look around.

Jeremy
Thanks but I see now such site, can you provide a URL? TIA.
Title: Re: DNG puzzle
Post by: Jeremy Roussak on October 16, 2018, 02:14:58 pm
Thanks but I see now such site, can you provide a URL? TIA.

This site, Andrew. LuLa. Look around the "user critiques" forum.

Jeremy
Title: Re: DNG puzzle
Post by: Telecaster on October 16, 2018, 05:03:41 pm
I invest way more effort into working with raw in terms of dialing in raw conversion parameters and various adjustments and IMHO parametric adjustments interpretation have no business being proprietary ! I cam easily take my raws to almost any raw converter w/o issues - but I can't take my work in ACR outside of Adobe realm ;) ... now think who is the worst offender really !

Not sure why you're interpreting (or at least seeming to do so) my comments here as being pro-Adobe. They're not.

-Dave-
Title: Re: DNG puzzle
Post by: Rob C on October 17, 2018, 03:44:47 am
My commercial work... (https://jeffschewe.myportfolio.com/commercial)

Not shy...just rarely show my old work.

Thank you for posting the link!

You have some very good work there and it backs up what I have sometimes pointed out in the past: still life is a creative form of photography where you have to start from nothing but a brief - or a layout - which itself takes a lot of creative input - and make something interesting happen. It is not a matter of just hanging around until God decides to switch on the pretty lighting.

On or two of the shots - the rabbit and hat one, in particular, seem familiar to me: did you also work with FPG or TIB?

Rob
Title: Re: DNG puzzle
Post by: Rob C on October 17, 2018, 04:02:58 am
Rob, I've know Jeff for over 20 years, been to his studio many times, stayed at his home many times, we are business partners and I know much of HIS work. It's that other guy (opgr) I was asking about. Tell me about his photography.

As for mine, some are here scattered in the forums (more recently in show your pets) but if you take the time to click on the web link to my site (http://digitaldog.net), there are some web galleries there if you want to look at say the Amazon shots I took while working with Michael/Chris and Jay Maisel (Jay and I were instructors on that trip).

As for my professional photography clients (alas, from the 1990's):
https://www.linkedin.com/in/digitaldog/?trk=hp-identity-name (https://www.linkedin.com/in/digitaldog/?trk=hp-identity-name)

We seem fated not to understand what the other writes. However, it looks as if Jeremy has answered that on my behalf.

I didn't see your shots in Pets Corner because I seldom go there (maybe because I was a Playboy fan and not a Penthouse one...) but I did try to access your site from your link.

Some of the sections were not open to me because of LinkedIn, which I don't want on the iPad though I did get involved via the computer a long time ago. However, the one to White Sands and the dogs was open, and it appears to be a nice location in which to shoot. Your dogs look very healthy and full of life - long may it continue.

It would be lovely to work there with the right girl, the right colours of chiffon fabrics flowing out in the breeze behind her... dream on: my fifteen minutes clocked out some time ago; Warhol could have been more generous.

;-)

Title: Re: DNG puzzle
Post by: digitaldog on October 17, 2018, 09:30:05 am
I didn't see your shots in Pets Corner because I seldom go there (maybe because I was a Playboy fan and not a Penthouse one...)
Then go back to my web site and click on the links for the Greg Gorman workshops I co-taught and you'll see the Playboy inspired stuff.  :P
Title: Re: DNG puzzle
Post by: PeterAit on October 17, 2018, 11:42:37 am
When I started this thread with a simple technical question, I could not have imagined how it would evolve. It's both horrifying and awe-inspiring.
Title: Re: DNG puzzle
Post by: digitaldog on October 17, 2018, 11:45:56 am
When I started this thread with a simple technical question, I could not have imagined how it would evolve. It's both horrifying and awe-inspiring.
What about post #2, specifically asking you about what settings you used and why, the DNG being smaller than the original proprietary raw in many cases is to be expected?
The rest is to be expected when DNG or something Adobe controls and provides comes up. Might as well ask, what wights more, a Canon or a Nikon, a Mac or a WIN PC; then all hell breaks loose.  ;)
Title: Re: DNG puzzle
Post by: Eric Myrvaagnes on October 17, 2018, 11:47:30 am
When I started this thread with a simple technical question, I could not have imagined how it would evolve. It's both horrifying and awe-inspiring.
From DNG to Playboy...   :D
Title: Re: DNG puzzle
Post by: Rob C on October 17, 2018, 12:32:45 pm
From DNG to Playboy...   :D

So true Eric: DNG etc. are all terrifying to consider instead of merely accepting; Playboy used to be inspiring, and then became awe-inspiring which wasn't quite as pleasant a vibe.

;-(
Title: Re: DNG puzzle
Post by: Denis de Gannes on October 17, 2018, 10:55:07 pm
When I started this thread with a simple technical question, I could not have imagined how it would evolve. It's both horrifying and awe-inspiring.
To me its a simple answer, if you you are committed to use only Adobe software then using the DNG option has some benefits, however if you are also utilising other software applications then you need to seriously consider if any pros will negate the cons you have to overcome.
Title: Re: DNG puzzle
Post by: digitaldog on October 17, 2018, 11:03:13 pm
To me its a simple answer, if you you are committed to use only Adobe software then using the DNG option has some benefits, however if you are also utilising other software applications then you need to seriously consider if any pros will negate the cons you have to overcome.
A simple answer to this exact question?
It's my understanding that the DNG format preserves all the info in the RAW, so how can I get a 50% file size reduction?
Hum..... Again, I believe post#2 had the simple answer prior to the political “answers” (to be kind).
Title: Re: DNG puzzle
Post by: Damon Lynch on October 19, 2018, 12:16:26 pm
A good RAW format should (among plenty of other criteria not listed below):


Why? If you know exactly what you (i.e. your software program) are looking for, and where to look for it, it's much quicker and much less prone to error. For example: if you have a RAW on a medium like a camera that is comparatively super-slow to read from, then being able to grab the image orientation, thumbnail, date/time and other key metadata quickly is extremely helpful. Just read a small chunk of the start of the file, and there is much you can do with it. If instead you have to fish around all throughout the file for this information, looking for field X in one file and field Y in another, then it's a total dog's breakfast.

The best mainstream format I know of with respect to these criteria is Canon's CR2 format (CRW was awful). The worst by far is Nikon's NEF format. (I'm ignoring the many marginal RAW formats, some of which are equally if not more awful). DNG is somewhat in the middle -- perhaps because it is created in post-processing as much as it is on the camera, there is more latitude for creativity to solve problems of the kinds that software developers must confront. Therefore it's more inconsistent than a format like CR2. That's not blaming anyone or casting aspersion on anyone's reputation -- I suspect it's an inevitable side-effect of the big-tent approach inherent to DNG.

Sadly, my preliminary impression of CR3 is that with respect to the criteria above, it regresses rather than improves on CR2. Bummer. It's highly annoying. Why did Canon go and screw it up?

For a discussion among software developers who are reverse engineering CR3 (DNG is mentioned), see https://github.com/Exiv2/exiv2/issues/236

As an aside, I'm writing this from the perspective of a software developer who confronts the issues mentioned above. I understand there are many other important attributes of a RAW format, for reasons already discussed. But these are the kinds of things I am confronted with, hence my two cents.

And as a final aside, good on Mr Knoll / Adobe for creating the DNG format and also the XMP standard. I wish more would stick to the XMP standard. I hate having to deal with variations on it because of smart-ass individual software developers who think their way is better.
Title: Re: DNG puzzle
Post by: Doug Gray on October 19, 2018, 06:55:22 pm
For what it's worth most RAW file use "tiff" containers. This makes DCRAW.C's job simpler. OEMs use a lot of proprietary tags and formats but the tiff spec defines how all this stuff is contained in the tiff container. Matlab can read the RAW tiff container and it will yield all sorts of fascinating stuff. Walking through dcraw.c as it processes a RAW file while looking at the contents in Matlab is an interesting exercise and will give an idea of what at least part of the RAW file contains and what it does but much remains unspecified by the OEM.