Pages: 1 2 3 [4] 5 6   Go Down

Author Topic: DNG puzzle  (Read 9850 times)

Jeremy Roussak

  • Administrator
  • Sr. Member
  • *****
  • Offline Offline
  • Posts: 8963
    • site
Re: DNG puzzle
« Reply #60 on: October 14, 2018, 07:32:11 am »

... since DCRAW, there is no risk that we won't be able to access original digital content.

That may well be true, but only as long as DCRAW is kept up to date with any new file formats and any changes in OS which might cause it to break.

I confess there are issues which cause me to lose more sleep than this one, though.

Jeremy
Logged

Bart_van_der_Wolf

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 8914
Re: DNG puzzle
« Reply #61 on: October 14, 2018, 07:50:01 am »

That may well be true, but only as long as DCRAW is kept up to date with any new file formats and any changes in OS which might cause it to break.

Hi Jeremy,

Same for any other software product, including DNG converters.

The difference is that DCRaw and siblings are open source projects.
The source code is available to anyone who wants to use, or improve it.

Cheers,
Bart
Logged
== If you do what you did, you'll get what you got. ==

Aram Hăvărneanu

  • Full Member
  • ***
  • Offline Offline
  • Posts: 214
Re: DNG puzzle
« Reply #62 on: October 14, 2018, 08:21:33 am »

only as long as DCRAW is kept up to date with any new file formats

Well, we were discussing image preservation here, a.i. images you already have, in already supported formats.

any changes in OS which might cause it to break.

Dcraw and libraw are standard ANSI C code. I'm much more convinced I'll be able to compile C code in 50 years, than I am convinced Adobe will still be around.
Logged

KLaban

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2451
    • Keith Laban Photography
Re: DNG puzzle
« Reply #63 on: October 14, 2018, 08:40:01 am »

Are there such things as proprietary DNG files? For example do the native Leica DNG files differ from other DNG files? And, if the answer is no then this is a good thing, right?

Merely the curiosity of a tog who is pig ignorant on such things.

Rob C

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 24074
Re: DNG puzzle
« Reply #64 on: October 14, 2018, 02:48:01 pm »

Are there such things as proprietary DNG files? For example do the native Leica DNG files differ from other DNG files? And, if the answer is no then this is a good thing, right?

Merely the curiosity of a tog who is pig ignorant on such things.

As I imagine many of us are. I just want things to remain simple, constant, and without challenging me to new departures and curved learning! Snaps are demanding enough without complications on top!

:-)

digitaldog

  • Sr. Member
  • ****
  • Online Online
  • Posts: 20652
  • Andrew Rodney
    • http://www.digitaldog.net/
Re: DNG puzzle
« Reply #65 on: October 14, 2018, 03:24:49 pm »

Are there such things as proprietary DNG files? For example do the native Leica DNG files differ from other DNG files? And, if the answer is no then this is a good thing, right?

Merely the curiosity of a tog who is pig ignorant on such things.
The answer should be no aside from proprietary metadata that can be stored within private tags. DNG is openly documented. That doesn't mean everyone follows the spec's or does so correctly (a recent post here about C1 is a case in point and the C1 folks can't answer if they've fixed such issues).
Logged
http://www.digitaldog.net/
Author "Color Management for Photographers".

Chris Kern

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2035
    • Chris Kern's Eponymous Website
Re: DNG puzzle
« Reply #66 on: October 14, 2018, 03:50:33 pm »

I guess you didn't read about the sustainability of digital objects and the risk of not being able to access original content for future generations. See, we have photography from the birth of the medium that we can access–as long as the original negs or positives survive but we WON'T be able to access original digital content if steps are not taken to preserve them.

This is a very significant issue for the professional preservationists.  Some years ago, I had work-related discussions regarding the preservation of digital data with specialists at the U.S. National Archives and the Library of Congress.  These guys think in terms of hundreds of years; both used the analogy of analog photography (we were talking about other forms of digital data) and the timelessness of appropriately stored negatives and prints.

National Public Radio (a private company despite its name, for those of you outside the United States) tackled the issue some years ago in an April 1* segment you can listen to here.  In case that link doesn't work for you, I've made the MPEG-3 audio file available here.

———
*Full disclosure: the part of "project curator" Alan Hovermale was played by my brother.

KLaban

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2451
    • Keith Laban Photography
Re: DNG puzzle
« Reply #67 on: October 14, 2018, 04:18:46 pm »

The answer should be no aside from proprietary metadata that can be stored within private tags. DNG is openly documented. That doesn't mean everyone follows the spec's or does so correctly (a recent post here about C1 is a case in point and the C1 folks can't answer if they've fixed such issues).

Many thanks.

Telecaster

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 3686
Re: DNG puzzle
« Reply #68 on: October 14, 2018, 05:45:34 pm »

IMO Raw formats have no business being proprietary, period. They can certainly be different from maker to maker. But all these formats should be publicly documented.

-Dave-
Logged

faberryman

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 4851
Re: DNG puzzle
« Reply #69 on: October 14, 2018, 05:54:28 pm »

IMO Raw formats have no business being proprietary, period. They can certainly be different from maker to maker. But all these formats should be publicly documented.
The can't be all that secret. Every editing program seems to be able to read them.
Logged

digitaldog

  • Sr. Member
  • ****
  • Online Online
  • Posts: 20652
  • Andrew Rodney
    • http://www.digitaldog.net/
Re: DNG puzzle
« Reply #70 on: October 14, 2018, 06:49:57 pm »

The can't be all that secret. Every editing program seems to be able to read them.
After being reverse engineered. Which takes all the 3rd party software companies time and money to do which of course the consumer ends up paying for. Waiting at the very least for this unnecessary step to take place. Not what we see with the camera JPEGs. So what's the point? It's political, not a technical issue.
Logged
http://www.digitaldog.net/
Author "Color Management for Photographers".

Aram Hăvărneanu

  • Full Member
  • ***
  • Offline Offline
  • Posts: 214
Re: DNG puzzle
« Reply #71 on: October 14, 2018, 06:55:47 pm »

Not to mention that even if "every" program can parse the file, every program renders it differently, because generally programs do not reverse-engineer the subjective look and interpolation/sharpening strategies that are supplied by other manufacturer. In fact, every one applies its own subjective look.

DNG has the same problem too, as identified here: https://forum.luminous-landscape.com/index.php?topic=103292.0

The meaning of various DNG fields is not specified more concretely that "whatever ACR does".
Logged

Bart_van_der_Wolf

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 8914
Re: DNG puzzle
« Reply #72 on: October 14, 2018, 07:31:30 pm »

Not to mention that even if "every" program can parse the file, every program renders it differently, because generally programs do not reverse-engineer the subjective look and interpolation/sharpening strategies that are supplied by other manufacturer. In fact, every one applies its own subjective look.

DNG has the same problem too, as identified here: https://forum.luminous-landscape.com/index.php?topic=103292.0

The meaning of various DNG fields is not specified more concretely that "whatever ACR does".

What many are not saying/understanding is that Adobe created DNG to benefit Adobe.

If camera manufacturers were to adopt DNG as their Raw container, it would mean less work for Adobe, and any issues with the DNG content would be for the OEMs to solve. Now, Adobe still have to create a translation from camera-specific parameters to generic DNG ones, so there is a time lag before new camera models are supported, because the DNG converter has to be updated.

DNG was designed to make things easier for Adobe, and create a lower threshold for using Adobe applications. Plain and simple. That's why the DNG converter is free. Image quality has not improved, because the camera specific innovations are just stored in a generic undocumented area of the DNG, and the Raw converter has basically not been improved.

Cheers,
Bart
Logged
== If you do what you did, you'll get what you got. ==

Schewe

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 6229
    • http:www.schewephoto.com
Re: DNG puzzle
« Reply #73 on: October 15, 2018, 01:48:37 am »

What many are not saying/understanding is that Adobe created DNG to benefit Adobe.

And your perception doesn't match my knowledge of the actual history...

Adobe didn't create DNG, Thomas Knoll did. At the time Thomas was tired of seeing how poorly the camera makers were at formulating both the metadata as well as the manner of packaging it inside a container.

Do you remember the first time Canon offered their pro digital camera, the 1D in late 2001? Remember what the file format extension for the raw was? It was a .tif. Now, Canon thought it was ok the use tiff because, well it was publicly documented and free to use but what Canon failed to grasp is that any software that could read tiffs would try to open the file and it would successfully open the embedded jpg thumbnail. The problem was if you saved the file as a tiff, it would save the jpeg thumbnail over writing the original raw file.

I know this was a problem because at the time I was a Canon Explorer of Light and was due to get an early camera to play with. Unfortunately (or fortunately for me), several other photographers got the camera first. A well known celeb photographer shot a bunch of celeb shots that were ruined because a digital tech opened the raw files into Photoshop (before Photoshop supported raw files) to check them and didn't realize by saving them, overwrote the originals. It required that photographer to do a reshoot...when the head of Canon USA found out about the problem, he was livid that the engineers were so stupid that they didn't anticipated using the .tif extension as a problem...it worked fine with the Canon software.

Ya see, when Canon left the Nikon/Canon/Kodak DCS camera club, Canon had to bootstrap a lot of digital engineering in a big hurry...seems that a little thing like raw file format extensions would cause any problems. They didn't really know what they were doing.

Canon then started using CR2 as their extension.

It was against this background that Thomas, who at the time worked with Dave Coffin and leveraged some of his work in Camera Raw, started to realize that somebody needed to at least attempt the creation of a standardized raw file format. He created DNG as an example of how a raw file format could be created and documented. Yes, Adobe allowed him to do it (and he got paid to do it) but what you need to understand about Adobe is if Thomas thinks it's important to do something, Adobe is pretty good and letting Thomas do what he thinks needs to be done.

You don't really think Adobe has gotten much of anything besides grief for creating DNG do you? DNG has not made Adobe's life (or Thomas' life) easier...it's actually added work to Thomas' workload because he has to oversee the DNG specs and SDK and DNG Converter. The job of reverse engineering still has to be done because only a few camera makers have adopted DNG. Those that have have received the benefits of not needing to spec a file format.

Adobe has already offered DNG to the ISO for use in an upcoming TIFF-EP file format update (things like ISO changes takes a long time so I don't know where that stands). I do know that all of the major camera makers use TIFF-EP and that TIFF-EP was derived from TIFF-6 and Adobe inherited the file format from Aldus and was given over to the ISO as an open standard.

The funny thing is some people think raw file formats contain "secret" stuff...they really don't. Everything in a raw file format can be reverse engineered and decoded–even when things may be encrypted...the claim that camera makers would have to give up private and proprietary secrets to use DNG is a spurious claim. DNG, like TIFF-EP and TIFF-6 provide standardized methods of encoding private maker notes and data.

The main reason that I'm aware of that the camera makers DON'T want to use DNG is that they simply don't want to be told what to do. They don't want to adhere to a standard if they don't have to. And the fact that the industry at large allows the camera makers to get away with using undocumented, proprietary raw file formats give them little reason to adopt DNG.

The fact is, DNG has already done the job Thomas really wanted it to do...teach the camera makers by example, how to create and use a well formed raw file format.
Logged

32BT

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 3095
    • Pictures
Re: DNG puzzle
« Reply #74 on: October 15, 2018, 06:47:08 am »

And your perception doesn't match my knowledge of the actual history...

Adobe didn't create DNG, Thomas Knoll did. At the time Thomas was tired of seeing how poorly the camera makers were at formulating both the metadata as well as the manner of packaging it inside a container.

Do you remember the first time Canon offered their pro digital camera, the 1D in late 2001? Remember what the file format extension for the raw was? It was a .tif. Now, Canon thought it was ok the use tiff because, well it was publicly documented and free to use but what Canon failed to grasp is that any software that could read tiffs would try to open the file and it would successfully open the embedded jpg thumbnail. The problem was if you saved the file as a tiff, it would save the jpeg thumbnail over writing the original raw file.

I know this was a problem because at the time I was a Canon Explorer of Light and was due to get an early camera to play with. Unfortunately (or fortunately for me), several other photographers got the camera first. A well known celeb photographer shot a bunch of celeb shots that were ruined because a digital tech opened the raw files into Photoshop (before Photoshop supported raw files) to check them and didn't realize by saving them, overwrote the originals. It required that photographer to do a reshoot...when the head of Canon USA found out about the problem, he was livid that the engineers were so stupid that they didn't anticipated using the .tif extension as a problem...it worked fine with the Canon software.

Ya see, when Canon left the Nikon/Canon/Kodak DCS camera club, Canon had to bootstrap a lot of digital engineering in a big hurry...seems that a little thing like raw file format extensions would cause any problems. They didn't really know what they were doing.

Canon then started using CR2 as their extension.

It was against this background that Thomas, who at the time worked with Dave Coffin and leveraged some of his work in Camera Raw, started to realize that somebody needed to at least attempt the creation of a standardized raw file format. He created DNG as an example of how a raw file format could be created and documented. Yes, Adobe allowed him to do it (and he got paid to do it) but what you need to understand about Adobe is if Thomas thinks it's important to do something, Adobe is pretty good and letting Thomas do what he thinks needs to be done.

You don't really think Adobe has gotten much of anything besides grief for creating DNG do you? DNG has not made Adobe's life (or Thomas' life) easier...it's actually added work to Thomas' workload because he has to oversee the DNG specs and SDK and DNG Converter. The job of reverse engineering still has to be done because only a few camera makers have adopted DNG. Those that have have received the benefits of not needing to spec a file format.

Adobe has already offered DNG to the ISO for use in an upcoming TIFF-EP file format update (things like ISO changes takes a long time so I don't know where that stands). I do know that all of the major camera makers use TIFF-EP and that TIFF-EP was derived from TIFF-6 and Adobe inherited the file format from Aldus and was given over to the ISO as an open standard.

The funny thing is some people think raw file formats contain "secret" stuff...they really don't. Everything in a raw file format can be reverse engineered and decoded–even when things may be encrypted...the claim that camera makers would have to give up private and proprietary secrets to use DNG is a spurious claim. DNG, like TIFF-EP and TIFF-6 provide standardized methods of encoding private maker notes and data.

The main reason that I'm aware of that the camera makers DON'T want to use DNG is that they simply don't want to be told what to do. They don't want to adhere to a standard if they don't have to. And the fact that the industry at large allows the camera makers to get away with using undocumented, proprietary raw file formats give them little reason to adopt DNG.

The fact is, DNG has already done the job Thomas really wanted it to do...teach the camera makers by example, how to create and use a well formed raw file format.

So basically DNG was a very round about way to have an extension changed?

Overwriting originals seems an operator error first and foremost, and a software error second. Trying to shove it into Canon's shoes is a real stretch of the imagination. Their tif was actual a proper tif, and, according to my memory, they changed the extension because it didn't play well when assigning apps to double-clicking the file. (Either ALL tifs opened in Canon's converter, or ALL tifs opened in Photoshop.)

Either way, DNG would have helped ALL rawconversion software, and thus it would have helped Adobe as well. The original assertion doesn't necessarily mean anything evil towards Adobe and therefore doesn't necessarily require a condescending history lesson or hail praise of saint thomas of adobe.

It will be funny though, once tiff-ep is accepted if ever, what extension would be proposed for files written in that format?
Logged
Regards,
~ O ~
If you can stomach it: pictures

Schewe

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 6229
    • http:www.schewephoto.com
Re: DNG puzzle
« Reply #75 on: October 15, 2018, 04:10:07 pm »

So basically DNG was a very round about way to have an extension changed?

Was that intended to be snarky?

But no...it wasn't merely a fix for file format extensions, it was also a lesson on how to safely pack private maker notes in a raw file format without resorting to "accidentally" encrypting the white balance metadata like Nikon did...remember that? Seems like the Nikon engineers were having problems getting the then new Nikon D2X camera's white balance data to be recognized by the Nikon software so they encrypted it. They didn't realize the act of encrypting was deemed copy protection therefore something Thomas Knoll refused to decode due to Digital Millennium Copyright Act implications...Pixels and Protocol

Oooops...Nikon didn't realize that would be a problem...oh, and BTW, is there some logic to the fact Nikon used the file extension .nef on both their scanners and digital cameras? That caused a bit of problems too, until Nikon got out of the scanner market.

Ya see, it's my considered option that hardware companies, by and large, are not very good when it comes to software. The camera makers cling to their file formats like there's something of value in them...there isn't. Can anybody point out any actual value to the .nef or .cr2 file formats? Is there anything Nikon or Canon would be loosing by publicly documenting how and where metadata is stored–even if the nature of the metadata remained private and proprietary?

Quote
The original assertion doesn't necessarily mean anything evil towards Adobe and therefore doesn't necessarily require a condescending history lesson or hail praise of saint thomas of adobe.

Yeah, ok...pretty sure THAT was supposed to be snarky...the original assertion was clearly a swipe at Adobe and so is the above.

Do you know Thomas? Have you ever argued the merits of this or that approach to software development? Have you ever seen him use trial and error to laboriously decode and reverse engineer a new camera? I have...I've also seen a lot of people attribute nefarious motives to both Thomas and Adobe that are simply bullshit. Thomas ain't a saint...he's far from perfect, but even YOU must admit that Thomas has done a lot for the industry-yeah, and gotten kinda rich for it. But if you knew Thomas you would realize he does what he does not for money or power...he does what he does because he thinks it's the right thing to do. In point of fact, he really can't help himself...and while he ain't a saint, he is a friggin' boy scout when it comes to digital imaging...an industry he kinda helped make.

Snak all ya want butthead...but I'm pretty sure Thomas has done far more for the industry than you have, right?
Logged

digitaldog

  • Sr. Member
  • ****
  • Online Online
  • Posts: 20652
  • Andrew Rodney
    • http://www.digitaldog.net/
Re: DNG puzzle
« Reply #76 on: October 15, 2018, 04:13:20 pm »

Snak all ya want butthead...but I'm pretty sure Thomas has done far more for the industry than you have, right?
We can only assume as all we know about this poster (despite you asking) is he's a male and lives in the Netherlands

Logged
http://www.digitaldog.net/
Author "Color Management for Photographers".

Rob C

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 24074
Re: DNG puzzle
« Reply #77 on: October 15, 2018, 05:30:42 pm »

And you should both know the danger in assumptions...

So much barely suppressed violence; quite disturbing to realise it abounds within the technical world.

At least he's capable of posting some pretty good photographs.

Rob

digitaldog

  • Sr. Member
  • ****
  • Online Online
  • Posts: 20652
  • Andrew Rodney
    • http://www.digitaldog.net/
Re: DNG puzzle
« Reply #78 on: October 15, 2018, 05:33:17 pm »

And you should both know the danger in assumptions...
Its about all you can do when the question asked are not answered (for whatever reasons).
Jeff tried.
His pretty good Photographs are where?
Logged
http://www.digitaldog.net/
Author "Color Management for Photographers".

Telecaster

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 3686
Re: DNG puzzle
« Reply #79 on: October 15, 2018, 05:55:46 pm »

So much barely suppressed violence; quite disturbing to realise it abounds within the technical world.

My first exposure to online raging came courtesy of the OS/2>—<Windows (aka IBM>—<Microsoft) flame wars of the late 1980s. (As discussed a bit elsewhere in LuLa Land recently.) IMO it has more than a little to do with the ability to create clockwork software worlds where everything unfolds according to design. From experience I can say there's a seductive, even intoxicating, aspect to this. By contrast the messy real world of human behavior & interaction can appear even more infuriating than otherwise.

-Dave-
Logged
Pages: 1 2 3 [4] 5 6   Go Up