Luminous Landscape Forum

The Art of Photography => Rantatorials [ARCHIVED] => Topic started by: torger on August 25, 2015, 10:25:50 am

Title: Yet some DNG comments (from a raw software developer)
Post by: torger on August 25, 2015, 10:25:50 am
I'd like to make some comments on the DNG format, from a raw software development perspective. I've been involved in writing Lumariver HDR, a DNG-supporting HDR software, longtime contributor to RawTherapee, an open-source raw conversion software which supports DNG, and now my own personal project DCamProf, a camera profiling software that can do both DNG profiles and ICC profiles for Capture One and others. So I know quite a bit of the internals of the format.

I've also reverse-engineered parts of the Phase One IIQ format, Leaf .mos format, Hasselblad 3FR format, and Kodak DCR format. Kodak was one of those companies that actually went out of business without ever releasing any documentation for their format by the way.

It's a popular view on the internet that it's easy to reverse-engineer raw formats and that we therefore DNG is unnecessary, just wait for some guy on the internet (someone like me) to reverse engineer. Well, reverse-engineering is an utmost waste of programming talent, and you can rarely be fully sure that you got it exactly right. In many cases you leave some aspects of the format unsupported. I certainly don't enjoy reverse-engineering, but as a MFD user and open-source user I've had to reverse-engineer just to be able to use my own gear.

Then we have the DNG format. I'm actually a little bit disappointed with it, from a technical aspect. It's not ideally composed for wide adoption in the industry, many aspects of it make it look like an Adobe-specific format rather than something anyone can adopt.

One rather provoking aspect is that many tags in the DNG spec are only weakly described in the documentation and are clearly just parameter settings for Adobe Camera Raw, such as the green split tag, which says: "This tag specifies, in arbitrary units, how closely the values of the green pixels in the blue/green rows track the values of the green pixels in the red/green rows." You can't have "arbitrary units" if you expect standard support, every tag must be precisely defined! There are many more examples of this.

The DNG format does not support much camera calibration data, frequently used in medium format camera formats such as IIQ and 3FR, things like flatfield correction. Adobe's own DNG converter solves this by cooking the file -- irreversibly applying the calibration data in the conversion process -- which is incompatible with some scientific applications where you indeed want the calibration data separate.

With the DNG format comes DNG camera profiles (DCP). These too have some issues:

- Can't alter hue/saturation of neutral axis (ICC can)
- LUT can't alter value of neutral axis (not really a problem of the format, but a problem of the ACR pipeline)
- Can't handle hue discontinuity (not really a problem of the format, but a problem of the ACR pipeline)
- Baseline exposure split between DNG file and DCP. This means that the camera profile can't specify the exposure offset on its own, but must relate to what tag the DNG converter has set. This is not good design.
- Blackrender not clearly defined how it affects rendering. A camera profile format that has tags of undefined meaning, and this is a central one -- not acceptable for a standard format.
- Matrices must be white-point preserving (generally what you want, but there are other ways to optimize a matrix)
- Color space cropped to ProPhoto primaries, which clips some of the human gamut (can be issue for scientific applications)
- White balance model built into the profile format, it's not a bad model but it's not so wise if you want others to adopt the format (which of course already have their own white balance models)

The whole DNG profile format is designed to work in an Adobe Camera Raw conversion pipeline with a ProPhoto primary RGB working space, and of course the given white balance model. I think Adobe should have excluded this from the format, or at least separate it more clearly so you could use DNG but instead use a different color conversion pipeline. You can of course by ignoring parts of the document, but if Adobe had wanted this to be adopted by others they shouldn't have integrated their own color model so deeply into the format.

That DNG profiles can't alter the neutral color, like making a Sepia look, Adobe has explained with that DCPs are intended for colorimetric corrections, not "looks", but that's not really true. Adobe don't make colorimetric profiles themselves, as soon as the tone curve is applied (all Adobe's profiles have tone curves, embedded or implicitly) the colorimetric idea is broken. They have indeed also added a LookTable (good!) which is intended to apply subjective looks. They should update their format to fully adopt the flexibility of both colorimetric and subjective renders that ICC profiles already have.

These are quite many issues, and I think Adobe would have to work with these in order to make the format more widely useful, rather than just being a format for Adobe Lightroom.

So my conclusion is that yes, I love the attempt to make DNG a widely adopted standard and I really don't like reverse-engineering formats, but DNG is far from perfect and I really think it needs more work to make it easier to adopt by others than Adobe themselves, both in terms of format but also the documentation.

Now the DNG format is like "here's how we do it (we didn't document it all though), take it or leave it". By putting some effort into it I think you can make a standardized raw format that can work for all the variety of raw conversion pipelines out there, and it's a zillion times better than having lots of different proprietary digital formats. But it's not only the others fault. We can't really demand manufacturers to adopt an Adobe-specific format, and today I think DNG is too much of that thing.

End of rant.
Title: Re: Yet some DNG comments (from a raw software developer)
Post by: AlterEgo on August 25, 2015, 10:54:16 am
With the DNG format comes DNG camera profiles (DCP).
I think that it is not correct to mix (as in put together) DCP with DNG in a rant... nobody forces a raw converter to use DCP model... you can use whatever you wish and DNG container can store ICC container(s).
Title: Re: Yet some DNG comments (from a raw software developer)
Post by: digitaldog on August 25, 2015, 10:58:25 am
No Rant that I see, just an opinion based on facts (unless other's who know than I do about this have a counter set of facts). Thanks.

I think the points about how DNG as it exists today isn't ideal is a better and more useful tactic to result in change then the anti-Adobe, anti-DNG real rants we here around the web on a regular basis. How can DNG be improved, not dismissed is far more useful to discuss. So thanks for that.
Title: Re: Yet some DNG comments (from a raw software developer)
Post by: torger on August 25, 2015, 11:16:50 am
I think that it is not correct to mix (as in put together) DCP with DNG in a rant... nobody forces a raw converter to use DCP model... you can use whatever you wish and DNG container can store ICC container(s).

Yes, it's a fair point. But I think the documentation is written in such a way that DCP and DNG is overlapping and the line is not at all clear.

I can agree that this aspect is more about fixing the documentation, rather than the format. Clearly separating what is providing the raw samples and calibration data etc, and then have DCP with their matrices, colorspaces and baseline offsets even in a separate document. Now all tags, belonging to the color model or not are just listed as "DNG tags".
Title: Re: Yet some DNG comments (from a raw software developer)
Post by: torger on August 25, 2015, 11:30:00 am
No Rant that I see, just an opinion based on facts (unless other's who know than I do about this have a counter set of facts). Thanks.

I think the points about how DNG as it exists today isn't ideal is a better and more useful tactic to result in change then the anti-Adobe, anti-DNG real rants we here around the web on a regular basis. How can DNG be improved, not dismissed is far more useful to discuss. So thanks for that.

I agree with that. I would certainly not say that because DNG has some issues it's great with proprietary raw formats, because it's not. Proprietary raw formats is just lazy, not fair to the customers, and not responsible in terms of archival.

When reverse-engineering the 3FR I as always asked for the spec, and I actually got a reply (there are some really nice folks at Hasselblad), but the reason they didn't want to share it was not because the format was really secret, but that they wanted to have the freedom to change their format with new camera models without having to spend resources on documenting and inform third parties (indeed if I want to support the new H5D-50 with the Sony sensor I have more reverse-engineering work to do...). Today they only share with Adobe so they can do the 3FR format in the DNG converter.

Indeed using a standard format can be more work for the manufacturer, at least for a starter. Still there are a few players that has already adopted the DNG format, Pentax, Sinar, Leica, probably some more. So I think DNG has potential, but I'd love to see some more work from Adobe to "shape it up" to be easier to adopt and look less Adobe-specific.
Title: Re: Yet some DNG comments (from a raw software developer)
Post by: sandymc on August 25, 2015, 11:42:27 am
Agree with Torger.

What I would add is that the problem, and I think the reason why we're unlikely to see it moving closer to a standard, is that DNG isn't really a raw format. It's closer to being Adobe's intermediate format. So when e.g., ACR loads a raw file, that raw file is converted to a representation that's consistent for all raw formats. That consistent representation, in a form that can be written to disk, is DNG.

The end result is that while DNG is very useful - a number of my products use it, both as an input and an output - in some situations it doesn't easily fit.

Sandy
Title: Re: Yet some DNG comments (from a raw software developer)
Post by: AlterEgo on August 25, 2015, 11:52:07 am
but the reason they didn't want to share it was not because the format was really secret, but that they wanted to have the freedom to change their format with new camera models without having to spend resources on documenting and inform third parties

exactly the point which "pro DNG" party prefers to ignore ... the issue is not the format, but that there is no reason to document or rather a reason not to document and have the ability to change things at will and w/o informing 3rd parties.
Title: Re: Yet some DNG comments (from a raw software developer)
Post by: AlterEgo on August 25, 2015, 11:53:06 am
Still there are a few players that has already adopted the DNG format, Pentax, Sinar, Leica, probably some more.

and some players in proper cameras realm decided to abandon DNG in favor of their own formats - for example Samsung... as for Pentax - there were 2 companies using DNG : Ricoh and Pentax (sold to Hoya, sold to Ricoh), now there is one - Ricoh... Pentax is just a brand...


however DNG has better chances in cell phones
Title: Re: Yet some DNG comments (from a raw software developer)
Post by: torger on August 25, 2015, 11:56:15 am
It's closer to being Adobe's intermediate format.

Very good point, I should have mentioned that. After a while working with the format it becomes quite clear that DNG is a sort of dump of what Lightroom uses internally as a common representation of all raw formats. All raw converters have this type of common representation format internally, but they can be quite different converter to converter.

It's not that hard to just ignore the Adobe-specific parts and just care about the features that is needed for getting the raw samples in. However I think it's almost a bit arrogant by Adobe to have such a strong Adobe-specific tone over it, as if they have not really asked themselves "what would we need to do to make this format acceptable by many?".

If I would be another strong player in the industry I would certainly be a bit provoked by that. For example if I would be Phase One I would of course think Capture One has the best color, and I would not like to get deep into a format which is tainted by Adobe's ideas of how colors should be rendered, which is drastically different from how Capture One does it.
Title: Re: Yet some DNG comments (from a raw software developer)
Post by: AlterEgo on August 25, 2015, 11:58:46 am
I think it's almost a bit arrogant by Adobe
or rather certain people there  8)
Title: Re: Yet some DNG comments (from a raw software developer)
Post by: Schewe on August 25, 2015, 02:51:40 pm
It's not that hard to just ignore the Adobe-specific parts and just care about the features that is needed for getting the raw samples in. However I think it's almost a bit arrogant by Adobe to have such a strong Adobe-specific tone over it, as if they have not really asked themselves "what would we need to do to make this format acceptable by many?".

That's not Adobe's doing...it's Thomas Knoll's doing. He personally oversees the development of DNG (with a hands off approach by Adobe) and Thomas does always try to do the right thing (he really does) but Adobe releases DNG for free so it's totally understandable that DNG has a Camera Raw pipeline centric approach. I'll ping Eric Chan about this thread (he doesn't post here lately). But he can pass along your "rant" to Thomas. It's useful to get real feedback about technical DNG issues (rather than political issues). Thomas has, in the past, made changes based on useful feedback.
Title: Re: Yet some DNG comments (from a raw software developer)
Post by: Telecaster on August 25, 2015, 03:41:26 pm
Proprietary raw formats is just lazy, not fair to the customers, and not responsible in terms of archival.

+ (somewhere near infinity on this) 1

-Dave-
Title: Re: Yet some DNG comments (from a raw software developer)
Post by: torger on August 25, 2015, 04:48:46 pm
That's not Adobe's doing...it's Thomas Knoll's doing. He personally oversees the development of DNG (with a hands off approach by Adobe) and Thomas does always try to do the right thing (he really does) but Adobe releases DNG for free so it's totally understandable that DNG has a Camera Raw pipeline centric approach. I'll ping Eric Chan about this thread (he doesn't post here lately). But he can pass along your "rant" to Thomas. It's useful to get real feedback about technical DNG issues (rather than political issues). Thomas has, in the past, made changes based on useful feedback.

I've only done raw conversion work for a few years, so I haven't followed DNG from start and really don't know the politics behind it, why they released it etc. I just thought that since they released it, it would be with the intention to make it a standard, but maybe it was for some other reason. In any case, if one wants it to be a standard, having it Adobe-centric is a bit problematic. One could say it's sort of a political issue, or shall we say diplomatic issue.

There are some real technical issues too though, should I provide serious feedback directly to Thomas Knoll I would need to sort it out a bit more to really think what the most important are.

Documentation first, that some tags are left with incomplete descriptions so that only Adobe's software can interpret them is the most serious aspect. As said I also think one should clearly separate out the color rendering aspects, which is the Adobe-centric stuff.

When it comes to the technical issues there may be differing opinions on what is the right thing to do.

I've most recently worked with the DCP part, so there is where I have the deepest technical insight.

- I think the baseline exposure should only belong to the DCP, not be split between DNG and DCP. I can't figure out a good reason to have it the way it is now. Now you can'ẗ specify the baseline in the DCP without knowing what the DNG writer (which may be some other software than Adobe's!) has added as baseline exposure.
- I think the pipeline should be more accepting on LUTs, as it is now it requires neutral axis to have 1,1,0 values, actually ACR ignores those LUT entries and have it hard-coded to 1,1,0. I also think that the pipeline should be able to handle discontinuity in the hue rotation which would allow more flexible hue control. When designing subjective look tables one can quite easily run into the hue discontinuity problem.
- Blackrender behavior must be precisely defined so other can replicate it, otherwise only Adobe can interpret those DCPs.
- I would rather have seen an unbounded floating point Lab space for the LUT rather than Prophoto RGB-HSV, but that is a quite big change... maybe just adding a new table type which is a basic 3D LUT just like in ICC profiles. Anything in anything out.
- Although it doesn't really hurt, one should know that having a tone curve embedded in the profile doesn't make that much sense. To make a sane color appearance result you cannot rely on a simple RGB-style curve (and using Adobe's RGB-HSV curve makes your look rather Adobeish which few except Adobe wants). You need to apply the curve in the LUT so you can have a more advanced tone reproduction operator, but since you can't scale the neutral axis in the LUT you need to employ the curve anyway, and then correct for its "wrongdoings" in the LUT, it works, but profile design becomes kind of strange. If I remember correctly it's actually not specified in the docs how the tone curve should work, but you can find it in the DNG SDK source code.

I really like the dual LUT approach, one for colorimetric corrections done pre-exposure (HueSatMap) and one for subjective looks (LookTable) applied post exposure/fill light etc. I think that split should be more clearly defined in the docs.
Title: Re: Yet some DNG comments (from a raw software developer)
Post by: digitaldog on August 25, 2015, 04:52:22 pm
There are some real technical issues too though, should I provide serious feedback directly to Thomas Knoll I would need to sort it out a bit more to really think what the most important are.
I suggest you post here, can't hurt and I'm pretty sure, certainly though Jeff, Thomas will get those data points.
Title: Re: Yet some DNG comments (from a raw software developer)
Post by: Schewe on August 25, 2015, 05:03:53 pm
There are some real technical issues too though, should I provide serious feedback directly to Thomas Knoll I would need to sort it out a bit more to really think what the most important are.

Yep...that would be the most efficient method of bringing about any changes :~)

I've already pinged Eric so any additional thoughts could also be posted here...
Title: Re: Yet some DNG comments (from a raw software developer)
Post by: torger on August 26, 2015, 03:10:04 am
As DNG involves both storing the raw samples and actual rendering, a good test to see if it can work as a standard is if a third party can based on the docs implement a pipeline that renders the same result.

There are currently a few challenges concerning this:


When it comes to the DNG as a raw container I don't have much to complain about (except for the docs on some tags), except for that I think it should have better support for calibration data. Being able to store IIQ and 3FR calibration data should be a good tester. I don't think it's ideal like now that the DNG converter applies the calibration data on the raw samples instead of storing it on the side as the original formats.

The DNG format has ICC support with the AsShotPreProfileMatrix and AsShotICCProfile tags, but I'm not sure if anyone is using them? In any case it needs to be more clearer defined how the pipeline should work in this case. Unfortunately ICC for cameras is in a sad state, as raw converters apply them so differently, so it would be hard to provide a set of tags that would work for all currently available raw pipelines. Capture One has one of the more elaborate ways, and actually two different ways, their own, and Leaf's method they inherited. Pre-matrix, curve, then exposure, fill light etc, and finally the ICC profile in the very end, and the ICC profile applies a gamma too. I don't think Capture One is a good role model in any case, whose design comes from the integer math days.
Title: Re: Yet some DNG comments (from a raw software developer)
Post by: BrianVS on August 26, 2015, 08:36:33 am
http://www.digitalpreservation.gov/formats/content/tiff_tags.shtml

I used to do a lot of data acquisition and image processing work when this stuff was new. NATO standard image transfer format was around, but it was by-and-large unique formats for each sensor. Now, writing a custom demosaic program for the Leica is just for fun. All I needed to do this for the Leica M8, M9, and M Monochrom was found in the page linked to above. It seems that most "raw" formats are TIFF 6.0 with proprietary extensions. The definitions for these proprietary extensions are not as well documented as DNG.

Kodak was happy to describe their DCS data format over the phone in 1993. Did not take much code to implement it.

It took 30 minutes to look at the NEF files from the Df and write code to make LR4.4 think it was a D4. ASCII "f" to ASCII "4". Stupid that anyone had to wait for an upgrade to LR to process Df files. Meaningless changes to Raw files to force an upgrade.
Title: Re: Yet some DNG comments (from a raw software developer)
Post by: torger on August 26, 2015, 10:35:15 am
The proprietary formats are riddled with these tiny silly upgrades, and requiring meta data from the side. Most irritating for us software developers.

In the open source world Dave Coffin with his DCRaw has made is still doing a massive contribution when it comes to keeping the proprietary formats accessible. I don't know what's going to happen when he stops, it will be a major blow to open source and many of the smaller commercial companies that rely on DCRaw for import.

Still many formats supported by DCRaw are only partially supported. There may be special shooting modes or subtle calibration data which is not applied etc. With the Kodak DCS it was already supported by DCRaw when I had a look at it, but there was some aspects around white balance I think that was not working properly.

The base structure of a proprietary format is usually simple, indeed all I know of are TIFF-based, but to support all tags for all models in all shooting modes can be a massive job, and what we usually end up with is a partially supported format. With the IIQ the open source world can't do compressed mode, can't do sensor+ mode, with the 3FR there are a number of models whose calibration data has not been reverse-engineered -- and I can guarantee that those aspects are not a 30 minute job :-/.

So indeed it would be great with a wider adoption of DNG.

As an added feature to DNG for archival I'd like to see that the camera SSF (=color filter response) could be embedded in tags, that would mean that raw converters could generate profiles on the fly, so you could get support for a new camera directly without even having a profile made for it. There are many ways to render color, but if you have the spectral sensitivity functions you have the source to do in any way you want.
Title: Re: Yet some DNG comments (from a raw software developer)
Post by: AlterEgo on August 26, 2015, 10:46:39 am
There may be special shooting modes or subtle calibration data which is not applied etc.
which DNG does not solve either... solution is not DNG, but documenting of whatever format is being used, that's it... and it is already shown that (dSLR/dSLM) manufacturers do not want or do not have compelling reasons to document...

PS: Ricoh/Pentax - what about "DNGPrivateData" tag in their DNG ?
Title: Re: Yet some DNG comments (from a raw software developer)
Post by: sandymc on August 26, 2015, 11:51:22 am
In the open source world Dave Coffin with his DCRaw has made is still doing a massive contribution when it comes to keeping the proprietary formats accessible. I don't know what's going to happen when he stops, it will be a major blow to open source and many of the smaller commercial companies that rely on DCRaw for import.

Dave is amazing. Hopefully, if he ever gets tired of doing DCRaw single handed, DCRaw could be formed up into a proper open source project, or perhaps e.g., LibRaw could take a more active role in developing new camera support. I for one would be happy to be part of that.

Sandy
Title: Re: Yet some DNG comments (from a raw software developer)
Post by: digitaldog on August 26, 2015, 11:53:50 am
Dave is amazing. Hopefully, if he ever gets tired of doing DCRaw single handed, DCRaw could be formed up into a proper open source project, or perhaps e.g., LibRaw could take a more active role in developing new camera support.
This begs the question, why should he have to do all this hard work in the first place?
Title: Re: Yet some DNG comments (from a raw software developer)
Post by: torger on August 26, 2015, 12:03:55 pm
which DNG does not solve either... solution is not DNG, but documenting of whatever format is being used, that's it... and it is already shown that (dSLR/dSLM) manufacturers do not want or do not have compelling reasons to document...

PS: Ricoh/Pentax - what about "DNGPrivateData" tag in their DNG ?

My background is in network communications and telecom. The internet today is founded on open well-documented standards. I know this is politically harder to achieve in the photography world as there are not the same commercial driving forces to actually be able to exchange data. In any case I'm not going to adopt the idea that hundreds of slightly different proprietary raw formats, documented or not, is a good solution. Although not perfect, DNG is the only attempt to actually get closer to something standardized, and there are already a few adopters.

To make it a real standard we can't have DNG managed by one person on Adobe, it must of course get into a standards body. But we can work from where we are now. Adobe stuff has ended up as standards before, such as the relative colorimetric black point compensation algorithm.

I have not been much involved in the "flame wars" that usually seems to arise around DNG vs proprietary raw, and they're still a mystery to me. How can you not want a standard as a photographer? I think it's fair to argue that DNG specifically may not be up to it in it's current state, but striving to get to a standard is a noble cause and noone which cares about photography should think that is a bad idea. It seems to me that some people don't like the idea of DNG is more about that they don't like Adobe as a company or their licensing policies or whatever rather than they hate standards for real. Those that do hate standards for real should be forced to work as an engineer and integrate various systems without any standard protocols. I'm sure they will change their minds after a while.

It's not that hard for the camera manufacturers to strive for a standard, even help out Adobe in refining DNG. The overhead of managing a standard format should be minimal compared to the cost of developing a new camera, and in the longer term you could actually save money. The only way to make it happen is if photographers start asking for it though, for the camera makers are in many ways really really conservative.
Title: Re: Yet some DNG comments (from a raw software developer)
Post by: digitaldog on August 26, 2015, 12:13:40 pm
To make it a real standard we can't have DNG managed by one person on Adobe, it must of course get into a standards body.
In the works I'm told. But that takes far too long but again, in the works.
Quote
I have not been much involved in the "flame wars" that usually seems to arise around DNG vs proprietary raw, and they're still a mystery to me. How can you not want a standard as a photographer?
I've asked that question numerous times in the "flame wars" and as yet haven't received an answer from the photographer’s side, only the manufacturers side through someone who is presumably a 'photographer'.
Quote
I think it's fair to argue that DNG specifically may not be up to it in it's current state, but striving to get to a standard is a noble cause and noone which cares about photography should think that is a bad idea
Many of us agree. Some don't. Which makes it more difficult to move this all forward. What's the old saying? You're either part of the solution or part of the problem.
Quote
It seems to me that some people don't like the idea of DNG is more about that they don't like Adobe as a company or their licensing policies or whatever rather than they hate standards for real. Those that do hate standards for real should be forced to work as an engineer and integrate various systems without any standard protocols. I'm sure they will change their minds after a while.
I can't see how it would be otherwise so I violently agree with you.
Quote
It's not that hard for the camera manufacturers to strive for a standard, even help out Adobe in refining DNG.
As I've said countless times, this isn't a technology issue, it's a political issue. Which makes it more difficult to accept so called photographers auguring for the manufacturers and not those of us that simply want control over OUR data.
Title: Re: Yet some DNG comments (from a raw software developer)
Post by: torger on August 26, 2015, 12:20:58 pm
This begs the question, why should he have to do all this hard work in the first place?

He doesn't do all hard work today, I'm one of those that sends him patches once in a while and I'm sure others help out too.

A very much respect Dave's work and very grateful about it, but it's not all good that he has the "central power". A key problem is that he has a very special coding style as seen in DCRaw, and I mean very special, which means that the code is very hard to get into and maintain and expand except for Dave himself. He thinks himself the exact opposite that his coding style is ideal, and it probably is if you're some kind of programming genius, but most of us aren't.

Although I thoroughly document my patches document what I have figured out and what is still open issues, Dave removes all comments when he includes it in a release (because if you're a programming genius you don't need comments ;) ).

So having it all concentrated in DCRaw is not an ideal situation for collaborative work.

To make it work as a broader collaborative open-source project I think the code should be restructured from scratch, and the code should be richly commented etc so it's clear what's supported and not, what is sure and what is guessed etc, and it should exist in the open at github or similar.

Maybe the libraw guys could step in at some point, I don't know. I think few can reverse-engineer camera formats as good as Dave can, and as long he is number one, DCRaw will be the reference.

When I reverse engineer a format it's with feelings of anger and frustration and my driving force is the obsession with "no f*****g manufacturer is going to lock in my image data". I do hope that Dave is one that actually enjoys the challenge though, otherwise he has a living hell :)
Title: Re: Yet some DNG comments (from a raw software developer)
Post by: AlterEgo on August 26, 2015, 12:26:26 pm
In any case I'm not going to adopt the idea that hundreds of slightly different proprietary raw formats, documented or not, is a good solution.

you are certainly entitled to your opinion

there are already a few adopters.

the sad fact is that those few adopters in dSLR/dSLM market are in place for ages (so "already" is not a proper word here - unless for a purely rhetorical purpose) and they are still - few... and for a good reason, the manufacturers either simply do not have a reason or do not want to have the hands tied/keep competition aware/etc.


How can you not want a standard as a photographer?

as a photographer (which I am not - being an amateur does not count) I do not care actually... otherwise I simply understand the reason why manufacturers do not want to use DNG or do not want to publish documentation publicly - but may as well share the info off the record/under NDA with for example Adobe itself or others... and I am hoping for a simple (for manufacurers) solution that is they will eventually get to the point where documenting their own formats will be beneficial for them, at least post factum (after release of a camera)... that's more realistic to expect... but for as long as the likes of pro-DNG people here will continue to buy cameras w/o DNG, for as long as the likes of Adobe will continue to support prop. formats and for as long as the masses will be using Adobe converters despite the fact that Adobe supports prop. formats (yes, that is actually the reason) then camera manufacturers are simply correct in that they do not need to bother.... I might not like it, but at least I am not hypocrite (hello guys).
Title: Re: Yet some DNG comments (from a raw software developer)
Post by: AlterEgo on August 26, 2015, 12:30:16 pm
It's not that hard for the camera manufacturers to strive for a standard, even help out Adobe in refining DNG.

not hard - but no reasons, more expenses, need to share developments with competition, hands are tied to implement changes fast... so ?

The overhead of managing a standard format should be minimal

there is no overhead not to do that... so ?

Title: Re: Yet some DNG comments (from a raw software developer)
Post by: AlterEgo on August 26, 2015, 12:39:05 pm
I've asked that question numerous times in the "flame wars"

and you were asked numerous times in the "flame wars" what Panasonic was supposed to do when they introduced additions related to software optics corrections in their raws... disclose the move to competition in advance ? lose money waiting (with the release of their new cameras) for Adobe to implement changes in DNG - do you recall how many month did it take for Adobe to support that in DNG ? and that was when changes in DNG are basically one Knoll-man decision.. now imagine changes when there is ISO deciding ? you never answered... for a reason - you simply do not have an answer
Title: Re: Yet some DNG comments (from a raw software developer)
Post by: AlterEgo on August 26, 2015, 12:41:52 pm
so I violently agree with you.
how about not buying a camera w/o DNG or not buying Adobe or any other raw converter that dares to support prop. formats ? put your money where your violent mouth is  ;D
Title: Re: Yet some DNG comments (from a raw software developer)
Post by: AlterEgo on August 26, 2015, 12:50:38 pm
When I reverse engineer a format it's with feelings of anger and frustration and my driving force is the obsession with "no f*****g manufacturer is going to lock in my image data". I do hope that Dave is one that actually enjoys the challenge though, otherwise he has a living hell :)

good feelings... I only can add that users of ACR/LR/C1/etc apparently are OK that "f.......g" Adobe/PhaseOne/erc lock their hard work on parametric raw conversion adjustments... so I 'd assume you only use open converters like rawtherapee, which does not do that... OR ?

actually for me that "f.......g" Adobe's (or PhaseOne's) policy matters much more than prop. raw formats  ;D... yet I understand the reasons why they want to not to have any open standard on access to parametric raw conversion adjustments (that is how to interpret them)... what's your stand on that ?
Title: Re: Yet some DNG comments (from a raw software developer)
Post by: sandymc on August 26, 2015, 01:03:14 pm
I do hope that Dave is one that actually enjoys the challenge though, otherwise he has a living hell :)

Yes, well, I think he would have to enjoy it in order to have done what he does for so long. But it would be better if DCRaw was a least broken out into a library and a main program. Even better if it was modularized.

But it's always going to be problematic. E.g., the raw engine I use for PhotoRaw/Accuraw is multithreaded, and optimized to minimize memory usage, both of which add a lot of complexity. If I were to be honest, I doubt that DCRaw (or LibRaw/any open source venture) would be willing to deal with that kind of complexity - for open source purposes, it's just not necessary.

Sandy
Title: Re: Yet some DNG comments (from a raw software developer)
Post by: AlterEgo on August 26, 2015, 01:08:02 pm
Yes, well, I think he would have to enjoy it in order to have done what he does for so long. But it would be better if DCRaw was a least broken out into a library and a main program. Even better if it was modularized.

but libraw, for example, already does that kind of refactoring...

I doubt that DCRaw (or LibRaw/any open source venture) would be willing to deal with that kind of complexity

even libraw code-man said a few days ago "А остальное - параллелим (потому FastRawViewer - fast), но не в опенсорсе."
Title: Re: Yet some DNG comments (from a raw software developer)
Post by: sandymc on August 26, 2015, 02:05:14 pm
but libraw, for example, already does that kind of refactoring...

Indeed. But the libraw guys are even slower to put out updates than Dave is! ( No insult to them - open source libraries are really hard)

Which is to say, I'd be very happy if there were a raw library that worked to commercial standards - clean, modular code, multithreaded, frequent updates. And I'd contribute code. But the incentives aren't really there for that happen.

BTW, on the topic of DNG, it would be entirely possible to integrate the DNG SDK with the front end of DCRaw. A major undertaking to pick apart DCRaw, certainly, but entirely doable. And the DNG SDK has multithreading hooks built in. But of course, licensing would be a problem.

Sandy
Title: Re: Yet some DNG comments (from a raw software developer)
Post by: AlterEgo on August 26, 2015, 02:12:37 pm
But the libraw guys are even slower to put out updates than Dave is!
public speed (or pace, rate) of updates : https://github.com/LibRaw/LibRaw/commits/master
Title: Re: Yet some DNG comments (from a raw software developer)
Post by: torger on August 26, 2015, 02:31:56 pm
AlterEgo, I think you're on the border of trolling.

As much as I enjoy the politics side of it, I don't want the technical aspect to get lost here. Even if DNG won't ever be a standard a see gains in my coding projects if the format and/or documentation is improved, as I must support both DNG and DCP in them.
Title: Re: Yet some DNG comments (from a raw software developer)
Post by: AlterEgo on August 26, 2015, 02:38:08 pm
I don't want the technical aspect to get lost here.
OK, let us stay on technical aspects - we will see how next will post anything non technical here  ;)
Title: Re: Yet some DNG comments (from a raw software developer)
Post by: digitaldog on August 26, 2015, 02:38:22 pm
AlterEgo, I think you're on the border of trolling.
Par for the course and the reason he's on my very short ignore list.
Please continue! Lots of excellent information thus far.
Title: Re: Yet some DNG comments (from a raw software developer)
Post by: AlterEgo on August 26, 2015, 02:48:45 pm
Par for the course and the reason he's on my very short ignore list.
Please continue! Lots of excellent information thus far.
so who is trolling here  :D ?
Title: Re: Yet some DNG comments (from a raw software developer)
Post by: BrianVS on August 26, 2015, 06:37:17 pm
TIFF-EP is set by the ISO body; no one seems to use it as you have to pay $300 or so to get a PDF of the standard.

Many companies spend a lot of money putting representatives into these bodies to gear the "international standard" towards their products. Telcos standards are infamous for this. Not all adopted standards are without problems, and some should have been thought out better. The "Ones-Complement 16-bit checksum" used in IPV4 should have been explained in the standard, it caused a lot of problems when V4 came online. Ones complement arithmetic to a CDC programmer has a different meaning than the one used by the IPV4 author. Fun Days. I implemented it both ways and let the error analyzer sort out which was the correct interpretation of the standard.

It would be nice to see DNG used in more cameras. But- I have to ask "How does this benefit me". Being able to look at a Hex dump and write code to pull images out of it- pays well.
Title: Re: Yet some DNG comments (from a raw software developer)
Post by: Schewe on August 26, 2015, 06:58:49 pm
To make it a real standard we can't have DNG managed by one person on Adobe, it must of course get into a standards body. But we can work from where we are now. Adobe stuff has ended up as standards before, such as the relative colorimetric black point compensation algorithm.

As far as I know, Adobe has already offered DNG to the ISO to be incorporated into an upcoming TIFF-EP update. How long that takes is anybody's guess.

In terms of background, Thomas originally created DNG for the purpose handling backwards compatibility to older software–a new camera file today can be can be converted to DNG and be opened as far back as Photoshop CS2. Thomas also wrote DNG for the purposes of educating the camera companies on how to create a well formed raw file format. While Nikon and Canon have not adopted DNG, they have evaluated DNG and learned from it.

The more modern raw file formats are really close to DNG in many respects compared to the early raw file formats like Canon's original .tif and .crw. I'm not sure about Nikon–although they caused a major problem when they encrypted the white balance metadata of the D2x (http://www.cnet.com/news/nikons-photo-encryption-reported-broken/)). Dave Coffin decrypted the metadata. Ironically, the reason why Nikon encrypted the metadata was they couldn't figure out how to pass the data from the camera to Nikon Capture so they encrypted it and used a key that the camera and software could read. To solve the problem and to allow Adobe to use the metadata, Nikon released a mini SDK to decrypt the metadata after Dave had broken it :~)
Title: Re: Yet some DNG comments (from a raw software developer)
Post by: jrp on August 26, 2015, 07:30:31 pm
Here is why one photographer no longer converts to DNG https://photographylife.com/why-i-no-longer-convert-raw-files-to-dng#more-116618 (https://photographylife.com/why-i-no-longer-convert-raw-files-to-dng#more-116618)

Title: Re: Yet some DNG comments (from a raw software developer)
Post by: BrianVS on August 26, 2015, 07:51:48 pm
Most people have not shot digital cameras long enough to have a manufacturer disappear and their raw file formats no longer supported by modern software.

I keep a Win95B machine around, with 5.25" floppy drives, 3.5" floppy drives, SCSI, ZIP, JAZ, and Clik. 30 year old 5.25" Floppy disks are readable, had Landsat data on them. Moved to a newer machine now. I remember Tintypes of family members that died long before I was born in my parents box of old pictures. I wonder what our relatives get passed to them 100 years from now.

20+ years ago my wife asked my to write code to unpack TIFF 6.0 images that made use of the multiple image stored in one file feature. Photoshop 3.0 would only process the first image. Writing the code was easy as the file format was documented. That is the advantage of DNG and any other standard that documents the file format. Why manufacturers of all cameras simply do not publish their file format is stupid and vain. It's just not hard to figure out, they would just be saving some people a lot of time. I still use the software written in the late 1980s to reverse engineer file formats. It's like a visual hex dumper that you write code to process the data as you go through it. A million and one uses.
Title: Re: Yet some DNG comments (from a raw software developer)
Post by: digitaldog on August 26, 2015, 07:54:23 pm
Here is why one photographer no longer converts to DNG https://photographylife.com/why-i-no-longer-convert-raw-files-to-dng#more-116618 (https://photographylife.com/why-i-no-longer-convert-raw-files-to-dng#more-116618)
I'm not buying it  ::) There isn't one point in the piece that can't easily be dismissed. None of the advantages of the DNG format, arguably most using an Adobe product was mentioned.
But then this was supposed to be a technical discussion of the format.
Title: Re: Yet some DNG comments (from a raw software developer)
Post by: torger on August 27, 2015, 03:34:17 am
One problem which is inbetween technical and political is that Adobe Camera Raw doesn't have that good reputation concerning color rendition, while say Capture One has a good reputation, at least in the medium format segment from which I have most experience. Many assume that this is because DCP is bad and ICC is good. However, I can assure you that the problem is not DCP, but how Adobe has chosen to design the look of their color profiles. Their tone curve also gives a special highlight rolloff behavior that has quite some effect on the look, which one may like or not like. Many of the others use a pure RGB curve in the highlight rolloff which gives another look (Capture One on of them). Personally I don't think any of them is ideal.

With the LookTable 3D LUT component in the DCP this can be worked around though, and you can create an entirely different look which I do in my own DCamProf project. It would have been smoother if the LUT was a bit more flexible as discussed earlier in this thread, but it's still possible to do most things.

If we think about long-term archiving ability, I really think the camera SSF should get in there. Without SSF the future raw converters will have to rely on the color conversion algorithms used in current software, like DCP. With SSF you have the key to do any color rendition, which doesn't need to use any of the current color vision models.
Title: Re: Yet some DNG comments (from a raw software developer)
Post by: BrianVS on August 27, 2015, 05:38:18 am
I looked up "SSF" and found it was a file format for storing GPS data- then saw that you referred to it as spectral response.

One of my first jobs (The Carter Years) was writing code to convert raw imagery into radiometrically calibrated data, first from film and later from a custom-made two-color IR sensor. Having the spectral response of the film and the sensor was required for this. These were measured and stored separately from the imagery. Today, most manufacturers hold the spectral response of their sensors as "proprietary". Kodak was the exception so I have the spectral response for the KAF-10500 and KAF-18500 used in the M8 and M9. Leica published the spectral response for the M Monochrom.

How many other cameras have published spectral response curves?

http://www.awaresystems.be/imaging/tiff/tifftags/privateifd/exif/spectralsensitivity.html

Not sure if the above tiff-ep tag is sufficient, or if it was meant to store a full spectral response curve.
Title: Re: Yet some DNG comments (from a raw software developer)
Post by: torger on August 27, 2015, 08:24:08 am
I looked up "SSF" and found it was a file format for storing GPS data- then saw that you referred to it as spectral response.

One of my first jobs (The Carter Years) was writing code to convert raw imagery into radiometrically calibrated data, first from film and later from a custom-made two-color IR sensor. Having the spectral response of the film and the sensor was required for this. These were measured and stored separately from the imagery. Today, most manufacturers hold the spectral response of their sensors as "proprietary". Kodak was the exception so I have the spectral response for the KAF-10500 and KAF-18500 used in the M8 and M9. Leica published the spectral response for the M Monochrom.

How many other cameras have published spectral response curves?

http://www.awaresystems.be/imaging/tiff/tifftags/privateifd/exif/spectralsensitivity.html

Not sure if the above tiff-ep tag is sufficient, or if it was meant to store a full spectral response curve.

I was not aware of that tiff tag, interesting. Could be worth looking into.

Yes, SSF, Spectral Sensitivity Functions, is a good search term if you want to look for scientific papers on the subject.

Spectral response is usually found in the spec sheets of sensors, but one needs to measure it on the camera as the IR-filter and possibly other aspects affects the curves greatly. With a spectrophotometer, integrating sphere and monochromator you can measure it, or use a quicker method like Image Engineering's camspecs product. This is $15k gear (unless you buy old stuff off Ebay and write some own code) so it's not for the private individual, but I would not be surprised if Adobe is already using this when profiling their cameras. So it would be nice if their DNG converter would embed this measurement.

(Of course the lens modulates the SSFs too, but it's not necessary to model that aspect to start with, SSF with a standard lens gets you very far while different lenses only make minor changes on top)

The spectral response cannot be hidden away, anyone with this gear can measure it, and all the camera manufacturers have it for sure so they can measure competitors cameras if they'd like (I don't think they do, I don't see any reason why they would), so there is no purpose whatsoever to have this "secret", as the only people that you hide it from is consumers that doesn't have the measurement gear.

My own profiling project DCamProf can generate profiles directly from SSFs, tried it and it works well.

You can find some SSFs on the net published via scientific papers, but it's not widely available as SSFs haven't yet become an established form to do camera profiling. In many ways camera color is very conservative, we're stuck using models from the 1990s and there seems to be little interest in developing new ideas. Not that SSF is actually new, but computer power today has made it feasible to use more data and computing intensive models.
Title: Re: Yet some DNG comments (from a raw software developer)
Post by: sandymc on August 27, 2015, 09:46:09 am
In many ways camera color is very conservative, we're stuck using models from the 1990s and there seems to be little interest in developing new ideas.

Not that I disagree about it being time that something new should be available, but I haven't seen any models that would in practice (emphasis on "in practice" as opposed to undeniable theoretical advantages that some models offer) give the majority of users better results than what they can get now........

Sandy
Title: Re: Yet some DNG comments (from a raw software developer)
Post by: Bart_van_der_Wolf on August 27, 2015, 11:41:14 am
My own profiling project DCamProf can generate profiles directly from SSFs, tried it and it works well.

You can find some SSFs on the net published via scientific papers, but it's not widely available as SSFs haven't yet become an established form to do camera profiling.

Hi Anders,

I haven't gotten around to doing it myself yet, it's somewhere on my Todo list (for later this year?), but I'm wondering if we can roll our own SSFs on a budget and get good enough results. Especially with the difficulty of shooting Color targets with reflective patches, it should be possible to do better optically/spectrally when using something like the "Star Analyser 100 (http://www.shelyak.com/produit.php?id_produit=6&id_rubrique=4)" and make an image of a light slit.

The light slit will be broken up in a nice rainbow of spectral colors by the diffraction grating, and when the Raw data (in linear gamma) is analyzed, a list of relative luminances by wavelength should become available (after some calibration, e.g. on the Fraunhofer lines of atmosphere filtered daylight). I know that the concept is easier in theory than executed in practice, but these seem challenges that can be solved for relatively little money.

I think that the SSF path is potentially much more reliable than shooting variable ColorCheckers with a lot of surface reflection issues.

Cheers,
Bart 
Title: Re: Yet some DNG comments (from a raw software developer)
Post by: AlterEgo on August 27, 2015, 01:18:39 pm
get good enough results.
for me good enough (with Sony A7) was in the end shooting just passport with matte patches (measuring it & measuring the actual illumination) and approximating SSF/CMF using a matlab script provided by people from RIT... granted not as good as elaborate designs with monochromators, etc discussed here (and elsewhere) recently, but for a solder-challenged me that's a solution.
Title: Re: Yet some DNG comments (from a raw software developer)
Post by: torger on August 27, 2015, 02:25:12 pm
Not that I disagree about it being time that something new should be available, but I haven't seen any models that would in practice (emphasis on "in practice" as opposed to undeniable theoretical advantages that some models offer) give the majority of users better results than what they can get now........

Sandy

Yes I think you're right. Doing anything new the first attempt would probably provide worse results than the current highly mature methods. If I had an established commercial product with many users I would be very very very careful to change anything that already works well.

What I'm thinking of improvements in this case is more automation, and less hand-tuning of profiles. However some sell the converters very much because of their hand-tuned profiles, the medium format manufacturers is the prime example I think.

What I'd like to see is profiles without those subjective tunings, just really scene-referred colorimetric, and then you would use spatially varying color appearance models to actually render the color image from that. There's 10+ years of research done in the academic world of this already, but no broadly established raw converter provides a possibility to work this way. I can't say for sure that it would work well though, but if I had an infinite amount of time I'd probably do some work in that direction.

DCP is often said to be better than ICC because DCP is scene-referred and ICC is not, but in practice DCP is just as output-referred as ICC. All Adobe's profiles apply a curve, and afaik all have subjective tunings too to provide a pleasing color appearance with that curve applied. You can't really work in Lightroom with a linear colorimetric profile and get good color appearance for general purpose photography, as all the tools to apply contrast will distort color in one way or another.
Title: Re: Yet some DNG comments (from a raw software developer)
Post by: digitaldog on August 27, 2015, 02:30:42 pm
If we think about long-term archiving ability, I really think the camera SSF should get in there. Without SSF the future raw converters will have to rely on the color conversion algorithms used in current software, like DCP.
This is an idea a pretty smart color geek named Eric Walowit has been proposing for years. If you don't know him or of him, ping me off list, I'll give you his email address.
He's on Linkedin too:
https://www.linkedin.com/pub/eric-walowit/4/4a0/793
Title: Re: Yet some DNG comments (from a raw software developer)
Post by: torger on August 27, 2015, 02:35:30 pm
Hi Anders,

I haven't gotten around to doing it myself yet, it's somewhere on my Todo list (for later this year?), but I'm wondering if we can roll our own SSFs on a budget and get good enough results. Especially with the difficulty of shooting Color targets with reflective patches, it should be possible to do better optically/spectrally when using something like the "Star Analyser 100 (http://www.shelyak.com/produit.php?id_produit=6&id_rubrique=4)" and make an image of a light slit.

The light slit will be broken up in a nice rainbow of spectral colors by the diffraction grating, and when the Raw data (in linear gamma) is analyzed, a list of relative luminances by wavelength should become available (after some calibration, e.g. on the Fraunhofer lines of atmosphere filtered daylight). I know that the concept is easier in theory than executed in practice, but these seem challenges that can be solved for relatively little money.

I think that the SSF path is potentially much more reliable than shooting variable ColorCheckers with a lot of surface reflection issues.

Yes it would be really nice with an SSF measurement method on a budget. I think the colorcheckers can work quite well though. The glare issues are very real though, I've seen that, it's extremely difficult to shoot glossy targets. When making general purpose profiles you may not want to correct lightness so much though, and if you make a 2.5D profile you don't need to measure that many dark patches, which makes it a bit simpler to do well.

One of many cool things with having SSFs though is that you can profile against a database of full spectrum measured colors, that is for example real human skin rather than a test target that tries to mimic it.
Title: Re: Yet some DNG comments (from a raw software developer)
Post by: AlterEgo on August 27, 2015, 03:07:20 pm
This is an idea a pretty smart color geek named Eric Walowit has been proposing for years.
and why 'd a smart geek mr. Knoll ignoring that for years ? when he can just add a tag to DNG specification (covering DCP profiles) at will ?
Title: Re: Yet some DNG comments (from a raw software developer)
Post by: sandymc on August 27, 2015, 03:31:41 pm
What I'm thinking of improvements in this case is more automation, and less hand-tuning of profiles. However some sell the converters very much because of their hand-tuned profiles, the medium format manufacturers is the prime example I think.

Yes, I'd agree with that. Or put another way - right now it's probably possible to make a lot of improvements without changing the color model as such, just by getting better at the camera profiling process. However I think that here ugly commercial reality get in the way; I don't think you can make much money from camera profiling software, because profiles can be copied freely.

Right now, the most companies that commercially produce camera profiling software either build very expensive professional packages, or do so as an accessory to hardware - e.g., X-Rite.

The best thing that e.g., Adobe could do for the world of camera profiling, IMHO, is to add the capability for licensing profiles to its software. That way software vendors could actually make money from selling software to build profiles, and we would see a lot of innovation.

Sandy
Title: Re: Yet some DNG comments (from a raw software developer)
Post by: AlterEgo on August 27, 2015, 04:02:12 pm
The best thing that e.g., Adobe could do for the world of camera profiling, IMHO, is to add the capability for licensing profiles to its software. That way software vendors could actually make money from selling software to build profiles, and we would see a lot of innovation.
people do buy (or pay subscriptions) Adobe products for totally different reasons nowadays... so Adobe in fact better served being a good samaritan and making it available for free.... yes.
Title: Re: Yet some DNG comments (from a raw software developer)
Post by: torger on August 27, 2015, 04:20:34 pm
The best thing that e.g., Adobe could do for the world of camera profiling, IMHO, is to add the capability for licensing profiles to its software. That way software vendors could actually make money from selling software to build profiles, and we would see a lot of innovation.

I'm not sure I understand. If profiles could be sold more easily than today, there would still be only a handful of profile producers, so the world market of profiling software would be like a handful :). Maybe we would see some inhouse profiling software, like all the manufacturers have today, in an independent profile producer. Huelight is one such individual today that sells profiles, but that's the only one I know of.

I think the profiling standstill is a little bit for the same reason of the DNG standstill -- photographers in general are not really interested. They're pleased with the current situation, and don't really have the knowledge of what's possible. I think most laymen think camera color sits 90% in the hardware and 10% in the profile, rather than the other way around which is closer to the truth. Camera makers have everything to gain from maintaining that myth too, especially those selling more exclusive cameras.

I think it's silly that cameras are made to have significantly different looks, and that you can't have a reproduction profile out of the box. How strange is it to expect that a camera can produce reasonably correct colors? The hardware can, but the profiling tradition doesn't allow it. Subjective looks is a good and convenient thing to have, but it can and should be added on top.

DCP actually has a wonderful layered approach which I use in full in DCamProf, first a matrix to get as close as possible to scene referred colorimetry, and then a LUT on top to refine that (HueSatMap tag). And then an additional LUT to add a look (LookTable tag). With RawTherepee the individual DCP components can be toggled on and off so you can choose if you want to use the colorimetric part only, with or without non-linear LUT corrections, and then toggle the look. What I've seen from Adobe's profiles they don't seem to design profiles this way themselves though, I don't know why. In ACR you can't toggle the individual parts anyway, so I guess it doesn't really matter from a user perspective, but seems strange to design the profile architecture this way and then not employ it.

An even better architecture I think would be to have just a colorimetric profile, which indeed would make all camera models look very similar, and then have look profiles entirely separate, which you can apply to any camera. The profile-does-it-all is legacy from the days when digital photography had to be packaged as an easy and familiar alternative to film photography, and choosing profile is like choosing film roll. I think we are past that period now though, and raw conversion could and should concentrate on being digital rather than mimicking an analog workflow.
Title: Re: Yet some DNG comments (from a raw software developer)
Post by: AlterEgo on August 27, 2015, 04:53:40 pm
but seems strange to design the profile architecture this way and then not employ it.

Adobe needs to design the UI to make the product usable by a lot of people, so between addition of some more (and more and more...) sliders/buttons/checkboxes and keeping it as it is ACR/LR actually is right, they gain a little market by adding those, but make UI more complex for hoi polloi... users aggravations do not worth that... RT tends to be overloaded by (useful indeed) features.

Title: Re: Yet some DNG comments (from a raw software developer)
Post by: AlterEgo on August 27, 2015, 04:56:48 pm
An even better architecture I think would be to have just
just a SSF (or multiple SSFs) and make a tab (in raw converter UI) with controls allowing you to build a profile on the fly (and save as preset)... but again - that shall be hidden from a regular Joe in order not to cause the tears.
Title: Re: Yet some DNG comments (from a raw software developer)
Post by: BrianVS on August 27, 2015, 05:29:11 pm
The problem becomes first to measure the spectral response for each sensor as implemented in the camera (accounting for IR cut filters, etc) , then to patch it into the DNG/TIFF-EP file. It might be easier to start an online database of spectral response curves for various cameras with measurements provided by end-users. Nikon, Canon, Sony, Olympus... do not provide the data so it must be measured. Anyone coming across files from the Leica M Digital cameras will have an easy time making up the curves as they are published. Same with my DCS200ir- the response curve is published. For most cameras: the curves are not published, and the cameras will be long inoperative when imagery from them is being recovered.
Title: Re: Yet some DNG comments (from a raw software developer)
Post by: torger on August 28, 2015, 03:25:59 am
just a SSF (or multiple SSFs) and make a tab (in raw converter UI) with controls allowing you to build a profile on the fly (and save as preset)... but again - that shall be hidden from a regular Joe in order not to cause the tears.

Yes, indeed. I think preset looks are good because regular Joe won't be able to design them even if they had the interest (which they 99% of the users won't have). I've worked with subjective profile design a little lately, and it's hard. I think I can do better subjective profiles than both Phase One and Hasselblad which has been my "role models" when experimenting (mostly Hasselblad), but I've not settled yet and I'd say it takes weeks of field testing before one has a look settled. And while I don't think you need super-human color vision, one need to have healthy color vision and quite some training to be able to comfortably move in the space of subtle subjective color adjustments.

Hasseblad's Phocus actually has a design where the look is separate from the profile. Their profile is not colorimetric but I'd say it's "95% realistic", that is they don't try to make any particular strong look (like most others do), and I've heard that all their cameras look almost exactly the same despite different sensor hardware (like it should be), and then they have selectable looks on top like "nature", "portrait" etc which are simply presets of the raw converter adjustments tools. It's a bit too simplistic though, I think the subjective looks require different more subtle tools than the typical adjustment tools, so it would be better with a separate panel.

Lightroom is one of the more "consumer-oriented" softwares, where ease of use is the top priority, and I think that is the best commercial strategy. 99% of the users don't care that for example Capture One has better color adjustment tools (which indeed are much more difficult to use).

With proper design it would be possible to make a modern color rendering pipeline and still maintain the ease of use though. There's just this die-hard idea that digital cameras are just the same as analog cameras, just faster to get to the print.

As a profile designer it's much you can do to make a technical progress, with DCamProf I just have to live with the rendering pipelines that are out there, which means that I must provide features to build subjective look into the profile itself although I think that is not how camera color should work. Supporting Capture One has been the worst, as their pipeline is a real old-school mess. For example they split the curve between a user-selectable curve and a fixed curve in the ICC profile (to minimize color shifts the RGB curve causes), and there's a bunch of workarounds to get decent precision out of the integer-only v2 ICC profiles. With their hand-tuned profiles they get fantastic color though, and of course as long as you hand-tune the final result the underlying pipeline can be anything. The end user only sees the final result of course, and does not care how it's implemented.

("My" raw converter, RawTherapee, is a disaster in terms of user friendliness of course, I call it an experimental box for technically interested photographers. It has lots of legacy too and many design issues, so I don't hold it up as a role model in any way... it's a free open source software with programmers that come and go and spend some of their limited spare time on, so it is what it is.)
Title: Re: Yet some DNG comments (from a raw software developer)
Post by: torger on August 28, 2015, 03:43:12 am
The problem becomes first to measure the spectral response for each sensor as implemented in the camera (accounting for IR cut filters, etc) , then to patch it into the DNG/TIFF-EP file. It might be easier to start an online database of spectral response curves for various cameras with measurements provided by end-users. Nikon, Canon, Sony, Olympus... do not provide the data so it must be measured. Anyone coming across files from the Leica M Digital cameras will have an easy time making up the curves as they are published. Same with my DCS200ir- the response curve is published. For most cameras: the curves are not published, and the cameras will be long inoperative when imagery from them is being recovered.

Yes it's a political challenge. Someone's gotta measure it, and they won't get paid for it. I think Adobe is in the best situation for doing this. Maybe they already do it (to render their profiles), and they need to support new cameras anyway as their commercial success builds on that. They get paid indirectly as they sell ACR subscriptions.

The reason to embed it in the DNG is simply to make the format truly self-contained and archival, I think it's a natural part of a "digital negative". It doesn't necessarily need to be mandatory, but I think it should be a clear and documented option for the format. If DNG ever evolves into an archival standard I think it would be a huge mistake to not have this.

The idea is that the DNG should contain all information needed to open it in a future raw converter that may use an entirely different color model than those used in raw converters today, which as discussed already feels old (at least I think so). I think it's unwise to consider DCP to be a "future-proof" format for color rendition. The SSF is the ideal future proof information.

Today we have a few archival formats for the raw converter output, TIFF, JPEG, JPEG2000, and DNG has potential to be it for the input. Although I do see it as a wet dream, I don't think it's realistic to get an archival format for parametric raw conversion though, but of those three I think it's the least important too. Output first (we already have that), and then the digital negative. When those two are established and raw conversion has matured more than it has today, there may be a possibility to actually strive for an archival format of parametric raw conversion. I'd use it.
Title: Re: Yet some DNG comments (from a raw software developer)
Post by: BrianVS on August 28, 2015, 03:21:09 pm
I was used to working with data sheets from film and later sensors to get the spectral response when converting intensity to "watts/Steradian". For the sensors, we did the spectral response measurements ourselves, stored "calibration files" and such. Such data is was not in the memory used by the digital sensors: spectral response data is not stored in the firmware of the camera. This would require that the manufactures measure the response, store the tables in non-volatile memory, and write the data to each image. I doubt they will do this. If the response curves were published, you could post-process the DNG files and patch the data in. This would require processing every file. I batch process DNG files from my Leica, very easy. The third, and most likely, back to collecting and publishing response curves for various cameras. Software would be written to read the curve and use it for color calibration.

Given the "Camera produces a Pretty Picture!" method of marketing cameras, it will be a miracle if manufacturers continue to provide uncompressed Raw image data (Sony A7 series, Leica M246, lower-end NEF, etc) in the future, let alone self-contained DNG files that have everything required to convert to real imagery.
Title: Re: Yet some DNG comments (from a raw software developer)
Post by: torger on August 29, 2015, 04:00:08 am
Given the "Camera produces a Pretty Picture!" method of marketing cameras, it will be a miracle if manufacturers continue to provide uncompressed Raw image data (Sony A7 series, Leica M246, lower-end NEF, etc) in the future, let alone self-contained DNG files that have everything required to convert to real imagery.

Yes indeed, things could be worse. That we get raw files at all rather than just JPEGs out of our cameras I would guess is one of those cases when the manufacturers actually listened to a minority of the customers. If the majority's we-don't-care attitude has been the deciding factor I think we'd still have only JPEGs.

Today raws are taken for granted and with user-friendly software like Adobe Lightroom I think even the majority uses raws. Hopefully we get to a time when a standard archival raw format is taken for granted too, but it's still a long way to go.
Title: Re: Yet some DNG comments (from a raw software developer)
Post by: amolitor on August 29, 2015, 01:23:57 pm
'The majority uses raws'?

The majority of whom? Obviously not the majority of people who take photographs. The majority of DSLR owners?

Title: Re: Yet some DNG comments (from a raw software developer)
Post by: torger on August 29, 2015, 02:16:54 pm
'The majority uses raws'?

The majority of whom? Obviously not the majority of people who take photographs. The majority of DSLR owners?



I'm speculating a bit of course as I have no real statistical information on the subject. I was thinking about the majority of professional photographers and enthusiasts.
Title: Re: Yet some DNG comments (from a raw software developer)
Post by: madmanchan on August 31, 2015, 12:52:04 pm
Hi torger and folks,

I'd like to confess that I haven't read the entire thread, but I've read the first couple of pages of posts and would like to help answer questions and clear up any confusion if possible.  I'll start by responding to a couple of items from torger's original post.  Please see inline below.

Thanks,
Eric

Quote
The DNG format does not support much camera calibration data, frequently used in medium format camera formats such as IIQ and 3FR, things like flatfield correction.

This is not true.  DNG supports many of these calibrations using an "opcode" mechanism (see Chapter 7 of the DNG 1.4 spec).  These are basic image processing directives that can be used to correct common sensor/lens issues, such as color shading.  The flat field correction you mention, for instance, can be handled using the GainMap opcode (page 94), either in Bayer space (use 4 opcodes, one per Bayer plane, in either opcode list 1 or 2) or in RGB space (after interpolation, in opcode list 3).  There are other opcodes for handling other issues such as per-tile/area scale factors (MapTable or MapPolynomial), fixing bad pixels or entire rows/columns/areas, and the like.

Quote
- Can't alter hue/saturation of neutral axis (ICC can)
- LUT can't alter value of neutral axis (not really a problem of the format, but a problem of the ACR pipeline)
- Can't handle hue discontinuity (not really a problem of the format, but a problem of the ACR pipeline)

I don't understand the concern/objection here.  I agree that the LUT as defined does not permit altering the hue/sat of the neutral axis, but I don't see sepia or warming/cooling "looks" or "rendering styles" as something to be covered by the DNG spec anyways.  My understanding and interpretation of DNG (yes, the DNG concept was long before I became involved) is that it's intended to provide enough tags to interpret sensor data, as well as to provide a useful and reproducible baseline rendering -- not to cover/describe how to do all possible renderings.

As you know, the value (brightness) of the neutral axis can be adjusted using other mechanisms, such as the optional tone curve of a profile.

Quote
- Baseline exposure split between DNG file and DCP. This means that the camera profile can't specify the exposure offset on its own, but must relate to what tag the DNG converter has set. This is not good design.

Well, I think that depends on the point of view.  If you're a camera vendor or DNG author trying to create a DNG file, you embed the profile into the DNG (so it travels with the file), and you have control over both tags.  For example, if you want no adjustment at all, just set both tags to zero (or omit both tags), and no exposure compensation will be done. 

The two tags are set up this way because it's generally not possible for a profile to "know" under what context it will be used.  For example, the BaselineExposure tag for a given camera model may vary from shot to shot (for instance, it may be ISO dependent).  Having a separate tag for the profile gives an opportunity to bias the default exposure one way or another, without having to "know" what the absolute/underlying BaselineExposure value from the file/negative is.

Quote
- Blackrender not clearly defined how it affects rendering. A camera profile format that has tags of undefined meaning, and this is a central one -- not acceptable for a standard format.

Point taken.

Quote
- Matrices must be white-point preserving (generally what you want, but there are other ways to optimize a matrix)

I don't see this as a limitation.  Preserving the white point is a useful constraint, which simply says that white has to map to white (and more generally the neutral axis should map to itself).  This leaves 6 degrees of freedom for optimizing a 3x3 matrix, and there's no restriction on how those remaining DOF can be optimized -- various samples, color metrics, etc. can be used.

Quote
Color space cropped to ProPhoto primaries, which clips some of the human gamut (can be issue for scientific applications)

True, though in practice, if it's important for an application, it can also be easily worked around (by specifying a ForwardMatrix that effectively makes the color matrix transform step a no-op, in other words preserving the native color primaries of the capture, rather than transforming it into another space).

Quote
- White balance model built into the profile format, it's not a bad model but it's not so wise if you want others to adopt the format (which of course already have their own white balance models)

I think that comes down to which aspects of the WB model you're referring to.  Certainly dealing with multiple illuminants leaves a lot of room for different methods.  Dealing with a single illuminant, with a single set of WB scale factors, is much more straightforward. 

Title: Re: Yet some DNG comments (from a raw software developer)
Post by: amolitor on August 31, 2015, 05:15:31 pm
If people spent as much time actually gathering up the known data about DNG and writing a standard around that, as they spent bitching and complaining in these threads, the standard would be done, reviewed, revised, and published.

It doesn't take an eternity to make something a Standard, unless you're in love with the idea of the CCITT or ISO or whomever owning it. It requires a small group of people to commit to doing a bit of work, and committing to sticking with it and forming a little committee with some rules and stuff. Set it up in a weekend. Writing the document would be a lot more work.

It won't make the many excellent reasons the majors have for not supporting it go away, but it would at least advance the cause slightly, which arguing on the internet won't.
Title: Re: Yet some DNG comments (from a raw software developer)
Post by: Schewe on September 02, 2015, 12:28:38 am
I'd like to confess that I haven't read the entire thread, but I've read the first couple of pages of posts and would like to help answer questions and clear up any confusion if possible.  I'll start by responding to a couple of items from torger's original post.

Eric,

Thanks for chiming in. To date, I think this is the best DNG File Format thread I've ever seen...some really good data has been exchanged and I hope it ends up being useful. In the grands scheme of things, arriving at a well documented raw file format will help the photo industry not retard it. Most of the FUD only muddies the waters and doesn't solve any issues. Kudos to all in the thread that kept it technical not political...this has been greatly needed (even if most of the tech is over my head–which I freely admit)!
Title: Re: Yet some DNG comments (from a raw software developer)
Post by: torger on September 03, 2015, 04:39:58 am
Thanks for the feedback Eric, I really appreciate it.

A few comments follows;

If the DNG format does support calibration data I would suggest to change the implementation of your DNG converter so that it embeds calibration data rather than applying it for formats like 3FR and IIQ. Users generally don't like "cooked" raws, and the only reason people haven't complained in this case is probably because they don't know it's happening.

One reason to change the neutral axis which is not related to look is if you want to use DNG profiles for reproduction photography and want to calibrate a fixed white balance setting. I also find the mix between "colorimetric scene-referred" and "subjective output-referred" a bit strange. These limitations are said to be because the format is intended to be scene-referred, but the look table, tone curve and indeed Adobe's own profiles says otherwise -- it's output-referred. When the actual use is output-referred with subjective tunings I find it odd to limit the neutral axis. With the neutral axis limited I also find it odd why these entries are included in the LUT at all, as their values must be set to 1,1,0. Adobe Camera Raw actually ignores those entries, I've tried to set them to other values but it has no effect.

You didn't comment on the hue discontinuity point. I did not explain that in detail either, here it comes. Hue is described as a circular value in the range 0-360 degrees, which means that 0 is the same as 360, +180 is the same as -180 etc. This makes the math a little tricky you need to take special care to make those additions right so you get the right result. In the DNG pipeline there is no such care taken. This means that if you have two neighboring entries in the LUT which differ more than 180 degrees, there will be an error in the calculation. Since RGB-HSV hue gets unstable when saturation is low it's not that unlikely you hit this when designing a profile, for example if the profile is designed in a CIECAM02 coordinate system. It's not too hard to work around though so it's not a huge deal, but the profile format would be easier to work with if the pipeline would have proper circular math when adding hues. If you don't want to fix this (you might not want to as it will potentially slow down the code), it should be a documented part of the standard. Now it just looks suspiciously similar to a bug ;)

I have indeed noted that some Adobe profiles use the forward matrix as a no-op, and that you essentially make all color correction in the LUT. The output space is still ProphotoRGB though so you're still limited by its boundaries. Of course you could make some own non-standard translation on top with other primaries, but it would not be the standard. So I still think it's just fair to say that DNG in its standard mode is limited to ProPhoto and is not intended to cover the full human gamut. If that's a problem or not is debatable of course. Personally I don't think it's a big problem, but I find it unlikely that a standard organization would choose this type of limitation today.

If using the forward-matrix as no-op becomes a de-facto standard to avoid clipping, maybe one should do like the ICC Lab LUT profile case, specify in the standard that the matrix should be a no-op when a LUT is available? At least it would be helpful for profile designers to present this option and why it's useful in the DNG spec.

Interesting note about baseline exposure that it's used for compensating changes caused by ISO settings. I think it should then be thoroughly documented in the DNG spec that baseline exposure should be used for that and nothing else, otherwise there's a risk that the tag is used for the wrong purpose. A wrong use would be if it's the same over the whole ISO range, or if there is a base offset that is the same (that is if it varies between 0.2 and 0.5, it should instead vary between 0.0 and 0.3 and the profile should add the 0.2). In the current specification there is nothing mentioned about variation over ISO.

I have a strong view that letting the baseline exposure be "anything" is bad, so I'm certainly not satisfied with the current status, and I think I have quite strong arguments for my view. To make it a standard you need to think more about third parties so it not just becomes an internal Adobe Camera Raw format. Breaking the dependency between the DNG raw file itself and and the DCP is an important aspect of that, and to do that the baseline exposure can't work as it does now, oh well maybe it can (I haven't checked in detail how DNG converter uses it), but the documentation must state how it should be used that is only as an ISO-correcting offset (or any other factors that make exposure vary) with 0.0 as base. My point is that it should definitely not be used as a fixed offset, that should be part of the profile. Otherwise you break interoperability.

There's another aspect that probably should be documented similar to the BlackRender part, namely how the exposure offset is converted to a tone curve, which is a good idea as far as I can see especially for negative offsets (otherwise clipped data would be brought into view). There's code for that in the DNG reference SDK, perhaps it should be documented in the spec?

I can certainly live with the white-point preserving thing, I optimize my matrices with that constraint. I'm just mentioning it because ICC doesn't have this limitation and others might have a problem with that.

Considering the white balance model it's true that you can just ignore it. It just feels like one of those things that is more like an internal Adobe thing that exists in the standard for the convenience of Adobe rather than it's useful for the standard. If it's only there for Adobe to use, I don't think it should be there. I need to think more about it though, I haven't a 100% complete view on the subject of white balance / whitepoint modeling.

One particular aspect that gives me a bit of a headache when designing profiles that people use with Adobe Lightroom is that when the user has set a custom white balance Lightroom remembers the temp/tint setting and then run it backwards through the profile to calculate the multipliers. This means that if my color matrices are not exactly the same as Adobe's profiles there will be a white balance shift (that is when you switch to my profile the tint of the image changes). So I just had to add the option to copy the color matrices from existing profiles for those users that didn't want this to happen. The problem is that estimating temp/tint with a camera matrices cannot be very exact so even if Adobe and I have the most exact methods it's likely that the result will differ several hundred degrees anyway. I haven't analyzed this in depth yet to suggest how it should be done, but I think that there is a problem currently. Any aspect that makes interoperability difficult is not good for a standard.
Title: Re: Yet some DNG comments (from a raw software developer)
Post by: rdonson on September 03, 2015, 12:06:24 pm
Eric,

Thanks for chiming in. To date, I think this is the best DNG File Format thread I've ever seen...some really good data has been exchanged and I hope it ends up being useful. In the grands scheme of things, arriving at a well documented raw file format will help the photo industry not retard it. Most of the FUD only muddies the waters and doesn't solve any issues. Kudos to all in the thread that kept it technical not political...this has been greatly needed (even if most of the tech is over my head–which I freely admit)!

Does this excellent thread point to the need for a public forum for the DNG community to exchange thoughts?
Title: Re: Yet some DNG comments (from a raw software developer)
Post by: sandymc on September 03, 2015, 12:51:32 pm
One particular aspect that gives me a bit of a headache when designing profiles that people use with Adobe Lightroom is that when the user has set a custom white balance Lightroom remembers the temp/tint setting and then run it backwards through the profile to calculate the multipliers. This means that if my color matrices are not exactly the same as Adobe's profiles there will be a white balance shift (that is when you switch to my profile the tint of the image changes). So I just had to add the option to copy the color matrices from existing profiles for those users that didn't want this to happen. The problem is that estimating temp/tint with a camera matrices cannot be very exact so even if Adobe and I have the most exact methods it's likely that the result will differ several hundred degrees anyway. I haven't analyzed this in depth yet to suggest how it should be done, but I think that there is a problem currently. Any aspect that makes interoperability difficult is not good for a standard.

I don't think that there's a good solution to this. As soon as you change matrixes, either the tint of the image as rendered changes, or the user's saved temp/tint will have to be changed to preserve appearance. So from the perspective of someone writing a raw developer, either way someone's going to be unhappy.

Sandy
Title: Re: Yet some DNG comments (from a raw software developer)
Post by: AlterEgo on September 03, 2015, 12:54:46 pm
So from the perspective of someone writing a raw developer, either way someone's going to be unhappy.

or as some people suggesting - just allow an option in UI in raw converter for people to switch from K/tint to "RGB" (pre/post demosaick) multipliers ... Iridient does that, no ?
Title: Re: Yet some DNG comments (from a raw software developer)
Post by: sandymc on September 03, 2015, 12:58:04 pm
Does this excellent thread point to the need for a public forum for the DNG community to exchange thoughts?

It would certainly aid in the adoption of DNG if the spec were to become both better specified, and less Adobe centric, as laid out earlier in this thread. But Adobe own the spec. Unless they're up for changes to both to the spec, and more important to the governance of the spec, there won't be much point to exchanging thoughts.

Sandy
Title: Re: Yet some DNG comments (from a raw software developer)
Post by: sandymc on September 03, 2015, 01:06:17 pm
or as some people suggesting - just allow an option in UI in raw converter for people to switch from K/tint to "RGB" (pre/post demosaick) multipliers ... Iridient does that, no ?

Yes, you could do that. The issues are, it's (a) yet another option, and more and more options is how you end up with a product like RPP (no insult to them, but RPP isn't a a mainstream product). And (b), it's a really exotic option that most average users won't understand. Also, (c) it's not just the matrixes - in the Adobe model, you could change color temp in the LUTs if you wanted. And makes implementing the option really tricky.

Sandy
Title: Re: Yet some DNG comments (from a raw software developer)
Post by: torger on September 03, 2015, 06:01:02 pm
Regarding the temp/tint issue;

The problem is that the model is really unstable. If two separate design processes would lead up to almost exactly the same temp/tint estimation it would be fine, but it can be quite vast differences, ie it's almost impossible to get two different profiles agree on what temp/tint it is.

ACR should have stored RGB multipliers instead and let the temp/tint change if you alter profile, but keep the actual white balance the same. This is what happens already today when you leave the white balance at "as shot". So I think it's easy. Just let the "as shot" behavior be the behavior for all temperature settings -- just let temp/tint be a user-friendly front-end to change RGB multipliers, but store the multipliers, not temp/tint.

Temp/tint can't be trusted anyway and therefore I think there is little value in keeping them constant when changing profile. Say if you could measure a light source in a room to "5000K tint 0", and then just dial in "5000K tint 0" on any camera to match that so you have the right white balance for that room, then I would understand the value of storing temp/tint instead of RGB multipliers. However the temp/tint model is not and cannot be that stable and reliable to provide that sort of function.

This is actually not a problem with the DNG standard itself, but how Adobe have implemented it. As Adobe is so dominant their implementation will serve as a key reference, and therefore I think it's important that it's done "right" by them.

Now when it is the way it is they can't change it for existing ACR, but they could change the behavior for a future revision.
Title: Re: Yet some DNG comments (from a raw software developer)
Post by: Lightsmith on March 25, 2019, 01:06:30 pm
Two standards for digital image formats have existed for decades, JPEG and TIFF. I can send a TIFF file anywhere in the world and it can be opened with any image editor or web press print software. DNG was developed by Adobe to save engineering labor and the delay in reverse engineering a new camera's RAW file. I spent time with someone who did the reverse engineering for Adobe and they would buy a camera to use to make the RAW files that would be reverse engineered. They chose not to use the camera manufacturer's API.
Title: Re: Yet some DNG comments (from a raw software developer)
Post by: Jeremy Roussak on March 25, 2019, 01:33:42 pm
Why are you re-opening a thread which has been dormant for three and a half years?

Jeremy
Title: Re: Yet some DNG comments (from a raw software developer)
Post by: digitaldog on March 25, 2019, 01:45:53 pm
Why are you re-opening a thread which has been dormant for three and a half years?

Jeremy
Good question, especially since the premise of the post is mostly false.  ;D  Didn't see any reason to go there but maybe that's why it was posted.
But is there any forum rules that forbid replying to a 'dormant' and old post?
Title: Re: Yet some DNG comments (from a raw software developer)
Post by: Jeremy Roussak on March 26, 2019, 04:14:24 am
Good question, especially since the premise of the post is mostly false.  ;D  Didn't see any reason to go there but maybe that's why it was posted.
But is there any forum rules that forbid replying to a 'dormant' and old post?

No. But that doesn't invalidate my question.

Jeremy
Title: Re: Yet some DNG comments (from a raw software developer)
Post by: BrianVS on April 12, 2019, 04:34:19 pm
Too funny. Since the last response to this thread, I wrote my own DNG processor for the M8, M9, and M Monochrom.

TIFF- Most "manufacturer Raw" formats are a variation of TIFF with manufacturer specific tags. NEF is a variation of TIFF, DNG is a variation of TIFF. Not too hard to reverse engineer.

Don't know where adobe moved the spec- used the wayback machine to get to the link on the Library of Congress page regarding DNG format. Used this to write my code.

https://web.archive.org/web/20170115213607/http://wwwimages.adobe.com/www.adobe.com/content/dam/Adobe/en/products/photoshop/pdfs/dng_spec.pdf

I had to write a TIFF processor for my wife over 20 years ago. TIFF allows multiple full digital images to be stored in one TIFF file. Their digital camera for the microscope, 1MPixel- a big deal in the 90s, stored all the images in a session in one TIFF file. Most software could only process the first image. Had to write code to read in the big TIFF and separate all images into separate TIFF files, with corrected header information. My "Honey-Do" list is always interesting. The hyper-spectral data that she works with now is a huge 3-D array of stacked images, no header information at all. No one told her the dimensions of the array.
Title: Re: Yet some DNG comments (from a raw software developer)
Post by: Bart_van_der_Wolf on April 14, 2019, 01:58:11 pm
Good question, especially since the premise of the post is mostly false.  ;D

Care to elaborate?

Cheers,
Bart
Title: Re: Yet some DNG comments (from a raw software developer)
Post by: digitaldog on April 14, 2019, 04:28:36 pm
Care to elaborate?

Cheers,
Bart
Already have, years ago too.