Poll

Do you save your originals as DNG or RAW?

DNG
- 22 (26.8%)
RAW
- 60 (73.2%)

Total Members Voted: 81


Pages: 1 ... 4 5 [6] 7 8 ... 15   Go Down

Author Topic: DNG or RAW  (Read 60641 times)

ErikKaffehr

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 11311
    • Echophoto
Re: DNG or RAW
« Reply #100 on: January 02, 2014, 12:55:58 pm »

Hi,

The image data is very simple, it just RGB values. To interpret the RGB values you need white balance (two numbers) and color conversion matrix (9 numbers). Additional data can be helpful. DNG can contain proprietary data. In fact you can embed a raw file as a whole in a DNG file.

Best regards
Erik



Hi Andrew,

So to put it another way you believe proprietary formats for data (including data you create) are socially benevolent when the algorithms that can be expected to process them proprietary too. Your expectation is that data that comes from a camera ought not to be proprietary because there already exist generic proprietary processes to manipulate the data, e.g. ACR. If there weren't presumably you wouldn't have problem because you're not against proprietary data formats per se.

From my perspective your argument is exceedingly narrow -- much more narrow for instance than the debates that emerged around the establishment of ISO standards for word processing and spreadsheet data formats, for instance.

Damon
Logged
Erik Kaffehr
 

Damon Lynch

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 330
    • http://www.damonlynch.net
Re: DNG or RAW
« Reply #101 on: January 02, 2014, 01:29:38 pm »

I simply want access from day one to the unrendered raw image data. I don't care about any of the other data. Proprietary or otherwise.

It seems to me that what you are arguing for is far narrower than aspects of the DNG spec itself. One of the the points of a metadata standard is to include information about how the data is to be transformed. That's why XMP files contain that data in an open format. That metadata about files we produce today is useful now, and will be useful to some people after all of us are long dead. It is still useful even though we don't know the precise implementation of the algorithms that actually transform the data because they are proprietary. In other words, you may not care about how a change in vibrancy is recorded, but there is already a standard for that. 

In the U.S. before there was a national standard for time, every town set their own time according to their local norms and regulations. That made making railway timetables an absolute nightmare. That spurred people on to standardize time nationally. Now we take it for granted, forgetting how different life once was. I hope one day in our lifetimes we will take it for granted that the data that comes from our cameras is not merely accessible, but that our degree of control of it allows the true freedom to do what we want with it, now and in the future. I fear however that the factors that Bart has identified are all too real, and unlike office document formats, not enough people care.

Logged

digitaldog

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 20630
  • Andrew Rodney
    • http://www.digitaldog.net/
Re: DNG or RAW
« Reply #102 on: January 02, 2014, 01:37:34 pm »

It seems to me that what you are arguing for is far narrower than aspects of the DNG spec itself.
As the bare minimum yes but simply saving out the spec'ed DNG is possible today and provides even MORE benifits to my workflow. How and why should any of this be different with the raw as with the JPEG? Treat them equally.
Logged
http://www.digitaldog.net/
Author "Color Management for Photographers".

hjulenissen

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2051
Re: DNG or RAW
« Reply #103 on: January 02, 2014, 02:00:56 pm »

Hi,

The image data is very simple, it just RGB values. To interpret the RGB values you need white balance (two numbers) and color conversion matrix (9 numbers). Additional data can be helpful. DNG can contain proprietary data. In fact you can embed a raw file as a whole in a DNG file.

Best regards
Erik
I believe that the RGB data may be lossless or lossy compressed, meaning that interpreting the bits representing "raw rgb data" may be non-trivial.

Of course, for manufacturers to not disclose the encoding of those rgb data, instead having people like Dave Coffin reverse-engineering it is bad for customers.

-h
Logged

Damon Lynch

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 330
    • http://www.damonlynch.net
Re: DNG or RAW
« Reply #104 on: January 02, 2014, 02:03:43 pm »

As the bare minimum yes but simply saving out the spec'ed DNG is possible today and provides even MORE benifits to my workflow. How and why should any of this be different with the raw as with the JPEG? Treat them equally.

What you're advocating is analogous to a word processing document format in which the textual content is saved in an open format but all the formatting (e.g. text formatting, page size etc.) is proprietary. It's not a vision I share.
Logged

digitaldog

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 20630
  • Andrew Rodney
    • http://www.digitaldog.net/
Re: DNG or RAW
« Reply #105 on: January 02, 2014, 02:07:15 pm »

What you're advocating is analogous to a word processing document format in which the textual content is saved in an open format but all the formatting (e.g. text formatting, page size etc.) is proprietary. It's not a vision I share.
That's your analogy, I made clear what I desire, a DNG from the camera as an option. If you prefer JPEG or the camera proprietary raw, I'm AOK with that.
Logged
http://www.digitaldog.net/
Author "Color Management for Photographers".

robgo2

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 388
    • Robert Goldstein Photography
Re: DNG or RAW
« Reply #106 on: January 02, 2014, 02:11:02 pm »

Pentax too. And Hasselblad.
Some Pentax cameras have raw DNG output, but the higher end models have PEF.

Rob
Logged

alain

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 465
Re: DNG or RAW
« Reply #107 on: January 02, 2014, 02:22:00 pm »

I believe that the RGB data may be lossless or lossy compressed, meaning that interpreting the bits representing "raw rgb data" may be non-trivial.

Of course, for manufacturers to not disclose the encoding of those rgb data, instead having people like Dave Coffin reverse-engineering it is bad for customers.

-h
It's a complete other thing to make the encoding available, even with specific contracts (including NDA's) or using a format that is published from another (American) company.

Logged

ErikKaffehr

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 11311
    • Echophoto
Re: DNG or RAW
« Reply #108 on: January 02, 2014, 02:43:23 pm »

Hi,

Customers who buy a system should have ownership of their data. To have to sign an NDA so you can process your images with the tool of your choice is a bit obscene.

Best regards
Erik

It's a complete other thing to make the encoding available, even with specific contracts (including NDA's) or using a format that is published from another (American) company.


Logged
Erik Kaffehr
 

alain

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 465
Re: DNG or RAW
« Reply #109 on: January 02, 2014, 02:47:24 pm »

Hi,

Customers who buy a system should have ownership of their data. To have to sign an NDA so you can process your images with the tool of your choice is a bit obscene.

Best regards
Erik

Erik

Can you point me to the location of the PS/Lightroom/dxo/C1 information how to get to the data, aka all parameters I have set?

Logged

ErikKaffehr

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 11311
    • Echophoto
Re: DNG or RAW
« Reply #110 on: January 02, 2014, 02:52:26 pm »

http://wwwimages.adobe.com/www.adobe.com/content/dam/Adobe/en/products/photoshop/pdfs/dng_spec_1.4.0.0.pdf

http://www.adobe.com/support/downloads/detail.jsp?ftpID=5474

Best regards
Erik


Erik

Can you point me to the location of the PS/Lightroom/dxo/C1 information how to get to the data, aka all parameters I have set?


Logged
Erik Kaffehr
 

alain

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 465
Re: DNG or RAW
« Reply #111 on: January 02, 2014, 02:58:53 pm »

http://wwwimages.adobe.com/www.adobe.com/content/dam/Adobe/en/products/photoshop/pdfs/dng_spec_1.4.0.0.pdf

http://www.adobe.com/support/downloads/detail.jsp?ftpID=5474

Best regards
Erik


Erik

That's not what I was asking.  I'm talking about complex changes to images on layers, like dodging and burning, spot healing, cloning, sharpning etc.  All data that's being made while PP's an image.
Logged

hjulenissen

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2051
Re: DNG or RAW
« Reply #112 on: January 03, 2014, 04:11:05 am »

It's a complete other thing to make the encoding available, even with specific contracts (including NDA's) or using a format that is published from another (American) company.
I don't see your point.

If I use a trivial 1970s compression scheme to compress my rgb data by 2:1, my users should not have to reverse-engineer actual files in order to be able to access their data. If I am a billion dollar camera company, I should allocate 3 days of engineering time to write a mock-up code (using some high-level inefficient language) example decoding the current scheme and release it as-is, e.g. open-source.

I am not asking Canon to tell us how they do noise reduction or demosaic or lense correction, as that is something that can fairly be considered trade-secrets.

-h
Logged

hjulenissen

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2051
Re: DNG or RAW
« Reply #113 on: January 03, 2014, 04:13:06 am »

Erik

That's not what I was asking.  I'm talking about complex changes to images on layers, like dodging and burning, spot healing, cloning, sharpning etc.  All data that's being made while PP's an image.
In the case of Lightroom, a surprising amount of verbal description of editing history is included as meta-data.

Of course, Adobe cannot tell you exactly how to translate that description into image manipulation, as that would effectively be documenting their proprietary image processing engine.

I believe strongly in opening up data, APIs, while I accept that your core intellectual property should be hidden. Not only because this is good for society, but because I think that it improves your business.

-h
« Last Edit: January 03, 2014, 04:15:53 am by hjulenissen »
Logged

ErikKaffehr

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 11311
    • Echophoto
Re: DNG or RAW
« Reply #114 on: January 03, 2014, 04:14:07 am »

Hi,

Working on an answer, but not there yet.

When saving a DNG file in say TIFF a 'side car' file is exported, like the one below:
http://echophoto.dnsalias.net/ekr/Articles/MFDJourney/RawImages/MFDB_VS_DSLR2/Femore2_20131124-CF044335.xmp

According to the DNG documentation I posted this is stored with tag=700 in a DNG file.


That file contains among other data settings for the gradient tool:
 <crs:GradientBasedCorrections>
    <rdf:Seq>
     <rdf:li>
      <rdf:Description
       crs:What="Correction"
       crs:CorrectionAmount="1.000000"
       crs:CorrectionActive="true"
       crs:LocalExposure="-0.124183"
       crs:LocalSaturation="0.294118"
       crs:LocalContrast="0.000000"
       crs:LocalClarity="0.000000"
       crs:LocalSharpness="0.000000"
       crs:LocalBrightness="-0.098039"
       crs:LocalToningHue="0.000000"
       crs:LocalToningSaturation="0.000000"
       crs:LocalExposure2012="-0.156453"
       crs:LocalContrast2012="0.000000"
       crs:LocalHighlights2012="-0.411765"
       crs:LocalShadows2012="0.000000"
       crs:LocalClarity2012="0.202614"
       crs:LocalLuminanceNoise="0.000000"
       crs:LocalMoire="0.000000"
       crs:LocalDefringe="0.000000"
       crs:LocalTemperature="0.000000"
       crs:LocalTint="0.000000">
      <crs:CorrectionMasks>
       <rdf:Seq>
        <rdf:li
         crs:What="Mask/Gradient"
         crs:MaskValue="1.000000"
         crs:ZeroX="0.596873"
         crs:ZeroY="0.631574"
         crs:FullX="0.595705"
         crs:FullY="0.375582"/>
       </rdf:Seq>
      </crs:CorrectionMasks>
      </rdf:Description>
     </rdf:li>
    </rdf:Seq>
   </crs:GradientBasedCorrections>

As you can see this information is readable. On the other hand it just lists parameters which are used, for instance:
crs:LocalHighlights2012="-0.411765"

Tells that LocalHiglights is at -0.411765, to implement it you need emulate higlight compression method in Lightroom.

You can easily transfer non "well know" parameters, like exposure, ranking and cropping. Indeed, many tools handling DNG files transfer exposure, cropping etc, but not proprietary methods.

Best regards
Erik

Erik

That's not what I was asking.  I'm talking about complex changes to images on layers, like dodging and burning, spot healing, cloning, sharpning etc.  All data that's being made while PP's an image.
Logged
Erik Kaffehr
 

Tim Lookingbill

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2436
Re: DNG or RAW
« Reply #115 on: January 03, 2014, 04:56:10 am »

Erik, is there anything in those xmp instructions that exposes how Adobe's color engine translates those instructions to the video card in order to evoke a response from the user editing and creating the instructions?

There's got to be some level of proprietary "secret sauce" even with Adobe that makes what looks like non-proprietary instructions in the form of xmp code create magic in one Raw converter but crap in another using the same instructions.

Converting to a DNG format doesn't show how it controls how that works in order to clearly indicate it's not considered proprietary. DNG is just a container of instructions and data no matter what Raw converter creates it and so if all camera manufacturers and software makers adopted DNG then the instructions written and embedded should deliver the same results across all converters regardless of the originator of the DNG.

IOW is the proprietary portion of a Raw workflow still most prominent in the source file or the converter that first creates and embeds the instructions that deliver the initial look of the image that can't be considered proprietary since everyone is now using the same DNG format (when and if that ever happens)?
Logged

Damon Lynch

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 330
    • http://www.damonlynch.net
Re: DNG or RAW
« Reply #116 on: January 03, 2014, 05:14:59 am »

DNG is just a container of instructions and data no matter what Raw converter creates it and so if all camera manufacturers and software makers adopted DNG then the instructions written and embedded should deliver the same results across all converters regardless of the originator of the DNG.
No. There should always be a fundamental difference between data and it's transformation. Transformation varies; the original data is the same. Metadata specifies how the data was transformed and with which version of the process.
Logged

digitaldog

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 20630
  • Andrew Rodney
    • http://www.digitaldog.net/
Re: DNG or RAW
« Reply #117 on: January 03, 2014, 11:02:43 am »

No. There should always be a fundamental difference between data and it's transformation. Transformation varies; the original data is the same. Metadata specifies how the data was transformed and with which version of the process.
I think it's important to put the various bits and pieces, in terms of what is and isn't (or will be) proprietary.
The container may be proprietary for a few months. Canon releases a 32DII Jan 1 2014. The raw data container is proprietary such only those with Canon's software can process their raw data. This is unacceptable for a lot of reasons. Imagine if Canon produced a raw file that only their raw converter could process. All the DNG naysayers would be shitting bricks (I'd love to see that).

Months after any or all raw converter manufacturers spend time and money to understand a container, with or without some other proprietary bits unnecessary for them to process that data, the container is understood by said raw converters. We waited months to unlock something that should not have been locked in the first place and cannot be locked such others can't have access to that data. WHY? The DNG container isn't proprietary. If on Jan 1 2014 the Canon 32DII spit out a DNG as the spec defines it, we'd be processing that raw in LR or C1 or Iridient Developer that very day.

There is proprietary processing. Nothing unusual, nothing unexpected. I don't expect C1 to handle the same raw data within it's processing engine, with proprietary tags or not as LR or Iridient Developer. No one (as yet) is asking for a universal raw converter that understands everyone else’s processing instructions (proprietary) and can handle that data the same way. Unnecessary. Even if we took 4 different raw converters, all having the same slider called Saturation, all having the same scale (0-100), expecting instructions for Saturation +7 to be applied in C1 from LR's instructions isn't going to happen and isn't expected. Even if it understood +7 Saturation, the entire proprietary raw processing engine is different and will produce a different result. Not an issue! I no more expect LR's proprietary Saturation to be the same as C1's as I expect the E6 lab in Santa Fe to produce identical processing as A&I in LA on all film types.

DNG allows the camera raw data, and other data that may be almost as important to be accessible to all the converters the day the camea ships. Not 3 months later when all the people writing code have to stop doing so in terms of making a better product, but solely to access my data, data I should have been able to feed to the processor of my choice on day one. I bring up JPEG and will do so again. Why isn't the raw data treated just like the JPEG in terms of my ability to access that data on day one? Not months later when the enviable happens as it has for years and years: I have access to my raw data because the engineers of my raw converter have hacked what was proprietary (mostly container data) that never needed to be in that state in the first place. Why?
Logged
http://www.digitaldog.net/
Author "Color Management for Photographers".

digitaldog

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 20630
  • Andrew Rodney
    • http://www.digitaldog.net/
Re: DNG or RAW
« Reply #118 on: January 03, 2014, 11:06:32 am »

Erik, is there anything in those xmp instructions that exposes how Adobe's color engine translates those instructions to the video card in order to evoke a response from the user editing and creating the instructions?
There's got to be some level of proprietary "secret sauce" even with Adobe that makes what looks like non-proprietary instructions in the form of xmp code create magic in one Raw converter but crap in another using the same instructions.
The entire processing is proprietary, there's lots of it. See my last post. You can produce XMP that says +18 Vibrance. You could have two products that actuallty have a silder called Vibrance but you'll get different results even if (big if) product A understood product B's XMP that has XMP with +18 Vibrance in it.
In a way, this is lovely, your raw image data can be molded as you desire based on a product you prefer for it's controls and rendering. Otherwise, raw is raw and the DNG part of all this is simply having a camera spit out a non proprietary document we can use.
Logged
http://www.digitaldog.net/
Author "Color Management for Photographers".

Bart_van_der_Wolf

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 8913
Re: DNG or RAW
« Reply #119 on: January 03, 2014, 12:09:25 pm »

Imagine if Canon produced a raw file that only their raw converter could process.

Hi Andrew,

That's a Red Herring statement. Even the Foveon format can be decoded by e.g. DCraw, but it's not as good as the dedicated converter can achieve. Any Raw format can (and will) be decodable. It's a non-argument.

Quote
We waited months to unlock something that should not have been locked in the first place and cannot be locked such others can't have access to that data. WHY?

Frankly, non-sense, because the camera came with a Raw converter. Maybe not you favorite one at that time, but your files would be accessible. Besides, putting the data in a DNG container makes no difference, the structure would still need to be understood. Raw or DNG, which is the topic of this thread, makes no difference. Remember the Fuji Xtrans 'quality' in LR/ACR, compared to Capture One which was pretty good from the get go?

Quote
The DNG container isn't proprietary.


But that's not what the thread is about. The data in the container can still be proprietary, so that container makes no difference whatsoever.

Quote
If on Jan 1 2014 the Canon 32DII spit out a DNG as the spec defines it, we'd be processing that raw in LR or C1 or Iridient Developer that very day.

Nope, it takes time to reverse engineer a proprietary set of 'data' before it becomes an image. The container makes no difference in that respect. The only thing that the DNG container allows is to work smoothly with e.g. Lightroom, in the sense of storing proprietary processing steps in the same container, and a few other nice features for some, but irrelevant for others.

I understand you do not like to be obstructed in access to your original image content, but that is a different discussion. Even film required a conversion process. Kodachrome also could not be processed in chemicals intended for another conversion, but a conversion will always be part of photography.

Cheers,
Bart
« Last Edit: January 03, 2014, 12:12:17 pm by BartvanderWolf »
Logged
== If you do what you did, you'll get what you got. ==
Pages: 1 ... 4 5 [6] 7 8 ... 15   Go Up