Pages: [1]   Go Down

Author Topic: DNG Converter tiling suppression on Phase One backs  (Read 5592 times)

torger

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 3267
DNG Converter tiling suppression on Phase One backs
« on: March 24, 2014, 09:52:36 am »

I'm working with improving IIQ format support in various raw software, ie provide patches to dcraw so it can find its way into various other smaller third party raw converters, both open source and not. As NDAs etc is not compatible with open source I have to rely on reverse engineering.

With P40+/P65+ and later backs tiling occurs as dcraw is not applying newer calibration data tags. I'm trying to figure out how these works and add that in to make the IIQ format a bit more open than it is today. I'm a strong supporter of open formats (I think the photographer has a right to the raw data), and while most closed formats are well reverse-engineered the IIQ has been lagging behind due to constantly new additions of various calibration data tags.

In this reverse-engineering process Adobe's DNG raw converter is of great help as this converter has support for the tags and one can so smoothly convert into the only open raw format there is (DNG) and then I can inspect the 'cooked' raw data (raw data with calibration tags applied), which gives much more information than seeing a readily processed image in Phase One's own software. However, while doing this it seems like the DNG converter in addition to applying the calibration tags also analyzes the file and refines the result to suppress tiling even more than calibration tags alone can do.

I've also seen examples where Capture One leaves faint tiling and Adobe Lightroom cleans them up successfully, which indicate that Adobe's converter does something more than just applying calibration data.

Can someone with insight into the Adobe internals confirm that Adobe DNG converter does additional analysis+refinement for tiling suppression on Phase One backs after calibration data has been applied, or not? It would greatly help me understand what happens.
« Last Edit: March 24, 2014, 09:56:33 am by torger »
Logged

Vladimirovich

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1311
Re: DNG Converter tiling suppression on Phase One backs
« Reply #1 on: March 24, 2014, 10:43:41 am »

Adobe DNG converter does additional analysis+refinement for tiling suppression on Phase One backs after calibration data has been applied

do you mean that Adobe DNG converter does not preserve the raw data read off sensor when it converts from IIQ to DNG  ?

Logged

torger

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 3267
Re: DNG Converter tiling suppression on Phase One backs
« Reply #2 on: March 24, 2014, 11:02:12 am »

do you mean that Adobe DNG converter does not preserve the raw data read off sensor when it converts from IIQ to DNG  ?

Yes, of that I'm 100% positive, that's easy to see. I don't think the DNG specification has support for the large amount of calibration data that IIQ packs (well, maybe with the most recent spec?), so the only possibility is to apply calibration data inside the converter and then save. Even if DNG format itself had/has support for calibration data few third-party DNG software would care to implement support for it as it would basically be only IIQ-converted files that would use it, so it's probably a good idea to keep the DNGs simple.

Actually, I think most other medium format CCD backs "cook" their data before writing to the file, like Hasselblad for example. Ie the back is individually measured in the factory and calibration data stored in the back, but calibration data is applied before writing the raw file, so you get a "cooked" raw (.3FR in the case of Hasselblad), just like the DNG in this case. Note that a CMOS sensor don't have the same need of calibration as a back based on a large CCD with multiple amplifiers for readout, so nearly all raw formats don't pack any calibration data at all.

From my perspective I'd prefer that the IIQ format had cooked data too, as it would make it a lot easier to support, but for scientific use cooked raws is not great of course, so doing it the IIQ way has its merits. The file is pretty ugly before applying calibration data though, one quadrant can have as much as 0.1 stop of more amplification than another etc. Calibration data only fixes dead pixels/columns, amplification offsets, amplifier linearity and slight pixel sensitivity variations over the sensor surface (flatfield). Simple scalings to fix linearity issues (ie all flatfield stuff and amplifier scalings) I would not consider to be that controversial even for scientific use (it's reversible), removing of dead pixels/columns is perhaps more so. As far as I understand almost all CCD sensor of this large size have some permanently bad pixels, so it's something that needs to be done in some stage for every back.
« Last Edit: March 24, 2014, 11:07:06 am by torger »
Logged

sandymc

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 350
Re: DNG Converter tiling suppression on Phase One backs
« Reply #3 on: March 24, 2014, 12:19:43 pm »

Are you sure that the possible additional clean-up occurs in DNG converter and not in Lightroom? If so, how are you sure? I'd be more inclined to believe that LR is doing the additional clean-up.

Sandy
Logged

torger

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 3267
Re: DNG Converter tiling suppression on Phase One backs
« Reply #4 on: March 24, 2014, 12:42:30 pm »

Are you sure that the possible additional clean-up occurs in DNG converter and not in Lightroom? If so, how are you sure? I'd be more inclined to believe that LR is doing the additional clean-up.

Yes I'm sure, on the development box I don't have Lightroom installed, only Adobe DNG Converter.

I convert the IIQ to DNG with Adobe DNG converter, then dump unmodified raw data out of the DNG using a modified dcraw which dumps at the earliest stage possible, ie no processing (demosacing etc) takes place, just unpacking the raw data and write it to a plain bitmap for further analysis.

So there's no other software involved than the DNG converter, and my dump-raw-data tool.
Logged

Vladimirovich

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1311
Re: DNG Converter tiling suppression on Phase One backs
« Reply #5 on: March 24, 2014, 01:17:20 pm »

Are you sure that the possible additional clean-up occurs in DNG converter and not in Lightroom? If so, how are you sure? I'd be more inclined to believe that LR is doing the additional clean-up.

Sandy

if he converts from IIQ to DNG then the whole premise of Adobe's workflow is that there shall be no (unless a bug) difference whatsoever if DNG was created by ACR or by LR or by Adobe DNG converter __for as long__ as they (ACR, LR, Adobe DNG converter) are of the same generation (official release, not RC or so).
Logged

Vladimirovich

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1311
Re: DNG Converter tiling suppression on Phase One backs
« Reply #6 on: March 24, 2014, 01:20:00 pm »

I don't think the DNG specification has support for the large amount of calibration data that IIQ packs (well, maybe with the most recent spec?)
I think that DNG v1.4 has those "opcodes" where you can store a big BLOB
Logged

torger

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 3267
Re: DNG Converter tiling suppression on Phase One backs
« Reply #7 on: March 25, 2014, 05:04:57 am »

I've figured out the relevant tags now, so the IIQ format just got a bit more open... still would be interesting to know if DNG converter has a "post-processing" step for extra tiling suppression, or if what I'm seeing is just some slight difference in how tags are applied.

Capture One can export DNG too. On a raw data level they differ quite much from the ones produced by DNG converter, and actually it seems like Capture One DNG export has some bug when it applies the narrow centerfold flatfield tag available on more recent backs (it leaves a bright line). Quite remarkable perhaps, but DNG is clearly not a prioritized format by Phase One so I'm not surprised that there can be some stray bug in there. That's why I've preferred using DNG converter as reference, I've tested lots of formats and Adobe generally do very good conversions.
Logged

madmanchan

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2115
    • Web
Re: DNG Converter tiling suppression on Phase One backs
« Reply #8 on: April 03, 2014, 12:40:21 pm »

torger, yes, the Adobe products (ACR, Lr, and DNG Converter) do contain extra logic to suppress/remove/minimize sensor-specific defects.  These include defective pixels (or pixel areas) and sensor tile seams.
Logged
Eric Chan

torger

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 3267
Re: DNG Converter tiling suppression on Phase One backs
« Reply #9 on: April 09, 2014, 09:13:46 am »

Thanks Eric, appreciate that.

A good tip for Capture One users that get tiling issues when they do strong contrast increases is convert in an Adobe product. Currently it seems like Capture One only applies calibration data with no extra logic, which has somewhat limited resolution and can only suppress tiling that much.
Logged
Pages: [1]   Go Up