Vuescan DNG problem

[jaapb]

Hi all,

I am having a problem with DNG's generated in Vuescan. I have a substantial amount of scans saved out as Vuescan DNG.
I now know this is not very wise if you want to postprocess in ACR and LR, because Vuescan DNG's are not real raws. They are rendered untagged files, this is why you don't want to process them in ACR and LR.
Not wanting to rescan many strips of film I decided to open them in Vuescan and save them out as tagged (prophoto and gray(for B&W)) TIFFs for postprocessing in ACR/LR.

However here comes the problem, the DNG's show up in Vuescan as only 300dpi (originally scanned at 4000dpi) and the resulting TIFF is very tiny and unusable (pixilated). These DNGs have been opened before in ACR and/or LR.

According to Ed Hamrick, the author and maintainer of Vuescan, once opened in ACR/LR a tag is written into the DNG, c716 for ACR and c717 for LR, alongside a low res preview (a 1:1 preview is in the LR database). This is causing Vuescan to only see the low res preview at 300dpi (my ACR workflow setting). Ed is not able to give a solution and regards these DNG's as being manipulated. Maybe Vuescans implementation of DNG is defying the purpose of DNG being an archival file format, but that is not solving my problem right now.

Is there a way to strip off all the tags and preview(s) written into the DNG to return the Vuescan DNG into its pristine state before opening into ACR/LR, because these untouched DNG's properly show up as 4000dpi files in Vuescan.

Any help is greatly appreciated.

Regards,

Jaap

[Bart_van_der_Wolf]

According to Ed Hamrick, the author and maintainer of Vuescan, once opened in ACR/LR a tag is written into the DNG, c716 for ACR and c717 for LR, alongside a low res preview (a 1:1 preview is in the LR database).

Hi Jaap,

This is serious, writing to a file and thus altering subsequent rendering, just by opening the DNG in ACR/LR. I haven't studied the DNG specifications so I don't know if it is even formally permitted to alter the file when only opening (seems very bad practice though). I also don't know what the tag showed before alteration, or if it was just added. Maybe this is a good question for the Lightroom forum as well?

Perhaps it is possible to create a DNG file with VueScan, make a copy and open that with ACR/LR. Then compare both files byte by byte what has changed and where, and see if changing those bytes will 'unchange' the original interpretation of the data in the DNG. Not something one would enjoy doing on a large number of files, but it might beat rescanning.

Hope you find a solution, it might save others a lot of agony as well.

Cheers,
Bart

[digitaldog]

I just have to ask (sorry). Since the DNGs are rendered (not true raw) why would you save out a DNG instead of a TIFF (and why would this software save out a DNG)? Seems pointless.

[Bart_van_der_Wolf]

I just have to ask (sorry). Since the DNGs are rendered (not true raw) why would you save out a DNG instead of a TIFF (and why would this software save out a DNG)? Seems pointless.

Hi Andrew,

In the light of archiving for posterity it seems totally logical to me, IF one assumes that DNG will be a standard file format that is possibly around longer than TIFF. Whether that will be the case is probably not something you and I can answer during our lifetimes.

It's an interesting question, but a bit off-topic for this thread, IMHO.

I'm more interested in the solution for the OP's predicament, and whether writing to a file upon opening it is even allowed under the DNG specifications.
 
Cheers,
Bart

[digitaldog]

In the light of archiving for posterity it seems totally logical to me, IF one assumes that DNG will be a standard file format that is possibly around longer than TIFF.

I’m not sure why. TIFF like DNG is owned by Adobe but both are openly documented and free for anyone to use in their products. Considering the history of TIFF, how many more applications can deal with TIFF, if the goal were to archive rendered data over the long haul, I’d stick with TIFF. I can open TIFFs I created in Photoshop 1.0.7 in CS5 and lots of other applications.

For an archival format for real raw data, DNG makes sense, but for rendered data? Not so sure when we have TIFF.

Sorry for the OT.

[sandymc]

whether writing to a file upon opening it is even allowed under the DNG specifications.

Yes - it's considered to be one of DNG's major advantages. E.g., for DNG's, LR writes all its edits to XMP metadata in the DNG, while for other raw files it has to use separate sidecar files. So with DNG, your edits and original images are always in one place - better for backups, etc.

But it is dangerous when the modified DNG's are used with software that doesn't support the full DNG spec. E.g., in the early days of Leica's M9, the then current version of Capture One would get color casts if the M9 files had previously been opened by LR.

Sandy

[jaapb]

Thank you all for responding.

I just have to ask (sorry). Since the DNGs are rendered (not true raw) why would you save out a DNG instead of a TIFF (and why would this software save out a DNG)? Seems pointless.

In fact it was the contribution of Andrew in threads in various forums ( I m afraid I forgot which forums, I thought photo dot net may have been one of them) that made me change from DNG as a file format for scan archiving to TIFF. I was just trying to simplify the workflow by adopting one program and file format to do pretty much all (LR). So I already switched to TIFF but still have a substantial part of my scanned archive in DNG file format.

This is serious, writing to a file and thus altering subsequent rendering, just by opening the DNG in ACR/LR. I haven't studied the DNG specifications so I don't know if it is even formally permitted to alter the file when only opening (seems very bad practice though). I also don't know what the tag showed before alteration, or if it was just added. Maybe this is a good question for the Lightroom forum as well?

This casts doubts on DNG being a true archival format, since an adobe tag written into the file renders it useless in the originating program.

I am not really an engineer to drill down into what is happening, obviously as an end user I am very much interested in the solution of this problem rather than pointing fingers to Adobe or Hamrick software.


Regards,

Jaap

[Frankomatic]

My recollection from archiving scanned Kodachrome slides using VueScan, is that VS uses the DNG file to store data about the scanner's IR channel (used for dust/scratch removal, if so equipped).  Perhaps it stores other scanner data as well, but the IR channel is what I was interested in at the time.  Thus one can later open the DNG in VS then "rescan" the image from "the file" and reapply dust/scratch removal at a different intensity.  I think VS also has an option to save a TIF at the same time, though the current IR settings are burned into the TIF.

During my archiving I saved a DNG for subsequent rescans in VS (if needed) as well as a TIF to use with other apps (usually Capture NX2, which doesn't use DNGs).  The VS DNG is quite handy actually.  If one finds his scan to be too soft once he loads the TIF into his image editor, he can rescan the image from the DNG, apply a weaker dust/scratch removal intensity, and then create a new TIF.

I really can't say if the VS DNG is meant to be used with other apps as my only use for them is with VS.  I would say vuescan's DNG is a valid archival format for VS, but have no idea if it would be considered a "true archival format" in the sense that it's compatible with any app that can open DNGs.  Sorry I can't offer a way for the OP to return his VS-generated files to their original state.

Regards,
Frank

[digitaldog]

In fact it was the contribution of Andrew in threads in various forums ( I m afraid I forgot which forums, I thought photo dot net may have been one of them) that made me change from DNG as a file format for scan archiving to TIFF.

Sorry :-(

I’m a proponent of DNG for raw data (from cameras) as opposed to proprietary raw files from the manufacturer. But a TIFF is a rendered image. And the file format is old, well supported and DNG in this case provides no advantages I can think of (and in this case, is causing issues with your scanning software). There’s a major difference between raw, un-rendered data and rendered data.

This casts doubts on DNG being a true archival format, since an adobe tag written into the file renders it useless in the originating program.

It casts doubts that I’ve had from day one when Adobe allowed DNG to encapsulate rendered data. I found it confusing then (how is a TIFF or rendered image inside a format called a ‘digital negative’ a digital negative? Its not). For raw data, its awesome. Scanner data isn’t what I’d call, at least today, raw data.

[Schewe]

This casts doubts on DNG being a true archival format, since an adobe tag written into the file renders it useless in the originating program.

No...this casts doubt's on Vuescan understanding and support for DNG. Adding some tags (which is what DNG is designed for) shouldn't invalidate a DNG. If Vuescan can't figure out how to parse a DNG, it should prolly not be in the business writing DNGs that break when touched by another DNG supporting software. I just can't see how this ISN'T Vuescan's responsibility to fix. And it's really not an issue of DNG being an archival format, it's an issue that software that supports DNG must actually support DNG.

[jaapb]

Jeff,

Apparently not all DNGs are created equally, and maybe you are right and Vuescans DNG implementation is faulty. But, as Andrew stated, it just might have been a bridge too far to have the DNG standard to encompass unrendered as well as rendered (untagged) data.
But I am not in a position to judge, I d rather have my problem solved.
I appreciate your comment though as this might let Ed acknowledge that other programs write in “his” DNG's and that he comes up with a solution.

Regards,

Jaap

[Schewe]

I appreciate your comment though as this might let Ed acknowledge that other programs write in “his” DNG's and that he comes up with a solution.

There are indeed different and various versions of DNG. But everything that is needed to fully and completely deal with DNG files is in the DNG spec.

If some venders only support a subset of DNG, that's not DNG's fault.

An app should be able to completely support DNG or else be VERY upfront with users EXACTLY what the app's limitations are.

Running a DNG through ACR/LR should seriously not disable the DNG support by another app. While the settings made in ACR/LR may not be read, not being able to read the full resolution is a bug, not a feature. No real way around that...

[Bart_van_der_Wolf]

There are indeed different and various versions of DNG. But everything that is needed to fully and completely deal with DNG files is in the DNG spec.

Hi Jeff,

Does that include what Adobe ACR/LR apparently writes to the file upon merely opening it? You seem to jump to the defence of Adobe very quickly. How sure are you that they are not writing undocumented Tag data? Really, how is a third party developer who is reading the specs supposed to know which tags are altered for no apparent reason (opening a file, for Pete's sake). Apparently he/she needs to assume the worst.

I'm not saying that Ed Hamrick is following all the specs (maybe he is, maybe he isn't), but you seem to be suggesting that he isn't. So what is it that you base that on?

Any solutions on offer to the OP's plight?

I can understand writing XMP data to a file (although an XMP file would be more logical), in fields that are designated to be read/write, but is that the case here? There seem to be a lot of assumptions flying about. How about some useful facts?

The OP has not been offered a real solution yet, maybe someone could suggest something useful, like reverting a file to the state before opening it with ACR/LR? What's needed to achieve that? C'mon folks.

Cheers,
Bart

[Schewe]

I can understand writing XMP data to a file (although an XMP file would be more logical), in fields that are designated to be read/write, but is that the case here? There seem to be a lot of assumptions flying about. How about some useful facts?

Read the DNG spec, have you?

It is a file format that is specifically designed to have xmp metadata written safely into the file. If an application barfs because metadata is safely written into the file, I would suggest the barfing app doesn't know how to correctly parse the DNG file format spec. Pretty simple...read the spec. Metadata written into the file should not invalidate the file. Vuescan is not parsing the DNG correctly.

The OP has not been offered a real solution yet, maybe someone could suggest something useful, like reverting a file to the state before opening it with ACR/LR? What's needed to achieve that? C'mon folks.

Sorry, but I'm pretty sure this is a Vuescan problem to fix. The OP is stuck between a rock and a hard place because full DNG isn't apparently in Vuescan. A limited and fragile support is worse than no support.

[Mark D Segal]

Hi all,

I am having a problem with DNG's generated in Vuescan. I have a substantial amount of scans saved out as Vuescan DNG.
I now know this is not very wise if you want to postprocess in ACR and LR, because Vuescan DNG's are not real raws. They are rendered untagged files, this is why you don't want to process them in ACR and LR.
Not wanting to rescan many strips of film I decided to open them in Vuescan and save them out as tagged (prophoto and gray(for B&W)) TIFFs for postprocessing in ACR/LR.

However here comes the problem, the DNG's show up in Vuescan as only 300dpi (originally scanned at 4000dpi) and the resulting TIFF is very tiny and unusable (pixilated). These DNGs have been opened before in ACR and/or LR.

According to Ed Hamrick, the author and maintainer of Vuescan, once opened in ACR/LR a tag is written into the DNG, c716 for ACR and c717 for LR, alongside a low res preview (a 1:1 preview is in the LR database). This is causing Vuescan to only see the low res preview at 300dpi (my ACR workflow setting). Ed is not able to give a solution and regards these DNG's as being manipulated. Maybe Vuescans implementation of DNG is defying the purpose of DNG being an archival file format, but that is not solving my problem right now.

Is there a way to strip off all the tags and preview(s) written into the DNG to return the Vuescan DNG into its pristine state before opening into ACR/LR, because these untouched DNG's properly show up as 4000dpi files in Vuescan.

Any help is greatly appreciated.

Regards,

Jaap

You are correct that these are not raw files in the sense that a DNG from a digital camera is a raw file. These are fully rendered RGB images, with a pixel count about the same as a TIFF of the same scan. I've experimented with this and seen it. Now, given that current versions of LR and ACR are capable of handling JPEG, TIFF and these rendered DNGs, have you tried further processing these Vuescan DNGs within LR or ACR, and then exporting them as 16 bit TIFFs to do any finishing touches in Photoshop? In other words, once you've imported those DNGs to LR/ACR, forget about Vuescan thenceforth. What happens to the integrity of the image using this workflow?

If what happens is no good and you need to revert the scan to its original state by stripping out tags, etc., an application engineer would need to guide you through the process of doing so, and unless it's Ed Hamrick himself, I don't know who else would be in a position to help with this. Only he knows his code and should know what happens to it when it opens in LR or ACR.

[jaapb]

In other words, once you've imported those DNGs to LR/ACR, forget about Vuescan thenceforth. What happens to the integrity of the image using this workflow?

Mark,

These files end up as 48 or 16 (B&W) bits full res tiffs with a colorprofile according to my PS colorsettings, so far so good. The problem is that the vuescan dng's are presented to ACR/LR as untagged rendered files.
As I understand now, ACR/LR assumes sRGB when an untagged file is opened and with all under the hood conversions in ACR/LR the file is never going to look as good as when a proper tagged tiff is opened in ACR/LR. This is why I abandoned Vuescans DNG's and have now switched to TIFF.
I am not getting the help from Ed as he considers "his" DNG's to be manipulated so I really hope someone in the know is able to help.

Regards,

Jaap

[Mark D Segal]

Jaap, I'm beginning to wonder whether the problem is with your settings either in Vuescan or in ACR or both. Let us be clear to start with that Vuescan's DNG files are NOT raw files regardless of what the program interface says. When you open such a file in ACR, ACR will not recognize a rendered file format and does not assign or convert a profile to the data until YOU do so on opening the image into Photoshop. You know the drop-down menu immediately under the image in ACR. That's the place where you select the rendered colour space, bit depth and resolution for the image. Have you gone into those settings to make sure you get the kind of output you are looking for? In the Vuescan Output tab have you made sure to leave all the file formats in the upper part of the menu UNchecked and only checked the Raw-DNG Format box further below?

I ask these questions because I was trying to replicate your problem and couldn't. I scanned a slide using the output settings I describe above, it opened in ACR, in the ACR Workflow Options menu described above I selected ProPhoto, 16 bit, 240 PPI, opened the image in Photoshop, and sure enough the ProPhoto profile was embedded, the 16 bit depth was recognized, and so was the file resolution. Having done all that, I would have no reason to ever return to Vuescan for that image, because Vuescan simply doesn't have an editing toolset of any use to me at that point.

[jaapb]

Mark,

Thank you for going into this effort to help clarify.
Checked my settings again, 16 bit, prophoto, 300dpi. DNG raw checked in VS. I too get a full res file in PS. But that is not the problem, I don't want to feed ACR/LR these DNG anymore because these DNG's are rendered untagged files. This has implications on image quality.
That is why I want my catalogue of VS DNG's converted into tagged TIFFs. before opening them into ACR/LR.
You could do that by scanning from file in VS and save them out as tagged TIFFs, but here the problem arises because the VS DNG once touched in ACR/LR show up as only 300 dpi files in VS resulting in very tiny TIFFs.

Just curious, can you replicate this when you open your slide DNG that is now touched by ACR/LR with Input|Scan from file in VS, can you save out a full res tagged TIFF from VS?

Thanks,

Jaap

[Mark D Segal]

Mark,

Thank you for going into this effort to help clarify.
Checked my settings again, 16 bit, prophoto, 300dpi. DNG raw checked in VS. I too get a full res file in PS. But that is not the problem, I don't want to feed ACR/LR these DNG anymore because these DNG's are rendered untagged files. This has implications on image quality.
That is why I want my catalogue of VS DNG's converted into tagged TIFFs. before opening them into ACR/LR.
You could do that by scanning from file in VS and save them out as tagged TIFFs, but here the problem arises because the VS DNG once touched in ACR/LR show up as only 300 dpi files in VS resulting in very tiny TIFFs.

Just curious, can you replicate this when you open your slide DNG that is now touched by ACR/LR with Input|Scan from file in VS, can you save out a full res tagged TIFF from VS?

Thanks,

Jaap

Jaap, I just tried what you said - point Vuescan to the "DNG" file I made and had previously opened in ACR. Then I turned off the output settings for DNG, and turned on the output setting for TIFF. I selected a Resolution on the Input page of 3200 dpi and on the output tab for print sie I selected scan sise which is a 100% magnification (i.e. no magnification). Then I pushed scan. Vuescan asked me to name the TIFF, which I did. Then when it opened in Photoshop, a missing profile warning came up asking me what to do with it, so I assigned Pro Photo RGB to it. Then it opened in Photoshop and I checked the "credentials". It did open as a full Tiff file, ProPhoto, 16bit, 3200 PPI, for a total pixel dimension of 66 MB which is completely consistent with the pixel dimensions of the so-called "raw DNG". With those dimensions in Photoshop, if I keep Resample unchecked in the Image Size dialog, I can ramp down PPI to say 360 and that will give me an enlargement of about 12*8 inches. All this means that previously opening the image in ACR did nothing to it - as expected, and outcomes are as they should be. So I still can't replicate your issue. Sorry about that.

[Bart_van_der_Wolf]

Just curious, can you replicate this when you open your slide DNG that is now touched by ACR/LR with Input|Scan from file in VS, can you save out a full res tagged TIFF from VS?

Hi Jaap,

I just did some testing, on a Windows platform, and a different scanner, etc., but anyway.

When I read a DNG Raw negative file from VueScan Pro (9.0.21)  (only opened with ACR, not saved) with VueScan again, Task=Scan to File,  Source=File, Scan resolution=Custom, I need to manually increase the Scan dpi to its maximum (the scan resolution that was used). Then on the Crop tab, I set the Crop size=Maximum. That allows me to Scan the 'Raw negative' to TIFF. I only miss 2 and 4 pixels at the edges, but can recreate almost the full original scan size in TIFF negative or rendered TIFF output format.

There are many variables, I know, so it might be different with a different file and OS, but do check the Custom Scan dpi setting.

Hope you can solve it.

I've looked at the altered DNG (it still escapes me why a file has to be altered upon opening, not saving) with a Hex editor comparison, and can pinpoint the changed, and added, bytes between files. However, besides a bit of readable XMP data added at the end of the file, there are also other changed bytes. Some are changed from null to something else, but there are also other changed bytes. Too hazardous to change without knowing the effects.

Cheers,
Bart