Luminous Landscape Forum

Raw & Post Processing, Printing => Digital Image Processing => Topic started by: saiguy on June 23, 2018, 10:29:58 pm

Title: How to run DNG validation
Post by: saiguy on June 23, 2018, 10:29:58 pm
Can someone tell me how to run DNG validation on folders. I read about it some time ago but can't find where. Jeff Schewe's book, The Digital Negative doesn't cover that. Would like to run it from Adobe Bridge if possible.

thanks,
Title: Re: How to run DNG validation
Post by: digitaldog on June 23, 2018, 10:48:26 pm
Can be done in Lightroom or with 3rd party software, not aware its possible in Bridge.
Title: Re: How to run DNG validation
Post by: Schewe on June 24, 2018, 03:04:33 am
Can someone tell me how to run DNG validation on folders. I read about it some time ago but can't find where. Jeff Schewe's book, The Digital Negative doesn't cover that. Would like to run it from Adobe Bridge if possible.


You can do a DNG Validate from either DNG Converter or from ACR/Bridge but the DNG Converter is easiest/quickest. Peter Krogh has written a tutorial available on the ASMP website: dpBestflow.org (https://dpbestflow.org/data-validation/dng-validation#lightroom)

You can also use Lightroom and use a command that was added in LR 5 under the Library menu/Validate DNG Files. Peter has written about this as well DNG VERIFICATION IN LIGHTROOM 5 (http://thedambook.com/dng-verification-in-lightroom-5/)
Title: Re: How to run DNG validation
Post by: digitaldog on June 24, 2018, 09:17:59 am
How do you do this from ACR Jeff?
I can't get that URL from Peter to open on my end.
Title: Re: How to run DNG validation
Post by: saiguy on June 24, 2018, 09:41:39 am
Thanks Andrew & Jeff.

My DNG's are SilverFast HDR files on an external drive. They are never in a LR catalog.

Can you tell how to run it from Bridge?
Title: Re: How to run DNG validation
Post by: Schewe on June 24, 2018, 11:20:12 am
Thanks Andrew & Jeff.

My DNG's are SilverFast HDR files on an external drive. They are never in a LR catalog.

Can you tell how to run it from Bridge?


You can’t, you need to open the images in ACR and resave as DNG. Or use Dng Converter to reconvert to DNG.
Title: Re: How to run DNG validation
Post by: digitaldog on June 24, 2018, 11:31:52 am

You can’t, you need to open the images in ACR and resave as DNG. Or use Dng Converter to reconvert to DNG.
Oh, Lightroom=more better ;D . Re-saving would force every file to be backed up again.
Title: Re: How to run DNG validation
Post by: saiguy on June 24, 2018, 01:38:12 pm
The SilverFast HDR files are saved as .dng  Can't find anything on validation from Bridge help menu.
Title: Re: How to run DNG validation
Post by: fdisilvestro on June 24, 2018, 04:21:40 pm
Hi, you cannot use Bridge. The options are:

- ACR - validate the image(s) currently opened in ACR
- Adobe DNG converter - you can check an entire folder by reconverting them to DNG - free
- Lightroom - Perhaps the best option, but you need the licence as in ACR

Now some observations:

The idea behind DNG validation is that a checksum of the original image portion of the file, the one that should never change, is calculated when creating the DNG and stored inside it. What validation does is to recalculate this checksum from the image bits and compare it to the original checksum.

It is important to note that DNGs created outside of the Adobe environment, may not contain this original checksum. This happens with many in-camera generated DNGs.

I do not know if the DNGs created by Silverfast contain this checksum or not. If the DNG does not contain a checksum, LR/ACR will just indicate "not validated". The option here is to regenerate a DNG with any of the Adobe tools.

Oh, Lightroom=more better ;D . Re-saving would force every file to be backed up again.

Yes, LR is the best option, but you need to have the licence. In relation to backup the files again, you can do it to a temporary folder and delete them after the process, so the only requirement is to have enough space on disk.

If starting with "non-verifiable" DNGs as explained previously, you may want to do this backup the first time, as it will not alter the raw image itself.

How do you do this from ACR Jeff?
I can't get that URL from Peter to open on my end.

The URL gives a warning because it it a https link with either an expired or unregistered digital certificate. In most browsers you can override this warning and get to the page. Since you don't have to input  any information in the page, this should not be an issue.
Title: Re: How to run DNG validation
Post by: digitaldog on June 24, 2018, 04:28:13 pm
How do you validate an image open in ACR, gotta rewrite ?
Title: Re: How to run DNG validation
Post by: digitaldog on June 24, 2018, 04:33:50 pm
What I’ve used in the past but now unsupported?
https://basepath.com/new/detail-ImageVerifier.php
Title: Re: How to run DNG validation
Post by: saiguy on June 24, 2018, 05:22:26 pm
Francisco,

Thanks for the detailed info. I have the PS/LR subscription. There is no reason to import the SF8 dng's to LR cat. I develop them with the SilverFast HDR Studio and then they become .tiff files. After passing them thru PS I import them in LR as the masters.

Looks like I should try the Adobe DNG Converter. Will this create new files? I have 4k slide scans so far and 5k more to do for the project I'm doing. Been using .dng for the validation function. Does Adobe DNG Converter act as a stand alone application?

thanks for your input,
Title: Re: How to run DNG validation
Post by: fdisilvestro on June 24, 2018, 05:25:25 pm
How do you validate an image open in ACR, gotta rewrite ?

ACR performs the validation when it opens the file. If there is an error it will display a dialogue with the warning. LR does the same when you open an image in the develop module.
This works well for one image at a time. If you want to check multiple images or an entire folder then it is better to use LR (Library -> Validate DNG or use DNG converter as mentioned previously
Title: Re: How to run DNG validation
Post by: fdisilvestro on June 24, 2018, 05:37:57 pm
Francisco,

Thanks for the detailed info. I have the PS/LR subscription. There is no reason to import the SF8 dng's to LR cat. I develop them with the SilverFast HDR Studio and then they become .tiff files. After passing them thru PS I import them in LR as the masters.

Looks like I should try the Adobe DNG Converter. Will this create new files? I have 4k slide scans so far and 5k more to do for the project I'm doing. Been using .dng for the validation function. Does Adobe DNG Converter act as a stand alone application?

thanks for your input,

Yes, DNG converter creates new files and it runs standalone.

One option that you might consider is to use LR importing with the options “add photos to catalog without moving them” and using minimal previews. In this way you can perform the validation periodically and will use minimal resources/storage.

One final note, the DNG validation is intended to check for data corruption over large periods of time. If you have validation issues just after creating the DNG, then whatever tool you used most certainly has a bug.
Title: Re: How to run DNG validation
Post by: saiguy on June 24, 2018, 06:02:14 pm
Thanks again Francisco.

Will try a separate LRcat for this purpose. Will import one folder first to see if SF does have a checksum viable dng. If not will have to consider the Converter route with duplicated files. Ouch!
Title: Re: How to run DNG validation
Post by: saiguy on June 24, 2018, 06:30:36 pm
Just created new LRcat. Imported 1 folder [added] of the SF8 dng's. Ran dng validation from Library. Results said Invalid DNG Files 0. Not Validated 117, which was all of them.

So this is confusing and not reliable. So there are no invalid files, yet none are validated.

Will have to consider going thru duplicating 9k files for the validation function via Adobe DNG Converter. The LR masters are tif. Are they not stable files? Then there is always the slides to fall back on.
Title: Re: How to run DNG validation
Post by: fdisilvestro on June 24, 2018, 06:38:43 pm
Just created new LRcat. Imported 1 folder [added] of the SF8 dng's. Ran dng validation from Library. Results said Invalid DNG Files 0. Not Validated 117, which was all of them.

So this is confusing and not reliable. So there are no invalid files, yet none are validated.

Will have to consider going thru duplicating 9k files for the validation function via Adobe DNG Converter. The LR masters are tif. Are they not stable files? Then there is always the slides to fall back on.

Hi,

This is just what I mentioned previously. The tool that created the DNGs is not including the checksum. This is normal in most applications / cameras outside the Adobe ecosystem.

If you want the DNG verification functionallity, you need to recreate the DNGs with DNG converter or LR but then you will be able to validate the DNGs from this event towards the future. There is no way to validate from the moment you initially created the DNG. This should not be an issue unless using a tool with a bug.

Data corruption such as "bit rot" may occur over large periods of time, and that is what DNG validation is about

Remember that DNG is just a container, what you include in it is up to the developer of the tool. Most data elements are optional. Not even Adobe includes all possible data. An example: When you convert Nikon NEFs to DNG, the Adobe converter will not include the masked pixels, maybe because Adobe thinks there is no use for them.
Title: Re: How to run DNG validation
Post by: digitaldog on June 24, 2018, 07:14:07 pm
Thanks again Francisco.

Will try a separate LRcat for this purpose. Will import one folder first to see if SF does have a checksum viable dng. If not will have to consider the Converter route with duplicated files. Ouch!
+1, good info worth archiving, didn't know ACR did this upon open even though most of my work takes place in LR.
Title: Re: How to run DNG validation
Post by: saiguy on June 24, 2018, 07:30:25 pm
Francisco,

Just ran the convert to dng function in LR on the above folder. Was fast and was able to validate after. Seems no file duplication is involved.

Thank you for this most useful solution.
Title: Re: How to run DNG validation
Post by: fdisilvestro on June 24, 2018, 07:47:42 pm
Francisco,

Just ran the convert to dng function in LR on the above folder. Was fast and was able to validate after. Seems no file duplication is involved.

Thank you for this most useful solution.

Hi,

Make sure it did not created new files, as this is the behaviour I know of LR. Just right click on any image inside LR and select "Show in Explorer" (in windows or equivalent in OSX) and make sure it is the same original folder and not a subfolder or a different one.
Title: Re: How to run DNG validation
Post by: saiguy on June 25, 2018, 08:50:27 am
Francisco,

Thanks for the warning. Just checked the external drive for a subfolder. No sub folder but disaster. Seems it replaced all the files and moved originals to the trash. They are now less than half in MB size. Probably stripped out the inferred channel and more.

Input profile ARGB, should be scanner profile. 8 bit tif instead of 16 bit. 20 x 29 inch @ 72 ppi. Should be about 5 x 7 @ 300 ppi. All 117 files with the same creation date of the same minute.

I was just going to start running all the folders until I saw your warning. You saved me.

Now what to do? Think I restore originals from trash or a backup and give up on the dng validation. There was no "put back" option from the trash.
Title: Re: How to run DNG validation
Post by: saiguy on June 25, 2018, 09:24:23 am
Clarification,

I opened, edit in PS, from LR. PS shows the correct size, ppi, but opened as tif and not in ACR. Maybe that's normal?

I opened the new dng from the external drive in SilverFast HDR Studio. For some reason it used a for web preset that gave the above mentioned sizes. File itself has correct sizes. You can not export as dng from HDR Studio. Must be a rendered file format.

The inferred iSRD tool was not available. Same with 2 others I checked. I always include IR in the scan so it must have gotten striped out. That is a deal breaker. And the scanner profile should be showing but is not.
Title: Re: How to run DNG validation
Post by: fdisilvestro on June 25, 2018, 12:27:44 pm
Saiguy,

I though you converted the images to DNG when importing, but now I realize you used the LR function "Convert to DNG" after your images were imported. In the dialog that appears, there is an option to "Delete orginals after succesfull conversion" which might have been the cause of your original images being deleted. The difference in size may be due to compression.

If all your files are on the trash, you should be able to copy or drag them to their original location.

Having said that, I checked a few things regarding the DNGs generated by Silverfast. First they are linear DNG or rendered files (not RAW), similar to tif, and it is very likely that the ifrared channel is not kept when you do the conversion. 

I did not realize before that you would need to process these files in HDR studio. It is very likely that the Silverfast suite prevents processing DNGs not created by their scanning software or modified by another app, so it is better to stay with the DNG generated in the scanner software.

In this scenario, the DNG validation by LR/ACR does not look as viable option, unfortunately
Title: Re: How to run DNG validation
Post by: digitaldog on June 25, 2018, 12:36:20 pm
Having said that, I checked a few things regarding the DNGs generated by Silverfast. First they are linear DNG or rendered files (not RAW), similar to tif, and it is very likely that the ifrared channel is not kept when you do the conversion. 
Which may be a bit OT, but I still do not understand why some companies feel the need to 'suggest' that a rendered image, linear or otherwise, stored into the DNG container is useful, it's not raw data.
I suppose if the scanning software were limited, one could argue for saving a 'rawer' rendered image to 'fix' later in say Photoshop. But SliverFast is quite good for a global editor of input scans. Call my old fashion, someone who was scanning back when the Leaf 45 first arrived with software that isn't anywhere as robust then, as SilverFast is today. We did all the heavy lifting at the scan stage. That includes working with really lovely and powerful scanning software later like LinoColor, ScanView etc.

Title: Re: How to run DNG validation
Post by: saiguy on June 25, 2018, 03:51:02 pm
Francisco,

I imported the HDR dng files to LR via add method. I do not remember seeing any dialog. Think they were just added. Then I used the LR function Convert to DNG.

I did restore the originals from the trash and they look to be OK.

Given your last reply and Andrew's, it seems the SF implementation of dng is useless if I can't run a validation. In future I will export the HDR raws in the tif file format. And yes, the work flow is to later develop them using their scanning tools in the HDR Studio.

Thank you for sharing & caring,
Title: Re: How to run DNG validation
Post by: digitaldog on June 25, 2018, 03:54:42 pm
Given your last reply and Andrew's, it seems the SF implementation of dng is useless if I can't run a validation.
Don't know about useless but questionable as to why one would do all this and save off as a DNG. If you have something that would validate a TIFF instead, well that's another compelling reason to save as TIFF rather than DNG which is a very close cousin of TIFF.
Title: Re: How to run DNG validation
Post by: kirkt on June 25, 2018, 04:21:43 pm
You can download the DNG SDK from Adobe.  Inside of the included project folders, there are target builds of the command line tool for dng_validate.  For example, in the Mac SDK folders:

... dng_sdk_1_4/dng_sdk/targets/mac/release64/dng_validate

and Windows:

... dng_sdk_1_4\dng_sdk\targets\win\release64\dng_validate.exe

kirk
Title: Re: How to run DNG validation
Post by: saiguy on June 25, 2018, 05:15:44 pm
Andrew,  Is there a tool to verify TIFF files?
Title: Re: How to run DNG validation
Post by: digitaldog on June 25, 2018, 05:40:12 pm
Andrew,  Is there a tool to verify TIFF files?
Yes posted what I’m using earlier.
Title: Re: How to run DNG validation
Post by: saiguy on June 25, 2018, 06:40:17 pm
Andrew, Can you please point to that post. Am very interested. Guess I missed it.

Sent a forum query to LSI asking if validation can be run on there HDR dng's. Will report here if they respond.
Title: Re: How to run DNG validation
Post by: digitaldog on June 25, 2018, 07:27:48 pm
Andrew, Can you please point to that post. Am very interested. Guess I missed it.

Sent a forum query to LSI asking if validation can be run on there HDR dng's. Will report here if they respond.

Re: How to run DNG validation
« Reply #11 on: June 24, 2018, 04:33:50 PM »

What I’ve used in the past but now unsupported?
https://basepath.com/new/detail-ImageVerifier.php
Title: Re: How to run DNG validation
Post by: fdisilvestro on June 26, 2018, 07:01:02 am
Which may be a bit OT, but I still do not understand why some companies feel the need to 'suggest' that a rendered image, linear or otherwise, stored into the DNG container is useful, it's not raw data.
I suppose if the scanning software were limited, one could argue for saving a 'rawer' rendered image to 'fix' later in say Photoshop. But SliverFast is quite good for a global editor of input scans. Call my old fashion, someone who was scanning back when the Leaf 45 first arrived with software that isn't anywhere as robust then, as SilverFast is today. We did all the heavy lifting at the scan stage. That includes working with really lovely and powerful scanning software later like LinoColor, ScanView etc.

Having been a past user of Silverfast, my take is that they included later the DNG option as a "me too" (was not there in version 6). They develped the HDR Studio with the the option to sptil in half the workflow. You can scan the files in batches and store all information plus the infrared channel in an intermediate file, which are the files considered in this thread.

You could then finalize the process at a later time in the HDR Studio suite. The issue here is that the file is rendered (you can open it in PS) and the suite will not accept a modified DNG to complete the process, so I agree that there is not much value in storing this as a DNG
Title: Re: How to run DNG validation
Post by: fdisilvestro on June 26, 2018, 07:02:35 am
You can download the DNG SDK from Adobe.  Inside of the included project folders, there are target builds of the command line tool for dng_validate.  For example, in the Mac SDK folders:

... dng_sdk_1_4/dng_sdk/targets/mac/release64/dng_validate

and Windows:

... dng_sdk_1_4\dng_sdk\targets\win\release64\dng_validate.exe

kirk

Hi, this works if you have the validation checksum stored in the DNG file. As was mentioned previously, most of the DNG's generated outside the Adobe applications do not include this piece of information
Title: Re: How to run DNG validation
Post by: fdisilvestro on June 26, 2018, 07:15:50 am
Is there a tool to verify TIFF files?

Just in case you want checksum validation for long term archiving, (which is what DNG validation does) most of the tools available online just calculate the checksum but I have found a tool that can perform periodic validations, which I'm testing at the moment, called Fixity (https://www.weareavp.com/products/fixity/).

This tool creates a database of checksum of multiples files in a folder structure which you can check periodically to ensure the data has not been changed. This works well only for archived files, meaning files that will not be updated in the future.

This is not exactly the same as what DNG validation does, as in the latter only the image portion of the DNG file is considered, (the raw data does not change if you edit the DNG)

Just in case, during my preliminary tests, the initial checsum generation for a folder three of 6,482 raw images totalling 260GB, took 2 hours 45 minutes. I'll perform a subsequent validation tomorrow and will report on the elapsed time.

One curious fact, is that the LR database stores a hash of each image file on first import and then another hash at the latest synchronization, which could be used for this validation with little programming effor, but I guess is not a priority.
Title: Re: How to run DNG validation
Post by: saiguy on June 26, 2018, 01:29:53 pm
Sent a question to SilverFast re: DNG validation. Got a reply from a "Senior Support & Test Expert". Said SF8 can not directly validate but other tools like LR can, but he was not 100% certain. Said he would do some testing and report the results.

I returned the mail telling what I experienced as stated above in this thread.

Will try Fixity today. Looks great as no conversion involved. Thanks Francisco.
Title: Re: How to run DNG validation
Post by: saiguy on June 26, 2018, 06:43:35 pm
Downloaded Fixity for Mac. Created a New Project but it would not recognize my mounted external drive. So I could not select the folder I wanted to use it on. Sent a query via their contact form. It would only show local sources on the Mac.
Title: Re: How to run DNG validation
Post by: fdisilvestro on June 26, 2018, 07:10:53 pm
Downloaded Fixity for Mac. Created a New Project but it would not recognize my mounted external drive. So I could not select the folder I wanted to use it on. Sent a query via their contact form. It would only show local sources on the Mac.

Check this thread External-volumes-Mac (https://groups.google.com/forum/#!msg/fixity-users/q4qIgH6VWUI/DFcTQz0WAQAJ;context-place=forum/fixity-users) in their user group to see if it solves your issue
Title: Re: How to run DNG validation
Post by: saiguy on June 27, 2018, 05:17:59 am
Thanks Francisco for the link to the user group. It said in the list of local choices there would be one for Volumes. But it was not there.
Title: Re: How to run DNG validation
Post by: saiguy on June 27, 2018, 10:57:26 am
Found it. When you go to select a directory it opens to the home folder. That is a drop down where if Macintosh HD is selected their is a Volumes choice from which I could navigate to the folder I wanted.

thanks again.
Title: Re: How to run DNG validation
Post by: fdisilvestro on June 27, 2018, 03:36:00 pm
Great! What I have seen so far is good. The program works well once you learn how to use it. There are some counter-intiuitive issues or things that are not clear in the documentation but it is being maintained by the developers. As my friends in marketing would say, it looks like a program developed by engineers, meaning most effort is put into function and less into form.

Regarding the tests, my scheduled run performed fine and it took about the same amount of time as the first run. One little advantage in windows is that you do not need to be logged on for the program to run according to schedule.

Overall, considering it is an free open-source program, my evaluation is positive so far
Title: Re: How to run DNG validation
Post by: saiguy on June 28, 2018, 08:12:02 am
I ran Fixity first time on a folder with many sub folders that is 97GB. Then changed the Project Settings to filter out DS_Store & thumbs.db. and ran again.

First run took 3 hours. Second run 9 hours 10 minutes. It did remove the filtered files from consideration.

Seems it runs thru 2 passes. First one with ex drive spinning, 2nd pass with ex drive spun down. I set Prefs for drives to shut down never and screen to sleep after 3 minutes. Twice I noticed what I call deep sleep where the screen goes black and the words MACBOOK PRO bounces around the screen. Makes me wonder if the app is still running then? Seems the app has its own way of shutting down the drive when not needed and over rides my temporary Pref for never.

The long 2nd run may have to do with the very many now filtered files to deal with???
Title: Re: How to run DNG validation
Post by: saiguy on June 28, 2018, 08:25:04 am
Forgot to mention above.

On the 2nd run the first pass took 2hr 40min. The 2nd pass took 6hr 30min.
Title: Re: How to run DNG validation
Post by: fdisilvestro on June 28, 2018, 03:48:55 pm
What checksum algorithm are you using? If it is SHA256, try changing it to MD5, which should be faster, and for validation of local files should be enough, otherwise contact support to see if they have any suggestion to  improve performance
Title: Re: How to run DNG validation
Post by: saiguy on June 28, 2018, 11:04:12 pm
Was using SHA256. Since it is their default thought they considered it best. Will try MD5, but dread going thru all that time again.

Was having contact with their support and quoted where you were getting rum time of 2hr 45min. on a folder 2&1/2 times larger than mine. She just said every system is different.

I have a robust MBP maxed out mid 2012 model. SSD and the ex drive is a G-RAID enterprise class, mirrored, with Thunderbolt connection.

Hoping to see your rerun speed test.

Wonder if I should start a New Project or just change the algorithm for the existing one. I'll delete the existing and start fresh with MD5 and filter out the DS_Store and thumbs. That will give a less cluttered comparison. Will do tomorrow and report findings.
Title: Re: How to run DNG validation
Post by: fdisilvestro on June 29, 2018, 09:12:47 am
It is difficult to tell what may be impacting performance and the code itself might not perform the same across different platforms. Just a few comments:

- MD5 should be around 30~40$ faster. Try with a smaller directory or another drive
- Filtering files should improve performance, as there are less hashes to compute
- In my tests, the execution time of the first run and subsequent runs is about the same. For the 260 GB, it was 2 hr 45 min first and then 2 hr 59 min for the following run

In regards to the difference in checksum algorithms, I made a test on a another with NMVe drive on a 9.3GB folder containing 237 files

MD5: First run 1 Min 51 Sec; second run 1 Min 1 Sec
SHA256: First run 3 Min 11 Sec; second run 3 Min 9 Sec
Title: Re: How to run DNG validation
Post by: saiguy on June 29, 2018, 10:05:27 am

Francisco, thanks for the good information. I'm going to redo with new Project  as mentioned above. You have already done the testing.
Title: Re: How to run DNG validation
Post by: saiguy on June 30, 2018, 11:15:08 am
Created a new Project with MD5 and filter out DS_Store and thumbs.db. In Mac Prefs, Energy Saver, set my usual Computer Sleep to 5min. and display sleep to 3min. There is a check box for Put hard discs to sleep when possible, which I checked on.

Same 97GB folder set. After 15 minutes into the run time I left for an hour. When I came back and the external drive was sleeping and so I think Fixity must have been paused. I thought the processing was so intense that the ex would not go to sleep. Don't like this behavior. So the following run time numbers may be up to 1hr longer than it could have been.

Took 10 hours to run. Seems too cumbersome for me.