Handling Large Image Files

[Michael Erlewine]

I have many large image files, most of them TIF files, the resulting images of years of focus stacking. As for the original layers, there are probably half a million or more of those. Let’s just look at these large TIF files.

Right, now I am trying to work (Windows 10) with the very large TIF files. I have tried Infraview, Bridge, PhotoMechanic, Windows Explorer, and so on. All of them seem to kind of choke up.

I have a pretty fast computer, yet nothing seems be very fluid. I am wondering what other photographers do with this situation. Here are some of my questions:

(1) What kind of external hard drives are best?

[I have tried SATA, SSDs, and all kinds of external drives, but none seem to be a charm, so to speak.]

(2) What size external hard drives are best?

(3) What viewing software do you use?

[As mentioned above, Adobe Bridge, Infraview, and so on see to bog down, take forever to load so I can see thumbnails, and generally prove almost unusable except at a snail’s pace.]

All suggestions are welcome as to how to deal with this ever-growing problem. Thanks.

[mcbroomf]

You are asking about software/hardware to allow you to browse thumbnails of your images right?

I gave up some time ago once I started creating large lightpainted images with many layers, especially as I have to use PSB files.  I will say that I have yet to try lightroom which can now give a psb thumb if smaller than a certain size.  I now rely on robust naming to allow me to find a file, and all print ready files are added as jpgs to my google photo site and I can find and check files and names quickly and then locate the tif/psb quickly fr printing or further editing.

What connections are you using to your external drives?  It makes a difference (USB2/3.1/3.2/TB2/TB3) although if you are using mechanical drives maybe not so much.

I use a pair of internal 4TB SSDs in Raid 0 for current work (about 1 years worth) and saving/opening files is pretty darn fast.  Of course all images (raws and print ready tiffs/psbs) are saved several times elsewhere as backups in case the SSDs die.

Mike

[Alan Klein]

Michael: I don't know if this would help because I never used it.  But there are Smart Previews that allow smaller files to be used for viewing and the larger files can be contained on separate larger drives.  Hope this solves your problem.  Alan.
https://helpx.adobe.com/lightroom-classic/help/lightroom-smart-previews.html

[Michael Erlewine]

My problem with Lightroom....I was an early adopter... is that twice Lightroom's catalog crashed and I lost ALL of my keywords, labels, etc.  That was years of work.

Now, I try to put all key information into the filename, like the lens used, and now I am slowly adding the subject-category like "Calla Lily," although I will shorten names to what will remind me, like "Calla," and so forth.

[Alan Klein]

My files aren't that big and I haven't had the same problems you've had. Maybe someone else has a solution.

[BobShaw]

My problem with Lightroom....I was an early adopter... is that twice Lightroom's catalog crashed and I lost ALL of my keywords, labels, etc.  That was years of work.

Now, I try to put all key information into the filename, like the lens used, and now I am slowly adding the subject-category like "Calla Lily," although I will shorten names to what will remind me, like "Calla," and so forth.

I am not a Lightroom user but I think adding keywords to a file name is self defeating. I only add the Shooting Date in Reverse order like 2020-08-22_IMG0021.cr2
That way you can always sort your images in the correct order.

If they are like other databases there will be a preference for the preview image size. Making that smaller might help.

[fdisilvestro]

Large tiffs are not easy to handle since they don’t have a high quality preview of the image, so they have to be read entirely to display a preview.

Even if you had bad experiences with Lightroom, I would say it is a good option to consider (using 100% previews or smart previews), and if you lost data due to a corrupt catalogue, then I suggest you review your backup & recovery procedures. You could also split your collection of images in 2 or more catalogues.

Anyway, I would also suggest you use the IPTC fields instead of the file names to store metadata.

Otherwise, even if you use Nmve SSDs (expensive), you will not have fast performance browsing large tiffs.

[Michael Erlewine]

Since I usually stack images from several to hundreds of layers, the resulting stacked image is a fabrication and does not carry all the info one finds in raw files, which is what I use for layering. I keep the raw files, so EXIF data is available if I am interested.

Actually, for my interest, I am only interested in a few pieces of data.

Filename
Lens Name
Keyword

I don’t really care about most of the other data because, in general I use APO lenses wide open (or nearly so), so since I know the F-Stop of the lens I am using. And most of the lenses I use are “exotics” meaning they do not usually interface to the camera to provide the standard EXIF data.

And I almost always use the lowest ISO, which for the Nikon cameras I use is ISO 64. So, the variable is the shutter speed and I don’t care about that.

An image filename might look like this:

8AE-0142-EL105-Morning-Glory

Where:
8AE-0142 =  the camera-generated filename
EL105 = APO El Nikkor 105 f/5.6
Morning Glory = the kind of subject

That all I need. From the filename (and the original NEFs (raw files) I can look up EXIF data if I need it.

However, what I am interested in is the particular lens used and not much else. I have a lot of lenses, mostly APO lenses, and their differing IQs vary, often subtly, so I want to know what lens was used.

[PeterAit]

I am puzzled as to why your files are so slow. I do focus stacking a lot and the final files - tiffs - are not any larger than the originals. You're not saving with layers, are you? That can lead to some huge files. Save the originals, of course, but just work with the final flattened image.

[Michael Erlewine]

I am puzzled as to why your files are so slow. I do focus stacking a lot and the final files - tiffs - are not any larger than the originals. You're not saving with layers, are you? That can lead to some huge files. Save the originals, of course, but just work with the final flattened image.

I use Zerene Stacker and I stack TIF files, result in the final stacked file that can be 250 MB or so.

That's not what I am talking about. I store the finished stacked TIF images and a folder of them is what I am talking about. That folder can contain 2500 large TIF file.

[kers]

I use Zerene Stacker and I stack TIF files, result in the final stacked file that can be 250 MB or so.

That's not what I am talking about. I store the finished stacked TIF images and a folder of them is what I am talking about. That folder can contain 2500 large TIF file.

If i understand correctly...

I would suggest only to keep the raw and the stack-result.
the tifs can be made anytime again from the raw...if you would ever need them again.

- I have the same volume-problem with panorama's... and dump the tifs as soon as the pano is ready.

[Michael Erlewine]

If i understand correctly...

I would suggest only to keep the raw and the stack-result.
the tifs can be made anytime again from the raw...if you would ever need them again.

- I have the same volume-problem with panorama's... and dump the tifs as soon as the pano is ready.

I have been doing this many years. I always throw away the TIFS used in stacking and keep the finished TIF and the NEF raw files.

[degrub]

look in the windows  - under control panel, system resource monitor and see what is going on with CPU usage, memory ( particularly page file usage ), and disk I/O.

i would suspect either the CPU cores are maxed out, possibly thermally throttling OR the system is paging memory (swapping in/out from memory to disk and vice versa). Sometimes adding more ram can help the latter or possibly using a faster interface (nVME versus SATA SSD for example)

[Michael Erlewine]

look in the windows  - under control panel, system resource monitor and see what is going on with CPU usage, memory ( particularly page file usage ), and disk I/O.

i would suspect either the CPU cores are maxed out, possibly thermally throttling OR the system is paging memory (swapping in/out from memory to disk and vice versa). Sometimes adding more ram can help the latter or possibly using a faster interface (nVME versus SATA SSD for example)

I have 64 GB RAM and pretty fast system. It's not working that hard. Just that the TIF files are large 250 MG and if I have 2500 of those in a folder, that's a lot to chew on. I just have to slow down a bit. I will use hard drive with SATA, take my time, and work through them.

[digitaldog]

My problem with Lightroom....I was an early adopter... is that twice Lightroom's catalog crashed and I lost ALL of my keywords, labels, etc.  That was years of work.

Any application can crash, data loss happens. It's why we backup our data. Every day, to multiple locations.
In this case, the catalog, the presets, the images, the DCP profiles; everything.

[LenR]

Hi Michael,

I know it's a shot in the dark but try changing "

8AE-0142-EL105-Morning-Glory

                  to

8AE_0142_EL105_Morning_Glory

It could be the system is refusing illegal characters.

Cheers

Len