New Hi Res Cameras- file size issues

[Gurtch]

I am a senior non-pro, seriously involved in photography since my first darkroom in 1959. I switched to digital capture around 2000. The files were inferior to my medium format scanned negative files. In 2002, I purchased a Canon 1DS, (full frame 11 mpixel),  for about $8,000; it was state of the art.  For each camera upgrade the files got bigger. My workflow was, and still is: keep recent raw files and Photoshop processed files on my PC internal drives, (both SSD drives, one for the operating system and programs, and one for active files). I eventually transfer the files to duplicate external storage. When I started out I used CDs, of which I have 157 pairs. The files got too large so I switched to DVDs, of which I have 92 pairs, then to Blueray discs of which I have 8 pairs. The camera files continued to get bigger, and I switched to duplicate external hard drives. I currently use a Sony A7RIV, and Fuji GFX 100S, and the cameras use lossless compressed files, but they are still huge. I know storage gets cheaper all the time, but is it harmful to allow the external drives to compress the already lossless compressed files? To complicate matters, I have another larger external drive, and Backup Program (called AShampoo), to once a week back up both internal SSDs, as well as one of the two external permanent file drives. If I were starting out today, and much younger (I am 84), the cloud storage, for a monthly fee would probably make sense. To give you a rough idea of my files, I have about 900 Photoshop "finished files", ready to print and frame from digital capture. Also about 150 likewise Photoshop finished files from scanned negatives. The A7Riv raw files are about 61,000 kb, the GFX 100s files are about 112,000 kb. Typical Photoshop finished files are about 352,000 kb for the Sony, and 600,000 for the Fuji. I will also post on the Medium Format Boad.

Thanks a million
Dave
Web site:   www.modernpictorials.com
 

[Joe Towner]

It's not an issue - it's a feature

Cloud stuff doesn't do too well with editing these giant files.  It does great backing them up though - it just takes time.

As to drives compressing already compressed files - there isn't any gain to the storage needed for the backup.  Software that compresses files will try, not gain any compression & move on to the next file.

I say that as I upgrade my local storage to from 50TB to 85TB - I  do multi-row panoramas & time lapses with MF cameras

[Dave Gurtcheff]

Thanks Joe. Still mulling around in my mind a sensible way to file and back up my images as well as my internal SSDs because they have my programs and a bunch of third party Photoshop "Actions" and "filters" that I down loaded and have no disks.
Dave

[Joe Towner]

How big are the drives?  Mac or PC?

I've got lots of ideas, happy to suggest a few.

[Dave Gurtcheff]

Well folks, after all the knowledgeable and generous suggestions, I decided to keep it as simple as I could, keeping in mind I am an amateur and not shooting as much as I once did. I shoot less, but the files are bigger, so I still produce a lot of GBs. Here is my setup: My PC is about 8 years old. I had it custom built by Puget Systems. About a year ago I had it upgraded to 32MB RAM, Windows 10 Pro, two internal drives, both SSDs. The C: drive is 500GB (nominal) and holds the operating system and all my programs. It has 251GB free of 465GB. The other drive (D:) is 1 TB (nominal) and holds all my files. It has 285 GB free of 932 GB. My photo files, as I produce them, both raw and Photoshop processed, are stored temporarily on my D: drive and simultaneously stored to two external USB drives. I also have a third USB drive where my back up program once a week stores copies of my photo files from the external drives, but also backs up my internal C: and D; drives.
So this is what I purchased during "CYBER MONDAY" sales: The two external drives for storing files permanantly are 5TB. The third drive to receive backups from those two drives plus my C: and D: drives is a powered desktop drive of 14TB.
I realize the capacities are over kill, but look at these prices from B&H:
Small USB external drives:
1TB = $55
2TB = $58
4TB = $100
5TB = $110
Powered USB external desktop drives:
8TB = $190
10TB = $210
14TB = $250
I will not install all these new drives until the present drives are filled.
I just cannot thank everyone enough who responded .
Dave Gurtcheff
Beach Haven NJ
www.modernpictorials.com
PS: in my original post I asked about compression to external drives, those files already losslessly compressed by the camera. What I was referring to is my present backup program is called AShampoo Pro 15. When setting up a backup plan, it asks if I want the files encrypted (I chose "no") and compressed (I think the default may have been "yes") but yesterday I changed that to "no"
Dave

[Joe Towner]

Awesome, have fun Dave.  Yes, we're still seeing the price effects of the Chia cryptocurrency on hard drives.  It will be back to 'normal' soon enough, where 8tb's are $120 as they were prior to 2021.

Honestly, what you are currently doing is what I recommend for most folks.  Internal drives, super fast for working space, then larger external(s) for archive.  It's all about having a good workflow, and you've got that going.

The only thing I would suggest is planning to rotate your internal SSD's - get a new 2tb drive to replace the 1tb drive, then reuse that 1tb drive for the 500gb drive.  Blame the GFX, the files are just huge - oh and Photoshop will happily consume any and all free space for swap :p

[Dave Gurtcheff]

Thanks Joe: What would you reccommend for back up software? I first used Acronis True Image after reading several reviews. For some reason, I gave that up (can't remember why) and switched to AShmpoo Pro 15. It is easy to set up a backup plan, and it sends an email report every week. Right now, for some reason, it is mis behaving. I told it to back up an external drive with 353GB to another external drive with 1.2 TB free space. It has been running for about 18 hours and still not finished!
Dave

[Joe Towner]

I'm mixed on backup software - and what you're looking for may not be best called a 'backup'.

What is on the external 353GB of data? Is it photos/files or a mix of stuff?  Are you going to be backing it up again to the same drive with 1.2TB space?

How are the 2 external drives plugged in? I would look at the USB connectors - should say 'SS' or be blue in color.  I would have them both plugged directly into the computer - USB3 hubs can really slow down a copy.

[BobShaw]

I think that the best thing you can do is to put all of your data and catalogues on one external drive and backup it religiously.
Ditch the DVD, CD, BlueRay, small drives etc.
That way if your computer dies you just plug it into another one and you are away.
Have at least two, preferably three backup drives and just rotate them with one offsite somewhere.
That is all that I have done for 10 years.

[Benny Profane]

I think that the best thing you can do is to put all of your data and catalogues on one external drive and backup it religiously.
Ditch the DVD, CD, BlueRay, small drives etc.
That way if your computer dies you just plug it into another one and you are away.
Have at least two, preferably three backup drives and just rotate them with one offsite somewhere.
That is all that I have done for 10 years.

Yup. I have to get my act together and start doing a third drive for off site. You never know.

I would never do Cloud. Services come and go.

[Joe Towner]

Cloud backup is awesome - it's just there, it works no matter if you're at home or on the road, it runs continuously in the background without you doing anything.  Should you do cloud only? No.  Will it cover you in a catastrophic event - yes(ish).

There is no harm in keeping old media around (cd's, dvd's, hard drives), but I would only consider them an option of last resort.  Having your entire archive on a small number of current tech drives is highly recommended - managing a single(few) 14tb USB3 drives is easier than a stack of disc's or assortment of 1-2tb HDD's.

As to software, I'm mixed a lot of different ways. Backup software wants to write to custom backup files.  You have to use the same software package to read those files & you have to restore them somewhere else to work on them.  Some backup products will make incremental backups, then consolidate them into larger files, and rely on the entire chain of files to be good to do a restore.  Hourly files get consolidated in to days, days consolidate into weeks, etc.  Some times these consolidations fail, and you'll need to start a whole new backup, on a new backup drive.

Backup is great for your c: drive, or your internal storage on a Mac.  Backup is good if your external drives are still small - say 4-5tb of data.  Once you get over 8-10tb of data, you don't want to 'backup' those files, you want to 'sync' or 'archive in place' them.  Products like Chronosync and GoodSync come to mind, but your needs may be different.  Instead of writing them to a backup file, they stay in their normal folder structure.  Then any 'changes' that are seen, the current file can either be overwritten, or archived separately so you can review them, or keep then.

It's all about the work flow - how you import photos, how you manage the cards that were just imported, etc.