LR 2.2 Library Backup too slow to ReadyNAS on Network

[budjames]

I'm a backup fanatic. I have have MacPro 8-core as my workstation with 4 x 1TB internal drives. 2 of these are in a RAID 0 configuration for my photo and video working storage. I backup up several times to a FW800 external drive (OWC dual drive external RAID 0) mirroring my internal RAID 0.

Once a week, I backup to a ReadyNAS NV+ with 4TB of storage in in RAID X configuration. What I noticed is that this backup (mirror the drive) takes about 8 hours. I found out that the culprit for the long back up times is the LR catalog file which has grown to 54+ GB. I've tried Chronosync and Synchronize X Pro using the "dissect packages" setting so that it only mirrors the components of the Catalog file that have changed. It still takes forever.

I'm thinking about moving the catalog to my boot HD so that it gets backup when I make a SuperDuper clone of my boot HD each week. I will exclude the Catalog file from my normal Time Machine backup as recommended by others.

I was curious how others are backing up their LR Catalog files?

Thanks.

Bud James
North Wales, PA

[francois]

…
 I found out that the culprit for the long back up times is the LR catalog file which has grown to 54+ GB. …

How many images do you store in your catalog? 54GB seems huge! Or.. do you also backup your previews?

Edit: I don't backup previews  as they can be regenerated.

[PeterAit]

I'm a backup fanatic. I have have MacPro 8-core as my workstation with 4 x 1TB internal drives. 2 of these are in a RAID 0 configuration for my photo and video working storage. I backup up several times to a FW800 external drive (OWC dual drive external RAID 0) mirroring my internal RAID 0.

Once a week, I backup to a ReadyNAS NV+ with 4TB of storage in in RAID X configuration. What I noticed is that this backup (mirror the drive) takes about 8 hours. I found out that the culprit for the long back up times is the LR catalog file which has grown to 54+ GB. I've tried Chronosync and Synchronize X Pro using the "dissect packages" setting so that it only mirrors the components of the Catalog file that have changed. It still takes forever.

I'm thinking about moving the catalog to my boot HD so that it gets backup when I make a SuperDuper clone of my boot HD each week. I will exclude the Catalog file from my normal Time Machine backup as recommended by others.

I was curious how others are backing up their LR Catalog files?

Thanks.

Bud James
North Wales, PA

I used to use LR's backup feature, so that it backed up once a week (or whatever) when you started LR - but that delays things just when you are ready to get to work, even if it's only for 5-10 minutes for my much smaller catalog. Now I use SyncToy (for PC, but I am sure something similar is available for the Mac) to back up not only the catalog but the original images on a regular basis. It runs at night so I don't really care how long it takes. Also, there's no need to back up your previews as they can be regenerated - and, at least in my case, the preview files take up about 7x as much space as the catalog itself.

Peter

[budjames]

I have about 65,000 images, mostly Canon RAW files from several vintages of DSLRs, in my database. My default for importing images is minimum preview size.

My backups are actually mirroring the "photos" folder on my RAID 0 that has the images and LR Catalog to a folder on my ReadyNAS NV+ on my gigabit LAN. Apparently the LR Catalog database contains thousands of files and folders which makes it time consuming to delete and copy files over my network.

I'm going to try moving my LR database to a different folder that I can backup separately to see if that speeds up making mirror images to the my ReadyNAS.

Bud James
North Wales, PA

[francois]

I have about 65,000 images, mostly Canon RAW files from several vintages of DSLRs, in my database. My default for importing images is minimum preview size.

My backups are actually mirroring the "photos" folder on my RAID 0 that has the images and LR Catalog to a folder on my ReadyNAS NV+ on my gigabit LAN. Apparently the LR Catalog database contains thousands of files and folders which makes it time consuming to delete and copy files over my network.

I'm going to try moving my LR database to a different folder that I can backup separately to see if that speeds up making mirror images to the my ReadyNAS.

Bud James
North Wales, PA

Bud,
I thought that you were talking about a 54GB .lrcat file... In my main catalog, I have about 35,000 and my catalog (the .lrcat file)is around 650MB only. Files consist of old scanned photos, Canon raw files (5D, 1D2, 1ds3, 400D) and a few Nikon NEFs.

[budjames]

Bud,
I thought that you were talking about a 54GB .lrcat file... In my main catalog, I have about 35,000 and my catalog (the .lrcat file)is around 650MB only. Files consist of old scanned photos, Canon raw files (5D, 1D2, 1ds3, 400D) and a few Nikon NEFs.

Francois, yes, my LR 2.2 folder containing the database is 54+GB. The actual images (RAW and some JPEGs) take about 500GB of drive space on my internal 2 x1TB RAID 0.

I tried rebuilding and optimizing the LR database and that didn't change the size noticeably.

Bud

[francois]

Francois, yes, my LR 2.2 folder containing the database is 54+GB. The actual images (RAW and some JPEGs) take about 500GB of drive space on my internal 2 x1TB RAID 0.

I tried rebuilding and optimizing the LR database and that didn't change the size noticeably.

Bud

Bud,
I believe that in your Lightroom folder, you must have two files. One is the Catalog (or Library) and the second one is the file that contains the previews. This file can be very large (in Gigabytes) but since Lightroom will regenerate your previews, there's no need to back it up.

I've enclosed a screenshot of the two files I have in my Lightroom folder... Your situation should be similar.

Keep me posted.

[budjames]

Bud,
I believe that in your Lightroom folder, you must have two files. One is the Catalog (or Library) and the second one is the file that contains the previews. This file can be very large (in Gigabytes) but since Lightroom will regenerate your previews, there's no need to back it up.

I've enclosed a screenshot of the two files I have in my Lightroom folder... Your situation should be similar.

Keep me posted.

Francois, you are brilliant and correct!

My LR database is about 1GB. The preview folder is 53GB+.

In order to speed up my mirrored backups, I excluded the previews folder from source files list. Now it works much faster.

Thanks.

Bud James

[jjj]

I seem to recall someone 'saved' some images that had corrupted [don't forget that with this sort of back up, it backs up any corruptions as well], by finding the 100% previews.

[francois]

Francois, you are brilliant and correct!

My LR database is about 1GB. The preview folder is 53GB+.

In order to speed up my mirrored backups, I excluded the previews folder from source files list. Now it works much faster.

Thanks.

Bud James

That's good to hear... Good backup!

[francois]

I seem to recall someone 'saved' some images that had corrupted [don't forget that with this sort of back up, it backs up any corruptions as well], by finding the 100% previews.

Correct, one can use the preview file to extract images, provided it's not corrupted but I wouldn't use it as my first line backup... (and I understand that it's not what you suggested    but naive users might think that it's a good shortcut for backups).

[jjj]

I doubt a naive user would even think of that method.
And it's anything but a shortcut.

[howardm]

Bud,

Are you *sure* you're running RAID 0?  That REALLY is not what you want.  I'm hoping you actually mean RAID 1.  In a RAID 0 configuration, a
failure of any disk destroys all data vs. RAID 1 which is a mirror/duplicate of the data.

When was the last time you tried to actually restore from a  real/simulated disk failure?

[budjames]

Bud,

Are you *sure* you're running RAID 0?  That REALLY is not what you want.  I'm hoping you actually mean RAID 1.  In a RAID 0 configuration, a
failure of any disk destroys all data vs. RAID 1 which is a mirror/duplicate of the data.

When was the last time you tried to actually restore from a  real/simulated disk failure?

Yep, it's a RAID 0, but for speed and capacity. The RAID 0 is backed daily to disks in a JBOD eSata tower.

Bud