Pages: [1]   Go Down

Author Topic: Lightroom 3 indexing and caching enhancements  (Read 3663 times)

Steve Weldon

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1479
    • Bangkok Images
Lightroom 3 indexing and caching enhancements
« on: June 12, 2010, 02:16:53 pm »

LR3 appears to have become refined and smooth enough where its now solidly my main tool for cataloging, viewing, and most run of the mill processing.  For finer work I use C1pro..

For grins I set up a new catalog on my Crucial C300 256g SSD.. built indexes and 1:1 high quality previews for 10,000 images just to see what it would be like.

Amazing instant "in the blink of an eye" navigation from frame to frame, collection to collection, etc.  If you spend a significant amount of time in LR3 you will instantly recognize the utility in this level of performance and the grin won't leave your face.

The 'test' taught me several things.

[blockquote][/indent]1.  From now on my indexes (catalogs) and previews will be on their own drive.

[blockquote][/blockquote]2.  I want this level of performance and I want 1:1 high quality previews available for my entire current archive (164,000 images)

[blockquote][/blockquote]3.  There's no way I'm going to spend the money necessary for a SSD of the necessary capacity (800-900g), even if you extend across 3-4 of them.  That's a serious amount of money for a smile..

[blockquote][/blockquote]I started looking at alternatives.  I came up with these.

[blockquote][/blockquote]1.  An inexpensive current model 1-2tb SATAII drive  (less than $200)  Least performance.

[blockquote][/blockquote]2.  (2) 600g Raptor SATA II's (about $575 at Newegg)   More performance.

[blockquote][/blockquote]3.  (2) new Seagate Momentus 2.5inch SATA II 500g Hybrid drives  (about $280 via Amazon pre-order)  Even better performance.

[indent][/blockquote]4.  (4) C300 Crucial 256g SATA III SSD's  ($2600 at Amazon not including 4 port SATA III controller, but can be used with less performance with a SATA II controller)  Best performance this side of PCIx SSD's at 3-4x the price.

The Seagate Momentus drives seem like the best compromise for performance and price.. Tests/reviews show them to exceed the performance of the Raptors but seriously lags a full SSD.  

A concern with any of these options is taking up a SATA port much less 2 or more.. but for the amount of time I spend in LR3 and the luxury of speed.. (grin not included) I'll give up 2 of my 10 SATA II ports on my MB..

These drives will hit the shelves the first week in July.. I can wait that long and use a ordinary 1tb SATA II HDD until then..

Before I always kept my indexes/previews to under 200g.. but now I'm willing to invest more and more importantly I think it will increase my productivity more than several of the options will cost.

Does anyone else have thoughts on this subject or perhaps a better alternative?  I know keeping 1:1's on so many images is borderline silly.. but if I can do it and get great performance for under $300. I'm willing to pay for it.
« Last Edit: June 12, 2010, 02:18:25 pm by Steve Weldon »
Logged
----------------------------------------------
http://www.BangkokImages.com

John R Smith

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1357
  • Still crazy, after all these years
Lightroom 3 indexing and caching enhancements
« Reply #1 on: June 12, 2010, 02:28:30 pm »

Well, Steve, you have a lot of images. I have maybe 1,000 tops, going back to 1980. (I just don't take many pictures). So for me, SSD would probably be the way to go.

Your assessment of the current drives seems very sound. But I would reckon SSD prices will drop by at least 30% by next year, maybe 50%. Look at what is happening with flash memory cards right now. So it might be best to sit it out for a year and see how things stack up then.

John
« Last Edit: June 12, 2010, 02:29:14 pm by John R Smith »
Logged
Hasselblad 500 C/M, SWC and CFV-39 DB
an

Steve Weldon

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1479
    • Bangkok Images
Lightroom 3 indexing and caching enhancements
« Reply #2 on: June 12, 2010, 02:37:19 pm »

Quote from: John R Smith
Well, Steve, you have a lot of images. I have maybe 1,000 tops, going back to 1980. (I just don't take many pictures). So for me, SSD would probably be the way to go.

Your assessment of the current drives seems very sound. But I would reckon SSD prices will drop by at least 30% by next year, maybe 50%. Look at what is happening with flash memory cards right now. So it might be best to sit it out for a year and see how things stack up then.

John
Sure, with a 1000 images you could probably fit them on your system SSD.

There's no doubt prices will drop.. but I put in too many hours on the system to wait a year with the hope of a price drop.. even 30% off $2600.. too much.  $100 each and now we'd be talking for this purpose.

On the other hand.. the performance increases and time savings I get from my current C300 256g SSD's on my main workstation and personal laptop.. are well worth their current price.

Sometimes we just can't have it all..    Though.. these hybrids will get us a lot closer..
Logged
----------------------------------------------
http://www.BangkokImages.com

NikoJorj

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1082
    • http://nikojorj.free.fr/
Lightroom 3 indexing and caching enhancements
« Reply #3 on: June 13, 2010, 08:25:30 am »

Quote from: Steve Weldon
3.  (2) new Seagate Momentus 2.5inch SATA II 500g Hybrid drives  (about $280 via Amazon pre-order)  Even better performance.
From what I've read, the hybrid Momentus has 4GB of SSD memory for 500g of HDD, where the most frequently accessed files are duplicated for faster access while reading (edited).
The 280$ question is : is 4GB of data enough to significantly fasten LR with a big catalog, ie many raw files and big previews?
« Last Edit: June 13, 2010, 08:36:41 am by NikoJorj »
Logged
Nicolas from Grenoble
A small gallery

francois

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 13769
Lightroom 3 indexing and caching enhancements
« Reply #4 on: June 13, 2010, 12:49:55 pm »

Quote from: NikoJorj
From what I've read, the hybrid Momentus has 4GB of SSD memory for 500g of HDD, where the most frequently accessed files are duplicated for faster access while reading (edited).
The 280$ question is : is 4GB of data enough to significantly fasten LR with a big catalog, ie many raw files and big previews?
I would be very interested in the hybrid 2.5" HDs but I've read conflicting reports about the improvements over a standard HD.
Logged
Francois

NikoJorj

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1082
    • http://nikojorj.free.fr/
Lightroom 3 indexing and caching enhancements
« Reply #5 on: June 13, 2010, 02:24:07 pm »

Quote from: francois
I've read conflicting reports about the improvements over a standard HD.
Me too, now...  
That may help, but is really no golden bullet at all, from what I can understand (or believe so).
Logged
Nicolas from Grenoble
A small gallery

Steve Weldon

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1479
    • Bangkok Images
Lightroom 3 indexing and caching enhancements
« Reply #6 on: June 14, 2010, 12:24:14 am »

Quote from: NikoJorj
From what I've read, the hybrid Momentus has 4GB of SSD memory for 500g of HDD, where the most frequently accessed files are duplicated for faster access while reading (edited).
The 280$ question is : is 4GB of data enough to significantly fasten LR with a big catalog, ie many raw files and big previews?
The article you linked is the one of several I read.   You're right, it's a 'read' function which really is perfect for an index/preview storage device.. How fast it saves previews is important, but not my priority.  You only create/save them once.  You access them many times.

If you carefully read the article you linked they spoke in depth about how the algorithms learn fast and don't hesitate much at all to change their priority of caching.. again ideal if not perfect for indexes/cache.

Will 4gb be enough.. well.. if you look at your average folder of images and the storage space required to store previews for an average folder.. 4gb is enough for several of my largest folders.  So.. if I go to that folder and peek around I'll be fine.  This will be fine for a purposed search/view.  If I'm just going in random probably not.
« Last Edit: June 14, 2010, 12:25:05 am by Steve Weldon »
Logged
----------------------------------------------
http://www.BangkokImages.com

Steve Weldon

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1479
    • Bangkok Images
Lightroom 3 indexing and caching enhancements
« Reply #7 on: June 14, 2010, 12:37:29 am »

Quote from: francois
I would be very interested in the hybrid 2.5" HDs but I've read conflicting reports about the improvements over a standard HD.
I think there are several things to look at here..

1.  Define standard hard drive.  If you compare this notebook cached drive against other non-cached notebook drives there is no conflict at all.  It's a huge night/day improvement.  Where the conflicts appear is when comparing a cached notebook drive with the fastest available desktop drive (the 600g Raptor).  In this comparison it depends on the purpose you're accessing the drive.  In most tests I've read the Raptor is better at some, worse at others, with the general consensus being the cached hybrid has an edge 'overall..'

2.  The cost of a 2.5 inch notebook 500gb 7200rpm drive hovers around $110.  The cost for this cached 2.5 inch notebook 500gb 7200rpm drive is currently $139 on Amazon.  A $20 difference for what is being reported as a world of difference.  This seems a no brainer for laptops/notebooks IF you're in the market for updating your hard disk.

3.  Any storage device, hard disk, SSD, hybrid.. its performance depends heavily on several factors including it's firmware and drivers..  The algorithms and programming are huge factors and basically define the intended use.  Server HDD's are set up differently than a consumer HDD for a home desktop.  This has always been.  However, when the performance figures multiply 10-20x these differences become that much more noticed/important.  SSD early adopters are finding this out in a big way, the read/write numbers are just a general look at how the drive will perform in your system.  You need to look very closely at the 4k cache, read/write, etc.. and how these numbers compare for your intended use.  With a cached hybrid.. this is even more important.  To take full advantage of the hybrid the drives intended use (algorithms/cache design/chip selection) really comes into play.

4.  I wonder if future hybrids will come packaged for different uses or perhaps have jumpers so a user can fine tune it to their uses?
Logged
----------------------------------------------
http://www.BangkokImages.com

DarkPenguin

  • Guest
Lightroom 3 indexing and caching enhancements
« Reply #8 on: June 14, 2010, 12:55:03 am »

My lightroom catalog is like 400-500mb.  The previews are in the 20gb or so range.  So if I'm trundling through a subset of images I should think that the 4gb would do fine.  On the other hand a $100 SSD would probably handle all that just as well.
Logged

Steve Weldon

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1479
    • Bangkok Images
Lightroom 3 indexing and caching enhancements
« Reply #9 on: June 14, 2010, 01:16:59 am »

Quote from: DarkPenguin
My lightroom catalog is like 400-500mb.  The previews are in the 20gb or so range.  So if I'm trundling through a subset of images I should think that the 4gb would do fine.  On the other hand a $100 SSD would probably handle all that just as well.
Yep.. if you don't think you'll grow much then that 40g SSD from Intel works well.  We just put one in my sons Dell 1720 (it's a notebook that holds two HDD's) this weekend and he's thrilled.. the difference is so profound you have to experience it to see..
Logged
----------------------------------------------
http://www.BangkokImages.com

francois

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 13769
Lightroom 3 indexing and caching enhancements
« Reply #10 on: June 14, 2010, 05:54:43 am »

Quote from: Steve Weldon
I think there are several things to look at here..

1.  Define standard hard drive.  If you compare this notebook cached drive against other non-cached notebook drives there is no conflict at all.  It's a huge night/day improvement.  Where the conflicts appear is when comparing a cached notebook drive with the fastest available desktop drive (the 600g Raptor).  In this comparison it depends on the purpose you're accessing the drive.  In most tests I've read the Raptor is better at some, worse at others, with the general consensus being the cached hybrid has an edge 'overall..'

2.  The cost of a 2.5 inch notebook 500gb 7200rpm drive hovers around $110.  The cost for this cached 2.5 inch notebook 500gb 7200rpm drive is currently $139 on Amazon.  A $20 difference for what is being reported as a world of difference.  This seems a no brainer for laptops/notebooks IF you're in the market for updating your hard disk.

3.  Any storage device, hard disk, SSD, hybrid.. its performance depends heavily on several factors including it's firmware and drivers..  The algorithms and programming are huge factors and basically define the intended use.  Server HDD's are set up differently than a consumer HDD for a home desktop.  This has always been.  However, when the performance figures multiply 10-20x these differences become that much more noticed/important.  SSD early adopters are finding this out in a big way, the read/write numbers are just a general look at how the drive will perform in your system.  You need to look very closely at the 4k cache, read/write, etc.. and how these numbers compare for your intended use.  With a cached hybrid.. this is even more important.  To take full advantage of the hybrid the drives intended use (algorithms/cache design/chip selection) really comes into play.

4.  I wonder if future hybrids will come packaged for different uses or perhaps have jumpers so a user can fine tune it to their uses?


1 & 2:My remark was hybrid vs laptop 2.5" HD. I'll upgrade very soon but the new hybrids haven't yet hit the swiss stores and I haven't seen test with Lightroom or/and Photoshop. Anyway, the asked price is, as you mentioned, not much more expensive than the non-hybrid HD, so the upgrade is a no-brainer for me.
Logged
Francois

Steve Weldon

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1479
    • Bangkok Images
Lightroom 3 indexing and caching enhancements
« Reply #11 on: June 14, 2010, 11:07:39 am »

Quote from: francois
1 & 2:My remark was hybrid vs laptop 2.5" HD. I'll upgrade very soon but the new hybrids haven't yet hit the swiss stores and I haven't seen test with Lightroom or/and Photoshop. Anyway, the asked price is, as you mentioned, not much more expensive than the non-hybrid HD, so the upgrade is a no-brainer for me.
I think for people who need that much capacity and don't want to lay out the cash for a solid performing SSD then the hybrids really are a no-brainer.

So far I've put SSD's in three laptops and they totally changed the character of the laptop.  It was like getting a new machine or better.  Two laptops were high-end C300's and the other was the low end 40g Intel and I didn't expect that much from it.. but the 4k read/writes really are where the performance is more than the sequential read/writes.. and the intel performs nearly as well as the 6x more expensive C300's.. though with only a fraction of the capacity.  Hybrids do well on the 4k tests so I expect them to significantly improve real life use for most laptop/notebook users.

I can't help but think a 600-1tb Raptor with hybrid capabilities would serve those wanting high performance really well, at a much lower price point than a full SSD.. plus with 2-3x the capacity.  They must be planning one.. or maybe maybe they don't want to cut into their new SSD sales.

I suppose I'm pretty pumped about the performance enhancements solid state storage delivers.  It's made more difference in performance than any computer/cpu/memory upgrade I can remember.

As a side note:  I've finally finished indexing and building 1:1 previews for my images.  I'm not sure why, perhaps LR3's database capabilities are that much better, or perhaps dedicating its own mechanical HDD for just the indexes/previews.. but performance is way better than LR2.7 with my old indexes/previews.

I hesitate to recommend anyone use this much time for anything.. but if you're unhappy with your LR indexing or preview rendering.. then maybe totally starting over with the indexes and previews would be a benefit.
Logged
----------------------------------------------
http://www.BangkokImages.com
Pages: [1]   Go Up