Pages: 1 [2] 3   Go Down

Author Topic: Storage systems and methods for photographers  (Read 18858 times)

Dan Wells

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1044
Re: Storage systems and methods for photographers
« Reply #20 on: February 27, 2017, 06:13:10 pm »

      Yes, QNAP is using the Thunderbolt networking feature - I have mixed feelings about that (it has the unpleasant feature of grabbing control of the Thunderbolt bus, but pleasantly it is much less of a problem on sudden disconnect than an enclosure)... You care much more about it grabbing the bus if you have a Mac with only one Thunderbolt port, because you can't daisy-chain anything to the QNAP. With two or more ports, simply put displays and/or other drives on a different port (this works very well on a 2015 MBP). They say it's actually 20 gigabits per second, not 10 (WHAT kind of disks would it take to test that???).
       The fact that it has a computer inside enables a couple of really nice features - it can back itself up with no interference from an external computer (mine has its own WD MyBook attached via USB, and it keeps a copy of itself on the MyBook updated daily). It'll do this even if I'm out of town for a week with my laptop. It's also accessible over the Internet like any other NAS - not at the blazing speed of a Thunderbolt connection, but try that with a G-tech RAID. A secondary feature, but a nice one, is that it'll control a UPS and shut itself down in a controlled manner if the power fails, resuming on its own when the power comes back. You can sort of get an enclosure to do that if you have it set to power up and down with the computer , and the computer is controlling the UPS(if the computer's not a laptop - a laptop will hum merrily along on its internal battery while the enclosure drains the UPS).
       The Thunderbolt connection saves at least $400 over trying to get a Mac to talk 10 Gb Ethernet ($1000 or more if you want to include a switch in the network)... I agree with Joe that it's basically a 10 Gb NAS that can run its network over Thunderbolt - but that's a crucial advantage if you have Macs, since the only way to get a Mac on 10 Gb Ethernet (other than an ancient Mac Pro with slots) is with an expensive Thunderbolt to 10 Gb adapter...
     So far, I haven't seen any major disadvantage to the QNAP compared to a conventional Thunderbolt RAID on a similar scale, and it's not more expensive than a good one (it IS more expensive than the low-end soft RAIDs like the Thunderbay series, but it's very competitive with higher-end LaCies and G-Techs).
    One day, Apple will make an iMac with 10 Gb Ethernet (who knows, maybe they'll put out a reasonably priced Thunderbolt to 10 Gb dongle at the same time), and then a regular 10 Gb NAS will work for creative pros (at least, as they upgrade their computers) - until then, kudos to QNAP for this odd machine.
« Last Edit: February 27, 2017, 06:21:26 pm by Dan Wells »
Logged

pflower

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 456
Re: Storage systems and methods for photographers
« Reply #21 on: April 20, 2017, 01:34:33 pm »

Thanks for your article which was most helpful, although I fear I might be asking a question you counsel against.

My present set up has evolved over a number of years and is pretty haphazard but works.  I do not do video and currently use a 4TB G-Tech drive on a 5k 27inch iMac (FW800 via a thunderbolt adapter) as my main drive.  Backups, or archives, are via a series of 2TB WD Passport drives.  I am actually quite disciplined in making sure that I have copied everything I do to at least 2 drives at the end of each week and never format a card without having done so.  So the worst that can happen (barring total destruction by fire) is that I lose a week's worth of edits but would still have the original files on a card.

Although I work with quite large files (mostly Hasselblad Raws but some 256mb Tiffs) I can't see that I really need thunderbolt speed - I am happy in that respect with my current set up.  But the 4tb is soon to be full.  So I was thinking about a cheap USB3 dock with bare 6tb drives.  One as main storage and 1 or 2 as backup/archives.

Anyone got any thoughts as to whether or not this is stupid?  If so can anyone recommend a better solution and if not any thoughts on reliability of drives or docks and suggestions on brands?

Thanks



I've been hard at work typing this up, so read to get a bit of an idea how I believe photographers and storage line up.  It's about 70% but I keep seeing incorrect or baseless articles come out and I needed to speak out.

https://medium.com/@PNWMF/storage-systems-and-methods-for-photographers-86e04f940013#.uzejsolsi
Logged

Joe Towner

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1365
Re: Storage systems and methods for photographers
« Reply #22 on: April 20, 2017, 04:29:52 pm »

Unless you start video editing, in reality, USB3 is perfectly fine when plugged directly into the Mac.  I wouldn't do storage off a USB3 hub unless required, but it wouldn't hurt too much.  Thunderbolt speeds really come into play when dealing with SSDs or multiple SSDs.

I wouldn't discount the enclosed 5/6tb drives, especially if you don't have a reason to swap 3.5" drives at this time.  The dock setup ones generally don't stack well  :D

The only bit I would ask is if you have a Flash or Fusion drive.  If you have a Fusion drive, I'd look at doing a 2 bay USB3 enclosure and put a SSD plus a 6tb drive in it.  Use the SSD as a 'working files' space, and leave the fusion drive for everything else.

Logged
t: @PNWMF

pflower

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 456
Re: Storage systems and methods for photographers
« Reply #23 on: April 21, 2017, 02:36:02 pm »

Thanks.  I do have a Fusion drive and also an external SSD directly connected to the iMac on which I have my Lightroom Catalog and the application.  All files however are on my 4TB drive.  I can see the sense in having "working files' on an SSD but frankly I do everything in Lightroom now and moving files around different drives within Lightroom drives me mad and I always end up making a mistake and have to root around in the Finder to find out where everything is.  But with the falling prices of SSDs I might have a look at this again and see if I can't work out a fool proof modus operandi.

I quite fancy the idea of a USB3 dock, and came across the idea from one of OWC's docks - the Drive Dock which sells for £250 or so.  Looking on Amazon there are a huge number of £30 docks.  I suppose for £30 or so it is worth taking a risk, but the price differential is a bit alarming and I am wondering what is the explanation.  There is no point buying something that really can't do the job.



Unless you start video editing, in reality, USB3 is perfectly fine when plugged directly into the Mac.  I wouldn't do storage off a USB3 hub unless required, but it wouldn't hurt too much.  Thunderbolt speeds really come into play when dealing with SSDs or multiple SSDs.

I wouldn't discount the enclosed 5/6tb drives, especially if you don't have a reason to swap 3.5" drives at this time.  The dock setup ones generally don't stack well  :D

The only bit I would ask is if you have a Flash or Fusion drive.  If you have a Fusion drive, I'd look at doing a 2 bay USB3 enclosure and put a SSD plus a 6tb drive in it.  Use the SSD as a 'working files' space, and leave the fusion drive for everything else.
Logged

traderjay

  • Jr. Member
  • **
  • Offline Offline
  • Posts: 85
Re: Storage systems and methods for photographers
« Reply #24 on: July 31, 2017, 05:00:19 pm »

One more thing to add - if you are going to use RAID with large capacity HDDs, RAID 6 is an ABSOLUTE must because you will run into URE (Unrecoverable Read Error) during the array rebuild process. If that happens you can kiss your entire array goodbye ;) 
Logged

Farmer

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2848
Re: Storage systems and methods for photographers
« Reply #25 on: July 31, 2017, 07:54:33 pm »

One more thing to add - if you are going to use RAID with large capacity HDDs, RAID 6 is an ABSOLUTE must because you will run into URE (Unrecoverable Read Error) during the array rebuild process. If that happens you can kiss your entire array goodbye ;)

With consumer level HDDs, this is a genuine risk.  Any array built with drives over 2TB in capacity and the chance of a URE starts to build rapidly.  Enterprise drives will generally be OK to between 3TB and 4TB, but there's still a real risk.  SSD is one way of getting around this, but prohibitively expensive.  RAID 6 is a good choice, but note that even that starts to have questionable reliability with consumer level HDDs after 5TB.

As always, RAID is not a backup solution - it's a redundancy solution.  You need backups that don't rely on RAID somewhere in the chain.
Logged
Phil Brown

Joe Towner

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1365
Re: Storage systems and methods for photographers
« Reply #26 on: August 02, 2017, 12:14:39 pm »

The URE situation has been around since the 2tb drives and older / slower NAS systems.  The fact we're at 10tb and still growing gives longer rebuild times, and thus more chances for failures.

I've done some work and need to update & add to the article.  At the end of the day, you have to treat a NAS and a single external hard drive the same.  The only difference is capacity, performance, and portability.  For each drive added, you have one more failure point.  For each layer of redundancy (RAID5/6, ZFS1/2/3) there is an added point of failure.  I think I  put it as the two options folks should consider today is RAID1 and RAID6.

As to consumer v enterprise drives, I'll just reference the BackBlaze articles https://www.backblaze.com/blog/hard-drive-failure-rates-q1-2017/ They've just added a whole bunch of enterprise drives due to a Seagate sale, but we'll see how they compare.
Logged
t: @PNWMF

traderjay

  • Jr. Member
  • **
  • Offline Offline
  • Posts: 85
Re: Storage systems and methods for photographers
« Reply #27 on: August 03, 2017, 10:59:14 am »

Yes that's why I stay away from consumer grade garbage NAS and stick with FreeNAS and its ZFS file system to minimize bit rot and parity errors. In addition, all my data are periodically backed up onto a separate HDD RAID 1 array in a different part of my house and then gets unplugged to act as cold storage. You can also buy cheap tape drive systems off ebay and do it that way to achieve true multiple redundancy :D
Logged

Jonathan Cross

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 644
Re: Storage systems and methods for photographers
« Reply #28 on: August 23, 2017, 08:17:03 am »

Dear All in this thread.  This has all been very interesting, but I am a novice compared with most of you experts so get blinded by many of the acronyms etc.  I would be grateful for advice/experience for someone thinking whether he is doing the right thing and whether there is a better way.

Currently I backup to two external hard drives via USB3 from my Windows PC and from a Windows laptop.  I backup my Lightroom catalog most times I use Lightroom and backup my images fairly often, particularly if I am clearing a camera drive.  I use Cobian for all my data, not just images and Lightroom catalog, which does an incremental backup 4 times and a full backup every 5th time.  I have been put off using the Cloud as I have been told that if I loose or have a problem with my password I will never get the data back.

I am wondering if it worth investing in a NAS drive with 2 bays using RAID 1.  My knowledge of this system is very basic.  I understand that I can get drives that will do the backups and allow access via USB, wireless, or over the internet.  I may change to Mac, so would need a system that is compatible with both Windows PCs and Macs.  I am also aware that, in an ideal world, off-site, non-networked, storage is desirable. 

One last point, I prefer to do backups when I decide, as I find that having an external hard drive permanently attached slows my boot and normal running speed.

Should I stay as I am or go for NAS, e.g. Synology?  Advice/experience for a novice would be appreciated.

Jonathan


Logged
Jonathan in UK

David Eichler

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 823
    • San Francisco Architectural and Interior Photographer
Re: Storage systems and methods for photographers
« Reply #29 on: August 28, 2017, 05:43:58 pm »

Dear All in this thread.  This has all been very interesting, but I am a novice compared with most of you experts so get blinded by many of the acronyms etc.  I would be grateful for advice/experience for someone thinking whether he is doing the right thing and whether there is a better way.

Currently I backup to two external hard drives via USB3 from my Windows PC and from a Windows laptop.  I backup my Lightroom catalog most times I use Lightroom and backup my images fairly often, particularly if I am clearing a camera drive.  I use Cobian for all my data, not just images and Lightroom catalog, which does an incremental backup 4 times and a full backup every 5th time.  I have been put off using the Cloud as I have been told that if I loose or have a problem with my password I will never get the data back.

I am wondering if it worth investing in a NAS drive with 2 bays using RAID 1.  My knowledge of this system is very basic.  I understand that I can get drives that will do the backups and allow access via USB, wireless, or over the internet.  I may change to Mac, so would need a system that is compatible with both Windows PCs and Macs.  I am also aware that, in an ideal world, off-site, non-networked, storage is desirable. 

One last point, I prefer to do backups when I decide, as I find that having an external hard drive permanently attached slows my boot and normal running speed.

Should I stay as I am or go for NAS, e.g. Synology?  Advice/experience for a novice would be appreciated.

Jonathan

RAID is not for backup. It is for speed or minimizing downtime from drive failure, or both.
Logged

Jim Kasson

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2370
    • The Last Word
Re: Storage systems and methods for photographers
« Reply #30 on: August 29, 2017, 10:46:40 am »

Dear All in this thread.  This has all been very interesting, but I am a novice compared with most of you experts so get blinded by many of the acronyms etc.  I would be grateful for advice/experience for someone thinking whether he is doing the right thing and whether there is a better way.

Currently I backup to two external hard drives via USB3 from my Windows PC and from a Windows laptop.  I backup my Lightroom catalog most times I use Lightroom and backup my images fairly often, particularly if I am clearing a camera drive.  I use Cobian for all my data, not just images and Lightroom catalog, which does an incremental backup 4 times and a full backup every 5th time.  I have been put off using the Cloud as I have been told that if I loose or have a problem with my password I will never get the data back.

I am wondering if it worth investing in a NAS drive with 2 bays using RAID 1.  My knowledge of this system is very basic.  I understand that I can get drives that will do the backups and allow access via USB, wireless, or over the internet.  I may change to Mac, so would need a system that is compatible with both Windows PCs and Macs.  I am also aware that, in an ideal world, off-site, non-networked, storage is desirable. 

One last point, I prefer to do backups when I decide, as I find that having an external hard drive permanently attached slows my boot and normal running speed.

Should I stay as I am or go for NAS, e.g. Synology?  Advice/experience for a novice would be appreciated.


I would not recommend a Synology NAS to a self-professed novice. Just yesterday, I had a Synology NAS drop out of a Windows network. I logged in to the NAS and checked the local time. It was wrong. I want to "Regional Options" and opened up the time control window, saw that it was synching with the right NTP server, which was a Windows DC, and clicked "Manual Sync". That fixed the problem, until the next time the Synonogy NAS decides to stop updating from that NTP server.

Does that sound like something you're ready for?

Jim

FabienP

  • Full Member
  • ***
  • Offline Offline
  • Posts: 192
Re: Storage systems and methods for photographers
« Reply #31 on: August 29, 2017, 05:06:42 pm »


Should I stay as I am or go for NAS, e.g. Synology?  Advice/experience for a novice would be appreciated.

Jonathan

I would like to add that the performance of a NAS on a typical Gbit network is inferior to that of a directly attached disk over USB3. You could typically get 175 MB/s on an external disk and only get 90 MB/s from the same disk installed in a NAS due to the network acting as a bottleneck.

If you do not need to regularly access your data from multiple devices and do not need more storage than what can be copied on today's largest available disk, you probably don't want to bother with a NAS.

Cheers,

Fabien
Logged

Joe Towner

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1365
Re: Storage systems and methods for photographers
« Reply #32 on: August 29, 2017, 10:03:49 pm »

Hey Jonathan,

Let's start with the easy questions - how much data are we talking about?  It's easier to add or upgrade to single disks (Costco has 6tb's for $130 each now).  Jim is right in that NAS systems aren't as easy to deal with external USB3 drives. 

Generally speaking, unless you have more than 6tb of data, a NAS isn't going to add much to a single pc setup.

-Joe
Logged
t: @PNWMF

BobShaw

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2218
    • Aspiration Images
Re: Storage systems and methods for photographers
« Reply #33 on: August 30, 2017, 02:10:25 am »

I would like to add that the performance of a NAS on a typical Gbit network is inferior to that of a directly attached disk over USB3. You could typically get 175 MB/s on an external disk and only get 90 MB/s from the same disk installed in a NAS due to the network acting as a bottleneck.
If you do not need to regularly access your data from multiple devices and do not need more storage than what can be copied on today's largest available disk, you probably don't want to bother with a NAS. Cheers,Fabien
Ditto. It is probably worse than that. I have my storage on a Mac Mini Server with a direct attached Firewire Drobo. That works OK. Backup is to a USB attached HDD. That also works well. Every second night it backs up to a Drobo NAS. Takes much much longer. Perhaps 5-10 times.
If you go to Mac and use the NAS as a backup then they should only be considered a backup of last resort. Time Machine sometimes does not recognise them properly and you can't just drag and drop to restore like you can with a direct attached. The only advantage is physical separation if you can get it far enough away to be worthwhile. Then you probably can't run Gigabit.
Logged
Website - http://AspirationImages.com
Studio and Commercial Photography

Joe Towner

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1365
Re: Storage systems and methods for photographers
« Reply #34 on: August 30, 2017, 02:15:58 pm »

So back to the performance discussion:

Disk performance is the combination of 2 factors: the media type and the connection type.  For hard drives, you can get 60-100MB/s with 7200rpm drives out performing the slower/cheaper 5400/5900rpm models.  For SSD's you're in the 500MB/s range.  SATA is in 1.5/3/6gbps ratings.  USB2 is horrid in it's CPU use, but tops out at 480mbps, Firewire 800 is 800mbps.  USB3 took a lot of the processing and pushed it to the controller and has a 5gbps max.

Yes, networking can be limited to 1gbps, but you get trade offs like larger disks and volumes that have things like redundancy and caching involved.  You can also trunk ports on NAS's to get 4gbps, and there are models now available with 10gbps ports, and there are a number of 10gbps cards that make it viable.

Internal disks will be the fastest performing due to their direct, high speed connection to the CPU.  NVMe's and SSD's make a computer much faster.  If you can't do it internally, my preference of external devices is:

-USB3.1 (aka ThunderBolt3)
-ThunderBolt2 / ThunderBolt1
-USB3
-Firewire800
-USB2

Yes, I left out eSATA mostly because it's rare & limited (more so than FW800).  Keep in mind there are times when you need to keep the bus empty, like when shooting tethered. 

For a vast majority of users, doing a second internal drive is the best, but an external USB3 drive plugged into a USB3 port will do just fine.  If you're not at 6tb of data, there is no reason to look at a NAS.
Logged
t: @PNWMF

Jonathan Cross

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 644
Re: Storage systems and methods for photographers
« Reply #35 on: September 01, 2017, 02:51:10 pm »

Thanks to all who replied to my post; you have saved me money!  I have a total of about 2TB of data, and will stick with USB 3 external drives.  My Windows PC is some 5 years old and was getting slower, both to boot and to run.  My tech guru thought the HD was getting past it with accumulating bad sectors.  He changed it for a faster drive of twice the capacity (2TB from 1TB), and it now boots and runs much faster. For backup and archiving I will stay with external USB 3 drives and not pay for NAS.  I do not need remote access, but could, of course, take a small drive with me if needed.
Thanks again, Jonathan
Logged
Jonathan in UK

Joe Towner

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1365
Re: Storage systems and methods for photographers
« Reply #36 on: September 02, 2017, 01:57:49 pm »

Thanks to all who replied to my post; you have saved me money!  I have a total of about 2TB of data, and will stick with USB 3 external drives.  My Windows PC is some 5 years old and was getting slower, both to boot and to run.  My tech guru thought the HD was getting past it with accumulating bad sectors.  He changed it for a faster drive of twice the capacity (2TB from 1TB), and it now boots and runs much faster. For backup and archiving I will stay with external USB 3 drives and not pay for NAS.  I do not need remote access, but could, of course, take a small drive with me if needed.
Thanks again, Jonathan

Hey Jonathan,

Hate to be the guy giving you bad news, but your tech didn't make things much better.  The biggest speed boost you'll ever see is upgrading that boot drive to a SSD.  EVERYTHING runs faster, even if your photos are on those external USB3 drives.  I'd almost say you could do a 480/500/512/525gb drive, leaving your photos on the second internal drive (that new 2tb drive would be perfect).

-Joe
Logged
t: @PNWMF

Jonathan Cross

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 644
Re: Storage systems and methods for photographers
« Reply #37 on: September 03, 2017, 07:52:34 am »

Yes, Joe, you are right.  I was aware of this option as my tech guru did talk to me about it, but Scrooge beat the chance of speed; I decided I would not pay the cost of SSD.  I am happy with the new HD and external drives.  When I change the whole machine I will go SSD.

Thanks,  Jonathan

 
Logged
Jonathan in UK

David Eichler

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 823
    • San Francisco Architectural and Interior Photographer
Re: Storage systems and methods for photographers
« Reply #38 on: September 03, 2017, 04:49:57 pm »

Internal disks will be the fastest performing due to their direct, high speed connection to the CPU.

All other things being equal, I guess this is probably correct. However, an outboard SSD of recent manufacture, connected by Thunderbolt 1, is somewhat faster than the original internal SSD in my late 2012 Mac Mini. Which leads me to wonder, if I were to put that outboard SSD inside that computer, would it be significantly faster than it is as an outboard drive, or does the design of that internal bus in a 5-year-old computer design not allow that?
Logged

Joe Towner

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1365
Re: Storage systems and methods for photographers
« Reply #39 on: September 04, 2017, 08:56:03 pm »

All other things being equal, I guess this is probably correct. However, an outboard SSD of recent manufacture, connected by Thunderbolt 1, is somewhat faster than the original internal SSD in my late 2012 Mac Mini. Which leads me to wonder, if I were to put that outboard SSD inside that computer, would it be significantly faster than it is as an outboard drive, or does the design of that internal bus in a 5-year-old computer design not allow that?

You could be seeing any number of things - wear on the internal drive, controller speed on the internal drive and chip technology on the internal drive.  The internal SATA port is 6Gb/s, so that isn't the limiter.  Newer SSD's have better controllers (remember the SandForce failures?) and faster internal pieces.  May I ask what drives are shown in MacOS?  Things like the Lacie Little Big Disk are cheating, where there are 2 drives working together.

You can also run the BlackMagic Disk Speed Test to compare their performance.
Logged
t: @PNWMF
Pages: 1 [2] 3   Go Up