Luminous Landscape Forum

Equipment & Techniques => Computers & Peripherals => Topic started by: Joe Towner on January 31, 2017, 03:22:45 pm

Title: Storage systems and methods for photographers
Post by: Joe Towner on January 31, 2017, 03:22:45 pm
I've been hard at work typing this up, so read to get a bit of an idea how I believe photographers and storage line up.  It's about 70% but I keep seeing incorrect or baseless articles come out and I needed to speak out.

https://medium.com/@PNWMF/storage-systems-and-methods-for-photographers-86e04f940013#.uzejsolsi
Title: Re: Storage systems and methods for photographers
Post by: rdonson on January 31, 2017, 04:30:43 pm
A good start, Joe!  I'm sure many people will benefit from this when you've finished.
Title: Re: Storage systems and methods for photographers
Post by: NancyP on January 31, 2017, 04:45:39 pm
Thanks!
Title: Re: Storage systems and methods for photographers
Post by: davidgp on February 01, 2017, 02:33:49 am
Hi Joe,

I just did a quick read to your article. When you are talking about the internal drive you said that all drives are SATA now, not exactly truth now a days. You can install PCIe mvne bus SSD drives now, maybe connected directly to a PCIe slot or to an m.2 connector (Apple uses instead a proprietary connector but it is just basically a PCIe drive).

If someone is buying a new PC (for macs if you select opt he SSD option it will be already one of this drives), it is better to buy one of these drives, such as a Samsung EVO 960... It will be around 3x faster than the SATA equivalent and just a bit more expensive.

Regards,

David


Enviado desde mi iPad utilizando Tapatalk
Title: Re: Storage systems and methods for photographers
Post by: Joe Towner on February 01, 2017, 11:39:21 am
Hi Joe,

I just did a quick read to your article. When you are talking about the internal drive you said that all drives are SATA now, not exactly truth now a days. You can install PCIe mvne bus SSD drives now, maybe connected directly to a PCIe slot or to an m.2 connector (Apple uses instead a proprietary connector but it is just basically a PCIe drive).

If someone is buying a new PC (for macs if you select opt he SSD option it will be already one of this drives), it is better to buy one of these drives, such as a Samsung EVO 960... It will be around 3x faster than the SATA equivalent and just a bit more expensive.

Regards,

David


Enviado desde mi iPad utilizando Tapatalk

Hey David,

Yep, NVMe drives are a whole different ball of wax, which I should state a bit clearer.  The price premium is totally worth it - I really wish they'd do some sort of slot much like the RAM on iMacs where you can easily upgrade to a larger size.  The speed and capacity of a 2280 drive is stunning.

Thanks for the feedback, I've got lots more to write, but I'm pulled many different directions.  I need to figure out how to do chapters or such on Medium.

-Joe
Title: Re: Storage systems and methods for photographers
Post by: Jim Kasson on February 01, 2017, 11:47:21 am
I've been hard at work typing this up, so read to get a bit of an idea how I believe photographers and storage line up.  It's about 70% but I keep seeing incorrect or baseless articles come out and I needed to speak out.

https://medium.com/@PNWMF/storage-systems-and-methods-for-photographers-86e04f940013#.uzejsolsi

Joe, I think this is great.

A few thoughts in general first.

You might want to back up a bit and talk about the difference between backup and archiving, like I did in this (unfortunately, by now somewhat dated) article:

http://blog.kasson.com/how-to/backing-up-photographic-images/

You should also talk about off-site vs on-site storage, and storage hierarchies (maybe using another name to make it easy on the less technical).

Now, some specific comments:

You say: “Direct Attach can run at a much faster speed than your wifi or wired network can (in most cases).”

You might want to amplify that. Specifically, I’ve found that Synology Rack NAS boxes attached with a single 1Gb/s Ethernet are generally as fast, and often faster, that USB3 drives. External USB3 drive enclosures sometimes provide much slower transfer speeds than my NAS connections, which run at wire speed. Even though USB3 wire speed is five times higher, I’ve never seen much improvement over NAS speeds. In addition, it seems that many USB3 implementations share bandwidth, so that other USB3 transfers can slow down (of course, that’s true if you only have a single Ethernet connection). 10Gb/s Ethernet is finally dropping in price, but that’s still beyond the scope of your intended audience, I think, as is Ethernet bonding. It is a surprise to me that the more complicated protocol stack for Ethernet and either IP or Windows Domain networking runs faster than the leaner USB3 DASD protocol, but that seems to be the case. Maybe more time to optimize the code? Maybe hardware restrictions in USB3 implementations?

You say: “Returning to the backup topic, most cloud backup platforms works fine with DAS disks, but not for NAS setups.”

I think you’re talking about the cloud backup services that use their own client software. I don’t use those. I’ve tried many, and found they are generally opaque, inflexible, and buggy. For cloud backup (and for on-site backup), I prefer a third part backup app like GoodSync, although that limits you to cloud backup services that don’t require their own apps, like Amazon S3 or Dropbox.

When you talk about Wi-Fi, you talk about it coming from a router that supports it. That’s not how I do it, and there are probably others that think like me. I have a router connected to my ISP. The LAN side of that router is connected to the WAN side of a SonicWall firewall, the LAN side of the firewall is connected, via Ethernet switches, to wireless access points distributed about the house. This allows for more reliable Wi-Fi and greater coverage than I could possible get if I bought a router with built-in Wi-Fi.

You talk about enterprise SAS drives (which I agree are overkill) and (by implication) consumer SATA drives, but you don’t mention my preferred choice for spinning in arrays, which is enterprise SATA drives. I’ve gotten no failures since I started to switch over to them about two years ago (I have about 35 drives running, so the sample size is small), and I’m always nervous during rebuilds, even with RAID 6 NAS boxes. On the remaining commercial drive arrays, I’ve configured hot spares, so if I don’t get to the server room for a week or two to notice a failure, the array has already rebuilt itself. I really should configure email alerts.

I like what you’ve said about striping. I used to be a big fan of three striped 8TB Helium disks for first-line photo storage. I’m beginning to think about SSD for that, but at present, that’s a lot of disks. Your remarks about software RAID also intrigue me. I’ll have to look at that. I’ve been using PCIe RAID controllers for the striping. I don’t use RAID5 and 6 on workstations, but aren’t there write speed problems with software versions of those, since the software has no NV RAM at its disposal? Or do the hardware RAIDs use NV RAM anymore?

You say: “if you have an Adaptec RAID card and it throws a fit and dies, you have to get a similar Adaptec RAID card to read the contents of the disk.” I don’t understand this. Why not just put in whatever the heck you want and restore from a backup? If the RAID controller dies, your array is out of service anyway, and will be for several days unless you have a spare controller on-site.

Anyway, that’s my two cents. I’m looking forward to reading the complete paper.

Jim





Title: Re: Storage systems and methods for photographers
Post by: Joe Towner on February 01, 2017, 04:12:13 pm
Thanks for the great feedback Jim.  I need to go into how separating your active work from your archive work is critical, mostly because RTO on the archive can be a week or two, while the active work is hours at worst.  I also need to talk about how to scale a system when you outgrow a NAS and need to either buy the next bigger size or hope their drive expansion works as promised.

As for enterprise SATA drives, I've seen some interesting info come out of Backblaze: https://www.backblaze.com/blog/hard-drive-benchmark-stats-2016/

I'm bugging OWC to give us a way to mix & match drives in their ThurnderBay product - mixing SSDs and drives of different sizes would make setup for end users much easier.  Then again, I'd love an 8 bay setup, mix in a few SSDs with some spinning rust and such.  Highpoint has one, but it's an external TB-SAS and then SAS expanders inside.

There's lots to talk about, and how tech keeps changing.  There are upsides to every decision, and there is an associated cost, and what we keep trying to do is maximize the value per dollar we put in.  The way some other blogs are talking about storage and needing RAID and all this stuff, it's making blanket assumptions.  Greatest example of non-RAID user is BCooter and his/their stacks of Lacie rugged external drives.
Title: Re: Storage systems and methods for photographers
Post by: Jim Kasson on February 01, 2017, 04:43:39 pm
Thanks for the great feedback Jim.  I need to go into how separating your active work from your archive work is critical, mostly because RTO on the archive can be a week or two, while the active work is hours at worst.  I also need to talk about how to scale a system when you outgrow a NAS and need to either buy the next bigger size or hope their drive expansion works as promised.

As for enterprise SATA drives, I've seen some interesting info come out of Backblaze: https://www.backblaze.com/blog/hard-drive-benchmark-stats-2016/

I'm bugging OWC to give us a way to mix & match drives in their ThurnderBay product - mixing SSDs and drives of different sizes would make setup for end users much easier.  Then again, I'd love an 8 bay setup, mix in a few SSDs with some spinning rust and such.  Highpoint has one, but it's an external TB-SAS and then SAS expanders inside.

There's lots to talk about, and how tech keeps changing.  There are upsides to every decision, and there is an associated cost, and what we keep trying to do is maximize the value per dollar we put in.  The way some other blogs are talking about storage and needing RAID and all this stuff, it's making blanket assumptions.  Greatest example of non-RAID user is BCooter and his/their stacks of Lacie rugged external drives.

Joe, there's another point that can't be emphasized too much: if you're not doing trial restores every so often (to backup, but direct attached, stores), the you don't know that your backup solution is working. I had an example of that occur to me recently, although not in a workstation environment. I was setting up three test WordPress instances. I created fresh instances, loaded UpDraft, and tried to restore from my Dropbox account. The restores failed with "corrupt zip" messages. I looked at the files on Dropbox, and they were indeed corrupt.  I restored from an ftp copy, and it worked. Then I told UpDraft to write the live site files to its own storage servers instead of Dropbox. I created new fresh instances and restored to them with UpDarft using the backup copies on its own servers. That worked. So somehow the files were being corrupted as UpDraft was writing them to Dropbox.

Let's say that I never did what amounted to a trial restore. And let's say that I didn't have ftp'd copies of all the files stored locally thanks to a GoodSync script. Now let's say I'd lost the live site. I'd have been SOL.

And there's a psychological reason to do trial restores. You can practice when you're not in a panic, and you'll know what to do when you have to do a restore in anger. In work environments in the past, I can't tell you how many times I've seen a bad situation made much worse by dumb mistakes caused by IT folks who were frantic and not thinking clearly.

Automated backup is wonderful, but you then have the possibility of having wholesale errors instead of the retail ones you get with less automation. Like having someone copy a corrupt set of files over your backup copy. Sounds impossible, doesn't it? Just have two ftp clients open simultaneously and get confused which is which, and it can happen.

Jim
Title: Re: Storage systems and methods for photographers
Post by: Joe Towner on February 01, 2017, 05:32:01 pm
if you're not doing trial restores every so often (to backup, but direct attached, stores), the you don't know that your backup solution is working.

...
Jim

+1000 - it's part of why I like sync rather than backup, partly because I don't have to find the license & install the software to recover a file, but I can literally just plug it in and browse the data at random.  Organization is the base of any workflow, the more organized you can be, the better.
Title: Re: Storage systems and methods for photographers
Post by: degrub on February 01, 2017, 06:05:17 pm
that is correct - sync is not backup. Once a file is corrupted on the sync media, the change can be picked up on the next sync. or vice versa depending on the trigger.
Frank
Title: Re: Storage systems and methods for photographers
Post by: scyth on February 01, 2017, 06:12:58 pm
you don't know that your backup solution is working.

just yesterday = https://www.theregister.co.uk/2017/02/01/gitlab_data_loss/
Title: Re: Storage systems and methods for photographers
Post by: Jim Kasson on February 01, 2017, 06:18:42 pm
+1000 - it's part of why I like sync rather than backup, partly because I don't have to find the license & install the software to recover a file, but I can literally just plug it in and browse the data at random.  Organization is the base of any workflow, the more organized you can be, the better.

I'm with you, except maybe for the terminology. In the apps that I use for backup, GoodSync and Vice Versa, the difference between synch and backup is just a checkbox when you set up each script. In either case, the backed up files are browsable and restorable randomly, using any program you please. The difference is that with synch, changes on the backup dise are propagated to the live side, which I consider dangerous.

But just looking at the files on the backup is not enough; you need to establish that you can restore them, and that they're not corrupted after the restore, partly so you know all that works, and partly to train yourself so you know what to do when you do it for real. Sounds like you agree with that, too.

OBTW, one of the big problems with doing backups by disk images is that you can hardly ever get anyone to do a trial restore on top of their perfectly good workstation.

Jim

Jim
Title: Re: Storage systems and methods for photographers
Post by: Tony Jay on February 01, 2017, 08:17:55 pm
I would definitely encourage you to complete the task!

I think that this particular aspect of digital asset management is probably the Achilles heel of a lot of photographers.
Perhaps even a couple of example back-up/archiving workflows could be constructed with relevant hardware and software components along with the relative pros and cons of each approach could be considered.

I appreciate the work that has already gone into explaining the various options and configurations that are practically available however to those who are perhaps not on the level of an IT professional it is a bit like presenting the occasional mechanic with a truck full of spare parts - put together correctly one gets a lamborghini - but to the occasional mechanic they are destined to forever remain a pile of spare parts!

I have so frequently come across individuals who fail to get the "big picture".
In a different, but related context, I recently had dealings with a photographer who run into trouble converting images to DNG. Her reasoning for doing this was to get rid of the annoying XMP side-car files accompanying her raw images. She was voicing concerns about how she would now have to "manually" update the previews in the DNG files to get accurate previews (this despite the fact that she only used Lightroom and had no intention of using any other software for viewing images). When I pointed out the mechanisms for doing this in Lightroom (as well as the point that it was irrelevant for her purposes) she primly pointed out that she did not want to write metadata back to file anyway. When I asked her what the origin of the XMP side-car files were, I got the strange answer that this was necessary to ensure that her edits were saved! When I explained that every edit, including the history was automatically saved into the catalog, it was clear that she had no idea.

I think this story illustrates a common problem in digital asset management where individuals are using the "tools" without really understanding the bigger picture and how things logically interrelate.
Even more interesting was that I discovered that her back-up and archiving regime was pretty good - in complete contradistinction to her application of digital asset management within Lightroom.
(I had fears that, given the first part of the story related above, that the back-up strategy would be a mess - but it was not the case.)
I am pretty sure that her back-ups were designed and implemented by somebody else though.

So, in summary, I feel that the background information you have shared is great, but would probably be lacking without a couple of concrete, if simply representative, workflow examples.

Tony Jay
Title: Re: Storage systems and methods for photographers
Post by: davidgp on February 02, 2017, 02:21:55 am
Hi Joe,

I see that Jim already gave you a lot of suggestions... Not much experience with RAID systems, so I can not comment there, but the topic of trying to restore files from your backup I fully support it.

Now, since you are mentioning NAS systems like Synology... I will suggest for those users, if possible, to use BTRFS filesystem over EXT4 since it is resilient to bit-roting https://arstechnica.com/information-technology/2014/01/bitrot-and-atomic-cows-inside-next-gen-filesystems/

For Apple users, looks like they are going to do a transition to a new filesystem this year or next one (long overdue), I hope they also add bit-rooting support, I think the beta versions available right now does not have it. Not sure for Windows if NTFS supports it, but I will suspect they don't


Enviado desde mi iPad utilizando Tapatalk
Title: Re: Storage systems and methods for photographers
Post by: Jim Kasson on February 02, 2017, 04:06:26 pm

As for enterprise SATA drives, I've seen some interesting info come out of Backblaze: https://www.backblaze.com/blog/hard-drive-benchmark-stats-2016/

Thanks. I switched to HGST drives for workstations the year before last, and HGST Helium drives for new hotswap arrays last year. I think that enterprise drives do make sense if not easily swappable, and may make sense if they are. Looks like HGST is their most reliable vendor, although they don't have much experience with HGST 8TB drives.

Jim
Title: Re: Storage systems and methods for photographers
Post by: davidgp on February 03, 2017, 03:11:03 am
Thanks. I switched to HGST drives for workstations the year before last, and HGST Helium drives for new hotswap arrays last year. I think that enterprise drives do make sense if not easily swappable, and may make sense if they are. Looks like HGST is their most reliable vendor, although they don't have much experience with HGST 8TB drives.

Jim

Hi Jim,

Being a Backblaze user I have being following their studies for a long time. Even if initial stats for them show that model X is better than model Y. The problem for them it is that maybe it is more cheaper and easy to get drive Y in large quantities than X... This is the reason you see they buy more Seagates than other brand, even after their problems with the 1.5 or 3 tb drives that they have years ago...

Also, it was last year that they started to buy 8TB drives in large quantities, after the ratio dollar per gigabyte was interesting for them (you also have to consider that they put 45 drives into each rack unit, so for them, using less rack units is part of the equation).

But, they always have some rack units with newer and expensive drives to test the waters before buying large quantities of an specific drive.

Regards,

David
Title: Re: Storage systems and methods for photographers
Post by: Dan Wells on February 17, 2017, 12:10:03 am
QNAP has an interesting new product out there that combines some of the advantages of NAS and DAS - it's a NAS (actually a series of them - they come in 4, 6 and 8 bay versions that are confusingly named the 682T (4-bay), 882T (6-bay) and 1282T (8-bay)) that connect via gigabit Ethernet, 10 Gig Ethernet (fast, but you don't have anything else that uses it) AND Thunderbolt. The odd naming scheme is because they're counting dedicated SSD caching bays (2 on the 682T and 882T, 4 on the 1282T) as drive bays - if you include the 2.5" SSD bays, a 682T DOES have 6 bays - only 4 of which accept regular hard drives... To further confuse things, it accepts 2 M.2 SSDs in addition to the 2 2.5"SSDs, and it doesn't count those slots as bays!

The advantage of the Thunderbolt connection is that it's extremely fast. I just got a 682T, and I've seen it break half a gigabyte per second consistently as I've transferred data to it (from the very quick PCIe SSD on my MacBook Pro). I've never seen another external drive go that fast in the real world, and the closest I've seen anything come is very expensive Fibre Channel RAIDs (I suspect a modern Thunderbolt DAS with SSD caching would be in the same range, and a Thunderbolt SSD might be close). Yes, USB 3.0 is theoretically rated that fast if you had enough fast drives connected to it, but I've never seen a USB 3.0 device come CLOSE to that - I've rarely seen one break 200 MB/s(even with a SSD in a USB 3.0 housing). USB of any variety always seems to have overhead that keeps it well below its rated speed, which only matters if your drives are very quick (fast RAID or SSD) to begin with, because USB 3.0 is faster than any single spinning drive. Standard gigabit NAS units are about as fast as most USB 3.0 drives - I've seen them get very close to their wire speed of 128 megabytes (1 gigabit) per second. The QNAP is more than 4 times faster than a conventional NAS, three times as fast as a fast USB 3.0 drive, and will hold its own against even the fastest Thunderbolt RAIDs (I've used an 8-bay GSPEED Shuttle, among the fastest Thunderbolt RAIDs around, and it isn't any faster).

Unlike a standard Thunderbolt enclosure, the QNAP system simultaneously functions as a NAS. It's designed to be hooked up over Thunderbolt and Ethernet at the same time, using the obscure networking function of Thunderbolt - it actually adds an extra, very fast network to your computer, running over Thunderbolt cabling (which means you can't daisy-chain anything else to the same Thunderbolt port - a problem if you only have one port).  . You can access the NAS from any other computer on the same network, and even use cloud-based functions in the same way as any other QNAP or Synology unit. Of course, those functions won't be blindingly fast - they'll depend on the network infrastructure like any other NAS - quite fast if it shares a gigabit network segment, very slow if you're running cloud-based functions over a cell connection. If you have a 10 gigabit network, it should run extremely fast, using its built-in 10 gigabit support (I haven't been able to test this, because I have no other device with 10 gigabit support).

Because it is a NAS that also happens to have a fast direct connection, it has most of the advantages and drawbacks of a NAS. It can perform a substantial number of functions on its own, including backing itself up to an external drive, offloading memory cards directly, and serving as a media server without tying up a computer. It can be set up as a Time Machine target, backing up all the Macs on the network (and making a multiple backup if it is further set up to back itself up to another drive). It will even share a printer over the network, although I wouldn't try this with a photo printer. On the other hand, it is comparable to any other NAS to set up, not to a Thunderbolt RAID (which is plug and go). The setup is wizard-based and takes less than half an hour, but it DOES require a modicum of knowledge and could intimidate less technical users.

Yes, it's an expensive unit, especially for smaller capacities where the very expensive enclosure will overwhelm the drive costs. The enclosure alone is $1900 for the 4-bay 682T, which is substantially more expensive than most NASs and many Thunderbolt RAIDs. Expensive Thunderbolt RAIDs come close to that price without drives, especially when you add features like sophisticated SSD caching, which the 682T and its big brothers have.

Once you add large drives (and there is no reason to use a machine like this with any but big, fast drives), it is comparable in price to higher-end Thunderbolt RAID units, which are its primary competitors (a 32 GB (raw capacity - 24 TB once RAID 5 steals a drive) unit with HGST enterprise drives is about $800 more than the OtherWorld 4-bay RAID, which uses your Mac to do the RAID calculations and uses cheaper drives , but is actually about $100 cheaper than the 4 bay, 32 TB GSPEED Studio, which does the work in the enclosure like the QNAP and uses the same drives). I wish QNAP made a version without the 10 gigabit Ethernet, a multi-hundred dollar feature that most photographers and videographers aren't set up to use, although perhaps a useful future-proofing.

As well as being a worthwhile consideration at the high end of the storage market, the QNAP Thunderbolt NAS series serve as a technology demonstration - QNAP or somebody else could put Thunderbolt networking in a less expensive NAS box that didn't feature 10 gigabit Ethernet, a very fast processor or SSD caching. A 4 bay unit in the $800 (empty) range should be possible by eliminating the most expensive non-Thunderbolt features from the present line? That would be entirely competitive with higher-end conventional NAS units and midrange Thunderbolt RAIDs, and the flexibility to be both at once would appeal to many users. A $1200 unit could keep the sophisticated SSD caching, but drop the 10 gigabit Ethernet and downgrade the processor somewhat. Something in that range would compete with the better Thunderbolt RAID units, and actually undercut many of them on price.
Title: Re: Storage systems and methods for photographers
Post by: Pictus on February 17, 2017, 09:50:01 am
Intel Atom C2000 chips are bricking products
http://www.guru3d.com/news-story/intel-atom-c2000-chips-are-bricking-products.html

"Folks with Synology DS1815+ NAS boxes have been reporting complete hardware failures; the DS1815+
is powered by an Intel Atom C2538. Other vendors using Atom C2000 chips include Asrock, Aaeon, HP,
Infortrend, Lanner, NEC, Newisys, Netgate, Netgear, Quanta, Supermicro, and ZNYX Networks.
The chipset is aimed at networking devices, storage systems, and microserver workloads."
Title: Re: Storage systems and methods for photographers
Post by: Dan Wells on February 17, 2017, 05:45:00 pm
This should get out broadly to the photographic community! Many low-end to midrange NAS boxes use the affected Atom chips (some similar unitsuse ARM chips, many midrange boxes use Pentium series chips one step above the Atoms, and faster NAS units often use a Core i3 or even higher). I was a bit surprised to find out that the popular Synology 1515+ and 1815+ units are Atom-based (I thought they were Pentiums). How many photographers have one of these (or any number of other NAS boxes) and don't even realize that their "disk drive" has a processor? They're easy enough to set up that many people don't think of them as servers (which is, of course, what they are).  Several of the WD My Cloud series use similar Atoms, and those are perhaps even more dangerous because they are always sold with disks in them, and they take even less user involvement to set up - my guess is that 80% of their owners have never paused to think about their processor...
Five years ago, servers were tricky enough to set up that nobody had a file server on their network without knowing what it was (unless an IT consultant had installed it without explanation)...  Now, you can buy a file server at Best Buy for under $200 for the least expensive WD and Seagate single-drive units, and just plug it into your router. They're a fantastic convenience, but something like this shows us that they're also complex devices that need to be understood.
Network hardware in general is getting cheap enough that misunderstood devices are cropping up everywhere. Routers plugged into routers (although, ironically, that configuration makes life difficult for hackers because of double address translation, so there's some reason to use two routers in succession - or the first router could be a cheapie provided by the cable company, while your whole network runs off the second), lots of "accidental servers" (not just file servers - the worst of it is all the TVs, thermostats and what have you that broadcast web interfaces - is the convenience of having your fridge online really worth the unsecured web server?), and even WiFi drives meant to provide extra storage to cell phones, but that connect to random networks.

Dan
Title: Re: Storage systems and methods for photographers
Post by: Joe Towner on February 22, 2017, 02:52:26 pm
I haven't seen the issues with the Atom processors, but it's a great reason to check your backups more frequently AND make sure you can work around an issue that crops up.

The QNAP boxes are interesting.  For starters, they aren't Thunderbolt enclosures, but 10Gig Ethernet NAS with a Thunderbolt port that'll let you do peer to peer networking at 10gbps.  Take a peak at these: https://www.asus.com/us/Motherboard-Accessory/ThunderboltEX_IIDUAL/   So basically they are removing the end user from setting up a very small 10gbps network, which is a good thing.  A true Thunderbolt enclosure takes the PCIe and works from there, doing a PCIe to PCIe bridge, then adapters & controller cards to devices like hard drives.  The easy test is if it works without a computer attached, it's got one inside.

10Gbps networking is expensive, with the switches costing ~$100 per port, and adapters ranging anywhere from $30 to $200 each.  I am actually looking forward to more of the 802.11bz aka 2.5gbps and 5gbps over cat5e & 6 respectively. 
Title: Re: Storage systems and methods for photographers
Post by: Dan Wells on February 27, 2017, 06:13:10 pm
      Yes, QNAP is using the Thunderbolt networking feature - I have mixed feelings about that (it has the unpleasant feature of grabbing control of the Thunderbolt bus, but pleasantly it is much less of a problem on sudden disconnect than an enclosure)... You care much more about it grabbing the bus if you have a Mac with only one Thunderbolt port, because you can't daisy-chain anything to the QNAP. With two or more ports, simply put displays and/or other drives on a different port (this works very well on a 2015 MBP). They say it's actually 20 gigabits per second, not 10 (WHAT kind of disks would it take to test that???).
       The fact that it has a computer inside enables a couple of really nice features - it can back itself up with no interference from an external computer (mine has its own WD MyBook attached via USB, and it keeps a copy of itself on the MyBook updated daily). It'll do this even if I'm out of town for a week with my laptop. It's also accessible over the Internet like any other NAS - not at the blazing speed of a Thunderbolt connection, but try that with a G-tech RAID. A secondary feature, but a nice one, is that it'll control a UPS and shut itself down in a controlled manner if the power fails, resuming on its own when the power comes back. You can sort of get an enclosure to do that if you have it set to power up and down with the computer , and the computer is controlling the UPS(if the computer's not a laptop - a laptop will hum merrily along on its internal battery while the enclosure drains the UPS).
       The Thunderbolt connection saves at least $400 over trying to get a Mac to talk 10 Gb Ethernet ($1000 or more if you want to include a switch in the network)... I agree with Joe that it's basically a 10 Gb NAS that can run its network over Thunderbolt - but that's a crucial advantage if you have Macs, since the only way to get a Mac on 10 Gb Ethernet (other than an ancient Mac Pro with slots) is with an expensive Thunderbolt to 10 Gb adapter...
     So far, I haven't seen any major disadvantage to the QNAP compared to a conventional Thunderbolt RAID on a similar scale, and it's not more expensive than a good one (it IS more expensive than the low-end soft RAIDs like the Thunderbay series, but it's very competitive with higher-end LaCies and G-Techs).
    One day, Apple will make an iMac with 10 Gb Ethernet (who knows, maybe they'll put out a reasonably priced Thunderbolt to 10 Gb dongle at the same time), and then a regular 10 Gb NAS will work for creative pros (at least, as they upgrade their computers) - until then, kudos to QNAP for this odd machine.
Title: Re: Storage systems and methods for photographers
Post by: pflower on April 20, 2017, 01:34:33 pm
Thanks for your article which was most helpful, although I fear I might be asking a question you counsel against.

My present set up has evolved over a number of years and is pretty haphazard but works.  I do not do video and currently use a 4TB G-Tech drive on a 5k 27inch iMac (FW800 via a thunderbolt adapter) as my main drive.  Backups, or archives, are via a series of 2TB WD Passport drives.  I am actually quite disciplined in making sure that I have copied everything I do to at least 2 drives at the end of each week and never format a card without having done so.  So the worst that can happen (barring total destruction by fire) is that I lose a week's worth of edits but would still have the original files on a card.

Although I work with quite large files (mostly Hasselblad Raws but some 256mb Tiffs) I can't see that I really need thunderbolt speed - I am happy in that respect with my current set up.  But the 4tb is soon to be full.  So I was thinking about a cheap USB3 dock with bare 6tb drives.  One as main storage and 1 or 2 as backup/archives.

Anyone got any thoughts as to whether or not this is stupid?  If so can anyone recommend a better solution and if not any thoughts on reliability of drives or docks and suggestions on brands?

Thanks



I've been hard at work typing this up, so read to get a bit of an idea how I believe photographers and storage line up.  It's about 70% but I keep seeing incorrect or baseless articles come out and I needed to speak out.

https://medium.com/@PNWMF/storage-systems-and-methods-for-photographers-86e04f940013#.uzejsolsi
Title: Re: Storage systems and methods for photographers
Post by: Joe Towner on April 20, 2017, 04:29:52 pm
Unless you start video editing, in reality, USB3 is perfectly fine when plugged directly into the Mac.  I wouldn't do storage off a USB3 hub unless required, but it wouldn't hurt too much.  Thunderbolt speeds really come into play when dealing with SSDs or multiple SSDs.

I wouldn't discount the enclosed 5/6tb drives, especially if you don't have a reason to swap 3.5" drives at this time.  The dock setup ones generally don't stack well  :D

The only bit I would ask is if you have a Flash or Fusion drive.  If you have a Fusion drive, I'd look at doing a 2 bay USB3 enclosure and put a SSD plus a 6tb drive in it.  Use the SSD as a 'working files' space, and leave the fusion drive for everything else.

Title: Re: Storage systems and methods for photographers
Post by: pflower on April 21, 2017, 02:36:02 pm
Thanks.  I do have a Fusion drive and also an external SSD directly connected to the iMac on which I have my Lightroom Catalog and the application.  All files however are on my 4TB drive.  I can see the sense in having "working files' on an SSD but frankly I do everything in Lightroom now and moving files around different drives within Lightroom drives me mad and I always end up making a mistake and have to root around in the Finder to find out where everything is.  But with the falling prices of SSDs I might have a look at this again and see if I can't work out a fool proof modus operandi.

I quite fancy the idea of a USB3 dock, and came across the idea from one of OWC's docks - the Drive Dock which sells for £250 or so.  Looking on Amazon there are a huge number of £30 docks.  I suppose for £30 or so it is worth taking a risk, but the price differential is a bit alarming and I am wondering what is the explanation.  There is no point buying something that really can't do the job.



Unless you start video editing, in reality, USB3 is perfectly fine when plugged directly into the Mac.  I wouldn't do storage off a USB3 hub unless required, but it wouldn't hurt too much.  Thunderbolt speeds really come into play when dealing with SSDs or multiple SSDs.

I wouldn't discount the enclosed 5/6tb drives, especially if you don't have a reason to swap 3.5" drives at this time.  The dock setup ones generally don't stack well  :D

The only bit I would ask is if you have a Flash or Fusion drive.  If you have a Fusion drive, I'd look at doing a 2 bay USB3 enclosure and put a SSD plus a 6tb drive in it.  Use the SSD as a 'working files' space, and leave the fusion drive for everything else.
Title: Re: Storage systems and methods for photographers
Post by: traderjay on July 31, 2017, 05:00:19 pm
One more thing to add - if you are going to use RAID with large capacity HDDs, RAID 6 is an ABSOLUTE must because you will run into URE (Unrecoverable Read Error) during the array rebuild process. If that happens you can kiss your entire array goodbye ;) 
Title: Re: Storage systems and methods for photographers
Post by: Farmer on July 31, 2017, 07:54:33 pm
One more thing to add - if you are going to use RAID with large capacity HDDs, RAID 6 is an ABSOLUTE must because you will run into URE (Unrecoverable Read Error) during the array rebuild process. If that happens you can kiss your entire array goodbye ;)

With consumer level HDDs, this is a genuine risk.  Any array built with drives over 2TB in capacity and the chance of a URE starts to build rapidly.  Enterprise drives will generally be OK to between 3TB and 4TB, but there's still a real risk.  SSD is one way of getting around this, but prohibitively expensive.  RAID 6 is a good choice, but note that even that starts to have questionable reliability with consumer level HDDs after 5TB.

As always, RAID is not a backup solution - it's a redundancy solution.  You need backups that don't rely on RAID somewhere in the chain.
Title: Re: Storage systems and methods for photographers
Post by: Joe Towner on August 02, 2017, 12:14:39 pm
The URE situation has been around since the 2tb drives and older / slower NAS systems.  The fact we're at 10tb and still growing gives longer rebuild times, and thus more chances for failures.

I've done some work and need to update & add to the article.  At the end of the day, you have to treat a NAS and a single external hard drive the same.  The only difference is capacity, performance, and portability.  For each drive added, you have one more failure point.  For each layer of redundancy (RAID5/6, ZFS1/2/3) there is an added point of failure.  I think I  put it as the two options folks should consider today is RAID1 and RAID6.

As to consumer v enterprise drives, I'll just reference the BackBlaze articles https://www.backblaze.com/blog/hard-drive-failure-rates-q1-2017/ They've just added a whole bunch of enterprise drives due to a Seagate sale, but we'll see how they compare.
Title: Re: Storage systems and methods for photographers
Post by: traderjay on August 03, 2017, 10:59:14 am
Yes that's why I stay away from consumer grade garbage NAS and stick with FreeNAS and its ZFS file system to minimize bit rot and parity errors. In addition, all my data are periodically backed up onto a separate HDD RAID 1 array in a different part of my house and then gets unplugged to act as cold storage. You can also buy cheap tape drive systems off ebay and do it that way to achieve true multiple redundancy :D
Title: Re: Storage systems and methods for photographers
Post by: Jonathan Cross on August 23, 2017, 08:17:03 am
Dear All in this thread.  This has all been very interesting, but I am a novice compared with most of you experts so get blinded by many of the acronyms etc.  I would be grateful for advice/experience for someone thinking whether he is doing the right thing and whether there is a better way.

Currently I backup to two external hard drives via USB3 from my Windows PC and from a Windows laptop.  I backup my Lightroom catalog most times I use Lightroom and backup my images fairly often, particularly if I am clearing a camera drive.  I use Cobian for all my data, not just images and Lightroom catalog, which does an incremental backup 4 times and a full backup every 5th time.  I have been put off using the Cloud as I have been told that if I loose or have a problem with my password I will never get the data back.

I am wondering if it worth investing in a NAS drive with 2 bays using RAID 1.  My knowledge of this system is very basic.  I understand that I can get drives that will do the backups and allow access via USB, wireless, or over the internet.  I may change to Mac, so would need a system that is compatible with both Windows PCs and Macs.  I am also aware that, in an ideal world, off-site, non-networked, storage is desirable. 

One last point, I prefer to do backups when I decide, as I find that having an external hard drive permanently attached slows my boot and normal running speed.

Should I stay as I am or go for NAS, e.g. Synology?  Advice/experience for a novice would be appreciated.

Jonathan


Title: Re: Storage systems and methods for photographers
Post by: David Eichler on August 28, 2017, 05:43:58 pm
Dear All in this thread.  This has all been very interesting, but I am a novice compared with most of you experts so get blinded by many of the acronyms etc.  I would be grateful for advice/experience for someone thinking whether he is doing the right thing and whether there is a better way.

Currently I backup to two external hard drives via USB3 from my Windows PC and from a Windows laptop.  I backup my Lightroom catalog most times I use Lightroom and backup my images fairly often, particularly if I am clearing a camera drive.  I use Cobian for all my data, not just images and Lightroom catalog, which does an incremental backup 4 times and a full backup every 5th time.  I have been put off using the Cloud as I have been told that if I loose or have a problem with my password I will never get the data back.

I am wondering if it worth investing in a NAS drive with 2 bays using RAID 1.  My knowledge of this system is very basic.  I understand that I can get drives that will do the backups and allow access via USB, wireless, or over the internet.  I may change to Mac, so would need a system that is compatible with both Windows PCs and Macs.  I am also aware that, in an ideal world, off-site, non-networked, storage is desirable. 

One last point, I prefer to do backups when I decide, as I find that having an external hard drive permanently attached slows my boot and normal running speed.

Should I stay as I am or go for NAS, e.g. Synology?  Advice/experience for a novice would be appreciated.

Jonathan

RAID is not for backup. It is for speed or minimizing downtime from drive failure, or both.
Title: Re: Storage systems and methods for photographers
Post by: Jim Kasson on August 29, 2017, 10:46:40 am
Dear All in this thread.  This has all been very interesting, but I am a novice compared with most of you experts so get blinded by many of the acronyms etc.  I would be grateful for advice/experience for someone thinking whether he is doing the right thing and whether there is a better way.

Currently I backup to two external hard drives via USB3 from my Windows PC and from a Windows laptop.  I backup my Lightroom catalog most times I use Lightroom and backup my images fairly often, particularly if I am clearing a camera drive.  I use Cobian for all my data, not just images and Lightroom catalog, which does an incremental backup 4 times and a full backup every 5th time.  I have been put off using the Cloud as I have been told that if I loose or have a problem with my password I will never get the data back.

I am wondering if it worth investing in a NAS drive with 2 bays using RAID 1.  My knowledge of this system is very basic.  I understand that I can get drives that will do the backups and allow access via USB, wireless, or over the internet.  I may change to Mac, so would need a system that is compatible with both Windows PCs and Macs.  I am also aware that, in an ideal world, off-site, non-networked, storage is desirable. 

One last point, I prefer to do backups when I decide, as I find that having an external hard drive permanently attached slows my boot and normal running speed.

Should I stay as I am or go for NAS, e.g. Synology?  Advice/experience for a novice would be appreciated.


I would not recommend a Synology NAS to a self-professed novice. Just yesterday, I had a Synology NAS drop out of a Windows network. I logged in to the NAS and checked the local time. It was wrong. I want to "Regional Options" and opened up the time control window, saw that it was synching with the right NTP server, which was a Windows DC, and clicked "Manual Sync". That fixed the problem, until the next time the Synonogy NAS decides to stop updating from that NTP server.

Does that sound like something you're ready for?

Jim
Title: Re: Storage systems and methods for photographers
Post by: FabienP on August 29, 2017, 05:06:42 pm

Should I stay as I am or go for NAS, e.g. Synology?  Advice/experience for a novice would be appreciated.

Jonathan

I would like to add that the performance of a NAS on a typical Gbit network is inferior to that of a directly attached disk over USB3. You could typically get 175 MB/s on an external disk and only get 90 MB/s from the same disk installed in a NAS due to the network acting as a bottleneck.

If you do not need to regularly access your data from multiple devices and do not need more storage than what can be copied on today's largest available disk, you probably don't want to bother with a NAS.

Cheers,

Fabien
Title: Re: Storage systems and methods for photographers
Post by: Joe Towner on August 29, 2017, 10:03:49 pm
Hey Jonathan,

Let's start with the easy questions - how much data are we talking about?  It's easier to add or upgrade to single disks (Costco has 6tb's for $130 each now).  Jim is right in that NAS systems aren't as easy to deal with external USB3 drives. 

Generally speaking, unless you have more than 6tb of data, a NAS isn't going to add much to a single pc setup.

-Joe
Title: Re: Storage systems and methods for photographers
Post by: BobShaw on August 30, 2017, 02:10:25 am
I would like to add that the performance of a NAS on a typical Gbit network is inferior to that of a directly attached disk over USB3. You could typically get 175 MB/s on an external disk and only get 90 MB/s from the same disk installed in a NAS due to the network acting as a bottleneck.
If you do not need to regularly access your data from multiple devices and do not need more storage than what can be copied on today's largest available disk, you probably don't want to bother with a NAS. Cheers,Fabien
Ditto. It is probably worse than that. I have my storage on a Mac Mini Server with a direct attached Firewire Drobo. That works OK. Backup is to a USB attached HDD. That also works well. Every second night it backs up to a Drobo NAS. Takes much much longer. Perhaps 5-10 times.
If you go to Mac and use the NAS as a backup then they should only be considered a backup of last resort. Time Machine sometimes does not recognise them properly and you can't just drag and drop to restore like you can with a direct attached. The only advantage is physical separation if you can get it far enough away to be worthwhile. Then you probably can't run Gigabit.
Title: Re: Storage systems and methods for photographers
Post by: Joe Towner on August 30, 2017, 02:15:58 pm
So back to the performance discussion:

Disk performance is the combination of 2 factors: the media type and the connection type.  For hard drives, you can get 60-100MB/s with 7200rpm drives out performing the slower/cheaper 5400/5900rpm models.  For SSD's you're in the 500MB/s range.  SATA is in 1.5/3/6gbps ratings.  USB2 is horrid in it's CPU use, but tops out at 480mbps, Firewire 800 is 800mbps.  USB3 took a lot of the processing and pushed it to the controller and has a 5gbps max.

Yes, networking can be limited to 1gbps, but you get trade offs like larger disks and volumes that have things like redundancy and caching involved.  You can also trunk ports on NAS's to get 4gbps, and there are models now available with 10gbps ports, and there are a number of 10gbps cards that make it viable.

Internal disks will be the fastest performing due to their direct, high speed connection to the CPU.  NVMe's and SSD's make a computer much faster.  If you can't do it internally, my preference of external devices is:

-USB3.1 (aka ThunderBolt3)
-ThunderBolt2 / ThunderBolt1
-USB3
-Firewire800
-USB2

Yes, I left out eSATA mostly because it's rare & limited (more so than FW800).  Keep in mind there are times when you need to keep the bus empty, like when shooting tethered. 

For a vast majority of users, doing a second internal drive is the best, but an external USB3 drive plugged into a USB3 port will do just fine.  If you're not at 6tb of data, there is no reason to look at a NAS.
Title: Re: Storage systems and methods for photographers
Post by: Jonathan Cross on September 01, 2017, 02:51:10 pm
Thanks to all who replied to my post; you have saved me money!  I have a total of about 2TB of data, and will stick with USB 3 external drives.  My Windows PC is some 5 years old and was getting slower, both to boot and to run.  My tech guru thought the HD was getting past it with accumulating bad sectors.  He changed it for a faster drive of twice the capacity (2TB from 1TB), and it now boots and runs much faster. For backup and archiving I will stay with external USB 3 drives and not pay for NAS.  I do not need remote access, but could, of course, take a small drive with me if needed.
Thanks again, Jonathan
Title: Re: Storage systems and methods for photographers
Post by: Joe Towner on September 02, 2017, 01:57:49 pm
Thanks to all who replied to my post; you have saved me money!  I have a total of about 2TB of data, and will stick with USB 3 external drives.  My Windows PC is some 5 years old and was getting slower, both to boot and to run.  My tech guru thought the HD was getting past it with accumulating bad sectors.  He changed it for a faster drive of twice the capacity (2TB from 1TB), and it now boots and runs much faster. For backup and archiving I will stay with external USB 3 drives and not pay for NAS.  I do not need remote access, but could, of course, take a small drive with me if needed.
Thanks again, Jonathan

Hey Jonathan,

Hate to be the guy giving you bad news, but your tech didn't make things much better.  The biggest speed boost you'll ever see is upgrading that boot drive to a SSD.  EVERYTHING runs faster, even if your photos are on those external USB3 drives.  I'd almost say you could do a 480/500/512/525gb drive, leaving your photos on the second internal drive (that new 2tb drive would be perfect).

-Joe
Title: Re: Storage systems and methods for photographers
Post by: Jonathan Cross on September 03, 2017, 07:52:34 am
Yes, Joe, you are right.  I was aware of this option as my tech guru did talk to me about it, but Scrooge beat the chance of speed; I decided I would not pay the cost of SSD.  I am happy with the new HD and external drives.  When I change the whole machine I will go SSD.

Thanks,  Jonathan

 
Title: Re: Storage systems and methods for photographers
Post by: David Eichler on September 03, 2017, 04:49:57 pm
Internal disks will be the fastest performing due to their direct, high speed connection to the CPU.

All other things being equal, I guess this is probably correct. However, an outboard SSD of recent manufacture, connected by Thunderbolt 1, is somewhat faster than the original internal SSD in my late 2012 Mac Mini. Which leads me to wonder, if I were to put that outboard SSD inside that computer, would it be significantly faster than it is as an outboard drive, or does the design of that internal bus in a 5-year-old computer design not allow that?
Title: Re: Storage systems and methods for photographers
Post by: Joe Towner on September 04, 2017, 08:56:03 pm
All other things being equal, I guess this is probably correct. However, an outboard SSD of recent manufacture, connected by Thunderbolt 1, is somewhat faster than the original internal SSD in my late 2012 Mac Mini. Which leads me to wonder, if I were to put that outboard SSD inside that computer, would it be significantly faster than it is as an outboard drive, or does the design of that internal bus in a 5-year-old computer design not allow that?

You could be seeing any number of things - wear on the internal drive, controller speed on the internal drive and chip technology on the internal drive.  The internal SATA port is 6Gb/s, so that isn't the limiter.  Newer SSD's have better controllers (remember the SandForce failures?) and faster internal pieces.  May I ask what drives are shown in MacOS?  Things like the Lacie Little Big Disk are cheating, where there are 2 drives working together.

You can also run the BlackMagic Disk Speed Test to compare their performance.
Title: Re: Storage systems and methods for photographers
Post by: David Eichler on September 04, 2017, 11:33:19 pm
You could be seeing any number of things - wear on the internal drive, controller speed on the internal drive and chip technology on the internal drive.  The internal SATA port is 6Gb/s, so that isn't the limiter.  Newer SSD's have better controllers (remember the SandForce failures?) and faster internal pieces.  May I ask what drives are shown in MacOS?  Things like the Lacie Little Big Disk are cheating, where there are 2 drives working together.

You can also run the BlackMagic Disk Speed Test to compare their performance.


The internal SSD is the original Apple one. The external SSD is a SanDisk purchased in the last year or so. The SanDisk is about 15% faster than the internal drive (per the Black Magic speed test), and that is with the internal drive less than half full.