To 10g or not to 10g ...

Started by mcbroomf, March 20, 2018, 12:34:19 pm

Joe Towner

The latest Synology 6.2 software does allow for multilink performace on the NFS shares, so I'll be testing that as well.

Did a quick one today, and just switching from SMB to AFP on the same share, same port doubled the speed as per the AJA speed test.

Joe Towner

I'm going to bump this thread, as there's good content and it's all related to setting up a 10gbps network.

Since we last chatted a bunch of new products have come to market that I want folks to be aware of.  Apple got into the 10gbps game with the latest 2018 MacMini - hopefully we'll see it in the next revisions of the iMac and MacPro's.  There are more TB3 adapters with both OWC and QNAP offering up adapters at $200 or less.  QNAP actually has 2 options, the standard RJ45 you've loved and the fancy SFP+ connection (more on this below).  For desktops, I'm still a huge fan of the eBay Mellanox cards with a SFP+ cable, but the Asus and new QNAP cards are both $99 or less.

On the NAS front, QNAP is doing a lot with SFP+ enabled units - starting for under $400! The TS-322x unit holds 3 drives, and 3 M.2 SATA drives and has a 10gb SFP+ port.  So you can do RAID5 over 3 disks, with a huge write cache in front of it with the SSD's - so copy your files on and start editing with the speed of the SSD, while it's getting written to the hard drives.  8 bay model with 2x SFP+ ports is under $800.

SFP+ adds a bit of consideration to this whole implementation, but it can make some things easier.  SFP+ lets you decide what type of connection you want, be it copper or fiber optics.  I know, you see the word fiber and you're thinking huge $$ but that's not the case anymore.  To connect from a SFP+ to a SFP+, there are SFP+ DAC cables - pre-made, fancy cables, at specific lengths that most likely cost about the same as a good network cable.  Here's a 20'/6m Generic SFP+ DAC cable for $25.  Plug it in and you're done.  If you wanted a longer connection, OM3 Multi-mode fiber is really cheap, even when you count in the SFP+ Fiber adapters.  To do a connection you'd need 2 of these Generic SFP+ 10gb adapters ($16 each) and a OM3 MM LC-LC fiber to connect them.  1m length ($2.80) all the way to 30m ($14) in length, it's an simple to see things at max speed.

Now that you see SFP+ as a viable, cost effective option, lets plug in a bunch of devices.  The QNAP switch has 12 ports, 4 SFP+ and 8 SFP+/RJ45's and is less than $500.  I've got one and love it, but there are times when you only have a few 10gb devices and a bunch of 1gb devies.  Here's the current steal - a 24port 1gb switch with 4 10gb SFP+ ports for $279!!  Yea, plug in a NAS and 3 workstations to 10gbps speed, plus everyone else can access the 10gbps network without the throttle of a single 1gbps port.  Even if you only have a NAS plugged in at 10gbps, and all other devices at 1gbps, you're able to have multiple machines connecting and not limited by the link from the switch to the NAS.

What else would you like to know or hear about?

Dan Wells

December 15, 2018, 10:13:30 am #22 Last Edit: December 15, 2018, 10:27:48 am by Dan Wells
10G has a very interesting feature - it's fast enough that there's no difference in speed between NAS and most local bulk storage (PCIe SSDs and SSD RAID can still outrun 10G and need to be connected by something faster - internal or Thunderbolt 3). Unfortunately, other than the dedicated link option, 10G is still very pricey because of the darned switches - there are still no switches under $500 unless you count refurbished bulky things with 24 standard gigabit ports and a couple of 10G uplink ports.

We're seeing more 10G ports built into higher-end computers (the iMac Pro is an example of interest to photographers, and there is even an option for $100 on the latest Mac Mini). Most Apple desktops should have 10G as either standard or optional from now on, as will  many creative-oriented PCs.  Higher-end NAS boxes have built-in 10G (I hadn't yet seen Joe's example of a $400 NAS, but I certainly believe it) , and it's not hugely expensive to add to a PC with a card, or even to a Mac as the Thunderbolt adapters have come down from $600 to $200 or less - but the switches are still pricey.

I think we'll see an explosion in interest in 10G among individual photographers and small studios when we see a $200ish switch with 3-5 10G ports and a few gigabit ports (or 8 10G ports with no gigabit). Most of us don't need 24 total ports, don't want to mess with SFP+, but want more than a couple of 10G ports. An average photographer wants 10G for a NAS and a couple of computers (desktop and a laptop with a TB adapter, for example), while a small studio may have 3-5 computers and maybe a second NAS on 10G. Gigabit ports are useful for additional computers that don't hit the NAS often, printers and routers...

If there was an affordable 8-port 10G switch (or even a 5-port), it would be easy enough to put the critical devices on 10G, linking it to the rest of the network with a gigabit uplink or two... Oddly, a switch that is mainly 10G with a couple of gigabit ports  is more useful in a small network than the opposite. Often, all the 10G devices will be in one room (or one room with a NAS in a closet nearby), with gigabit devices spread out - if the photography gear is in a home, there are likely to be gigabit devices all over the house (although many of them may be WiFi or powerline).

Depending on how you're set up, the 10G segment may be all patch cables straight to the switch (even good patch cables are relatively cheap), with any in-wall cabling being gigabit. Even gigabit internet is really rare - since the internet link is likely to be 100 megabits or so, gigabit connections work fine for anything that is mainly accessing the internet. Even big photo printers and high-speed laser printers (within reason - I don't know the data rates on 100 page per minute lasers or the like, but how many photographers worry about those?) are well under a gigabit, and won't benefit from a faster connection - a surprising number of printers had 100 megabit Ethernet until gigabit chips became so cheap that there's no reason to use anything else. Streaming boxes, game consoles and the like can use a lot of bandwidth, but are limited by the speed of the internet connection - they don't do much on the LAN in most cases.  The only real use for 10G is hitting local NAS boxes at high speed or transferring data between local computers - and those are probably all close together...


Quote from: Dan Wells on December 15, 2018, 10:13:30 am
An average photographer wants 10G for a NAS and a couple of computers (desktop and a laptop with a TB adapter, for example),

then that average photographer can simply build DIY NAS using generic (from Ebay) NAS case and using motherboard with 2 x 10g ports - hence no switch needed at all ...

Joe Towner

While things like XigmaNAS (NAS4Free renamed since they couldn't trademark), and FreeNAS (iXsystems) make the build your own NAS fairly easy, the savings compared to purchasing a diskless NAS are quickly lost if you ever have an issue.  It's just a matter of your individual tech abilities, and how you value your time.  If you have to go find someone who can help rescue your ZFS Pool, it's going to be costly.  Plus the cost savings just isn't there compared to the ease of doing a Synology (lots of personal experience) or QNAP (will be playing with soon).

As for the switch, we're still only seeing the beginning of 10gbps or 2.5/5/10gbps (NBaseT) models.  The model for $279 with dual power supplies is a bit more than most home users, but the price makes up for dealing with SFP+.  Netgear has the switch I think that matches what Dan wants    
MS510TX for $270
10 ports broken down a bit:  4x1gbps, 2x2.5gbps, 2x5gbps, 1x10gbps RJ45, 1x10gbps SFP+  Only catch is that you need the newest 10gb cards that speak 2.5/5gbps as well.

Yes, the direct connect works fine, just remember who you're walking thru setting static IP's and making sure any mounts or traffic goes over the faster link.  I'll take 2.5gbps or 5gpbs any day over 1gbps.


Quote from: Joe Towner on December 16, 2018, 02:28:25 pm
Plus the cost savings just isn't there compared to the ease of doing a Synology (lots of personal experience) or QNAP (will be playing with soon).

you always pay more for the ease if you can't DIY, that's true...


Well what you also get is a lot more power. Especially the Synology are quite weak on CPU power. While it works for most cases you can built a mich more powerfull System if you know what your doing.

I agree that freeNAS and co is nothing for people who haven't built and configured their own computers. For these applications a Synology or Qnap is the way to go.

Sent from my iPhone using Tapatalk
Christopher Hauser

Dan Wells

I might or might not ever tackle a NAS build - and I've built many computers over the years. The hardware doesn't scare me at all, but dealing with a NAS OS without "walk me through" configuration is at least mildly scary (and I've configured all manner of RAID, including Fibre Channel for a friend - back in the days when you couldn't easily get anything else over 10 TB with decent speed). I'm also an ex-IT professional (very ex - it was 15+ years ago, when I was right out of college).

I'd guess that most photographers probably don't have that level of computer skill - some certainly have much more than I do (isn't Bill Atkinson, who wrote much of the original Mac software, a photographer)? Maybe I'm overestimating how hard it is to build a NAS, and I'd be fine...

When I think about the kind of NAS I'd want, were I to bother to build one, it would be an 8-bay or 12-bay with 10 GB Ethernet. Ideally, it would have a second Ethernet port (doesn't have to be 10GB), so it could be on two network segments - one 10Gb for the big Mac, and one right into the router so that other things can hit it over WiFi or gigabit.

It would be a straight-out storage server, so it doesn't need a ton of computing power or RAM - I don't need to transcode on the NAS or anything like that, and I'm not running my own email or web servers... Right now, I have a 6-bay QNAP Thunderbolt hybrid NAS that serves that function, and the only reason I would replace it is that I have a remarkably storage-hungry Nikon Z7 that chews through 32 GB or more (I've come close to filling a 64 GB card in a day, and it's not even my high-shooting season) on an average day of shooting! I have about 8 TB free on the QNAP, so I probably have about a year or two before the Z7 eats it up to the point where I need something bigger.

There are some NAS case options out there - the 8 bay options I've found (I looked just to see how hard it would be) are either Mini ITX, which is a challenge because there doesn't seem to be a Mini ITX motherboard with 10 GB networking, either Intel or AMD-based, or they're conventional big server cases, which are a lot bigger than Synology and QNAP cases. Of course, I could use a Mini-ITX case and a PCIe LAN card... Once you're looking for 12 hot-swap bays, I haven't come across anything that isn't a big ol' server case (or rackmount, which will scream like a banshee in anything except a server closet).



December 20, 2018, 05:13:10 am #28 Last Edit: December 20, 2018, 05:16:40 am by DP
Quote from: Dan Wells on December 19, 2018, 07:16:38 pm
Once you're looking for 12 hot-swap bays, I haven't come across anything that isn't a big ol' server case (or rackmount, which will scream like a banshee in anything except a server closet).

FRV developer just built himself a DIY NAS (plain FreeBSD OS) with 2x10g supermicro mb


and ultimately you can just put Windows Workstation (Enterprise and Pro) with Resilient File System (ReFS) @ storage spaces... certainly it is not FreeBSD & ZFS - but Windows... so less hassle if you don't want to touch plain FreeBSD or it's FreeNAS-like incarnations

Dan Wells

Has anyone seen a "name" version of a case like that? Something like a Fractal Design, Silverstone, Corsair, SuperMicro or whatever sold by Newegg or Amazon, shipped from the US? Or will this always be a game of dealing with overseas suppliers with little guarantee and shipping costs more than half the cost of the product? I suppose it would be possible to take out the moving parts (primarily the fans and any included power supplies) and replace them with trusted models - but there's a backplane in there that's probably a custom PCB. I wouldn't trust my photos to anything that didn't come from a company I'd heard of with a reputation for quality (if it were a media server where everything was also in the cloud, I might feel differently). Of course I back everything up, but I also want known high-quality systems...


Quote from: Dan Wells on December 20, 2018, 03:08:55 pm
Has anyone seen a "name" version of a case like that?

well, he did a research... his comments were essentially why in the world chineese workers can't do some 3rd shift pumping brand-name looking cases over the fence for cash  ;D ... but still agree - it is a decent looking case and small...

you can try to wrestle (some idioms and there are many, can't be translated still no matter how big Google's data grown) with google translate @ ...

Joe Towner

A motherboard that has dual 10gbps NIC's on it is going to cost a couple hundred dollars - cheapest is $275 on Newegg, though ASRock has a few with a single 2.6gbps NIC.  So you're much better purchasing a rock solid board and using a PCIe NIC.

Back on topic - Ubiquiti has a 10gbps SFP+ to RJ45 adapter I want to try -  $65 to convert one of the 10gbps QNAP's from SFP+ to RJ45 would be huge to a lot of people.