Pages: [1] 2   Go Down

Author Topic: To 10g or not to 10g ...  (Read 3211 times)

mcbroomf

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 708
    • Mike Broomfield
To 10g or not to 10g ...
« on: March 20, 2018, 12:34:19 PM »

In the discussion about TB3 disk arrays Joe mentioned a 10g network setup he is eyeing
http://forum.luminous-landscape.com/index.php?topic=123716.msg1033107#msg1033107
As I have just received the Akitio TB3 enclosure (the Quad X) coincident to a new PC build arriving tomorrow I poked around a little bit and found an Akitio TB3 -> 10g adapter and started wondering about moving my network to 10g (the PC does not have a 10g card but neither is my router).  I exchanged a PM with Joe and though it might be interesting see if others run 10g, or are thinking about it.

I think for the moment I am NOT planning to do this (although  I can always be persuaded to spend money on tech goodies).  The new PC is a desktop with a case and MB that will allow me slip in SATA 6 hard drives.  The OS and LR/PS cache are handled by 2 separate M.2 cards on the MB and my primary/current RAWS will be edited on 2x2TB raid 0 SATA SSDs.  All the RAWS will be duplicated to an on-board spinning HD together with all my archived RAWs and final print files and large .psb files on a separate drive with non-photo files on a 3rd.  The Akitio TB3 will be my primary backup for all 3 disks (migrating from a slow USB enclosure) and my Synology NAS will be my 2nd backup.  So as you can see my NAS is at the end of the; fastest -> fast -> slower -> slowest chain.  while it might be nice to bump it to 10g with either the TB3-10g adapter, or a new 10g PCI card + a 10g router I'm not sure I see a reason.  My desktop and all peripherals sit at my feet.  I do use a laptop but it's specifically for use in the kitchen/dining/living rooms so I run it purely through WiFi and have never hard wired it to my router.  I use it for PostP when on a trip but never at home.

So, do you run 10g?  If so how is it setup and why do you have 10g?  Do you want something faster  ;D  I imagine if NAS is your primary storage there's a good reason but I'd be interested in hearing how other folk have their networks setup (for photo or non-photo application).

PS.  I don't really know the 1st thing about networks, eg I don't know what NBase-T is ...

Cheers,
Logged

Joe Towner

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 951
Re: To 10g or not to 10g ...
« Reply #1 on: March 21, 2018, 10:52:22 AM »

Thanks for moving the thread over - I think others will get a kick out of it.

NBase-T otherwise known as 802.3bz, is the brand name way to talk about the new wired network speeds of 2.5gbps, 5gbps over existing cabling.  There are network switches and adapters that will work at this faster than 1gbps speed, without the costly re-wiring of a building.  Originally targeted at wifi access points for high density locations, it's a great tool for folks who move lots of data, like us.  We can fully saturate (use) a 1gbps network connection, especially if working off a SSD or other fast storage system.  The cost and wiring requirements for 10gbps networking is high - CAT7 cabling, fully shielded, each port on a network switch.  So they've developed a great middle ground.  The gear is still being slowly released, but Netgear seems to have jumped in with both feet with a full line up options based on need.   https://www.netgear.com/landings/multi-gig/

In IT, we compare network switches on a Price per Port, as in how much each port of a certain speed switch will be.  A 10gbps port is going to cost most than a 5gbps port, which costs more than the 2.5gbps port and 1gbps ports are just thrown around willy-nilly.  But by matching the actual need per port, you can get away with fewer of the expensive 10gbps ports, and get more of the 2.5 & 5gbps ports.

Once you start to copy around 100's of gigabytes of data, gaining a 2x or 5x speed advantage becomes huge.  For me, the advantage is to gain access to lots of storage (on my NAS) at higher speeds than I can otherwise on my Mac laptop.  By using a 25' CAT7 network cable and a tb3-10gbps adapter, I can sit on my couch and access my NAS at the fastest speed.

-Joe
Logged
t: @PNWMF

Joe Towner

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 951
Re: To 10g or not to 10g ...
« Reply #2 on: March 21, 2018, 12:32:21 PM »

To further discuss speeds, I want to lay out some of the limitations we're currently working within.  There are 2 types of storage mediums out there, flash or solid state and spinning hard drives.  The spinning hard drives are by far the slowest, but they offer up high capacity at a low price.  The other strong points of hard drives is that you don't have a write limitation, and you can do data recovery if needed.

As to solid state drives, the speed is huge, but there are different connectors that can dramatically slow things down.  Right now, the fastest setup is NVMe drives, which are solid state drives connected directly to PCIe lanes.  These can run over 2 GB/s, compared to SSD's connected to a SATA port top out around 500 MB/s.  The NVMe drives are either directly on a motherboard or via PCIe slot adapter.  The third way to connect a NVMe drive is via TB3, which is what the OWC ThunderBlade V4 is doing - https://eshop.macsales.com/shop/owc-thunderblade-v4

So, while a SSD is fast, a SSD connected via PCIe is even faster.
Logged
t: @PNWMF

mcbroomf

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 708
    • Mike Broomfield
Re: To 10g or not to 10g ...
« Reply #3 on: March 21, 2018, 12:54:42 PM »

Thanks for that info Joe.  How will you be wired (10g), ie is that 25' Cat7 running across the floor from one room to another or do/will you have room to room outlets running through the basement so that you can keep it neat?

Somewhat off topic ... while connecting a 3rd Cat6 cable to my router in prep for my new PC I depressed the little power button on the back (ASUS RT-AC66).  Now when I try to turn it back on the button won't stay depressed and as soon as I release it the power dies.  Grrr.... talk about timing!  To get round it I jammed a toothpick alongside the button while depressed to keep it in the On position, but I'm shopping for a new router.  Looks like I will upgrade to the ASUS RT-AC86 which is a bit faster and has a USB 3.1 port which might be handy.
Logged

Joe Towner

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 951
Re: To 10g or not to 10g ...
« Reply #4 on: March 21, 2018, 02:55:03 PM »

The cat7 will run across the floor - my NAS sits behind the tv in the living room.  I've done new construction with cat5e as recently as 2012, but I see cat6 is most new residential locations.  With the cable length of 25', I can get away with standard Cat6, but the price jump to Cat7 is only a few extra dollars.

One thing I didn't mention is the fun in between CAT6, CAT6a and CAT7.  In practical application, CAT6 can go 10gbps for 55 meters, CAT6a can do 10gbps for 100 meters.  CAT7 can do 10gbps and higher at 100 meters, it's more a limit of the devices plugged in at both ends.  Here's a bit more of a post - https://community.fs.com/blog/should-we-use-cat6-or-cat6a-for-10gbase-t.html

Sorry to hear about the power switch - it's very odd that it stopped working correctly.  I don't recommend sharing a drive off your home router USB3 port, there are much better solutions that do the job better.

-Joe
Logged
t: @PNWMF

mrenters

  • Jr. Member
  • **
  • Offline Offline
  • Posts: 79
    • http://www.teckelworks.com/
Re: To 10g or not to 10g ...
« Reply #5 on: March 24, 2018, 09:28:56 PM »

You might be able to get by using Cat5e for shorter distances. I've got 10g running on Cat5e over a distance of 10m or so at home and I've seen reports of it being good for up to 45m.

I'm using an Intel X550-T2 card in my FreeNAS 11.1 server and Asus XG-C100C cards in my Windows boxes. Both of these cards will do multi-gigabit (1g, 2.5g, 5g) but negotiate and maintain 10g just fine.

My bottleneck is now the CPU (Atom C2758) and number of hard drive spindles, currently 2x10GB WD Golds in mirrored configuration. For data reads that comes out of RAM cache I've seen speeds of 550MB/sec over SMB.

Martin
Logged

Joe Towner

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 951
Re: To 10g or not to 10g ...
« Reply #6 on: March 25, 2018, 10:00:43 PM »

Great to hear it's nice and fast, until you have to go to disk of course :)  I've got a 16 bay SAS exclosure that I think about using, doing a Z3 setup across 8 or 10 tb disks.  Then I remember it's $4,800 for 18 of them...
Logged
t: @PNWMF

Endeavour

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 291
Re: To 10g or not to 10g ...
« Reply #7 on: April 05, 2018, 12:09:55 PM »

cat5e is far more forgiving for home network installs.
Cat6 requires a LOT more care and attention, failure to adhere to the best practices and you've paid cat6 prices for cat5 performance

for example, one sharp bend in your cabling, can reduce the throughput by 50% (a bend with a radius less than one inch is going to cause you issues)

Cat6 cabling is delicate, and terminating correctly is a lot more difficult than cat5e


Logged

FabienP

  • Full Member
  • ***
  • Offline Offline
  • Posts: 111
Re: To 10g or not to 10g ...
« Reply #8 on: April 06, 2018, 05:06:46 PM »

(...)
The gear is still being slowly released, but Netgear seems to have jumped in with both feet with a full line up options based on need.   https://www.netgear.com/landings/multi-gig/
(...)

-Joe

Thanks for the link to those Netgear switches, Joe. I didn't realise that affordable multi-gigabit switches were now widely available. Too bad that these still require active cooling, but the smaller model should be barely audible in a home / SOHO environment.

I am starting to run out of excuses for not migrating to 10GBASE-T or at least 5GBASE-T. However I am still using a six years old Ivy Bridge Xeon E3 workstation and am missing PCI Express lanes to add a faster network card. I would at least need a 4x port to get enough bandwidth for the NIC.

A new workstation is planned for next year, when Spectre (and Meltdown for Intel) will hopefully be corrected in silicon!

Cheers,

Fabien
Logged

Joe Towner

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 951
Re: To 10g or not to 10g ...
« Reply #9 on: April 09, 2018, 11:43:46 AM »

Cat6 cabling is delicate, and terminating correctly is a lot more difficult than cat5e

I wouldn't call it delicate, I'd call it a pain in the ***.  The inner shield really screws with your fingertips, but it terminates into patch panels and jacks the same way that cat5e does.  Making Cat6 patch cables is a PITA, purchase pre-made ones and save your sanity.
Logged
t: @PNWMF

Joe Towner

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 951
Re: To 10g or not to 10g ...
« Reply #10 on: April 09, 2018, 05:03:40 PM »

I hate to have left my favorite brand out of this discussion, so I'll loop back in the Ubiquiti products, and the Unifi 16XG - https://www.ubnt.com/unifi-switching/unifi-switch-16-xg/

It's 16 10gbps ports, 12 are SFP+, while 4 are RJ45, and $600.  The SFP+ ports don't help you much, until you see what used/system pull Mellanox 10gbps SFP+ cards cost on eBay - $24! Part number is MNPA19-XTR and you can get them with the SFP+ cable for that $24 price.  Need a longer cable, 7m SFP+ cables from FS.com are $42.  Works in Windows 7 and 10, plus NAS4Free, so if you're a home-built NAS kind of person, you have lots of options.
Logged
t: @PNWMF

paulster

  • Newbie
  • *
  • Offline Offline
  • Posts: 25
Re: To 10g or not to 10g ...
« Reply #11 on: April 09, 2018, 05:55:32 PM »

I hate to have left my favorite brand out of this discussion, so I'll loop back in the Ubiquiti products, and the Unifi 16XG - https://www.ubnt.com/unifi-switching/unifi-switch-16-xg/

It's 16 10gbps ports, 12 are SFP+, while 4 are RJ45, and $600.  The SFP+ ports don't help you much, until you see what used/system pull Mellanox 10gbps SFP+ cards cost on eBay - $24! Part number is MNPA19-XTR and you can get them with the SFP+ cable for that $24 price.  Need a longer cable, 7m SFP+ cables from FS.com are $42.  Works in Windows 7 and 10, plus NAS4Free, so if you're a home-built NAS kind of person, you have lots of options.
Are those coming with two SFP+ modules at those prices?  The pictures suggest they do.  And do you know if those SFP+ modules are compatible with the 16XG?  That would make for a steal of a solution.
Logged

Joe Towner

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 951
Re: To 10g or not to 10g ...
« Reply #12 on: April 09, 2018, 06:52:53 PM »

It's an SFP+ to SFP+ cable, in the generic sense.  The auctions don't list a cable length, but even at 2m, it's quite a steal.  You can get SFP+ DAC cables, or you can do fiber extremely cheap.

For a fiber setup, pair these Ubiquiti MM 10gb SFP+ at $16, https://www.fs.com/products/65336.html
with these Mellanox ones https://www.fs.com/products/65334.html for the same $16,
and a 10GBase-SR fiber for $12 (15m) or less! https://www.fs.com/products/44158.html

So card ($24) plus SFP+ adapters (2 x $16) plus fiber ($12) puts it at $78 per machine.  The Ubiquiti actually comes in 2 flavors, the Unifi one which I love because it plays into all the cool network stuff I already do, or their EdgeSwitch version which is more stand-alone.

-Joe
Logged
t: @PNWMF

Joe Towner

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 951
Re: To 10g or not to 10g ...
« Reply #13 on: May 09, 2018, 12:15:53 AM »

Once I can find one to purchase, I'll have some testing to share.  TB3 10gbps Ethernet - $199 msrp

http://www.sonnettech.com/product/solo-10g-tb3.html
Logged
t: @PNWMF

paulster

  • Newbie
  • *
  • Offline Offline
  • Posts: 25
Re: To 10g or not to 10g ...
« Reply #14 on: May 09, 2018, 05:35:17 PM »

That's a great deal!

I've been looking at the Promise SanLink3 SL3T1GL as a potential option, which is a bit smaller, but $100 more.
Logged

Joe Towner

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 951
Re: To 10g or not to 10g ...
« Reply #15 on: May 25, 2018, 03:15:01 PM »

I've got my Solo 10g and a few of the Mellanox cards, with the Asus and some switches on their way.  Will setup trunking on my Synology and test (DS1817+ w/ SSD caching). 

One addition is the unmanaged QNap 10gb switch, https://www.qnap.com/en-us/product/qsw-1208-8c - it's 12 ports, with 8 being either SFP+ or RJ45.  So while port trunking isn't it's thing it can do, it can aggregate other 10gbps connections.
Logged
t: @PNWMF

paulster

  • Newbie
  • *
  • Offline Offline
  • Posts: 25
Re: To 10g or not to 10g ...
« Reply #16 on: May 29, 2018, 02:21:27 PM »

I'm interested to hear how the Solo works out, as I'm seriously thinking about pulling the trigger on one for my laptop.

I also use a DS1817+ and I've found that without SSD caching I can still get WD Blue 3D SSDs on my client workstations to thermally throttle (the Samsung Pros don't) when using the 10Gbe link, so I'm not sure that the SSD cache would really add a whole lot, at least for the large files that I'm moving around.  Did you ever test with and without it?

From my experience link aggregation on the Synology is only useful when you have multiple users hitting it at the same time.  SMB file transfers occur over a single session using one interface only.  Even backups from one Synology to another don't capitalize on the LAGs, so the benefit may not be as great as you'd expect.

Although if you have multiple users hitting it and the SSD cache then it should fly!
Logged

FabienP

  • Full Member
  • ***
  • Offline Offline
  • Posts: 111
Re: To 10g or not to 10g ...
« Reply #17 on: May 30, 2018, 06:20:33 PM »

(...)

From my experience link aggregation on the Synology is only useful when you have multiple users hitting it at the same time.  SMB file transfers occur over a single session using one interface only.  Even backups from one Synology to another don't capitalize on the LAGs, so the benefit may not be as great as you'd expect.

Although if you have multiple users hitting it and the SSD cache then it should fly!

This is to be expected with plain SMB connections. One would have to use SMB multichannel connections to get more bandwidth for a single client. The Synology appliances probably use Samba to serve SMB connections. The last time I checked, Samba only supports SMB multichannel as an experimental feature and will not be activated in stable releases used in NAS appliances.

LAGG interfaces remain useful as a failover.

Cheers,

Fabien
Logged

armand

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2818
    • Photos
Re: To 10g or not to 10g ...
« Reply #18 on: May 30, 2018, 07:24:20 PM »

I don't want to hijack the thread but this might be helpful.
I was reading this review: https://photographylife.com/reviews/synology-ds1618 where they mentioned that you could just get a crossover cable between that particular NAS and the desktop so you can access it it from the desktop at 10G and at 1G from the rest of the computers or at AC wireless speeds.
Sounds like a lower/ less complicated investment while getting most of the benefits, what do you think?
I'm quite tempted, for me it will solve the workable storage issue (I use a volume comprised of several SSDs on SATA, so a similar speed) while having easier access from everywhere else on the net.

PS. I currently use a DS412+ as a backup only

Joe Towner

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 951
Re: To 10g or not to 10g ...
« Reply #19 on: May 30, 2018, 11:40:47 PM »

I don't want to hijack the thread but this might be helpful.
I was reading this review: https://photographylife.com/reviews/synology-ds1618 where they mentioned that you could just get a crossover cable between that particular NAS and the desktop so you can access it it from the desktop at 10G and at 1G from the rest of the computers or at AC wireless speeds.
Sounds like a lower/ less complicated investment while getting most of the benefits, what do you think?
I'm quite tempted, for me it will solve the workable storage issue (I use a volume comprised of several SSDs on SATA, so a similar speed) while having easier access from everywhere else on the net.

PS. I currently use a DS412+ as a backup only

Yes, for a single connection doing a normal cable to between 2 10gbps port is much simpler, though you'd want to statically assign the IP's in a different network and do the mount via IP rather than hostname.  One thing most folks don't remember is that gigabit and faster are auto-crossover (MDI/MDI-X), so don't waste money on a CAT6a crossover cable  ;D

I wish the DS1618+ was more in line with the DS918+ in the dedicated, NVMe slots, but it lacks 10gbps networking.  If anything both need to be relaunched with NBASE-T ports that do 2.5/5gbps ports.
Logged
t: @PNWMF
Pages: [1] 2   Go Up