Notes about 100G Ethernet switches for a home lab, looking for either:
- Cheap low port count new switches.
- Available second hand switches with possibly more ports than required.
- Options for mixing some low speed ports, as have some other 10G devices.
Found on Reddit "Cheap" 100G switches, pros and cons of each option? for initial suggestions.
Manufacturers product page says:
There are four 100 Gigabit QSFP28 ports. Use them to connect other 100 Gigabit devices or grab a breakout cable to connect multiple 25 Gigabit devices: each port is connected to four 25 Gigabit lines, allowing you to connect 16 x 25 Gigabit devices. In fact, you can mix and match all kinds of connections this way: 100 Gigabits on one port, 40 Gigabits on the other, 4x25 Gigabits on the third, and maybe even some 10G or 1G cables on the fourth. The mighty Marvell DX4310 switch-chip will handle everything without breaking a sweat!
New price on Amazon - Mikrotik CRS504-4XQ-IN QSFP28 M RM £627.35
MikroTik CRS504-4XQ-IN Review Momentus 4x 100GbE and 25GbE Desktop Switch contains:
The other big one is a huge missing feature. With 100GbE, 200GbE, 400GbE, and so forth networks, RDMA has become a mainstream feature and RoCE (V1/V2) have become extremely popular. This is largely due to the amount of data that has to move through the network stack. Bypassing that network stack via direct memory to NIC transactions saves an enormous amount of CPU cycles. We could not figure out how to get features like PFC working on this switch. We went to the MikroTik help docs and searched for RDMA, and we got absolutely nothing.
Second hand on ebay “Very Clean, Fully Tested Arista DCS-7160-48YC6-R Switch Rear-to-Front Airflow. Arista 7160, High Capacity 48 x 25GbE SFP and 6 x 100GbE QSFP switch, rear to front air, 2 x AC” £849.00
Manufacturers 7160 Series Data Center Switch datasheet has the overview:
7160-48YC6: 48 port 25G SFP and 6 port 100G QSFP
- Offer 48 wirespeed 10/25G ports with 6 100G ports for up to 72 total 25G or 10G ports
- Easy migration from 1/10G to 1/10/25G using a familiar SFP connection
- Six 100G QSFP ports allow choice of 6x 100GbE, 24x 25GbE, 6x 40GbE, or 24x 10GbE
- 5 speeds for flexible 10GbE, 25GbE, 40GbE, 50GbE and 100GbE with optics or cables
- 3.6Tbps of wire speed performance with 24MB of buffer
Second hand on ebay Dell Z9100-ON , 32 Port 100Gb QSFP28 2x 10Gb SFP+ Switch, Tested, Warranty £495.00
Output shows running Dell EMC Application Software Version: 9.14(1.14). Possible license issues with trying to update to OS10:
- Looking for a cheap 100G network switch
- Dell Z9100-ON
To use OS10 you will need a license, and in many cases the license will be lost when buying a second-hand device, so you can only get an "indefinite" trial system (cannot be used after a certain period of time, needs to be reinstalled to "trial" again)
Dell datasheet says:
The compact Z9100-ON design provides industry-leading density of either 32 ports of 100GbE, 64 ports of 50GbE, 32 ports of 40GbE, 128 ports of 25GbE or 128 ports 10GbE and two SFP+ ports of 10GbE/1GbE/100MbE to conserve rack space while enabling denser footprints and simplifying migration to 100Gbps in the data center core. Priority-based flow control (PFC), data center bridge exchange (DCBX) and enhanced transmission selection (ETS) make the Z9100-ON ideally suited for DCB environments.
1RU high-density 10/25/40/50/100GbE fixed switch with choice of up to 32 ports of 100GbE (QSFP28), 64 ports of 50GbE (QSFP+), 32 ports of 40GbE (QSFP+), 128 ports of 25GbE (SFP28) or 128+2 ports of 10GbE (using breakout cable)
Some configurartion documentation - Dell EMC Networking OS Configuration Guide for the Z9100–ON System 9.14.2.4:
Splitting 100G Ports The platform supports splitting a single 100G QSFP 28 port into any of the following ports:
- Two 50G ports
- Four 25G ports
- One 40G port
- Four 10G ports
FEC Configuration FEC configurations are available on 100–Gigbit, 50–Gigabit and 25–Gigabit Ethernet interfaces. To configure FEC, use the following commands.
Enable FEC.
INTERFACE mode fec enableThe CL91 keyword enables RS-FEC on 100G interfaces as per IEEE. The system displays an error message when this is applied on 25G or 50G interfaces.
The CL74 keyword enables BaseR-FEC on 25G or 50G interfaces as per 25G/50G Ethernet Consortium. The system displays an error message when this is applied on 100G interfaces.
To disable FEC, use the
no fec enablecommand.
From Need advice on complete homelab refresh:
- I run a Celestica Seastone DX010 since 2021 and I've made a few posts and Github issues about it. It is fiddly and doesn't exactly "just work", especially if you're making frequent changes. I used to own a Dell Z9100 (sold it here) which "just worked" but it was picky about transceivers and wasn't nearly as flexible. If you do pick up a DX010 I will help you. Using RoCE/SMB Direct over 100G has been a game changer for the way I access storage.
Product page Half-width Rackmount 100GbE L3 Lite Managed Switch, cost-optimized for enterprise IT:
The QSW-M7308R-4X is a 100GbE L3 Lite Managed Switch with four 100GbE QSFP28 and eight 25GbE SFP28 ports.
The 100GbE ports support breakout into 4x 25GbE links.
The datasheet says the supported standards includes IEEE 802.3ae 10G Fiber, but isn't clear if the supports running at 10 GbE.
QNAP QSW-M7308R-4X 8x 25GbE and 4x 100GbE Managed Switch Review says uses a Marvell Prestera 98DX7324 switch chip. The 98DX73xx product brief contains:
Multi-Rate 1/2.5/5/10/25/40/50/100/200/400G Ethernet ports
Amazon QSW-M7308R-4X Ultra-high-speed managed switch with four 100GbE QSFP28 and eight 25GbE SFP28 ports £1,341.47
Second hand on Ebay Dell PowerSwitch S4112F-ON 10GbE SFP+ OS10 Perpetual | 3 Year Warranty £600.00
From the listing:
| Feature | Description |
|---|---|
| Series | S4100 |
| Ports | 12 x 10GbE SFP+ |
| Accessory Ports | 3 x 100GbE QSFP28 |
| Power Supplies Installed | 2 x Integrated AC Power Supply Units |
| Airflow | Reverse Airflow - PSU to IO |
| Operating System | Open Network Installation Enviroment (ONIE) |
| Licenses installed | Open Network- No Dell OS Installed (OS10 Available on Request) |
| Switching Capacity | 840Gb/s |
| Management | Console |
| Chasis Type | 1U Rack Mounted |
| Rack Mounting Kit | Dell 1U Tandem Switch Tray Not Included |
| Additional Information | Supports 3rd Party OS |
The Dell PowerSwitch S4100-ON Series Spec Sheet has the following for the S4112F-ON:
- Ports: 12xSFP+ 3xQSFP28
- Max 10GbE density: 24 (12 SFP+ and 12 via QSFP28 breakout)
- Max 25GbE density: 12 via QSFP28 breakout
- Max 40GbE density: 3
- Max 50GbE density: 6
- Max 100GbE density: 3
For licenses OS10 licensing logic? on the Dell community has the following from the Moderator DELL-Young E:
Are there any other major differences between the above editions? >> Os10 Enterprise edition is a license required and trial edition works only 120 days after that the switch reboot by itself 70 hrs once.
The base Module of OS10 is the Open Edition. It includes the Linux distribution, as well as SAI (Open Compute project) and CPS APIs, providing Abstraction for NPU as well as the Control Plane. we will discuss these, in a bit. Open Edition is also Dell EMC’s contribution to Open Source Community, via the Open Switch Project & Sonic Project.
you can include these point also with previous answer.
OS10’s Open Edition, which is Dell EMC’s contribution to the Linux Community, is Free. The path to Custom Applications and build, is open to the customers with Open Edition.
If OS10 is factory-loaded on your switch, you do not need to install an OS10 license as it is pre-loaded with a perpetual license. If you download OS10 on a trial basis, OS10 comes with a 120-day trial license. ONIE installation on a factory-loaded Dell switch wipes out the license partition. Hence, the switch on reload comes up with a default 120 day trial license. Purchase and install a perpetual license after the trial period https://dell.to/3H3rGhB continue with uninterrupted use, purchase and install a perpetual license to avoid the OS10 device rebooting every 72 hours after 120 days.
To sum up, yes you are correct. For open edition, your switch will need to be in "ON" model, which in short for Open Networking, in such like S4148F-ON or S5212F-ON to be able to install Open Networking OS. Open Networking is 3rd party networking OS. An example: https://dell.to/3NS9Tfm
You will not be able to back up your OS image or firmware on the switch.