My colleagues and I recently built a filer using OI 151a/ZFS and wanted to increase the throughput by aggregating multiple NICs to a common IP address that is dedicated for data. We did it by physically adding a card with 4 NICs, aggregating the ports on server and then updating the switch to enable link aggregation (port trunking) on the 2 connections.
Step 1 – Add the PCI card
For this filer, we added a 4 NIC PCI/e mezannine card to the server.
Step 2 – Aggregate the NICs on the server
I then logged into the server as root and aggregating the NIC’s. This process was slightly different than the what I found on the web because OI 151a is based on Solaris 11. Here is what I did.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 |
% # Turn off the automatic network management % svcadm disable /network/physical:nwam % # Show the links % ifconfig -a % dladm show-link % # Unplumb the current physical devices (from show-link) % ifconfig bnx0 unplumb % ifconfig bnx1 unplumb % # Create the aggregation, name it "aggr1" % dladm create-aggr -l bnx0 -l bnx1 aggr1 % # Create the plumbing for it % ifconfig aggr plumb % # Assign the IP address % ifconfig aggr 192.168.1.200 netmask 255.255.255.0 up % # Verify that it is up % ifconfig -a % dladm show-aggr |
Step 3 – Program the Switch
Finally we needed to enable link aggregation (aka Port Trunking) on the switch to make sure that it can handle multiple MACs for a single IP address. If you don’t do this, performance will degrade to be worse than a single port because the switch will keep getting confused by different MACs.
This step is different for all switches. I have done it with a Cisco Catalyst 3750 and one of my colleagues did it with a Netgear GS176T. The instructions for doing it can be found in the user manuals.
On line 13 you mention aggr1, but on lines 16 and 19 aggr (without 1). Is that correct?
We’re also using an aggregation but based on LACP by adding options “-P L2,L3 -L active” on line 13. I’m very curious in how you configured the etherchannel on the Cisco 3750. Can you publish the relevant configuration pieces to make this post complete?
BTW. Great post due to it’s simplicity…