My colleagues and I recently built a filer using OI 151a/ZFS and wanted to increase the throughput by aggregating multiple NICs to a common IP address that is dedicated for data. We did it by physically adding a card with 4 NICs, aggregating the ports on server and then updating the switch to enable link aggregation (port trunking) on the 2 connections.
Step 1 – Add the PCI card
For this filer, we added a 4 NIC PCI/e mezannine card to the server.
Step 2 – Aggregate the NICs on the server
I then logged into the server as root and aggregating the NIC’s. This process was slightly different than the what I found on the web because OI 151a is based on Solaris 11. Here is what I did.
% # Turn off the automatic network management
% svcadm disable /network/physical:nwam
% # Show the links
% ifconfig -a
% dladm show-link
% # Unplumb the current physical devices (from show-link)
% ifconfig bnx0 unplumb
% ifconfig bnx1 unplumb
% # Create the aggregation, name it "aggr1"
% dladm create-aggr -l bnx0 -l bnx1 aggr1
% # Create the plumbing for it
% ifconfig aggr plumb
% # Assign the IP address
% ifconfig aggr 192.168.1.200 netmask 255.255.255.0 up
% # Verify that it is up
% ifconfig -a
% dladm show-aggr
Step 3 – Program the Switch
Finally we needed to enable link aggregation (aka Port Trunking) on the switch to make sure that it can handle multiple MACs for a single IP address. If you don’t do this, performance will degrade to be worse than a single port because the switch will keep getting confused by different MACs.
This step is different for all switches. I have done it with a Cisco Catalyst 3750 and one of my colleagues did it with a Netgear GS176T. The instructions for doing it can be found in the user manuals.