It has been a little more than a decade since the introduction of the first consumer computer with gigabit Ethernet and there have been drastic increases in CPU processing power and storage throughput. Gigabit Ethernet has become an obvious bottleneck for high throughput applications. 10Gbps is the next step for Ethernet, but is not the only option when looking for greater network speed. A potentially faster and less expensive alternative to 10GbE is InfiniBand. This guide shows how to configure IPoIB on an Ubuntu 12.04.1 or similar host.
Overview
Ubuntu 12.04.1 LTS loads the ib_mthca and mlx4_core modules by default when a corresponding host channel adapter (HCA) is detected at boot. The ib_mthca module provides support for older Mellanox InfiniHost III HCAs and the mlx4_* set of modules provides support for ConnectX and newer Mellanox HCAs. When using ConnectX and newer HCAs, two modules are required, mlx4_core and mlx4_ib. With Ubuntu the latter modules is not loaded by default so it must be loaded manually. The last required module for an IPoIB node is the ib_ipoib module itself. Another useful module to load on every node is the user space management datagram module, ib_umad. This module lets you use scripts like ibstat and ibhosts from the infiniband-diags package. Last, a subnet manager is required for any Infiniband fabric. OpenSM is a cross platform open source subnet manager and under Linux requires the ib_umad module to function.
[toggle_box title=”Required Modules (click to expand)” width=”600″]ib_mthca – InfiniHost III and older HCAs
mlx4_core and mlx4_ib – ConnectX and newer HCA
ib_ipoib – the IPoIB module
ib_umad – required by OpenSM, useful on clients for diagnostics (ibstat, ibhosts)[/toggle_box]
Ephemeral Ubuntu Mellanox IPoIB Setup
1. Load the required modules using modprobe.
If using a ConnectX or newer HCA, load mlx4_ib.
[box_light]max@IBClient:~ $ sudo modprobe mlx4_ib[/box_light]
Load ib_ipoib and ib_umad.
[box_light]max@IBClient:~ $ sudo modprobe ib_umad
max@IBClient:~ $ sudo modprobe ib_ipoib[/box_light]
Confirm the modules have been loaded.
[box_light]max@IBClient:~ $ lsmod | egrep ‘mlx4|mthca|ib_ipoib|ib_umad'[/box_light]
1a. On one node, install OpenSM.
[box_light]max@IBClient:~ $ sudo apt-get install opensm[/box_light]
2. Once multiple nodes are up, check the link with ibstat. The example output below is from a dual port ConnectX HCA.
[box_light]max@IBClient:~ $ ibstat
CA ‘mlx4_0’
CA type: MT25418
Number of ports: 2
Firmware version: 2.9.1000
Hardware version: a0
Node GUID: 0x0008f104039a07fc
System image GUID: 0x0008f104039a07ff
Port 1:
State: Initializing
Physical state: LinkUp
Rate: 20
Base lid: 0
LMC: 0
SM lid: 0
Capability mask: 0x02510868
Port GUID: 0x0008f104039a07fd
Port 2:
State: Down
Physical state: Polling
Rate: 28
Base lid: 0
LMC: 0
SM lid: 0
Capability mask: 0x02510868
Port GUID: 0x0008f104039a07fe[/box_light]
3. Check for the IPoIB interface with ifconfig. The example output below is from dual port ConnectX HCA.
[box_light]max@IBClient:~ $ ifconfig -a
…
ib0 Link encap:UNSPEC HWaddr 80-00-00-48-FE-80-00-00-00-00-00-00-00-00-00-00
BROADCAST MULTICAST MTU:2044 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:256
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
ib1 Link encap:UNSPEC HWaddr 80-00-00-49-FE-80-00-00-00-00-00-00-00-00-00-00
BROADCAST MULTICAST MTU:2044 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:256
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
…[/box_light]
4. Configure the IPoIP interface ib0 like an regular Ethernet interface.
[box_light]max@IBClient:~ $ sudo ifconfig ib0 10.10.10.1 netmask 255.255.255.0[/box_light]
For connected mode, also run the following two commands.
[box_light]max@IBClient:~ $ echo ‘connected’ | sudo tee /sys/class/net/ib0/mode > /dev/null[/box_light]
To change the MTU in connected mode run the following, where X is the desired MTU <= 65520.
[box_light]max@IBClient:~ $ sudo ifconfig ib0 mtu X[/box_light]
Repeat the procedure on other nodes using different IP addresses and without installing OpenSM.
Permanent Ubuntu Mellanox IPoIB Setup
0. For the Infiniband network to function, a subnet manager is needed on one of the nodes. When setting up a point to point network or using an Infiniband switch without a subnet manager, install OpenSM on one of the nodes.
[box_light]max@IBClient:~ $ sudo apt-get install opensm[/box_light]
1. Add required modules to /etc/modules. If using a ConnectX or newer HCA, also add mlx4_ib.
[box_light]max@IBClient:~ $ sudo -s
root@IBClient:~ # printf “mlx4_ib” >> /etc/modules
root@IBClient:~ # printf “ib_umad” >> /etc/modules
root@IBClient:~ # printf “ib_ipoib” >> /etc/modules[/box_light]
Here is an example /etc/modules for a machine with a ConnectX-2 HCA.
[box_light]root@IBClient:~ # cat /etc/modules
# /etc/modules: kernel modules to load at boot time.
#
# This file contains the names of kernel modules that should be loaded
# at boot time, one per line. Lines beginning with “#” are ignored.
loop
lp
rtc
mlx4_ib
ib_umad
ib_ipoib[/box_light]
2. Edit /etc/network/interfaces adding the IPoIB interface and the desired IP and subnetmask. It is always a good idea to make a backup, /etc/network/interfaces.bak, to revert to if anything goes wrong.
[box_light]root@IBClient:~ # cp /etc/network/interfaces{,.bak}[/box_light]
To make it simple, printf(1) is used to add the new lines to /etc/network/interfaces. Replace XXX.XXX.XXX.XXX with the IP and YYY.YYY.YYY.YYY with the subnetmask.
[box_light]root@IBClient:~ # printf “\n\nauto ib0” >> /etc/network/interfaces
root@IBClient:~ # printf “iface ib0 inet static” >> /etc/network/interfaces
root@IBClient:~ # printf “\taddress XXX.XXX.XXX.XXX” >> /etc/network/interfaces
root@IBClient:~ # printf “\tnetmask YYY.YYY.YYY.YYY” >> /etc/network/interfaces[/box_light]
For connected mode, also add the following two lines where Z is the desired MTU(<= 65520).
[box_light]root@IBClient:~ # printf “post-up echo connected > /sys/class/net/ib0/mode” >> /etc/network/interfaces
root@IBClient:~ # printf “post-up /sbin/ifconfig $IFACE mtu Z >> /etc/network/interfaces[/box_light]
3. Check your /etc/network/interfaces file with cat(1). Below is an example with ib0 configured in connected mode with max MTU.
[box_light]root@IBClient:~ # cat /etc/network/interaces
…
auto ib0
iface ib0 inet static
address 10.0.0.1
netmask 255.255.255.0
post-up echo connected > /sys/class/net/ib0/mode
post-up /sbin/ifconfig $IFACE mtu 65520
…
[/box_light]
Reboot for everything to take effect. Repeat steps 1 through 3 on the other node with different IP address.
Conclusion:
IPoIB configuration nicely mimics regular Ethernet except the extra required modules. The mainline kernel has superb support for Mellanox InfiniBand hardware and helps make set up uneventful with a vanilla Ubuntu Server install. The next installment will compare the performance of multiple HCA generations.
Thanks in advance for your detailed explanation. It is really hard to find such an step by step tutorial for IB all over internet!
I have HP DL360 G7, Mellanox Connectx-2 40GB, SUN switch (i guess with subnet manager on it) and couple of local comuter asrender node. Installing the drivers on the ubuntu server 14.04 based on your tutorial, I got to the point that I can see my configured static ip address and other info after cat /etc/network/interfaces. But, other local clients only can access to the shared samba folders through the primary em1 (ethernt) and not IB at all.
I am pretty new to Linux (3 days or so) and eny help would be highly appreciated.
Thanks again
Ehsan