User:Zucca/wip/InfiniBand/IP-over-InfiniBand/draft

IP over InfiniBand (IPoIB)

Users of InfiniBand can also use it to carry IP-networking packets, thus allowing it to replace ethernet in some cases. From at least kernel version 4.10 onwards users can compile IP-over-IB in-kernel (CONFIG_INFINIBAND_IPOIB). However if that's not the case and the support is compiled as kernel module it may be possible that the module isn't automatically loaded. The module is named ib_ipoib:

root #modprobe ib_ipoib

InfiniBand network interfaces are usually named as ib<number>:

user $ifconfig ib0

ib0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 2044
Infiniband hardware address can be incorrect! Please read BUGS section in ifconfig(8).
        infiniband 00:00:04:04:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00  txqueuelen 256  (InfiniBand)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

Performance tuning

When using IP over InfiniBand ther performance is usually low by default. This is because the default mtu of each InfiniBand IP interface is set to a low value.

Changing the mtu of InfiniBand HCA

Automatic

The most conveivent way is to chance mtu automatically when kernel adds the interface to the system. Before mtu can be changed the mode of the interface must be changed to 'connected' from 'datagram' Next example uses udev rules to accoplish that.

FILE /etc/udev/rules.d/2-InfiniBand.rulesudev rules for InfiniBand IP networking interfaces

# Rules to set InfiniBand device attributes.

# Change mode to connected and change mtu to maximum for best performance.
ACTION=="add", KERNEL=="ib[0-9]*", SUBSYSTEM=="net", ATTR{mode}="connected", ATTR{mtu}="65520"

It has been reported that the rule above does not work. Users with this problem may use the following instead:

FILE /etc/udev/rules.d/2-InfiniBand.rulesudev rules for InfiniBand IP networking interfaces

# Rules to set InfiniBand device attributes.

# Change mode to connected and change mtu to maximum for best performance.
ACTION=="add", KERNEL=="ib[0-9]*", SUBSYSTEM=="net", RUN+="/bin/sh -c 'echo connected > /sys/class/net/%k/mode && echo 65520 > /sys/class/net/%k/mtu'"

Manual

Mode and mtu can also be changed manually in run time. Next commands assume the interface in question is named ib0.

root #echo connected > /sys/class/net/ib0/mode

root #echo 65520 > /sys/class/net/ib0/mtu