IPoIB – Using TCP/IP on an InfiniBand Network

Existing applications can take advantage of the higher bandwidth and lower latency of InfiniBand by use of IPoIB, Internet Protocol over InfiniBand. When the driver for IPoIB is loaded virtual network interfaces are made visible to the operating system. These devices appear is if they were Ethernet device and can be manipulated in the same way as a regular Ethernet device. The virtual devices use the name “ib” and are enumerated from 0: ib0, ib1, etc.

Enabling IPoIB using OFED

The configuration for which drivers are loaded when using OFED is located in /etc/infiniband/openib.conf. In this file set these values:

IPOIB_LOAD=yes
SET_IPOIB_CM=yes

Restart the openibd service:

$ service openibd restart

Check for success by running ‘ifconfig -a’ and looking for devices starting with “ib”.

Enabling IPoIB using RHEL or CentOS Provided Software

The configuration for the distribution provided InfiniBand software is located in /etc/rdma/rdma.conf. In this file, set this value:

IPOIB_LOAD=yes

Restart the rdma service:

$ service rdma restart

Check for success by running ‘ifconfig -a’ and looking for devices starting with “ib”.

Configure The IP Information

Once the ib network devices are present, the IPoIB interface can be configured using the standard network configuration tools: system-config-network or by creating/editing the files directly in /etc/sysconfig/network-scripts.