- EU-Vietnam Business Network (EVBN)

C H A P T E R

Internet Protocol Over InfiniBand

This chapter describes configuration aspects of running the Internet Protocol over InfiniBand (IPoIB).

Configuring IPoIB on Linux

You might decide to change your IPoIB configuration for a variety of reasons, including the installation of an additional InfiniBand ExpressModule (IB EM).

To Install IPoIB Driver

1. Detemine whether the IPoIB driver is already installed by typing the

lsmod | grep ib command.

The output from this command shows all the IB drivers.

In the following sample output, note that the driver, ib_ipoib, is not listed.

> lsmod | grep ib
ib_sdp       45340 0
rdma_cm      26760 1 ib_sdp
ib_addr      10504 1 rdma_cm
ib_local_sa  14232 1 rdma_cm
findex       6528 1 ib_local_sa
ib_ipath     70552 0
ipath_core   179652 1 ib_ipath
ib_mthca     139184 0
ib_uverbs    47536 0
ib_umad      19888 0
ib_ucm       21512 0
ib_sa        18196 2 rdma_cm,ib_local_sa
ib_cm        39952 2 rdma_cm,ib_ucm
ib_mad       43176 5 ib_local_sa,ib_mthca,ib_umad,ib_sa,ib_cm
ib_core      59520 11
ib_sdp,rdma_cm,ib_local_sa,ib_ipath,ib_mthca,ib_uverbs,ib_umad, ib_ucm,ib_sa,ib_cm,ib_mad

2. To install the IPoIB driver, enter the modprobe command:

> modprobe ib_ipoib

3. Enter the lsmod | grep ib command again and note that ib_ipoib is now listed.

> lsmod | grep ib
ib_ipoib     59800 0
ib_sdp       45340 0
rdma_cm      26760 1 ib_sdp
ib_addr      10504 1 rdma_cm
ib_local_sa  14232 1 rdma_cm
.
.
.
ib_core      59520 11
ib_sdp,rdma_cm,ib_local_sa,ib_ipath,ib_mthca,ib_uverbs,ib_umad, ib_ucm,ib_sa,ib_cm,ib_mad

4. Type the ifconfig command to check for network interface ib0.

> ifconfig ib0
ib0     Link encap:UNSPEC  HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
        BROADCAST MULTICAST  MTU:2044  Metric:1
        RX packets:0 errors:0 dropped:0 overruns:0 frame:0
        TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
        collisions:0 txqueuelen:128
        RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)

Note that network interface ib0 is present but has no valid IP address.

To assign an address, see To Change IPoIB Configuration Without Rebooting.

To Change IPoIB Configuration Without Rebooting

caution icon
Caution – The changes made by this procedure are only temporary. These changes are lost during a reboot. To make permanent changes, see To Change IB Startup Behavior on Linux.

1. Start the InfiniBand IP network by typing the ifconfig command and assigning a valid IP address for ib0.

> ifconfig ib0 10.0.0.50/24
ib0     Link encap:UNSPEC  HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
        inet addr:10.0.0.50  Bcast: 10.0.0.255  Mask:255.255.255.0
        UP BROADCAST RUNNING MULTICAST  MTU:2044  Metric:1
        RX packets:0 errors:0 dropped:0 overruns:0 frame:0
        TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
        collisions:0 txqueuelen:128
        RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)

If ib0 is unconfigured, it appears without an IP address as shown in the following output.

> ifconfig ib0
ib0     Link encap:UNSPEC  HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
        BROADCAST MULTICAST  MTU:2044  Metric:1 
        RX packets:0 errors:0 dropped:0 overruns:0 frame:0
        TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
        collisions:0 txqueuelen:128
        RX bytes:0 (0.0 b)  TX bytes:0 ( 0.0 b)

2. Type the route command to verify that the 10.0.0 subnet is available.

The following output shows subnet 10.0.0 present and routed through ib0.

> route
Kernel IP routing table
Destination Gateway    Genmask        Flags  Metric Ref  Use  Iface
10.0.0.0    *          255.255.255.0  U      0      0    0    ib0
10.8.134.0  *          255.255.255.0         0      0    0    eth0
169.254.0.0 *          255.255.0.0    U      0      0    0    eth0
default     ban3rtr0d0 0.0.0.0        UG     0      0    0    eth0

3. As shown in the following example, you can enter the ping command to see another IPoIB node on the 10.0.0 subnet:

> ping 10.0.0.79
PING 10.0.0.79 (10.0.0.79) 56(84) bytes of data.
64 bytes from 10.0.0.79: icmp_seq=0 ttl=255 time=1.82 ms
64 bytes from 10.0.0.79: icmp_seq=1 ttl=255 time=0.082 ms
64 bytes from 10.0.0.79: icmp_seq=2 ttl=255 time= 0.062 ms
 
--- 10.0.0.79 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2001ms
rtt min/avg/max/mdev = 0.062/0.655/1.823/0.825 ms, pipe 2

At this point, the IPoIB network is active and properly configured without rebooting.

To Change IB Startup Behavior on Linux

You can change the InfiniBand startup behavior by editing the openib.conf configuration file. (The full path is /etc/infiniband/openib.conf).

In the following example, openib.conf specifies that whenever the system boots, the InfiniBand services, IPoIB, and the SDP IP service are to start up automatically (ONBOOT=yes, IPOIB_LOAD=yes, SDP_LOAD=yes). However, openib.conf specifies that the SRP service is NOT to start up automatically (SRP_LOAD=no). You can alter any and all of these parameters.

1. Edit ipoib.conf.

The following is an example of the ipoib.conf file.

# Start HCA driver upon boot
ONBOOT=yes
# Load UCM module
UCM_LOAD=no
# Load RDMA_CM module
RDMA_CM_LOAD=no
# Load RDMA_UCM module
RDMA_UCM_LOAD=no
# Load MTHCA
MTHCA_LOAD=yes
# Load IPATH
IPATH_LOAD=yes
# Load IPoIB
IPOIB_LOAD=yes
# Load SDP module
SDP_LOAD=yes
# Load SRP module
SRP_LOAD=no
# Load RDS module
RDS_LOAD=no

2. Create (or edit) the ifcfg-ibn file to configure an individual network interface.

For each InfiniBand network interface, you will need a corresponding startup file (ifcfg-ibn) in your startup scripts directory.

The directory for startup scripts might be /etc/sysconfig/network/, /etc/sysconfig/network-scripts/, or similar, depending on your version of Linux distribution.

As an example, the startup file for ib0, might look something like the following.

more /etc/sysconfig/network-scripts/ifcfg-ib0
 DEVICE=ib0
 BOOTPROTO=static
 IPADDR=10.0.0.50
 NETMASK=255.255.255.0
 NETWORK=10.0.0.0
 BROADCAST=10.0.0.255
 ONBOOT=yes

The ONBOOT=yes parameter indicates that the corresponding IP network interface is to automatically start up when the system boots. Specifying ONBOOT=no will configure the interface but not start it.

Generally, if you have enabled IPoIB services, the IB stack installation scripts automatically create the ifcfg-ibn configuration files for all IB network interfaces present. If you install an IB EM after you have installed the IB stack, you might need to manually create the ifcfg-ibn files for the newly installed network interfaces.