RoCE Configuration for Ubuntu

To configure RoCE on the adapter for an Ubuntu host, RDMA must be installed and configured on the Ubuntu host. To configure and set up RoCE for Ubuntu 14.04.5/16.04.1 Linux:

1. When you begin installing the Ubuntu server, verify if the basic packages, modules, and tools are available for Ethernet and RDMA. Log in as ROOT,and install all required packages.

   a. Install the basic packages required for Ubuntu:

   ```bash
   apt-get install build-essential cmake gcc \
   pkg-config vlan automake autoconf dkms git ninja-build \
   libudev-dev libnl-3-dev libnl-route-3-dev valgrind python3-dev cython3 python3-docutils pandoc

   b. Install the following RDMA packages required for Ubuntu:
   

   apt-get install –f libibverbs* librdma* libibcm.* libibmad.* libibumad*
   ```

   Or, Install from source code

   git clone	https://github.com/linux-rdma/rdma-core.git
   

   bash [ -d build ] || mkdir build cd build cmake -DCMAKE_INSTALL_PREFIX=/usr -DCMAKE_BUILD_TYPE=Release .. cmake --build . sudo cmake --build . --target install

   c.Install RDMA user space tools and libraries required for Ubuntu:

   apt-get install -f libtool ibutils ibverbs-utils \
   rdmacm-utils infiniband-diags perftest librdmacm-dev \
   libibverbs-dev numactl libnuma-dev libnl-3-200 \
   libnl-route-3-200 libnl-route-3-dev libnl-utils 
   

2. If the file /etc/udev/rules.d/40-rdma.rules does not exist, create it with the following content:

   cat > /etc/udev/rules.d/40-rdma.rules << "EOF"
   KERNEL=="umad*", NAME="infiniband/%k"
   KERNEL=="issm*", NAME="infiniband/%k"
   KERNEL=="ucm*", NAME="infiniband/%k", MODE="0666"
   KERNEL=="uverbs*", NAME="infiniband/%k", MODE="0666"
   KERNEL=="ucma", NAME="infiniband/%k", MODE="0666"
   KERNEL=="rdma_cm", NAME="infiniband/%k", MODE="0666"
   EOF
   

3. Edit the /etc/security/limits.conf file to increase the size of memory, which can be locked by a non-root user. Add the following lines, and then log out:

   cat >> /etc/security/limits.conf << "EOF"
   * soft memlock unlimited
   * hard memlock unlimited
   root soft memlock unlimited
   root hard memlock unlimited
   EOF
   

4. Log into the system again, or verify after reboot. Then issue the following command:
   bash ulimit -l
   You should get the output as unlimited .

5. Reboot the system.

6. To allow the device to be recognized as an InfiniBand device that can be used by OFED, install the FastLinQ package by issuing the following commands:
   bash cd fastlinq-X.X.X.X make clean make install

7. Install the libqedr libraries to work with RDMA user space applications using one of the following command options:
   Option 1:

   cd fastlinq-X.X.X.X
   make libqedr_install
   

   Option 2:

   cd fastlinq-X.X.X.X/libqedr-X.X.X.X/
   ./configure --prefix=/usr --libdir=${exec_prefix}/lib --sysconfdir=/etc make install
   

8. Before loading the QLogic Ethernet and RDMA drivers, uninstall the existing out-of-box or inbox drivers by issuing the following commands:

   modprobe –r qede
   depmod –a
   modprobe -v qedr
   

9. Load the RDMA modules by issuing the following commands. You must perform this step whenever you reboot the system.

    modprobe rdma_cm
    modprobe ib_uverbs
    modprobe rdma_ucm
    modprobe ib_ucm
    modprobe ib_umad
   

10. To list RoCE devices, issue the following command:

    ibv_devinfo