## RoCE Configuration for Ubuntu

To configure RoCE on the adapter for an Ubuntu host, RDMA must be installed and configured on the Ubuntu host. To configure and set up RoCE for Ubuntu 14.04.5/16.04.1 Linux:

1. When you begin installing the Ubuntu server, verify if the basic packages, modules, and tools are available for Ethernet and RDMA. Log in as ROOT,and install all required packages.

	a. Install the basic packages required for Ubuntu:
    
    ```bash
    apt-get install build-essential cmake gcc \
     pkg-config vlan automake autoconf dkms git  ninja-build  \
    libudev-dev libnl-3-dev libnl-route-3-dev valgrind python3-dev cython3 python3-docutils pandoc
    
    ```
    b. Install the following RDMA packages required for Ubuntu:
    ```bash
	apt-get install –f libibverbs* librdma* libibcm.* libibmad.* libibumad*
  	```
    
     Or, Install from source code
           
    ```bash
    git clone	https://github.com/linux-rdma/rdma-core.git
    ```
    
    ```bash
    [ -d build ] || mkdir build
    cd build
    cmake -DCMAKE_INSTALL_PREFIX=/usr -DCMAKE_BUILD_TYPE=Release  ..
    cmake --build .
    sudo cmake --build . --target install
     ```
     
    c.Install RDMA user space tools and libraries required for Ubuntu:
    
    ```bash
    apt-get install -f libtool ibutils ibverbs-utils \
    rdmacm-utils infiniband-diags perftest librdmacm-dev \
    libibverbs-dev numactl libnuma-dev libnl-3-200 \
    libnl-route-3-200 libnl-route-3-dev libnl-utils 
    ```
    
2. If the file /etc/udev/rules.d/40-rdma.rules does not exist, create it with the following content:

    ```bash
cat > /etc/udev/rules.d/40-rdma.rules << "EOF"
KERNEL=="umad*", NAME="infiniband/%k"
KERNEL=="issm*", NAME="infiniband/%k"
KERNEL=="ucm*", NAME="infiniband/%k", MODE="0666"
KERNEL=="uverbs*", NAME="infiniband/%k", MODE="0666"
KERNEL=="ucma", NAME="infiniband/%k", MODE="0666"
KERNEL=="rdma_cm", NAME="infiniband/%k", MODE="0666"
EOF
```

3. Edit the /etc/security/limits.conf file to increase the size of memory, which can be locked by a non-root user. Add the following lines, and then log out:

    ```bash
    cat >> /etc/security/limits.conf << "EOF"
    * soft memlock unlimited
    * hard memlock unlimited
    root soft memlock unlimited
    root hard memlock unlimited
    EOF
    ```
4. Log into the system again, or verify after reboot. Then issue the following command:
 ```bash
ulimit -l
 ```
You should get the output as `unlimited` .

5. Reboot the system.

6. To allow the device to be recognized as an InfiniBand device that can be used by OFED, install the FastLinQ package by issuing the following commands:
 ```bash
cd fastlinq-X.X.X.X
make clean
make install
```
7. Install the libqedr libraries to work with RDMA user space applications using one of the following command options:
    Option 1:

    ```bash
    cd fastlinq-X.X.X.X
    make libqedr_install
    ```

    Option 2:

    ```bash
    cd fastlinq-X.X.X.X/libqedr-X.X.X.X/
    ./configure --prefix=/usr --libdir=${exec_prefix}/lib --sysconfdir=/etc make install
    ```

8. Before loading the QLogic Ethernet and RDMA drivers, uninstall the existing out-of-box or inbox drivers by issuing the following commands:

    ```bash
    modprobe –r qede
    depmod –a
    modprobe -v qedr
    ```

9. Load the RDMA modules by issuing the following commands. You must perform this step whenever you reboot the system.

    ```bash
     modprobe rdma_cm
     modprobe ib_uverbs
     modprobe rdma_ucm
     modprobe ib_ucm
     modprobe ib_umad
    ```

10. To list RoCE devices, issue the following command:

    ```bash
    ibv_devinfo
    ```