1a7752129SBenoît Ganne# RDMA (ibverb) Ethernet driver {#rdma_doc}
2a7752129SBenoît Ganne
3a7752129SBenoît GanneThis driver relies on Linux rdma-core (libibverb) userspace poll-mode driver
4a7752129SBenoît Ganneto rx/tx Ethernet packets. Despite using the RDMA APIs, this is **not** about
5a7752129SBenoît GanneRDMA (no Infiniband, no RoCE, no iWARP), just pure traditional Ethernet
6a7752129SBenoît Gannepackets.
7a7752129SBenoît Ganne
8a7752129SBenoît Ganne## Maturity level
9a7752129SBenoît GanneUnder development: it should work, but has not been thoroughly tested.
10a7752129SBenoît Ganne
11a7752129SBenoît Ganne## Supported Hardware
12a7752129SBenoît Ganne - Mellanox ConnectX-4
13a7752129SBenoît Ganne - Mellanox ConnectX-5
14a7752129SBenoît Ganne
15a7752129SBenoît Ganne## Features
16a7752129SBenoît Ganne - bifurcation: MAC based flow steering for transparent sharing of a single
17a7752129SBenoît Gannephysical port between multiple virtual interfaces including Linux netdev
18a7752129SBenoît Ganne - multiqueue
19a7752129SBenoît Ganne
20a7752129SBenoît Ganne## Security considerations
21a7752129SBenoît GanneWhen creating a rdma interface, it will receive all packets to the MAC address
22a7752129SBenoît Ganneattributed to the interface plus a copy of all broadcast and multicast
23a7752129SBenoît Gannetraffic.
24a7752129SBenoît GanneThe MAC address is under the control of VPP: **the user controlling VPP can
25a7752129SBenoît Gannedivert all traffic of any MAC address to the VPP process, including the Linux
26a7752129SBenoît Gannenetdev MAC address as long as it can create a rdma interface**.
27a7752129SBenoît GanneThe rights to create a rdma interface are controlled by the access rights of
28a7752129SBenoît Gannethe `/dev/infiniband/uverbs[0-9]+`device nodes.
29a7752129SBenoît Ganne
30a7752129SBenoît Ganne## Quickstart
31a7752129SBenoît Ganne1. Make sure the `ib_uverbs` module is loaded:
32a7752129SBenoît Ganne```
33a7752129SBenoît Ganne~# modprobe ib_uverbs
34a7752129SBenoît Ganne```
35a7752129SBenoît Ganne2. In VPP, create a new rdma virtual interface tied to the Linux netdev of the
36a7752129SBenoît Gannephysical port you want to use (`enp94s0f0` in this example):
37a7752129SBenoît Ganne```
38a7752129SBenoît Gannevpp# create int rdma host-if enp94s0f0 name rdma-0
39a7752129SBenoît Ganne```
40a7752129SBenoît Ganne3. Use the interface as usual, eg.:
41a7752129SBenoît Ganne```
42a7752129SBenoît Gannevpp# set int ip addr rdma-0 1.1.1.1/24
43a7752129SBenoît Gannevpp# set int st rdma-0 up
44a7752129SBenoît Gannevpp# ping 1.1.1.100`
45a7752129SBenoît Ganne```
46a7752129SBenoît Ganne
47dc812d9aSBenoît Ganne## Containers support
48a7752129SBenoît GanneIt should work in containers as long as:
49a7752129SBenoît Ganne - the `ib_uverbs` module is loaded
50a7752129SBenoît Ganne - the device nodes `/dev/infiniband/uverbs[0-9]+` are usable from the
51a7752129SBenoît Ganne   container (but see [security considerations](#Security considerations))
520dcafcc5SBenoît Ganne
53dc812d9aSBenoît Ganne## SR-IOV VFs support
540dcafcc5SBenoît GanneIt should work on SR-IOV VFs the same way it does with PFs. Because of VFs
550dcafcc5SBenoît Gannesecurity containment features, make sure the MAC address of the rdma VPP
560dcafcc5SBenoît Ganneinterface matches the MAC address assigned to the underlying VF.
570dcafcc5SBenoît GanneFor example:
580dcafcc5SBenoît Ganne```
590dcafcc5SBenoît Gannehost# echo 1 > /sys/class/infiniband/mlx5_0/device/sriov_numvfs
600dcafcc5SBenoît Gannehost# ip l set dev enp94s0f0 vf 0 mac 92:5d:f5:df:b1:6f spoof on trust off
610dcafcc5SBenoît Gannehost# ip l set dev enp94s0f2 up
620dcafcc5SBenoît Gannevpp# create int rdma host-if enp94s0f2 name rdma-0
630dcafcc5SBenoît Gannevpp# set int mac address rdma-0 92:5d:f5:df:b1:6f
640dcafcc5SBenoît Ganne```
650dcafcc5SBenoît GanneIf you plan to use L2 features such as switching, make sure the underlying
660dcafcc5SBenoît GanneVF is configured in trusted mode and spoof-checking is disabled (of course, be
670dcafcc5SBenoît Ganneaware of the [security considerations](#Security considerations)):
680dcafcc5SBenoît Ganne```
690dcafcc5SBenoît Gannehost# ip l set dev enp94s0f0 vf 0 spoof off trust on
700dcafcc5SBenoît Ganne```
71dc812d9aSBenoît Ganne
72dc812d9aSBenoît Ganne## Direct Verb mode
73dc812d9aSBenoît GanneDirect Verb allows the driver to access the NIC HW RX/TX rings directly
74dc812d9aSBenoît Ganneinstead of having to go through libibverb and suffering associated overhead.
75dc812d9aSBenoît GanneIt will be automatically selected if the adapter supports it.
76