1..  BSD LICENSE
2    Copyright (C) Cavium networks Ltd. 2016.
3    All rights reserved.
4
5    Redistribution and use in source and binary forms, with or without
6    modification, are permitted provided that the following conditions
7    are met:
8
9    * Redistributions of source code must retain the above copyright
10    notice, this list of conditions and the following disclaimer.
11    * Redistributions in binary form must reproduce the above copyright
12    notice, this list of conditions and the following disclaimer in
13    the documentation and/or other materials provided with the
14    distribution.
15    * Neither the name of Cavium networks nor the names of its
16    contributors may be used to endorse or promote products derived
17    from this software without specific prior written permission.
18
19    THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
20    "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
21    LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
22    A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
23    OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
24    SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
25    LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
26    DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
27    THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
28    (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
29    OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
30
31ThunderX NICVF Poll Mode Driver
32===============================
33
34The ThunderX NICVF PMD (**librte_pmd_thunderx_nicvf**) provides poll mode driver
35support for the inbuilt NIC found in the **Cavium ThunderX** SoC family
36as well as their virtual functions (VF) in SR-IOV context.
37
38More information can be found at `Cavium Networks Official Website
39<http://www.cavium.com/ThunderX_ARM_Processors.html>`_.
40
41Features
42--------
43
44Features of the ThunderX PMD are:
45
46- Multiple queues for TX and RX
47- Receive Side Scaling (RSS)
48- Packet type information
49- Checksum offload
50- Promiscuous mode
51- Multicast mode
52- Port hardware statistics
53- Jumbo frames
54- Link state information
55- Scattered and gather for TX and RX
56- VLAN stripping
57- SR-IOV VF
58- NUMA support
59- Multi queue set support (up to 96 queues (12 queue sets)) per port
60
61Supported ThunderX SoCs
62-----------------------
63- CN88xx
64- CN81xx
65- CN83xx
66
67Prerequisites
68-------------
69- Follow the DPDK :ref:`Getting Started Guide for Linux <linux_gsg>` to setup the basic DPDK environment.
70
71Pre-Installation Configuration
72------------------------------
73
74Config File Options
75~~~~~~~~~~~~~~~~~~~
76
77The following options can be modified in the ``config`` file.
78Please note that enabling debugging options may affect system performance.
79
80- ``CONFIG_RTE_LIBRTE_THUNDERX_NICVF_PMD`` (default ``n``)
81
82  By default it is enabled only for defconfig_arm64-thunderx-* config.
83  Toggle compilation of the ``librte_pmd_thunderx_nicvf`` driver.
84
85- ``CONFIG_RTE_LIBRTE_THUNDERX_NICVF_DEBUG_INIT`` (default ``n``)
86
87  Toggle display of initialization related messages.
88
89- ``CONFIG_RTE_LIBRTE_THUNDERX_NICVF_DEBUG_RX`` (default ``n``)
90
91  Toggle display of receive fast path run-time message
92
93- ``CONFIG_RTE_LIBRTE_THUNDERX_NICVF_DEBUG_TX`` (default ``n``)
94
95  Toggle display of transmit fast path run-time message
96
97- ``CONFIG_RTE_LIBRTE_THUNDERX_NICVF_DEBUG_DRIVER`` (default ``n``)
98
99  Toggle display of generic debugging messages
100
101- ``CONFIG_RTE_LIBRTE_THUNDERX_NICVF_DEBUG_MBOX`` (default ``n``)
102
103  Toggle display of PF mailbox related run-time check messages
104
105Driver Compilation
106~~~~~~~~~~~~~~~~~~
107
108To compile the ThunderX NICVF PMD for Linux arm64 gcc target, run the
109following “make” command:
110
111.. code-block:: console
112
113   cd <DPDK-source-directory>
114   make config T=arm64-thunderx-linuxapp-gcc install
115
116Linux
117-----
118
119.. _thunderx_testpmd_example:
120
121Running testpmd
122~~~~~~~~~~~~~~~
123
124This section demonstrates how to launch ``testpmd`` with ThunderX NIC VF device
125managed by ``librte_pmd_thunderx_nicvf`` in the Linux operating system.
126
127#. Load ``vfio-pci`` driver:
128
129   .. code-block:: console
130
131      modprobe vfio-pci
132
133   .. _thunderx_vfio_noiommu:
134
135#. Enable **VFIO-NOIOMMU** mode (optional):
136
137   .. code-block:: console
138
139      echo 1 > /sys/module/vfio/parameters/enable_unsafe_noiommu_mode
140
141   .. note::
142
143      **VFIO-NOIOMMU** is required only when running in VM context and should not be enabled otherwise.
144      See also :ref:`SR-IOV: Prerequisites and sample Application Notes <thunderx_sriov_example>`.
145
146#. Bind the ThunderX NIC VF device to ``vfio-pci`` loaded in the previous step:
147
148   Setup VFIO permissions for regular users and then bind to ``vfio-pci``:
149
150   .. code-block:: console
151
152      ./tools/dpdk-devbind.py --bind vfio-pci 0002:01:00.2
153
154#. Start ``testpmd`` with basic parameters:
155
156   .. code-block:: console
157
158      ./arm64-thunderx-linuxapp-gcc/app/testpmd -c 0xf -n 4 -w 0002:01:00.2 \
159        -- -i --disable-hw-vlan-filter --no-flush-rx \
160        --port-topology=loop
161
162   Example output:
163
164   .. code-block:: console
165
166      ...
167
168      PMD: rte_nicvf_pmd_init(): librte_pmd_thunderx nicvf version 1.0
169
170      ...
171      EAL:   probe driver: 177d:11 rte_nicvf_pmd
172      EAL:   using IOMMU type 1 (Type 1)
173      EAL:   PCI memory mapped at 0x3ffade50000
174      EAL: Trying to map BAR 4 that contains the MSI-X table.
175           Trying offsets: 0x40000000000:0x0000, 0x10000:0x1f0000
176      EAL:   PCI memory mapped at 0x3ffadc60000
177      PMD: nicvf_eth_dev_init(): nicvf: device (177d:11) 2:1:0:2
178      PMD: nicvf_eth_dev_init(): node=0 vf=1 mode=tns-bypass sqs=false
179           loopback_supported=true
180      PMD: nicvf_eth_dev_init(): Port 0 (177d:11) mac=a6:c6:d9:17:78:01
181      Interactive-mode selected
182      Configuring Port 0 (socket 0)
183      ...
184
185      PMD: nicvf_dev_configure(): Configured ethdev port0 hwcap=0x0
186      Port 0: A6:C6:D9:17:78:01
187      Checking link statuses...
188      Port 0 Link Up - speed 10000 Mbps - full-duplex
189      Done
190      testpmd>
191
192.. _thunderx_sriov_example:
193
194SR-IOV: Prerequisites and sample Application Notes
195~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
196
197Current ThunderX NIC PF/VF kernel modules maps each physical Ethernet port
198automatically to virtual function (VF) and presented them as PCIe-like SR-IOV device.
199This section provides instructions to configure SR-IOV with Linux OS.
200
201#. Verify PF devices capabilities using ``lspci``:
202
203   .. code-block:: console
204
205      lspci -vvv
206
207   Example output:
208
209   .. code-block:: console
210
211      0002:01:00.0 Ethernet controller: Cavium Networks Device a01e (rev 01)
212      ...
213      Capabilities: [100 v1] Alternative Routing-ID Interpretation (ARI)
214      ...
215      Capabilities: [180 v1] Single Root I/O Virtualization (SR-IOV)
216      ...
217      Kernel driver in use: thunder-nic
218      ...
219
220   .. note::
221
222      Unless ``thunder-nic`` driver is in use make sure your kernel config includes ``CONFIG_THUNDER_NIC_PF`` setting.
223
224#. Verify VF devices capabilities and drivers using ``lspci``:
225
226   .. code-block:: console
227
228      lspci -vvv
229
230   Example output:
231
232   .. code-block:: console
233
234      0002:01:00.1 Ethernet controller: Cavium Networks Device 0011 (rev 01)
235      ...
236      Capabilities: [100 v1] Alternative Routing-ID Interpretation (ARI)
237      ...
238      Kernel driver in use: thunder-nicvf
239      ...
240
241      0002:01:00.2 Ethernet controller: Cavium Networks Device 0011 (rev 01)
242      ...
243      Capabilities: [100 v1] Alternative Routing-ID Interpretation (ARI)
244      ...
245      Kernel driver in use: thunder-nicvf
246      ...
247
248   .. note::
249
250      Unless ``thunder-nicvf`` driver is in use make sure your kernel config includes ``CONFIG_THUNDER_NIC_VF`` setting.
251
252#. Verify PF/VF bind using ``dpdk-devbind.py``:
253
254   .. code-block:: console
255
256      ./tools/dpdk-devbind.py --status
257
258   Example output:
259
260   .. code-block:: console
261
262      ...
263      0002:01:00.0 'Device a01e' if= drv=thunder-nic unused=vfio-pci
264      0002:01:00.1 'Device 0011' if=eth0 drv=thunder-nicvf unused=vfio-pci
265      0002:01:00.2 'Device 0011' if=eth1 drv=thunder-nicvf unused=vfio-pci
266      ...
267
268#. Load ``vfio-pci`` driver:
269
270   .. code-block:: console
271
272      modprobe vfio-pci
273
274#. Bind VF devices to ``vfio-pci`` using ``dpdk-devbind.py``:
275
276   .. code-block:: console
277
278      ./tools/dpdk-devbind.py --bind vfio-pci 0002:01:00.1
279      ./tools/dpdk-devbind.py --bind vfio-pci 0002:01:00.2
280
281#. Verify VF bind using ``dpdk-devbind.py``:
282
283   .. code-block:: console
284
285      ./tools/dpdk-devbind.py --status
286
287   Example output:
288
289   .. code-block:: console
290
291      ...
292      0002:01:00.1 'Device 0011' drv=vfio-pci unused=
293      0002:01:00.2 'Device 0011' drv=vfio-pci unused=
294      ...
295      0002:01:00.0 'Device a01e' if= drv=thunder-nic unused=vfio-pci
296      ...
297
298#. Pass VF device to VM context (PCIe Passthrough):
299
300   The VF devices may be passed through to the guest VM using qemu or
301   virt-manager or virsh etc.
302   ``librte_pmd_thunderx_nicvf`` or ``thunder-nicvf`` should be used to bind
303   the VF devices in the guest VM in :ref:`VFIO-NOIOMMU <thunderx_vfio_noiommu>` mode.
304
305   Example qemu guest launch command:
306
307   .. code-block:: console
308
309      sudo qemu-system-aarch64 -name vm1 \
310      -machine virt,gic_version=3,accel=kvm,usb=off \
311      -cpu host -m 4096 \
312      -smp 4,sockets=1,cores=8,threads=1 \
313      -nographic -nodefaults \
314      -kernel <kernel image> \
315      -append "root=/dev/vda console=ttyAMA0 rw hugepagesz=512M hugepages=3" \
316      -device vfio-pci,host=0002:01:00.1 \
317      -drive file=<rootfs.ext3>,if=none,id=disk1,format=raw  \
318      -device virtio-blk-device,scsi=off,drive=disk1,id=virtio-disk1,bootindex=1 \
319      -netdev tap,id=net0,ifname=tap0,script=/etc/qemu-ifup_thunder \
320      -device virtio-net-device,netdev=net0 \
321      -serial stdio \
322      -mem-path /dev/huge
323
324#. Refer to section :ref:`Running testpmd <thunderx_testpmd_example>` for instruction
325   how to launch ``testpmd`` application.
326
327Multiple Queue Set per DPDK port configuration
328~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
329
330There are two types of VFs:
331
332- Primary VF
333- Secondary VF
334
335Each port consists of a primary VF and n secondary VF(s). Each VF provides 8 Tx/Rx queues to a port.
336When a given port is configured to use more than 8 queues, it requires one (or more) secondary VF.
337Each secondary VF adds 8 additional queues to the queue set.
338
339During PMD driver initialization, the primary VF's are enumerated by checking the
340specific flag (see sqs message in DPDK boot log - sqs indicates secondary queue set).
341They are at the beginning of VF list (the remain ones are secondary VF's).
342
343The primary VFs are used as master queue sets. Secondary VFs provide
344additional queue sets for primary ones. If a port is configured for more then
3458 queues than it will request for additional queues from secondary VFs.
346
347Secondary VFs cannot be shared between primary VFs.
348
349Primary VFs are present on the beginning of the 'Network devices using kernel
350driver' list, secondary VFs are on the remaining on the remaining part of the list.
351
352   .. note::
353
354      The VNIC driver in the multiqueue setup works differently than other drivers like `ixgbe`.
355      We need to bind separately each specific queue set device with the ``tools/dpdk-devbind.py`` utility.
356
357   .. note::
358
359      Depending on the hardware used, the kernel driver sets a threshold ``vf_id``. VFs that try to attached with an id below or equal to
360      this boundary are considered primary VFs. VFs that try to attach with an id above this boundary are considered secondary VFs.
361
362
363Example device binding
364~~~~~~~~~~~~~~~~~~~~~~
365
366If a system has three interfaces, a total of 18 VF devices will be created
367on a non-NUMA machine.
368
369   .. note::
370
371      NUMA systems have 12 VFs per port and non-NUMA 6 VFs per port.
372
373   .. code-block:: console
374
375      # tools/dpdk-devbind.py --status
376
377      Network devices using DPDK-compatible driver
378      ============================================
379      <none>
380
381      Network devices using kernel driver
382      ===================================
383      0000:01:10.0 'Device a026' if= drv=thunder-BGX unused=vfio-pci,uio_pci_generic
384      0000:01:10.1 'Device a026' if= drv=thunder-BGX unused=vfio-pci,uio_pci_generic
385      0002:01:00.0 'Device a01e' if= drv=thunder-nic unused=vfio-pci,uio_pci_generic
386      0002:01:00.1 'Device 0011' if=eth0 drv=thunder-nicvf unused=vfio-pci,uio_pci_generic
387      0002:01:00.2 'Device 0011' if=eth1 drv=thunder-nicvf unused=vfio-pci,uio_pci_generic
388      0002:01:00.3 'Device 0011' if=eth2 drv=thunder-nicvf unused=vfio-pci,uio_pci_generic
389      0002:01:00.4 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic
390      0002:01:00.5 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic
391      0002:01:00.6 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic
392      0002:01:00.7 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic
393      0002:01:01.0 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic
394      0002:01:01.1 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic
395      0002:01:01.2 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic
396      0002:01:01.3 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic
397      0002:01:01.4 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic
398      0002:01:01.5 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic
399      0002:01:01.6 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic
400      0002:01:01.7 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic
401      0002:01:02.0 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic
402      0002:01:02.1 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic
403      0002:01:02.2 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic
404
405      Other network devices
406      =====================
407      0002:00:03.0 'Device a01f' unused=vfio-pci,uio_pci_generic
408
409
410We want to bind two physical interfaces with 24 queues each device, we attach two primary VFs
411and four secondary queues. In our example we choose two 10G interfaces eth1 (0002:01:00.2) and eth2 (0002:01:00.3).
412We will choose four secondary queue sets from the ending of the list (0002:01:01.7-0002:01:02.2).
413
414
415#. Bind two primary VFs to the ``vfio-pci`` driver:
416
417   .. code-block:: console
418
419      tools/dpdk-devbind.py -b vfio-pci 0002:01:00.2
420      tools/dpdk-devbind.py -b vfio-pci 0002:01:00.3
421
422#. Bind four primary VFs to the ``vfio-pci`` driver:
423
424   .. code-block:: console
425
426      tools/dpdk-devbind.py -b vfio-pci 0002:01:01.7
427      tools/dpdk-devbind.py -b vfio-pci 0002:01:02.0
428      tools/dpdk-devbind.py -b vfio-pci 0002:01:02.1
429      tools/dpdk-devbind.py -b vfio-pci 0002:01:02.2
430
431The nicvf thunderx driver will make use of attached secondary VFs automatically during the interface configuration stage.
432
433Limitations
434-----------
435
436CRC striping
437~~~~~~~~~~~~
438
439The ThunderX SoC family NICs strip the CRC for every packets coming into the
440host interface. So, CRC will be stripped even when the
441``rxmode.hw_strip_crc`` member is set to 0 in ``struct rte_eth_conf``.
442
443Maximum packet length
444~~~~~~~~~~~~~~~~~~~~~
445
446The ThunderX SoC family NICs support a maximum of a 9K jumbo frame. The value
447is fixed and cannot be changed. So, even when the ``rxmode.max_rx_pkt_len``
448member of ``struct rte_eth_conf`` is set to a value lower than 9200, frames
449up to 9200 bytes can still reach the host interface.
450
451Maximum packet segments
452~~~~~~~~~~~~~~~~~~~~~~~
453
454The ThunderX SoC family NICs support up to 12 segments per packet when working
455in scatter/gather mode. So, setting MTU will result with ``EINVAL`` when the
456frame size does not fit in the maximum number of segments.
457