thunderx.rst revision 8b25d1ad
1..  BSD LICENSE
2    Copyright (C) Cavium networks Ltd. 2016.
3    All rights reserved.
4
5    Redistribution and use in source and binary forms, with or without
6    modification, are permitted provided that the following conditions
7    are met:
8
9    * Redistributions of source code must retain the above copyright
10    notice, this list of conditions and the following disclaimer.
11    * Redistributions in binary form must reproduce the above copyright
12    notice, this list of conditions and the following disclaimer in
13    the documentation and/or other materials provided with the
14    distribution.
15    * Neither the name of Cavium networks nor the names of its
16    contributors may be used to endorse or promote products derived
17    from this software without specific prior written permission.
18
19    THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
20    "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
21    LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
22    A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
23    OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
24    SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
25    LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
26    DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
27    THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
28    (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
29    OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
30
31ThunderX NICVF Poll Mode Driver
32===============================
33
34The ThunderX NICVF PMD (**librte_pmd_thunderx_nicvf**) provides poll mode driver
35support for the inbuilt NIC found in the **Cavium ThunderX** SoC family
36as well as their virtual functions (VF) in SR-IOV context.
37
38More information can be found at `Cavium Networks Official Website
39<http://www.cavium.com/ThunderX_ARM_Processors.html>`_.
40
41Features
42--------
43
44Features of the ThunderX PMD are:
45
46- Multiple queues for TX and RX
47- Receive Side Scaling (RSS)
48- Packet type information
49- Checksum offload
50- Promiscuous mode
51- Multicast mode
52- Port hardware statistics
53- Jumbo frames
54- Link state information
55- Scattered and gather for TX and RX
56- VLAN stripping
57- SR-IOV VF
58- NUMA support
59
60Supported ThunderX SoCs
61-----------------------
62- CN88xx
63
64Prerequisites
65-------------
66- Follow the DPDK :ref:`Getting Started Guide for Linux <linux_gsg>` to setup the basic DPDK environment.
67
68Pre-Installation Configuration
69------------------------------
70
71Config File Options
72~~~~~~~~~~~~~~~~~~~
73
74The following options can be modified in the ``config`` file.
75Please note that enabling debugging options may affect system performance.
76
77- ``CONFIG_RTE_LIBRTE_THUNDERX_NICVF_PMD`` (default ``n``)
78
79  By default it is enabled only for defconfig_arm64-thunderx-* config.
80  Toggle compilation of the ``librte_pmd_thunderx_nicvf`` driver.
81
82- ``CONFIG_RTE_LIBRTE_THUNDERX_NICVF_DEBUG_INIT`` (default ``n``)
83
84  Toggle display of initialization related messages.
85
86- ``CONFIG_RTE_LIBRTE_THUNDERX_NICVF_DEBUG_RX`` (default ``n``)
87
88  Toggle display of receive fast path run-time message
89
90- ``CONFIG_RTE_LIBRTE_THUNDERX_NICVF_DEBUG_TX`` (default ``n``)
91
92  Toggle display of transmit fast path run-time message
93
94- ``CONFIG_RTE_LIBRTE_THUNDERX_NICVF_DEBUG_DRIVER`` (default ``n``)
95
96  Toggle display of generic debugging messages
97
98- ``CONFIG_RTE_LIBRTE_THUNDERX_NICVF_DEBUG_MBOX`` (default ``n``)
99
100  Toggle display of PF mailbox related run-time check messages
101
102Driver Compilation
103~~~~~~~~~~~~~~~~~~
104
105To compile the ThunderX NICVF PMD for Linux arm64 gcc target, run the
106following “make” command:
107
108.. code-block:: console
109
110   cd <DPDK-source-directory>
111   make config T=arm64-thunderx-linuxapp-gcc install
112
113Linux
114-----
115
116.. _thunderx_testpmd_example:
117
118Running testpmd
119~~~~~~~~~~~~~~~
120
121This section demonstrates how to launch ``testpmd`` with ThunderX NIC VF device
122managed by ``librte_pmd_thunderx_nicvf`` in the Linux operating system.
123
124#. Load ``vfio-pci`` driver:
125
126   .. code-block:: console
127
128      modprobe vfio-pci
129
130   .. _thunderx_vfio_noiommu:
131
132#. Enable **VFIO-NOIOMMU** mode (optional):
133
134   .. code-block:: console
135
136      echo 1 > /sys/module/vfio/parameters/enable_unsafe_noiommu_mode
137
138   .. note::
139
140      **VFIO-NOIOMMU** is required only when running in VM context and should not be enabled otherwise.
141      See also :ref:`SR-IOV: Prerequisites and sample Application Notes <thunderx_sriov_example>`.
142
143#. Bind the ThunderX NIC VF device to ``vfio-pci`` loaded in the previous step:
144
145   Setup VFIO permissions for regular users and then bind to ``vfio-pci``:
146
147   .. code-block:: console
148
149      ./tools/dpdk_nic_bind.py --bind vfio-pci 0002:01:00.2
150
151#. Start ``testpmd`` with basic parameters:
152
153   .. code-block:: console
154
155      ./arm64-thunderx-linuxapp-gcc/app/testpmd -c 0xf -n 4 -w 0002:01:00.2 \
156        -- -i --disable-hw-vlan-filter --crc-strip --no-flush-rx \
157        --port-topology=loop
158
159   Example output:
160
161   .. code-block:: console
162
163      ...
164
165      PMD: rte_nicvf_pmd_init(): librte_pmd_thunderx nicvf version 1.0
166
167      ...
168      EAL:   probe driver: 177d:11 rte_nicvf_pmd
169      EAL:   using IOMMU type 1 (Type 1)
170      EAL:   PCI memory mapped at 0x3ffade50000
171      EAL: Trying to map BAR 4 that contains the MSI-X table.
172           Trying offsets: 0x40000000000:0x0000, 0x10000:0x1f0000
173      EAL:   PCI memory mapped at 0x3ffadc60000
174      PMD: nicvf_eth_dev_init(): nicvf: device (177d:11) 2:1:0:2
175      PMD: nicvf_eth_dev_init(): node=0 vf=1 mode=tns-bypass sqs=false
176           loopback_supported=true
177      PMD: nicvf_eth_dev_init(): Port 0 (177d:11) mac=a6:c6:d9:17:78:01
178      Interactive-mode selected
179      Configuring Port 0 (socket 0)
180      ...
181
182      PMD: nicvf_dev_configure(): Configured ethdev port0 hwcap=0x0
183      Port 0: A6:C6:D9:17:78:01
184      Checking link statuses...
185      Port 0 Link Up - speed 10000 Mbps - full-duplex
186      Done
187      testpmd>
188
189.. _thunderx_sriov_example:
190
191SR-IOV: Prerequisites and sample Application Notes
192~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
193
194Current ThunderX NIC PF/VF kernel modules maps each physical Ethernet port
195automatically to virtual function (VF) and presented them as PCIe-like SR-IOV device.
196This section provides instructions to configure SR-IOV with Linux OS.
197
198#. Verify PF devices capabilities using ``lspci``:
199
200   .. code-block:: console
201
202      lspci -vvv
203
204   Example output:
205
206   .. code-block:: console
207
208      0002:01:00.0 Ethernet controller: Cavium Networks Device a01e (rev 01)
209              ...
210              Capabilities: [100 v1] Alternative Routing-ID Interpretation (ARI)
211              ...
212              Capabilities: [180 v1] Single Root I/O Virtualization (SR-IOV)
213              ...
214              Kernel driver in use: thunder-nic
215              ...
216
217   .. note::
218
219      Unless ``thunder-nic`` driver is in use make sure your kernel config includes ``CONFIG_THUNDER_NIC_PF`` setting.
220
221#. Verify VF devices capabilities and drivers using ``lspci``:
222
223   .. code-block:: console
224
225      lspci -vvv
226
227   Example output:
228
229   .. code-block:: console
230
231      0002:01:00.1 Ethernet controller: Cavium Networks Device 0011 (rev 01)
232              ...
233              Capabilities: [100 v1] Alternative Routing-ID Interpretation (ARI)
234              ...
235              Kernel driver in use: thunder-nicvf
236              ...
237
238      0002:01:00.2 Ethernet controller: Cavium Networks Device 0011 (rev 01)
239              ...
240              Capabilities: [100 v1] Alternative Routing-ID Interpretation (ARI)
241              ...
242              Kernel driver in use: thunder-nicvf
243              ...
244
245   .. note::
246
247      Unless ``thunder-nicvf`` driver is in use make sure your kernel config includes ``CONFIG_THUNDER_NIC_VF`` setting.
248
249#. Verify PF/VF bind using ``dpdk_nic_bind.py``:
250
251   .. code-block:: console
252
253      ./tools/dpdk_nic_bind.py --status
254
255   Example output:
256
257   .. code-block:: console
258
259      ...
260      0002:01:00.0 'Device a01e' if= drv=thunder-nic unused=vfio-pci
261      0002:01:00.1 'Device 0011' if=eth0 drv=thunder-nicvf unused=vfio-pci
262      0002:01:00.2 'Device 0011' if=eth1 drv=thunder-nicvf unused=vfio-pci
263      ...
264
265#. Load ``vfio-pci`` driver:
266
267   .. code-block:: console
268
269      modprobe vfio-pci
270
271#. Bind VF devices to ``vfio-pci`` using ``dpdk_nic_bind.py``:
272
273   .. code-block:: console
274
275      ./tools/dpdk_nic_bind.py --bind vfio-pci 0002:01:00.1
276      ./tools/dpdk_nic_bind.py --bind vfio-pci 0002:01:00.2
277
278#. Verify VF bind using ``dpdk_nic_bind.py``:
279
280   .. code-block:: console
281
282      ./tools/dpdk_nic_bind.py --status
283
284   Example output:
285
286   .. code-block:: console
287
288      ...
289      0002:01:00.1 'Device 0011' drv=vfio-pci unused=
290      0002:01:00.2 'Device 0011' drv=vfio-pci unused=
291      ...
292      0002:01:00.0 'Device a01e' if= drv=thunder-nic unused=vfio-pci
293      ...
294
295#. Pass VF device to VM context (PCIe Passthrough):
296
297   The VF devices may be passed through to the guest VM using qemu or
298   virt-manager or virsh etc.
299   ``librte_pmd_thunderx_nicvf`` or ``thunder-nicvf`` should be used to bind
300   the VF devices in the guest VM in :ref:`VFIO-NOIOMMU <thunderx_vfio_noiommu>` mode.
301
302   Example qemu guest launch command:
303
304   .. code-block:: console
305
306      sudo qemu-system-aarch64 -name vm1 \
307      -machine virt,gic_version=3,accel=kvm,usb=off \
308      -cpu host -m 4096 \
309      -smp 4,sockets=1,cores=8,threads=1 \
310      -nographic -nodefaults \
311      -kernel <kernel image> \
312      -append "root=/dev/vda console=ttyAMA0 rw hugepagesz=512M hugepages=3" \
313      -device vfio-pci,host=0002:01:00.1 \
314      -drive file=<rootfs.ext3>,if=none,id=disk1,format=raw  \
315      -device virtio-blk-device,scsi=off,drive=disk1,id=virtio-disk1,bootindex=1 \
316      -netdev tap,id=net0,ifname=tap0,script=/etc/qemu-ifup_thunder \
317      -device virtio-net-device,netdev=net0 \
318      -serial stdio \
319      -mem-path /dev/huge
320
321#. Refer to section :ref:`Running testpmd <thunderx_testpmd_example>` for instruction
322   how to launch ``testpmd`` application.
323
324Limitations
325-----------
326
327CRC striping
328~~~~~~~~~~~~
329
330The ThunderX SoC family NICs strip the CRC for every packets coming into the
331host interface. So, CRC will be stripped even when the
332``rxmode.hw_strip_crc`` member is set to 0 in ``struct rte_eth_conf``.
333
334Maximum packet length
335~~~~~~~~~~~~~~~~~~~~~
336
337The ThunderX SoC family NICs support a maximum of a 9K jumbo frame. The value
338is fixed and cannot be changed. So, even when the ``rxmode.max_rx_pkt_len``
339member of ``struct rte_eth_conf`` is set to a value lower than 9200, frames
340up to 9200 bytes can still reach the host interface.
341
342Maximum packet segments
343~~~~~~~~~~~~~~~~~~~~~~~
344
345The ThunderX SoC family NICs support up to 12 segments per packet when working
346in scatter/gather mode. So, setting MTU will result with ``EINVAL`` when the
347frame size does not fit in the maximum number of segments.
348
349Limited VFs
350~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
351
352The ThunderX SoC family NICs has 128VFs and each VF has 8/8 queues
353for RX/TX respectively. Current driver implementation has one to one mapping
354between physical port and VF hence only limited VFs can be used.
355