1809f0800SChristian Ehrhardt..  BSD LICENSE
2809f0800SChristian Ehrhardt    Copyright (C) Cavium networks Ltd. 2016.
3809f0800SChristian Ehrhardt    All rights reserved.
4809f0800SChristian Ehrhardt
5809f0800SChristian Ehrhardt    Redistribution and use in source and binary forms, with or without
6809f0800SChristian Ehrhardt    modification, are permitted provided that the following conditions
7809f0800SChristian Ehrhardt    are met:
8809f0800SChristian Ehrhardt
9809f0800SChristian Ehrhardt    * Redistributions of source code must retain the above copyright
10809f0800SChristian Ehrhardt    notice, this list of conditions and the following disclaimer.
11809f0800SChristian Ehrhardt    * Redistributions in binary form must reproduce the above copyright
12809f0800SChristian Ehrhardt    notice, this list of conditions and the following disclaimer in
13809f0800SChristian Ehrhardt    the documentation and/or other materials provided with the
14809f0800SChristian Ehrhardt    distribution.
15809f0800SChristian Ehrhardt    * Neither the name of Cavium networks nor the names of its
16809f0800SChristian Ehrhardt    contributors may be used to endorse or promote products derived
17809f0800SChristian Ehrhardt    from this software without specific prior written permission.
18809f0800SChristian Ehrhardt
19809f0800SChristian Ehrhardt    THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
20809f0800SChristian Ehrhardt    "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
21809f0800SChristian Ehrhardt    LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
22809f0800SChristian Ehrhardt    A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
23809f0800SChristian Ehrhardt    OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
24809f0800SChristian Ehrhardt    SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
25809f0800SChristian Ehrhardt    LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
26809f0800SChristian Ehrhardt    DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
27809f0800SChristian Ehrhardt    THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
28809f0800SChristian Ehrhardt    (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
29809f0800SChristian Ehrhardt    OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
30809f0800SChristian Ehrhardt
31809f0800SChristian EhrhardtThunderX NICVF Poll Mode Driver
32809f0800SChristian Ehrhardt===============================
33809f0800SChristian Ehrhardt
34809f0800SChristian EhrhardtThe ThunderX NICVF PMD (**librte_pmd_thunderx_nicvf**) provides poll mode driver
35809f0800SChristian Ehrhardtsupport for the inbuilt NIC found in the **Cavium ThunderX** SoC family
36809f0800SChristian Ehrhardtas well as their virtual functions (VF) in SR-IOV context.
37809f0800SChristian Ehrhardt
38809f0800SChristian EhrhardtMore information can be found at `Cavium Networks Official Website
39809f0800SChristian Ehrhardt<http://www.cavium.com/ThunderX_ARM_Processors.html>`_.
40809f0800SChristian Ehrhardt
41809f0800SChristian EhrhardtFeatures
42809f0800SChristian Ehrhardt--------
43809f0800SChristian Ehrhardt
44809f0800SChristian EhrhardtFeatures of the ThunderX PMD are:
45809f0800SChristian Ehrhardt
46809f0800SChristian Ehrhardt- Multiple queues for TX and RX
47809f0800SChristian Ehrhardt- Receive Side Scaling (RSS)
48809f0800SChristian Ehrhardt- Packet type information
49809f0800SChristian Ehrhardt- Checksum offload
50809f0800SChristian Ehrhardt- Promiscuous mode
51809f0800SChristian Ehrhardt- Multicast mode
52809f0800SChristian Ehrhardt- Port hardware statistics
53809f0800SChristian Ehrhardt- Jumbo frames
54809f0800SChristian Ehrhardt- Link state information
55809f0800SChristian Ehrhardt- Scattered and gather for TX and RX
56809f0800SChristian Ehrhardt- VLAN stripping
57809f0800SChristian Ehrhardt- SR-IOV VF
58809f0800SChristian Ehrhardt- NUMA support
593d9b7210SChristian Ehrhardt- Multi queue set support (up to 96 queues (12 queue sets)) per port
60809f0800SChristian Ehrhardt
61809f0800SChristian EhrhardtSupported ThunderX SoCs
62809f0800SChristian Ehrhardt-----------------------
63809f0800SChristian Ehrhardt- CN88xx
643d9b7210SChristian Ehrhardt- CN81xx
653d9b7210SChristian Ehrhardt- CN83xx
66809f0800SChristian Ehrhardt
67809f0800SChristian EhrhardtPrerequisites
68809f0800SChristian Ehrhardt-------------
69809f0800SChristian Ehrhardt- Follow the DPDK :ref:`Getting Started Guide for Linux <linux_gsg>` to setup the basic DPDK environment.
70809f0800SChristian Ehrhardt
71809f0800SChristian EhrhardtPre-Installation Configuration
72809f0800SChristian Ehrhardt------------------------------
73809f0800SChristian Ehrhardt
74809f0800SChristian EhrhardtConfig File Options
75809f0800SChristian Ehrhardt~~~~~~~~~~~~~~~~~~~
76809f0800SChristian Ehrhardt
77809f0800SChristian EhrhardtThe following options can be modified in the ``config`` file.
78809f0800SChristian EhrhardtPlease note that enabling debugging options may affect system performance.
79809f0800SChristian Ehrhardt
80809f0800SChristian Ehrhardt- ``CONFIG_RTE_LIBRTE_THUNDERX_NICVF_PMD`` (default ``n``)
81809f0800SChristian Ehrhardt
82809f0800SChristian Ehrhardt  By default it is enabled only for defconfig_arm64-thunderx-* config.
83809f0800SChristian Ehrhardt  Toggle compilation of the ``librte_pmd_thunderx_nicvf`` driver.
84809f0800SChristian Ehrhardt
85809f0800SChristian Ehrhardt- ``CONFIG_RTE_LIBRTE_THUNDERX_NICVF_DEBUG_INIT`` (default ``n``)
86809f0800SChristian Ehrhardt
87809f0800SChristian Ehrhardt  Toggle display of initialization related messages.
88809f0800SChristian Ehrhardt
89809f0800SChristian Ehrhardt- ``CONFIG_RTE_LIBRTE_THUNDERX_NICVF_DEBUG_RX`` (default ``n``)
90809f0800SChristian Ehrhardt
91809f0800SChristian Ehrhardt  Toggle display of receive fast path run-time message
92809f0800SChristian Ehrhardt
93809f0800SChristian Ehrhardt- ``CONFIG_RTE_LIBRTE_THUNDERX_NICVF_DEBUG_TX`` (default ``n``)
94809f0800SChristian Ehrhardt
95809f0800SChristian Ehrhardt  Toggle display of transmit fast path run-time message
96809f0800SChristian Ehrhardt
97809f0800SChristian Ehrhardt- ``CONFIG_RTE_LIBRTE_THUNDERX_NICVF_DEBUG_DRIVER`` (default ``n``)
98809f0800SChristian Ehrhardt
99809f0800SChristian Ehrhardt  Toggle display of generic debugging messages
100809f0800SChristian Ehrhardt
101809f0800SChristian Ehrhardt- ``CONFIG_RTE_LIBRTE_THUNDERX_NICVF_DEBUG_MBOX`` (default ``n``)
102809f0800SChristian Ehrhardt
103809f0800SChristian Ehrhardt  Toggle display of PF mailbox related run-time check messages
104809f0800SChristian Ehrhardt
105809f0800SChristian EhrhardtDriver Compilation
106809f0800SChristian Ehrhardt~~~~~~~~~~~~~~~~~~
107809f0800SChristian Ehrhardt
108809f0800SChristian EhrhardtTo compile the ThunderX NICVF PMD for Linux arm64 gcc target, run the
109809f0800SChristian Ehrhardtfollowing “make” command:
110809f0800SChristian Ehrhardt
111809f0800SChristian Ehrhardt.. code-block:: console
112809f0800SChristian Ehrhardt
113809f0800SChristian Ehrhardt   cd <DPDK-source-directory>
114809f0800SChristian Ehrhardt   make config T=arm64-thunderx-linuxapp-gcc install
115809f0800SChristian Ehrhardt
116809f0800SChristian EhrhardtLinux
117809f0800SChristian Ehrhardt-----
118809f0800SChristian Ehrhardt
119809f0800SChristian Ehrhardt.. _thunderx_testpmd_example:
120809f0800SChristian Ehrhardt
121809f0800SChristian EhrhardtRunning testpmd
122809f0800SChristian Ehrhardt~~~~~~~~~~~~~~~
123809f0800SChristian Ehrhardt
124809f0800SChristian EhrhardtThis section demonstrates how to launch ``testpmd`` with ThunderX NIC VF device
125809f0800SChristian Ehrhardtmanaged by ``librte_pmd_thunderx_nicvf`` in the Linux operating system.
126809f0800SChristian Ehrhardt
127809f0800SChristian Ehrhardt#. Load ``vfio-pci`` driver:
128809f0800SChristian Ehrhardt
129809f0800SChristian Ehrhardt   .. code-block:: console
130809f0800SChristian Ehrhardt
131809f0800SChristian Ehrhardt      modprobe vfio-pci
132809f0800SChristian Ehrhardt
133809f0800SChristian Ehrhardt   .. _thunderx_vfio_noiommu:
134809f0800SChristian Ehrhardt
135809f0800SChristian Ehrhardt#. Enable **VFIO-NOIOMMU** mode (optional):
136809f0800SChristian Ehrhardt
137809f0800SChristian Ehrhardt   .. code-block:: console
138809f0800SChristian Ehrhardt
139809f0800SChristian Ehrhardt      echo 1 > /sys/module/vfio/parameters/enable_unsafe_noiommu_mode
140809f0800SChristian Ehrhardt
141809f0800SChristian Ehrhardt   .. note::
142809f0800SChristian Ehrhardt
143809f0800SChristian Ehrhardt      **VFIO-NOIOMMU** is required only when running in VM context and should not be enabled otherwise.
144809f0800SChristian Ehrhardt      See also :ref:`SR-IOV: Prerequisites and sample Application Notes <thunderx_sriov_example>`.
145809f0800SChristian Ehrhardt
146809f0800SChristian Ehrhardt#. Bind the ThunderX NIC VF device to ``vfio-pci`` loaded in the previous step:
147809f0800SChristian Ehrhardt
148809f0800SChristian Ehrhardt   Setup VFIO permissions for regular users and then bind to ``vfio-pci``:
149809f0800SChristian Ehrhardt
150809f0800SChristian Ehrhardt   .. code-block:: console
151809f0800SChristian Ehrhardt
1525b1ff351SRicardo Salveti      ./tools/dpdk-devbind.py --bind vfio-pci 0002:01:00.2
153809f0800SChristian Ehrhardt
154809f0800SChristian Ehrhardt#. Start ``testpmd`` with basic parameters:
155809f0800SChristian Ehrhardt
156809f0800SChristian Ehrhardt   .. code-block:: console
157809f0800SChristian Ehrhardt
158809f0800SChristian Ehrhardt      ./arm64-thunderx-linuxapp-gcc/app/testpmd -c 0xf -n 4 -w 0002:01:00.2 \
1599365d6cfSChristian Ehrhardt        -- -i --disable-hw-vlan-filter --no-flush-rx \
160809f0800SChristian Ehrhardt        --port-topology=loop
161809f0800SChristian Ehrhardt
162809f0800SChristian Ehrhardt   Example output:
163809f0800SChristian Ehrhardt
164809f0800SChristian Ehrhardt   .. code-block:: console
165809f0800SChristian Ehrhardt
166809f0800SChristian Ehrhardt      ...
167809f0800SChristian Ehrhardt
168809f0800SChristian Ehrhardt      PMD: rte_nicvf_pmd_init(): librte_pmd_thunderx nicvf version 1.0
169809f0800SChristian Ehrhardt
170809f0800SChristian Ehrhardt      ...
171809f0800SChristian Ehrhardt      EAL:   probe driver: 177d:11 rte_nicvf_pmd
172809f0800SChristian Ehrhardt      EAL:   using IOMMU type 1 (Type 1)
173809f0800SChristian Ehrhardt      EAL:   PCI memory mapped at 0x3ffade50000
174809f0800SChristian Ehrhardt      EAL: Trying to map BAR 4 that contains the MSI-X table.
175809f0800SChristian Ehrhardt           Trying offsets: 0x40000000000:0x0000, 0x10000:0x1f0000
176809f0800SChristian Ehrhardt      EAL:   PCI memory mapped at 0x3ffadc60000
177809f0800SChristian Ehrhardt      PMD: nicvf_eth_dev_init(): nicvf: device (177d:11) 2:1:0:2
178809f0800SChristian Ehrhardt      PMD: nicvf_eth_dev_init(): node=0 vf=1 mode=tns-bypass sqs=false
179809f0800SChristian Ehrhardt           loopback_supported=true
180809f0800SChristian Ehrhardt      PMD: nicvf_eth_dev_init(): Port 0 (177d:11) mac=a6:c6:d9:17:78:01
181809f0800SChristian Ehrhardt      Interactive-mode selected
182809f0800SChristian Ehrhardt      Configuring Port 0 (socket 0)
183809f0800SChristian Ehrhardt      ...
184809f0800SChristian Ehrhardt
185809f0800SChristian Ehrhardt      PMD: nicvf_dev_configure(): Configured ethdev port0 hwcap=0x0
186809f0800SChristian Ehrhardt      Port 0: A6:C6:D9:17:78:01
187809f0800SChristian Ehrhardt      Checking link statuses...
188809f0800SChristian Ehrhardt      Port 0 Link Up - speed 10000 Mbps - full-duplex
189809f0800SChristian Ehrhardt      Done
190809f0800SChristian Ehrhardt      testpmd>
191809f0800SChristian Ehrhardt
192809f0800SChristian Ehrhardt.. _thunderx_sriov_example:
193809f0800SChristian Ehrhardt
194809f0800SChristian EhrhardtSR-IOV: Prerequisites and sample Application Notes
195809f0800SChristian Ehrhardt~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
196809f0800SChristian Ehrhardt
197809f0800SChristian EhrhardtCurrent ThunderX NIC PF/VF kernel modules maps each physical Ethernet port
198809f0800SChristian Ehrhardtautomatically to virtual function (VF) and presented them as PCIe-like SR-IOV device.
199809f0800SChristian EhrhardtThis section provides instructions to configure SR-IOV with Linux OS.
200809f0800SChristian Ehrhardt
201809f0800SChristian Ehrhardt#. Verify PF devices capabilities using ``lspci``:
202809f0800SChristian Ehrhardt
203809f0800SChristian Ehrhardt   .. code-block:: console
204809f0800SChristian Ehrhardt
205809f0800SChristian Ehrhardt      lspci -vvv
206809f0800SChristian Ehrhardt
207809f0800SChristian Ehrhardt   Example output:
208809f0800SChristian Ehrhardt
209809f0800SChristian Ehrhardt   .. code-block:: console
210809f0800SChristian Ehrhardt
211809f0800SChristian Ehrhardt      0002:01:00.0 Ethernet controller: Cavium Networks Device a01e (rev 01)
2123d9b7210SChristian Ehrhardt      ...
2133d9b7210SChristian Ehrhardt      Capabilities: [100 v1] Alternative Routing-ID Interpretation (ARI)
2143d9b7210SChristian Ehrhardt      ...
2153d9b7210SChristian Ehrhardt      Capabilities: [180 v1] Single Root I/O Virtualization (SR-IOV)
2163d9b7210SChristian Ehrhardt      ...
2173d9b7210SChristian Ehrhardt      Kernel driver in use: thunder-nic
2183d9b7210SChristian Ehrhardt      ...
219809f0800SChristian Ehrhardt
220809f0800SChristian Ehrhardt   .. note::
221809f0800SChristian Ehrhardt
222809f0800SChristian Ehrhardt      Unless ``thunder-nic`` driver is in use make sure your kernel config includes ``CONFIG_THUNDER_NIC_PF`` setting.
223809f0800SChristian Ehrhardt
224809f0800SChristian Ehrhardt#. Verify VF devices capabilities and drivers using ``lspci``:
225809f0800SChristian Ehrhardt
226809f0800SChristian Ehrhardt   .. code-block:: console
227809f0800SChristian Ehrhardt
228809f0800SChristian Ehrhardt      lspci -vvv
229809f0800SChristian Ehrhardt
230809f0800SChristian Ehrhardt   Example output:
231809f0800SChristian Ehrhardt
232809f0800SChristian Ehrhardt   .. code-block:: console
233809f0800SChristian Ehrhardt
234809f0800SChristian Ehrhardt      0002:01:00.1 Ethernet controller: Cavium Networks Device 0011 (rev 01)
2353d9b7210SChristian Ehrhardt      ...
2363d9b7210SChristian Ehrhardt      Capabilities: [100 v1] Alternative Routing-ID Interpretation (ARI)
2373d9b7210SChristian Ehrhardt      ...
2383d9b7210SChristian Ehrhardt      Kernel driver in use: thunder-nicvf
2393d9b7210SChristian Ehrhardt      ...
240809f0800SChristian Ehrhardt
241809f0800SChristian Ehrhardt      0002:01:00.2 Ethernet controller: Cavium Networks Device 0011 (rev 01)
2423d9b7210SChristian Ehrhardt      ...
2433d9b7210SChristian Ehrhardt      Capabilities: [100 v1] Alternative Routing-ID Interpretation (ARI)
2443d9b7210SChristian Ehrhardt      ...
2453d9b7210SChristian Ehrhardt      Kernel driver in use: thunder-nicvf
2463d9b7210SChristian Ehrhardt      ...
247809f0800SChristian Ehrhardt
248809f0800SChristian Ehrhardt   .. note::
249809f0800SChristian Ehrhardt
250809f0800SChristian Ehrhardt      Unless ``thunder-nicvf`` driver is in use make sure your kernel config includes ``CONFIG_THUNDER_NIC_VF`` setting.
251809f0800SChristian Ehrhardt
2525b1ff351SRicardo Salveti#. Verify PF/VF bind using ``dpdk-devbind.py``:
253809f0800SChristian Ehrhardt
254809f0800SChristian Ehrhardt   .. code-block:: console
255809f0800SChristian Ehrhardt
2565b1ff351SRicardo Salveti      ./tools/dpdk-devbind.py --status
257809f0800SChristian Ehrhardt
258809f0800SChristian Ehrhardt   Example output:
259809f0800SChristian Ehrhardt
260809f0800SChristian Ehrhardt   .. code-block:: console
261809f0800SChristian Ehrhardt
262809f0800SChristian Ehrhardt      ...
263809f0800SChristian Ehrhardt      0002:01:00.0 'Device a01e' if= drv=thunder-nic unused=vfio-pci
264809f0800SChristian Ehrhardt      0002:01:00.1 'Device 0011' if=eth0 drv=thunder-nicvf unused=vfio-pci
265809f0800SChristian Ehrhardt      0002:01:00.2 'Device 0011' if=eth1 drv=thunder-nicvf unused=vfio-pci
266809f0800SChristian Ehrhardt      ...
267809f0800SChristian Ehrhardt
268809f0800SChristian Ehrhardt#. Load ``vfio-pci`` driver:
269809f0800SChristian Ehrhardt
270809f0800SChristian Ehrhardt   .. code-block:: console
271809f0800SChristian Ehrhardt
272809f0800SChristian Ehrhardt      modprobe vfio-pci
273809f0800SChristian Ehrhardt
2745b1ff351SRicardo Salveti#. Bind VF devices to ``vfio-pci`` using ``dpdk-devbind.py``:
275809f0800SChristian Ehrhardt
276809f0800SChristian Ehrhardt   .. code-block:: console
277809f0800SChristian Ehrhardt
2785b1ff351SRicardo Salveti      ./tools/dpdk-devbind.py --bind vfio-pci 0002:01:00.1
2795b1ff351SRicardo Salveti      ./tools/dpdk-devbind.py --bind vfio-pci 0002:01:00.2
280809f0800SChristian Ehrhardt
2815b1ff351SRicardo Salveti#. Verify VF bind using ``dpdk-devbind.py``:
282809f0800SChristian Ehrhardt
283809f0800SChristian Ehrhardt   .. code-block:: console
284809f0800SChristian Ehrhardt
2855b1ff351SRicardo Salveti      ./tools/dpdk-devbind.py --status
286809f0800SChristian Ehrhardt
287809f0800SChristian Ehrhardt   Example output:
288809f0800SChristian Ehrhardt
289809f0800SChristian Ehrhardt   .. code-block:: console
290809f0800SChristian Ehrhardt
291809f0800SChristian Ehrhardt      ...
292809f0800SChristian Ehrhardt      0002:01:00.1 'Device 0011' drv=vfio-pci unused=
293809f0800SChristian Ehrhardt      0002:01:00.2 'Device 0011' drv=vfio-pci unused=
294809f0800SChristian Ehrhardt      ...
295809f0800SChristian Ehrhardt      0002:01:00.0 'Device a01e' if= drv=thunder-nic unused=vfio-pci
296809f0800SChristian Ehrhardt      ...
297809f0800SChristian Ehrhardt
298809f0800SChristian Ehrhardt#. Pass VF device to VM context (PCIe Passthrough):
299809f0800SChristian Ehrhardt
300809f0800SChristian Ehrhardt   The VF devices may be passed through to the guest VM using qemu or
301809f0800SChristian Ehrhardt   virt-manager or virsh etc.
302809f0800SChristian Ehrhardt   ``librte_pmd_thunderx_nicvf`` or ``thunder-nicvf`` should be used to bind
303809f0800SChristian Ehrhardt   the VF devices in the guest VM in :ref:`VFIO-NOIOMMU <thunderx_vfio_noiommu>` mode.
304809f0800SChristian Ehrhardt
305809f0800SChristian Ehrhardt   Example qemu guest launch command:
306809f0800SChristian Ehrhardt
307809f0800SChristian Ehrhardt   .. code-block:: console
308809f0800SChristian Ehrhardt
309809f0800SChristian Ehrhardt      sudo qemu-system-aarch64 -name vm1 \
310809f0800SChristian Ehrhardt      -machine virt,gic_version=3,accel=kvm,usb=off \
311809f0800SChristian Ehrhardt      -cpu host -m 4096 \
312809f0800SChristian Ehrhardt      -smp 4,sockets=1,cores=8,threads=1 \
313809f0800SChristian Ehrhardt      -nographic -nodefaults \
314809f0800SChristian Ehrhardt      -kernel <kernel image> \
315809f0800SChristian Ehrhardt      -append "root=/dev/vda console=ttyAMA0 rw hugepagesz=512M hugepages=3" \
316809f0800SChristian Ehrhardt      -device vfio-pci,host=0002:01:00.1 \
317809f0800SChristian Ehrhardt      -drive file=<rootfs.ext3>,if=none,id=disk1,format=raw  \
318809f0800SChristian Ehrhardt      -device virtio-blk-device,scsi=off,drive=disk1,id=virtio-disk1,bootindex=1 \
319809f0800SChristian Ehrhardt      -netdev tap,id=net0,ifname=tap0,script=/etc/qemu-ifup_thunder \
320809f0800SChristian Ehrhardt      -device virtio-net-device,netdev=net0 \
321809f0800SChristian Ehrhardt      -serial stdio \
322809f0800SChristian Ehrhardt      -mem-path /dev/huge
323809f0800SChristian Ehrhardt
324809f0800SChristian Ehrhardt#. Refer to section :ref:`Running testpmd <thunderx_testpmd_example>` for instruction
325809f0800SChristian Ehrhardt   how to launch ``testpmd`` application.
326809f0800SChristian Ehrhardt
3273d9b7210SChristian EhrhardtMultiple Queue Set per DPDK port configuration
3283d9b7210SChristian Ehrhardt~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
3293d9b7210SChristian Ehrhardt
3303d9b7210SChristian EhrhardtThere are two types of VFs:
3313d9b7210SChristian Ehrhardt
3323d9b7210SChristian Ehrhardt- Primary VF
3333d9b7210SChristian Ehrhardt- Secondary VF
3343d9b7210SChristian Ehrhardt
3353d9b7210SChristian EhrhardtEach port consists of a primary VF and n secondary VF(s). Each VF provides 8 Tx/Rx queues to a port.
3363d9b7210SChristian EhrhardtWhen a given port is configured to use more than 8 queues, it requires one (or more) secondary VF.
3373d9b7210SChristian EhrhardtEach secondary VF adds 8 additional queues to the queue set.
3383d9b7210SChristian Ehrhardt
3393d9b7210SChristian EhrhardtDuring PMD driver initialization, the primary VF's are enumerated by checking the
3403d9b7210SChristian Ehrhardtspecific flag (see sqs message in DPDK boot log - sqs indicates secondary queue set).
3413d9b7210SChristian EhrhardtThey are at the beginning of VF list (the remain ones are secondary VF's).
3423d9b7210SChristian Ehrhardt
3433d9b7210SChristian EhrhardtThe primary VFs are used as master queue sets. Secondary VFs provide
3443d9b7210SChristian Ehrhardtadditional queue sets for primary ones. If a port is configured for more then
3453d9b7210SChristian Ehrhardt8 queues than it will request for additional queues from secondary VFs.
3463d9b7210SChristian Ehrhardt
3473d9b7210SChristian EhrhardtSecondary VFs cannot be shared between primary VFs.
3483d9b7210SChristian Ehrhardt
3493d9b7210SChristian EhrhardtPrimary VFs are present on the beginning of the 'Network devices using kernel
3503d9b7210SChristian Ehrhardtdriver' list, secondary VFs are on the remaining on the remaining part of the list.
3513d9b7210SChristian Ehrhardt
3523d9b7210SChristian Ehrhardt   .. note::
3533d9b7210SChristian Ehrhardt
3543d9b7210SChristian Ehrhardt      The VNIC driver in the multiqueue setup works differently than other drivers like `ixgbe`.
3553d9b7210SChristian Ehrhardt      We need to bind separately each specific queue set device with the ``tools/dpdk-devbind.py`` utility.
3563d9b7210SChristian Ehrhardt
3573d9b7210SChristian Ehrhardt   .. note::
3583d9b7210SChristian Ehrhardt
3593d9b7210SChristian Ehrhardt      Depending on the hardware used, the kernel driver sets a threshold ``vf_id``. VFs that try to attached with an id below or equal to
3603d9b7210SChristian Ehrhardt      this boundary are considered primary VFs. VFs that try to attach with an id above this boundary are considered secondary VFs.
3613d9b7210SChristian Ehrhardt
3623d9b7210SChristian Ehrhardt
3633d9b7210SChristian EhrhardtExample device binding
3643d9b7210SChristian Ehrhardt~~~~~~~~~~~~~~~~~~~~~~
3653d9b7210SChristian Ehrhardt
3663d9b7210SChristian EhrhardtIf a system has three interfaces, a total of 18 VF devices will be created
3673d9b7210SChristian Ehrhardton a non-NUMA machine.
3683d9b7210SChristian Ehrhardt
3693d9b7210SChristian Ehrhardt   .. note::
3703d9b7210SChristian Ehrhardt
3713d9b7210SChristian Ehrhardt      NUMA systems have 12 VFs per port and non-NUMA 6 VFs per port.
3723d9b7210SChristian Ehrhardt
3733d9b7210SChristian Ehrhardt   .. code-block:: console
3743d9b7210SChristian Ehrhardt
3753d9b7210SChristian Ehrhardt      # tools/dpdk-devbind.py --status
3763d9b7210SChristian Ehrhardt
3773d9b7210SChristian Ehrhardt      Network devices using DPDK-compatible driver
3783d9b7210SChristian Ehrhardt      ============================================
3793d9b7210SChristian Ehrhardt      <none>
3803d9b7210SChristian Ehrhardt
3813d9b7210SChristian Ehrhardt      Network devices using kernel driver
3823d9b7210SChristian Ehrhardt      ===================================
3833d9b7210SChristian Ehrhardt      0000:01:10.0 'Device a026' if= drv=thunder-BGX unused=vfio-pci,uio_pci_generic
3843d9b7210SChristian Ehrhardt      0000:01:10.1 'Device a026' if= drv=thunder-BGX unused=vfio-pci,uio_pci_generic
3853d9b7210SChristian Ehrhardt      0002:01:00.0 'Device a01e' if= drv=thunder-nic unused=vfio-pci,uio_pci_generic
3863d9b7210SChristian Ehrhardt      0002:01:00.1 'Device 0011' if=eth0 drv=thunder-nicvf unused=vfio-pci,uio_pci_generic
3873d9b7210SChristian Ehrhardt      0002:01:00.2 'Device 0011' if=eth1 drv=thunder-nicvf unused=vfio-pci,uio_pci_generic
3883d9b7210SChristian Ehrhardt      0002:01:00.3 'Device 0011' if=eth2 drv=thunder-nicvf unused=vfio-pci,uio_pci_generic
3893d9b7210SChristian Ehrhardt      0002:01:00.4 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic
3903d9b7210SChristian Ehrhardt      0002:01:00.5 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic
3913d9b7210SChristian Ehrhardt      0002:01:00.6 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic
3923d9b7210SChristian Ehrhardt      0002:01:00.7 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic
3933d9b7210SChristian Ehrhardt      0002:01:01.0 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic
3943d9b7210SChristian Ehrhardt      0002:01:01.1 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic
3953d9b7210SChristian Ehrhardt      0002:01:01.2 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic
3963d9b7210SChristian Ehrhardt      0002:01:01.3 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic
3973d9b7210SChristian Ehrhardt      0002:01:01.4 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic
3983d9b7210SChristian Ehrhardt      0002:01:01.5 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic
3993d9b7210SChristian Ehrhardt      0002:01:01.6 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic
4003d9b7210SChristian Ehrhardt      0002:01:01.7 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic
4013d9b7210SChristian Ehrhardt      0002:01:02.0 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic
4023d9b7210SChristian Ehrhardt      0002:01:02.1 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic
4033d9b7210SChristian Ehrhardt      0002:01:02.2 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic
4043d9b7210SChristian Ehrhardt
4053d9b7210SChristian Ehrhardt      Other network devices
4063d9b7210SChristian Ehrhardt      =====================
4073d9b7210SChristian Ehrhardt      0002:00:03.0 'Device a01f' unused=vfio-pci,uio_pci_generic
4083d9b7210SChristian Ehrhardt
4093d9b7210SChristian Ehrhardt
4103d9b7210SChristian EhrhardtWe want to bind two physical interfaces with 24 queues each device, we attach two primary VFs
4113d9b7210SChristian Ehrhardtand four secondary queues. In our example we choose two 10G interfaces eth1 (0002:01:00.2) and eth2 (0002:01:00.3).
4123d9b7210SChristian EhrhardtWe will choose four secondary queue sets from the ending of the list (0002:01:01.7-0002:01:02.2).
4133d9b7210SChristian Ehrhardt
4143d9b7210SChristian Ehrhardt
4153d9b7210SChristian Ehrhardt#. Bind two primary VFs to the ``vfio-pci`` driver:
4163d9b7210SChristian Ehrhardt
4173d9b7210SChristian Ehrhardt   .. code-block:: console
4183d9b7210SChristian Ehrhardt
4193d9b7210SChristian Ehrhardt      tools/dpdk-devbind.py -b vfio-pci 0002:01:00.2
4203d9b7210SChristian Ehrhardt      tools/dpdk-devbind.py -b vfio-pci 0002:01:00.3
4213d9b7210SChristian Ehrhardt
4223d9b7210SChristian Ehrhardt#. Bind four primary VFs to the ``vfio-pci`` driver:
4233d9b7210SChristian Ehrhardt
4243d9b7210SChristian Ehrhardt   .. code-block:: console
4253d9b7210SChristian Ehrhardt
4263d9b7210SChristian Ehrhardt      tools/dpdk-devbind.py -b vfio-pci 0002:01:01.7
4273d9b7210SChristian Ehrhardt      tools/dpdk-devbind.py -b vfio-pci 0002:01:02.0
4283d9b7210SChristian Ehrhardt      tools/dpdk-devbind.py -b vfio-pci 0002:01:02.1
4293d9b7210SChristian Ehrhardt      tools/dpdk-devbind.py -b vfio-pci 0002:01:02.2
4303d9b7210SChristian Ehrhardt
4313d9b7210SChristian EhrhardtThe nicvf thunderx driver will make use of attached secondary VFs automatically during the interface configuration stage.
4323d9b7210SChristian Ehrhardt
433809f0800SChristian EhrhardtLimitations
434809f0800SChristian Ehrhardt-----------
435809f0800SChristian Ehrhardt
436809f0800SChristian EhrhardtCRC striping
437809f0800SChristian Ehrhardt~~~~~~~~~~~~
438809f0800SChristian Ehrhardt
439809f0800SChristian EhrhardtThe ThunderX SoC family NICs strip the CRC for every packets coming into the
440809f0800SChristian Ehrhardthost interface. So, CRC will be stripped even when the
441809f0800SChristian Ehrhardt``rxmode.hw_strip_crc`` member is set to 0 in ``struct rte_eth_conf``.
442809f0800SChristian Ehrhardt
443809f0800SChristian EhrhardtMaximum packet length
444809f0800SChristian Ehrhardt~~~~~~~~~~~~~~~~~~~~~
445809f0800SChristian Ehrhardt
446809f0800SChristian EhrhardtThe ThunderX SoC family NICs support a maximum of a 9K jumbo frame. The value
447809f0800SChristian Ehrhardtis fixed and cannot be changed. So, even when the ``rxmode.max_rx_pkt_len``
448809f0800SChristian Ehrhardtmember of ``struct rte_eth_conf`` is set to a value lower than 9200, frames
449809f0800SChristian Ehrhardtup to 9200 bytes can still reach the host interface.
450809f0800SChristian Ehrhardt
451809f0800SChristian EhrhardtMaximum packet segments
452809f0800SChristian Ehrhardt~~~~~~~~~~~~~~~~~~~~~~~
453809f0800SChristian Ehrhardt
454809f0800SChristian EhrhardtThe ThunderX SoC family NICs support up to 12 segments per packet when working
455809f0800SChristian Ehrhardtin scatter/gather mode. So, setting MTU will result with ``EINVAL`` when the
456809f0800SChristian Ehrhardtframe size does not fit in the maximum number of segments.
457