15129044dSC.J. Collier..  BSD LICENSE
25129044dSC.J. Collier    Copyright (c) 2015 QLogic Corporation
35129044dSC.J. Collier    All rights reserved.
45129044dSC.J. Collier
55129044dSC.J. Collier    Redistribution and use in source and binary forms, with or without
65129044dSC.J. Collier    modification, are permitted provided that the following conditions
75129044dSC.J. Collier    are met:
85129044dSC.J. Collier
95129044dSC.J. Collier    * Redistributions of source code must retain the above copyright
105129044dSC.J. Collier    notice, this list of conditions and the following disclaimer.
115129044dSC.J. Collier    * Redistributions in binary form must reproduce the above copyright
125129044dSC.J. Collier    notice, this list of conditions and the following disclaimer in
135129044dSC.J. Collier    the documentation and/or other materials provided with the
145129044dSC.J. Collier    distribution.
155129044dSC.J. Collier    * Neither the name of QLogic Corporation nor the names of its
165129044dSC.J. Collier    contributors may be used to endorse or promote products derived
175129044dSC.J. Collier    from this software without specific prior written permission.
185129044dSC.J. Collier
195129044dSC.J. Collier    THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
205129044dSC.J. Collier    "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
215129044dSC.J. Collier    LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
225129044dSC.J. Collier    A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
235129044dSC.J. Collier    OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
245129044dSC.J. Collier    SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
255129044dSC.J. Collier    LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
265129044dSC.J. Collier    DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
275129044dSC.J. Collier    THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
285129044dSC.J. Collier    (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
295129044dSC.J. Collier    OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
305129044dSC.J. Collier
315129044dSC.J. CollierBNX2X Poll Mode Driver
325129044dSC.J. Collier======================
335129044dSC.J. Collier
345129044dSC.J. CollierThe BNX2X poll mode driver library (**librte_pmd_bnx2x**) implements support
355129044dSC.J. Collierfor **QLogic 578xx** 10/20 Gbps family of adapters as well as their virtual
365129044dSC.J. Collierfunctions (VF) in SR-IOV context. It is supported on several standard Linux
375129044dSC.J. Collierdistros like Red Hat 7.x and SLES12 OS. It is compile-tested under FreeBSD OS.
385129044dSC.J. Collier
395129044dSC.J. CollierMore information can be found at `QLogic Corporation's Official Website
405129044dSC.J. Collier<http://www.qlogic.com>`_.
415129044dSC.J. Collier
425129044dSC.J. CollierSupported Features
435129044dSC.J. Collier------------------
445129044dSC.J. Collier
455129044dSC.J. CollierBNX2X PMD has support for:
465129044dSC.J. Collier
475129044dSC.J. Collier- Base L2 features
485129044dSC.J. Collier- Unicast/multicast filtering
495129044dSC.J. Collier- Promiscuous mode
505129044dSC.J. Collier- Port hardware statistics
515129044dSC.J. Collier- SR-IOV VF
525129044dSC.J. Collier
535129044dSC.J. CollierNon-supported Features
545129044dSC.J. Collier----------------------
555129044dSC.J. Collier
565129044dSC.J. CollierThe features not yet supported include:
575129044dSC.J. Collier
585129044dSC.J. Collier- TSS (Transmit Side Scaling)
595129044dSC.J. Collier- RSS (Receive Side Scaling)
605129044dSC.J. Collier- LRO/TSO offload
615129044dSC.J. Collier- Checksum offload
625129044dSC.J. Collier- SR-IOV PF
635129044dSC.J. Collier- Rx TX scatter gather
645129044dSC.J. Collier
655129044dSC.J. CollierCo-existence considerations
665129044dSC.J. Collier---------------------------
675129044dSC.J. Collier
685129044dSC.J. Collier- BCM578xx being a CNA can have both NIC and Storage personalities.
695129044dSC.J. Collier  However, coexistence with storage protocol drivers (cnic, bnx2fc and
705129044dSC.J. Collier  bnx2fi) is not supported on the same adapter. So storage personality
715129044dSC.J. Collier  has to be disabled on that adapter when used in DPDK applications.
725129044dSC.J. Collier
735129044dSC.J. Collier- For SR-IOV case, bnx2x PMD will be used to bind to SR-IOV VF device and
745129044dSC.J. Collier  Linux native kernel driver (bnx2x) will be attached to SR-IOV PF.
755129044dSC.J. Collier
765129044dSC.J. Collier
775129044dSC.J. CollierSupported QLogic NICs
785129044dSC.J. Collier---------------------
795129044dSC.J. Collier
805129044dSC.J. Collier- 578xx
815129044dSC.J. Collier
825129044dSC.J. CollierPrerequisites
835129044dSC.J. Collier-------------
845129044dSC.J. Collier
855129044dSC.J. Collier- Requires firmware version **7.2.51.0**. It is included in most of the
865129044dSC.J. Collier  standard Linux distros. If it is not available visit
875129044dSC.J. Collier  `QLogic Driver Download Center <http://driverdownloads.qlogic.com>`_
885129044dSC.J. Collier  to get the required firmware.
895129044dSC.J. Collier
905129044dSC.J. CollierPre-Installation Configuration
915129044dSC.J. Collier------------------------------
925129044dSC.J. Collier
935129044dSC.J. CollierConfig File Options
945129044dSC.J. Collier~~~~~~~~~~~~~~~~~~~
955129044dSC.J. Collier
965129044dSC.J. CollierThe following options can be modified in the ``.config`` file. Please note that
975129044dSC.J. Collierenabling debugging options may affect system performance.
985129044dSC.J. Collier
999365d6cfSChristian Ehrhardt- ``CONFIG_RTE_LIBRTE_BNX2X_PMD`` (default **n**)
1005129044dSC.J. Collier
1019365d6cfSChristian Ehrhardt  Toggle compilation of bnx2x driver. To use bnx2x PMD set this config parameter
1029365d6cfSChristian Ehrhardt  to 'y'. Also, in order for firmware binary to load user will need zlib devel
1039365d6cfSChristian Ehrhardt  package installed.
1045129044dSC.J. Collier
1055129044dSC.J. Collier- ``CONFIG_RTE_LIBRTE_BNX2X_DEBUG`` (default **n**)
1065129044dSC.J. Collier
1075129044dSC.J. Collier  Toggle display of generic debugging messages.
1085129044dSC.J. Collier
1095129044dSC.J. Collier- ``CONFIG_RTE_LIBRTE_BNX2X_DEBUG_INIT`` (default **n**)
1105129044dSC.J. Collier
1115129044dSC.J. Collier  Toggle display of initialization related messages.
1125129044dSC.J. Collier
1135129044dSC.J. Collier- ``CONFIG_RTE_LIBRTE_BNX2X_DEBUG_TX`` (default **n**)
1145129044dSC.J. Collier
1155129044dSC.J. Collier  Toggle display of transmit fast path run-time messages.
1165129044dSC.J. Collier
1175129044dSC.J. Collier- ``CONFIG_RTE_LIBRTE_BNX2X_DEBUG_RX`` (default **n**)
1185129044dSC.J. Collier
1195129044dSC.J. Collier  Toggle display of receive fast path run-time messages.
1205129044dSC.J. Collier
1215129044dSC.J. Collier- ``CONFIG_RTE_LIBRTE_BNX2X_DEBUG_PERIODIC`` (default **n**)
1225129044dSC.J. Collier
1235129044dSC.J. Collier  Toggle display of register reads and writes.
1245129044dSC.J. Collier
1255129044dSC.J. Collier
1265129044dSC.J. Collier.. _bnx2x_driver-compilation:
1275129044dSC.J. Collier
1285129044dSC.J. CollierDriver Compilation
1295129044dSC.J. Collier~~~~~~~~~~~~~~~~~~
1305129044dSC.J. Collier
1315129044dSC.J. CollierBNX2X PMD for Linux x86_64 gcc target, run the following "make"
1325129044dSC.J. Colliercommand::
1335129044dSC.J. Collier
1345129044dSC.J. Collier   cd <DPDK-source-directory>
1355129044dSC.J. Collier   make config T=x86_64-native-linuxapp-gcc install
1365129044dSC.J. Collier
1375129044dSC.J. CollierTo compile BNX2X PMD for Linux x86_64 clang target, run the following "make"
1385129044dSC.J. Colliercommand::
1395129044dSC.J. Collier
1405129044dSC.J. Collier   cd <DPDK-source-directory>
1415129044dSC.J. Collier   make config T=x86_64-native-linuxapp-clang install
1425129044dSC.J. Collier
1435129044dSC.J. CollierTo compile BNX2X PMD for Linux i686 gcc target, run the following "make"
1445129044dSC.J. Colliercommand::
1455129044dSC.J. Collier
1465129044dSC.J. Collier   cd <DPDK-source-directory>
1475129044dSC.J. Collier   make config T=i686-native-linuxapp-gcc install
1485129044dSC.J. Collier
1495129044dSC.J. CollierTo compile BNX2X PMD for Linux i686 gcc target, run the following "make"
1505129044dSC.J. Colliercommand:
1515129044dSC.J. Collier
1525129044dSC.J. Collier.. code-block:: console
1535129044dSC.J. Collier
1545129044dSC.J. Collier   cd <DPDK-source-directory>
1555129044dSC.J. Collier   make config T=i686-native-linuxapp-gcc install
1565129044dSC.J. Collier
1575129044dSC.J. CollierTo compile BNX2X PMD for FreeBSD x86_64 clang target, run the following "gmake"
1585129044dSC.J. Colliercommand::
1595129044dSC.J. Collier
1605129044dSC.J. Collier   cd <DPDK-source-directory>
1615129044dSC.J. Collier   gmake config T=x86_64-native-bsdapp-clang install
1625129044dSC.J. Collier
1635129044dSC.J. CollierTo compile BNX2X PMD for FreeBSD x86_64 gcc target, run the following "gmake"
1645129044dSC.J. Colliercommand::
1655129044dSC.J. Collier
1665129044dSC.J. Collier   cd <DPDK-source-directory>
1673d9b7210SChristian Ehrhardt   gmake config T=x86_64-native-bsdapp-gcc install -Wl,-rpath=/usr/local/lib/gcc49 CC=gcc49
1685129044dSC.J. Collier
1695129044dSC.J. CollierTo compile BNX2X PMD for FreeBSD x86_64 gcc target, run the following "gmake"
1705129044dSC.J. Colliercommand:
1715129044dSC.J. Collier
1725129044dSC.J. Collier.. code-block:: console
1735129044dSC.J. Collier
1745129044dSC.J. Collier   cd <DPDK-source-directory>
1753d9b7210SChristian Ehrhardt   gmake config T=x86_64-native-bsdapp-gcc install -Wl,-rpath=/usr/local/lib/gcc49 CC=gcc49
1765129044dSC.J. Collier
1775129044dSC.J. CollierLinux
1785129044dSC.J. Collier-----
1795129044dSC.J. Collier
1805129044dSC.J. Collier.. _bnx2x_Linux-installation:
1815129044dSC.J. Collier
1825129044dSC.J. CollierLinux Installation
1835129044dSC.J. Collier~~~~~~~~~~~~~~~~~~
1845129044dSC.J. Collier
1855129044dSC.J. CollierSample Application Notes
1865129044dSC.J. Collier~~~~~~~~~~~~~~~~~~~~~~~~
1875129044dSC.J. Collier
1885129044dSC.J. CollierThis section demonstrates how to launch ``testpmd`` with QLogic 578xx
1895129044dSC.J. Collierdevices managed by ``librte_pmd_bnx2x`` in Linux operating system.
1905129044dSC.J. Collier
1915129044dSC.J. Collier#. Request huge pages:
1925129044dSC.J. Collier
1935129044dSC.J. Collier   .. code-block:: console
1945129044dSC.J. Collier
1955129044dSC.J. Collier      echo 1024 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages/nr_hugepages
1965129044dSC.J. Collier
1975129044dSC.J. Collier#. Load ``igb_uio`` or ``vfio-pci`` driver:
1985129044dSC.J. Collier
1995129044dSC.J. Collier   .. code-block:: console
2005129044dSC.J. Collier
2015129044dSC.J. Collier      insmod ./x86_64-native-linuxapp-gcc/kmod/igb_uio.ko
2025129044dSC.J. Collier
2035129044dSC.J. Collier   or
2045129044dSC.J. Collier
2055129044dSC.J. Collier   .. code-block:: console
2065129044dSC.J. Collier
2075129044dSC.J. Collier      modprobe vfio-pci
2085129044dSC.J. Collier
2095129044dSC.J. Collier#. Bind the QLogic adapters to ``igb_uio`` or ``vfio-pci`` loaded in the
2105129044dSC.J. Collier   previous step::
2115129044dSC.J. Collier
2125b1ff351SRicardo Salveti      ./tools/dpdk-devbind.py --bind igb_uio 0000:84:00.0 0000:84:00.1
2135129044dSC.J. Collier
2145129044dSC.J. Collier   or
2155129044dSC.J. Collier
2165129044dSC.J. Collier   Setup VFIO permissions for regular users and then bind to ``vfio-pci``:
2175129044dSC.J. Collier
2185129044dSC.J. Collier   .. code-block:: console
2195129044dSC.J. Collier
2205129044dSC.J. Collier      sudo chmod a+x /dev/vfio
2215129044dSC.J. Collier
2225129044dSC.J. Collier      sudo chmod 0666 /dev/vfio/*
2235129044dSC.J. Collier
2245b1ff351SRicardo Salveti      ./tools/dpdk-devbind.py --bind vfio-pci 0000:84:00.0 0000:84:00.1
2255129044dSC.J. Collier
2265129044dSC.J. Collier#. Start ``testpmd`` with basic parameters:
2275129044dSC.J. Collier
2285129044dSC.J. Collier   .. code-block:: console
2295129044dSC.J. Collier
2305129044dSC.J. Collier      ./x86_64-native-linuxapp-gcc/app/testpmd -c 0xf -n 4 -- -i
2315129044dSC.J. Collier
2325129044dSC.J. Collier   Example output:
2335129044dSC.J. Collier
2345129044dSC.J. Collier   .. code-block:: console
2355129044dSC.J. Collier
2365129044dSC.J. Collier      [...]
2375129044dSC.J. Collier      EAL: PCI device 0000:84:00.0 on NUMA socket 1
2385129044dSC.J. Collier      EAL:   probe driver: 14e4:168e rte_bnx2x_pmd
2395129044dSC.J. Collier      EAL:   PCI memory mapped at 0x7f14f6fe5000
2405129044dSC.J. Collier      EAL:   PCI memory mapped at 0x7f14f67e5000
2415129044dSC.J. Collier      EAL:   PCI memory mapped at 0x7f15fbd9b000
2425129044dSC.J. Collier      EAL: PCI device 0000:84:00.1 on NUMA socket 1
2435129044dSC.J. Collier      EAL:   probe driver: 14e4:168e rte_bnx2x_pmd
2445129044dSC.J. Collier      EAL:   PCI memory mapped at 0x7f14f5fe5000
2455129044dSC.J. Collier      EAL:   PCI memory mapped at 0x7f14f57e5000
2465129044dSC.J. Collier      EAL:   PCI memory mapped at 0x7f15fbd4f000
2475129044dSC.J. Collier      Interactive-mode selected
2485129044dSC.J. Collier      Configuring Port 0 (socket 0)
2495129044dSC.J. Collier      PMD: bnx2x_dev_tx_queue_setup(): fp[00] req_bd=512, thresh=512,
2505129044dSC.J. Collier                   usable_bd=1020, total_bd=1024,
2515129044dSC.J. Collier                                tx_pages=4
2525129044dSC.J. Collier      PMD: bnx2x_dev_rx_queue_setup(): fp[00] req_bd=128, thresh=0,
2535129044dSC.J. Collier                   usable_bd=510, total_bd=512,
2545129044dSC.J. Collier                                rx_pages=1, cq_pages=8
2555129044dSC.J. Collier      PMD: bnx2x_print_adapter_info():
2565129044dSC.J. Collier      [...]
2575129044dSC.J. Collier      Checking link statuses...
2585129044dSC.J. Collier      Port 0 Link Up - speed 10000 Mbps - full-duplex
2595129044dSC.J. Collier      Port 1 Link Up - speed 10000 Mbps - full-duplex
2605129044dSC.J. Collier      Done
2615129044dSC.J. Collier      testpmd>
2625129044dSC.J. Collier
2635129044dSC.J. CollierSR-IOV: Prerequisites and sample Application Notes
2645129044dSC.J. Collier~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2655129044dSC.J. Collier
2665129044dSC.J. CollierThis section provides instructions to configure SR-IOV with Linux OS.
2675129044dSC.J. Collier
2685129044dSC.J. Collier#. Verify SR-IOV and ARI capabilities are enabled on the adapter using ``lspci``:
2695129044dSC.J. Collier
2705129044dSC.J. Collier   .. code-block:: console
2715129044dSC.J. Collier
2725129044dSC.J. Collier      lspci -s <slot> -vvv
2735129044dSC.J. Collier
2745129044dSC.J. Collier   Example output:
2755129044dSC.J. Collier
2765129044dSC.J. Collier   .. code-block:: console
2775129044dSC.J. Collier
2785129044dSC.J. Collier      [...]
2795129044dSC.J. Collier      Capabilities: [1b8 v1] Alternative Routing-ID Interpretation (ARI)
2805129044dSC.J. Collier      [...]
2815129044dSC.J. Collier      Capabilities: [1c0 v1] Single Root I/O Virtualization (SR-IOV)
2825129044dSC.J. Collier      [...]
2835129044dSC.J. Collier      Kernel driver in use: igb_uio
2845129044dSC.J. Collier
2855129044dSC.J. Collier#. Load the kernel module:
2865129044dSC.J. Collier
2875129044dSC.J. Collier   .. code-block:: console
2885129044dSC.J. Collier
2895129044dSC.J. Collier      modprobe bnx2x
2905129044dSC.J. Collier
2915129044dSC.J. Collier   Example output:
2925129044dSC.J. Collier
2935129044dSC.J. Collier   .. code-block:: console
2945129044dSC.J. Collier
2955129044dSC.J. Collier      systemd-udevd[4848]: renamed network interface eth0 to ens5f0
2965129044dSC.J. Collier      systemd-udevd[4848]: renamed network interface eth1 to ens5f1
2975129044dSC.J. Collier
2985129044dSC.J. Collier#. Bring up the PF ports:
2995129044dSC.J. Collier
3005129044dSC.J. Collier   .. code-block:: console
3015129044dSC.J. Collier
3025129044dSC.J. Collier      ifconfig ens5f0 up
3035129044dSC.J. Collier      ifconfig ens5f1 up
3045129044dSC.J. Collier
3055129044dSC.J. Collier#. Create VF device(s):
3065129044dSC.J. Collier
3075129044dSC.J. Collier   Echo the number of VFs to be created into "sriov_numvfs" sysfs entry
3085129044dSC.J. Collier   of the parent PF.
3095129044dSC.J. Collier
3105129044dSC.J. Collier   Example output:
3115129044dSC.J. Collier
3125129044dSC.J. Collier   .. code-block:: console
3135129044dSC.J. Collier
3145129044dSC.J. Collier      echo 2 > /sys/devices/pci0000:00/0000:00:03.0/0000:81:00.0/sriov_numvfs
3155129044dSC.J. Collier
3165129044dSC.J. Collier
3175129044dSC.J. Collier#. Assign VF MAC address:
3185129044dSC.J. Collier
3195129044dSC.J. Collier   Assign MAC address to the VF using iproute2 utility. The syntax is:
3205129044dSC.J. Collier   ip link set <PF iface> vf <VF id> mac <macaddr>
3215129044dSC.J. Collier
3225129044dSC.J. Collier   Example output:
3235129044dSC.J. Collier
3245129044dSC.J. Collier   .. code-block:: console
3255129044dSC.J. Collier
3265129044dSC.J. Collier      ip link set ens5f0 vf 0 mac 52:54:00:2f:9d:e8
3275129044dSC.J. Collier
3285129044dSC.J. Collier
3295129044dSC.J. Collier#. PCI Passthrough:
3305129044dSC.J. Collier
3315129044dSC.J. Collier   The VF devices may be passed through to the guest VM using virt-manager or
3325129044dSC.J. Collier   virsh etc. bnx2x PMD should be used to bind the VF devices in the guest VM
3335129044dSC.J. Collier   using the instructions outlined in the Application notes below.
334