15129044dSC.J. Collier..  BSD LICENSE
25129044dSC.J. Collier    Copyright 2015 Chelsio Communications.
35129044dSC.J. Collier    All rights reserved.
45129044dSC.J. Collier
55129044dSC.J. Collier    Redistribution and use in source and binary forms, with or without
65129044dSC.J. Collier    modification, are permitted provided that the following conditions
75129044dSC.J. Collier    are met:
85129044dSC.J. Collier
95129044dSC.J. Collier    * Redistributions of source code must retain the above copyright
105129044dSC.J. Collier    notice, this list of conditions and the following disclaimer.
115129044dSC.J. Collier    * Redistributions in binary form must reproduce the above copyright
125129044dSC.J. Collier    notice, this list of conditions and the following disclaimer in
135129044dSC.J. Collier    the documentation and/or other materials provided with the
145129044dSC.J. Collier    distribution.
155129044dSC.J. Collier    * Neither the name of Chelsio Communications nor the names of its
165129044dSC.J. Collier    contributors may be used to endorse or promote products derived
175129044dSC.J. Collier    from this software without specific prior written permission.
185129044dSC.J. Collier
305129044dSC.J. Collier
315129044dSC.J. CollierCXGBE Poll Mode Driver
325129044dSC.J. Collier======================
335129044dSC.J. Collier
345129044dSC.J. CollierThe CXGBE PMD (**librte_pmd_cxgbe**) provides poll mode driver support
355129044dSC.J. Collierfor **Chelsio T5** 10/40 Gbps family of adapters. CXGBE PMD has support
365129044dSC.J. Collierfor the latest Linux and FreeBSD operating systems.
375129044dSC.J. Collier
385129044dSC.J. CollierMore information can be found at `Chelsio Communications Official Website
395129044dSC.J. Collier<http://www.chelsio.com>`_.
405129044dSC.J. Collier
415129044dSC.J. CollierFeatures
425129044dSC.J. Collier--------
435129044dSC.J. Collier
445129044dSC.J. CollierCXGBE PMD has support for:
455129044dSC.J. Collier
465129044dSC.J. Collier- Multiple queues for TX and RX
475129044dSC.J. Collier- Receiver Side Steering (RSS)
485129044dSC.J. Collier- VLAN filtering
495129044dSC.J. Collier- Checksum offload
505129044dSC.J. Collier- Promiscuous mode
515129044dSC.J. Collier- All multicast mode
525129044dSC.J. Collier- Port hardware statistics
535129044dSC.J. Collier- Jumbo frames
545129044dSC.J. Collier
555129044dSC.J. CollierLimitations
565129044dSC.J. Collier-----------
575129044dSC.J. Collier
585129044dSC.J. CollierThe Chelsio T5 devices provide two/four ports but expose a single PCI bus
595129044dSC.J. Collieraddress, thus, librte_pmd_cxgbe registers itself as a
605129044dSC.J. CollierPCI driver that allocates one Ethernet device per detected port.
615129044dSC.J. Collier
625129044dSC.J. CollierFor this reason, one cannot whitelist/blacklist a single port without
635129044dSC.J. Collierwhitelisting/blacklisting the other ports on the same device.
645129044dSC.J. Collier
655129044dSC.J. CollierSupported Chelsio T5 NICs
665129044dSC.J. Collier-------------------------
675129044dSC.J. Collier
685129044dSC.J. Collier- 1G NICs: T502-BT
695129044dSC.J. Collier- 10G NICs: T520-BT, T520-CR, T520-LL-CR, T520-SO-CR, T540-CR
705129044dSC.J. Collier- 40G NICs: T580-CR, T580-LP-CR, T580-SO-CR
715129044dSC.J. Collier- Other T5 NICs: T522-CR
725129044dSC.J. Collier
735129044dSC.J. CollierPrerequisites
745129044dSC.J. Collier-------------
755129044dSC.J. Collier
765129044dSC.J. Collier- Requires firmware version **** and higher. Visit
775129044dSC.J. Collier  `Chelsio Download Center <http://service.chelsio.com>`_ to get latest firmware
785129044dSC.J. Collier  bundled with the latest Chelsio Unified Wire package.
795129044dSC.J. Collier
805129044dSC.J. Collier  For Linux, installing and loading the latest cxgb4 kernel driver from the
815129044dSC.J. Collier  Chelsio Unified Wire package should get you the latest firmware. More
825129044dSC.J. Collier  information can be obtained from the User Guide that is bundled with the
835129044dSC.J. Collier  Chelsio Unified Wire package.
845129044dSC.J. Collier
855129044dSC.J. Collier  For FreeBSD, the latest firmware obtained from the Chelsio Unified Wire
865129044dSC.J. Collier  package must be manually flashed via cxgbetool available in FreeBSD source
875129044dSC.J. Collier  repository.
885129044dSC.J. Collier
895129044dSC.J. Collier  Instructions on how to manually flash the firmware are given in section
905129044dSC.J. Collier  :ref:`linux-installation` for Linux and section :ref:`freebsd-installation`
915129044dSC.J. Collier  for FreeBSD.
925129044dSC.J. Collier
935129044dSC.J. CollierPre-Installation Configuration
945129044dSC.J. Collier------------------------------
955129044dSC.J. Collier
965129044dSC.J. CollierConfig File Options
975129044dSC.J. Collier~~~~~~~~~~~~~~~~~~~
985129044dSC.J. Collier
995129044dSC.J. CollierThe following options can be modified in the ``.config`` file. Please note that
1005129044dSC.J. Collierenabling debugging options may affect system performance.
1015129044dSC.J. Collier
1025129044dSC.J. Collier- ``CONFIG_RTE_LIBRTE_CXGBE_PMD`` (default **y**)
1035129044dSC.J. Collier
1045129044dSC.J. Collier  Toggle compilation of librte_pmd_cxgbe driver.
1055129044dSC.J. Collier
1065129044dSC.J. Collier- ``CONFIG_RTE_LIBRTE_CXGBE_DEBUG`` (default **n**)
1075129044dSC.J. Collier
1085129044dSC.J. Collier  Toggle display of generic debugging messages.
1095129044dSC.J. Collier
1105129044dSC.J. Collier- ``CONFIG_RTE_LIBRTE_CXGBE_DEBUG_REG`` (default **n**)
1115129044dSC.J. Collier
1125129044dSC.J. Collier  Toggle display of registers related run-time check messages.
1135129044dSC.J. Collier
1145129044dSC.J. Collier- ``CONFIG_RTE_LIBRTE_CXGBE_DEBUG_MBOX`` (default **n**)
1155129044dSC.J. Collier
1165129044dSC.J. Collier  Toggle display of firmware mailbox related run-time check messages.
1175129044dSC.J. Collier
1185129044dSC.J. Collier- ``CONFIG_RTE_LIBRTE_CXGBE_DEBUG_TX`` (default **n**)
1195129044dSC.J. Collier
1205129044dSC.J. Collier  Toggle display of transmission data path run-time check messages.
1215129044dSC.J. Collier
1225129044dSC.J. Collier- ``CONFIG_RTE_LIBRTE_CXGBE_DEBUG_RX`` (default **n**)
1235129044dSC.J. Collier
1245129044dSC.J. Collier  Toggle display of receiving data path run-time check messages.
1255129044dSC.J. Collier
1265129044dSC.J. Collier.. _driver-compilation:
1275129044dSC.J. Collier
1285129044dSC.J. CollierDriver Compilation
1295129044dSC.J. Collier~~~~~~~~~~~~~~~~~~
1305129044dSC.J. Collier
1315129044dSC.J. CollierTo compile CXGBE PMD for Linux x86_64 gcc target, run the following "make"
1325129044dSC.J. Colliercommand:
1335129044dSC.J. Collier
1345129044dSC.J. Collier.. code-block:: console
1355129044dSC.J. Collier
1365129044dSC.J. Collier   cd <DPDK-source-directory>
1375129044dSC.J. Collier   make config T=x86_64-native-linuxapp-gcc install
1385129044dSC.J. Collier
1395129044dSC.J. CollierTo compile CXGBE PMD for FreeBSD x86_64 clang target, run the following "gmake"
1405129044dSC.J. Colliercommand:
1415129044dSC.J. Collier
1425129044dSC.J. Collier.. code-block:: console
1435129044dSC.J. Collier
1445129044dSC.J. Collier   cd <DPDK-source-directory>
1455129044dSC.J. Collier   gmake config T=x86_64-native-bsdapp-clang install
1465129044dSC.J. Collier
1475129044dSC.J. CollierLinux
1485129044dSC.J. Collier-----
1495129044dSC.J. Collier
1505129044dSC.J. Collier.. _linux-installation:
1515129044dSC.J. Collier
1525129044dSC.J. CollierLinux Installation
1535129044dSC.J. Collier~~~~~~~~~~~~~~~~~~
1545129044dSC.J. Collier
1555129044dSC.J. CollierSteps to manually install the latest firmware from the downloaded Chelsio
1565129044dSC.J. CollierUnified Wire package for Linux operating system are as follows:
1575129044dSC.J. Collier
1585129044dSC.J. Collier#. Load the kernel module:
1595129044dSC.J. Collier
1605129044dSC.J. Collier   .. code-block:: console
1615129044dSC.J. Collier
1625129044dSC.J. Collier      modprobe cxgb4
1635129044dSC.J. Collier
1645129044dSC.J. Collier#. Use ifconfig to get the interface name assigned to Chelsio card:
1655129044dSC.J. Collier
1665129044dSC.J. Collier   .. code-block:: console
1675129044dSC.J. Collier
1685129044dSC.J. Collier      ifconfig -a | grep "00:07:43"
1695129044dSC.J. Collier
1705129044dSC.J. Collier   Example output:
1715129044dSC.J. Collier
1725129044dSC.J. Collier   .. code-block:: console
1735129044dSC.J. Collier
1745129044dSC.J. Collier      p1p1      Link encap:Ethernet  HWaddr 00:07:43:2D:EA:C0
1755129044dSC.J. Collier      p1p2      Link encap:Ethernet  HWaddr 00:07:43:2D:EA:C8
1765129044dSC.J. Collier
1775129044dSC.J. Collier#. Install cxgbtool:
1785129044dSC.J. Collier
1795129044dSC.J. Collier   .. code-block:: console
1805129044dSC.J. Collier
1815129044dSC.J. Collier      cd <path_to_uwire>/tools/cxgbtool
1825129044dSC.J. Collier      make install
1835129044dSC.J. Collier
1845129044dSC.J. Collier#. Use cxgbtool to load the firmware config file onto the card:
1855129044dSC.J. Collier
1865129044dSC.J. Collier   .. code-block:: console
1875129044dSC.J. Collier
1885129044dSC.J. Collier      cxgbtool p1p1 loadcfg <path_to_uwire>/src/network/firmware/t5-config.txt
1895129044dSC.J. Collier
1905129044dSC.J. Collier#. Use cxgbtool to load the firmware image onto the card:
1915129044dSC.J. Collier
1925129044dSC.J. Collier   .. code-block:: console
1935129044dSC.J. Collier
1945129044dSC.J. Collier      cxgbtool p1p1 loadfw <path_to_uwire>/src/network/firmware/t5fw-*.bin
1955129044dSC.J. Collier
1965129044dSC.J. Collier#. Unload and reload the kernel module:
1975129044dSC.J. Collier
1985129044dSC.J. Collier   .. code-block:: console
1995129044dSC.J. Collier
2005129044dSC.J. Collier      modprobe -r cxgb4
2015129044dSC.J. Collier      modprobe cxgb4
2025129044dSC.J. Collier
2035129044dSC.J. Collier#. Verify with ethtool:
2045129044dSC.J. Collier
2055129044dSC.J. Collier   .. code-block:: console
2065129044dSC.J. Collier
2075129044dSC.J. Collier      ethtool -i p1p1 | grep "firmware"
2085129044dSC.J. Collier
2095129044dSC.J. Collier   Example output:
2105129044dSC.J. Collier
2115129044dSC.J. Collier   .. code-block:: console
2125129044dSC.J. Collier
2135129044dSC.J. Collier      firmware-version:, TP
2145129044dSC.J. Collier
2155129044dSC.J. CollierRunning testpmd
2165129044dSC.J. Collier~~~~~~~~~~~~~~~
2175129044dSC.J. Collier
2185129044dSC.J. CollierThis section demonstrates how to launch **testpmd** with Chelsio T5
2195129044dSC.J. Collierdevices managed by librte_pmd_cxgbe in Linux operating system.
2205129044dSC.J. Collier
2215129044dSC.J. Collier#. Change to DPDK source directory where the target has been compiled in
2225129044dSC.J. Collier   section :ref:`driver-compilation`:
2235129044dSC.J. Collier
2245129044dSC.J. Collier   .. code-block:: console
2255129044dSC.J. Collier
2265129044dSC.J. Collier      cd <DPDK-source-directory>
2275129044dSC.J. Collier
2285129044dSC.J. Collier#. Load the kernel module:
2295129044dSC.J. Collier
2305129044dSC.J. Collier   .. code-block:: console
2315129044dSC.J. Collier
2325129044dSC.J. Collier      modprobe cxgb4
2335129044dSC.J. Collier
2345129044dSC.J. Collier#. Get the PCI bus addresses of the interfaces bound to cxgb4 driver:
2355129044dSC.J. Collier
2365129044dSC.J. Collier   .. code-block:: console
2375129044dSC.J. Collier
2385129044dSC.J. Collier      dmesg | tail -2
2395129044dSC.J. Collier
2405129044dSC.J. Collier   Example output:
2415129044dSC.J. Collier
2425129044dSC.J. Collier   .. code-block:: console
2435129044dSC.J. Collier
2445129044dSC.J. Collier      cxgb4 0000:02:00.4 p1p1: renamed from eth0
2455129044dSC.J. Collier      cxgb4 0000:02:00.4 p1p2: renamed from eth1
2465129044dSC.J. Collier
2475129044dSC.J. Collier   .. note::
2485129044dSC.J. Collier
2495129044dSC.J. Collier      Both the interfaces of a Chelsio T5 2-port adapter are bound to the
2505129044dSC.J. Collier      same PCI bus address.
2515129044dSC.J. Collier
2525129044dSC.J. Collier#. Unload the kernel module:
2535129044dSC.J. Collier
2545129044dSC.J. Collier   .. code-block:: console
2555129044dSC.J. Collier
2565129044dSC.J. Collier      modprobe -ar cxgb4 csiostor
2575129044dSC.J. Collier
2585129044dSC.J. Collier#. Request huge pages:
2595129044dSC.J. Collier
2605129044dSC.J. Collier   .. code-block:: console
2615129044dSC.J. Collier
2625129044dSC.J. Collier      echo 1024 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages/nr_hugepages
2635129044dSC.J. Collier
2645129044dSC.J. Collier#. Mount huge pages:
2655129044dSC.J. Collier
2665129044dSC.J. Collier   .. code-block:: console
2675129044dSC.J. Collier
2685129044dSC.J. Collier      mkdir /mnt/huge
2695129044dSC.J. Collier      mount -t hugetlbfs nodev /mnt/huge
2705129044dSC.J. Collier
2715129044dSC.J. Collier#. Load igb_uio or vfio-pci driver:
2725129044dSC.J. Collier
2735129044dSC.J. Collier   .. code-block:: console
2745129044dSC.J. Collier
2755129044dSC.J. Collier      insmod ./x86_64-native-linuxapp-gcc/kmod/igb_uio.ko
2765129044dSC.J. Collier
2775129044dSC.J. Collier   or
2785129044dSC.J. Collier
2795129044dSC.J. Collier   .. code-block:: console
2805129044dSC.J. Collier
2815129044dSC.J. Collier      modprobe vfio-pci
2825129044dSC.J. Collier
2835129044dSC.J. Collier#. Bind the Chelsio T5 adapters to igb_uio or vfio-pci loaded in the previous
2845129044dSC.J. Collier   step:
2855129044dSC.J. Collier
2865129044dSC.J. Collier   .. code-block:: console
2875129044dSC.J. Collier
2885b1ff351SRicardo Salveti      ./tools/dpdk-devbind.py --bind igb_uio 0000:02:00.4
2895129044dSC.J. Collier
2905129044dSC.J. Collier   or
2915129044dSC.J. Collier
2925129044dSC.J. Collier   Setup VFIO permissions for regular users and then bind to vfio-pci:
2935129044dSC.J. Collier
2945129044dSC.J. Collier   .. code-block:: console
2955129044dSC.J. Collier
2965129044dSC.J. Collier      sudo chmod a+x /dev/vfio
2975129044dSC.J. Collier
2985129044dSC.J. Collier      sudo chmod 0666 /dev/vfio/*
2995129044dSC.J. Collier
3005b1ff351SRicardo Salveti      ./tools/dpdk-devbind.py --bind vfio-pci 0000:02:00.4
3015129044dSC.J. Collier
3025129044dSC.J. Collier   .. note::
3035129044dSC.J. Collier
3045129044dSC.J. Collier      Currently, CXGBE PMD only supports the binding of PF4 for Chelsio T5 NICs.
3055129044dSC.J. Collier
3065129044dSC.J. Collier#. Start testpmd with basic parameters:
3075129044dSC.J. Collier
3085129044dSC.J. Collier   .. code-block:: console
3095129044dSC.J. Collier
3105129044dSC.J. Collier      ./x86_64-native-linuxapp-gcc/app/testpmd -c 0xf -n 4 -w 0000:02:00.4 -- -i
3115129044dSC.J. Collier
3125129044dSC.J. Collier   Example output:
3135129044dSC.J. Collier
3145129044dSC.J. Collier   .. code-block:: console
3155129044dSC.J. Collier
3165129044dSC.J. Collier      [...]
3175129044dSC.J. Collier      EAL: PCI device 0000:02:00.4 on NUMA socket -1
3185129044dSC.J. Collier      EAL:   probe driver: 1425:5401 rte_cxgbe_pmd
3195129044dSC.J. Collier      EAL:   PCI memory mapped at 0x7fd7c0200000
3205129044dSC.J. Collier      EAL:   PCI memory mapped at 0x7fd77cdfd000
3215129044dSC.J. Collier      EAL:   PCI memory mapped at 0x7fd7c10b7000
3225129044dSC.J. Collier      PMD: rte_cxgbe_pmd: fw:, TP:
3235129044dSC.J. Collier      PMD: rte_cxgbe_pmd: Coming up as MASTER: Initializing adapter
3245129044dSC.J. Collier      Interactive-mode selected
3255129044dSC.J. Collier      Configuring Port 0 (socket 0)
3265129044dSC.J. Collier      Port 0: 00:07:43:2D:EA:C0
3275129044dSC.J. Collier      Configuring Port 1 (socket 0)
3285129044dSC.J. Collier      Port 1: 00:07:43:2D:EA:C8
3295129044dSC.J. Collier      Checking link statuses...
3305129044dSC.J. Collier      PMD: rte_cxgbe_pmd: Port0: passive DA port module inserted
3315129044dSC.J. Collier      PMD: rte_cxgbe_pmd: Port1: passive DA port module inserted
3325129044dSC.J. Collier      Port 0 Link Up - speed 10000 Mbps - full-duplex
3335129044dSC.J. Collier      Port 1 Link Up - speed 10000 Mbps - full-duplex
3345129044dSC.J. Collier      Done
3355129044dSC.J. Collier      testpmd>
3365129044dSC.J. Collier
3375129044dSC.J. Collier.. note::
3385129044dSC.J. Collier
3395129044dSC.J. Collier   Flow control pause TX/RX is disabled by default and can be enabled via
3405129044dSC.J. Collier   testpmd. Refer section :ref:`flow-control` for more details.
3415129044dSC.J. Collier
3425129044dSC.J. CollierFreeBSD
3435129044dSC.J. Collier-------
3445129044dSC.J. Collier
3455129044dSC.J. Collier.. _freebsd-installation:
3465129044dSC.J. Collier
3475129044dSC.J. CollierFreeBSD Installation
3485129044dSC.J. Collier~~~~~~~~~~~~~~~~~~~~
3495129044dSC.J. Collier
3505129044dSC.J. CollierSteps to manually install the latest firmware from the downloaded Chelsio
3515129044dSC.J. CollierUnified Wire package for FreeBSD operating system are as follows:
3525129044dSC.J. Collier
3535129044dSC.J. Collier#. Load the kernel module:
3545129044dSC.J. Collier
3555129044dSC.J. Collier   .. code-block:: console
3565129044dSC.J. Collier
3575129044dSC.J. Collier      kldload if_cxgbe
3585129044dSC.J. Collier
3595129044dSC.J. Collier#. Use dmesg to get the t5nex instance assigned to the Chelsio card:
3605129044dSC.J. Collier
3615129044dSC.J. Collier   .. code-block:: console
3625129044dSC.J. Collier
3635129044dSC.J. Collier      dmesg | grep "t5nex"
3645129044dSC.J. Collier
3655129044dSC.J. Collier   Example output:
3665129044dSC.J. Collier
3675129044dSC.J. Collier   .. code-block:: console
3685129044dSC.J. Collier
3695129044dSC.J. Collier      t5nex0: <Chelsio T520-CR> irq 16 at device 0.4 on pci2
3705129044dSC.J. Collier      cxl0: <port 0> on t5nex0
3715129044dSC.J. Collier      cxl1: <port 1> on t5nex0
3725129044dSC.J. Collier      t5nex0: PCIe x8, 2 ports, 14 MSI-X interrupts, 31 eq, 13 iq
3735129044dSC.J. Collier
3745129044dSC.J. Collier   In the example above, a Chelsio T520-CR card is bound to a t5nex0 instance.
3755129044dSC.J. Collier
3765129044dSC.J. Collier#. Install cxgbetool from FreeBSD source repository:
3775129044dSC.J. Collier
3785129044dSC.J. Collier   .. code-block:: console
3795129044dSC.J. Collier
3805129044dSC.J. Collier      cd <path_to_FreeBSD_source>/tools/tools/cxgbetool/
3815129044dSC.J. Collier      make && make install
3825129044dSC.J. Collier
3835129044dSC.J. Collier#. Use cxgbetool to load the firmware image onto the card:
3845129044dSC.J. Collier
3855129044dSC.J. Collier   .. code-block:: console
3865129044dSC.J. Collier
3875129044dSC.J. Collier      cxgbetool t5nex0 loadfw <path_to_uwire>/src/network/firmware/t5fw-*.bin
3885129044dSC.J. Collier
3895129044dSC.J. Collier#. Unload and reload the kernel module:
3905129044dSC.J. Collier
3915129044dSC.J. Collier   .. code-block:: console
3925129044dSC.J. Collier
3935129044dSC.J. Collier      kldunload if_cxgbe
3945129044dSC.J. Collier      kldload if_cxgbe
3955129044dSC.J. Collier
3965129044dSC.J. Collier#. Verify with sysctl:
3975129044dSC.J. Collier
3985129044dSC.J. Collier   .. code-block:: console
3995129044dSC.J. Collier
4005129044dSC.J. Collier      sysctl -a | grep "t5nex" | grep "firmware"
4015129044dSC.J. Collier
4025129044dSC.J. Collier   Example output:
4035129044dSC.J. Collier
4045129044dSC.J. Collier   .. code-block:: console
4055129044dSC.J. Collier
4065129044dSC.J. Collier      dev.t5nex.0.firmware_version:
4075129044dSC.J. Collier
4085129044dSC.J. CollierRunning testpmd
4095129044dSC.J. Collier~~~~~~~~~~~~~~~
4105129044dSC.J. Collier
4115129044dSC.J. CollierThis section demonstrates how to launch **testpmd** with Chelsio T5
4125129044dSC.J. Collierdevices managed by librte_pmd_cxgbe in FreeBSD operating system.
4135129044dSC.J. Collier
4145129044dSC.J. Collier#. Change to DPDK source directory where the target has been compiled in
4155129044dSC.J. Collier   section :ref:`driver-compilation`:
4165129044dSC.J. Collier
4175129044dSC.J. Collier   .. code-block:: console
4185129044dSC.J. Collier
4195129044dSC.J. Collier      cd <DPDK-source-directory>
4205129044dSC.J. Collier
4215129044dSC.J. Collier#. Copy the contigmem kernel module to /boot/kernel directory:
4225129044dSC.J. Collier
4235129044dSC.J. Collier   .. code-block:: console
4245129044dSC.J. Collier
4255129044dSC.J. Collier      cp x86_64-native-bsdapp-clang/kmod/contigmem.ko /boot/kernel/
4265129044dSC.J. Collier
4275129044dSC.J. Collier#. Add the following lines to /boot/loader.conf:
4285129044dSC.J. Collier
4295129044dSC.J. Collier   .. code-block:: console
4305129044dSC.J. Collier
4315129044dSC.J. Collier      # reserve 2 x 1G blocks of contiguous memory using contigmem driver
4325129044dSC.J. Collier      hw.contigmem.num_buffers=2
4335129044dSC.J. Collier      hw.contigmem.buffer_size=1073741824
4345129044dSC.J. Collier      # load contigmem module during boot process
4355129044dSC.J. Collier      contigmem_load="YES"
4365129044dSC.J. Collier
4375129044dSC.J. Collier   The above lines load the contigmem kernel module during boot process and
4385129044dSC.J. Collier   allocate 2 x 1G blocks of contiguous memory to be used for DPDK later on.
4395129044dSC.J. Collier   This is to avoid issues with potential memory fragmentation during later
4405129044dSC.J. Collier   system up time, which may result in failure of allocating the contiguous
4415129044dSC.J. Collier   memory required for the contigmem kernel module.
4425129044dSC.J. Collier
4435129044dSC.J. Collier#. Restart the system and ensure the contigmem module is loaded successfully:
4445129044dSC.J. Collier
4455129044dSC.J. Collier   .. code-block:: console
4465129044dSC.J. Collier
4475129044dSC.J. Collier      reboot
4485129044dSC.J. Collier      kldstat | grep "contigmem"
4495129044dSC.J. Collier
4505129044dSC.J. Collier   Example output:
4515129044dSC.J. Collier
4525129044dSC.J. Collier   .. code-block:: console
4535129044dSC.J. Collier
4545129044dSC.J. Collier      2    1 0xffffffff817f1000 3118     contigmem.ko
4555129044dSC.J. Collier
4565129044dSC.J. Collier#. Repeat step 1 to ensure that you are in the DPDK source directory.
4575129044dSC.J. Collier
4585129044dSC.J. Collier#. Load the cxgbe kernel module:
4595129044dSC.J. Collier
4605129044dSC.J. Collier   .. code-block:: console
4615129044dSC.J. Collier
4625129044dSC.J. Collier      kldload if_cxgbe
4635129044dSC.J. Collier
4645129044dSC.J. Collier#. Get the PCI bus addresses of the interfaces bound to t5nex driver:
4655129044dSC.J. Collier
4665129044dSC.J. Collier   .. code-block:: console
4675129044dSC.J. Collier
4685129044dSC.J. Collier      pciconf -l | grep "t5nex"
4695129044dSC.J. Collier
4705129044dSC.J. Collier   Example output:
4715129044dSC.J. Collier
4725129044dSC.J. Collier   .. code-block:: console
4735129044dSC.J. Collier
4745129044dSC.J. Collier      t5nex0@pci0:2:0:4: class=0x020000 card=0x00001425 chip=0x54011425 rev=0x00
4755129044dSC.J. Collier
4765129044dSC.J. Collier   In the above example, the t5nex0 is bound to 2:0:4 bus address.
4775129044dSC.J. Collier
4785129044dSC.J. Collier   .. note::
4795129044dSC.J. Collier
4805129044dSC.J. Collier      Both the interfaces of a Chelsio T5 2-port adapter are bound to the
4815129044dSC.J. Collier      same PCI bus address.
4825129044dSC.J. Collier
4835129044dSC.J. Collier#. Unload the kernel module:
4845129044dSC.J. Collier
4855129044dSC.J. Collier   .. code-block:: console
4865129044dSC.J. Collier
4875129044dSC.J. Collier      kldunload if_cxgbe
4885129044dSC.J. Collier
4895129044dSC.J. Collier#. Set the PCI bus addresses to hw.nic_uio.bdfs kernel environment parameter:
4905129044dSC.J. Collier
4915129044dSC.J. Collier   .. code-block:: console
4925129044dSC.J. Collier
4935129044dSC.J. Collier      kenv hw.nic_uio.bdfs="2:0:4"
4945129044dSC.J. Collier
4955129044dSC.J. Collier   This automatically binds 2:0:4 to nic_uio kernel driver when it is loaded in
4965129044dSC.J. Collier   the next step.
4975129044dSC.J. Collier
4985129044dSC.J. Collier   .. note::
4995129044dSC.J. Collier
5005129044dSC.J. Collier      Currently, CXGBE PMD only supports the binding of PF4 for Chelsio T5 NICs.
5015129044dSC.J. Collier
5025129044dSC.J. Collier#. Load nic_uio kernel driver:
5035129044dSC.J. Collier
5045129044dSC.J. Collier   .. code-block:: console
5055129044dSC.J. Collier
5065129044dSC.J. Collier      kldload ./x86_64-native-bsdapp-clang/kmod/nic_uio.ko
5075129044dSC.J. Collier
5085129044dSC.J. Collier#. Start testpmd with basic parameters:
5095129044dSC.J. Collier
5105129044dSC.J. Collier   .. code-block:: console
5115129044dSC.J. Collier
5125129044dSC.J. Collier      ./x86_64-native-bsdapp-clang/app/testpmd -c 0xf -n 4 -w 0000:02:00.4 -- -i
5135129044dSC.J. Collier
5145129044dSC.J. Collier   Example output:
5155129044dSC.J. Collier
5165129044dSC.J. Collier   .. code-block:: console
5175129044dSC.J. Collier
5185129044dSC.J. Collier      [...]
5195129044dSC.J. Collier      EAL: PCI device 0000:02:00.4 on NUMA socket 0
5205129044dSC.J. Collier      EAL:   probe driver: 1425:5401 rte_cxgbe_pmd
5215129044dSC.J. Collier      EAL:   PCI memory mapped at 0x8007ec000
5225129044dSC.J. Collier      EAL:   PCI memory mapped at 0x842800000
5235129044dSC.J. Collier      EAL:   PCI memory mapped at 0x80086c000
5245129044dSC.J. Collier      PMD: rte_cxgbe_pmd: fw:, TP:
5255129044dSC.J. Collier      PMD: rte_cxgbe_pmd: Coming up as MASTER: Initializing adapter
5265129044dSC.J. Collier      Interactive-mode selected
5275129044dSC.J. Collier      Configuring Port 0 (socket 0)
5285129044dSC.J. Collier      Port 0: 00:07:43:2D:EA:C0
5295129044dSC.J. Collier      Configuring Port 1 (socket 0)
5305129044dSC.J. Collier      Port 1: 00:07:43:2D:EA:C8
5315129044dSC.J. Collier      Checking link statuses...
5325129044dSC.J. Collier      PMD: rte_cxgbe_pmd: Port0: passive DA port module inserted
5335129044dSC.J. Collier      PMD: rte_cxgbe_pmd: Port1: passive DA port module inserted
5345129044dSC.J. Collier      Port 0 Link Up - speed 10000 Mbps - full-duplex
5355129044dSC.J. Collier      Port 1 Link Up - speed 10000 Mbps - full-duplex
5365129044dSC.J. Collier      Done
5375129044dSC.J. Collier      testpmd>
5385129044dSC.J. Collier
5395129044dSC.J. Collier.. note::
5405129044dSC.J. Collier
5415129044dSC.J. Collier   Flow control pause TX/RX is disabled by default and can be enabled via
5425129044dSC.J. Collier   testpmd. Refer section :ref:`flow-control` for more details.
5435129044dSC.J. Collier
5445129044dSC.J. CollierSample Application Notes
5455129044dSC.J. Collier------------------------
5465129044dSC.J. Collier
5475129044dSC.J. Collier.. _flow-control:
5485129044dSC.J. Collier
5495129044dSC.J. CollierEnable/Disable Flow Control
5505129044dSC.J. Collier~~~~~~~~~~~~~~~~~~~~~~~~~~~
5515129044dSC.J. Collier
5525129044dSC.J. CollierFlow control pause TX/RX is disabled by default and can be enabled via
5535129044dSC.J. Colliertestpmd as follows:
5545129044dSC.J. Collier
5555129044dSC.J. Collier.. code-block:: console
5565129044dSC.J. Collier
5575129044dSC.J. Collier   testpmd> set flow_ctrl rx on tx on 0 0 0 0 mac_ctrl_frame_fwd off autoneg on 0
5585129044dSC.J. Collier   testpmd> set flow_ctrl rx on tx on 0 0 0 0 mac_ctrl_frame_fwd off autoneg on 1
5595129044dSC.J. Collier
5605129044dSC.J. CollierTo disable again, run:
5615129044dSC.J. Collier
5625129044dSC.J. Collier.. code-block:: console
5635129044dSC.J. Collier
5645129044dSC.J. Collier   testpmd> set flow_ctrl rx off tx off 0 0 0 0 mac_ctrl_frame_fwd off autoneg off 0
5655129044dSC.J. Collier   testpmd> set flow_ctrl rx off tx off 0 0 0 0 mac_ctrl_frame_fwd off autoneg off 1
5665129044dSC.J. Collier
5675129044dSC.J. CollierJumbo Mode
5685129044dSC.J. Collier~~~~~~~~~~
5695129044dSC.J. Collier
5705129044dSC.J. CollierThere are two ways to enable sending and receiving of jumbo frames via testpmd.
5715129044dSC.J. CollierOne method involves using the **mtu** command, which changes the mtu of an
5725129044dSC.J. Collierindividual port without having to stop the selected port. Another method
5735129044dSC.J. Collierinvolves stopping all the ports first and then running **max-pkt-len** command
5745129044dSC.J. Collierto configure the mtu of all the ports with a single command.
5755129044dSC.J. Collier
5765129044dSC.J. Collier- To configure each port individually, run the mtu command as follows:
5775129044dSC.J. Collier
5785129044dSC.J. Collier  .. code-block:: console
5795129044dSC.J. Collier
5805129044dSC.J. Collier     testpmd> port config mtu 0 9000
5815129044dSC.J. Collier     testpmd> port config mtu 1 9000
5825129044dSC.J. Collier
5835129044dSC.J. Collier- To configure all the ports at once, stop all the ports first and run the
5845129044dSC.J. Collier  max-pkt-len command as follows:
5855129044dSC.J. Collier
5865129044dSC.J. Collier  .. code-block:: console
5875129044dSC.J. Collier
5885129044dSC.J. Collier     testpmd> port stop all
5895129044dSC.J. Collier     testpmd> port config all max-pkt-len 9000