197f17497SC.J. Collier.. BSD LICENSE
297f17497SC.J. Collier
397f17497SC.J. Collier    Copyright (c) 2015-2016 Amazon.com, Inc. or its affiliates.
497f17497SC.J. Collier    All rights reserved.
597f17497SC.J. Collier
697f17497SC.J. Collier    Redistribution and use in source and binary forms, with or without
797f17497SC.J. Collier    modification, are permitted provided that the following conditions
897f17497SC.J. Collier    are met:
997f17497SC.J. Collier
1097f17497SC.J. Collier    * Redistributions of source code must retain the above copyright
1197f17497SC.J. Collier    notice, this list of conditions and the following disclaimer.
1297f17497SC.J. Collier    * Redistributions in binary form must reproduce the above copyright
1397f17497SC.J. Collier    notice, this list of conditions and the following disclaimer in
1497f17497SC.J. Collier    the documentation and/or other materials provided with the
1597f17497SC.J. Collier    distribution.
1697f17497SC.J. Collier    * Neither the name of Amazon.com, Inc. nor the names of its
1797f17497SC.J. Collier    contributors may be used to endorse or promote products derived
1897f17497SC.J. Collier    from this software without specific prior written permission.
1997f17497SC.J. Collier
2097f17497SC.J. Collier    THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
2197f17497SC.J. Collier    "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
2297f17497SC.J. Collier    LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
2397f17497SC.J. Collier    A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
2497f17497SC.J. Collier    OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
2597f17497SC.J. Collier    SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
2697f17497SC.J. Collier    LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
2797f17497SC.J. Collier    DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
2897f17497SC.J. Collier    THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
2997f17497SC.J. Collier    (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
3097f17497SC.J. Collier    OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
3197f17497SC.J. Collier
3297f17497SC.J. CollierENA Poll Mode Driver
3397f17497SC.J. Collier====================
3497f17497SC.J. Collier
3597f17497SC.J. CollierThe ENA PMD is a DPDK poll-mode driver for the Amazon Elastic
3697f17497SC.J. CollierNetwork Adapter (ENA) family.
3797f17497SC.J. Collier
3897f17497SC.J. CollierOverview
3997f17497SC.J. Collier--------
4097f17497SC.J. Collier
4197f17497SC.J. CollierThe ENA driver exposes a lightweight management interface with a
4297f17497SC.J. Collierminimal set of memory mapped registers and an extendable command set
4397f17497SC.J. Collierthrough an Admin Queue.
4497f17497SC.J. Collier
4597f17497SC.J. CollierThe driver supports a wide range of ENA adapters, is link-speed
4697f17497SC.J. Collierindependent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE,
4797f17497SC.J. Collieretc.), and it negotiates and supports an extendable feature set.
4897f17497SC.J. Collier
4997f17497SC.J. CollierENA adapters allow high speed and low overhead Ethernet traffic
5097f17497SC.J. Collierprocessing by providing a dedicated Tx/Rx queue pair per CPU core.
5197f17497SC.J. Collier
5297f17497SC.J. CollierThe ENA driver supports industry standard TCP/IP offload features such
5397f17497SC.J. Collieras checksum offload and TCP transmit segmentation offload (TSO).
5497f17497SC.J. Collier
5597f17497SC.J. CollierReceive-side scaling (RSS) is supported for multi-core scaling.
5697f17497SC.J. Collier
5797f17497SC.J. CollierSome of the ENA devices support a working mode called Low-latency
5897f17497SC.J. CollierQueue (LLQ), which saves several more microseconds.
5997f17497SC.J. Collier
6097f17497SC.J. CollierManagement Interface
6197f17497SC.J. Collier--------------------
6297f17497SC.J. Collier
6397f17497SC.J. CollierENA management interface is exposed by means of:
6497f17497SC.J. Collier
6597f17497SC.J. Collier* Device Registers
6697f17497SC.J. Collier* Admin Queue (AQ) and Admin Completion Queue (ACQ)
6797f17497SC.J. Collier
6897f17497SC.J. CollierENA device memory-mapped PCIe space for registers (MMIO registers)
6997f17497SC.J. Collierare accessed only during driver initialization and are not involved
7097f17497SC.J. Collierin further normal device operation.
7197f17497SC.J. Collier
7297f17497SC.J. CollierAQ is used for submitting management commands, and the
7397f17497SC.J. Collierresults/responses are reported asynchronously through ACQ.
7497f17497SC.J. Collier
7597f17497SC.J. CollierENA introduces a very small set of management commands with room for
7697f17497SC.J. Colliervendor-specific extensions. Most of the management operations are
7797f17497SC.J. Collierframed in a generic Get/Set feature command.
7897f17497SC.J. Collier
7997f17497SC.J. CollierThe following admin queue commands are supported:
8097f17497SC.J. Collier
8197f17497SC.J. Collier* Create I/O submission queue
8297f17497SC.J. Collier* Create I/O completion queue
8397f17497SC.J. Collier* Destroy I/O submission queue
8497f17497SC.J. Collier* Destroy I/O completion queue
8597f17497SC.J. Collier* Get feature
8697f17497SC.J. Collier* Set feature
8797f17497SC.J. Collier* Get statistics
8897f17497SC.J. Collier
8997f17497SC.J. CollierRefer to ``ena_admin_defs.h`` for the list of supported Get/Set Feature
9097f17497SC.J. Collierproperties.
9197f17497SC.J. Collier
9297f17497SC.J. CollierData Path Interface
9397f17497SC.J. Collier-------------------
9497f17497SC.J. Collier
9597f17497SC.J. CollierI/O operations are based on Tx and Rx Submission Queues (Tx SQ and Rx
9697f17497SC.J. CollierSQ correspondingly). Each SQ has a completion queue (CQ) associated
9797f17497SC.J. Collierwith it.
9897f17497SC.J. Collier
9997f17497SC.J. CollierThe SQs and CQs are implemented as descriptor rings in contiguous
10097f17497SC.J. Collierphysical memory.
10197f17497SC.J. Collier
10297f17497SC.J. CollierRefer to ``ena_eth_io_defs.h`` for the detailed structure of the descriptor
10397f17497SC.J. Collier
10497f17497SC.J. CollierThe driver supports multi-queue for both Tx and Rx.
10597f17497SC.J. Collier
10697f17497SC.J. CollierConfiguration information
10797f17497SC.J. Collier-------------------------
10897f17497SC.J. Collier
10997f17497SC.J. Collier**DPDK Configuration Parameters**
11097f17497SC.J. Collier
11197f17497SC.J. Collier  The following configuration options are available for the ENA PMD:
11297f17497SC.J. Collier
11397f17497SC.J. Collier   * **CONFIG_RTE_LIBRTE_ENA_PMD** (default y): Enables or disables inclusion
11497f17497SC.J. Collier     of the ENA PMD driver in the DPDK compilation.
11597f17497SC.J. Collier
11697f17497SC.J. Collier
11797f17497SC.J. Collier   * **CONFIG_RTE_LIBRTE_ENA_DEBUG_INIT** (default y): Enables or disables debug
11897f17497SC.J. Collier     logging of device initialization within the ENA PMD driver.
11997f17497SC.J. Collier
12097f17497SC.J. Collier   * **CONFIG_RTE_LIBRTE_ENA_DEBUG_RX** (default n): Enables or disables debug
12197f17497SC.J. Collier     logging of RX logic within the ENA PMD driver.
12297f17497SC.J. Collier
12397f17497SC.J. Collier   * **CONFIG_RTE_LIBRTE_ENA_DEBUG_TX** (default n): Enables or disables debug
12497f17497SC.J. Collier     logging of TX logic within the ENA PMD driver.
12597f17497SC.J. Collier
12697f17497SC.J. Collier   * **CONFIG_RTE_LIBRTE_ENA_COM_DEBUG** (default n): Enables or disables debug
12797f17497SC.J. Collier     logging of low level tx/rx logic in ena_com(base) within the ENA PMD driver.
12897f17497SC.J. Collier
12997f17497SC.J. Collier**ENA Configuration Parameters**
13097f17497SC.J. Collier
13197f17497SC.J. Collier   * **Number of Queues**
13297f17497SC.J. Collier
13397f17497SC.J. Collier     This is the requested number of queues upon initialization, however, the actual
13497f17497SC.J. Collier     number of receive and transmit queues to be created will be the minimum between
13597f17497SC.J. Collier     the maximal number supported by the device and number of queues requested.
13697f17497SC.J. Collier
13797f17497SC.J. Collier   * **Size of Queues**
13897f17497SC.J. Collier
13997f17497SC.J. Collier     This is the requested size of receive/transmit queues, while the actual size
14097f17497SC.J. Collier     will be the minimum between the requested size and the maximal receive/transmit
14197f17497SC.J. Collier     supported by the device.
14297f17497SC.J. Collier
14397f17497SC.J. CollierBuilding DPDK
14497f17497SC.J. Collier-------------
14597f17497SC.J. Collier
14697f17497SC.J. CollierSee the :ref:`DPDK Getting Started Guide for Linux <linux_gsg>` for
14797f17497SC.J. Collierinstructions on how to build DPDK.
14897f17497SC.J. Collier
14997f17497SC.J. CollierBy default the ENA PMD library will be built into the DPDK library.
15097f17497SC.J. Collier
15197f17497SC.J. CollierFor configuring and using UIO and VFIO frameworks, please also refer :ref:`the
15297f17497SC.J. Collierdocumentation that comes with DPDK suite <linux_gsg>`.
15397f17497SC.J. Collier
15497f17497SC.J. CollierSupported ENA adapters
15597f17497SC.J. Collier----------------------
15697f17497SC.J. Collier
15797f17497SC.J. CollierCurrent ENA PMD supports the following ENA adapters including:
15897f17497SC.J. Collier
15997f17497SC.J. Collier* ``1d0f:ec20`` - ENA VF
16097f17497SC.J. Collier* ``1d0f:ec21`` - ENA VF with LLQ support
16197f17497SC.J. Collier
16297f17497SC.J. CollierSupported Operating Systems
16397f17497SC.J. Collier---------------------------
16497f17497SC.J. Collier
16597f17497SC.J. CollierAny Linux distribution fulfilling the conditions described in ``System Requirements``
16697f17497SC.J. Colliersection of :ref:`the DPDK documentation <linux_gsg>` or refer to *DPDK Release Notes*.
16797f17497SC.J. Collier
16897f17497SC.J. CollierSupported features
16997f17497SC.J. Collier------------------
17097f17497SC.J. Collier
17197f17497SC.J. Collier* Jumbo frames up to 9K
17297f17497SC.J. Collier* Port Hardware Statistics
17397f17497SC.J. Collier* IPv4/TCP/UDP checksum offload
17497f17497SC.J. Collier* TSO offload
17597f17497SC.J. Collier* Multiple receive and transmit queues
17697f17497SC.J. Collier* RSS
17797f17497SC.J. Collier* Low Latency Queue for Tx
17897f17497SC.J. Collier
17997f17497SC.J. CollierUnsupported features
18097f17497SC.J. Collier--------------------
18197f17497SC.J. Collier
18297f17497SC.J. CollierThe features supported by the device and not yet supported by this PMD include:
18397f17497SC.J. Collier
18497f17497SC.J. Collier* Asynchronous Event Notification Queue (AENQ)
18597f17497SC.J. Collier
18697f17497SC.J. CollierPrerequisites
18797f17497SC.J. Collier-------------
18897f17497SC.J. Collier
18997f17497SC.J. Collier#. Prepare the system as recommended by DPDK suite.  This includes environment
190f7a9461eSLuca Boccassi   variables, hugepages configuration, tool-chains and configuration.
19197f17497SC.J. Collier
192f7a9461eSLuca Boccassi#. ENA PMD can operate with ``vfio-pci`` or ``igb_uio`` driver.
19397f17497SC.J. Collier
194f7a9461eSLuca Boccassi#. Insert ``vfio-pci`` or ``igb_uio`` kernel module using the command
195f7a9461eSLuca Boccassi   ``modprobe vfio-pci`` or ``modprobe igb_uio`` respectively.
196f7a9461eSLuca Boccassi
197f7a9461eSLuca Boccassi#. For ``vfio-pci`` users only:
198f7a9461eSLuca Boccassi   Please make sure that ``IOMMU`` is enabled in your system,
199f7a9461eSLuca Boccassi   or use ``vfio`` driver in ``noiommu`` mode::
200f7a9461eSLuca Boccassi
201f7a9461eSLuca Boccassi     echo 1 > /sys/module/vfio/parameters/enable_unsafe_noiommu_mode
202f7a9461eSLuca Boccassi
203f7a9461eSLuca Boccassi#. Bind the intended ENA device to ``vfio-pci`` or ``igb_uio`` module.
20497f17497SC.J. Collier
20597f17497SC.J. Collier
20697f17497SC.J. CollierAt this point the system should be ready to run DPDK applications. Once the
20797f17497SC.J. Collierapplication runs to completion, the ENA can be detached from igb_uio if necessary.
20897f17497SC.J. Collier
20997f17497SC.J. CollierUsage example
21097f17497SC.J. Collier-------------
21197f17497SC.J. Collier
21297f17497SC.J. CollierThis section demonstrates how to launch **testpmd** with Amazon ENA
21397f17497SC.J. Collierdevices managed by librte_pmd_ena.
21497f17497SC.J. Collier
21597f17497SC.J. Collier#. Load the kernel modules:
21697f17497SC.J. Collier
21797f17497SC.J. Collier   .. code-block:: console
21897f17497SC.J. Collier
21997f17497SC.J. Collier      modprobe uio
22097f17497SC.J. Collier      insmod ./x86_64-native-linuxapp-gcc/kmod/igb_uio.ko
22197f17497SC.J. Collier
22297f17497SC.J. Collier   .. note::
22397f17497SC.J. Collier
22497f17497SC.J. Collier      Currently Amazon ENA PMD driver depends on igb_uio user space I/O kernel module
22597f17497SC.J. Collier
22697f17497SC.J. Collier#. Mount and request huge pages:
22797f17497SC.J. Collier
22897f17497SC.J. Collier   .. code-block:: console
22997f17497SC.J. Collier
23097f17497SC.J. Collier      mount -t hugetlbfs nodev /mnt/hugepages
23197f17497SC.J. Collier      echo 1024 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
23297f17497SC.J. Collier
23397f17497SC.J. Collier#. Bind UIO driver to ENA device (using provided by DPDK binding tool):
23497f17497SC.J. Collier
23597f17497SC.J. Collier   .. code-block:: console
23697f17497SC.J. Collier
2377b53c036SRicardo Salveti      ./tools/dpdk-devbind.py --bind=igb_uio 0000:02:00.1
23897f17497SC.J. Collier
23997f17497SC.J. Collier#. Start testpmd with basic parameters:
24097f17497SC.J. Collier
24197f17497SC.J. Collier   .. code-block:: console
24297f17497SC.J. Collier
24397f17497SC.J. Collier      ./x86_64-native-linuxapp-gcc/app/testpmd -c 0xf -n 4 -- -i
24497f17497SC.J. Collier
24597f17497SC.J. Collier   Example output:
24697f17497SC.J. Collier
24797f17497SC.J. Collier   .. code-block:: console
24897f17497SC.J. Collier
24997f17497SC.J. Collier      [...]
25097f17497SC.J. Collier      EAL: PCI device 0000:02:00.1 on NUMA socket -1
25197f17497SC.J. Collier      EAL:   probe driver: 1d0f:ec20 rte_ena_pmd
25297f17497SC.J. Collier      EAL:   PCI memory mapped at 0x7f9b6c400000
25397f17497SC.J. Collier      PMD: eth_ena_dev_init(): Initializing 0:2:0.1
25497f17497SC.J. Collier      Interactive-mode selected
25597f17497SC.J. Collier      Configuring Port 0 (socket 0)
25697f17497SC.J. Collier      Port 0: 00:00:00:11:00:01
25797f17497SC.J. Collier      Checking link statuses...
25897f17497SC.J. Collier      Port 0 Link Up - speed 10000 Mbps - full-duplex
25997f17497SC.J. Collier      Done
26097f17497SC.J. Collier      testpmd>
261