ena.rst revision f7a9461e
1.. BSD LICENSE
2
3    Copyright (c) 2015-2016 Amazon.com, Inc. or its affiliates.
4    All rights reserved.
5
6    Redistribution and use in source and binary forms, with or without
7    modification, are permitted provided that the following conditions
8    are met:
9
10    * Redistributions of source code must retain the above copyright
11    notice, this list of conditions and the following disclaimer.
12    * Redistributions in binary form must reproduce the above copyright
13    notice, this list of conditions and the following disclaimer in
14    the documentation and/or other materials provided with the
15    distribution.
16    * Neither the name of Amazon.com, Inc. nor the names of its
17    contributors may be used to endorse or promote products derived
18    from this software without specific prior written permission.
19
20    THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
21    "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
22    LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
23    A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
24    OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
25    SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
26    LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
27    DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
28    THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
29    (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
30    OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
31
32ENA Poll Mode Driver
33====================
34
35The ENA PMD is a DPDK poll-mode driver for the Amazon Elastic
36Network Adapter (ENA) family.
37
38Overview
39--------
40
41The ENA driver exposes a lightweight management interface with a
42minimal set of memory mapped registers and an extendable command set
43through an Admin Queue.
44
45The driver supports a wide range of ENA adapters, is link-speed
46independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE,
47etc.), and it negotiates and supports an extendable feature set.
48
49ENA adapters allow high speed and low overhead Ethernet traffic
50processing by providing a dedicated Tx/Rx queue pair per CPU core.
51
52The ENA driver supports industry standard TCP/IP offload features such
53as checksum offload and TCP transmit segmentation offload (TSO).
54
55Receive-side scaling (RSS) is supported for multi-core scaling.
56
57Some of the ENA devices support a working mode called Low-latency
58Queue (LLQ), which saves several more microseconds.
59
60Management Interface
61--------------------
62
63ENA management interface is exposed by means of:
64
65* Device Registers
66* Admin Queue (AQ) and Admin Completion Queue (ACQ)
67
68ENA device memory-mapped PCIe space for registers (MMIO registers)
69are accessed only during driver initialization and are not involved
70in further normal device operation.
71
72AQ is used for submitting management commands, and the
73results/responses are reported asynchronously through ACQ.
74
75ENA introduces a very small set of management commands with room for
76vendor-specific extensions. Most of the management operations are
77framed in a generic Get/Set feature command.
78
79The following admin queue commands are supported:
80
81* Create I/O submission queue
82* Create I/O completion queue
83* Destroy I/O submission queue
84* Destroy I/O completion queue
85* Get feature
86* Set feature
87* Get statistics
88
89Refer to ``ena_admin_defs.h`` for the list of supported Get/Set Feature
90properties.
91
92Data Path Interface
93-------------------
94
95I/O operations are based on Tx and Rx Submission Queues (Tx SQ and Rx
96SQ correspondingly). Each SQ has a completion queue (CQ) associated
97with it.
98
99The SQs and CQs are implemented as descriptor rings in contiguous
100physical memory.
101
102Refer to ``ena_eth_io_defs.h`` for the detailed structure of the descriptor
103
104The driver supports multi-queue for both Tx and Rx.
105
106Configuration information
107-------------------------
108
109**DPDK Configuration Parameters**
110
111  The following configuration options are available for the ENA PMD:
112
113   * **CONFIG_RTE_LIBRTE_ENA_PMD** (default y): Enables or disables inclusion
114     of the ENA PMD driver in the DPDK compilation.
115
116
117   * **CONFIG_RTE_LIBRTE_ENA_DEBUG_INIT** (default y): Enables or disables debug
118     logging of device initialization within the ENA PMD driver.
119
120   * **CONFIG_RTE_LIBRTE_ENA_DEBUG_RX** (default n): Enables or disables debug
121     logging of RX logic within the ENA PMD driver.
122
123   * **CONFIG_RTE_LIBRTE_ENA_DEBUG_TX** (default n): Enables or disables debug
124     logging of TX logic within the ENA PMD driver.
125
126   * **CONFIG_RTE_LIBRTE_ENA_COM_DEBUG** (default n): Enables or disables debug
127     logging of low level tx/rx logic in ena_com(base) within the ENA PMD driver.
128
129**ENA Configuration Parameters**
130
131   * **Number of Queues**
132
133     This is the requested number of queues upon initialization, however, the actual
134     number of receive and transmit queues to be created will be the minimum between
135     the maximal number supported by the device and number of queues requested.
136
137   * **Size of Queues**
138
139     This is the requested size of receive/transmit queues, while the actual size
140     will be the minimum between the requested size and the maximal receive/transmit
141     supported by the device.
142
143Building DPDK
144-------------
145
146See the :ref:`DPDK Getting Started Guide for Linux <linux_gsg>` for
147instructions on how to build DPDK.
148
149By default the ENA PMD library will be built into the DPDK library.
150
151For configuring and using UIO and VFIO frameworks, please also refer :ref:`the
152documentation that comes with DPDK suite <linux_gsg>`.
153
154Supported ENA adapters
155----------------------
156
157Current ENA PMD supports the following ENA adapters including:
158
159* ``1d0f:ec20`` - ENA VF
160* ``1d0f:ec21`` - ENA VF with LLQ support
161
162Supported Operating Systems
163---------------------------
164
165Any Linux distribution fulfilling the conditions described in ``System Requirements``
166section of :ref:`the DPDK documentation <linux_gsg>` or refer to *DPDK Release Notes*.
167
168Supported features
169------------------
170
171* Jumbo frames up to 9K
172* Port Hardware Statistics
173* IPv4/TCP/UDP checksum offload
174* TSO offload
175* Multiple receive and transmit queues
176* RSS
177* Low Latency Queue for Tx
178
179Unsupported features
180--------------------
181
182The features supported by the device and not yet supported by this PMD include:
183
184* Asynchronous Event Notification Queue (AENQ)
185
186Prerequisites
187-------------
188
189#. Prepare the system as recommended by DPDK suite.  This includes environment
190   variables, hugepages configuration, tool-chains and configuration.
191
192#. ENA PMD can operate with ``vfio-pci`` or ``igb_uio`` driver.
193
194#. Insert ``vfio-pci`` or ``igb_uio`` kernel module using the command
195   ``modprobe vfio-pci`` or ``modprobe igb_uio`` respectively.
196
197#. For ``vfio-pci`` users only:
198   Please make sure that ``IOMMU`` is enabled in your system,
199   or use ``vfio`` driver in ``noiommu`` mode::
200
201     echo 1 > /sys/module/vfio/parameters/enable_unsafe_noiommu_mode
202
203#. Bind the intended ENA device to ``vfio-pci`` or ``igb_uio`` module.
204
205
206At this point the system should be ready to run DPDK applications. Once the
207application runs to completion, the ENA can be detached from igb_uio if necessary.
208
209Usage example
210-------------
211
212This section demonstrates how to launch **testpmd** with Amazon ENA
213devices managed by librte_pmd_ena.
214
215#. Load the kernel modules:
216
217   .. code-block:: console
218
219      modprobe uio
220      insmod ./x86_64-native-linuxapp-gcc/kmod/igb_uio.ko
221
222   .. note::
223
224      Currently Amazon ENA PMD driver depends on igb_uio user space I/O kernel module
225
226#. Mount and request huge pages:
227
228   .. code-block:: console
229
230      mount -t hugetlbfs nodev /mnt/hugepages
231      echo 1024 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
232
233#. Bind UIO driver to ENA device (using provided by DPDK binding tool):
234
235   .. code-block:: console
236
237      ./tools/dpdk-devbind.py --bind=igb_uio 0000:02:00.1
238
239#. Start testpmd with basic parameters:
240
241   .. code-block:: console
242
243      ./x86_64-native-linuxapp-gcc/app/testpmd -c 0xf -n 4 -- -i
244
245   Example output:
246
247   .. code-block:: console
248
249      [...]
250      EAL: PCI device 0000:02:00.1 on NUMA socket -1
251      EAL:   probe driver: 1d0f:ec20 rte_ena_pmd
252      EAL:   PCI memory mapped at 0x7f9b6c400000
253      PMD: eth_ena_dev_init(): Initializing 0:2:0.1
254      Interactive-mode selected
255      Configuring Port 0 (socket 0)
256      Port 0: 00:00:00:11:00:01
257      Checking link statuses...
258      Port 0 Link Up - speed 10000 Mbps - full-duplex
259      Done
260      testpmd>
261