l2_forward_real_virtual.rst revision 7b53c036
1..  BSD LICENSE
2    Copyright(c) 2010-2014 Intel Corporation. All rights reserved.
3    All rights reserved.
4
5    Redistribution and use in source and binary forms, with or without
6    modification, are permitted provided that the following conditions
7    are met:
8
9    * Redistributions of source code must retain the above copyright
10    notice, this list of conditions and the following disclaimer.
11    * Redistributions in binary form must reproduce the above copyright
12    notice, this list of conditions and the following disclaimer in
13    the documentation and/or other materials provided with the
14    distribution.
15    * Neither the name of Intel Corporation nor the names of its
16    contributors may be used to endorse or promote products derived
17    from this software without specific prior written permission.
18
19    THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
20    "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
21    LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
22    A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
23    OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
24    SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
25    LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
26    DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
27    THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
28    (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
29    OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
30
31.. _l2_fwd_app_real_and_virtual:
32
33L2 Forwarding Sample Application (in Real and Virtualized Environments)
34=======================================================================
35
36The L2 Forwarding sample application is a simple example of packet processing using
37the Data Plane Development Kit (DPDK) which
38also takes advantage of Single Root I/O Virtualization (SR-IOV) features in a virtualized environment.
39
40.. note::
41
42    Please note that previously a separate L2 Forwarding in Virtualized Environments sample application was used,
43    however, in later DPDK versions these sample applications have been merged.
44
45Overview
46--------
47
48The L2 Forwarding sample application, which can operate in real and virtualized environments,
49performs L2 forwarding for each packet that is received on an RX_PORT.
50The destination port is the adjacent port from the enabled portmask, that is,
51if the first four ports are enabled (portmask 0xf),
52ports 1 and 2 forward into each other, and ports 3 and 4 forward into each other.
53Also, the MAC addresses are affected as follows:
54
55*   The source MAC address is replaced by the TX_PORT MAC address
56
57*   The destination MAC address is replaced by  02:00:00:00:00:TX_PORT_ID
58
59This application can be used to benchmark performance using a traffic-generator, as shown in the :numref:`figure_l2_fwd_benchmark_setup`.
60
61The application can also be used in a virtualized environment as shown in :numref:`figure_l2_fwd_virtenv_benchmark_setup`.
62
63The L2 Forwarding application can also be used as a starting point for developing a new application based on the DPDK.
64
65.. _figure_l2_fwd_benchmark_setup:
66
67.. figure:: img/l2_fwd_benchmark_setup.*
68
69   Performance Benchmark Setup (Basic Environment)
70
71
72.. _figure_l2_fwd_virtenv_benchmark_setup:
73
74.. figure:: img/l2_fwd_virtenv_benchmark_setup.*
75
76   Performance Benchmark Setup (Virtualized Environment)
77
78.. _l2_fwd_vf_setup:
79
80Virtual Function Setup Instructions
81~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
82
83This application can use the virtual function available in the system and
84therefore can be used in a virtual machine without passing through
85the whole Network Device into a guest machine in a virtualized scenario.
86The virtual functions can be enabled in the host machine or the hypervisor with the respective physical function driver.
87
88For example, in a Linux* host machine, it is possible to enable a virtual function using the following command:
89
90.. code-block:: console
91
92    modprobe ixgbe max_vfs=2,2
93
94This command enables two Virtual Functions on each of Physical Function of the NIC,
95with two physical ports in the PCI configuration space.
96It is important to note that enabled Virtual Function 0 and 2 would belong to Physical Function 0
97and Virtual Function 1 and 3 would belong to Physical Function 1,
98in this case enabling a total of four Virtual Functions.
99
100Compiling the Application
101-------------------------
102
103#.  Go to the example directory:
104
105    .. code-block:: console
106
107        export RTE_SDK=/path/to/rte_sdk
108        cd ${RTE_SDK}/examples/l2fwd
109
110#.  Set the target (a default target is used if not specified). For example:
111
112    .. code-block:: console
113
114        export RTE_TARGET=x86_64-native-linuxapp-gcc
115
116    *See the DPDK Getting Started Guide* for possible RTE_TARGET values.
117
118#.  Build the application:
119
120    .. code-block:: console
121
122        make
123
124Running the Application
125-----------------------
126
127The application requires a number of command line options:
128
129.. code-block:: console
130
131    ./build/l2fwd [EAL options] -- -p PORTMASK [-q NQ]
132
133where,
134
135*   p PORTMASK: A hexadecimal bitmask of the ports to configure
136
137*   q NQ: A number of queues (=ports) per lcore (default is 1)
138
139To run the application in linuxapp environment with 4 lcores, 16 ports and 8 RX queues per lcore, issue the command:
140
141.. code-block:: console
142
143    $ ./build/l2fwd -c f -n 4 -- -q 8 -p ffff
144
145Refer to the *DPDK Getting Started Guide* for general information on running applications
146and the Environment Abstraction Layer (EAL) options.
147
148Explanation
149-----------
150
151The following sections provide some explanation of the code.
152
153.. _l2_fwd_app_cmd_arguments:
154
155Command Line Arguments
156~~~~~~~~~~~~~~~~~~~~~~
157
158The L2 Forwarding sample application takes specific parameters,
159in addition to Environment Abstraction Layer (EAL) arguments.
160The preferred way to parse parameters is to use the getopt() function,
161since it is part of a well-defined and portable library.
162
163The parsing of arguments is done in the l2fwd_parse_args() function.
164The method of argument parsing is not described here.
165Refer to the *glibc getopt(3)* man page for details.
166
167EAL arguments are parsed first, then application-specific arguments.
168This is done at the beginning of the main() function:
169
170.. code-block:: c
171
172    /* init EAL */
173
174    ret = rte_eal_init(argc, argv);
175    if (ret < 0)
176        rte_exit(EXIT_FAILURE, "Invalid EAL arguments\n");
177
178    argc -= ret;
179    argv += ret;
180
181    /* parse application arguments (after the EAL ones) */
182
183    ret = l2fwd_parse_args(argc, argv);
184    if (ret < 0)
185        rte_exit(EXIT_FAILURE, "Invalid L2FWD arguments\n");
186
187.. _l2_fwd_app_mbuf_init:
188
189Mbuf Pool Initialization
190~~~~~~~~~~~~~~~~~~~~~~~~
191
192Once the arguments are parsed, the mbuf pool is created.
193The mbuf pool contains a set of mbuf objects that will be used by the driver
194and the application to store network packet data:
195
196.. code-block:: c
197
198    /* create the mbuf pool */
199
200    l2fwd_pktmbuf_pool = rte_mempool_create("mbuf_pool", NB_MBUF, MBUF_SIZE, 32, sizeof(struct rte_pktmbuf_pool_private),
201        rte_pktmbuf_pool_init, NULL, rte_pktmbuf_init, NULL, SOCKET0, 0);
202
203    if (l2fwd_pktmbuf_pool == NULL)
204        rte_panic("Cannot init mbuf pool\n");
205
206The rte_mempool is a generic structure used to handle pools of objects.
207In this case, it is necessary to create a pool that will be used by the driver,
208which expects to have some reserved space in the mempool structure,
209sizeof(struct rte_pktmbuf_pool_private) bytes.
210The number of allocated pkt mbufs is NB_MBUF, with a size of MBUF_SIZE each.
211A per-lcore cache of 32 mbufs is kept.
212The memory is allocated in NUMA socket 0,
213but it is possible to extend this code to allocate one mbuf pool per socket.
214
215Two callback pointers are also given to the rte_mempool_create() function:
216
217*   The first callback pointer is to rte_pktmbuf_pool_init() and is used
218    to initialize the private data of the mempool, which is needed by the driver.
219    This function is provided by the mbuf API, but can be copied and extended by the developer.
220
221*   The second callback pointer given to rte_mempool_create() is the mbuf initializer.
222    The default is used, that is, rte_pktmbuf_init(), which is provided in the rte_mbuf library.
223    If a more complex application wants to extend the rte_pktmbuf structure for its own needs,
224    a new function derived from rte_pktmbuf_init( ) can be created.
225
226.. _l2_fwd_app_dvr_init:
227
228Driver Initialization
229~~~~~~~~~~~~~~~~~~~~~
230
231The main part of the code in the main() function relates to the initialization of the driver.
232To fully understand this code, it is recommended to study the chapters that related to the Poll Mode Driver
233in the *DPDK Programmer's Guide* - Rel 1.4 EAR and the *DPDK API Reference*.
234
235.. code-block:: c
236
237    if (rte_eal_pci_probe() < 0)
238        rte_exit(EXIT_FAILURE, "Cannot probe PCI\n");
239
240    nb_ports = rte_eth_dev_count();
241
242    if (nb_ports == 0)
243        rte_exit(EXIT_FAILURE, "No Ethernet ports - bye\n");
244
245    /* reset l2fwd_dst_ports */
246
247    for (portid = 0; portid < RTE_MAX_ETHPORTS; portid++)
248        l2fwd_dst_ports[portid] = 0;
249
250    last_port = 0;
251
252    /*
253     * Each logical core is assigned a dedicated TX queue on each port.
254     */
255
256    for (portid = 0; portid < nb_ports; portid++) {
257        /* skip ports that are not enabled */
258
259        if ((l2fwd_enabled_port_mask & (1 << portid)) == 0)
260           continue;
261
262        if (nb_ports_in_mask % 2) {
263            l2fwd_dst_ports[portid] = last_port;
264            l2fwd_dst_ports[last_port] = portid;
265        }
266        else
267           last_port = portid;
268
269        nb_ports_in_mask++;
270
271        rte_eth_dev_info_get((uint8_t) portid, &dev_info);
272    }
273
274Observe that:
275
276*   rte_igb_pmd_init_all() simultaneously registers the driver as a PCI driver and as an Ethernet* Poll Mode Driver.
277
278*   rte_eal_pci_probe() parses the devices on the PCI bus and initializes recognized devices.
279
280The next step is to configure the RX and TX queues.
281For each port, there is only one RX queue (only one lcore is able to poll a given port).
282The number of TX queues depends on the number of available lcores.
283The rte_eth_dev_configure() function is used to configure the number of queues for a port:
284
285.. code-block:: c
286
287    ret = rte_eth_dev_configure((uint8_t)portid, 1, 1, &port_conf);
288    if (ret < 0)
289        rte_exit(EXIT_FAILURE, "Cannot configure device: "
290            "err=%d, port=%u\n",
291            ret, portid);
292
293The global configuration is stored in a static structure:
294
295.. code-block:: c
296
297    static const struct rte_eth_conf port_conf = {
298        .rxmode = {
299            .split_hdr_size = 0,
300            .header_split = 0,   /**< Header Split disabled */
301            .hw_ip_checksum = 0, /**< IP checksum offload disabled */
302            .hw_vlan_filter = 0, /**< VLAN filtering disabled */
303            .jumbo_frame = 0,    /**< Jumbo Frame Support disabled */
304            .hw_strip_crc= 0,    /**< CRC stripped by hardware */
305        },
306
307        .txmode = {
308            .mq_mode = ETH_DCB_NONE
309        },
310    };
311
312.. _l2_fwd_app_rx_init:
313
314RX Queue Initialization
315~~~~~~~~~~~~~~~~~~~~~~~
316
317The application uses one lcore to poll one or several ports, depending on the -q option,
318which specifies the number of queues per lcore.
319
320For example, if the user specifies -q 4, the application is able to poll four ports with one lcore.
321If there are 16 ports on the target (and if the portmask argument is -p ffff ),
322the application will need four lcores to poll all the ports.
323
324.. code-block:: c
325
326    ret = rte_eth_rx_queue_setup((uint8_t) portid, 0, nb_rxd, SOCKET0, &rx_conf, l2fwd_pktmbuf_pool);
327    if (ret < 0)
328
329        rte_exit(EXIT_FAILURE, "rte_eth_rx_queue_setup: "
330            "err=%d, port=%u\n",
331            ret, portid);
332
333The list of queues that must be polled for a given lcore is stored in a private structure called struct lcore_queue_conf.
334
335.. code-block:: c
336
337    struct lcore_queue_conf {
338        unsigned n_rx_port;
339        unsigned rx_port_list[MAX_RX_QUEUE_PER_LCORE];
340        struct mbuf_table tx_mbufs[L2FWD_MAX_PORTS];
341    } rte_cache_aligned;
342
343    struct lcore_queue_conf lcore_queue_conf[RTE_MAX_LCORE];
344
345The values n_rx_port and rx_port_list[] are used in the main packet processing loop
346(see :ref:`l2_fwd_app_rx_tx_packets`).
347
348The global configuration for the RX queues is stored in a static structure:
349
350.. code-block:: c
351
352    static const struct rte_eth_rxconf rx_conf = {
353        .rx_thresh = {
354            .pthresh = RX_PTHRESH,
355            .hthresh = RX_HTHRESH,
356            .wthresh = RX_WTHRESH,
357        },
358    };
359
360.. _l2_fwd_app_tx_init:
361
362TX Queue Initialization
363~~~~~~~~~~~~~~~~~~~~~~~
364
365Each lcore should be able to transmit on any port. For every port, a single TX queue is initialized.
366
367.. code-block:: c
368
369    /* init one TX queue on each port */
370
371    fflush(stdout);
372
373    ret = rte_eth_tx_queue_setup((uint8_t) portid, 0, nb_txd, rte_eth_dev_socket_id(portid), &tx_conf);
374    if (ret < 0)
375        rte_exit(EXIT_FAILURE, "rte_eth_tx_queue_setup:err=%d, port=%u\n", ret, (unsigned) portid);
376
377The global configuration for TX queues is stored in a static structure:
378
379.. code-block:: c
380
381    static const struct rte_eth_txconf tx_conf = {
382        .tx_thresh = {
383            .pthresh = TX_PTHRESH,
384            .hthresh = TX_HTHRESH,
385            .wthresh = TX_WTHRESH,
386        },
387        .tx_free_thresh = RTE_TEST_TX_DESC_DEFAULT + 1, /* disable feature */
388    };
389
390.. _l2_fwd_app_rx_tx_packets:
391
392Receive, Process and Transmit Packets
393~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
394
395In the l2fwd_main_loop() function, the main task is to read ingress packets from the RX queues.
396This is done using the following code:
397
398.. code-block:: c
399
400    /*
401     * Read packet from RX queues
402     */
403
404    for (i = 0; i < qconf->n_rx_port; i++) {
405        portid = qconf->rx_port_list[i];
406        nb_rx = rte_eth_rx_burst((uint8_t) portid, 0,  pkts_burst, MAX_PKT_BURST);
407
408        for (j = 0; j < nb_rx; j++) {
409            m = pkts_burst[j];
410            rte_prefetch0[rte_pktmbuf_mtod(m, void *)); l2fwd_simple_forward(m, portid);
411        }
412    }
413
414Packets are read in a burst of size MAX_PKT_BURST.
415The rte_eth_rx_burst() function writes the mbuf pointers in a local table and returns the number of available mbufs in the table.
416
417Then, each mbuf in the table is processed by the l2fwd_simple_forward() function.
418The processing is very simple: process the TX port from the RX port, then replace the source and destination MAC addresses.
419
420.. note::
421
422    In the following code, one line for getting the output port requires some explanation.
423
424During the initialization process, a static array of destination ports (l2fwd_dst_ports[]) is filled such that for each source port,
425a destination port is assigned that is either the next or previous enabled port from the portmask.
426Naturally, the number of ports in the portmask must be even, otherwise, the application exits.
427
428.. code-block:: c
429
430    static void
431    l2fwd_simple_forward(struct rte_mbuf *m, unsigned portid)
432    {
433        struct ether_hdr *eth;
434        void *tmp;
435        unsigned dst_port;
436
437        dst_port = l2fwd_dst_ports[portid];
438
439        eth = rte_pktmbuf_mtod(m, struct ether_hdr *);
440
441        /* 02:00:00:00:00:xx */
442
443        tmp = &eth->d_addr.addr_bytes[0];
444
445        *((uint64_t *)tmp) = 0x000000000002 + ((uint64_t) dst_port << 40);
446
447        /* src addr */
448
449        ether_addr_copy(&l2fwd_ports_eth_addr[dst_port], &eth->s_addr);
450
451        l2fwd_send_packet(m, (uint8_t) dst_port);
452    }
453
454Then, the packet is sent using the l2fwd_send_packet (m, dst_port) function.
455For this test application, the processing is exactly the same for all packets arriving on the same RX port.
456Therefore, it would have been possible to call the l2fwd_send_burst() function directly from the main loop
457to send all the received packets on the same TX port,
458using the burst-oriented send function, which is more efficient.
459
460However, in real-life applications (such as, L3 routing),
461packet N is not necessarily forwarded on the same port as packet N-1.
462The application is implemented to illustrate that, so the same approach can be reused in a more complex application.
463
464The l2fwd_send_packet() function stores the packet in a per-lcore and per-txport table.
465If the table is full, the whole packets table is transmitted using the l2fwd_send_burst() function:
466
467.. code-block:: c
468
469    /* Send the packet on an output interface */
470
471    static int
472    l2fwd_send_packet(struct rte_mbuf *m, uint8_t port)
473    {
474        unsigned lcore_id, len;
475        struct lcore_queue_conf *qconf;
476
477        lcore_id = rte_lcore_id();
478        qconf = &lcore_queue_conf[lcore_id];
479        len = qconf->tx_mbufs[port].len;
480        qconf->tx_mbufs[port].m_table[len] = m;
481        len++;
482
483        /* enough pkts to be sent */
484
485        if (unlikely(len == MAX_PKT_BURST)) {
486            l2fwd_send_burst(qconf, MAX_PKT_BURST, port);
487            len = 0;
488        }
489
490        qconf->tx_mbufs[port].len = len; return 0;
491    }
492
493To ensure that no packets remain in the tables, each lcore does a draining of TX queue in its main loop.
494This technique introduces some latency when there are not many packets to send,
495however it improves performance:
496
497.. code-block:: c
498
499    cur_tsc = rte_rdtsc();
500
501    /*
502     *   TX burst queue drain
503     */
504
505    diff_tsc = cur_tsc - prev_tsc;
506
507    if (unlikely(diff_tsc > drain_tsc)) {
508        for (portid = 0; portid < RTE_MAX_ETHPORTS; portid++) {
509            if (qconf->tx_mbufs[portid].len == 0)
510                continue;
511
512            l2fwd_send_burst(&lcore_queue_conf[lcore_id], qconf->tx_mbufs[portid].len, (uint8_t) portid);
513
514            qconf->tx_mbufs[portid].len = 0;
515        }
516
517        /* if timer is enabled */
518
519        if (timer_period > 0) {
520            /* advance the timer */
521
522            timer_tsc += diff_tsc;
523
524            /* if timer has reached its timeout */
525
526            if (unlikely(timer_tsc >= (uint64_t) timer_period)) {
527                /* do this only on master core */
528
529                if (lcore_id == rte_get_master_lcore()) {
530                    print_stats();
531
532                    /* reset the timer */
533                    timer_tsc = 0;
534                }
535            }
536        }
537
538        prev_tsc = cur_tsc;
539    }
540