link_status_intr.rst revision 8b25d1ad
1..  BSD LICENSE
2    Copyright(c) 2010-2014 Intel Corporation. All rights reserved.
3    All rights reserved.
4
5    Redistribution and use in source and binary forms, with or without
6    modification, are permitted provided that the following conditions
7    are met:
8
9    * Redistributions of source code must retain the above copyright
10    notice, this list of conditions and the following disclaimer.
11    * Redistributions in binary form must reproduce the above copyright
12    notice, this list of conditions and the following disclaimer in
13    the documentation and/or other materials provided with the
14    distribution.
15    * Neither the name of Intel Corporation nor the names of its
16    contributors may be used to endorse or promote products derived
17    from this software without specific prior written permission.
18
19    THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
20    "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
21    LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
22    A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
23    OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
24    SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
25    LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
26    DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
27    THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
28    (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
29    OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
30
31Link Status Interrupt Sample Application
32========================================
33
34The Link Status Interrupt sample application is a simple example of packet processing using
35the Data Plane Development Kit (DPDK) that
36demonstrates how network link status changes for a network port can be captured and
37used by a DPDK application.
38
39Overview
40--------
41
42The Link Status Interrupt sample application registers a user space callback for the link status interrupt of each port
43and performs L2 forwarding for each packet that is received on an RX_PORT.
44The following operations are performed:
45
46*   RX_PORT and TX_PORT are paired with available ports one-by-one according to the core mask
47
48*   The source MAC address is replaced by the TX_PORT MAC address
49
50*   The destination MAC address is replaced by 02:00:00:00:00:TX_PORT_ID
51
52This application can be used to demonstrate the usage of link status interrupt and its user space callbacks
53and the behavior of L2 forwarding each time the link status changes.
54
55Compiling the Application
56-------------------------
57
58#.  Go to the example directory:
59
60    .. code-block:: console
61
62        export RTE_SDK=/path/to/rte_sdk
63        cd ${RTE_SDK}/examples/link_status_interrupt
64
65#.  Set the target (a default target is used if not specified). For example:
66
67    .. code-block:: console
68
69        export RTE_TARGET=x86_64-native-linuxapp-gcc
70
71    See the *DPDK Getting Started Guide* for possible RTE_TARGET values.
72
73#.  Build the application:
74
75    .. code-block:: console
76
77        make
78
79.. note::
80
81    The compiled application is written to the build subdirectory.
82    To have the application written to a different location,
83    the O=/path/to/build/directory option may be specified on the make command line.
84
85Running the Application
86-----------------------
87
88The application requires a number of command line options:
89
90.. code-block:: console
91
92    ./build/link_status_interrupt [EAL options] -- -p PORTMASK [-q NQ][-T PERIOD]
93
94where,
95
96*   -p PORTMASK: A hexadecimal bitmask of the ports to configure
97
98*   -q NQ: A number of queues (=ports) per lcore (default is 1)
99
100*   -T PERIOD: statistics will be refreshed each PERIOD seconds (0 to disable, 10 default)
101
102To run the application in a linuxapp environment with 4 lcores, 4 memory channels, 16 ports and 8 RX queues per lcore,
103issue the command:
104
105.. code-block:: console
106
107    $ ./build/link_status_interrupt -c f -n 4-- -q 8 -p ffff
108
109Refer to the *DPDK Getting Started Guide* for general information on running applications
110and the Environment Abstraction Layer (EAL) options.
111
112Explanation
113-----------
114
115The following sections provide some explanation of the code.
116
117Command Line Arguments
118~~~~~~~~~~~~~~~~~~~~~~
119
120The Link Status Interrupt sample application takes specific parameters,
121in addition to Environment Abstraction Layer (EAL) arguments (see Section `Running the Application`_).
122
123Command line parsing is done in the same way as it is done in the L2 Forwarding Sample Application.
124See :ref:`l2_fwd_app_cmd_arguments` for more information.
125
126Mbuf Pool Initialization
127~~~~~~~~~~~~~~~~~~~~~~~~
128
129Mbuf pool initialization is done in the same way as it is done in the L2 Forwarding Sample Application.
130See :ref:`l2_fwd_app_mbuf_init` for more information.
131
132Driver Initialization
133~~~~~~~~~~~~~~~~~~~~~
134
135The main part of the code in the main() function relates to the initialization of the driver.
136To fully understand this code, it is recommended to study the chapters that related to the Poll Mode Driver in the
137*DPDK Programmer's Guide and the DPDK API Reference*.
138
139.. code-block:: c
140
141    if (rte_eal_pci_probe() < 0)
142        rte_exit(EXIT_FAILURE, "Cannot probe PCI\n");
143
144    nb_ports = rte_eth_dev_count();
145    if (nb_ports == 0)
146        rte_exit(EXIT_FAILURE, "No Ethernet ports - bye\n");
147
148    /*
149     * Each logical core is assigned a dedicated TX queue on each port.
150     */
151
152    for (portid = 0; portid < nb_ports; portid++) {
153        /* skip ports that are not enabled */
154
155        if ((lsi_enabled_port_mask & (1 << portid)) == 0)
156            continue;
157
158        /* save the destination port id */
159
160        if (nb_ports_in_mask % 2) {
161            lsi_dst_ports[portid] = portid_last;
162            lsi_dst_ports[portid_last] = portid;
163        }
164        else
165            portid_last = portid;
166
167        nb_ports_in_mask++;
168
169        rte_eth_dev_info_get((uint8_t) portid, &dev_info);
170    }
171
172Observe that:
173
174*   rte_eal_pci_probe()  parses the devices on the PCI bus and initializes recognized devices.
175
176The next step is to configure the RX and TX queues.
177For each port, there is only one RX queue (only one lcore is able to poll a given port).
178The number of TX queues depends on the number of available lcores.
179The rte_eth_dev_configure() function is used to configure the number of queues for a port:
180
181.. code-block:: c
182
183    ret = rte_eth_dev_configure((uint8_t) portid, 1, 1, &port_conf);
184    if (ret < 0)
185        rte_exit(EXIT_FAILURE, "Cannot configure device: err=%d, port=%u\n", ret, portid);
186
187The global configuration is stored in a static structure:
188
189.. code-block:: c
190
191    static const struct rte_eth_conf port_conf = {
192        .rxmode = {
193            .split_hdr_size = 0,
194            .header_split = 0,   /**< Header Split disabled */
195            .hw_ip_checksum = 0, /**< IP checksum offload disabled */
196            .hw_vlan_filter = 0, /**< VLAN filtering disabled */
197            .hw_strip_crc= 0,    /**< CRC stripped by hardware */
198        },
199        .txmode = {},
200        .intr_conf = {
201            .lsc = 1, /**< link status interrupt feature enabled */
202        },
203    };
204
205Configuring lsc to 0 (the default) disables the generation of any link status change interrupts in kernel space
206and no user space interrupt event is received.
207The public interface rte_eth_link_get() accesses the NIC registers directly to update the link status.
208Configuring lsc to non-zero enables the generation of link status change interrupts in kernel space
209when a link status change is present and calls the user space callbacks registered by the application.
210The public interface rte_eth_link_get() just reads the link status in a global structure
211that would be updated in the interrupt host thread only.
212
213Interrupt Callback Registration
214~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
215
216The application can register one or more callbacks to a specific port and interrupt event.
217An example callback function that has been written as indicated below.
218
219.. code-block:: c
220
221    static void
222    lsi_event_callback(uint8_t port_id, enum rte_eth_event_type type, void *param)
223    {
224        struct rte_eth_link link;
225
226        RTE_SET_USED(param);
227
228        printf("\n\nIn registered callback...\n");
229
230        printf("Event type: %s\n", type == RTE_ETH_EVENT_INTR_LSC ? "LSC interrupt" : "unknown event");
231
232        rte_eth_link_get_nowait(port_id, &link);
233
234        if (link.link_status) {
235            printf("Port %d Link Up - speed %u Mbps - %s\n\n", port_id, (unsigned)link.link_speed,
236                  (link.link_duplex == ETH_LINK_FULL_DUPLEX) ? ("full-duplex") : ("half-duplex"));
237        } else
238            printf("Port %d Link Down\n\n", port_id);
239    }
240
241This function is called when a link status interrupt is present for the right port.
242The port_id indicates which port the interrupt applies to.
243The type parameter identifies the interrupt event type,
244which currently can be RTE_ETH_EVENT_INTR_LSC only, but other types can be added in the future.
245The param parameter is the address of the parameter for the callback.
246This function should be implemented with care since it will be called in the interrupt host thread,
247which is different from the main thread of its caller.
248
249The application registers the lsi_event_callback and a NULL parameter to the link status interrupt event on each port:
250
251.. code-block:: c
252
253    rte_eth_dev_callback_register((uint8_t)portid, RTE_ETH_EVENT_INTR_LSC, lsi_event_callback, NULL);
254
255This registration can be done only after calling the rte_eth_dev_configure() function and before calling any other function.
256If lsc is initialized with 0, the callback is never called since no interrupt event would ever be present.
257
258RX Queue Initialization
259~~~~~~~~~~~~~~~~~~~~~~~
260
261The application uses one lcore to poll one or several ports, depending on the -q option,
262which specifies the number of queues per lcore.
263
264For example, if the user specifies -q 4, the application is able to poll four ports with one lcore.
265If there are 16 ports on the target (and if the portmask argument is -p ffff),
266the application will need four lcores to poll all the ports.
267
268.. code-block:: c
269
270    ret = rte_eth_rx_queue_setup((uint8_t) portid, 0, nb_rxd, SOCKET0, &rx_conf, lsi_pktmbuf_pool);
271    if (ret < 0)
272        rte_exit(EXIT_FAILURE, "rte_eth_rx_queue_setup: err=%d, port=%u\n", ret, portid);
273
274The list of queues that must be polled for a given lcore is stored in a private structure called struct lcore_queue_conf.
275
276.. code-block:: c
277
278    struct lcore_queue_conf {
279        unsigned n_rx_port;
280        unsigned rx_port_list[MAX_RX_QUEUE_PER_LCORE]; unsigned tx_queue_id;
281        struct mbuf_table tx_mbufs[LSI_MAX_PORTS];
282    } rte_cache_aligned;
283
284    struct lcore_queue_conf lcore_queue_conf[RTE_MAX_LCORE];
285
286The n_rx_port and rx_port_list[] fields are used in the main packet processing loop
287(see `Receive, Process and Transmit Packets`_).
288
289The global configuration for the RX queues is stored in a static structure:
290
291.. code-block:: c
292
293    static const struct rte_eth_rxconf rx_conf = {
294        .rx_thresh = {
295            .pthresh = RX_PTHRESH,
296            .hthresh = RX_HTHRESH,
297            .wthresh = RX_WTHRESH,
298        },
299    };
300
301TX Queue Initialization
302~~~~~~~~~~~~~~~~~~~~~~~
303
304Each lcore should be able to transmit on any port.
305For every port, a single TX queue is initialized.
306
307.. code-block:: c
308
309    /* init one TX queue logical core on each port */
310
311    fflush(stdout);
312
313    ret = rte_eth_tx_queue_setup(portid, 0, nb_txd, rte_eth_dev_socket_id(portid), &tx_conf);
314    if (ret < 0)
315        rte_exit(EXIT_FAILURE, "rte_eth_tx_queue_setup: err=%d,port=%u\n", ret, (unsigned) portid);
316
317The global configuration for TX queues is stored in a static structure:
318
319.. code-block:: c
320
321    static const struct rte_eth_txconf tx_conf = {
322        .tx_thresh = {
323            .pthresh = TX_PTHRESH,
324            .hthresh = TX_HTHRESH,
325            .wthresh = TX_WTHRESH,
326        },
327        .tx_free_thresh = RTE_TEST_TX_DESC_DEFAULT + 1, /* disable feature */
328    };
329
330Receive, Process and Transmit Packets
331~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
332
333In the lsi_main_loop() function, the main task is to read ingress packets from the RX queues.
334This is done using the following code:
335
336.. code-block:: c
337
338    /*
339     *   Read packet from RX queues
340     */
341
342    for (i = 0; i < qconf->n_rx_port; i++) {
343        portid = qconf->rx_port_list[i];
344        nb_rx = rte_eth_rx_burst((uint8_t) portid, 0, pkts_burst, MAX_PKT_BURST);
345        port_statistics[portid].rx += nb_rx;
346
347        for (j = 0; j < nb_rx; j++) {
348            m = pkts_burst[j];
349            rte_prefetch0(rte_pktmbuf_mtod(m, void *));
350            lsi_simple_forward(m, portid);
351        }
352    }
353
354Packets are read in a burst of size MAX_PKT_BURST.
355The rte_eth_rx_burst() function writes the mbuf pointers in a local table and returns the number of available mbufs in the table.
356
357Then, each mbuf in the table is processed by the lsi_simple_forward() function.
358The processing is very simple: processes the TX port from the RX port and then replaces the source and destination MAC addresses.
359
360.. note::
361
362    In the following code, the two lines for calculating the output port require some explanation.
363    If portId is even, the first line does nothing (as portid & 1 will be 0), and the second line adds 1.
364    If portId is odd, the first line subtracts one and the second line does nothing.
365    Therefore, 0 goes to 1, and 1 to 0, 2 goes to 3 and 3 to 2, and so on.
366
367.. code-block:: c
368
369    static void
370    lsi_simple_forward(struct rte_mbuf *m, unsigned portid)
371    {
372        struct ether_hdr *eth;
373        void *tmp;
374        unsigned dst_port = lsi_dst_ports[portid];
375
376        eth = rte_pktmbuf_mtod(m, struct ether_hdr *);
377
378        /* 02:00:00:00:00:xx */
379
380        tmp = &eth->d_addr.addr_bytes[0];
381
382        *((uint64_t *)tmp) = 0x000000000002 + (dst_port << 40);
383
384        /* src addr */
385        ether_addr_copy(&lsi_ports_eth_addr[dst_port], &eth->s_addr);
386
387        lsi_send_packet(m, dst_port);
388    }
389
390Then, the packet is sent using the lsi_send_packet(m, dst_port) function.
391For this test application, the processing is exactly the same for all packets arriving on the same RX port.
392Therefore, it would have been possible to call the lsi_send_burst() function directly from the main loop
393to send all the received packets on the same TX port using
394the burst-oriented send function, which is more efficient.
395
396However, in real-life applications (such as, L3 routing),
397packet N is not necessarily forwarded on the same port as packet N-1.
398The application is implemented to illustrate that so the same approach can be reused in a more complex application.
399
400The lsi_send_packet() function stores the packet in a per-lcore and per-txport table.
401If the table is full, the whole packets table is transmitted using the lsi_send_burst() function:
402
403.. code-block:: c
404
405    /* Send the packet on an output interface */
406
407    static int
408    lsi_send_packet(struct rte_mbuf *m, uint8_t port)
409    {
410        unsigned lcore_id, len;
411        struct lcore_queue_conf *qconf;
412
413        lcore_id = rte_lcore_id();
414        qconf = &lcore_queue_conf[lcore_id];
415        len = qconf->tx_mbufs[port].len;
416        qconf->tx_mbufs[port].m_table[len] = m;
417        len++;
418
419        /* enough pkts to be sent */
420
421        if (unlikely(len == MAX_PKT_BURST)) {
422            lsi_send_burst(qconf, MAX_PKT_BURST, port);
423            len = 0;
424        }
425        qconf->tx_mbufs[port].len = len;
426
427        return 0;
428    }
429
430To ensure that no packets remain in the tables, each lcore does a draining of the TX queue in its main loop.
431This technique introduces some latency when there are not many packets to send.
432However, it improves performance:
433
434.. code-block:: c
435
436    cur_tsc = rte_rdtsc();
437
438    /*
439     *    TX burst queue drain
440     */
441
442    diff_tsc = cur_tsc - prev_tsc;
443
444    if (unlikely(diff_tsc > drain_tsc)) {
445        /* this could be optimized (use queueid instead of * portid), but it is not called so often */
446
447        for (portid = 0; portid < RTE_MAX_ETHPORTS; portid++) {
448            if (qconf->tx_mbufs[portid].len == 0)
449                continue;
450
451            lsi_send_burst(&lcore_queue_conf[lcore_id],
452            qconf->tx_mbufs[portid].len, (uint8_t) portid);
453            qconf->tx_mbufs[portid].len = 0;
454        }
455
456        /* if timer is enabled */
457
458        if (timer_period > 0) {
459            /* advance the timer */
460
461            timer_tsc += diff_tsc;
462
463            /* if timer has reached its timeout */
464
465            if (unlikely(timer_tsc >= (uint64_t) timer_period)) {
466                /* do this only on master core */
467
468                if (lcore_id == rte_get_master_lcore()) {
469                    print_stats();
470
471                    /* reset the timer */
472                    timer_tsc = 0;
473                }
474            }
475        }
476        prev_tsc = cur_tsc;
477   }
478