l2_forward_cat.rst revision 97f17497
1..  BSD LICENSE
2    Copyright(c) 2016 Intel Corporation. All rights reserved.
3    All rights reserved.
4
5    Redistribution and use in source and binary forms, with or without
6    modification, are permitted provided that the following conditions
7    are met:
8
9    * Redistributions of source code must retain the above copyright
10    notice, this list of conditions and the following disclaimer.
11    * Redistributions in binary form must reproduce the above copyright
12    notice, this list of conditions and the following disclaimer in
13    the documentation and/or other materials provided with the
14    distribution.
15    * Neither the name of Intel Corporation nor the names of its
16    contributors may be used to endorse or promote products derived
17    from this software without specific prior written permission.
18
19    THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
20    "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
21    LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
22    A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
23    OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
24    SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
25    LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
26    DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
27    THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
28    (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
29    OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
30
31
32L2 Forwarding Sample Application with Cache Allocation Technology (CAT)
33=======================================================================
34
35Basic Forwarding sample application is a simple *skeleton* example of
36a forwarding application. It has been extended to make use of CAT via extended
37command line options and linking against the libpqos library.
38
39It is intended as a demonstration of the basic components of a DPDK forwarding
40application and use of the libpqos library to program CAT.
41For more detailed implementations see the L2 and L3 forwarding
42sample applications.
43
44CAT and Code Data Prioritization (CDP) features allow management of the CPU's
45last level cache. CAT introduces classes of service (COS) that are essentially
46bitmasks. In current CAT implementations, a bit in a COS bitmask corresponds to
47one cache way in last level cache.
48A CPU core is always assigned to one of the CAT classes.
49By programming CPU core assignment and COS bitmasks, applications can be given
50exclusive, shared, or mixed access to the CPU's last level cache.
51CDP extends CAT so that there are two bitmasks per COS,
52one for data and one for code.
53The number of classes and number of valid bits in a COS bitmask is CPU model
54specific and COS bitmasks need to be contiguous. Sample code calls this bitmask
55``cbm`` or capacity bitmask.
56By default, after reset, all CPU cores are assigned to COS 0 and all classes
57are programmed to allow fill into all cache ways.
58CDP is off by default.
59
60For more information about CAT please see:
61
62* https://github.com/01org/intel-cmt-cat
63
64White paper demonstrating example use case:
65
66* `Increasing Platform Determinism with Platform Quality of Service for the Data Plane Development Kit <http://www.intel.com/content/www/us/en/communications/increasing-platform-determinism-pqos-dpdk-white-paper.html>`_
67
68Compiling the Application
69-------------------------
70
71Requires ``libpqos`` from Intel's
72`intel-cmt-cat software package <https://github.com/01org/intel-cmt-cat>`_
73hosted on GitHub repository. For installation notes, please see ``README`` file.
74
75GIT:
76
77* https://github.com/01org/intel-cmt-cat
78
79To compile the application export the path to PQoS lib
80and the DPDK source tree and go to the example directory:
81
82.. code-block:: console
83
84    export PQOS_INSTALL_PATH=/path/to/libpqos
85    export RTE_SDK=/path/to/rte_sdk
86
87    cd ${RTE_SDK}/examples/l2fwd-cat
88
89Set the target, for example:
90
91.. code-block:: console
92
93    export RTE_TARGET=x86_64-native-linuxapp-gcc
94
95See the *DPDK Getting Started* Guide for possible ``RTE_TARGET`` values.
96
97Build the application as follows:
98
99.. code-block:: console
100
101    make
102
103
104Running the Application
105-----------------------
106
107To run the example in a ``linuxapp`` environment and enable CAT on cpus 0-2:
108
109.. code-block:: console
110
111    ./build/l2fwd-cat -c 2 -n 4 -- --l3ca="0x3@(0-2)"
112
113or to enable CAT and CDP on cpus 1,3:
114
115.. code-block:: console
116
117    ./build/l2fwd-cat -c 2 -n 4 -- --l3ca="(0x00C00,0x00300)@(1,3)"
118
119If CDP is not supported it will fail with following error message:
120
121.. code-block:: console
122
123    PQOS: CDP requested but not supported.
124    PQOS: Requested CAT configuration is not valid!
125    PQOS: Shutting down PQoS library...
126    EAL: Error - exiting with code: 1
127      Cause: PQOS: L3CA init failed!
128
129The option to enable CAT is:
130
131* ``--l3ca='<common_cbm@cpus>[,<(code_cbm,data_cbm)@cpus>...]'``:
132
133  where ``cbm`` stands for capacity bitmask and must be expressed in
134  hexadecimal form.
135
136  ``common_cbm`` is a single mask, for a CDP enabled system, a group of two
137  masks (``code_cbm`` and ``data_cbm``) is used.
138
139  ``(`` and ``)`` are necessary if it's a group.
140
141  ``cpus`` could be a single digit/range or a group and must be expressed in
142  decimal form.
143
144  ``(`` and ``)`` are necessary if it's a group.
145
146  e.g. ``--l3ca='0x00F00@(1,3),0x0FF00@(4-6),0xF0000@7'``
147
148  * cpus 1 and 3 share its 4 ways with cpus 4, 5 and 6;
149
150  * cpus 4, 5 and 6 share half (4 out of 8 ways) of its L3 with cpus 1 and 3;
151
152  * cpus 4, 5 and 6 have exclusive access to 4 out of 8 ways;
153
154  * cpu 7 has exclusive access to all of its 4 ways;
155
156  e.g. ``--l3ca='(0x00C00,0x00300)@(1,3)'`` for CDP enabled system
157
158  * cpus 1 and 3 have access to 2 ways for code and 2 ways for data, code and
159    data ways are not overlapping.
160
161
162Refer to *DPDK Getting Started Guide* for general information on running
163applications and the Environment Abstraction Layer (EAL) options.
164
165
166To reset or list CAT configuration and control CDP please use ``pqos`` tool
167from Intel's
168`intel-cmt-cat software package <https://github.com/01org/intel-cmt-cat>`_.
169
170To enabled or disable CDP:
171
172.. code-block:: console
173
174    sudo ./pqos -S cdp-on
175
176    sudo ./pqos -S cdp-off
177
178to reset CAT configuration:
179
180.. code-block:: console
181
182    sudo ./pqos -R
183
184to list CAT config:
185
186.. code-block:: console
187
188    sudo ./pqos -s
189
190For more info about ``pqos`` tool please see its man page or
191`intel-cmt-cat wiki <https://github.com/01org/intel-cmt-cat/wiki>`_.
192
193
194Explanation
195-----------
196
197The following sections provide an explanation of the main components of the
198code.
199
200All DPDK library functions used in the sample code are prefixed with ``rte_``
201and are explained in detail in the *DPDK API Documentation*.
202
203
204The Main Function
205~~~~~~~~~~~~~~~~~
206
207The ``main()`` function performs the initialization and calls the execution
208threads for each lcore.
209
210The first task is to initialize the Environment Abstraction Layer (EAL).  The
211``argc`` and ``argv`` arguments are provided to the ``rte_eal_init()``
212function. The value returned is the number of parsed arguments:
213
214.. code-block:: c
215
216    int ret = rte_eal_init(argc, argv);
217    if (ret < 0)
218        rte_exit(EXIT_FAILURE, "Error with EAL initialization\n");
219
220The next task is to initialize the PQoS library and configure CAT. The
221``argc`` and ``argv`` arguments are provided to the ``cat_init()``
222function. The value returned is the number of parsed arguments:
223
224.. code-block:: c
225
226    int ret = cat_init(argc, argv);
227    if (ret < 0)
228        rte_exit(EXIT_FAILURE, "PQOS: L3CA init failed!\n");
229
230``cat_init()`` is a wrapper function which parses the command, validates
231the requested parameters and configures CAT accordingly.
232
233Parsing of command line arguments is done in ``parse_args(...)``.
234libpqos is then initialized with the ``pqos_init(...)`` call. Next, libpqos is
235queried for system CPU information and L3CA capabilities via
236``pqos_cap_get(...)`` and ``pqos_cap_get_type(..., PQOS_CAP_TYPE_L3CA, ...)``
237calls. When all capability and topology information is collected, the requested
238CAT configuration is validated. A check is then performed (on per socket basis)
239for a sufficient number of un-associated COS. COS are selected and
240configured via the ``pqos_l3ca_set(...)`` call. Finally, COS are associated to
241relevant CPUs via ``pqos_l3ca_assoc_set(...)`` calls.
242
243``atexit(...)`` is used to register ``cat_exit(...)`` to be called on
244a clean exit. ``cat_exit(...)`` performs a simple CAT clean-up, by associating
245COS 0 to all involved CPUs via ``pqos_l3ca_assoc_set(...)`` calls.