build_dpdk.rst revision 97f17497
1..  BSD LICENSE
2    Copyright(c) 2010-2014 Intel Corporation. All rights reserved.
3    All rights reserved.
4
5    Redistribution and use in source and binary forms, with or without
6    modification, are permitted provided that the following conditions
7    are met:
8
9    * Redistributions of source code must retain the above copyright
10    notice, this list of conditions and the following disclaimer.
11    * Redistributions in binary form must reproduce the above copyright
12    notice, this list of conditions and the following disclaimer in
13    the documentation and/or other materials provided with the
14    distribution.
15    * Neither the name of Intel Corporation nor the names of its
16    contributors may be used to endorse or promote products derived
17    from this software without specific prior written permission.
18
19    THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
20    "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
21    LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
22    A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
23    OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
24    SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
25    LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
26    DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
27    THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
28    (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
29    OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
30
31.. _building_from_source:
32
33Compiling the DPDK Target from Source
34=====================================
35
36System Requirements
37-------------------
38
39The DPDK and its applications require the GNU make system (gmake)
40to build on FreeBSD. Optionally, gcc may also be used in place of clang
41to build the DPDK, in which case it too must be installed prior to
42compiling the DPDK. The installation of these tools is covered in this
43section.
44
45Compiling the DPDK requires the FreeBSD kernel sources, which should be
46included during the installation of FreeBSD on the development platform.
47The DPDK also requires the use of FreeBSD ports to compile and function.
48
49To use the FreeBSD ports system, it is required to update and extract the FreeBSD
50ports tree by issuing the following commands:
51
52.. code-block:: console
53
54    portsnap fetch
55    portsnap extract
56
57If the environment requires proxies for external communication, these can be set
58using:
59
60.. code-block:: console
61
62    setenv http_proxy <my_proxy_host>:<port>
63    setenv ftp_proxy <my_proxy_host>:<port>
64
65The FreeBSD ports below need to be installed prior to building the DPDK.
66In general these can be installed using the following set of commands::
67
68   cd /usr/ports/<port_location>
69
70   make config-recursive
71
72   make install
73
74   make clean
75
76Each port location can be found using::
77
78   whereis <port_name>
79
80The ports required and their locations are as follows:
81
82* dialog4ports: ``/usr/ports/ports-mgmt/dialog4ports``
83
84* GNU make(gmake): ``/usr/ports/devel/gmake``
85
86* coreutils: ``/usr/ports/sysutils/coreutils``
87
88For compiling and using the DPDK with gcc, the compiler must be installed
89from the ports collection:
90
91* gcc: version 4.8 is recommended ``/usr/ports/lang/gcc48``.
92  Ensure that ``CPU_OPTS`` is selected (default is OFF).
93
94When running the make config-recursive command, a dialog may be presented to the
95user. For the installation of the DPDK, the default options were used.
96
97.. note::
98
99    To avoid multiple dialogs being presented to the user during make install,
100    it is advisable before running the make install command to re-run the
101    make config-recursive command until no more dialogs are seen.
102
103
104Install the DPDK and Browse Sources
105-----------------------------------
106
107First, uncompress the archive and move to the DPDK source directory:
108
109.. code-block:: console
110
111    unzip DPDK-<version>.zip
112    cd DPDK-<version>
113
114    ls
115    app/ config/ examples/ lib/ LICENSE.GPL LICENSE.LGPL Makefile
116    mk/ scripts/ tools/
117
118The DPDK is composed of several directories:
119
120*   lib: Source code of DPDK libraries
121
122*   app: Source code of DPDK applications (automatic tests)
123
124*   examples: Source code of DPDK applications
125
126*   config, tools, scripts, mk: Framework-related makefiles, scripts and configuration
127
128Installation of the DPDK Target Environments
129--------------------------------------------
130
131The format of a DPDK target is::
132
133   ARCH-MACHINE-EXECENV-TOOLCHAIN
134
135Where:
136
137* ``ARCH`` is: ``x86_64``
138
139* ``MACHINE`` is: ``native``
140
141* ``EXECENV`` is: ``bsdapp``
142
143* ``TOOLCHAIN`` is: ``gcc`` | ``clang``
144
145The configuration files for the DPDK targets can be found in the DPDK/config
146directory in the form of::
147
148    defconfig_ARCH-MACHINE-EXECENV-TOOLCHAIN
149
150.. note::
151
152   Configuration files are provided with the ``RTE_MACHINE`` optimization level set.
153   Within the configuration files, the ``RTE_MACHINE`` configuration value is set
154   to native, which means that the compiled software is tuned for the platform
155   on which it is built.  For more information on this setting, and its
156   possible values, see the *DPDK Programmers Guide*.
157
158To make the target, use ``gmake install T=<target>``.
159
160For example to compile for FreeBSD use:
161
162.. code-block:: console
163
164    gmake install T=x86_64-native-bsdapp-clang
165
166.. note::
167
168   If the compiler binary to be used does not correspond to that given in the
169   TOOLCHAIN part of the target, the compiler command may need to be explicitly
170   specified. For example, if compiling for gcc, where the gcc binary is called
171   gcc4.8, the command would need to be ``gmake install T=<target> CC=gcc4.8``.
172
173Browsing the Installed DPDK Environment Target
174----------------------------------------------
175
176Once a target is created, it contains all the libraries and header files for the
177DPDK environment that are required to build customer applications.
178In addition, the test and testpmd applications are built under the build/app
179directory, which may be used for testing.  A kmod directory is also present that
180contains the kernel modules to install:
181
182.. code-block:: console
183
184    ls x86_64-native-bsdapp-gcc
185
186    app build hostapp include kmod lib Makefile
187
188
189.. _loading_contigmem:
190
191Loading the DPDK contigmem Module
192---------------------------------
193
194To run a DPDK application, physically contiguous memory is required.
195In the absence of non-transparent superpages, the included sources for the
196contigmem kernel module provides the ability to present contiguous blocks of
197memory for the DPDK to use. The contigmem module must be loaded into the
198running kernel before any DPDK is run.  The module is found in the kmod
199sub-directory of the DPDK target directory.
200
201The amount of physically contiguous memory along with the number of physically
202contiguous blocks to be reserved by the module can be set at runtime prior to
203module loading using:
204
205.. code-block:: console
206
207    kenv hw.contigmem.num_buffers=n
208    kenv hw.contigmem.buffer_size=m
209
210The kernel environment variables can also be specified during boot by placing the
211following in ``/boot/loader.conf``::
212
213    hw.contigmem.num_buffers=n hw.contigmem.buffer_size=m
214
215The variables can be inspected using the following command:
216
217.. code-block:: console
218
219    sysctl -a hw.contigmem
220
221Where n is the number of blocks and m is the size in bytes of each area of
222contiguous memory.  A default of two buffers of size 1073741824 bytes (1 Gigabyte)
223each is set during module load if they are not specified in the environment.
224
225The module can then be loaded using kldload (assuming that the current directory
226is the DPDK target directory):
227
228.. code-block:: console
229
230    kldload ./kmod/contigmem.ko
231
232It is advisable to include the loading of the contigmem module during the boot
233process to avoid issues with potential memory fragmentation during later system
234up time.  This can be achieved by copying the module to the ``/boot/kernel/``
235directory and placing the following into ``/boot/loader.conf``::
236
237    contigmem_load="YES"
238
239.. note::
240
241    The contigmem_load directive should be placed after any definitions of
242    ``hw.contigmem.num_buffers`` and ``hw.contigmem.buffer_size`` if the default values
243    are not to be used.
244
245An error such as:
246
247.. code-block:: console
248
249    kldload: can't load ./x86_64-native-bsdapp-gcc/kmod/contigmem.ko:
250             Exec format error
251
252is generally attributed to not having enough contiguous memory
253available and can be verified via dmesg or ``/var/log/messages``:
254
255.. code-block:: console
256
257    kernel: contigmalloc failed for buffer <n>
258
259To avoid this error, reduce the number of buffers or the buffer size.
260
261.. _loading_nic_uio:
262
263Loading the DPDK nic_uio Module
264-------------------------------
265
266After loading the contigmem module, the ``nic_uio must`` also be loaded into the
267running kernel prior to running any DPDK application.  This module must
268be loaded using the kldload command as shown below (assuming that the current
269directory is the DPDK target directory).
270
271.. code-block:: console
272
273    kldload ./kmod/nic_uio.ko
274
275.. note::
276
277    If the ports to be used are currently bound to a existing kernel driver
278    then the ``hw.nic_uio.bdfs sysctl`` value will need to be set before loading the
279    module. Setting this value is described in the next section below.
280
281Currently loaded modules can be seen by using the ``kldstat`` command and a module
282can be removed from the running kernel by using ``kldunload <module_name>``.
283
284To load the module during boot, copy the ``nic_uio`` module to ``/boot/kernel``
285and place the following into ``/boot/loader.conf``::
286
287    nic_uio_load="YES"
288
289.. note::
290
291    ``nic_uio_load="YES"`` must appear after the contigmem_load directive, if it exists.
292
293By default, the ``nic_uio`` module will take ownership of network ports if they are
294recognized DPDK devices and are not owned by another module. However, since
295the FreeBSD kernel includes support, either built-in, or via a separate driver
296module, for most network card devices, it is likely that the ports to be used are
297already bound to a driver other than ``nic_uio``. The following sub-section describe
298how to query and modify the device ownership of the ports to be used by
299DPDK applications.
300
301.. _binding_network_ports:
302
303Binding Network Ports to the nic_uio Module
304~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
305
306Device ownership can be viewed using the pciconf -l command. The example below shows
307four Intel® 82599 network ports under ``if_ixgbe`` module ownership.
308
309.. code-block:: console
310
311    pciconf -l
312    ix0@pci0:1:0:0: class=0x020000 card=0x00038086 chip=0x10fb8086 rev=0x01 hdr=0x00
313    ix1@pci0:1:0:1: class=0x020000 card=0x00038086 chip=0x10fb8086 rev=0x01 hdr=0x00
314    ix2@pci0:2:0:0: class=0x020000 card=0x00038086 chip=0x10fb8086 rev=0x01 hdr=0x00
315    ix3@pci0:2:0:1: class=0x020000 card=0x00038086 chip=0x10fb8086 rev=0x01 hdr=0x00
316
317The first column constitutes three components:
318
319#. Device name: ``ixN``
320
321#. Unit name: ``pci0``
322
323#. Selector (Bus:Device:Function): ``1:0:0``
324
325Where no driver is associated with a device, the device name will be ``none``.
326
327By default, the FreeBSD kernel will include built-in drivers for the most common
328devices; a kernel rebuild would normally be required to either remove the drivers
329or configure them as loadable modules.
330
331To avoid building a custom kernel, the ``nic_uio`` module can detach a network port
332from its current device driver. This is achieved by setting the ``hw.nic_uio.bdfs``
333kernel environment variable prior to loading ``nic_uio``, as follows::
334
335    hw.nic_uio.bdfs="b:d:f,b:d:f,..."
336
337Where a comma separated list of selectors is set, the list must not contain any
338whitespace.
339
340For example to re-bind ``ix2@pci0:2:0:0`` and ``ix3@pci0:2:0:1`` to the ``nic_uio`` module
341upon loading, use the following command::
342
343    kenv hw.nic_uio.bdfs="2:0:0,2:0:1"
344
345The variable can also be specified during boot by placing the following into
346``/boot/loader.conf``, before the previously-described ``nic_uio_load`` line - as
347shown::
348
349    hw.nic_uio.bdfs="2:0:0,2:0:1"
350    nic_uio_load="YES"
351
352Binding Network Ports Back to their Original Kernel Driver
353~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
354
355If the original driver for a network port has been compiled into the kernel,
356it is necessary to reboot FreeBSD to restore the original device binding. Before
357doing so, update or remove the ``hw.nic_uio.bdfs`` in ``/boot/loader.conf``.
358
359If rebinding to a driver that is a loadable module, the network port binding can
360be reset without rebooting. To do so, unload both the target kernel module and the
361``nic_uio`` module, modify or clear the ``hw.nic_uio.bdfs`` kernel environment (kenv)
362value, and reload the two drivers - first the original kernel driver, and then
363the ``nic_uio driver``. Note: the latter does not need to be reloaded unless there are
364ports that are still to be bound to it.
365
366Example commands to perform these steps are shown below:
367
368.. code-block:: console
369
370    kldunload nic_uio
371    kldunload <original_driver>
372
373    # To clear the value completely:
374    kenv -u hw.nic_uio.bdfs
375
376    # To update the list of ports to bind:
377    kenv hw.nic_uio.bdfs="b:d:f,b:d:f,..."
378
379    kldload <original_driver>
380
381    kldload nic_uio  # optional
382