build_dpdk.rst revision 6b3e017e
1..  BSD LICENSE
2    Copyright(c) 2010-2014 Intel Corporation. All rights reserved.
3    All rights reserved.
4
5    Redistribution and use in source and binary forms, with or without
6    modification, are permitted provided that the following conditions
7    are met:
8
9    * Redistributions of source code must retain the above copyright
10    notice, this list of conditions and the following disclaimer.
11    * Redistributions in binary form must reproduce the above copyright
12    notice, this list of conditions and the following disclaimer in
13    the documentation and/or other materials provided with the
14    distribution.
15    * Neither the name of Intel Corporation nor the names of its
16    contributors may be used to endorse or promote products derived
17    from this software without specific prior written permission.
18
19    THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
20    "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
21    LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
22    A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
23    OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
24    SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
25    LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
26    DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
27    THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
28    (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
29    OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
30
31.. _building_from_source:
32
33Compiling the DPDK Target from Source
34=====================================
35
36System Requirements
37-------------------
38
39The DPDK and its applications require the GNU make system (gmake)
40to build on FreeBSD. Optionally, gcc may also be used in place of clang
41to build the DPDK, in which case it too must be installed prior to
42compiling the DPDK. The installation of these tools is covered in this
43section.
44
45Compiling the DPDK requires the FreeBSD kernel sources, which should be
46included during the installation of FreeBSD on the development platform.
47The DPDK also requires the use of FreeBSD ports to compile and function.
48
49To use the FreeBSD ports system, it is required to update and extract the FreeBSD
50ports tree by issuing the following commands:
51
52.. code-block:: console
53
54    portsnap fetch
55    portsnap extract
56
57If the environment requires proxies for external communication, these can be set
58using:
59
60.. code-block:: console
61
62    setenv http_proxy <my_proxy_host>:<port>
63    setenv ftp_proxy <my_proxy_host>:<port>
64
65The FreeBSD ports below need to be installed prior to building the DPDK.
66In general these can be installed using the following set of commands::
67
68   cd /usr/ports/<port_location>
69
70   make config-recursive
71
72   make install
73
74   make clean
75
76Each port location can be found using::
77
78   whereis <port_name>
79
80The ports required and their locations are as follows:
81
82* dialog4ports: ``/usr/ports/ports-mgmt/dialog4ports``
83
84* GNU make(gmake): ``/usr/ports/devel/gmake``
85
86* coreutils: ``/usr/ports/sysutils/coreutils``
87
88For compiling and using the DPDK with gcc, the compiler must be installed
89from the ports collection:
90
91* gcc: version 4.9 is recommended ``/usr/ports/lang/gcc49``.
92  Ensure that ``CPU_OPTS`` is selected (default is OFF).
93
94When running the make config-recursive command, a dialog may be presented to the
95user. For the installation of the DPDK, the default options were used.
96
97.. note::
98
99    To avoid multiple dialogs being presented to the user during make install,
100    it is advisable before running the make install command to re-run the
101    make config-recursive command until no more dialogs are seen.
102
103
104Install the DPDK and Browse Sources
105-----------------------------------
106
107First, uncompress the archive and move to the DPDK source directory:
108
109.. code-block:: console
110
111    unzip DPDK-<version>.zip
112    cd DPDK-<version>
113
114The DPDK is composed of several directories:
115
116*   lib: Source code of DPDK libraries
117
118*   app: Source code of DPDK applications (automatic tests)
119
120*   examples: Source code of DPDK applications
121
122*   config, tools, scripts, mk: Framework-related makefiles, scripts and configuration
123
124Installation of the DPDK Target Environments
125--------------------------------------------
126
127The format of a DPDK target is::
128
129   ARCH-MACHINE-EXECENV-TOOLCHAIN
130
131Where:
132
133* ``ARCH`` is: ``x86_64``
134
135* ``MACHINE`` is: ``native``
136
137* ``EXECENV`` is: ``bsdapp``
138
139* ``TOOLCHAIN`` is: ``gcc`` | ``clang``
140
141The configuration files for the DPDK targets can be found in the DPDK/config
142directory in the form of::
143
144    defconfig_ARCH-MACHINE-EXECENV-TOOLCHAIN
145
146.. note::
147
148   Configuration files are provided with the ``RTE_MACHINE`` optimization level set.
149   Within the configuration files, the ``RTE_MACHINE`` configuration value is set
150   to native, which means that the compiled software is tuned for the platform
151   on which it is built.  For more information on this setting, and its
152   possible values, see the *DPDK Programmers Guide*.
153
154To make the target, use ``gmake install T=<target>``.
155
156For example to compile for FreeBSD use:
157
158.. code-block:: console
159
160    gmake install T=x86_64-native-bsdapp-clang
161
162.. note::
163
164   If the compiler binary to be used does not correspond to that given in the
165   TOOLCHAIN part of the target, the compiler command may need to be explicitly
166   specified. For example, if compiling for gcc, where the gcc binary is called
167   gcc4.9, the command would need to be ``gmake install T=<target> CC=gcc4.9``.
168
169Browsing the Installed DPDK Environment Target
170----------------------------------------------
171
172Once a target is created, it contains all the libraries and header files for the
173DPDK environment that are required to build customer applications.
174In addition, the test and testpmd applications are built under the build/app
175directory, which may be used for testing.  A kmod directory is also present that
176contains the kernel modules to install.
177
178.. _loading_contigmem:
179
180Loading the DPDK contigmem Module
181---------------------------------
182
183To run a DPDK application, physically contiguous memory is required.
184In the absence of non-transparent superpages, the included sources for the
185contigmem kernel module provides the ability to present contiguous blocks of
186memory for the DPDK to use. The contigmem module must be loaded into the
187running kernel before any DPDK is run.  The module is found in the kmod
188sub-directory of the DPDK target directory.
189
190The amount of physically contiguous memory along with the number of physically
191contiguous blocks to be reserved by the module can be set at runtime prior to
192module loading using:
193
194.. code-block:: console
195
196    kenv hw.contigmem.num_buffers=n
197    kenv hw.contigmem.buffer_size=m
198
199The kernel environment variables can also be specified during boot by placing the
200following in ``/boot/loader.conf``::
201
202    hw.contigmem.num_buffers=n hw.contigmem.buffer_size=m
203
204The variables can be inspected using the following command:
205
206.. code-block:: console
207
208    sysctl -a hw.contigmem
209
210Where n is the number of blocks and m is the size in bytes of each area of
211contiguous memory.  A default of two buffers of size 1073741824 bytes (1 Gigabyte)
212each is set during module load if they are not specified in the environment.
213
214The module can then be loaded using kldload (assuming that the current directory
215is the DPDK target directory):
216
217.. code-block:: console
218
219    kldload ./kmod/contigmem.ko
220
221It is advisable to include the loading of the contigmem module during the boot
222process to avoid issues with potential memory fragmentation during later system
223up time.  This can be achieved by copying the module to the ``/boot/kernel/``
224directory and placing the following into ``/boot/loader.conf``::
225
226    contigmem_load="YES"
227
228.. note::
229
230    The contigmem_load directive should be placed after any definitions of
231    ``hw.contigmem.num_buffers`` and ``hw.contigmem.buffer_size`` if the default values
232    are not to be used.
233
234An error such as:
235
236.. code-block:: console
237
238    kldload: can't load ./x86_64-native-bsdapp-gcc/kmod/contigmem.ko:
239             Exec format error
240
241is generally attributed to not having enough contiguous memory
242available and can be verified via dmesg or ``/var/log/messages``:
243
244.. code-block:: console
245
246    kernel: contigmalloc failed for buffer <n>
247
248To avoid this error, reduce the number of buffers or the buffer size.
249
250.. _loading_nic_uio:
251
252Loading the DPDK nic_uio Module
253-------------------------------
254
255After loading the contigmem module, the ``nic_uio`` module must also be loaded into the
256running kernel prior to running any DPDK application.  This module must
257be loaded using the kldload command as shown below (assuming that the current
258directory is the DPDK target directory).
259
260.. code-block:: console
261
262    kldload ./kmod/nic_uio.ko
263
264.. note::
265
266    If the ports to be used are currently bound to a existing kernel driver
267    then the ``hw.nic_uio.bdfs sysctl`` value will need to be set before loading the
268    module. Setting this value is described in the next section below.
269
270Currently loaded modules can be seen by using the ``kldstat`` command and a module
271can be removed from the running kernel by using ``kldunload <module_name>``.
272
273To load the module during boot, copy the ``nic_uio`` module to ``/boot/kernel``
274and place the following into ``/boot/loader.conf``::
275
276    nic_uio_load="YES"
277
278.. note::
279
280    ``nic_uio_load="YES"`` must appear after the contigmem_load directive, if it exists.
281
282By default, the ``nic_uio`` module will take ownership of network ports if they are
283recognized DPDK devices and are not owned by another module. However, since
284the FreeBSD kernel includes support, either built-in, or via a separate driver
285module, for most network card devices, it is likely that the ports to be used are
286already bound to a driver other than ``nic_uio``. The following sub-section describe
287how to query and modify the device ownership of the ports to be used by
288DPDK applications.
289
290.. _binding_network_ports:
291
292Binding Network Ports to the nic_uio Module
293~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
294
295Device ownership can be viewed using the pciconf -l command. The example below shows
296four Intel® 82599 network ports under ``if_ixgbe`` module ownership.
297
298.. code-block:: console
299
300    pciconf -l
301    ix0@pci0:1:0:0: class=0x020000 card=0x00038086 chip=0x10fb8086 rev=0x01 hdr=0x00
302    ix1@pci0:1:0:1: class=0x020000 card=0x00038086 chip=0x10fb8086 rev=0x01 hdr=0x00
303    ix2@pci0:2:0:0: class=0x020000 card=0x00038086 chip=0x10fb8086 rev=0x01 hdr=0x00
304    ix3@pci0:2:0:1: class=0x020000 card=0x00038086 chip=0x10fb8086 rev=0x01 hdr=0x00
305
306The first column constitutes three components:
307
308#. Device name: ``ixN``
309
310#. Unit name: ``pci0``
311
312#. Selector (Bus:Device:Function): ``1:0:0``
313
314Where no driver is associated with a device, the device name will be ``none``.
315
316By default, the FreeBSD kernel will include built-in drivers for the most common
317devices; a kernel rebuild would normally be required to either remove the drivers
318or configure them as loadable modules.
319
320To avoid building a custom kernel, the ``nic_uio`` module can detach a network port
321from its current device driver. This is achieved by setting the ``hw.nic_uio.bdfs``
322kernel environment variable prior to loading ``nic_uio``, as follows::
323
324    hw.nic_uio.bdfs="b:d:f,b:d:f,..."
325
326Where a comma separated list of selectors is set, the list must not contain any
327whitespace.
328
329For example to re-bind ``ix2@pci0:2:0:0`` and ``ix3@pci0:2:0:1`` to the ``nic_uio`` module
330upon loading, use the following command::
331
332    kenv hw.nic_uio.bdfs="2:0:0,2:0:1"
333
334The variable can also be specified during boot by placing the following into
335``/boot/loader.conf``, before the previously-described ``nic_uio_load`` line - as
336shown::
337
338    hw.nic_uio.bdfs="2:0:0,2:0:1"
339    nic_uio_load="YES"
340
341Binding Network Ports Back to their Original Kernel Driver
342~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
343
344If the original driver for a network port has been compiled into the kernel,
345it is necessary to reboot FreeBSD to restore the original device binding. Before
346doing so, update or remove the ``hw.nic_uio.bdfs`` in ``/boot/loader.conf``.
347
348If rebinding to a driver that is a loadable module, the network port binding can
349be reset without rebooting. To do so, unload both the target kernel module and the
350``nic_uio`` module, modify or clear the ``hw.nic_uio.bdfs`` kernel environment (kenv)
351value, and reload the two drivers - first the original kernel driver, and then
352the ``nic_uio driver``. Note: the latter does not need to be reloaded unless there are
353ports that are still to be bound to it.
354
355Example commands to perform these steps are shown below:
356
357.. code-block:: console
358
359    kldunload nic_uio
360    kldunload <original_driver>
361
362    # To clear the value completely:
363    kenv -u hw.nic_uio.bdfs
364
365    # To update the list of ports to bind:
366    kenv hw.nic_uio.bdfs="b:d:f,b:d:f,..."
367
368    kldload <original_driver>
369
370    kldload nic_uio  # optional
371