multi_proc_support.rst revision 6b3e017e
1..  BSD LICENSE
2    Copyright(c) 2010-2014 Intel Corporation. All rights reserved.
3    All rights reserved.
4
5    Redistribution and use in source and binary forms, with or without
6    modification, are permitted provided that the following conditions
7    are met:
8
9    * Redistributions of source code must retain the above copyright
10    notice, this list of conditions and the following disclaimer.
11    * Redistributions in binary form must reproduce the above copyright
12    notice, this list of conditions and the following disclaimer in
13    the documentation and/or other materials provided with the
14    distribution.
15    * Neither the name of Intel Corporation nor the names of its
16    contributors may be used to endorse or promote products derived
17    from this software without specific prior written permission.
18
19    THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
20    "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
21    LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
22    A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
23    OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
24    SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
25    LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
26    DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
27    THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
28    (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
29    OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
30
31.. _Multi-process_Support:
32
33Multi-process Support
34=====================
35
36In the DPDK, multi-process support is designed to allow a group of DPDK processes
37to work together in a simple transparent manner to perform packet processing,
38or other workloads.
39To support this functionality,
40a number of additions have been made to the core DPDK Environment Abstraction Layer (EAL).
41
42The EAL has been modified to allow different types of DPDK processes to be spawned,
43each with different permissions on the hugepage memory used by the applications.
44For now, there are two types of process specified:
45
46*   primary processes, which can initialize and which have full permissions on shared memory
47
48*   secondary processes, which cannot initialize shared memory,
49    but can attach to pre- initialized shared memory and create objects in it.
50
51Standalone DPDK processes are primary processes,
52while secondary processes can only run alongside a primary process or
53after a primary process has already configured the hugepage shared memory for them.
54
55To support these two process types, and other multi-process setups described later,
56two additional command-line parameters are available to the EAL:
57
58*   ``--proc-type:`` for specifying a given process instance as the primary or secondary DPDK instance
59
60*   ``--file-prefix:`` to allow processes that do not want to co-operate to have different memory regions
61
62A number of example applications are provided that demonstrate how multiple DPDK processes can be used together.
63These are more fully documented in the "Multi- process Sample Application" chapter
64in the *DPDK Sample Application's User Guide*.
65
66Memory Sharing
67--------------
68
69The key element in getting a multi-process application working using the DPDK is to ensure that
70memory resources are properly shared among the processes making up the multi-process application.
71Once there are blocks of shared memory available that can be accessed by multiple processes,
72then issues such as inter-process communication (IPC) becomes much simpler.
73
74On application start-up in a primary or standalone process,
75the DPDK records to memory-mapped files the details of the memory configuration it is using - hugepages in use,
76the virtual addresses they are mapped at, the number of memory channels present, etc.
77When a secondary process is started, these files are read and the EAL recreates the same memory configuration
78in the secondary process so that all memory zones are shared between processes and all pointers to that memory are valid,
79and point to the same objects, in both processes.
80
81.. note::
82
83    Refer to `Multi-process Limitations`_ for details of
84    how Linux kernel Address-Space Layout Randomization (ASLR) can affect memory sharing.
85
86.. _figure_multi_process_memory:
87
88.. figure:: img/multi_process_memory.*
89
90   Memory Sharing in the DPDK Multi-process Sample Application
91
92
93The EAL also supports an auto-detection mode (set by EAL ``--proc-type=auto`` flag ),
94whereby an DPDK process is started as a secondary instance if a primary instance is already running.
95
96Deployment Models
97-----------------
98
99Symmetric/Peer Processes
100~~~~~~~~~~~~~~~~~~~~~~~~
101
102DPDK multi-process support can be used to create a set of peer processes where each process performs the same workload.
103This model is equivalent to having multiple threads each running the same main-loop function,
104as is done in most of the supplied DPDK sample applications.
105In this model, the first of the processes spawned should be spawned using the ``--proc-type=primary`` EAL flag,
106while all subsequent instances should be spawned using the ``--proc-type=secondary`` flag.
107
108The simple_mp and symmetric_mp sample applications demonstrate this usage model.
109They are described in the "Multi-process Sample Application" chapter in the *DPDK Sample Application's User Guide*.
110
111Asymmetric/Non-Peer Processes
112~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
113
114An alternative deployment model that can be used for multi-process applications
115is to have a single primary process instance that acts as a load-balancer or
116server distributing received packets among worker or client threads, which are run as secondary processes.
117In this case, extensive use of rte_ring objects is made, which are located in shared hugepage memory.
118
119The client_server_mp sample application shows this usage model.
120It is described in the "Multi-process Sample Application" chapter in the *DPDK Sample Application's User Guide*.
121
122Running Multiple Independent DPDK Applications
123~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
124
125In addition to the above scenarios involving multiple DPDK processes working together,
126it is possible to run multiple DPDK processes side-by-side,
127where those processes are all working independently.
128Support for this usage scenario is provided using the ``--file-prefix`` parameter to the EAL.
129
130By default, the EAL creates hugepage files on each hugetlbfs filesystem using the rtemap_X filename,
131where X is in the range 0 to the maximum number of hugepages -1.
132Similarly, it creates shared configuration files, memory mapped in each process, using the /var/run/.rte_config filename,
133when run as root (or $HOME/.rte_config when run as a non-root user;
134if filesystem and device permissions are set up to allow this).
135The rte part of the filenames of each of the above is configurable using the file-prefix parameter.
136
137In addition to specifying the file-prefix parameter,
138any DPDK applications that are to be run side-by-side must explicitly limit their memory use.
139This is done by passing the -m flag to each process to specify how much hugepage memory, in megabytes,
140each process can use (or passing ``--socket-mem`` to specify how much hugepage memory on each socket each process can use).
141
142.. note::
143
144    Independent DPDK instances running side-by-side on a single machine cannot share any network ports.
145    Any network ports being used by one process should be blacklisted in every other process.
146
147Running Multiple Independent Groups of DPDK Applications
148~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
149
150In the same way that it is possible to run independent DPDK applications side- by-side on a single system,
151this can be trivially extended to multi-process groups of DPDK applications running side-by-side.
152In this case, the secondary processes must use the same ``--file-prefix`` parameter
153as the primary process whose shared memory they are connecting to.
154
155.. note::
156
157    All restrictions and issues with multiple independent DPDK processes running side-by-side
158    apply in this usage scenario also.
159
160Multi-process Limitations
161-------------------------
162
163There are a number of limitations to what can be done when running DPDK multi-process applications.
164Some of these are documented below:
165
166*   The multi-process feature requires that the exact same hugepage memory mappings be present in all applications.
167    The Linux security feature - Address-Space Layout Randomization (ASLR) can interfere with this mapping,
168    so it may be necessary to disable this feature in order to reliably run multi-process applications.
169
170.. warning::
171
172    Disabling Address-Space Layout Randomization (ASLR) may have security implications,
173    so it is recommended that it be disabled only when absolutely necessary,
174    and only when the implications of this change have been understood.
175
176*   All DPDK processes running as a single application and using shared memory must have distinct coremask arguments.
177    It is not possible to have a primary and secondary instance, or two secondary instances,
178    using any of the same logical cores.
179    Attempting to do so can cause corruption of memory pool caches, among other issues.
180
181*   The delivery of interrupts, such as Ethernet* device link status interrupts, do not work in secondary processes.
182    All interrupts are triggered inside the primary process only.
183    Any application needing interrupt notification in multiple processes should provide its own mechanism
184    to transfer the interrupt information from the primary process to any secondary process that needs the information.
185
186*   The use of function pointers between multiple processes running based of different compiled binaries is not supported,
187    since the location of a given function in one process may be different to its location in a second.
188    This prevents the librte_hash library from behaving properly as in a multi-threaded instance,
189    since it uses a pointer to the hash function internally.
190
191To work around this issue, it is recommended that multi-process applications perform the hash calculations by directly calling
192the hashing function from the code and then using the rte_hash_add_with_hash()/rte_hash_lookup_with_hash() functions
193instead of the functions which do the hashing internally, such as rte_hash_add()/rte_hash_lookup().
194
195*   Depending upon the hardware in use, and the number of DPDK processes used,
196    it may not be possible to have HPET timers available in each DPDK instance.
197    The minimum number of HPET comparators available to Linux* userspace can be just a single comparator,
198    which means that only the first, primary DPDK process instance can open and mmap  /dev/hpet.
199    If the number of required DPDK processes exceeds that of the number of available HPET comparators,
200    the TSC (which is the default timer in this release) must be used as a time source across all processes instead of the HPET.
201