intel_vf.rst revision 97f17497
1..  BSD LICENSE
2    Copyright(c) 2010-2014 Intel Corporation. All rights reserved.
3    All rights reserved.
4
5    Redistribution and use in source and binary forms, with or without
6    modification, are permitted provided that the following conditions
7    are met:
8
9    * Redistributions of source code must retain the above copyright
10    notice, this list of conditions and the following disclaimer.
11    * Redistributions in binary form must reproduce the above copyright
12    notice, this list of conditions and the following disclaimer in
13    the documentation and/or other materials provided with the
14    distribution.
15    * Neither the name of Intel Corporation nor the names of its
16    contributors may be used to endorse or promote products derived
17    from this software without specific prior written permission.
18
19    THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
20    "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
21    LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
22    A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
23    OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
24    SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
25    LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
26    DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
27    THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
28    (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
29    OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
30
31I40E/IXGBE/IGB Virtual Function Driver
32======================================
33
34Supported Intel® Ethernet Controllers (see the *DPDK Release Notes* for details)
35support the following modes of operation in a virtualized environment:
36
37*   **SR-IOV mode**: Involves direct assignment of part of the port resources to different guest operating systems
38    using the PCI-SIG Single Root I/O Virtualization (SR IOV) standard,
39    also known as "native mode" or "pass-through" mode.
40    In this chapter, this mode is referred to as IOV mode.
41
42*   **VMDq mode**: Involves central management of the networking resources by an IO Virtual Machine (IOVM) or
43    a Virtual Machine Monitor (VMM), also known as software switch acceleration mode.
44    In this chapter, this mode is referred to as the Next Generation VMDq mode.
45
46SR-IOV Mode Utilization in a DPDK Environment
47---------------------------------------------
48
49The DPDK uses the SR-IOV feature for hardware-based I/O sharing in IOV mode.
50Therefore, it is possible to partition SR-IOV capability on Ethernet controller NIC resources logically and
51expose them to a virtual machine as a separate PCI function called a "Virtual Function".
52Refer to :numref:`figure_single_port_nic`.
53
54Therefore, a NIC is logically distributed among multiple virtual machines (as shown in :numref:`figure_single_port_nic`),
55while still having global data in common to share with the Physical Function and other Virtual Functions.
56The DPDK fm10kvf, i40evf, igbvf or ixgbevf as a Poll Mode Driver (PMD) serves for the Intel® 82576 Gigabit Ethernet Controller,
57Intel® Ethernet Controller I350 family, Intel® 82599 10 Gigabit Ethernet Controller NIC,
58Intel® Fortville 10/40 Gigabit Ethernet Controller NIC's virtual PCI function, or PCIe host-interface of the Intel Ethernet Switch
59FM10000 Series.
60Meanwhile the DPDK Poll Mode Driver (PMD) also supports "Physical Function" of such NIC's on the host.
61
62The DPDK PF/VF Poll Mode Driver (PMD) supports the Layer 2 switch on Intel® 82576 Gigabit Ethernet Controller,
63Intel® Ethernet Controller I350 family, Intel® 82599 10 Gigabit Ethernet Controller,
64and Intel® Fortville 10/40 Gigabit Ethernet Controller NICs so that guest can choose it for inter virtual machine traffic in SR-IOV mode.
65
66For more detail on SR-IOV, please refer to the following documents:
67
68*   `SR-IOV provides hardware based I/O sharing <http://www.intel.com/network/connectivity/solutions/vmdc.htm>`_
69
70*   `PCI-SIG-Single Root I/O Virtualization Support on IA
71    <http://www.intel.com/content/www/us/en/pci-express/pci-sig-single-root-io-virtualization-support-in-virtualization-technology-for-connectivity-paper.html>`_
72
73*   `Scalable I/O Virtualized Servers <http://www.intel.com/content/www/us/en/virtualization/server-virtualization/scalable-i-o-virtualized-servers-paper.html>`_
74
75.. _figure_single_port_nic:
76
77.. figure:: img/single_port_nic.*
78
79   Virtualization for a Single Port NIC in SR-IOV Mode
80
81
82Physical and Virtual Function Infrastructure
83~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
84
85The following describes the Physical Function and Virtual Functions infrastructure for the supported Ethernet Controller NICs.
86
87Virtual Functions operate under the respective Physical Function on the same NIC Port and therefore have no access
88to the global NIC resources that are shared between other functions for the same NIC port.
89
90A Virtual Function has basic access to the queue resources and control structures of the queues assigned to it.
91For global resource access, a Virtual Function has to send a request to the Physical Function for that port,
92and the Physical Function operates on the global resources on behalf of the Virtual Function.
93For this out-of-band communication, an SR-IOV enabled NIC provides a memory buffer for each Virtual Function,
94which is called a "Mailbox".
95
96The PCIE host-interface of Intel Ethernet Switch FM10000 Series VF infrastructure
97^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
98
99In a virtualized environment, the programmer can enable a maximum of *64 Virtual Functions (VF)*
100globally per PCIE host-interface of the Intel Ethernet Switch FM10000 Series device.
101Each VF can have a maximum of 16 queue pairs.
102The Physical Function in host could be only configured by the Linux* fm10k driver
103(in the case of the Linux Kernel-based Virtual Machine [KVM]), DPDK PMD PF driver doesn't support it yet.
104
105For example,
106
107*   Using Linux* fm10k driver:
108
109    .. code-block:: console
110
111        rmmod fm10k (To remove the fm10k module)
112        insmod fm0k.ko max_vfs=2,2 (To enable two Virtual Functions per port)
113
114Virtual Function enumeration is performed in the following sequence by the Linux* pci driver for a dual-port NIC.
115When you enable the four Virtual Functions with the above command, the four enabled functions have a Function#
116represented by (Bus#, Device#, Function#) in sequence starting from 0 to 3.
117However:
118
119*   Virtual Functions 0 and 2 belong to Physical Function 0
120
121*   Virtual Functions 1 and 3 belong to Physical Function 1
122
123.. note::
124
125    The above is an important consideration to take into account when targeting specific packets to a selected port.
126
127Intel® Fortville 10/40 Gigabit Ethernet Controller VF Infrastructure
128^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
129
130In a virtualized environment, the programmer can enable a maximum of *128 Virtual Functions (VF)*
131globally per Intel® Fortville 10/40 Gigabit Ethernet Controller NIC device.
132Each VF can have a maximum of 16 queue pairs.
133The Physical Function in host could be either configured by the Linux* i40e driver
134(in the case of the Linux Kernel-based Virtual Machine [KVM]) or by DPDK PMD PF driver.
135When using both DPDK PMD PF/VF drivers, the whole NIC will be taken over by DPDK based application.
136
137For example,
138
139*   Using Linux* i40e  driver:
140
141    .. code-block:: console
142
143        rmmod i40e (To remove the i40e module)
144        insmod i40e.ko max_vfs=2,2 (To enable two Virtual Functions per port)
145
146*   Using the DPDK PMD PF i40e driver:
147
148    Kernel Params: iommu=pt, intel_iommu=on
149
150    .. code-block:: console
151
152        modprobe uio
153        insmod igb_uio
154        ./dpdk_nic_bind.py -b igb_uio bb:ss.f
155        echo 2 > /sys/bus/pci/devices/0000\:bb\:ss.f/max_vfs (To enable two VFs on a specific PCI device)
156
157    Launch the DPDK testpmd/example or your own host daemon application using the DPDK PMD library.
158
159*   Using the DPDK PMD PF ixgbe driver to enable VF RSS:
160
161    Same steps as above to install the modules of uio, igb_uio, specify max_vfs for PCI device, and
162    launch the DPDK testpmd/example or your own host daemon application using the DPDK PMD library.
163
164    The available queue number(at most 4) per VF depends on the total number of pool, which is
165    determined by the max number of VF at PF initialization stage and the number of queue specified
166    in config:
167
168    *   If the max number of VF is set in the range of 1 to 32:
169
170        If the number of rxq is specified as 4(e.g. '--rxq 4' in testpmd), then there are totally 32
171        pools(ETH_32_POOLS), and each VF could have 4 or less(e.g. 2) queues;
172
173        If the number of rxq is specified as 2(e.g. '--rxq 2' in testpmd), then there are totally 32
174        pools(ETH_32_POOLS), and each VF could have 2 queues;
175
176    *   If the max number of VF is in the range of 33 to 64:
177
178        If the number of rxq is 4 ('--rxq 4' in testpmd), then error message is expected as rxq is not
179        correct at this case;
180
181        If the number of rxq is 2 ('--rxq 2' in testpmd), then there is totally 64 pools(ETH_64_POOLS),
182        and each VF have 2 queues;
183
184    On host, to enable VF RSS functionality, rx mq mode should be set as ETH_MQ_RX_VMDQ_RSS
185    or ETH_MQ_RX_RSS mode, and SRIOV mode should be activated(max_vfs >= 1).
186    It also needs config VF RSS information like hash function, RSS key, RSS key length.
187
188    .. code-block:: console
189
190        testpmd -c 0xffff -n 4 -- --coremask=<core-mask> --rxq=4 --txq=4 -i
191
192    The limitation for VF RSS on Intel® 82599 10 Gigabit Ethernet Controller is:
193    The hash and key are shared among PF and all VF, the RETA table with 128 entries is also shared
194    among PF and all VF; So it could not to provide a method to query the hash and reta content per
195    VF on guest, while, if possible, please query them on host(PF) for the shared RETA information.
196
197Virtual Function enumeration is performed in the following sequence by the Linux* pci driver for a dual-port NIC.
198When you enable the four Virtual Functions with the above command, the four enabled functions have a Function#
199represented by (Bus#, Device#, Function#) in sequence starting from 0 to 3.
200However:
201
202*   Virtual Functions 0 and 2 belong to Physical Function 0
203
204*   Virtual Functions 1 and 3 belong to Physical Function 1
205
206.. note::
207
208    The above is an important consideration to take into account when targeting specific packets to a selected port.
209
210Intel® 82599 10 Gigabit Ethernet Controller VF Infrastructure
211^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
212
213The programmer can enable a maximum of *63 Virtual Functions* and there must be *one Physical Function* per Intel® 82599
21410 Gigabit Ethernet Controller NIC port.
215The reason for this is that the device allows for a maximum of 128 queues per port and a virtual/physical function has to
216have at least one queue pair (RX/TX).
217The current implementation of the DPDK ixgbevf driver supports a single queue pair (RX/TX) per Virtual Function.
218The Physical Function in host could be either configured by the Linux* ixgbe driver
219(in the case of the Linux Kernel-based Virtual Machine [KVM]) or by DPDK PMD PF driver.
220When using both DPDK PMD PF/VF drivers, the whole NIC will be taken over by DPDK based application.
221
222For example,
223
224*   Using Linux* ixgbe driver:
225
226    .. code-block:: console
227
228        rmmod ixgbe (To remove the ixgbe module)
229        insmod ixgbe max_vfs=2,2 (To enable two Virtual Functions per port)
230
231*   Using the DPDK PMD PF ixgbe driver:
232
233    Kernel Params: iommu=pt, intel_iommu=on
234
235    .. code-block:: console
236
237        modprobe uio
238        insmod igb_uio
239        ./dpdk_nic_bind.py -b igb_uio bb:ss.f
240        echo 2 > /sys/bus/pci/devices/0000\:bb\:ss.f/max_vfs (To enable two VFs on a specific PCI device)
241
242    Launch the DPDK testpmd/example or your own host daemon application using the DPDK PMD library.
243
244Virtual Function enumeration is performed in the following sequence by the Linux* pci driver for a dual-port NIC.
245When you enable the four Virtual Functions with the above command, the four enabled functions have a Function#
246represented by (Bus#, Device#, Function#) in sequence starting from 0 to 3.
247However:
248
249*   Virtual Functions 0 and 2 belong to Physical Function 0
250
251*   Virtual Functions 1 and 3 belong to Physical Function 1
252
253.. note::
254
255    The above is an important consideration to take into account when targeting specific packets to a selected port.
256
257Intel® 82576 Gigabit Ethernet Controller and Intel® Ethernet Controller I350 Family VF Infrastructure
258^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
259
260In a virtualized environment, an Intel® 82576 Gigabit Ethernet Controller serves up to eight virtual machines (VMs).
261The controller has 16 TX and 16 RX queues.
262They are generally referred to (or thought of) as queue pairs (one TX and one RX queue).
263This gives the controller 16 queue pairs.
264
265A pool is a group of queue pairs for assignment to the same VF, used for transmit and receive operations.
266The controller has eight pools, with each pool containing two queue pairs, that is, two TX and two RX queues assigned to each VF.
267
268In a virtualized environment, an Intel® Ethernet Controller I350 family device serves up to eight virtual machines (VMs) per port.
269The eight queues can be accessed by eight different VMs if configured correctly (the i350 has 4x1GbE ports each with 8T X and 8 RX queues),
270that means, one Transmit and one Receive queue assigned to each VF.
271
272For example,
273
274*   Using Linux* igb driver:
275
276    .. code-block:: console
277
278        rmmod igb (To remove the igb module)
279        insmod igb max_vfs=2,2 (To enable two Virtual Functions per port)
280
281*   Using DPDK PMD PF igb driver:
282
283    Kernel Params: iommu=pt, intel_iommu=on modprobe uio
284
285    .. code-block:: console
286
287        insmod igb_uio
288        ./dpdk_nic_bind.py -b igb_uio bb:ss.f
289        echo 2 > /sys/bus/pci/devices/0000\:bb\:ss.f/max_vfs (To enable two VFs on a specific pci device)
290
291    Launch DPDK testpmd/example or your own host daemon application using the DPDK PMD library.
292
293Virtual Function enumeration is performed in the following sequence by the Linux* pci driver for a four-port NIC.
294When you enable the four Virtual Functions with the above command, the four enabled functions have a Function#
295represented by (Bus#, Device#, Function#) in sequence, starting from 0 to 7.
296However:
297
298*   Virtual Functions 0 and 4 belong to Physical Function 0
299
300*   Virtual Functions 1 and 5 belong to Physical Function 1
301
302*   Virtual Functions 2 and 6 belong to Physical Function 2
303
304*   Virtual Functions 3 and 7 belong to Physical Function 3
305
306.. note::
307
308    The above is an important consideration to take into account when targeting specific packets to a selected port.
309
310Validated Hypervisors
311~~~~~~~~~~~~~~~~~~~~~
312
313The validated hypervisor is:
314
315*   KVM (Kernel Virtual Machine) with  Qemu, version 0.14.0
316
317However, the hypervisor is bypassed to configure the Virtual Function devices using the Mailbox interface,
318the solution is hypervisor-agnostic.
319Xen* and VMware* (when SR- IOV is supported) will also be able to support the DPDK with Virtual Function driver support.
320
321Expected Guest Operating System in Virtual Machine
322~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
323
324The expected guest operating systems in a virtualized environment are:
325
326*   Fedora* 14 (64-bit)
327
328*   Ubuntu* 10.04 (64-bit)
329
330For supported kernel versions, refer to the *DPDK Release Notes*.
331
332Setting Up a KVM Virtual Machine Monitor
333----------------------------------------
334
335The following describes a target environment:
336
337*   Host Operating System: Fedora 14
338
339*   Hypervisor: KVM (Kernel Virtual Machine) with Qemu  version 0.14.0
340
341*   Guest Operating System: Fedora 14
342
343*   Linux Kernel Version: Refer to the  *DPDK Getting Started Guide*
344
345*   Target Applications:  l2fwd, l3fwd-vf
346
347The setup procedure is as follows:
348
349#.  Before booting the Host OS, open **BIOS setup** and enable **Intel® VT features**.
350
351#.  While booting the Host OS kernel, pass the intel_iommu=on kernel command line argument using GRUB.
352    When using DPDK PF driver on host, pass the iommu=pt kernel command line argument in GRUB.
353
354#.  Download qemu-kvm-0.14.0 from
355    `http://sourceforge.net/projects/kvm/files/qemu-kvm/ <http://sourceforge.net/projects/kvm/files/qemu-kvm/>`_
356    and install it in the Host OS using the following steps:
357
358    When using a recent kernel (2.6.25+) with kvm modules included:
359
360    .. code-block:: console
361
362        tar xzf qemu-kvm-release.tar.gz
363        cd qemu-kvm-release
364        ./configure --prefix=/usr/local/kvm
365        make
366        sudo make install
367        sudo /sbin/modprobe kvm-intel
368
369    When using an older kernel, or a kernel from a distribution without the kvm modules,
370    you must download (from the same link), compile and install the modules yourself:
371
372    .. code-block:: console
373
374        tar xjf kvm-kmod-release.tar.bz2
375        cd kvm-kmod-release
376        ./configure
377        make
378        sudo make install
379        sudo /sbin/modprobe kvm-intel
380
381    qemu-kvm installs in the /usr/local/bin directory.
382
383    For more details about KVM configuration and usage, please refer to:
384
385    `http://www.linux-kvm.org/page/HOWTO1 <http://www.linux-kvm.org/page/HOWTO1>`_.
386
387#.  Create a Virtual Machine and install Fedora 14 on the Virtual Machine.
388    This is referred to as the Guest Operating System (Guest OS).
389
390#.  Download and install the latest ixgbe driver from:
391
392    `http://downloadcenter.intel.com/Detail_Desc.aspx?agr=Y&DwnldID=14687 <http://downloadcenter.intel.com/Detail_Desc.aspx?agr=Y&DwnldID=14687>`_
393
394#.  In the Host OS
395
396    When using Linux kernel ixgbe driver, unload the Linux ixgbe driver and reload it with the max_vfs=2,2 argument:
397
398    .. code-block:: console
399
400        rmmod ixgbe
401        modprobe ixgbe max_vfs=2,2
402
403    When using DPDK PMD PF driver, insert DPDK kernel module igb_uio and set the number of VF by sysfs max_vfs:
404
405    .. code-block:: console
406
407        modprobe uio
408        insmod igb_uio
409        ./dpdk_nic_bind.py -b igb_uio 02:00.0 02:00.1 0e:00.0 0e:00.1
410        echo 2 > /sys/bus/pci/devices/0000\:02\:00.0/max_vfs
411        echo 2 > /sys/bus/pci/devices/0000\:02\:00.1/max_vfs
412        echo 2 > /sys/bus/pci/devices/0000\:0e\:00.0/max_vfs
413        echo 2 > /sys/bus/pci/devices/0000\:0e\:00.1/max_vfs
414
415    .. note::
416
417        You need to explicitly specify number of vfs for each port, for example,
418        in the command above, it creates two vfs for the first two ixgbe ports.
419
420    Let say we have a machine with four physical ixgbe ports:
421
422
423        0000:02:00.0
424
425        0000:02:00.1
426
427        0000:0e:00.0
428
429        0000:0e:00.1
430
431    The command above creates two vfs for device 0000:02:00.0:
432
433    .. code-block:: console
434
435        ls -alrt /sys/bus/pci/devices/0000\:02\:00.0/virt*
436        lrwxrwxrwx. 1 root root 0 Apr 13 05:40 /sys/bus/pci/devices/0000:02:00.0/virtfn1 -> ../0000:02:10.2
437        lrwxrwxrwx. 1 root root 0 Apr 13 05:40 /sys/bus/pci/devices/0000:02:00.0/virtfn0 -> ../0000:02:10.0
438
439    It also creates two vfs for device 0000:02:00.1:
440
441    .. code-block:: console
442
443        ls -alrt /sys/bus/pci/devices/0000\:02\:00.1/virt*
444        lrwxrwxrwx. 1 root root 0 Apr 13 05:51 /sys/bus/pci/devices/0000:02:00.1/virtfn1 -> ../0000:02:10.3
445        lrwxrwxrwx. 1 root root 0 Apr 13 05:51 /sys/bus/pci/devices/0000:02:00.1/virtfn0 -> ../0000:02:10.1
446
447#.  List the PCI devices connected and notice that the Host OS shows two Physical Functions (traditional ports)
448    and four Virtual Functions (two for each port).
449    This is the result of the previous step.
450
451#.  Insert the pci_stub module to hold the PCI devices that are freed from the default driver using the following command
452    (see http://www.linux-kvm.org/page/How_to_assign_devices_with_VT-d_in_KVM Section 4 for more information):
453
454    .. code-block:: console
455
456        sudo /sbin/modprobe pci-stub
457
458    Unbind the default driver from the PCI devices representing the Virtual Functions.
459    A script to perform this action is as follows:
460
461    .. code-block:: console
462
463        echo "8086 10ed" > /sys/bus/pci/drivers/pci-stub/new_id
464        echo 0000:08:10.0 > /sys/bus/pci/devices/0000:08:10.0/driver/unbind
465        echo 0000:08:10.0 > /sys/bus/pci/drivers/pci-stub/bind
466
467    where, 0000:08:10.0 belongs to the Virtual Function visible in the Host OS.
468
469#.  Now, start the Virtual Machine by running the following command:
470
471    .. code-block:: console
472
473        /usr/local/kvm/bin/qemu-system-x86_64 -m 4096 -smp 4 -boot c -hda lucid.qcow2 -device pci-assign,host=08:10.0
474
475    where:
476
477        — -m = memory to assign
478
479        — -smp = number of smp cores
480
481        — -boot = boot option
482
483        — -hda = virtual disk image
484
485        — -device = device to attach
486
487    .. note::
488
489        — The pci-assign,host=08:10.0 alue indicates that you want to attach a PCI device
490        to a Virtual Machine and the respective (Bus:Device.Function)
491        numbers should be passed for the Virtual Function to be attached.
492
493        — qemu-kvm-0.14.0 allows a maximum of four PCI devices assigned to a VM,
494        but this is qemu-kvm version dependent since qemu-kvm-0.14.1 allows a maximum of five PCI devices.
495
496        — qemu-system-x86_64 also has a -cpu command line option that is used to select the cpu_model
497        to emulate in a Virtual Machine. Therefore, it can be used as:
498
499        .. code-block:: console
500
501            /usr/local/kvm/bin/qemu-system-x86_64 -cpu ?
502
503            (to list all available cpu_models)
504
505            /usr/local/kvm/bin/qemu-system-x86_64 -m 4096 -cpu host -smp 4 -boot c -hda lucid.qcow2 -device pci-assign,host=08:10.0
506
507            (to use the same cpu_model equivalent to the host cpu)
508
509        For more information, please refer to: `http://wiki.qemu.org/Features/CPUModels <http://wiki.qemu.org/Features/CPUModels>`_.
510
511#.  Install and run DPDK host app to take  over the Physical Function. Eg.
512
513    .. code-block:: console
514
515        make install T=x86_64-native-linuxapp-gcc
516        ./x86_64-native-linuxapp-gcc/app/testpmd -c f -n 4 -- -i
517
518#.  Finally, access the Guest OS using vncviewer with the localhost:5900 port and check the lspci command output in the Guest OS.
519    The virtual functions will be listed as available for use.
520
521#.  Configure and install the DPDK with an x86_64-native-linuxapp-gcc configuration on the Guest OS as normal,
522    that is, there is no change to the normal installation procedure.
523
524    .. code-block:: console
525
526        make config T=x86_64-native-linuxapp-gcc O=x86_64-native-linuxapp-gcc
527        cd x86_64-native-linuxapp-gcc
528        make
529
530.. note::
531
532    If you are unable to compile the DPDK and you are getting "error: CPU you selected does not support x86-64 instruction set",
533    power off the Guest OS and start the virtual machine with the correct -cpu option in the qemu- system-x86_64 command as shown in step 9.
534    You must select the best x86_64 cpu_model to emulate or you can select host option if available.
535
536.. note::
537
538    Run the DPDK l2fwd sample application in the Guest OS with Hugepages enabled.
539    For the expected benchmark performance, you must pin the cores from the Guest OS to the Host OS (taskset can be used to do this) and
540    you must also look at the PCI Bus layout on the board to ensure you are not running the traffic over the QPI Interface.
541
542.. note::
543
544    *   The Virtual Machine Manager (the Fedora package name is virt-manager) is a utility for virtual machine management
545        that can also be used to create, start, stop and delete virtual machines.
546        If this option is used, step 2 and 6 in the instructions provided will be different.
547
548    *   virsh, a command line utility for virtual machine management,
549        can also be used to bind and unbind devices to a virtual machine in Ubuntu.
550        If this option is used, step 6 in the instructions provided will be different.
551
552    *   The Virtual Machine Monitor (see :numref:`figure_perf_benchmark`) is equivalent to a Host OS with KVM installed as described in the instructions.
553
554.. _figure_perf_benchmark:
555
556.. figure:: img/perf_benchmark.*
557
558   Performance Benchmark Setup
559
560
561DPDK SR-IOV PMD PF/VF Driver Usage Model
562----------------------------------------
563
564Fast Host-based Packet Processing
565~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
566
567Software Defined Network (SDN) trends are demanding fast host-based packet handling.
568In a virtualization environment,
569the DPDK VF PMD driver performs the same throughput result as a non-VT native environment.
570
571With such host instance fast packet processing, lots of services such as filtering, QoS,
572DPI can be offloaded on the host fast path.
573
574:numref:`figure_fast_pkt_proc` shows the scenario where some VMs directly communicate externally via a VFs,
575while others connect to a virtual switch and share the same uplink bandwidth.
576
577.. _figure_fast_pkt_proc:
578
579.. figure:: img/fast_pkt_proc.*
580
581   Fast Host-based Packet Processing
582
583
584SR-IOV (PF/VF) Approach for Inter-VM Communication
585--------------------------------------------------
586
587Inter-VM data communication is one of the traffic bottle necks in virtualization platforms.
588SR-IOV device assignment helps a VM to attach the real device, taking advantage of the bridge in the NIC.
589So VF-to-VF traffic within the same physical port (VM0<->VM1) have hardware acceleration.
590However, when VF crosses physical ports (VM0<->VM2), there is no such hardware bridge.
591In this case, the DPDK PMD PF driver provides host forwarding between such VMs.
592
593:numref:`figure_inter_vm_comms` shows an example.
594In this case an update of the MAC address lookup tables in both the NIC and host DPDK application is required.
595
596In the NIC, writing the destination of a MAC address belongs to another cross device VM to the PF specific pool.
597So when a packet comes in, its destination MAC address will match and forward to the host DPDK PMD application.
598
599In the host DPDK application, the behavior is similar to L2 forwarding,
600that is, the packet is forwarded to the correct PF pool.
601The SR-IOV NIC switch forwards the packet to a specific VM according to the MAC destination address
602which belongs to the destination VF on the VM.
603
604.. _figure_inter_vm_comms:
605
606.. figure:: img/inter_vm_comms.*
607
608   Inter-VM Communication
609