15129044dSC.J. Collier..  BSD LICENSE
25129044dSC.J. Collier    Copyright(c) 2010-2014 Intel Corporation. All rights reserved.
35129044dSC.J. Collier    All rights reserved.
45129044dSC.J. Collier
55129044dSC.J. Collier    Redistribution and use in source and binary forms, with or without
65129044dSC.J. Collier    modification, are permitted provided that the following conditions
75129044dSC.J. Collier    are met:
85129044dSC.J. Collier
95129044dSC.J. Collier    * Redistributions of source code must retain the above copyright
105129044dSC.J. Collier    notice, this list of conditions and the following disclaimer.
115129044dSC.J. Collier    * Redistributions in binary form must reproduce the above copyright
125129044dSC.J. Collier    notice, this list of conditions and the following disclaimer in
135129044dSC.J. Collier    the documentation and/or other materials provided with the
145129044dSC.J. Collier    distribution.
155129044dSC.J. Collier    * Neither the name of Intel Corporation nor the names of its
165129044dSC.J. Collier    contributors may be used to endorse or promote products derived
175129044dSC.J. Collier    from this software without specific prior written permission.
185129044dSC.J. Collier
195129044dSC.J. Collier    THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
205129044dSC.J. Collier    "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
215129044dSC.J. Collier    LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
225129044dSC.J. Collier    A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
235129044dSC.J. Collier    OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
245129044dSC.J. Collier    SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
255129044dSC.J. Collier    LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
265129044dSC.J. Collier    DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
275129044dSC.J. Collier    THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
285129044dSC.J. Collier    (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
295129044dSC.J. Collier    OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
305129044dSC.J. Collier
315129044dSC.J. CollierPoll Mode Driver for Paravirtual VMXNET3 NIC
325129044dSC.J. Collier============================================
335129044dSC.J. Collier
345129044dSC.J. CollierThe VMXNET3 adapter is the next generation of a paravirtualized NIC, introduced by VMware* ESXi.
353d9b7210SChristian EhrhardtIt is designed for performance, offers all the features available in VMXNET2, and adds several new features such as,
365129044dSC.J. Colliermulti-queue support (also known as Receive Side Scaling, RSS),
375129044dSC.J. CollierIPv6 offloads, and MSI/MSI-X interrupt delivery.
385129044dSC.J. CollierOne can use the same device in a DPDK application with VMXNET3 PMD introduced in DPDK API.
395129044dSC.J. Collier
405129044dSC.J. CollierIn this chapter, two setups with the use of the VMXNET3 PMD are demonstrated:
415129044dSC.J. Collier
425129044dSC.J. Collier#.  Vmxnet3 with a native NIC connected to a vSwitch
435129044dSC.J. Collier
445129044dSC.J. Collier#.  Vmxnet3 chaining VMs connected to a vSwitch
455129044dSC.J. Collier
465129044dSC.J. CollierVMXNET3 Implementation in the DPDK
475129044dSC.J. Collier----------------------------------
485129044dSC.J. Collier
495129044dSC.J. CollierFor details on the VMXNET3 device, refer to the VMXNET3 driver's vmxnet3 directory and support manual from VMware*.
505129044dSC.J. Collier
515129044dSC.J. CollierFor performance details, refer to the following link from VMware:
525129044dSC.J. Collier
535129044dSC.J. Collier`http://www.vmware.com/pdf/vsp_4_vmxnet3_perf.pdf <http://www.vmware.com/pdf/vsp_4_vmxnet3_perf.pdf>`_
545129044dSC.J. Collier
555129044dSC.J. CollierAs a PMD, the VMXNET3 driver provides the packet reception and transmission callbacks, vmxnet3_recv_pkts and vmxnet3_xmit_pkts.
565129044dSC.J. Collier
575129044dSC.J. CollierThe VMXNET3 PMD handles all the packet buffer memory allocation and resides in guest address space
585129044dSC.J. Collierand it is solely responsible to free that memory when not needed.
595129044dSC.J. CollierThe packet buffers and features to be supported are made available to hypervisor via VMXNET3 PCI configuration space BARs.
605129044dSC.J. CollierDuring RX/TX, the packet buffers are exchanged by their GPAs,
615129044dSC.J. Collierand the hypervisor loads the buffers with packets in the RX case and sends packets to vSwitch in the TX case.
625129044dSC.J. Collier
635129044dSC.J. CollierThe VMXNET3 PMD is compiled with vmxnet3 device headers.
645129044dSC.J. CollierThe interface is similar to that of the other PMDs available in the DPDK API.
655129044dSC.J. CollierThe driver pre-allocates the packet buffers and loads the command ring descriptors in advance.
665129044dSC.J. CollierThe hypervisor fills those packet buffers on packet arrival and write completion ring descriptors,
675129044dSC.J. Collierwhich are eventually pulled by the PMD.
685129044dSC.J. CollierAfter reception, the DPDK application frees the descriptors and loads new packet buffers for the coming packets.
695129044dSC.J. CollierThe interrupts are disabled and there is no notification required.
705129044dSC.J. CollierThis keeps performance up on the RX side, even though the device provides a notification feature.
715129044dSC.J. Collier
725129044dSC.J. CollierIn the transmit routine, the DPDK application fills packet buffer pointers in the descriptors of the command ring
735129044dSC.J. Collierand notifies the hypervisor.
743d9b7210SChristian EhrhardtIn response the hypervisor takes packets and passes them to the vSwitch, It writes into the completion descriptors ring.
755129044dSC.J. CollierThe rings are read by the PMD in the next transmit routine call and the buffers and descriptors are freed from memory.
765129044dSC.J. Collier
775129044dSC.J. CollierFeatures and Limitations of VMXNET3 PMD
785129044dSC.J. Collier---------------------------------------
795129044dSC.J. Collier
805129044dSC.J. CollierIn release 1.6.0, the VMXNET3 PMD provides the basic functionality of packet reception and transmission.
815129044dSC.J. CollierThere are several options available for filtering packets at VMXNET3 device level including:
825129044dSC.J. Collier
835129044dSC.J. Collier#.  MAC Address based filtering:
845129044dSC.J. Collier
855129044dSC.J. Collier    *   Unicast, Broadcast, All Multicast modes - SUPPORTED BY DEFAULT
865129044dSC.J. Collier
875129044dSC.J. Collier    *   Multicast with Multicast Filter table - NOT SUPPORTED
885129044dSC.J. Collier
895129044dSC.J. Collier    *   Promiscuous mode - SUPPORTED
905129044dSC.J. Collier
915129044dSC.J. Collier    *   RSS based load balancing between queues - SUPPORTED
925129044dSC.J. Collier
935129044dSC.J. Collier#.  VLAN filtering:
945129044dSC.J. Collier
955129044dSC.J. Collier    *   VLAN tag based filtering without load balancing - SUPPORTED
965129044dSC.J. Collier
975129044dSC.J. Collier.. note::
985129044dSC.J. Collier
995129044dSC.J. Collier
1005129044dSC.J. Collier    *   Release 1.6.0 does not support separate headers and body receive cmd_ring and hence,
1015129044dSC.J. Collier        multiple segment buffers are not supported.
1025129044dSC.J. Collier        Only cmd_ring_0 is used for packet buffers, one for each descriptor.
1035129044dSC.J. Collier
1045129044dSC.J. Collier    *   Receive and transmit of scattered packets is not supported.
1055129044dSC.J. Collier
1065129044dSC.J. Collier    *   Multicast with Multicast Filter table is not supported.
1075129044dSC.J. Collier
1085129044dSC.J. CollierPrerequisites
1095129044dSC.J. Collier-------------
1105129044dSC.J. Collier
1115129044dSC.J. CollierThe following prerequisites apply:
1125129044dSC.J. Collier
1135129044dSC.J. Collier*   Before starting a VM, a VMXNET3 interface to a VM through VMware vSphere Client must be assigned.
1145129044dSC.J. Collier    This is shown in the figure below.
1155129044dSC.J. Collier
1165129044dSC.J. Collier.. _figure_vmxnet3_int:
1175129044dSC.J. Collier
1185129044dSC.J. Collier.. figure:: img/vmxnet3_int.*
1195129044dSC.J. Collier
1205129044dSC.J. Collier   Assigning a VMXNET3 interface to a VM using VMware vSphere Client
1215129044dSC.J. Collier
1225129044dSC.J. Collier.. note::
1235129044dSC.J. Collier
1245129044dSC.J. Collier    Depending on the Virtual Machine type, the VMware vSphere Client shows Ethernet adaptors while adding an Ethernet device.
1255129044dSC.J. Collier    Ensure that the VM type used offers a VMXNET3 device. Refer to the VMware documentation for a listed of VMs.
1265129044dSC.J. Collier
1275129044dSC.J. Collier.. note::
1285129044dSC.J. Collier
1295129044dSC.J. Collier    Follow the *DPDK Getting Started Guide* to setup the basic DPDK environment.
1305129044dSC.J. Collier
1315129044dSC.J. Collier.. note::
1325129044dSC.J. Collier
1335129044dSC.J. Collier    Follow the *DPDK Sample Application's User Guide*, L2 Forwarding/L3 Forwarding and
1345129044dSC.J. Collier    TestPMD for instructions on how to run a DPDK application using an assigned VMXNET3 device.
1355129044dSC.J. Collier
1365129044dSC.J. CollierVMXNET3 with a Native NIC Connected to a vSwitch
1375129044dSC.J. Collier------------------------------------------------
1385129044dSC.J. Collier
1395129044dSC.J. CollierThis section describes an example setup for Phy-vSwitch-VM-Phy communication.
1405129044dSC.J. Collier
1415129044dSC.J. Collier.. _figure_vswitch_vm:
1425129044dSC.J. Collier
1435129044dSC.J. Collier.. figure:: img/vswitch_vm.*
1445129044dSC.J. Collier
1455129044dSC.J. Collier   VMXNET3 with a Native NIC Connected to a vSwitch
1465129044dSC.J. Collier
1475129044dSC.J. Collier.. note::
1485129044dSC.J. Collier
1495129044dSC.J. Collier    Other instructions on preparing to use DPDK such as, hugepage enabling, uio port binding are not listed here.
1505129044dSC.J. Collier    Please refer to *DPDK Getting Started Guide and DPDK Sample Application's User Guide* for detailed instructions.
1515129044dSC.J. Collier
1525129044dSC.J. CollierThe packet reception and transmission flow path is::
1535129044dSC.J. Collier
1545129044dSC.J. Collier    Packet generator -> 82576
1555129044dSC.J. Collier                     -> VMware ESXi vSwitch
1565129044dSC.J. Collier                     -> VMXNET3 device
1575129044dSC.J. Collier                     -> Guest VM VMXNET3 port 0 rx burst
1585129044dSC.J. Collier                     -> Guest VM 82599 VF port 0 tx burst
1595129044dSC.J. Collier                     -> 82599 VF
1605129044dSC.J. Collier                     -> Packet generator
1615129044dSC.J. Collier
1625129044dSC.J. CollierVMXNET3 Chaining VMs Connected to a vSwitch
1635129044dSC.J. Collier-------------------------------------------
1645129044dSC.J. Collier
1655129044dSC.J. CollierThe following figure shows an example VM-to-VM communication over a Phy-VM-vSwitch-VM-Phy communication channel.
1665129044dSC.J. Collier
1675129044dSC.J. Collier.. _figure_vm_vm_comms:
1685129044dSC.J. Collier
1695129044dSC.J. Collier.. figure:: img/vm_vm_comms.*
1705129044dSC.J. Collier
1715129044dSC.J. Collier   VMXNET3 Chaining VMs Connected to a vSwitch
1725129044dSC.J. Collier
1735129044dSC.J. Collier.. note::
1745129044dSC.J. Collier
1755129044dSC.J. Collier    When using the L2 Forwarding or L3 Forwarding applications,
1765129044dSC.J. Collier    a destination MAC address needs to be written in packets to hit the other VM's VMXNET3 interface.
1775129044dSC.J. Collier
1785129044dSC.J. CollierIn this example, the packet flow path is::
1795129044dSC.J. Collier
1805129044dSC.J. Collier    Packet generator -> 82599 VF
1815129044dSC.J. Collier                     -> Guest VM 82599 port 0 rx burst
1825129044dSC.J. Collier                     -> Guest VM VMXNET3 port 1 tx burst
1835129044dSC.J. Collier                     -> VMXNET3 device
1845129044dSC.J. Collier                     -> VMware ESXi vSwitch
1855129044dSC.J. Collier                     -> VMXNET3 device
1865129044dSC.J. Collier                     -> Guest VM VMXNET3 port 0 rx burst
1875129044dSC.J. Collier                     -> Guest VM 82599 VF port 1 tx burst
1885129044dSC.J. Collier                     -> 82599 VF
1895129044dSC.J. Collier                     -> Packet generator
190