lm_bond_virtio_sriov.rst revision 6b3e017e
1..  BSD LICENSE
2    Copyright(c) 2016 Intel Corporation. All rights reserved.
3    All rights reserved.
4
5    Redistribution and use in source and binary forms, with or without
6    modification, are permitted provided that the following conditions
7    are met:
8
9    * Redistributions of source code must retain the above copyright
10    notice, this list of conditions and the following disclaimer.
11    * Redistributions in binary form must reproduce the above copyright
12    notice, this list of conditions and the following disclaimer in
13    the documentation and/or other materials provided with the
14    distribution.
15    * Neither the name of Intel Corporation nor the names of its
16    contributors may be used to endorse or promote products derived
17    from this software without specific prior written permission.
18
19    THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
20    "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
21    LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
22    A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
23    OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
24    SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
25    LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
26    DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
27    THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
28    (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
29    OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
30
31Live Migration of VM with SR-IOV VF
32===================================
33
34Overview
35--------
36
37It is not possible to migrate a Virtual Machine which has an SR-IOV Virtual Function (VF).
38
39To get around this problem the bonding PMD is used.
40
41The following sections show an example of how to do this.
42
43Test Setup
44----------
45
46A bonded device is created in the VM.
47The virtio and VF PMD's are added as slaves to the bonded device.
48The VF is set as the primary slave of the bonded device.
49
50A bridge must be set up on the Host connecting the tap device, which is the
51backend of the Virtio device and the Physical Function (PF) device.
52
53To test the Live Migration two servers with identical operating systems installed are used.
54KVM and Qemu 2.3 is also required on the servers.
55
56In this example, the servers have Niantic and or Fortville NIC's installed.
57The NIC's on both servers are connected to a switch
58which is also connected to the traffic generator.
59
60The switch is configured to broadcast traffic on all the NIC ports.
61A :ref:`Sample switch configuration <lm_bond_virtio_sriov_switch_conf>`
62can be found in this section.
63
64The host is running the Kernel PF driver (ixgbe or i40e).
65
66The ip address of host_server_1 is 10.237.212.46
67
68The ip address of host_server_2 is 10.237.212.131
69
70.. _figure_lm_bond_virtio_sriov:
71
72.. figure:: img/lm_bond_virtio_sriov.*
73
74Live Migration steps
75--------------------
76
77The sample scripts mentioned in the steps below can be found in the
78:ref:`Sample host scripts <lm_bond_virtio_sriov_host_scripts>` and
79:ref:`Sample VM scripts <lm_bond_virtio_sriov_vm_scripts>` sections.
80
81On host_server_1: Terminal 1
82~~~~~~~~~~~~~~~~~~~~~~~~~~~~
83
84.. code-block:: console
85
86   cd /root/dpdk/host_scripts
87   ./setup_vf_on_212_46.sh
88
89For Fortville NIC
90
91.. code-block:: console
92
93   ./vm_virtio_vf_i40e_212_46.sh
94
95For Niantic NIC
96
97.. code-block:: console
98
99   ./vm_virtio_vf_one_212_46.sh
100
101On host_server_1: Terminal 2
102~~~~~~~~~~~~~~~~~~~~~~~~~~~~
103
104.. code-block:: console
105
106   cd /root/dpdk/host_scripts
107   ./setup_bridge_on_212_46.sh
108   ./connect_to_qemu_mon_on_host.sh
109   (qemu)
110
111On host_server_1: Terminal 1
112~~~~~~~~~~~~~~~~~~~~~~~~~~~~
113
114**In VM on host_server_1:**
115
116.. code-block:: console
117
118   cd /root/dpdk/vm_scripts
119   ./setup_dpdk_in_vm.sh
120   ./run_testpmd_bonding_in_vm.sh
121
122   testpmd> show port info all
123
124The ``mac_addr`` command only works with kernel PF for Niantic
125
126.. code-block:: console
127
128   testpmd> mac_addr add port 1 vf 0 AA:BB:CC:DD:EE:FF
129
130The syntax of the ``testpmd`` command is:
131
132Create bonded device (mode) (socket).
133
134Mode 1 is active backup.
135
136Virtio is port 0 (P0).
137
138VF is port 1 (P1).
139
140Bonding is port 2 (P2).
141
142.. code-block:: console
143
144   testpmd> create bonded device 1 0
145   Created new bonded device net_bond_testpmd_0 on (port 2).
146   testpmd> add bonding slave 0 2
147   testpmd> add bonding slave 1 2
148   testpmd> show bonding config 2
149
150The syntax of the ``testpmd`` command is:
151
152set bonding primary (slave id) (port id)
153
154Set primary to P1 before starting bonding port.
155
156.. code-block:: console
157
158   testpmd> set bonding primary 1 2
159   testpmd> show bonding config 2
160   testpmd> port start 2
161   Port 2: 02:09:C0:68:99:A5
162   Checking link statuses...
163   Port 0 Link Up - speed 10000 Mbps - full-duplex
164   Port 1 Link Up - speed 10000 Mbps - full-duplex
165   Port 2 Link Up - speed 10000 Mbps - full-duplex
166
167   testpmd> show bonding config 2
168
169Primary is now P1. There are 2 active slaves.
170
171Use P2 only for forwarding.
172
173.. code-block:: console
174
175   testpmd> set portlist 2
176   testpmd> show config fwd
177   testpmd> set fwd mac
178   testpmd> start
179   testpmd> show bonding config 2
180
181Primary is now P1. There are 2 active slaves.
182
183.. code-block:: console
184
185   testpmd> show port stats all
186
187VF traffic is seen at P1 and P2.
188
189.. code-block:: console
190
191   testpmd> clear port stats all
192   testpmd> set bonding primary 0 2
193   testpmd> remove bonding slave 1 2
194   testpmd> show bonding config 2
195
196Primary is now P0. There is 1 active slave.
197
198.. code-block:: console
199
200   testpmd> clear port stats all
201   testpmd> show port stats all
202
203No VF traffic is seen at P0 and P2, VF MAC address still present.
204
205.. code-block:: console
206
207   testpmd> port stop 1
208   testpmd> port close 1
209
210Port close should remove VF MAC address, it does not remove perm_addr.
211
212The ``mac_addr`` command only works with the kernel PF for Niantic.
213
214.. code-block:: console
215
216   testpmd> mac_addr remove 1 AA:BB:CC:DD:EE:FF
217   testpmd> port detach 1
218   Port '0000:00:04.0' is detached. Now total ports is 2
219   testpmd> show port stats all
220
221No VF traffic is seen at P0 and P2.
222
223On host_server_1: Terminal 2
224~~~~~~~~~~~~~~~~~~~~~~~~~~~~
225
226.. code-block:: console
227
228   (qemu) device_del vf1
229
230
231On host_server_1: Terminal 1
232~~~~~~~~~~~~~~~~~~~~~~~~~~~~
233
234**In VM on host_server_1:**
235
236.. code-block:: console
237
238   testpmd> show bonding config 2
239
240Primary is now P0. There is 1 active slave.
241
242.. code-block:: console
243
244   testpmd> show port info all
245   testpmd> show port stats all
246
247On host_server_2: Terminal 1
248~~~~~~~~~~~~~~~~~~~~~~~~~~~~
249
250.. code-block:: console
251
252   cd /root/dpdk/host_scripts
253   ./setup_vf_on_212_131.sh
254   ./vm_virtio_one_migrate.sh
255
256On host_server_2: Terminal 2
257~~~~~~~~~~~~~~~~~~~~~~~~~~~~
258
259.. code-block:: console
260
261   ./setup_bridge_on_212_131.sh
262   ./connect_to_qemu_mon_on_host.sh
263   (qemu) info status
264   VM status: paused (inmigrate)
265   (qemu)
266
267On host_server_1: Terminal 2
268~~~~~~~~~~~~~~~~~~~~~~~~~~~~
269
270Check that the switch is up before migrating.
271
272.. code-block:: console
273
274   (qemu) migrate tcp:10.237.212.131:5555
275   (qemu) info status
276   VM status: paused (postmigrate)
277
278For the Niantic NIC.
279
280.. code-block:: console
281
282   (qemu) info migrate
283   capabilities: xbzrle: off rdma-pin-all: off auto-converge: off zero-blocks: off
284   Migration status: completed
285   total time: 11834 milliseconds
286   downtime: 18 milliseconds
287   setup: 3 milliseconds
288   transferred ram: 389137 kbytes
289   throughput: 269.49 mbps
290   remaining ram: 0 kbytes
291   total ram: 1590088 kbytes
292   duplicate: 301620 pages
293   skipped: 0 pages
294   normal: 96433 pages
295   normal bytes: 385732 kbytes
296   dirty sync count: 2
297   (qemu) quit
298
299For the Fortville NIC.
300
301.. code-block:: console
302
303   (qemu) info migrate
304   capabilities: xbzrle: off rdma-pin-all: off auto-converge: off zero-blocks: off
305   Migration status: completed
306   total time: 11619 milliseconds
307   downtime: 5 milliseconds
308   setup: 7 milliseconds
309   transferred ram: 379699 kbytes
310   throughput: 267.82 mbps
311   remaining ram: 0 kbytes
312   total ram: 1590088 kbytes
313   duplicate: 303985 pages
314   skipped: 0 pages
315   normal: 94073 pages
316   normal bytes: 376292 kbytes
317   dirty sync count: 2
318   (qemu) quit
319
320On host_server_2: Terminal 1
321~~~~~~~~~~~~~~~~~~~~~~~~~~~~
322
323**In VM on host_server_2:**
324
325   Hit Enter key. This brings the user to the testpmd prompt.
326
327.. code-block:: console
328
329   testpmd>
330
331On host_server_2: Terminal 2
332~~~~~~~~~~~~~~~~~~~~~~~~~~~~
333
334.. code-block:: console
335
336   (qemu) info status
337   VM status: running
338
339For the Niantic NIC.
340
341.. code-block:: console
342
343   (qemu) device_add pci-assign,host=06:10.0,id=vf1
344
345For the Fortville NIC.
346
347.. code-block:: console
348
349   (qemu) device_add pci-assign,host=03:02.0,id=vf1
350
351On host_server_2: Terminal 1
352~~~~~~~~~~~~~~~~~~~~~~~~~~~~
353
354**In VM on host_server_2:**
355
356.. code-block:: console
357
358   testomd> show port info all
359   testpmd> show port stats all
360   testpmd> show bonding config 2
361   testpmd> port attach 0000:00:04.0
362   Port 1 is attached.
363   Now total ports is 3
364   Done
365
366   testpmd> port start 1
367
368The ``mac_addr`` command only works with the Kernel PF for Niantic.
369
370.. code-block:: console
371
372   testpmd> mac_addr add port 1 vf 0 AA:BB:CC:DD:EE:FF
373   testpmd> show port stats all.
374   testpmd> show config fwd
375   testpmd> show bonding config 2
376   testpmd> add bonding slave 1 2
377   testpmd> set bonding primary 1 2
378   testpmd> show bonding config 2
379   testpmd> show port stats all
380
381VF traffic is seen at P1 (VF) and P2 (Bonded device).
382
383.. code-block:: console
384
385   testpmd> remove bonding slave 0 2
386   testpmd> show bonding config 2
387   testpmd> port stop 0
388   testpmd> port close 0
389   testpmd> port detach 0
390   Port '0000:00:03.0' is detached. Now total ports is 2
391
392   testpmd> show port info all
393   testpmd> show config fwd
394   testpmd> show port stats all
395
396VF traffic is seen at P1 (VF) and P2 (Bonded device).
397
398.. _lm_bond_virtio_sriov_host_scripts:
399
400Sample host scripts
401-------------------
402
403setup_vf_on_212_46.sh
404~~~~~~~~~~~~~~~~~~~~~
405Set up Virtual Functions on host_server_1
406
407.. code-block:: sh
408
409   #!/bin/sh
410   # This script is run on the host 10.237.212.46 to setup the VF
411
412   # set up Niantic VF
413   cat /sys/bus/pci/devices/0000\:09\:00.0/sriov_numvfs
414   echo 1 > /sys/bus/pci/devices/0000\:09\:00.0/sriov_numvfs
415   cat /sys/bus/pci/devices/0000\:09\:00.0/sriov_numvfs
416   rmmod ixgbevf
417
418   # set up Fortville VF
419   cat /sys/bus/pci/devices/0000\:02\:00.0/sriov_numvfs
420   echo 1 > /sys/bus/pci/devices/0000\:02\:00.0/sriov_numvfs
421   cat /sys/bus/pci/devices/0000\:02\:00.0/sriov_numvfs
422   rmmod i40evf
423
424vm_virtio_vf_one_212_46.sh
425~~~~~~~~~~~~~~~~~~~~~~~~~~
426
427Setup Virtual Machine on host_server_1
428
429.. code-block:: sh
430
431   #!/bin/sh
432
433   # Path to KVM tool
434   KVM_PATH="/usr/bin/qemu-system-x86_64"
435
436   # Guest Disk image
437   DISK_IMG="/home/username/disk_image/virt1_sml.disk"
438
439   # Number of guest cpus
440   VCPUS_NR="4"
441
442   # Memory
443   MEM=1536
444
445   taskset -c 1-5 $KVM_PATH \
446    -enable-kvm \
447    -m $MEM \
448    -smp $VCPUS_NR \
449    -cpu host \
450    -name VM1 \
451    -no-reboot \
452    -net none \
453    -vnc none -nographic \
454    -hda $DISK_IMG \
455    -netdev type=tap,id=net1,script=no,downscript=no,ifname=tap1 \
456    -device virtio-net-pci,netdev=net1,mac=CC:BB:BB:BB:BB:BB \
457    -device pci-assign,host=09:10.0,id=vf1 \
458    -monitor telnet::3333,server,nowait
459
460setup_bridge_on_212_46.sh
461~~~~~~~~~~~~~~~~~~~~~~~~~
462
463Setup bridge on host_server_1
464
465.. code-block:: sh
466
467   #!/bin/sh
468   # This script is run on the host 10.237.212.46 to setup the bridge
469   # for the Tap device and the PF device.
470   # This enables traffic to go from the PF to the Tap to the Virtio PMD in the VM.
471
472   # ens3f0 is the Niantic NIC
473   # ens6f0 is the Fortville NIC
474
475   ifconfig ens3f0 down
476   ifconfig tap1 down
477   ifconfig ens6f0 down
478   ifconfig virbr0 down
479
480   brctl show virbr0
481   brctl addif virbr0 ens3f0
482   brctl addif virbr0 ens6f0
483   brctl addif virbr0 tap1
484   brctl show virbr0
485
486   ifconfig ens3f0 up
487   ifconfig tap1 up
488   ifconfig ens6f0 up
489   ifconfig virbr0 up
490
491connect_to_qemu_mon_on_host.sh
492~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
493
494.. code-block:: sh
495
496   #!/bin/sh
497   # This script is run on both hosts when the VM is up,
498   # to connect to the Qemu Monitor.
499
500   telnet 0 3333
501
502setup_vf_on_212_131.sh
503~~~~~~~~~~~~~~~~~~~~~~
504
505Set up Virtual Functions on host_server_2
506
507.. code-block:: sh
508
509   #!/bin/sh
510   # This script is run on the host 10.237.212.131 to setup the VF
511
512   # set up Niantic VF
513   cat /sys/bus/pci/devices/0000\:06\:00.0/sriov_numvfs
514   echo 1 > /sys/bus/pci/devices/0000\:06\:00.0/sriov_numvfs
515   cat /sys/bus/pci/devices/0000\:06\:00.0/sriov_numvfs
516   rmmod ixgbevf
517
518   # set up Fortville VF
519   cat /sys/bus/pci/devices/0000\:03\:00.0/sriov_numvfs
520   echo 1 > /sys/bus/pci/devices/0000\:03\:00.0/sriov_numvfs
521   cat /sys/bus/pci/devices/0000\:03\:00.0/sriov_numvfs
522   rmmod i40evf
523
524vm_virtio_one_migrate.sh
525~~~~~~~~~~~~~~~~~~~~~~~~
526
527Setup Virtual Machine on host_server_2
528
529.. code-block:: sh
530
531   #!/bin/sh
532   # Start the VM on host_server_2 with the same parameters except without the VF
533   # parameters, as the VM on host_server_1, in migration-listen mode
534   # (-incoming tcp:0:5555)
535
536   # Path to KVM tool
537   KVM_PATH="/usr/bin/qemu-system-x86_64"
538
539   # Guest Disk image
540   DISK_IMG="/home/username/disk_image/virt1_sml.disk"
541
542   # Number of guest cpus
543   VCPUS_NR="4"
544
545   # Memory
546   MEM=1536
547
548   taskset -c 1-5 $KVM_PATH \
549    -enable-kvm \
550    -m $MEM \
551    -smp $VCPUS_NR \
552    -cpu host \
553    -name VM1 \
554    -no-reboot \
555    -net none \
556    -vnc none -nographic \
557    -hda $DISK_IMG \
558    -netdev type=tap,id=net1,script=no,downscript=no,ifname=tap1 \
559    -device virtio-net-pci,netdev=net1,mac=CC:BB:BB:BB:BB:BB \
560    -incoming tcp:0:5555 \
561    -monitor telnet::3333,server,nowait
562
563setup_bridge_on_212_131.sh
564~~~~~~~~~~~~~~~~~~~~~~~~~~
565
566Setup bridge on host_server_2
567
568.. code-block:: sh
569
570   #!/bin/sh
571   # This script is run on the host to setup the bridge
572   # for the Tap device and the PF device.
573   # This enables traffic to go from the PF to the Tap to the Virtio PMD in the VM.
574
575   # ens4f0 is the Niantic NIC
576   # ens5f0 is the Fortville NIC
577
578   ifconfig ens4f0 down
579   ifconfig tap1 down
580   ifconfig ens5f0 down
581   ifconfig virbr0 down
582
583   brctl show virbr0
584   brctl addif virbr0 ens4f0
585   brctl addif virbr0 ens5f0
586   brctl addif virbr0 tap1
587   brctl show virbr0
588
589   ifconfig ens4f0 up
590   ifconfig tap1 up
591   ifconfig ens5f0 up
592   ifconfig virbr0 up
593
594.. _lm_bond_virtio_sriov_vm_scripts:
595
596Sample VM scripts
597-----------------
598
599setup_dpdk_in_vm.sh
600~~~~~~~~~~~~~~~~~~~
601
602Set up DPDK in the Virtual Machine
603
604.. code-block:: sh
605
606   #!/bin/sh
607   # this script matches the vm_virtio_vf_one script
608   # virtio port is 03
609   # vf port is 04
610
611   cat  /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
612   echo 1024 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
613   cat  /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
614
615   ifconfig -a
616   /root/dpdk/tools/dpdk-devbind.py --status
617
618   rmmod virtio-pci ixgbevf
619
620   modprobe uio
621   insmod /root/dpdk/x86_64-default-linuxapp-gcc/kmod/igb_uio.ko
622
623   /root/dpdk/tools/dpdk-devbind.py -b igb_uio 0000:00:03.0
624   /root/dpdk/tools/dpdk-devbind.py -b igb_uio 0000:00:04.0
625
626   /root/dpdk/tools/dpdk-devbind.py --status
627
628run_testpmd_bonding_in_vm.sh
629~~~~~~~~~~~~~~~~~~~~~~~~~~~~
630
631Run testpmd in the Virtual Machine.
632
633.. code-block:: sh
634
635   #!/bin/sh
636   # Run testpmd in the VM
637
638   # The test system has 8 cpus (0-7), use cpus 2-7 for VM
639   # Use taskset -pc <core number> <thread_id>
640
641   # use for bonding of virtio and vf tests in VM
642
643   /root/dpdk/x86_64-default-linuxapp-gcc/app/testpmd \
644   -c f -n 4 --socket-mem 350 --  --i --port-topology=chained
645
646.. _lm_bond_virtio_sriov_switch_conf:
647
648Sample switch configuration
649---------------------------
650
651The Intel switch is used to connect the traffic generator to the
652NIC's on host_server_1 and host_server_2.
653
654In order to run the switch configuration two console windows are required.
655
656Log in as root in both windows.
657
658TestPointShared, run_switch.sh and load /root/switch_config must be executed
659in the sequence below.
660
661On Switch: Terminal 1
662~~~~~~~~~~~~~~~~~~~~~
663
664run TestPointShared
665
666.. code-block:: console
667
668   /usr/bin/TestPointShared
669
670On Switch: Terminal 2
671~~~~~~~~~~~~~~~~~~~~~
672
673execute run_switch.sh
674
675.. code-block:: console
676
677   /root/run_switch.sh
678
679On Switch: Terminal 1
680~~~~~~~~~~~~~~~~~~~~~
681
682load switch configuration
683
684.. code-block:: console
685
686   load /root/switch_config
687
688Sample switch configuration script
689~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
690
691The ``/root/switch_config`` script:
692
693.. code-block:: sh
694
695   # TestPoint History
696   show port 1,5,9,13,17,21,25
697   set port 1,5,9,13,17,21,25 up
698   show port 1,5,9,13,17,21,25
699   del acl 1
700   create acl 1
701   create acl-port-set
702   create acl-port-set
703   add port port-set 1 0
704   add port port-set 5,9,13,17,21,25 1
705   create acl-rule 1 1
706   add acl-rule condition 1 1 port-set 1
707   add acl-rule action 1 1 redirect 1
708   apply acl
709   create vlan 1000
710   add vlan port 1000 1,5,9,13,17,21,25
711   set vlan tagging 1000 1,5,9,13,17,21,25 tag
712   set switch config flood_ucast fwd
713   show port stats all 1,5,9,13,17,21,25
714