trex_book.asciidoc revision 05136015
1TRex 
2====
3:author: hhaim 
4:email: <hhaim@cisco.com> 
5:revnumber: 2.1
6:revdate: 2017-02-21-a
7:quotes.++:
8:numbered:
9:web_server_url: http://trex-tgn.cisco.com/trex
10:local_web_server_url: csi-wiki-01:8181/trex
11:toclevels: 4
12
13include::trex_ga.asciidoc[]
14
15
16== Introduction
17
18=== A word on traffic generators
19
20Traditionally, routers have been tested using commercial traffic generators, while performance typically has been measured using packets per second (PPS) metrics. As router functionality and services have become more complex, stateful traffic generators have become necessary to provide more realistic traffic scenarios.
21
22Advantages of realistic traffic generators:
23
24* Accurate performance metrics.
25* Discovering bottlenecks in realistic traffic scenarios.
26
27==== Current Challenges:
28
29* *Cost*: Commercial stateful traffic generators are very expensive.
30* *Scale*: Bandwidth does not scale up well with feature complexity.
31* *Standardization*: Lack of standardization of traffic patterns and methodologies.
32* *Flexibility*: Commercial tools are not sufficiently agile when flexibility and customization are needed.
33
34==== Implications 
35
36* High capital expenditure (capex) spent by different teams.
37* Testing in low scale and extrapolation became a common practice. This is non-ideal and fails to indicate bottlenecks that appear in real-world scenarios.
38* Teams use different benchmark methodologies, so results are not standardized.
39* Delays in development and testing due to dependence on testing tool features.
40* Resource and effort investment in developing different ad hoc tools and test methodologies.
41
42=== Overview of TRex
43
44TRex addresses the problems associated with commercial stateful traffic generators, through an innovative and extendable software implementation, and by leveraging standard and open software and x86/UCS hardware.
45
46* Generates and analyzes L4-7 traffic. In one package, provides capabilities of commercial L7 tools.
47* Stateful traffic generator based on pre-processing and smart replay of real traffic templates.
48* Generates and *amplifies* both client- and server-side traffic.
49* Customized functionality can be added.
50* Scales to 200Gb/sec for one UCS (using Intel 40Gb/sec NICs).
51* Low cost.
52* Self-contained package that can be easily installed and deployed.
53* Virtual interface support enables TRex to be used in a fully virtual environment without physical NICs. Example use cases:
54** Amazon AWS
55** Cisco LaaS
56// Which LaaS is this? Location as a service? Linux?
57** TRex on your laptop
58
59
60
61.TRex Hardware 
62[options="header",cols="1^,1^"]
63|=================
64|Cisco UCS Platform | Intel NIC 
65| image:images/ucs200_2.png[title="platform"] | image:images/Intel520.png[title="NIC"]  
66|=================
67
68=== Purpose of this guide
69
70This guide explains the use of TRex internals and the use of TRex together with Cisco ASR1000 Series routers. The examples illustrate novel traffic generation techniques made possible by TRex. 
71
72== Download and installation
73
74=== Hardware recommendations
75
76TRex operates in a Linux application environment, interacting with Linux kernel modules. 
77TRex curretly works on x86 architecture and can operate well on Cisco UCS hardware. The following platforms have been tested and are recommended for operating TRex. 
78
79[NOTE]
80=====================================
81 A high-end UCS platform is not required for operating TRex in its current version, but may be required for future versions.
82=====================================
83
84[NOTE]
85=====================================
86 Not all supported DPDK interfaces are supported by TRex.
87=====================================
88 
89
90.Preferred UCS hardware
91[options="header",cols="1,3"]
92|=================
93| UCS Type | Comments 
94| UCS C220 Mx  | *Preferred Low-End*. Supports up to 40Gb/sec with 540-D2. With newer Intel NIC (recommended), supports 80Gb/sec with 1RU. See table below describing components.
95| UCS C200| Early UCS model.
96| UCS C210 Mx | Supports up to 40Gb/sec PCIe3.0.
97| UCS C240 Mx | *Preferred, High-End* Supports up to 200Gb/sec. 6x XL710 NICS (PCIex8) or 2xFM10K (PCIex16). See table below describing components.
98| UCS C260M2 | Supports up to 30Gb/sec (limited by V2 PCIe).
99|=================
100
101.Low-end UCS C220 Mx - Internal components
102[options="header",cols="1,2",width="60%"]
103|=================
104| Components |  Details 
105| CPU  | 2x E5-2620 @ 2.0 GHz. 
106| CPU Configuration | 2-Socket CPU configurations (also works with 1 CPU).
107| Memory | 2x4 banks f.or each CPU. Total of 32GB in 8 banks.	 
108| RAID | No RAID.
109|=================
110
111.High-end C240 Mx - Internal components 
112[options="header",cols="1,2",width="60%"]
113|=================
114| Components |  Details 
115| CPU  | 2x E5-2667 @ 3.20 GHz.
116| PCIe | 1x Riser PCI expansion card option A PID UCSC-PCI-1A-240M4 enables 2 PCIex16.
117| CPU Configuration | 2-Socket CPU configurations (also works with 1 CPU).
118| Memory | 2x4 banks for each CPU. Total of 32GB in 8 banks. 
119| RAID | No RAID.
120| Riser 1/2 | Both left and right should support x16 PCIe. Right (Riser1) should be from option A x16 and Left (Riser2) should be x16. Need to order both.
121|=================
122 
123.Supported NICs
124[options="header",cols="1,1,4",width="90%"]
125|=================
126| Chipset              | Bandwidth  (Gb/sec)  |  Example
127| Intel I350           | 1   | Intel 4x1GE 350-T4 NIC
128| Intel 82599          | 10  |   Cisco part ID:N2XX-AIPCI01 Intel x520-D2, Intel X520 Dual Port 10Gb SFP+ Adapter
129| Intel 82599 VF       | x  | 
130| Intel X710           | 10  | Cisco part ID:UCSC-PCIE-IQ10GF link:https://en.wikipedia.org/wiki/Small_form-factor_pluggable_transceiver[SFP+], *Preferred* support per stream stats in hardware link:http://www.silicom-usa.com/PE310G4i71L_Quad_Port_Fiber_SFP+_10_Gigabit_Ethernet_PCI_Express_Server_Adapter_49[Silicom PE310G4i71L]
131| Intel XL710          | 40  | Cisco part ID:UCSC-PCIE-ID40GF, link:https://en.wikipedia.org/wiki/QSFP[QSFP+] (copper/optical)
132| Intel XL710/X710 VF  | x  | 
133| Intel 82599 VF       | x  | 
134| Intel FM10420        | 25/100 | QSFP28,  by Silicom link:http://www.silicom-usa.com/100_Gigabit_Dual_Port_Fiber_Ethernet_PCI_Express_PE3100G2DQiR_96[Silicom PE3100G2DQiR_96] (*in development*) 
135| Mellanox ConnectX-4  | 25/40/50/56/100 | QSFP28, link:http://www.mellanox.com/page/products_dyn?product_family=201&[ConnectX-4] link:http://www.mellanox.com/related-docs/prod_adapter_cards/PB_ConnectX-4_VPI_Card.pdf[ConnectX-4-brief] (copper/optical) supported from v2.11 more details xref:connectx_support[TRex Support]
136| Mellanox ConnectX-5  | 25/40/50/56/100 | Supported 
137| Cisco 1300 series    | 40              | QSFP+, VIC 1380,  VIC 1385, VIC 1387 see more xref:ciscovic_support[TRex Support]
138| VMXNET / +
139VMXNET3 (see notes) | VMware paravirtualized  | Connect using VMware vSwitch
140| E1000    | paravirtualized  | VMware/KVM/VirtualBox 
141| Virtio | paravirtualized  | KVM
142|=================
143
144// in table above, is it correct to list "paravirtualized" as chipset? Also, what is QSFP28? It does not appear on the lined URL. Clarify: is Intel X710 the preferred NIC?
145
146.SFP+ support 
147[options="header",cols="2,1,1,1",width="90%"]
148|=================
149| link:https://en.wikipedia.org/wiki/Small_form-factor_pluggable_transceiver[SFP+]  | Intel Ethernet Converged X710-DAX |  Silicom link:http://www.silicom-usa.com/PE310G4i71L_Quad_Port_Fiber_SFP+_10_Gigabit_Ethernet_PCI_Express_Server_Adapter_49[PE310G4i71L] (Open optic) | 82599EB 10-Gigabit
150| link:http://www.cisco.com/c/en/us/products/collateral/interfaces-modules/transceiver-modules/data_sheet_c78-455693.html[Cisco SFP-10G-SR] | Not supported  | [green]*Supported* | [green]*Supported*
151| link:http://www.cisco.com/c/en/us/products/collateral/interfaces-modules/transceiver-modules/data_sheet_c78-455693.html[Cisco SFP-10G-LR] | Not supported     | [green]*Supported* | [green]*Supported*
152| link:http://www.cisco.com/c/en/us/products/collateral/interfaces-modules/transceiver-modules/data_sheet_c78-455693.html[Cisco SFP-H10GB-CU1M]| [green]*Supported* | [green]*Supported* | [green]*Supported*
153| link:http://www.cisco.com/c/en/us/products/collateral/interfaces-modules/transceiver-modules/data_sheet_c78-455693.html[Cisco SFP-10G-AOC1M] | [green]*Supported* | [green]*Supported* | [green]*Supported*
154|=================
155
156[NOTE]
157=====================================
158 Intel X710 NIC (example: FH X710DA4FHBLK) operates *only* with Intel SFP+. For open optic, use the Silicom PE310G4i71L NIC, available here:
159 http://www.silicom-usa.com/PE310G4i71L_Quad_Port_Fiber_SFP+_10_Gigabit_Ethernet_PCI_Express_Server_Adapter_49
160=====================================
161// it appears that link:<link> doesn't work in a note, so i changed the wording
162
163// clarify above table and note
164
165.XL710 NIC base QSFP+ support 
166[options="header",cols="1,1,1",width="90%"]
167|=================
168| link:https://en.wikipedia.org/wiki/QSFP[QSFP+]             | Intel Ethernet Converged XL710-QDAX                                          | Silicom link:http://www.silicom-usa.com/Dual_Port_Fiber_40_Gigabit_Ethernet_PCI_Express_Server_Adapter_PE340G2Qi71_83[PE340G2Qi71] Open optic 
169| QSFP+ SR4 optics                                           | [green]*Supported*: APPROVED OPTICS. *Not supported*: Cisco QSFP-40G-SR4-S   | [green]*Supported*: Cisco QSFP-40G-SR4-S  
170| QSFP+ LR-4 Optics                                          | [green]*Supported*: APPROVED OPTICS. *Not supported*: Cisco QSFP-40G-LR4-S   | [green]*Supported*: Cisco QSFP-40G-LR4-S 
171| QSFP Active Optical Cables (AoC)                           | [green]*Supported*: Cisco QSFP-H40G-AOC                                      | [green]*Supported*: Cisco QSFP-H40G-AOC  
172| QSFP+ Intel Ethernet Modular Supported                     | N/A                                                                          | N/A
173| QSFP+ DA twin-ax cables                                    | N/A                                                                          | N/A  
174| Active QSFP+ Copper Cables                                 | [green]*Supported*: Cisco QSFP-4SFP10G-CU                                    | [green]*Supported*: Cisco QSFP-4SFP10G-CU
175|=================
176
177[NOTE]
178=====================================
179 For Intel XL710 NICs, Cisco SR4/LR QSFP+ does not operate. Use Silicom with Open Optic.
180=====================================
181
182
183.ConnectX-4 NIC base QSFP28 support (100gb)
184[options="header",cols="1,2",width="90%"]
185|=================
186| link:https://en.wikipedia.org/wiki/QSFP[QSFP28]             | ConnectX-4
187| QSFP28 SR4 optics                                           | N/A
188| QSFP28 LR-4 Optics                                          | N/A 
189| QSFP28 (AoC)                                                | [green]*Supported*: Cisco QSFP-100G-AOCxM    
190| QSFP28 DA twin-ax cables                                    | [green]*Supported*: Cisco QSFP-100G-CUxM
191|=================
192
193.Cisco VIC NIC base QSFP+ support 
194[options="header",cols="1,2",width="90%"]
195|=================
196| link:https://en.wikipedia.org/wiki/QSFP[QSFP+]             | Intel Ethernet Converged XL710-QDAX 
197| QSFP+ SR4 optics                                           | N/A
198| QSFP+ LR-4 Optics                                          | N/A
199| QSFP Active Optical Cables (AoC)                           | [green]*Supported*: Cisco QSFP-H40G-AOC
200| QSFP+ Intel Ethernet Modular Optics                        | N/A            
201| QSFP+ DA twin-ax cables                                    | N/A  
202| Active QSFP+ Copper Cables                                 | N/A
203|=================
204
205
206// clarify above table and note. let's discuss.
207.FM10K QSFP28 support 
208[options="header",cols="1,1",width="70%"]
209|=================
210| QSFP28             | Example 
211| todo  |  todo
212|=================
213
214// do we want to show "todo"? maybe "pending"
215
216
217[IMPORTANT]
218=====================================
219* Intel SFP+ 10Gb/sec is the only one supported by default on the standard Linux driver. TRex also supports Cisco 10Gb/sec SFP+. 
220// above, replace "only one" with "only mode"?
221* For operating high speed throughput (example: several Intel XL710 40Gb/sec), use different link:https://en.wikipedia.org/wiki/Non-uniform_memory_access[NUMA] nodes for different NICs. +
222    To verify NUMA and NIC topology: `lstopo (yum install hwloc)` +
223    To display CPU info, including NUMA node: `lscpu` +
224    NUMA usage: xref:numa-example[example]
225* For the Intel XL710 NIC, verify that the NVM is v5.04. xref:xl710-firmware[Info].
226**  `> sudo ./t-rex-64 -f cap2/dns.yaml -d 0 *-v 6* --nc | grep NVM` +
227    `PMD:  FW 5.0 API 1.5 NVM 05.00.04  eetrack 800013fc` 
228=====================================
229
230// above, maybe rename the bullet points "NIC usage notes"? should we create a subsection for NICs? Maybe it would be under "2.1 Hardware recommendations" as a subsection.
231
232
233.Sample order for recommended low-end Cisco UCSC-C220-M3S with 4x10Gb ports
234[options="header",cols="1,1",width="70%"]
235|=================
236| Component | Quantity 
237| UCSC-C220-M3S    |  1
238| UCS-CPU-E5-2650  |  2
239| UCS-MR-1X041RY-A |  8
240| A03-D500GC3      |  1
241| N2XX-AIPCI01     |  2
242| UCSC-PSU-650W    |  1
243| SFS-250V-10A-IS  |  1
244| UCSC-CMA1        |  1
245| UCSC-HS-C220M3   |  2
246| N20-BBLKD        |  7
247| UCSC-PSU-BLKP    |  1
248| UCSC-RAIL1       |  1
249|=================
250
251// should table above say "low-end Cisco UCS C220 M3S" instead of "low-end USCS-C220-M3S"?
252
253NOTE: Purchase the 10Gb/sec SFP+ separately. Cisco would be fine with TRex (but not for plain Linux driver).
254// does note above mean "TRex operates with 10Gb/sec SFP+ components, but plain Linux does not provide drivers."? if so, how does purchasing separately solve this? where do they get drivers?
255
256=== Installing OS 
257
258==== Supported versions
259
260Supported Linux versions:
261
262* Fedora 20-23, 64-bit kernel (not 32-bit)
263* Ubuntu 14.04.1 LTS, 64-bit kernel (not 32-bit)
264* Ubuntu 16.xx LTS, 64-bit kernel (not 32-bit) -- Not fully supported.
265// clarify "not fully supported"
266* CentOs/RedHat 7.2 LTS, 64-bit kernel (not 32-bit) -- This is the only working option for ConnectX-4.
267
268NOTE: Additional OS versions may be supported by compiling the necessary drivers.
269
270To check whether a kernel is 64-bit, verify that the ouput of the following command is `x86_64`.
271
272[source,bash]
273----
274$uname -m
275x86_64 
276----
277
278
279==== Download Linux
280
281ISO images for supported Linux releases can be downloaded from:
282
283.Supported Linux ISO image links
284[options="header",cols="1^,2^",width="50%"]
285|======================================
286| Distribution                                                                                                                                | SHA256 Checksum
287| link:http://archives.fedoraproject.org/pub/archive/fedora/linux/releases/20/Fedora/x86_64/iso/Fedora-20-x86_64-DVD.iso[Fedora 20]           | 
288 link:http://archives.fedoraproject.org/pub/archive/fedora/linux/releases/20/Fedora/x86_64/iso/Fedora-20-x86_64-CHECKSUM[Fedora 20 CHECKSUM] 
289| link:http://fedora-mirror01.rbc.ru/pub/fedora/linux/releases/21/Server/x86_64/iso/Fedora-Server-DVD-x86_64-21.iso[Fedora 21]                |
290 link:http://fedora-mirror01.rbc.ru/pub/fedora/linux/releases/21/Server/x86_64/iso/Fedora-Server-21-x86_64-CHECKSUM[Fedora 21 CHECKSUM] 
291| link:http://old-releases.ubuntu.com/releases/14.04.1/ubuntu-14.04-desktop-amd64.iso[Ubuntu 14.04.1]                                         |
292 http://old-releases.ubuntu.com/releases/14.04.1/SHA256SUMS[Ubuntu 14.04* CHECKSUMs] 
293| link:http://releases.ubuntu.com/16.04.1/ubuntu-16.04.1-server-amd64.iso[Ubuntu 16.04.1]                                                     |
294 http://releases.ubuntu.com/16.04.1/SHA256SUMS[Ubuntu 16.04* CHECKSUMs]
295|======================================
296
297
298For Fedora downloads...
299
300* Select a mirror close to your location: +
301https://admin.fedoraproject.org/mirrormanager/mirrors/Fedora +
302Choose: "Fedora Linux http" -> releases -> <version number> -> Server -> x86_64 -> iso -> Fedora-Server-DVD-x86_64-<version number>.iso
303
304* Verify that the link:https://en.wikipedia.org/wiki/SHA-2[SHA-256] checksum value of the downloaded file matches the linked checksum values with the `sha256sum` command. Example:
305
306[source,bash]
307----
308$sha256sum Fedora-18-x86_64-DVD.iso
30991c5f0aca391acf76a047e284144f90d66d3d5f5dcd26b01f368a43236832c03 
310----
311
312
313==== Install Linux 
314
315Ask your lab admin to install the Linux using CIMC, assign an IP, and set the DNS. Request the sudo or super user password to enable you to ping and SSH.
316
317xref:fedora21_example[Example of installing Fedora 21 Server]
318
319[NOTE]
320=====================================
321 * Requirement for using TRex: sudo or root password for the machine.
322 * Upgrading the linux Kernel using `yum upgrade` requires building the TRex drivers. 
323 * In Ubuntu 16, auto-updater is enabled by default. It is recommended to turn it off. Updating kernel requires re-compiling the DPDK .ko file. +
324To disable auto-updater: +
325 > sudo apt-get remove unattended-upgrades
326=====================================
327
328==== Verify Intel NIC installation
329
330Use `lspci` to verify the NIC installation. 
331
332Example: 4x 10Gb/sec TRex configuration (see output below):
333
334* I350 management port
335
336* 4x Intel Ethernet Converged Network Adapter model x520-D2 (82599 chipset)
337
338[source,bash]
339----
340$[root@trex]lspci | grep Ethernet
34101:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)                #<1>
34201:00.1 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)                #<2>
34303:00.0 Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01) #<3>
34403:00.1 Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01)
34582:00.0 Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01)
34682:00.1 Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01)
347----
348<1> Management port 
349<2> CIMC port
350<3> 10Gb/sec traffic ports (Intel 82599EB) 
351
352=== Obtaining the TRex package
353
354Use `ssh` to connect to the TRex machine and execute the commands described below.
355
356NOTE: Prerequisite: *$WEB_URL* is *{web_server_url}* or *{local_web_server_url}* (Cisco internal)
357
358// Clarify the note above and probably should not have the Cisco internal part
359
360Latest release:
361
362// want to call that "Latest stable release" ?
363
364[source,bash]
365----
366$mkdir trex
367$cd trex
368$wget --no-cache $WEB_URL/release/latest 
369$tar -xzvf latest 
370----
371
372
373Bleeding edge version:
374[source,bash]
375----
376$wget --no-cache $WEB_URL/release/be_latest 
377----
378
379To obtain a specific version:
380[source,bash]
381----
382$wget --no-cache $WEB_URL/release/vX.XX.tar.gz #<1>
383----
384
385<1> X.XX = Version number
386
387== First time Running
388
389=== Configuring for loopback
390
391Before connecting TRex to your DUT, it is strongly advised to verify that TRex and the NICs work correctly in loopback. +
392
393[NOTE]
394=====================================================================
3951. For best performance, loopback the interfaces on the same NUMA (controlled by the same physical processor). If you are unable to check this, proceed without this step.
396
3972. If you are using a 10Gbs NIC based on an Intel 520-D2 NIC, and you loopback ports on the same NIC using SFP+, the device might not sync, causing a failure to link up. +
398Many types of SFP+ (Intel/Cisco/SR/LR) have been verified to work. +
399If you encounter link issues, try to loopback interfaces from different NICs, or use link:http://www.fiberopticshare.com/tag/cisco-10g-twinax[Cisco twinax copper cable].
400=====================================================================
401
402.Loopback example
403image:images/loopback_example.png[title="Loopback example"] 
404
405==== Identify the ports 
406
407Use the following command to identify ports.
408
409[source,bash]
410----
411 $>sudo ./dpdk_setup_ports.py -s
412
413 Network devices using DPDK-compatible driver
414 ============================================
415
416 Network devices using kernel driver
417 ===================================
418 0000:03:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection' drv= unused=ixgb #<1>
419 0000:03:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection' drv= unused=ixgb
420 0000:13:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection' drv= unused=ixgb
421 0000:13:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection' drv= unused=ixgb
422 0000:02:00.0 '82545EM Gigabit Ethernet Controller (Copper)' if=eth2 drv=e1000 unused=igb_uio *Active* #<2>
423
424 Other network devices
425 =====================
426 <none>
427----
428
429<1> If you have not run any DPDK applications, the command output shows a list of interfaces bound to the kernel or not bound at all.
430<2> The interface marked 'active' is the one used by your ssh connection. *Never* put this interface into TRex config file.
431
432// possible to clarify "*Never* put this interface into TRex config file." , such as what putting this into the config file entails and what the consequences are.
433
434Choose the ports to use and follow the instructions in the next section to create a configuration file.
435
436==== Creating minimum configuration file
437
438Default configuration file name: `/etc/trex_cfg.yaml`.
439
440For a full list of YAML configuration file options, see xref:trex_config_yaml_config_file[YAML Configuration File].
441
442For many purposes, it is convenient to begin with a copy of the basic configuration file template, available in the cfg folder:
443
444[source,bash]
445----
446$cp  cfg/simple_cfg.yaml /etc/trex_cfg.yaml
447----
448
449Next, edit the configuration file, adding the interface and IP address details.
450
451Example:
452
453[source,bash]
454----
455<none>
456- port_limit      : 2
457  version         : 2
458#List of interfaces. Change according to your setup. Use ./dpdk_setup_ports.py -s to see available options.
459interfaces    : ["03:00.0", "03:00.1"]  #<1>
460port_info       :  # Port IPs. Change according to your needs. In case of loopback, you can leave as is.
461          - ip         : 1.1.1.1
462            default_gw : 2.2.2.2
463          - ip         : 2.2.2.2
464            default_gw : 1.1.1.1
465----
466<1> Edit this line to match the interfaces you are using.
467All NICs must have the same type - do not mix different NIC types in one config file. For more info, see link:http://trex-tgn.cisco.com/youtrack/issue/trex-201[trex-201].
468
469
470=== Script for creating config file
471
472A script is available to automate the process of tailoring the basic configuration file to your needs. The script gets you started, and then you can then edit the resulting configuration file directly for advanced options. For details, see xref:trex_config_yaml_config_file[YAML Configuration File].
473
474There are two ways to run the script:
475
476* Interactive mode: Script pormpts you for parameters.
477* Command line mode: Provide all parameters using command line options.
478
479==== Interactive mode
480
481The script provides a list of available interfaces with interface-related information. Follow the instructions to create a basic config file.
482
483[source,bash]
484----
485sudo ./dpdk_setup_ports.py -i
486----
487
488==== Command line mode
489
490Run the following command to display a list of all interfaces and interface-related information:
491
492[source,bash]
493----
494sudo ./dpdk_setup_ports.py -t
495----
496
497* In case of *Loopback* and/or only *L1-L2 Switches* on the way, IPs and destination MACs are not required. The script assumes the following interface connections: 0&#8596;1, 2&#8596;3 etc. +
498
499// clarify "on the way" above
500
501Run the following:
502
503[source,bash]
504----
505sudo ./dpdk_setup_ports.py -c <TRex interface 0> <TRex interface 1> ...
506----
507
508* In case of a *Router* (or other next hop device, such as *L3 Switch*), specify the TRex IPs and default gateways, or MACs of the router, as described below.
509
510.Command line options for the configuration file creation script (dpdk_setup_ports.py -c)
511[options="header",cols="2,5,3",width="100%"]
512|=================
513| Argument | Description | Example
514| -c  | Create a configuration file by specified interfaces (PCI address or Linux names: eth1 etc.) | -c 03:00.1 eth1 eth4 84:00.0
515| --dump | Dump created configuration to screen. |
516| -o | Output the configuration to a file. | -o /etc/trex_cfg.yaml
517| --dest-macs | Destination MACs to be used per each interface. Use this option for MAC-based configuration instead of IP-based. Do not use this option together with --ip and --def_gw | --dest-macs 11:11:11:11:11:11 22:22:22:22:22:22
518| --ip | List of IPs to use for each interface. If --ip and --dest-macs are not specified, the script assumes loopback connections (0&#8596;1, 2&#8596;3 etc.). | --ip 1.2.3.4 5.6.7.8
519|--def-gw | List of default gateways to use for each interface. When using the --ip option, also use the --def_gw option. | --def-gw 3.4.5.6 7.8.9.10
520| --ci | Cores include: White list of cores to use. Include enough cores for each NUMA. | --ci 0 2 4 5 6
521| --ce | Cores exclude: Black list of cores to exclude. When excluding cores, ensure that enough remain for each NUMA. | --ci 10 11 12
522| --no-ht | No HyperThreading: Use only one thread of each core specified in the configuration file. |
523| --prefix | (Advanced option) Prefix to be used in TRex configuration in case of parallel instances. | --prefix first_instance
524| --zmq-pub-port | (Advanced option) ZMQ Publisher port to be used in TRex configuration in case of parallel instances. | --zmq-pub-port 4000
525| --zmq-rpc-port | (Advanced option) ZMQ RPC port to be used in the TRex configuration in case of parallel instances. | --zmq-rpc-port
526| --ignore-numa | (Advanced option) Ignore NUMAs for configuration creation. This option may reduce performance. Use only if necessary - for example, in case of a pair of interfaces at different NUMAs. |
527|=================
528
529=== TRex on ESXi
530
531General recommendation: For best performance, run TRex on "bare metal" hardware, without any type of VM. Bandwidth on a VM may be limited, and IPv6 may not be fully supported.
532
533In special cases, it may be reasonable or advantageous to run TRex on VM:
534
535* If you already have VM installed, and do not require high performance.
536* Virtual NICs can be used to bridge between TRex and NICs not supported by TRex.
537
538==== Configuring ESXi for running TRex
539
5401. Click the host machine, then select Configuration -> Networking.
541
542a. One of the NICs must be connected to the main vSwitch network for an "outside" connection for the TRex client and ssh: +
543image:images/vSwitch_main.png[title="vSwitch_main"]
544
545b. Other NICs that are used for TRex traffic must be in a separate vSwitch: +
546// In the line above, i changed "distinguish" to "a separate". please verify.
547image:images/vSwitch_loopback.png[title="vSwitch_loopback"]
548
5492. Right-click the guest machine -> Edit settings -> Ensure the NICs are set to their networks: +
550image:images/vSwitch_networks.png[title="vSwitch_networks"]
551
552[NOTE]
553=====================================================================
554Before version 2.10, the following command did not function correctly:
555[subs="quotes"]
556....
557sudo ./t-rex-64 -f cap2/dns.yaml *--lm 1 --lo* -l 1000 -d 100
558....
559The vSwitch did not route packets correctly. This issue was resolved in version 2.10 when TRex started to support ARP.
560=====================================================================
561// in the note above, verify "did not route packets correctly" and clarify.
562
563==== Configuring Pass-through
564
565Pass-through enables direct use of host machine NICs from within the VM. Pass-through access is generally limited only by the NIC/hardware itself, but there may be occasional spikes in latency (~10ms). Passthrough settings cannot be saved to OVA.
566
5671. Click the host machine. Enter Configuration -> Advanced settings -> Edit. 
568
5692. Mark the desired NICs. +
570image:images/passthrough_marking.png[title="passthrough_marking"]
571
5723. Reboot the ESXi to apply.
573
5744. Right-click the guest machine. Edit settings -> Add -> *PCI device* -> Select the NICs individually. +
575image:images/passthrough_adding.png[title="passthrough_adding"]
576
577=== Configuring for running with router (or other L3 device) as DUT
578
579You can follow link:trex_config_guide.html[this presentation] for an example of how to configure the router as a DUT.
580
581=== Running TRex, understanding output
582
583After configuration is complete, use the following command to start basic TRex run for 10 seconds
584(it will use the default config file name /etc/trex_cfg.yaml):
585[source,bash]
586----
587$sudo ./t-rex-64 -f cap2/dns.yaml -c 4 -m 1 -d 10  -l 1000
588----
589
590==== TRex output
591
592After running TRex successfully, the output will be similar to the following:
593
594[source,python]
595----
596$ sudo ./t-rex-64 -f cap2/dns.yaml -d 10 -l 1000
597Starting  TRex 2.09 please wait  ...
598zmq publisher at: tcp://*:4500 
599 number of ports found : 4
600  port : 0 
601  ------------
602  link         :  link : Link Up - speed 10000 Mbps - full-duplex      <1>
603  promiscuous  : 0 
604  port : 1 
605  ------------
606  link         :  link : Link Up - speed 10000 Mbps - full-duplex
607  promiscuous  : 0 
608  port : 2 
609  ------------
610  link         :  link : Link Up - speed 10000 Mbps - full-duplex
611  promiscuous  : 0 
612  port : 3 
613  ------------
614  link         :  link : Link Up - speed 10000 Mbps - full-duplex
615  promiscuous  : 0 
616
617
618 -Per port stats table 
619      ports |               0 |               1 |               2 |               3 
620 -------------------------------------------------------------------------------------
621   opackets |            1003 |            1003 |            1002 |            1002 
622     obytes |           66213 |           66229 |           66132 |           66132 
623   ipackets |            1003 |            1003 |            1002 |            1002 
624     ibytes |           66225 |           66209 |           66132 |           66132 
625    ierrors |               0 |               0 |               0 |               0 
626    oerrors |               0 |               0 |               0 |               0 
627      Tx Bw |     217.09 Kbps |     217.14 Kbps |     216.83 Kbps |     216.83 Kbps 
628
629 -Global stats enabled 
630 Cpu Utilization : 0.0  % <2>  29.7 Gb/core <3>
631 Platform_factor : 1.0    
632 Total-Tx        :     867.89 Kbps                                             <4>
633 Total-Rx        :     867.86 Kbps                                             <5>
634 Total-PPS       :       1.64 Kpps  
635 Total-CPS       :       0.50  cps  
636
637 Expected-PPS    :       2.00  pps   <6>
638 Expected-CPS    :       1.00  cps   <7>
639 Expected-BPS    :       1.36 Kbps   <8>
640
641 Active-flows    :        0 <9> Clients :      510   Socket-util  : 0.0000 %
642 Open-flows      :        1 <10> Servers :      254   Socket   :        1  Socket/Clients :  0.0
643 drop-rate       :       0.00  bps   <11>
644 current time    : 5.3 sec  
645 test duration   : 94.7 sec  
646
647 -Latency stats enabled 
648 Cpu Utilization : 0.2 %  <12>
649 if|   tx_ok , rx_ok  , rx   ,error,    average   ,   max         , Jitter ,  max window 
650   |         ,        , check,     , latency(usec),latency (usec) ,(usec)  ,             
651 --------------------------------------------------------------------------------------------------
652 0 |     1002,    1002,         0,   0,         51  ,      69,       0      |   0  69  67    <13>
653 1 |     1002,    1002,         0,   0,         53  ,     196,       0      |   0  196  53
654 2 |     1002,    1002,         0,   0,         54  ,      71,       0      |   0  71  69 
655 3 |     1002,    1002,         0,   0,         53  ,     193,       0      |   0  193  52 
656----
657<1> Link must be up for TRex to work.
658<2> Average CPU utilization of transmitters threads. For best results it should be lower than 80%.
659<3> Gb/sec generated per core of DP. Higher is better.
660<4> Total Tx must be the same as Rx at the end of the run.
661<5> Total Rx must be the same as Tx at the end of the run.
662<6> Expected number of packets per second (calculated without latency packets).
663<7> Expected number of connections per second (calculated without latency packets).
664<8> Expected number of bits per second (calculated without latency packets).
665<9> Number of TRex active "flows". Could be different than the number of router flows, due to aging issues. Usualy the TRex number of active flows is much lower than that of the router because the router ages flows slower.
666<10> Total number of TRex flows opened since startup (including active ones, and ones already closed).
667<11> Drop rate.
668<12> Rx and latency thread CPU utilization.
669<13> Tx_ok on port 0 should equal Rx_ok on port 1, and vice versa.
670
671// the formatting of the latency stats table in the output above is difficult to read
672
673==== Additional information about statistics in output
674
675*socket*::  Same as the active flows.  
676
677*Socket/Clients*:: Average of active flows per client, calculated as active_flows/#clients.
678
679*Socket-util*:: Estimate of number of L4 ports (sockets) used per client IP. This is approximately (100*active_flows/#clients)/64K, calculated as (average active flows per client*100/64K). Utilization of more than 50% means that TRex is generating too many flows per single client, and that more clients must be added in the generator configuration.
680// clarify above, especially the formula
681
682*Max window*:: Maximum latency within a time window of 500 msec. There are few values shown per port.
683 The earliest value is on the left, and latest value (last 500msec) on the right. This can help in identifying spikes of high latency clearing after some time. Maximum latency is the total maximum over the entire test duration. To best understand this, run TRex with the latency option (-l) and watch the results with this section in mind.
684
685// clarify the values. in the table, it looks like there are 3 values: left, middle, right
686
687*Platform_factor*:: In some cases, users duplicate traffic using a splitter/switch. In this scenario, it is useful for all numbers displayed by TRex to be multiplied by this factor, so that TRex counters will match the DUT counters.
688
689// in the above "multiplied by this factor" - which factor? 2?
690
691WARNING: If you do not see Rx packets, review the MAC address configuration.
692
693include::trex_book_basic.asciidoc[]
694
695// not sure what the include thing above is for
696
697== Advanced features
698
699=== VLAN (dot1q) support
700
701To add a VLAN tag to all traffic generated by TRex, add a ``vlan'' keyword in each port section in the platform config file, as described in the xref:trex_config_yaml_config_file[YAML Configuration File] section. +
702
703You can specify a different VLAN tag for each port, or use VLAN only on some ports. +
704
705One useful application of this can be in a lab setup where you have one TRex and many DUTs, and you want to test a different DUT on each run, without changing cable connections. You can put each DUT on a VLAN of its own, and use different TRex platform configuration files with different VLANs on each run.
706
707
708=== Utilizing maximum port bandwidth in case of asymmetric traffic profile 
709
710anchor:trex_load_bal[]
711
712[NOTE]
713If you want simple VLAN support, this is probably *not* the feature to use. This feature is used for load balancing. To configure VLAN support, see the ``vlan'' field in the xref:trex_config_yaml_config_file[YAML Configuration File] section.
714
715
716The VLAN Trunk TRex feature attempts to solve the router port bandwidth limitation when the traffic profile is asymmetric (example: Asymmetric SFR profile).
717
718This feature converts asymmetric traffic to symmetric, from the port perspective, using router sub-interfaces. This requires TRex to send the traffic on two VLANs, as described below.
719
720// the paragraph above mentions the "VLAN Trunk TRex feature" but I don't see any description or use of the term, "VLAN Trunk".
721
722.YAML format - This goes in the traffic YAML file.
723[source,python]
724----
725  vlan       : { enable : 1  ,  vlan0 : 100 , vlan1 : 200 }
726----
727
728
729.Example
730[source,python]
731----
732- duration : 0.1
733  vlan       : { enable : 1  ,  vlan0 : 100 , vlan1 : 200 }   <1>
734----
735<1> Enable load balance feature: vlan0==100 , vlan1==200 +
736For a full file example, see the TRex source in: scripts/cap2/ipv4_load_balance.yaml
737
738// THIS LINE IS HERE ONLY TO HELP RENDITION OF THE LINE BELOW. due to a rendition bug, the next line otherwise does not appear, probably because it follows the footnote text above.
739
740*Problem definition:*::
741
742Scenario: TRex with two ports and an SFR traffic profile.
743
744.Without VLAN/sub interfaces, all client emulated traffic is sent on port 0, and all server emulated traffic (example: HTTP response) on port 1.
745[source,python]
746----
747TRex port 0 ( client) <-> [  DUT ] <-> TRex port 1 ( server)
748----
749Without VLAN support, the traffic is asymmetric. 10% of the traffic is sent from port 0 (client side), 90% from port 1 (server). Port 1 is the bottlneck (10Gb/s limit).
750
751.With VLAN/sub interfaces 
752[source,python]
753----
754TRex port 0 ( client VLAN0) <->  | DUT  | <-> TRex port 1 ( server-VLAN0)
755TRex port 0 ( server VLAN1) <->  | DUT  | <-> TRex port 1 ( client-VLAN1)
756----
757
758In this case, traffic on vlan0 is sent as before, while for traffic on vlan1, the order is reversed (client traffic sent on port1 and server traffic on port0).
759TRex divides the flows evenly between the vlans. This results in an equal amount of traffic on each port.
760
761*Router configuation:*::
762[source,python]
763----
764        !
765        interface TenGigabitEthernet1/0/0      <1>
766         mac-address 0000.0001.0000   
767         mtu 4000
768         no ip address
769         load-interval 30
770        !
771        i
772        interface TenGigabitEthernet1/0/0.100
773         encapsulation dot1Q 100               <2> 
774         ip address 11.77.11.1 255.255.255.0
775         ip nbar protocol-discovery
776         ip policy route-map vlan_100_p1_to_p2 <3> 
777        !
778        interface TenGigabitEthernet1/0/0.200
779         encapsulation dot1Q 200               <4>
780         ip address 11.88.11.1 255.255.255.0
781         ip nbar protocol-discovery
782         ip policy route-map vlan_200_p1_to_p2 <5> 
783        !
784        interface TenGigabitEthernet1/1/0
785         mac-address 0000.0001.0000
786         mtu 4000
787         no ip address
788         load-interval 30
789        !
790        interface TenGigabitEthernet1/1/0.100
791         encapsulation dot1Q 100
792         ip address 22.77.11.1 255.255.255.0
793         ip nbar protocol-discovery
794         ip policy route-map vlan_100_p2_to_p1
795        !
796        interface TenGigabitEthernet1/1/0.200
797         encapsulation dot1Q 200
798         ip address 22.88.11.1 255.255.255.0
799         ip nbar protocol-discovery
800         ip policy route-map vlan_200_p2_to_p1
801        !
802        
803        arp 11.77.11.12 0000.0001.0000 ARPA      <6>
804        arp 22.77.11.12 0000.0001.0000 ARPA
805        
806        route-map vlan_100_p1_to_p2 permit 10    <7>
807         set ip next-hop 22.77.11.12
808        !
809        route-map vlan_100_p2_to_p1 permit 10
810         set ip next-hop 11.77.11.12
811        !
812        
813        route-map vlan_200_p1_to_p2 permit 10
814         set ip next-hop 22.88.11.12
815        !
816        route-map vlan_200_p2_to_p1 permit 10
817         set ip next-hop 11.88.11.12
818        !
819----
820<1> Main interface must not have IP address.
821<2> Enable VLAN1
822<3> PBR configuration
823<4> Enable VLAN2  
824<5> PBR configuration
825<6> TRex destination port MAC address
826<7> PBR configuration rules
827
828=== Static source MAC address setting  
829
830With this feature, TRex replaces the source MAC address with the client IP address.
831
832 Note: This feature was requested by the Cisco ISG group. 
833
834
835*YAML:*::
836[source,python]
837----
838 mac_override_by_ip : true
839----
840
841.Example
842[source,python]
843----
844- duration : 0.1
845 ..
846  mac_override_by_ip : true <1>
847----
848<1> In this case, the client side MAC address looks like this:
849SRC_MAC = IPV4(IP) + 00:00  
850
851=== IPv6 support 
852
853Support for IPv6 includes:
854
8551. Support for pcap files containing IPv6 packets.
8562. Ability to generate IPv6 traffic from pcap files containing IPv4 packets. +
857The `--ipv6` command line option enables this feature.
858The keywords `src_ipv6` and `dst_ipv6` specify the most significant 96 bits of the IPv6 address. Example:
859
860[source,python]
861----
862      src_ipv6 : [0xFE80,0x0232,0x1002,0x0051,0x0000,0x0000]
863      dst_ipv6 : [0x2001,0x0DB8,0x0003,0x0004,0x0000,0x0000]
864----
865
866The IPv6 address is formed by placing what would typically be the IPv4
867address into the least significant 32 bits and copying the value provided
868in the src_ipv6/dst_ipv6 keywords into the most signficant 96 bits.
869If src_ipv6 and dst_ipv6 are not specified, the default
870is to form IPv4-compatible addresses (most signifcant 96 bits are zero).
871
872There is IPv6 support for all plugins.
873  
874*Example:*::
875[source,bash]
876----
877$sudo ./t-rex-64 -f cap2l/sfr_delay_10_1g.yaml -c 4 -p -l 100 -d 100000 -m 30  --ipv6
878----
879
880*Limitations:*::
881
882* TRex cannot generate both IPv4 and IPv6 traffic.
883* The `--ipv6` switch must be specified even when using a pcap file containing only IPv6 packets.
884
885
886*Router configuration:*::
887
888[source,python]
889----
890interface TenGigabitEthernet1/0/0
891 mac-address 0000.0001.0000
892 mtu 4000
893 ip address 11.11.11.11 255.255.255.0
894 ip policy route-map p1_to_p2
895 load-interval 30
896 ipv6 enable   ==> IPv6
897 ipv6 address 2001:DB8:1111:2222::1/64                  <1>
898 ipv6 policy route-map ipv6_p1_to_p2                    <2>
899!
900
901
902ipv6 unicast-routing                                    <3>
903
904ipv6 neighbor 3001::2 TenGigabitEthernet0/1/0 0000.0002.0002   <4>
905ipv6 neighbor 2001::2 TenGigabitEthernet0/0/0 0000.0003.0002
906
907route-map ipv6_p1_to_p2 permit 10                              <5>
908 set ipv6 next-hop 2001::2
909!
910route-map ipv6_p2_to_p1 permit 10
911 set ipv6 next-hop 3001::2
912!
913
914
915asr1k(config)#ipv6 route 4000::/64 2001::2                 
916asr1k(config)#ipv6 route 5000::/64 3001::2 
917----
918<1> Enable IPv6 
919<2> Add pbr 
920<3> Enable IPv6 routing 
921<4> MAC address setting. Should be TRex MAC.
922<5> PBR configuraion
923
924
925=== Client clustering configuration
926
927TRex supports testing complex topologies with more than one DUT, using a feature called "client clustering". This feature allows specifying the distribution of clients that TRex emulates.
928
929Consider the following topology:
930
931.Topology example 
932image:images/topology.png[title="Client Clustering",width=850]
933
934There are two clusters of DUTs. Using the configuration file, you can partition TRex emulated clients into groups, and define how they will be spread between the DUT clusters.
935
936Group configuration includes:
937
938* IP start range.
939* IP end range.
940* Initiator side configuration: Parameters affecting packets sent from client side.
941* Responder side configuration: Parameters affecting packets sent from server side.
942
943[NOTE]
944It is important to understand that this is *complimentary* to the client generator
945configured per profile. It only defines how the clients will be spread between clusters.
946
947In the following example, a profile defines a client generator.
948
949[source,bash]
950----
951$cat cap2/dns.yaml 
952- duration : 10.0
953  generator :  
954          distribution : "seq"           
955          clients_start : "16.0.0.1"
956          clients_end   : "16.0.0.255"   
957          servers_start : "48.0.0.1"
958          servers_end   : "48.0.0.255"   
959          dual_port_mask : "1.0.0.0" 
960  cap_info : 
961     - name: cap2/dns.pcap
962       cps : 1.0          
963       ipg : 10000        
964       rtt : 10000        
965       w   : 1            
966----
967
968Goal:
969
970* Create two clusters with 4 and 3 devices, respectively. 
971* Send *80%* of the traffic to the upper cluster and *20%* to the lower cluster. Specify the DUT to which the packet will be sent by MAC address or IP. (The following example uses the MAC address. The instructions after the example indicate how to change to IP-based.)
972
973Create the following cluster configuration file:
974
975[source,bash]
976----
977#
978# Client configuration example file
979# The file must contain the following fields
980#
981# 'vlan'   - if the entire configuration uses VLAN,
982#            each client group must include vlan
983#            configuration
984#
985# 'groups' - each client group must contain range of IPs
986#            and initiator and responder section
987#            'count' represents the number of different DUTs
988#            in the group.
989#
990
991# 'true' means each group must contain VLAN configuration. 'false' means no VLAN config allowed.
992vlan: true
993
994groups:
995
996-    ip_start  : 16.0.0.1
997     ip_end    : 16.0.0.204
998     initiator :
999                 vlan    : 100
1000                 dst_mac : "00:00:00:01:00:00"
1001     responder :
1002                 vlan    : 200
1003                 dst_mac : "00:00:00:02:00:00"
1004
1005     count     : 4
1006
1007-    ip_start  : 16.0.0.205
1008     ip_end    : 16.0.0.255
1009     initiator :
1010                 vlan    : 101
1011                 dst_mac : "00:00:01:00:00:00"
1012
1013     responder:
1014                 vlan    : 201
1015                 dst_mac : "00:00:02:00:00:00"
1016
1017     count     : 3
1018
1019----
1020
1021The above configuration divides the generator range of 255 clients to two clusters. The range
1022of IPs in all groups in the client configuration file must cover the entire range of client IPs
1023from the traffic profile file.
1024
1025MAC addresses will be allocated incrementally, with a wrap around after ``count'' addresses. 
1026
1027Example:
1028
1029*Initiator side (packets with source in 16.x.x.x net):*
1030
1031* 16.0.0.1 -> 48.x.x.x - dst_mac: 00:00:00:01:00:00  vlan: 100 
1032* 16.0.0.2 -> 48.x.x.x - dst_mac: 00:00:00:01:00:01  vlan: 100 
1033* 16.0.0.3 -> 48.x.x.x - dst_mac: 00:00:00:01:00:02  vlan: 100 
1034* 16.0.0.4 -> 48.x.x.x - dst_mac: 00:00:00:01:00:03  vlan: 100 
1035* 16.0.0.5 -> 48.x.x.x - dst_mac: 00:00:00:01:00:00  vlan: 100 
1036* 16.0.0.6 -> 48.x.x.x - dst_mac: 00:00:00:01:00:01  vlan: 100 
1037
1038*Responder side (packets with source in 48.x.x.x net):* 
1039
1040* 48.x.x.x -> 16.0.0.1  - dst_mac(from responder) : "00:00:00:02:00:00" , vlan:200
1041* 48.x.x.x -> 16.0.0.2  - dst_mac(from responder) : "00:00:00:02:00:01" , vlan:200
1042
1043and so on. +
1044
1045The MAC addresses of DUTs must be changed to be sequential. Another option is to replace:
1046`dst_mac : <ip-address>` +
1047with: +
1048`next_hop : <ip-address>`
1049
1050// clarify "another option" above.
1051
1052For example, the first group in the configuration file would be:
1053
1054[source,bash]
1055----
1056-    ip_start  : 16.0.0.1
1057     ip_end    : 16.0.0.204
1058     initiator :
1059                 vlan     : 100
1060                 next_hop : 1.1.1.1
1061                 src_ip   : 1.1.1.100
1062     responder :
1063                 vlan     : 200
1064                 next_hop : 2.2.2.1
1065                 src_ip   : 2.2.2.100
1066
1067     count     : 4
1068----
1069
1070In this case, TRex attempts to resolve the following addresses using ARP:
1071
10721.1.1.1, 1.1.1.2, 1.1.1.3, 1.1.1.4 (and the range 2.2.2.1-2.2.2.4)
1073
1074If not all IPs are resolved, TRex exits with an error message. 
1075
1076`src_ip` is used to send link:https://en.wikipedia.org/wiki/Address_Resolution_Protocol[gratuitous ARP], and for filling relevant fields in ARP request. If no `src_ip` is given, TRex looks for the source IP in the relevant port section in the platform configuration file (/etc/trex_cfg.yaml). If none is found, TRex exits with an error message. 
1077
1078If a client config file is given, TRex ignores the `dest_mac` and `default_gw` parameters from the platform configuration file.
1079
1080Now, streams will look like: +
1081
1082// clarify the line above
1083
1084*Initiator side (packets with source in 16.x.x.x net):*
1085
1086* 16.0.0.1 -> 48.x.x.x - dst_mac: MAC of 1.1.1.1  vlan: 100
1087* 16.0.0.2 -> 48.x.x.x - dst_mac: MAC of 1.1.1.2  vlan: 100
1088* 16.0.0.3 -> 48.x.x.x - dst_mac: MAC of 1.1.1.3  vlan: 100
1089* 16.0.0.4 -> 48.x.x.x - dst_mac: MAC of 1.1.1.4  vlan: 100
1090* 16.0.0.5 -> 48.x.x.x - dst_mac: MAC of 1.1.1.1  vlan: 100
1091* 16.0.0.6 -> 48.x.x.x - dst_mac: MAC of 1.1.1.2  vlan: 100
1092
1093*Responder side (packets with source in 48.x.x.x net):*
1094
1095* 48.x.x.x -> 16.0.0.1  - dst_mac: MAC of 2.2.2.1 , vlan:200
1096* 48.x.x.x -> 16.0.0.2  - dst_mac: MAC of 2.2.2.2 , vlan:200
1097
1098
1099[NOTE]
1100=====================================================================
1101It is important to understand that the IP to MAC coupling (with either MAC-based or IP-based configuration) is done at the beginning and never changes. For example, in a MAC-based configuration:
1102
1103* Packets with source IP 16.0.0.2 will always have VLAN 100 and dst MAC 00:00:00:01:00:01. 
1104* Packets with destination IP 16.0.0.2 will always have VLAN 200 and dst MAC 00:00:00:02:00:01.
1105
1106Consequently, you can predict exactly which packet (and how many packets) will go to each DUT.
1107=====================================================================
1108
1109// the logic of the note above is not completely clear to me
1110
1111*Usage:*
1112
1113[source,bash]
1114----
1115sudo ./t-rex-64 -f cap2/dns.yaml --client_cfg my_cfg.yaml
1116----
1117
1118=== NAT support 
1119
1120TRex can learn dynamic NAT/PAT translation. To enable this feature, use the +
1121`--learn-mode <mode>` +
1122switch at the command line. To learn the NAT translation, TRex must embed information describing which flow a packet belongs to, in the first packet of each flow. TRex can do this using one of several methods, depending on the chosen <mode>.
1123
1124// is this logic correct in the paragraph above: "TRex must embed information describing which flow a packet belongs to, in the first packet of each flow"
1125
1126*Mode 1:*::
1127
1128`--learn-mode 1` +
1129*TCP flow*: Flow information is embedded in the ACK of the first TCP SYN. +
1130*UDP flow*: Flow information is embedded in the IP identification field of the first packet in the flow. +
1131This mode was developed for testing NAT with firewalls (which usually do not work with mode 2). In this mode, TRex also learns and compensates for TCP sequence number randomization that might be done by the DUT. TRex can learn and compensate for seq num randomization in both directions of the connection.
1132
1133*Mode 2:*::
1134
1135`--learn-mode 2` +
1136Flow information is added in a special IPv4 option header (8 bytes long 0x10 id). This option header is added only to the first packet in the flow. This mode does not work with DUTs that drop packets with IP options (for example, Cisco ASA firewall).
1137
1138*Mode 3:*::
1139
1140`--learn-mode 3` +
1141Similar to mode 1, but TRex does not learn the seq num randomization in the server->client direction.
1142This mode can provide better connections-per-second performance than mode 1. But for all existing firewalls, the mode 1 cps rate is adequate.
1143
1144==== Examples
1145
1146*simple HTTP traffic*
1147
1148[source,bash]
1149----
1150$sudo ./t-rex-64 -f cap2/http_simple.yaml -c 4  -l 1000 -d 100000 -m 30  --learn-mode 1
1151----
1152
1153*SFR traffic without bundling/ALG support*
1154
1155[source,bash]
1156----
1157$sudo ./t-rex-64 -f avl/sfr_delay_10_1g_no_bundling.yaml -c 4  -l 1000 -d 100000 -m 10  --learn-mode 2
1158----
1159
1160*NAT terminal counters:*::
1161
1162[source,python]
1163----
1164-Global stats enabled 
1165 Cpu Utilization : 0.6  %  33.4 Gb/core 
1166 Platform_factor : 1.0
1167 Total-Tx        :       3.77 Gbps   NAT time out    :      917 <1> (0 in wait for syn+ack) <5>
1168 Total-Rx        :       3.77 Gbps   NAT aged flow id:        0 <2>
1169 Total-PPS       :     505.72 Kpps   Total NAT active:      163 <3> (12 waiting for syn) <6>
1170 Total-CPS       :      13.43 Kcps   Total NAT opened:    82677 <4>
1171----
1172<1> Number of connections for which TRex had to send the next packet in the flow, but did not learn the NAT translation yet. Should be 0. Usually, a value different than 0 is seen if the DUT drops the flow (probably because it cannot handle the number of connections).
1173<2> Number of flows for which the flow had already aged out by the time TRex received the translation info. A value other than 0 is rare. Can occur only when there is very high latency in the DUT input/output queue.
1174<3> Number of flows for which TRex sent the first packet before learning the NAT translation. The value depends on the connection per second rate and round trip time.
1175<4> Total number of translations over the lifetime of the TRex instance. May be different from the total number of flows if template is uni-directional (and consequently does not need translation).
1176<5> Out of the timed-out flows, the number that were timed out while waiting to learn the TCP seq num randomization of the server->client from the SYN+ACK packet. Seen only in --learn-mode 1.
1177<6> Out of the active NAT sessions, the number that are waiting to learn the client->server translation from the SYN packet. (Others are waiting for SYN+ACK from server.) Seen only in --learn-mode 1.
1178
1179
1180// THIS COMMENT LINE IS HERE TO HELP CORRECT RENDITION OF THE LINE BELOW. Without this, rendition omits the following several lines.
1181
1182*Configuration for Cisco ASR1000 Series:*::
1183
1184This feature was tested with the following configuration and the +
1185sfr_delay_10_1g_no_bundling.yaml +
1186traffic profile. The client address range is 16.0.0.1 to 16.0.0.255 
1187
1188[source,python]
1189----
1190interface TenGigabitEthernet1/0/0            <1>
1191 mac-address 0000.0001.0000
1192 mtu 4000
1193 ip address 11.11.11.11 255.255.255.0
1194 ip policy route-map p1_to_p2
1195 ip nat inside                               <2>
1196 load-interval 30
1197!
1198
1199interface TenGigabitEthernet1/1/0
1200 mac-address 0000.0001.0000
1201 mtu 4000
1202 ip address 11.11.11.11 255.255.255.0
1203 ip policy route-map p1_to_p2
1204 ip nat outside                              <3>
1205 load-interval 30
1206
1207ip  nat pool my 200.0.0.0 200.0.0.255 netmask 255.255.255.0  <4>
1208
1209ip nat inside source list 7 pool my overload 
1210access-list 7 permit 16.0.0.0 0.0.0.255                      <5>
1211
1212ip nat inside source list 8 pool my overload                 <6>
1213access-list 8 permit 17.0.0.0 0.0.0.255                      
1214----
1215<1> Must be connected to TRex Client port (router inside port)
1216<2> NAT inside 
1217<3> NAT outside
1218<4> Pool of outside address with overload
1219<5> Match TRex YAML client range
1220<6> In case of dual port TRex
1221
1222// verify 1 and 5 above; rephrased. should #4 say "addresses" ?
1223
1224// THIS COMMENT LINE IS HERE TO HELP CORRECT RENDITION OF THE LINE BELOW. Without this, rendition omits the following several lines.
1225
1226
1227*Limitations:*::
1228
1229. The IPv6-IPv6 NAT feature does not exist on routers, so this feature can work only with IPv4.
1230. Does not support NAT64. 
1231. Bundling/plugin is not fully supported. Consequently, sfr_delay_10.yaml does not work. Use sfr_delay_10_no_bundling.yaml instead.
1232
1233[NOTE]
1234=====================================================================
1235* `--learn-verify` is a TRex debug mechanism for testing the TRex learn mechanism.
1236* Need to run it when DUT is configured without NAT. It will verify that the inside_ip==outside_ip and inside_port==outside_port.
1237=====================================================================
1238
1239=== Flow order/latency verification 
1240
1241In normal mode (without the feature enabled), received traffic is not checked by software. Hardware (Intel NIC) testing for dropped packets occurs at the end of the test. The only exception is the Latency/Jitter packets. This is one reason that with TRex, you *cannot* check features that terminate traffic (for example TCP Proxy).
1242
1243To enable this feature, add 
1244`--rx-check <sample>` 
1245to the command line options, where <sample> is the sample rate. The number of flows that will be sent to the software for verification is (1/(sample_rate). For 40Gb/sec traffic you can use a sample rate of 1/128. Watch for Rx CPU% utilization.
1246
1247[NOTE]
1248============
1249This feature changes the TTL of the sampled flows to 255 and expects to receive packets with TTL 254 or 255 (one routing hop). If you have more than one hop in your setup, use `--hops` to change it to a higher value. More than one hop is possible if there are number of routers betwean TRex client side and TRex server side.
1250============
1251
1252This feature ensures that:
1253
1254* Packets get out of DUT in order (from each flow perspective).
1255* There are no packet drops (no need to wait for the end of the test). Without this flag, you must wait for the end of the test in order to identify packet drops, because there is always a difference between TX and Rx, due to RTT.
1256
1257
1258.Full example 
1259[source,bash]
1260----
1261$sudo ./t-rex-64 -f avl/sfr_delay_10_1g.yaml -c 4 -p -l 100 -d 100000 -m 30  --rx-check 128
1262----
1263
1264[source,python]
1265----
1266Cpu Utilization : 0.1 %                                                                       <1>
1267 if|   tx_ok , rx_ok  , rx   ,error,    average   ,   max         , Jitter ,  max window 
1268   |         ,        , check,     , latency(usec),latency (usec) ,(usec)  ,             
1269 --------------------------------------------------------------------------------
1270 0 |     1002,    1002,      2501,   0,         61  ,      70,       3      |  60
1271 1 |     1002,    1002,      2012,   0,         56  ,      63,       2      |  50
1272 2 |     1002,    1002,      2322,   0,         66  ,      74,       5      |  68
1273 3 |     1002,    1002,      1727,   0,         58  ,      68,       2      |  52
1274
1275 Rx Check stats enabled                                                                       <2>
1276 -------------------------------------------------------------------------------------------
1277 rx check:  avg/max/jitter latency,       94  ,     744,       49      |  252  287  309       <3>
1278 
1279 active flows: <6>      10, fif: <5>     308,  drop:        0, errors:        0                <4>
1280 -------------------------------------------------------------------------------------------
1281----
1282<1> CPU% of the Rx thread. If it is too high, *increase* the sample rate.
1283<2> Rx Check section. For more detailed info, press 'r' during the test or at the end of the test.
1284<3> Average latency, max latency, jitter on the template flows in microseconds. This is usually *higher* than the latency check packet because the feature works more on this packet.
1285<4> Drop counters and errors counter should be zero. If not, press 'r' to see the full report or view the report at the end of the test.
1286<5> fif - First in flow. Number of new flows handled by the Rx thread.
1287<6> active flows - number of active flows handled by rx thread 
1288
1289.Press R to Display Full Report
1290[source,python]
1291----
1292 m_total_rx                              : 2 
1293 m_lookup                                : 2 
1294 m_found                                 : 1 
1295 m_fif                                   : 1 
1296 m_add                                   : 1 
1297 m_remove                                : 1 
1298 m_active                                : 0 
1299                                                        <1>
1300 0  0  0  0  1041  0  0  0  0  0  0  0  0  min_delta  : 10 usec 
1301 cnt        : 2 
1302 high_cnt   : 2 
1303 max_d_time : 1041 usec
1304 sliding_average    : 1 usec                            <2>
1305 precent    : 100.0 %
1306 histogram 
1307 -----------
1308 h[1000]  :  2 
1309 tempate_id_ 0 , errors:       0,  jitter: 61           <3>
1310 tempate_id_ 1 , errors:       0,  jitter: 0 
1311 tempate_id_ 2 , errors:       0,  jitter: 0 
1312 tempate_id_ 3 , errors:       0,  jitter: 0 
1313 tempate_id_ 4 , errors:       0,  jitter: 0 
1314 tempate_id_ 5 , errors:       0,  jitter: 0 
1315 tempate_id_ 6 , errors:       0,  jitter: 0 
1316 tempate_id_ 7 , errors:       0,  jitter: 0 
1317 tempate_id_ 8 , errors:       0,  jitter: 0 
1318 tempate_id_ 9 , errors:       0,  jitter: 0 
1319 tempate_id_10 , errors:       0,  jitter: 0 
1320 tempate_id_11 , errors:       0,  jitter: 0 
1321 tempate_id_12 , errors:       0,  jitter: 0 
1322 tempate_id_13 , errors:       0,  jitter: 0 
1323 tempate_id_14 , errors:       0,  jitter: 0 
1324 tempate_id_15 , errors:       0,  jitter: 0 
1325 ager :
1326 m_st_alloc                                 : 1 
1327 m_st_free                                  : 0 
1328 m_st_start                                 : 2 
1329 m_st_stop                                  : 1 
1330 m_st_handle                                : 0 
1331----
1332<1> Errors, if any, shown here
1333<2> Low pass filter on the active average of latency events 
1334<3> Error per template info
1335
1336// IGNORE: this line added to help rendition. Without this line, the "Notes and Limitations" section below does not appear.
1337
1338*Notes and Limitations:*::
1339
1340** To receive the packets TRex does the following:
1341*** Changes the TTL to 0xff and expects 0xFF (loopback) or oxFE (route). (Use `--hop` to configure this value.)
1342*** Adds 24 bytes of metadata as ipv4/ipv6 option header.
1343// clarify "ipv4/ipv6 option header" above
1344
1345== Reference
1346
1347=== Traffic YAML (parameter of -f option)
1348
1349==== Global Traffic YAML section 
1350
1351[source,python]
1352----
1353- duration : 10.0                          <1>
1354  generator :                              <2>
1355          distribution : "seq"           
1356          clients_start : "16.0.0.1"     
1357          clients_end   : "16.0.0.255"   
1358          servers_start : "48.0.0.1"     
1359          servers_end   : "48.0.0.255"   
1360          clients_per_gb : 201
1361          min_clients    : 101
1362          dual_port_mask : "1.0.0.0" 
1363          tcp_aging      : 1
1364          udp_aging      : 1
1365  cap_ipg    : true                            <3>
1366  cap_ipg_min    : 30                          <4>
1367  cap_override_ipg    : 200                    <5>
1368  vlan       : { enable : 1  ,  vlan0 : 100 , vlan1 : 200 } <6>
1369  mac_override_by_ip : true  <7>
1370----
1371<1> Test duration (seconds). Can be overridden using the `-d` option.
1372<2> See full explanation on generator section link:trex_manual.html#_clients_servers_ip_allocation_scheme[here].
1373<3> true (default) indicates that the IPG is taken from the cap file (also taking into account cap_ipg_min and cap_override_ipg if they exist). false indicates that IPG is taken from per template section.
1374<4> The following two options can set the min ipg in microseconds: (if (pkt_ipg<cap_ipg_min) { pkt_ipg=cap_override_ipg} )
1375<5> Value to override (microseconds), as described in note above.
1376<6> Enable load balance feature. See xref:trex_load_bal[trex load balance section] for info.
1377<7> Enable MAC address replacement by client IP.
1378
1379==== Timer Wheel section configuration  
1380
1381(from v2.13)
1382see xref:timer_w[Timer Wheel section] 
1383
1384==== Per template section 
1385// clarify "per template"  
1386
1387[source,python]
1388----
1389     - name: cap2/dns.pcap <1>
1390       cps : 10.0          <2>
1391       ipg : 10000         <3>
1392       rtt : 10000         <4>
1393       w   : 1             <5>
1394       server_addr : "48.0.0.7"    <6>
1395       one_app_server : true       <7>
1396       
1397----
1398<1> The name of the template pcap file. Can be relative path from the t-rex-64 image directory, or an absolute path. The pcap file should include only one flow. (Exception: in case of plug-ins).
1399<2> Connection per second. This is the value that will be used if specifying -m 1 from command line (giving -m x will multiply this 
1400<3> If the global section of the YAML file includes `cap_ipg    : false`, this line sets the inter-packet gap in microseconds. 
1401<4> Should be set to the same value as ipg (microseconds). 
1402<5> Default value: w=1. This indicates to the IP generator how to generate the flows. If w=2, two flows from the same template will be generated in a burst (more for HTTP that has burst of flows).
1403<6> If `one_app_server` is set to true, then all templates will use the same server.
1404<7> If the same server address is required, set this value to true.
1405
1406
1407
1408=== YAML Configuration File (parameter of --cfg option)
1409
1410anchor:trex_config[]
1411anchor:trex_config_yaml_config_file[]
1412
1413The configuration file, in YAML format, configures TRex behavior, including:
1414 
1415- IP address or MAC address for each port (source and destination).
1416- Masked interfaces, to ensure that TRex does not try to use the management ports as traffic ports.
1417- Changing the zmq/telnet TCP port.
1418
1419You specify which config file to use by adding --cfg <file name> to the command line arguments. +
1420If no --cfg given, the default `/etc/trex_cfg.yaml` is used. +
1421Configuration file examples can be found in the `$TREX_ROOT/scripts/cfg` folder.
1422
1423==== Basic Configuration
1424
1425[source,python]
1426----
1427     - port_limit    : 2    #mandatory <1>
1428       version       : 2    #mandatory <2>
1429       interfaces    : ["03:00.0", "03:00.1"]   #mandatory <3>
1430       #enable_zmq_pub  : true #optional <4>
1431       #zmq_pub_port    : 4500 #optional <5>
1432       #prefix          : setup1 #optional <6>
1433       #limit_memory    : 1024 #optional <7>
1434       c               : 4 #optional <8>
1435       port_bandwidth_gb : 10 #optional <9>
1436       port_info       :  # set eh mac addr  mandatory
1437            - default_gw : 1.1.1.1   # port 0 <10>
1438              dest_mac   : '00:00:00:01:00:00' # Either default_gw or dest_mac is mandatory <10>
1439              src_mac    : '00:00:00:02:00:00' # optional <11>
1440              ip         : 2.2.2.2 # optional <12>
1441              vlan       : 15 # optional <13>
1442            - dest_mac   : '00:00:00:03:00:00'  # port 1
1443              src_mac    : '00:00:00:04:00:00'
1444            - dest_mac   : '00:00:00:05:00:00'  # port 2
1445              src_mac    : '00:00:00:06:00:00'
1446            - dest_mac   :   [0x0,0x0,0x0,0x7,0x0,0x01]  # port 3 <14>
1447              src_mac    :   [0x0,0x0,0x0,0x8,0x0,0x02] # <14>
1448----
1449<1>  Number of ports. Should be equal to the number of interfaces listed in 3. - mandatory
1450<2>  Must be set to 2. - mandatory
1451<3>  List of interfaces to use. Run `sudo ./dpdk_setup_ports.py --show` to see the list you can choose from. - mandatory
1452<4>  Enable the ZMQ publisher for stats data, default is true. 
1453<5>  ZMQ port number. Default value is good. If running two TRex instances on the same machine, each should be given distinct number. Otherwise, can remove this line.
1454<6>  If running two TRex instances on the same machine, each should be given distinct name. Otherwise, can remove this line. ( Passed to DPDK as --file-prefix arg)
1455<7>  Limit the amount of packet memory used. (Passed to dpdk as -m arg)
1456<8> Number of threads (cores) TRex will use per interface pair ( Can be overridden by -c command line option )
1457<9> The bandwidth of each interface in Gbs. In this example we have 10Gbs interfaces. For VM, put 1. Used to tune the amount of memory allocated by TRex.
1458<10> TRex need to know the destination MAC address to use on each port. You can specify this in one of two ways: +
1459Specify dest_mac directly. +
1460Specify default_gw (since version 2.10). In this case (only if no dest_mac given), TRex will issue ARP request to this IP, and will use
1461the result as dest MAC. If no dest_mac given, and no ARP response received, TRex will exit.
1462
1463<11> Source MAC to use when sending packets from this interface. If not given (since version 2.10), MAC address of the port will be used.
1464<12> If given (since version 2.10), TRex will issue gratitues ARP for the ip + src MAC pair on appropriate port. In stateful mode,
1465gratitues ARP for each ip will be sent every 120 seconds (Can be changed using --arp-refresh-period argument).
1466<13> If given (since version 2.18), all traffic on the port will be sent with this VLAN tag.
1467<14> Old MAC address format. New format is supported since version v2.09.
1468
1469[NOTE]
1470=========================================================================================
1471If you use version earlier than 2.10, or choose to omit the ``ip''
1472and have mac based configuration, be aware that TRex will not send any
1473gratitues ARP and will not answer ARP requests. In this case, you must configure static
1474ARP entries pointing to TRex port on your DUT. For an example config, you can look
1475xref:trex_config[here].
1476=========================================================================================
1477
1478To find out which interfaces (NIC ports) can be used, perform the following:
1479
1480[source,bash]
1481----
1482 $>sudo ./dpdk_setup_ports.py --show
1483
1484 Network devices using DPDK-compatible driver
1485 ============================================
1486
1487 Network devices using kernel driver
1488 ===================================
1489 0000:02:00.0 '82545EM Gigabit Ethernet Controller' if=eth2 drv=e1000 unused=igb_uio *Active* #<1>
1490 0000:03:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection' drv= unused=ixgb #<2>
1491 0000:03:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection' drv= unused=ixgb
1492 0000:13:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection' drv= unused=ixgb
1493 0000:13:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection' drv= unused=ixgb
1494
1495 Other network devices
1496 =====================
1497 <none>
1498----
1499<1> We see that 02:00.0 is active (our management port).
1500<2> All other NIC ports (03:00.0, 03:00.1, 13:00.0, 13:00.1) can be used.
1501
1502minimum configuration file is:
1503
1504[source,bash]
1505----
1506<none>
1507- port_limit    : 4         
1508  version       : 2
1509  interfaces    : ["03:00.0","03:00.1","13:00.1","13:00.0"]
1510----
1511
1512==== Memory section configuration  
1513
1514The memory section is optional. It is used when there is a need to tune the amount of memory used by TRex packet manager.
1515Default values (from the TRex source code), are usually good for most users. Unless you have some unusual needs, you can
1516eliminate this section.
1517
1518[source,python]
1519----
1520        - port_limit      : 2                                                                           
1521          version       : 2                                                                             
1522          interfaces    : ["03:00.0","03:00.1"]                                                         
1523          memory    :                                           <1>
1524             mbuf_64     : 16380                                <2>
1525             mbuf_128    : 8190
1526             mbuf_256    : 8190
1527             mbuf_512    : 8190
1528             mbuf_1024   : 8190
1529             mbuf_2048   : 4096
1530             traffic_mbuf_64     : 16380                        <3>
1531             traffic_mbuf_128    : 8190
1532             traffic_mbuf_256    : 8190
1533             traffic_mbuf_512    : 8190
1534             traffic_mbuf_1024   : 8190
1535             traffic_mbuf_2048   : 4096
1536             dp_flows    : 1048576                              <4>
1537             global_flows : 10240                               <5>
1538----
1539<1> Memory section header
1540<2> Numbers of memory buffers allocated for packets in transit, per port pair. Numbers are specified per packet size.
1541<3> Numbers of memory buffers allocated for holding the part of the packet which is remained unchanged per template.
1542You should increase numbers here, only if you have very large amount of templates.
1543<4> Number of TRex flow objects allocated (To get best performance they are allocated upfront, and not dynamically).
1544If you expect more concurrent flows than the default (1048576), enlarge this.
1545<5> Number objects TRex allocates for holding NAT ``in transit'' connections. In stateful mode, TRex learn NAT
1546translation by looking at the address changes done by the DUT to the first packet of each flow. So, these are the
1547number of flows for which TRex sent the first flow packet, but did not learn the translation yet. Again, default
1548here (10240) should be good. Increase only if you use NAT and see issues.
1549
1550
1551==== Platform section configuration  
1552
1553The platform section is optional. It is used to tune the performance and allocate the cores to the right NUMA  
1554a configuration file now has the folowing struct to support multi instance 
1555
1556[source,python]
1557----
1558- version       : 2
1559  interfaces    : ["03:00.0","03:00.1"]
1560  port_limit    : 2 
1561....
1562  platform :                                                    <1>
1563        master_thread_id  : 0                                   <2>
1564        latency_thread_id : 5                                   <3>
1565        dual_if   :                                             <4>
1566             - socket   : 0                                     <5>
1567               threads  : [1,2,3,4]                             <6>
1568----
1569<1> Platform section header.
1570<2> Hardware thread_id for control thread.
1571<3> Hardware thread_id for RX thread.
1572<4> ``dual_if'' section defines info for interface pairs (according to the order in ``interfaces'' list).
1573each section, starting with ``- socket'' defines info for different interface pair.
1574<5> The NUMA node from which memory will be allocated for use by the interface pair.
1575<6> Hardware threads to be used for sending packets for the interface pair. Threads are pinned to cores, so specifying threads
1576actually determines the hardware cores.
1577
1578*Real example:* anchor:numa-example[]
1579
1580We connected 2 Intel XL710 NICs close to each other on the motherboard. They shared the same NUMA:
1581
1582image:images/same_numa.png[title="2_NICSs_same_NUMA"]
1583
1584CPU utilization was very high ~100%, with c=2 and c=4 the results were same.
1585
1586Then, we moved the cards to different NUMAs:
1587
1588image:images/different_numa.png[title="2_NICSs_different_NUMAs"]
1589
1590*+*
1591We added configuration to the /etc/trex_cfg.yaml:
1592
1593[source,python]
1594  platform :
1595        master_thread_id  : 0
1596        latency_thread_id : 8
1597        dual_if   :
1598             - socket   : 0
1599               threads  : [1, 2, 3, 4, 5, 6, 7]
1600             - socket   : 1
1601               threads  : [9, 10, 11, 12, 13, 14, 15]
1602
1603This gave best results: with *\~98 Gb/s* TX BW and c=7, CPU utilization became *~21%*! (40% with c=4)
1604
1605
1606==== Timer Wheeel  section configuration  
1607
1608anchor:timer_w[]
1609
1610The memory section is optional. It is used when there is a need to tune the amount of memory used by TRex packet manager.
1611Default values (from the TRex source code), are usually good for most users. Unless you have some unusual needs, you can
1612eliminate this section.
1613
1614==== Timer Wheel section configuration  
1615The flow scheduler uses timer wheel to schedule flows. To tune it for a large number of flows it is possible to change the default values.
1616This is an advance configuration, don't use it if you don't know what you are doing. it can be configure in trex_cfg file and trex traffic profile.
1617
1618[source,python]
1619----
1620  tw :                             
1621     buckets : 1024                <1>
1622     levels  : 3                   <2>
1623     bucket_time_usec : 20.0       <3>
1624----
1625<1> the number of buckets in each level, higher number will improve performance, but will reduce the maximum levels.
1626<2> how many levels.  
1627<3> bucket time in usec. higher number will create more bursts  
1628
1629
1630=== Command line options 
1631
1632anchor:cml-line[]
1633
1634*--active-flows*::
1635    An experimental switch to scale up or down the number of active flows.
1636    It is not accurate due to the quantization of flow scheduler and in some cases does not work.
1637    Example: --active-flows 500000 wil set the ballpark of the active flows to be ~0.5M.
1638
1639*--allow-coredump*::
1640Allow creation of core dump.
1641
1642*--arp-refresh-period <num>*::
1643Period in seconds between sending of gratuitous ARP for our addresses. Value of 0 means ``never send``.
1644
1645*-c <num>*::
1646Number of hardware threads to use per interface pair. Use at least 4 for TRex 40Gbs. +
1647TRex uses 2 threads for inner needs. Rest of the threads can be used. Maximum number here, can be number of free threads
1648divided by number of interface pairs. +
1649For virtual NICs on VM, we always use one thread per interface pair.
1650
1651*--cfg <file name>*::
1652TRex configuration file to use. See relevant manual section for all config file options.
1653
1654*--checksum-offload*::
1655Enable IP, TCP and UDP tx checksum offloading, using DPDK. This requires all used interfaces to support this.
1656
1657*--client_cfg <file>*::
1658YAML file describing clients configuration. Look link:trex_manual.html#_client_clustering_configuration[here] for details.
1659
1660*-d <num>*::
1661Duration of the test in seconds.
1662
1663*-e*::
1664  Same as `-p`, but change the src/dst IP according to the port. Using this, you will get all the packets of the
1665  same flow from the same port, and with the same src/dst IP. +
1666  It will not work good with NBAR as it expects all clients ip to be sent from same direction.
1667
1668*-f <yaml file>*::
1669Specify traffic YAML configuration file to use. Mandatory option for stateful mode.
1670
1671*--hops <num>*::
1672   Provide number of hops in the setup (default is one hop). Relevant only if the Rx check is enabled.
1673   Look link:trex_manual.html#_flow_order_latency_verification[here] for details.
1674
1675*--iom <mode>*::
1676        I/O mode. Possible values: 0 (silent), 1 (normal), 2 (short).
1677
1678*--ipv6*::
1679       Convert templates to IPv6 mode.
1680
1681*-k <num>*::
1682   Run ``warm up'' traffic for num seconds before starting the test. This is needed if TRex is connected to switch running
1683   spanning tree. You want the switch to see traffic from all relevant source MAC addresses before starting to send real
1684   data. Traffic sent is the same used for the latency test (-l option) +
1685   Current limitation (holds for TRex version 1.82): does not work properly on VM.
1686
1687*-l <rate>*::
1688    In parallel to the test, run latency check, sending packets at rate/sec from each interface.
1689
1690*--learn-mode <mode>*::
1691    Learn the dynamic NAT translation. Look link:trex_manual.html#_nat_support[here] for details.
1692
1693*--learn-verify*::
1694   Used for testing the NAT learning mechanism. Do the learning as if DUT is doing NAT, but verify that packets
1695   are not actually changed.
1696
1697*--limit-ports <port num>*::
1698   Limit the number of ports used. Overrides the ``port_limit'' from config file.
1699
1700*--lm <hex bit mask>*::
1701Mask specifying which ports will send traffic. For example, 0x1 - Only port 0 will send. 0x4 - only port 2 will send.
1702This can be used to verify port connectivity. You can send packets from one port, and look at counters on the DUT.
1703
1704*--lo*::
1705   Latency only - Send only latency packets. Do not send packets from the templates/pcap files.
1706
1707*-m <num>*::
1708   Rate multiplier. TRex will multiply the CPS rate of each template by num.
1709
1710*--nc*::
1711    If set, will terminate exacly at the end of the specified duration.
1712    This provides faster, more accurate TRex termination.
1713    By default (without this option), TRex waits for all flows to terminate gracefully. In case of a very long flow, termination might prolong. 
1714
1715*--no-flow-control-change*::
1716    Since version 2.21. +
1717    Prevents TRex from changing flow control. By default (without this option), TRex disables flow control at startup for all cards, except for the Intel XL710 40G card.
1718
1719*--no-hw-flow-stat*::
1720    Relevant only for Intel x710 stateless mode. Do not use HW counters for flow stats. +
1721    Enabling this will support lower traffic rate, but will also report RX byte count statistics.
1722
1723*--no-key*:: Daemon mode, don't get input from keyboard.
1724
1725*--no-watchdog*:: Disable watchdog.
1726    
1727*-p*::
1728Send all packets of the same flow from the same direction. For each flow, TRex will randomly choose between client port and
1729server port, and send all the packets from this port. src/dst IPs keep their values as if packets are sent from two ports.
1730Meaning, we get on the same port packets from client to server, and from server to client. +
1731If you are using this with a router, you can not relay on routing rules to pass traffic to TRex, you must configure policy
1732based routes to pass all traffic from one DUT port to the other. +
1733
1734*-pm <num>*::
1735   Platform factor. If the setup includes splitter, you can multiply all statistic number displayed by TRex by this factor, so that they will match the DUT counters.
1736  
1737*-pubd*::
1738  Disable ZMQ monitor's publishers.
1739
1740*--rx-check <sample rate>*::
1741        Enable Rx check module. Using this, each thread randomly samples 1/sample_rate of the flows and checks packet order, latency, and additional statistics for the sampled flows.
1742        Note: This feature works on the RX thread.
1743
1744*--software*::
1745    Since version 2.21. +
1746    Do not configure any hardware rules. In this mode, all RX packets will be processed by software. No HW assist for dropping (while counting) packets will be used.
1747    This mode is good for enabling features like link:trex_stateless.html#_tutorial_per_stream_statistics[per stream statistics], and
1748    link:trex_stateless.html#_tutorial_per_stream_latency_jitter_packet_errors[latency], support packet types, not supported by HW flow director rules (For example QinQ).
1749    Drawback of this is that because software has to handle all received packets, total rate of RX streams is significantly lower.
1750    Currently, this mode is also limited to using only one TX core (and one RX core as usual).
1751
1752*-v <verbosity level>*::
1753   Show debug info. Value of 1 shows debug info on startup. Value of 3, shows debug info during run at some cases. Might slow down operation.
1754
1755*--vlan*:: Relevant only for stateless mode with Intel 82599 10G NIC.
1756   When configuring flow stat and latency per stream rules, assume all streams uses VLAN.
1757
1758*-w <num seconds>*::
1759   Wait additional time between NICs initialization and sending traffic. Can be useful if DUT needs extra setup time. Default is 1 second.
1760
1761
1762ifndef::backend-docbook[]
1763
1764
1765endif::backend-docbook[]
1766
1767== Appendix
1768
1769=== Simulator 
1770 
1771The TRex simulator is a linux application (no DPDK needed) that can run on any Linux (it can also run on TRex machine itself).
1772you can create output pcap file from input of traffic YAML.
1773
1774====  Simulator 
1775
1776
1777[source,bash]
1778----
1779
1780$./bp-sim-64-debug -f avl/sfr_delay_10_1g.yaml -v 1
1781
1782 -- loading cap file avl/delay_10_http_get_0.pcap 
1783 -- loading cap file avl/delay_10_http_post_0.pcap 
1784 -- loading cap file avl/delay_10_https_0.pcap 
1785 -- loading cap file avl/delay_10_http_browsing_0.pcap 
1786 -- loading cap file avl/delay_10_exchange_0.pcap 
1787 -- loading cap file avl/delay_10_mail_pop_0.pcap 
1788 -- loading cap file avl/delay_10_mail_pop_1.pcap 
1789 -- loading cap file avl/delay_10_mail_pop_2.pcap 
1790 -- loading cap file avl/delay_10_oracle_0.pcap 
1791 -- loading cap file avl/delay_10_rtp_160k_full.pcap 
1792 -- loading cap file avl/delay_10_rtp_250k_full.pcap 
1793 -- loading cap file avl/delay_10_smtp_0.pcap 
1794 -- loading cap file avl/delay_10_smtp_1.pcap 
1795 -- loading cap file avl/delay_10_smtp_2.pcap 
1796 -- loading cap file avl/delay_10_video_call_0.pcap 
1797 -- loading cap file avl/delay_10_sip_video_call_full.pcap 
1798 -- loading cap file avl/delay_10_citrix_0.pcap 
1799 -- loading cap file avl/delay_10_dns_0.pcap 
1800 id,name                                    , tps, cps,f-pkts,f-bytes, duration,   Mb/sec,   MB/sec,   c-flows,  PPS,total-Mbytes-duration,errors,flows    #<2>
1801 00, avl/delay_10_http_get_0.pcap             ,404.52,404.52,    44 ,   37830 ,   0.17 , 122.42 ,   15.30 ,         67 , 17799 ,       2 , 0 , 1 
1802 01, avl/delay_10_http_post_0.pcap            ,404.52,404.52,    54 ,   48468 ,   0.21 , 156.85 ,   19.61 ,         85 , 21844 ,       2 , 0 , 1 
1803 02, avl/delay_10_https_0.pcap                ,130.87,130.87,    96 ,   91619 ,   0.22 ,  95.92 ,   11.99 ,         29 , 12564 ,       1 , 0 , 1 
1804 03, avl/delay_10_http_browsing_0.pcap        ,709.89,709.89,    37 ,   34425 ,   0.13 , 195.50 ,   24.44 ,         94 , 26266 ,       2 , 0 , 1 
1805 04, avl/delay_10_exchange_0.pcap             ,253.81,253.81,    43 ,    9848 ,   1.57 ,  20.00 ,    2.50 ,        400 , 10914 ,       0 , 0 , 1 
1806 05, avl/delay_10_mail_pop_0.pcap             ,4.76,4.76,    20 ,    5603 ,   0.17 ,   0.21 ,    0.03 ,          1 ,    95 ,       0 , 0 , 1 
1807 06, avl/delay_10_mail_pop_1.pcap             ,4.76,4.76,   114 ,  101517 ,   0.25 ,   3.86 ,    0.48 ,          1 ,   543 ,       0 , 0 , 1 
1808 07, avl/delay_10_mail_pop_2.pcap             ,4.76,4.76,    30 ,   15630 ,   0.19 ,   0.60 ,    0.07 ,          1 ,   143 ,       0 , 0 , 1 
1809 08, avl/delay_10_oracle_0.pcap               ,79.32,79.32,   302 ,   56131 ,   6.86 ,  35.62 ,    4.45 ,        544 , 23954 ,       0 , 0 , 1 
1810 09, avl/delay_10_rtp_160k_full.pcap          ,2.78,8.33,  1354 , 1232757 ,  61.24 ,  27.38 ,    3.42 ,        170 ,  3759 ,       0 , 0 , 3 
1811 10, avl/delay_10_rtp_250k_full.pcap          ,1.98,5.95,  2069 , 1922000 ,  61.38 ,  30.48 ,    3.81 ,        122 ,  4101 ,       0 , 0 , 3 
1812 11, avl/delay_10_smtp_0.pcap                 ,7.34,7.34,    22 ,    5618 ,   0.19 ,   0.33 ,    0.04 ,          1 ,   161 ,       0 , 0 , 1 
1813 12, avl/delay_10_smtp_1.pcap                 ,7.34,7.34,    35 ,   18344 ,   0.21 ,   1.08 ,    0.13 ,          2 ,   257 ,       0 , 0 , 1 
1814 13, avl/delay_10_smtp_2.pcap                 ,7.34,7.34,   110 ,   96544 ,   0.27 ,   5.67 ,    0.71 ,          2 ,   807 ,       0 , 0 , 1 
1815 14, avl/delay_10_video_call_0.pcap           ,11.90,11.90,  2325 , 2532577 ,  36.56 , 241.05 ,   30.13 ,        435 , 27662 ,       3 , 0 , 1 
1816 15, avl/delay_10_sip_video_call_full.pcap    ,29.35,58.69,  1651 ,  120315 ,  24.56 ,  28.25 ,    3.53 ,        721 , 48452 ,       0 , 0 , 2 
1817 16, avl/delay_10_citrix_0.pcap               ,43.62,43.62,   272 ,   84553 ,   6.23 ,  29.51 ,    3.69 ,        272 , 11866 ,       0 , 0 , 1 
1818 17, avl/delay_10_dns_0.pcap                  ,1975.02,1975.02,     2 ,     162 ,   0.01 ,   2.56 ,    0.32 ,         22 ,  3950 ,       0 , 0 , 1 
1819
1820 00, sum                                      ,4083.86,93928.84,  8580 , 6413941 ,   0.00 , 997.28 ,  124.66 ,       2966 , 215136 ,      12 , 0 , 23 
1821 Memory usage 
1822 size_64        : 1687 
1823 size_128       : 222 
1824 size_256       : 798 
1825 size_512       : 1028 
1826 size_1024      : 86 
1827 size_2048      : 4086 
1828 Total    :       8.89 Mbytes  159% util #<1>
1829
1830----
1831<1> the memory usage of the templates 
1832<2> CSV for all the templates 
1833
1834
1835=== firmware update to XL710/X710 
1836anchor:xl710-firmware[]
1837 
1838To upgrade the firmware  follow  this
1839
1840==== Download the driver 
1841
1842*Download driver i40e from link:https://downloadcenter.intel.com/download/24411/Network-Adapter-Driver-for-PCI-E-40-Gigabit-Network-Connections-under-Linux-[here]
1843*Build the kernel module
1844
1845[source,bash]
1846----
1847$tar -xvzf i40e-1.3.47
1848$cd i40e-1.3.47/src
1849$make 
1850$sudo insmod  i40e.ko
1851----
1852
1853
1854====  Bind the NIC to Linux
1855
1856In this stage we bind the NIC to Linux (take it from DPDK)
1857
1858[source,bash]
1859----
1860$sudo ./dpdk_nic_bind.py --status # show the ports 
1861
1862Network devices using DPDK-compatible driver
1863============================================
18640000:02:00.0 'Device 1583' drv=igb_uio unused=      #<1>
18650000:02:00.1 'Device 1583' drv=igb_uio unused=      #<2>
18660000:87:00.0 'Device 1583' drv=igb_uio unused=
18670000:87:00.1 'Device 1583' drv=igb_uio unused=
1868
1869$sudo dpdk_nic_bind.py -u 02:00.0  02:00.1          #<3> 
1870
1871$sudo dpdk_nic_bind.py -b i40e 02:00.0 02:00.1      #<4>
1872
1873$ethtool -i p1p2                                    #<5>  
1874
1875driver: i40e
1876version: 1.3.47
1877firmware-version: 4.24 0x800013fc 0.0.0             #<6>
1878bus-info: 0000:02:00.1
1879supports-statistics: yes
1880supports-test: yes
1881supports-eeprom-access: yes
1882supports-register-dump: yes
1883supports-priv-flags: yes
1884
1885   
1886$ethtool -S p1p2 
1887$lspci -s 02:00.0 -vvv                              #<7>
1888
1889
1890----
1891<1> XL710 ports that need to unbind from DPDK
1892<2> XL710 ports that need to unbind from DPDK
1893<3> Unbind from DPDK using this command
1894<4> Bind to linux to i40e driver 
1895<5> Show firmware version throw linux driver 
1896<6> Firmare version
1897<7> More info 
1898
1899
1900====  Upgrade 
1901
1902Download NVMUpdatePackage.zip from Intel site link:http://downloadcenter.intel.com/download/24769/NVM-Update-Utility-for-Intel-Ethernet-Converged-Network-Adapter-XL710-X710-Series[here]  
1903It includes the utility `nvmupdate64e`
1904
1905Run this:
1906
1907[source,bash]
1908----
1909$sudo ./nvmupdate64e  
1910----
1911
1912You might need a power cycle and to run this command a few times to get the latest firmware  
1913
1914====  QSFP+ support for XL710
1915
1916see link:https://www.google.co.il/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&cad=rja&uact=8&ved=0ahUKEwjJhPSH3b3LAhUp7nIKHSkACUYQFggaMAA&url=http%3A%2F%2Fwww.intel.co.id%2Fcontent%2Fdam%2Fwww%2Fpublic%2Fus%2Fen%2Fdocuments%2Frelease-notes%2Fxl710-ethernet-controller-feature-matrix.pdf&usg=AFQjCNFhwozfz-XuKGMOy9_MJDbetw15Og&sig2=ce7YU9F9Et6xf6KvqSFBxg&bvm=bv.116636494,d.bGs[QSFP+ support] for QSFP+ support and Firmware requirement for XL710
1917
1918
1919=== TRex with ASA 5585
1920
1921When running TRex aginst ASA 5585, you have to notice following things:
1922
1923* ASA can't forward ipv4 options, so there is a need to use --learn-mode 1 (or 3) in case of NAT. In this mode, bidirectional UDP flows are not supported.
1924--learn-mode 1 support TCP sequence number randomization in both sides of the connection (client to server and server client). For this to work, TRex must learn
1925the translation of packets from both sides, so this mode reduce the amount of connections per second TRex can generate (The number is still high enough to test
1926any existing firewall). If you need higher cps rate, you can use --learn-mode 3. This mode handles sequence number randomization on client->server side only.
1927* Latency should be tested using ICMP with `--l-pkt-mode 2`
1928
1929====  ASA 5585 sample configuration
1930
1931[source,bash]
1932----
1933ciscoasa# show running-config 
1934: Saved
1935
1936: 
1937: Serial Number: JAD194801KX
1938: Hardware:   ASA5585-SSP-10, 6144 MB RAM, CPU Xeon 5500 series 2000 MHz, 1 CPU (4 cores)
1939:
1940ASA Version 9.5(2) 
1941!
1942hostname ciscoasa
1943enable password 8Ry2YjIyt7RRXU24 encrypted
1944passwd 2KFQnbNIdI.2KYOU encrypted
1945names
1946!
1947interface Management0/0
1948 management-only
1949 nameif management
1950 security-level 100
1951 ip address 10.56.216.106 255.255.255.0 
1952!             
1953interface TenGigabitEthernet0/8
1954 nameif inside
1955 security-level 100
1956 ip address 15.0.0.1 255.255.255.0 
1957!             
1958interface TenGigabitEthernet0/9
1959 nameif outside
1960 security-level 0
1961 ip address 40.0.0.1 255.255.255.0 
1962!             
1963boot system disk0:/asa952-smp-k8.bin
1964ftp mode passive
1965pager lines 24
1966logging asdm informational
1967mtu management 1500
1968mtu inside 9000
1969mtu outside 9000
1970no failover   
1971no monitor-interface service-module 
1972icmp unreachable rate-limit 1 burst-size 1
1973no asdm history enable
1974arp outside 40.0.0.2 90e2.baae.87d1 
1975arp inside 15.0.0.2 90e2.baae.87d0 
1976arp timeout 14400
1977no arp permit-nonconnected
1978route management 0.0.0.0 0.0.0.0 10.56.216.1 1
1979route inside 16.0.0.0 255.0.0.0 15.0.0.2 1
1980route outside 48.0.0.0 255.0.0.0 40.0.0.2 1
1981timeout xlate 3:00:00
1982timeout pat-xlate 0:00:30
1983timeout conn 1:00:00 half-closed 0:10:00 udp 0:02:00 sctp 0:02:00 icmp 0:00:02
1984timeout sunrpc 0:10:00 h323 0:05:00 h225 1:00:00 mgcp 0:05:00 mgcp-pat 0:05:00
1985timeout sip 0:30:00 sip_media 0:02:00 sip-invite 0:03:00 sip-disconnect 0:02:00
1986timeout sip-provisional-media 0:02:00 uauth 0:05:00 absolute
1987timeout tcp-proxy-reassembly 0:01:00
1988timeout floating-conn 0:00:00
1989user-identity default-domain LOCAL
1990http server enable
1991http 192.168.1.0 255.255.255.0 management
1992no snmp-server location
1993no snmp-server contact
1994crypto ipsec security-association pmtu-aging infinite
1995crypto ca trustpool policy
1996telnet 0.0.0.0 0.0.0.0 management
1997telnet timeout 5
1998ssh stricthostkeycheck
1999ssh timeout 5 
2000ssh key-exchange group dh-group1-sha1
2001console timeout 0
2002!             
2003tls-proxy maximum-session 1000
2004!             
2005threat-detection basic-threat
2006threat-detection statistics access-list
2007no threat-detection statistics tcp-intercept
2008dynamic-access-policy-record DfltAccessPolicy
2009!             
2010class-map icmp-class
2011 match default-inspection-traffic
2012class-map inspection_default
2013 match default-inspection-traffic
2014!             
2015!             
2016policy-map type inspect dns preset_dns_map
2017 parameters   
2018  message-length maximum client auto
2019  message-length maximum 512
2020policy-map icmp_policy
2021 class icmp-class
2022  inspect icmp 
2023policy-map global_policy
2024 class inspection_default
2025  inspect dns preset_dns_map 
2026  inspect ftp 
2027  inspect h323 h225 
2028  inspect h323 ras 
2029  inspect rsh 
2030  inspect rtsp 
2031  inspect esmtp 
2032  inspect sqlnet 
2033  inspect skinny  
2034  inspect sunrpc 
2035  inspect xdmcp 
2036  inspect sip  
2037  inspect netbios 
2038  inspect tftp 
2039  inspect ip-options 
2040!             
2041service-policy global_policy global
2042service-policy icmp_policy interface outside
2043prompt hostname context 
2044!             
2045jumbo-frame reservation
2046!             
2047no call-home reporting anonymous
2048: end         
2049ciscoasa# 
2050----
2051
2052====  TRex commands example 
2053
2054Using these commands the configuration is:
2055
20561. NAT learn mode (TCP-ACK)
20572. Delay of 1 second at start up (-k 1). It was added because ASA drops the first packets.
20583. Latency is configured to ICMP reply mode (--l-pkt-mode 2).
2059
2060
2061*Simple HTTP:*::
2062[source,bash]
2063----
2064$sudo ./t-rex-64 -f cap2/http_simple.yaml -d 1000 -l 1000 --l-pkt-mode 2 -m 1000  --learn-mode 1 -k 1
2065----
2066
2067This is more realistic traffic for enterprise (we removed from SFR file the bidirectional UDP traffic templates, which (as described above), are not supported in this mode).
2068
2069*Enterprise profile:*::
2070[source,bash]
2071----
2072$sudo ./t-rex-64 -f avl/sfr_delay_10_1g_asa_nat.yaml -d 1000 -l 1000 --l-pkt-mode 2 -m 4 --learn-mode 1 -k 1
2073----
2074
2075The TRex output
2076
2077[source,bash]
2078----
2079-Per port stats table 
2080      ports |               0 |               1 
2081 -----------------------------------------------------------------------------------------
2082   opackets |       106347896 |       118369678 
2083     obytes |     33508291818 |    118433748567 
2084   ipackets |       118378757 |       106338782 
2085     ibytes |    118434305375 |     33507698915 
2086    ierrors |               0 |               0 
2087    oerrors |               0 |               0 
2088      Tx Bw |     656.26 Mbps |       2.27 Gbps 
2089
2090-Global stats enabled 
2091 Cpu Utilization : 18.4  %  31.7 Gb/core 
2092 Platform_factor : 1.0  
2093 Total-Tx        :       2.92 Gbps   NAT time out    :        0 #<1> (0 in wait for syn+ack) #<1>
2094 Total-Rx        :       2.92 Gbps   NAT aged flow id:        0 #<1>
2095 Total-PPS       :     542.29 Kpps   Total NAT active:      163  (12 waiting for syn)
2096 Total-CPS       :       8.30 Kcps   Nat_learn_errors:        0
2097
2098 Expected-PPS    :     539.85 Kpps
2099 Expected-CPS    :       8.29 Kcps  
2100 Expected-BPS    :       2.90 Gbps  
2101
2102 Active-flows    :     7860  Clients :      255   Socket-util : 0.0489 %    
2103 Open-flows      :  3481234  Servers :     5375   Socket :     7860 Socket/Clients :  30.8 
2104 drop-rate       :       0.00  bps   #<1>
2105 current time    : 425.1 sec  
2106 test duration   : 574.9 sec  
2107
2108-Latency stats enabled 
2109 Cpu Utilization : 0.3 %  
2110 if|   tx_ok , rx_ok  , rx   ,error,    average   ,   max         , Jitter ,  max window 
2111   |         ,        , check,     , latency(usec),latency (usec) ,(usec)  ,             
2112 ---------------------------------------------------------------------------------------------------------------- 
2113 0 |   420510,  420495,         0,   1,         58  ,    1555,      14      |  240  257  258  258  219  930  732  896  830  472  190  207  729 
2114 1 |   420496,  420509,         0,   1,         51  ,    1551,      13      |  234  253  257  258  214  926  727  893  826  468  187  204  724
2115----
2116<1>  These counters should be zero 
2117
2118anchor:fedora21_example[]
2119
2120=== Fedora 21 Server installation
2121
2122Download the .iso file from link above, boot with it using Hypervisor or CIMC console. +
2123Troubleshooting -> install in basic graphics mode
2124
2125* In packages selection, choose:
2126
2127** C Development Tools and Libraries
2128
2129** Development Tools
2130
2131** System Tools
2132
2133* Set Ethernet configuration if needed
2134
2135* Use default hard-drive partitions, reclaim space if needed
2136
2137* After installation, edit file /etc/selinux/config +
2138set: +
2139SELINUX=disabled
2140
2141* Run: +
2142systemctl disable firewalld 
2143
2144* Edit file /etc/yum.repos.d/fedora-updates.repo +
2145set everywhere: +
2146enabled=0
2147
2148* Reboot
2149
2150=== Configure Linux host as network emulator
2151
2152There are lots of Linux tutorials on the web, so this will not be full tutorial, only highlighting some key points. Commands
2153were checked on Ubuntu system.
2154
2155For this example:
2156
21571. TRex Client side network is 16.0.0.x 
21582. TRex Server side network is 48.0.0.x 
21593. Linux Client side network eth0 is configured with IPv4 as 172.168.0.1 
21604. Linux Server side network eth1 is configured with IPv4 as 10.0.0.1 
2161
2162[source,bash]
2163----
2164
2165  TRex-0 (16.0.0.1->48.0.0.1 )   <-->
2166
2167                ( 172.168.0.1/255.255.0.0)-eth0 [linux] -( 10.0.0.1/255.255.0.0)-eth1 
2168
2169                <--> TRex-1 (16.0.0.1<-48.0.0.1)
2170  
2171----
2172
2173
2174==== Enable forwarding
2175One time (will be discarded after reboot): +
2176
2177[source,bash]
2178----
2179echo 1 > /proc/sys/net/ipv4/ip_forward
2180----
2181To make this permanent, add the following line to the file /etc/sysctl.conf: +
2182----
2183net.ipv4.ip_forward=1
2184----
2185
2186==== Add static routes
2187Example if for the default TRex networks, 48.0.0.0 and 16.0.0.0.
2188
2189Routing all traffic from 48.0.0.0 to the gateway 10.0.0.100
2190[source,bash]
2191----
2192route add -net 48.0.0.0 netmask 255.255.0.0 gw 10.0.0.100
2193----
2194
2195Routing all traffic from 16.0.0.0 to the gateway 172.168.0.100
2196[source,bash]
2197----
2198route add -net 16.0.0.0 netmask 255.255.0.0 gw 172.168.0.100
2199----
2200If you use stateless mode, and decide to add route only in one direction, remember to disable reverse path check. +
2201For example, to disable on all interfaces:
2202[source,bash]
2203----
2204for i in /proc/sys/net/ipv4/conf/*/rp_filter ; do
2205  echo 0 > $i 
2206done
2207----
2208
2209Alternatively, you can edit /etc/network/interfaces, and add something like this for both ports connected to TRex.
2210This will take effect, only after restarting networking (rebooting the machine in an alternative also).
2211----
2212auto eth1
2213iface eth1 inet static
2214address 16.0.0.100
2215netmask 255.0.0.0
2216network 16.0.0.0
2217broadcast 16.255.255.255
2218... same for 48.0.0.0
2219----
2220
2221==== Add static ARP entries
2222[source,bash]
2223----
2224sudo arp -s 10.0.0.100 <Second TRex port MAC>
2225sudo arp -s 172.168.0.100 <TRex side the NICs are not visible to ifconfig, run:
2226----
2227
2228=== Configure Linux to use VF on Intel X710 and 82599 NICs
2229
2230TRex supports paravirtualized interfaces such as VMXNET3/virtio/E1000 however when connected to a vSwitch, the vSwitch limits the performance. VPP or OVS-DPDK can improve the performance but require more software resources to handle the rate.
2231SR-IOV can accelerate the performance and reduce CPU resource usage as well as latency by utilizing NIC hardware switch capability (the switching  is done by hardware).
2232TRex version 2.15 now includes SR-IOV support for XL710 and X710.
2233The following diagram compares between vSwitch and SR-IOV.
2234
2235image:images/sr_iov_vswitch.png[title="vSwitch_main",width=850]
2236
2237One use case which shows the performance gain that can be acheived by using SR-IOV is when a user wants to create a pool of TRex VMs that tests a pool of virtual DUTs (e.g. ASAv,CSR etc.)
2238When using newly supported SR-IOV, compute, storage and networking resources can be controlled dynamically (e.g by using OpenStack)
2239 
2240image:images/sr_iov_trex.png[title="vSwitch_main",width=850]
2241
2242The above diagram is an example of one server with two NICS. TRex VMs can be allocated on one NIC while the DUTs can be allocated on another.
2243
2244
2245Following are some links we used and lessons we learned while putting up an environment for testing TRex with VF interfaces (using SR-IOV).
2246This is by no means a full toturial of VF usage, and different Linux distributions might need slightly different handling.
2247
2248==== Links and resources
2249link:http://www.intel.com/content/dam/www/public/us/en/documents/technology-briefs/xl710-sr-iov-config-guide-gbe-linux-brief.pdf[This]
2250is a good tutorial by Intel of SR-IOV and how to configure. +
2251
2252link:http://dpdk.org/doc/guides/nics/intel_vf.html[This] is a tutorial from DPDK documentation. +
2253
2254==== Linux configuration
2255First, need to verify BIOS support for the feature. Can consult link:http://kpanic.de/node/8[this link] for directions. +
2256Second, need to make sure you have the correct kernel options. +
2257We added the following options to the kernel boot command on Grub: ``iommu=pt intel_iommu=on pci_pt_e820_access=on''. This
2258was needed on Fedora and Ubuntu. When using Centos, adding these options was not needed. +
2259To load the kernel module with the correct VF parameters after reboot add following line ``options i40e max_vfs=1,1'' to some file in ``/etc/modprobe.d/'' +
2260On Centos, we needed to also add the following file (example for x710 ): +
2261
2262[source,bash]
2263----
2264cat /etc/sysconfig/modules/i40e.modules
2265#!/bin/sh
2266rmmod i40e >/dev/null 2>&1
2267exec /sbin/modprobe i40e >/dev/null 2>&1
2268----
2269
2270==== x710 specific instructions
2271For x710 (i40e driver), we needed to download latest kernel driver. On all distributions we were using, existing driver was not new enough. +
2272To make the system use your new compiled driver with the correct parameters: +
2273Copy the .ko file to /lib/modules/Your kernel version as seen by uname -r/kernel/drivers/net/ethernet/intel/i40e/i40e.ko +
2274
2275==== 82599 specific instructions
2276In order to make VF interfaces work correctly, we had to increase mtu on related PF interfaces. +
2277For example, if you run with max_vfs=1,1 (one VF per PF), you will have something like this:
2278
2279[source,bash]
2280----
2281sudo ./dpdk_nic_bind.py -s
2282Network devices using DPDK-compatible driver
2283============================================
22840000:03:10.0 '82599 Ethernet Controller Virtual Function' drv=igb_uio unused=
22850000:03:10.1 '82599 Ethernet Controller Virtual Function' drv=igb_uio unused=
2286
2287Network devices using kernel driver
2288===================================
22890000:01:00.0 'I350 Gigabit Network Connection' if=eth0 drv=igb unused=igb_uio *Active*
22900000:03:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection' if=eth2 drv=ixgbe unused=igb_uio
22910000:03:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection' if=eth3 drv=ixgbe unused=igb_uio
2292----
2293
2294In order to work with 0000:03:10.0 and 0000:03:10.1, you will have to run the following +
2295[source,bash]
2296----
2297sudo ifconfig eth3 up mtu 9000
2298sudo ifconfig eth2 up mtu 9000
2299----
2300
2301
2302TRex stateful performance::
2303
2304Using the following command, running on x710 card with VF driver, we can see that TRex can reach 30GBps, using only one core. We can also see that the average latency is around 20 usec, which is pretty much the same value we get on loopback ports with x710 physical function without VF.
2305$sudo ./t-rex-64 -f cap2/http_simple.yaml -m 40000 -l 100 -c 1 -p
2306
2307[source,python]
2308----
2309
2310$sudo ./t-rex-64 -f cap2/http_simple.yaml -m 40000 -l 100 -c 1 -p
2311
2312  -Per port stats table
2313      ports |               0 |               1
2314  -----------------------------------------------------------------------------------------
2315   opackets |       106573954 |       107433792
2316     obytes |     99570878833 |    100374254956
2317   ipackets |       107413075 |       106594490
2318     ibytes | 100354899813    |     99590070585
2319    ierrors |            1038 |            1027
2320    oerrors |               0 |               0
2321      Tx Bw |      15.33 Gbps |      15.45 Gbps
2322 
2323-Global stats enabled
2324Cpu Utilization : 91.5  %  67.3 Gb/core
2325Platform_factor : 1.0
2326Total-Tx :      30.79 Gbps
2327Total-Rx :      30.79 Gbps
2328Total-PPS :       4.12 Mpps
2329Total-CPS :     111.32 Kcps
2330 
2331Expected-PPS :       4.11 Mpps
2332Expected-CPS :     111.04 Kcps
2333Expected-BPS :      30.71 Gbps
2334 
2335Active-flows :    14651  Clients : 255   Socket-util : 0.0912 %
2336Open-flows :  5795073  Servers : 65535   Socket :    14652 Socket/Clients :  57.5
2337drop-rate :       0.00 bps
2338current time : 53.9 sec
2339test duration : 3546.1 sec
2340 
2341 -Latency stats enabled
2342Cpu Utilization : 23.4 %
2343if| tx_ok , rx_ok  , rx check ,error,       latency (usec) ,    Jitter         
2344   | ,        ,          , ,   average   , max  ,    (usec)
2345  -------------------------------------------------------------------------------
23460 | 5233,    5233,         0, 0,         19  , 580,       5      | 37  37  37 4
23471 | 5233,    5233,         0, 0,         22  , 577,       5      | 38  40  39 3
2348----
2349
2350TRex stateless performance::
2351
2352[source,python]
2353----
2354
2355$sudo ./t-rex-64 -i -c 1
2356
2357trex>portattr
2358Port Status
2359 
2360     port |          0           |          1
2361  -------------------------------------------------------------
2362  driver          | net_i40e_vf      |     net_i40e_vf
2363  description     | XL710/X710 Virtual  |  XL710/X710 Virtual
2364 
2365With the console command:
2366start -f stl/imix.py -m 8mpps --force --port 0
2367We can see, that we can reach 8M packet per second, which in this case is around 24.28 Gbit/second.
2368 
2369Global Statistics
2370 
2371connection   : localhost, Port 4501                  total_tx_L2  : 24.28 Gb/sec
2372version      : v2.15 total_tx_L1  : 25.55 Gb/sec
2373cpu_util.    : 80.6% @ 1 cores (1 per port)          total_rx     : 24.28 Gb/sec
2374rx_cpu_util. : 66.8%                                 total_pps    : 7.99 Mpkt/sec
2375async_util.  : 0.18% / 1.84 KB/sec                   drop_rate    : 0.00 b/sec
2376queue_full   : 3,467 pkts
2377 
2378Port Statistics
2379 
2380   port    |         0         |         1         | total
2381  ----------------------------------------------------------------------
2382  owner      |           ibarnea |           ibarnea |
2383  link       |                UP |                UP |
2384  state      | TRANSMITTING      |              IDLE |
2385  speed      |           40 Gb/s |           40 Gb/s |
2386  CPU util.  | 80.6%             |              0.0% |
2387  --         |                   |                   |
2388  Tx bps L2  | 24.28 Gbps        |          0.00 bps |        24.28 Gbps
2389  Tx bps L1  | 25.55 Gbps        |             0 bps |        25.55 Gbps
2390  Tx pps     | 7.99 Mpps         |          0.00 pps |         7.99 Mpps
2391  Line Util. |           63.89 % |            0.00 % |
2392  ---        |                   |                   |
2393  Rx bps     | 0.00 bps          |        24.28 Gbps |        24.28 Gbps
2394  Rx pps     | 0.00 pps          |         7.99 Mpps |         7.99 Mpps
2395  ----       |                   |                   |
2396  opackets   | 658532501         |                 0 |         658532501
2397  ipackets   |                 0 |         658612569 |         658612569
2398  obytes     | 250039721918      |                 0 |      250039721918
2399  ibytes     |                 0 |      250070124150 |      250070124150
2400  tx-bytes   | 250.04 GB         |               0 B |         250.04 GB
2401  rx-bytes   |               0 B |         250.07 GB |         250.07 GB
2402  tx-pkts    | 658.53 Mpkts      |            0 pkts |      658.53 Mpkts
2403  rx-pkts    | 0 pkts            |      658.61 Mpkts |      658.61 Mpkts
2404  -----      |                   |                   |
2405  oerrors    |                 0 |                 0 |                 0
2406  ierrors    |                 0 |            15,539 |            15,539
2407----
2408
2409
2410==== Performance
2411See the performance tests we did link:trex_vm_bench.html[here]
2412
2413=== Mellanox ConnectX-4/5 support 
2414
2415anchor:connectx_support[]
2416
2417Mellanox ConnectX-4/5 adapter family supports 100/56/40/25/10 Gb/s Ethernet speeds. 
2418Its DPDK support is a bit different from Intel DPDK support, more information can be found link:http://dpdk.org/doc/guides/nics/mlx5.html[here].
2419Intel NICs do not require additional kernel drivers (except for igb_uio which is already supported in most distributions). ConnectX-4 works on top of Infiniband API (verbs) and requires special kernel modules/user space libs. 
2420This means that it is required to install OFED package to be able to work with this NIC.
2421Installing the full OFED package is the simplest way to make it work (trying to install part of the package can work too but didn't work for us).
2422The advantage of this model is that you can control it using standard Linux tools (ethtool and ifconfig will work).
2423The disadvantage is the OFED dependency.
2424
2425==== Installation  
2426
2427==== Install Linux 
2428
2429We tested the following distro with TRex and OFED. Others might work too.
2430
2431* CentOS 7.2 
2432
2433Following distro was tested and did *not* work for us.
2434
2435* Fedora 21 (3.17.4-301.fc21.x86_64)  
2436* Ubuntu 14.04.3 LTS (GNU/Linux 3.19.0-25-generic x86_64)  -- crash when RSS was enabled link:https://trex-tgn.cisco.com/youtrack/issue/trex-294[MLX RSS issue]
2437
2438==== Install OFED 
2439
2440Information was taken from  link:http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers[Install OFED]
2441
2442* Download 4.0 OFED tar for your distro  
2443
2444[IMPORTANT]
2445=====================================
2446The version must be *MLNX_OFED_LINUX-4.0* or higher (4.0.x)
2447=====================================
2448
2449[IMPORTANT]
2450=====================================
2451Make sure you have an internet connection without firewalls for HTTPS/HTTP - required by yum/apt-get
2452=====================================
2453
2454.Verify md5
2455[source,bash]
2456----
2457$md5sum md5sum MLNX_OFED_LINUX-4.0.0.0.0-rhel7.2-x86_64.tgz
245858b9fb369d7c62cedbc855661a89a9fd  MLNX_OFED_LINUX-4.0-xx.0.0.0-rhel7.2-x86_64.tgz
2459----
2460
2461.Open the tar
2462[source,bash]
2463----
2464$tar -xvzf MLNX_OFED_LINUX-4.0-x.0.0.0-rhel7.2-x86_64.tgz
2465$cd MLNX_OFED_LINUX-4.0-x.0.0.0-rhel7.2-x86_64
2466----
2467
2468.Run Install script
2469[source,bash]
2470----
2471$sudo ./mlnxofedinstall  
2472
2473
2474Log: /tmp/ofed.build.log
2475Logs dir: /tmp/MLNX_OFED_LINUX.10406.logs
2476
2477Below is the list of MLNX_OFED_LINUX packages that you have chosen
2478(some may have been added by the installer due to package dependencies):
2479
2480ofed-scripts
2481mlnx-ofed-kernel-utils
2482mlnx-ofed-kernel-dkms
2483iser-dkms
2484srp-dkms
2485mlnx-sdp-dkms
2486mlnx-rds-dkms
2487mlnx-nfsrdma-dkms
2488libibverbs1
2489ibverbs-utils
2490libibverbs-dev
2491libibverbs1-dbg
2492libmlx4-1
2493libmlx4-dev
2494libmlx4-1-dbg
2495libmlx5-1
2496libmlx5-dev
2497libmlx5-1-dbg
2498libibumad
2499libibumad-static
2500libibumad-devel
2501ibacm
2502ibacm-dev
2503librdmacm1
2504librdmacm-utils
2505librdmacm-dev
2506mstflint
2507ibdump
2508libibmad
2509libibmad-static
2510libibmad-devel
2511libopensm
2512opensm
2513opensm-doc
2514libopensm-devel
2515infiniband-diags
2516infiniband-diags-compat
2517mft
2518kernel-mft-dkms
2519libibcm1
2520libibcm-dev
2521perftest
2522ibutils2
2523libibdm1
2524ibutils
2525cc-mgr
2526ar-mgr
2527dump-pr
2528ibsim
2529ibsim-doc
2530knem-dkms
2531mxm
2532fca
2533sharp
2534hcoll
2535openmpi
2536mpitests
2537knem
2538rds-tools
2539libdapl2
2540dapl2-utils
2541libdapl-dev
2542srptools
2543mlnx-ethtool
2544libsdp1
2545libsdp-dev
2546sdpnetstat
2547
2548This program will install the MLNX_OFED_LINUX package on your machine.
2549Note that all other Mellanox, OEM, OFED, or Distribution IB packages will be removed.
2550Do you want to continue?[y/N]:y
2551
2552Checking SW Requirements...
2553
2554One or more required packages for installing MLNX_OFED_LINUX are missing.
2555Attempting to install the following missing packages:
2556autotools-dev tcl debhelper dkms tk8.4 libgfortran3 graphviz chrpath automake dpatch flex bison autoconf quilt m4 tcl8.4 libltdl-dev pkg-config pytho
2557bxml2 tk swig gfortran libnl1
2558
2559..
2560
2561Removing old packages...
2562Installing new packages
2563Installing ofed-scripts-3.4...
2564Installing mlnx-ofed-kernel-utils-3.4...
2565Installing mlnx-ofed-kernel-dkms-3.4...
2566
2567Removing old packages...
2568Installing new packages
2569Installing ofed-scripts-3.4...
2570Installing mlnx-ofed-kernel-utils-3.4...
2571Installing mlnx-ofed-kernel-dkms-3.4...
2572Installing iser-dkms-1.8.1...
2573Installing srp-dkms-1.6.1...
2574Installing mlnx-sdp-dkms-3.4...
2575Installing mlnx-rds-dkms-3.4...
2576Installing mlnx-nfsrdma-dkms-3.4...
2577Installing libibverbs1-1.2.1mlnx1...
2578Installing ibverbs-utils-1.2.1mlnx1...
2579Installing libibverbs-dev-1.2.1mlnx1...
2580Installing libibverbs1-dbg-1.2.1mlnx1...
2581Installing libmlx4-1-1.2.1mlnx1...
2582Installing libmlx4-dev-1.2.1mlnx1...
2583Installing libmlx4-1-dbg-1.2.1mlnx1...
2584Installing libmlx5-1-1.2.1mlnx1...
2585Installing libmlx5-dev-1.2.1mlnx1...
2586Installing libmlx5-1-dbg-1.2.1mlnx1...
2587Installing libibumad-1.3.10.2.MLNX20150406.966500d...
2588Installing libibumad-static-1.3.10.2.MLNX20150406.966500d...
2589Installing libibumad-devel-1.3.10.2.MLNX20150406.966500d...
2590Installing ibacm-1.2.1mlnx1...
2591Installing ibacm-dev-1.2.1mlnx1...
2592Installing librdmacm1-1.1.0mlnx...
2593Installing librdmacm-utils-1.1.0mlnx...
2594Installing librdmacm-dev-1.1.0mlnx...
2595Installing mstflint-4.5.0...
2596Installing ibdump-4.0.0...
2597Installing libibmad-1.3.12.MLNX20160814.4f078cc...
2598Installing libibmad-static-1.3.12.MLNX20160814.4f078cc...
2599Installing libibmad-devel-1.3.12.MLNX20160814.4f078cc...
2600Installing libopensm-4.8.0.MLNX20160906.32a95b6...
2601Installing opensm-4.8.0.MLNX20160906.32a95b6...
2602Installing opensm-doc-4.8.0.MLNX20160906.32a95b6...
2603Installing libopensm-devel-4.8.0.MLNX20160906.32a95b6...
2604Installing infiniband-diags-1.6.6.MLNX20160814.999c7b2...
2605Installing infiniband-diags-compat-1.6.6.MLNX20160814.999c7b2...
2606Installing mft-4.5.0...
2607Installing kernel-mft-dkms-4.5.0...
2608Installing libibcm1-1.0.5mlnx2...
2609Installing libibcm-dev-1.0.5mlnx2...
2610Installing perftest-3.0...
2611Installing ibutils2-2.1.1...
2612Installing libibdm1-1.5.7.1...
2613Installing ibutils-1.5.7.1...
2614Installing cc-mgr-1.0...
2615Installing ar-mgr-1.0...
2616Installing dump-pr-1.0...
2617Installing ibsim-0.6...
2618Installing ibsim-doc-0.6...
2619Installing knem-dkms-1.1.2.90mlnx1...
2620Installing mxm-3.5.220c57f...
2621Installing fca-2.5.2431...
2622Installing sharp-1.1.1.MLNX20160915.8763a35...
2623Installing hcoll-3.6.1228...
2624Installing openmpi-1.10.5a1...
2625Installing mpitests-3.2.18...
2626Installing knem-1.1.2.90mlnx1...
2627Installing rds-tools-2.0.7...
2628Installing libdapl2-2.1.9mlnx...
2629Installing dapl2-utils-2.1.9mlnx...
2630Installing libdapl-dev-2.1.9mlnx...
2631Installing srptools-1.0.3...
2632Installing mlnx-ethtool-4.2...
2633Installing libsdp1-1.1.108...
2634Installing libsdp-dev-1.1.108...
2635Installing sdpnetstat-1.60...
2636Selecting previously unselected package mlnx-fw-updater.
2637(Reading database ... 70592 files and directories currently installed.)
2638Preparing to unpack .../mlnx-fw-updater_3.4-1.0.0.0_amd64.deb ...
2639Unpacking mlnx-fw-updater (3.4-1.0.0.0) ...
2640Setting up mlnx-fw-updater (3.4-1.0.0.0) ...
2641
2642Added RUN_FW_UPDATER_ONBOOT=no to /etc/infiniband/openib.conf
2643
2644Attempting to perform Firmware update...
2645Querying Mellanox devices firmware ...
2646
2647Device #1:
2648
2649  Device Type:      ConnectX4
2650  Part Number:      MCX416A-CCA_Ax
2651  Description:      ConnectX-4 EN network interface card; 100GbE dual-port QSFP28; PCIe3.0 x16; ROHS R6
2652  PSID:             MT_2150110033
2653  PCI Device Name:  03:00.0
2654  Base GUID:        248a07030014fc60
2655  Base MAC:         0000248a0714fc60
2656  Versions:         Current        Available     
2657     FW             12.16.1006     12.17.1010    
2658     PXE            3.4.0812       3.4.0903      
2659
2660  Status:           Update required
2661
2662
2663Found 1 device(s) requiring firmware update...
2664
2665Device #1: Updating FW ... Done
2666
2667Restart needed for updates to take effect.
2668Log File: /tmp/MLNX_OFED_LINUX.16084.logs/fw_update.log
2669Please reboot your system for the changes to take effect.
2670Device (03:00.0):
2671        03:00.0 Ethernet controller: Mellanox Technologies MT27620 Family
2672        Link Width: x16
2673        PCI Link Speed: 8GT/s
2674
2675Device (03:00.1):
2676        03:00.1 Ethernet controller: Mellanox Technologies MT27620 Family
2677        Link Width: x16
2678        PCI Link Speed: 8GT/s
2679
2680Installation passed successfully
2681To load the new driver, run:
2682/etc/init.d/openibd restart
2683-----
2684
2685
2686.Reboot 
2687[source,bash]
2688----
2689$sudo  reboot 
2690----
2691
2692
2693.After reboot 
2694[source,bash]
2695----
2696$ibv_devinfo 
2697hca_id: mlx5_1
2698        transport:                      InfiniBand (0)
2699        fw_ver:                         12.17.1010             << 12.17.00
2700        node_guid:                      248a:0703:0014:fc61
2701        sys_image_guid:                 248a:0703:0014:fc60
2702        vendor_id:                      0x02c9
2703        vendor_part_id:                 4115
2704        hw_ver:                         0x0
2705        board_id:                       MT_2150110033
2706        phys_port_cnt:                  1
2707        Device ports:
2708                port:   1
2709                        state:                  PORT_DOWN (1)
2710                        max_mtu:                4096 (5)
2711                        active_mtu:             1024 (3)
2712                        sm_lid:                 0
2713                        port_lid:               0
2714                        port_lmc:               0x00
2715                        link_layer:             Ethernet
2716
2717hca_id: mlx5_0
2718        transport:                      InfiniBand (0)
2719        fw_ver:                         12.17.1010
2720        node_guid:                      248a:0703:0014:fc60
2721        sys_image_guid:                 248a:0703:0014:fc60
2722        vendor_id:                      0x02c9
2723        vendor_part_id:                 4115
2724        hw_ver:                         0x0
2725        board_id:                       MT_2150110033
2726        phys_port_cnt:                  1
2727        Device ports:
2728                port:   1
2729                        state:                  PORT_DOWN (1)
2730                        max_mtu:                4096 (5)
2731                        active_mtu:             1024 (3)
2732                        sm_lid:                 0
2733                        port_lid:               0
2734                        port_lmc:               0x00
2735                        link_layer:             Ethernet
2736
2737----
2738
2739.ibdev2netdev
2740[source,bash]
2741-----
2742$ibdev2netdev
2743mlx5_0 port 1 ==> eth6 (Down)
2744mlx5_1 port 1 ==> eth7 (Down)
2745-----
2746
2747.find the ports 
2748[source,bash]
2749-----
2750
2751        $sudo ./dpdk_setup_ports.py -t
2752  +----+------+---------++---------------------------------------------
2753  | ID | NUMA |   PCI   ||                      Name     |  Driver   | 
2754  +====+======+=========++===============================+===========+=
2755  | 0  | 0    | 06:00.0 || VIC Ethernet NIC              | enic      | 
2756  +----+------+---------++-------------------------------+-----------+-
2757  | 1  | 0    | 07:00.0 || VIC Ethernet NIC              | enic      | 
2758  +----+------+---------++-------------------------------+-----------+-
2759  | 2  | 0    | 0a:00.0 || 82599ES 10-Gigabit SFI/SFP+ Ne| ixgbe     | 
2760  +----+------+---------++-------------------------------+-----------+-
2761  | 3  | 0    | 0a:00.1 || 82599ES 10-Gigabit SFI/SFP+ Ne| ixgbe     | 
2762  +----+------+---------++-------------------------------+-----------+-
2763  | 4  | 0    | 0d:00.0 || Device 15d0                   |           | 
2764  +----+------+---------++-------------------------------+-----------+-
2765  | 5  | 0    | 10:00.0 || I350 Gigabit Network Connectio| igb       | 
2766  +----+------+---------++-------------------------------+-----------+-
2767  | 6  | 0    | 10:00.1 || I350 Gigabit Network Connectio| igb       | 
2768  +----+------+---------++-------------------------------+-----------+-
2769  | 7  | 1    | 85:00.0 || 82599ES 10-Gigabit SFI/SFP+ Ne| ixgbe     | 
2770  +----+------+---------++-------------------------------+-----------+-
2771  | 8  | 1    | 85:00.1 || 82599ES 10-Gigabit SFI/SFP+ Ne| ixgbe     | 
2772  +----+------+---------++-------------------------------+-----------+-
2773  | 9  | 1    | 87:00.0 || MT27700 Family [ConnectX-4]   | mlx5_core |  #<1>
2774  +----+------+---------++-------------------------------+-----------+-
2775  | 10 | 1    | 87:00.1 || MT27700 Family [ConnectX-4]   | mlx5_core |  #<2>
2776  +----+------+---------++---------------------------------------------
2777-----
2778<1>  ConnectX-4 port 0
2779<2>  ConnectX-4 port 1
2780
2781
2782.Config file example
2783[source,bash]
2784-----
2785### Config file generated by dpdk_setup_ports.py ###
2786
2787 - port_limit: 2
2788   version: 2
2789   interfaces: ['87:00.0', '87:00.1']
2790   port_info:
2791      - ip: 1.1.1.1
2792        default_gw: 2.2.2.2
2793      - ip: 2.2.2.2
2794        default_gw: 1.1.1.1
2795
2796   platform:
2797      master_thread_id: 0
2798      latency_thread_id: 1
2799      dual_if:
2800        - socket: 1
2801          threads: [8,9,10,11,12,13,14,15,24,25,26,27,28,29,30,31]
2802-----
2803
2804
2805
2806
2807==== TRex specific implementation details 
2808
2809TRex uses flow director filter to steer specific packets to specific queues. 
2810To support that, we change IPv4.TOS/Ipv6.TC  LSB to *1* for packets we want to handle by software (Other packets will be dropped). So latency packets will have this bit turned on (This is true for all NIC types, not only for ConnectX-4).
2811This means taht if the DUT for some reason clears this bit (change TOS LSB to 0, e.g. change it from 0x3 to 0x2 for example) some TRex features (latency measurement for example) will not work properly.
2812
2813==== Which NIC to buy?
2814
2815NIC with two ports will work better from performance prospective, so it is better to have MCX456A-ECAT(dual 100gb ports) and *not* the  MCX455A-ECAT (single 100gb port).
2816
2817==== Limitation/Issues  
2818
2819* Stateless mode ``per stream statistics'' feature is handled in software (No hardware support like in X710 card).
2820* link:https://trex-tgn.cisco.com/youtrack/issue/trex-261[Latency issue]
2821* link:https://trex-tgn.cisco.com/youtrack/issue/trex-262[Statful RX out of order]
2822* link:https://trex-tgn.cisco.com/youtrack/issue/trex-273[Fedora 21 & OFED 3.4.1]
2823
2824
2825==== Performance Cycles/Packet ConnectX-4 vs Intel XL710
2826
2827For TRex version v2.11, these are the comparison results between XL710 and ConnectX-4 for various scenarios.
2828
2829.Stateless MPPS/Core [Preliminary]
2830image:images/xl710_vs_mlx5_64b.png[title="Stateless 64B"] 
2831
2832.Stateless Gb/Core [Preliminary]
2833image:images/xl710_vs_mlx5_var_size.png[title="Stateless variable size packet"] 
2834
2835*Comments*::
2836 
28371. MLX5 can reach ~50MPPS while XL710 is limited to 35MPPS. (With potential future fix it will be ~65MPPS)
28382. For Stateless/Stateful 256B profiles, ConnectX-4 uses half of the CPU cycles per packet. ConnectX-4 probably can handle in a better way chained mbufs (scatter gather).
28393. In the average stateful scenario, ConnectX-4 is the same as XL710.
28404. For Stateless 64B/IMIX profiles, ConnectX-4 uses 50-90% more CPU cycles per packet (it is actually even more because there is the TRex scheduler overhead) - it means that in worst case scenario, you will need x2 CPU for the same total MPPS.
2841
2842
2843[NOTE]
2844=====================================
2845There is a task to automate the production of thess reports
2846=====================================
2847
2848==== Troubleshooting
2849
2850* Before running TRex make sure the commands    `ibv_devinfo` and  `ibdev2netdev` present the NICS
2851* `ifconfig` should work too, you need to be able to ping from those ports
2852* run TRex server with '-v 7' for example `$sudo ./t-rex-64 -i -v 7`
2853* In case the link_layer is not set to *Ethernet* you should run this command
2854
2855[source,bash]
2856----
2857$sudo mlxconfig -d /dev/mst/mt4115_pciconf0 set LINK_TYPE_P1=2 LINK_TYPE_P2=2
2858$sudo mlxconfig -d /dev/mst/mt4115_pciconf1 set LINK_TYPE_P1=2 LINK_TYPE_P2=2
2859----
2860
2861see link:https://community.mellanox.com/docs/DOC-2294[ConnectX-4 getting Started]
2862
2863* It is possible to change the link speed  (e.g. 50Gb/40Gb/25Gb) see link:https://community.mellanox.com/docs/DOC-2334[change speed]
2864
2865for example to change to 50Gb speed
2866
2867[source,bash]
2868----
2869$sudo ethtool -s enp135s0f1 speed 50000 autoneg off
2870----
2871
2872
2873 
2874
2875
2876==== Limitations/Issues
2877
2878* The order of the mlx5 PCI addrees in /etc/trex_cfg.yaml should be in the same order reported by `./dpdk_setup_ports.py` tool see link:https://groups.google.com/forum/#!searchin/trex-tgn/unable$20to$20run$20rx-check$20with$204$20port%7Csort:relevance/trex-tgn/DsORbw3AbaU/IT-KLcZbDgAJ[issue_thread] and link:https://trex-tgn.cisco.com/youtrack/issue/trex-295[trex-295] else there would be reported this error.
2879
2880.Will work
2881[source,bash]
2882----
2883 - version         : 2
2884   interfaces      : ["03:00.0","03:00.1"]
2885   port_limit      : 2 
2886----
2887
2888.Will not work
2889[source,bash]
2890----
2891 - version         : 2
2892   interfaces      : ["03:00.1","03:00.0"]
2893   port_limit      : 2 
2894----
2895
2896.The error 
2897[source,bash]
2898----
2899 PMD: net_mlx5: 0x7ff2dcfcb2c0: flow director mode 0 not supported
2900 EAL: Error - exiting with code: 1
2901   Cause: rte_eth_dev_filter_ctrl: err=-22, port=2
2902----
2903
2904==== Build with native OFED 
2905
2906In some case there is a need to build the dpdk-mlx5 with different OFED (not just 4.0 maybe newer) 
2907to do so run this on native machine 
2908
2909[source,bash]
2910---- 
2911[csi-trex-07]> ./b configure
2912Setting top to                           : /auto/srg-sce-swinfra-usr/emb/users/hhaim/work/depot/asr1k/emb/private/hhaim/bp_sim_git/trex-core 
2913Setting out to                           : /auto/srg-sce-swinfra-usr/emb/users/hhaim/work/depot/asr1k/emb/private/hhaim/bp_sim_git/trex-core/linux_dpdk/build_dpdk 
2914Checking for program 'g++, c++'          : /bin/g++ 
2915Checking for program 'ar'                : /bin/ar 
2916Checking for program 'gcc, cc'           : /bin/gcc 
2917Checking for program 'ar'                : /bin/ar 
2918Checking for program 'ldd'               : /bin/ldd 
2919Checking for library z                   : yes 
2920Checking for OFED                        : Found needed version 4.0   #<1> 
2921Checking for library ibverbs             : yes 
2922'configure' finished successfully (1.826s)
2923---- 
2924<1> make sure it was identify 
2925
2926
2927.Code change need for new OFED 
2928[source,python]
2929---- 
2930        index fba7540..a55fe6b 100755
2931        --- a/linux_dpdk/ws_main.py
2932        +++ b/linux_dpdk/ws_main.py
2933        @@ -143,8 +143,11 @@ def missing_pkg_msg(fedora, ubuntu):
2934         def check_ofed(ctx):
2935             ctx.start_msg('Checking for OFED')
2936             ofed_info='/usr/bin/ofed_info'
2937        -    ofed_ver= '-3.4-'
2938        -    ofed_ver_show= 'v3.4'
2939        +
2940        +    ofed_ver_re = re.compile('.*[-](\d)[.](\d)[-].*')
2941        +
2942        +    ofed_ver= 40                                     <1>
2943        +    ofed_ver_show= '4.0'
2944        
2945        
2946        --- a/scripts/dpdk_setup_ports.py
2947        +++ b/scripts/dpdk_setup_ports.py
2948        @@ -366,8 +366,8 @@ Other network devices
2949         
2950                 ofed_ver_re = re.compile('.*[-](\d)[.](\d)[-].*')
2951         
2952        -        ofed_ver= 34
2953        -        ofed_ver_show= '3.4-1'
2954        +        ofed_ver= 40                                 <2>
2955        +        ofed_ver_show= '4.0'
2956---- 
2957<1> change to new version 
2958<2> change to new version 
2959 
2960
2961
2962=== Cisco VIC support 
2963
2964anchor:ciscovic_support[]
2965
2966* Supported from TRex version v2.12.
2967* Since version 2.21, all VIC card types supported by DPDK are supported by TRex, using ``--software'' command line argument.
2968Notice that if using ``--software'', no HW assist is used, causing supported packet rate to be much lower.
2969Since we do not have all cards in our lab, we could not test all of them. Will be glad for feedback on this (good or bad).
2970
2971* If not using ``--software'', following limitations apply:
2972** Only 1300 series Cisco adapter supported.
2973** Must have VIC firmware version 2.0(13) for UCS C-series servers. Will be GA in Febuary 2017.
2974** Must have VIC firmware version 3.1(2) for blade servers (which supports more filtering capabilities).
2975** The feature can be enabled via Cisco CIMC or USCM with the 'advanced filters' radio button.  When enabled, these additional flow director modes are available:
2976        RTE_ETH_FLOW_NONFRAG_IPV4_OTHER
2977        RTE_ETH_FLOW_NONFRAG_IPV4_SCTP
2978        RTE_ETH_FLOW_NONFRAG_IPV6_UDP
2979        RTE_ETH_FLOW_NONFRAG_IPV6_TCP
2980        RTE_ETH_FLOW_NONFRAG_IPV6_SCTP
2981        RTE_ETH_FLOW_NONFRAG_IPV6_OTHER
2982
2983==== vNIC Configuration Parameters
2984
2985*Number of Queues*::
2986  The maximum number of receive queues (RQs), work queues (WQs) and completion queues (CQs) are configurable on a per vNIC basis through the Cisco UCS Manager (CIMC or UCSM).
2987  These values should be configured as follows:
2988  * The number of WQs should be greater or equal to the number of threads (-c value) plus 1
2989  * The number of RQs should be greater than 5
2990  * The number of CQs should set to WQs + RQs 
2991  * Unless there is a lack of resources due to creating many vNICs, it is recommended that the WQ and RQ sizes be set to the *maximum*. 
2992
2993*Advanced filters*::
2994  advanced filter should be enabled 
2995
2996*MTU*::
2997  set the MTU to maximum 9000-9190 (Depends on the FW version)
2998  
2999more information could be found here link:http://www.dpdk.org/doc/guides/nics/enic.html?highlight=enic[enic DPDK]
3000 
3001image:images/UCS-B-adapter_policy_1.jpg[title="vic configuration",align="center",width=800]
3002image:images/UCS-B-adapter_policy_2.jpg[title="vic configuration",align="center",width=800]
3003
3004In case it is not configured correctly, this error will be seen
3005
3006.VIC error in case of wrong RQ/WQ 
3007[source,bash]
3008----
3009Starting  TRex v2.15 please wait  ...
3010no client generator pool configured, using default pool
3011no server generator pool configured, using default pool
3012zmq publisher at: tcp://*:4500
3013Number of ports found: 2
3014set driver name rte_enic_pmd
3015EAL: Error - exiting with code: 1
3016  Cause: Cannot configure device: err=-22, port=0     #<1>
3017----
3018<1>There is not enough queues 
3019
3020
3021running it with verbose mode with CLI  `-v 7`  
3022
3023[source,bash]
3024----
3025$sudo ./t-rex-64 -f cap2/dns.yaml -c 1 -m 1 -d 10  -l 1000 -v 7
3026----
3027
3028will give move info 
3029
3030[source,bash]
3031----
3032EAL:   probe driver: 1137:43 rte_enic_pmd
3033PMD: rte_enic_pmd: Advanced Filters available
3034PMD: rte_enic_pmd: vNIC MAC addr 00:25:b5:99:00:4c wq/rq 256/512 mtu 1500, max mtu:9190
3035PMD: rte_enic_pmd: vNIC csum tx/rx yes/yes rss yes intr mode any type min 
3036PMD: rte_enic_pmd: vNIC resources avail: wq 2 rq 2 cq 4 intr 6                  #<1>
3037EAL: PCI device 0000:0f:00.0 on NUMA socket 0
3038EAL:   probe driver: 1137:43 rte_enic_pmd
3039PMD: rte_enic_pmd: Advanced Filters available
3040PMD: rte_enic_pmd: vNIC MAC addr 00:25:b5:99:00:5c wq/rq 256/512 mtu 1500, max 
3041----
3042<1> rq is 2 which mean 1 input queue which is less than minimum required by trex (rq should be at least 5)
3043
3044
3045==== Limitations/Issues
3046
3047* Stateless mode ``per stream statistics'' feature is handled in software (No hardware support like in X710 card).
3048* link:https://trex-tgn.cisco.com/youtrack/issue/trex-272[QSFP+ issue]
3049* VLAN 0 Priority Tagging
3050If a vNIC is configured in TRUNK mode by the UCS manager, the adapter will priority tag egress packets according to 802.1Q if they were not already VLAN tagged by software. If the adapter is connected to a properly configured switch, there will be no unexpected behavior.
3051In test setups where an Ethernet port of a Cisco adapter in TRUNK mode is connected point-to-point to another adapter port or connected though a router instead of a switch, all ingress packets will be VLAN tagged. TRex can work with that see more link:http://www.cisco.com/c/en/us/support/docs/servers-unified-computing/ucs-c-series-rack-servers/117637-technote-UCS-00.html[upstream VIC]
3052Upstream the VIC always tags packets with an 802.1p header.In downstream it is possible to remove the tag (not supported by TRex yet)
3053
3054
3055=== More active flows   
3056
3057From version v2.13 there is a new Stateful scheduler that works better in the case of high concurrent/active flows.
3058In case of EMIX 70% better performance was observed.
3059In this tutorial there are 14 DP cores & up to 8M flows.  
3060There is a  special config file to enlarge the number of flows. This tutorial  present the difference in performance between the old scheduler and the new. 
3061
3062==== Setup details
3063
3064[cols="1,5"]
3065|=================
3066| Server: | UCSC-C240-M4SX
3067| CPU:    | 2 x Intel(R) Xeon(R) CPU E5-2667 v3 @ 3.20GHz
3068| RAM:    | 65536 @ 2133 MHz
3069| NICs:   | 2 x Intel Corporation Ethernet Controller XL710 for 40GbE QSFP+ (rev 01)
3070| QSFP:   | Cisco QSFP-H40G-AOC1M
3071| OS:     | Fedora 18
3072| Switch: | Cisco Nexus 3172 Chassis, System version: 6.0(2)U5(2).
3073| TRex:   | v2.13/v2.12 using 7 cores per dual interface.
3074|=================
3075
3076==== Traffic profile 
3077
3078.cap2/cur_flow_single.yaml
3079[source,python]
3080----
3081- duration : 0.1
3082  generator :  
3083          distribution : "seq"
3084          clients_start : "16.0.0.1"
3085          clients_end   : "16.0.0.255"
3086          servers_start : "48.0.0.1"
3087          servers_end   : "48.0.255.255"
3088          clients_per_gb : 201
3089          min_clients    : 101
3090          dual_port_mask : "1.0.0.0" 
3091  cap_info : 
3092     - name: cap2/udp_10_pkts.pcap  <1>
3093       cps : 100
3094       ipg : 200
3095       rtt : 200
3096       w   : 1
3097----
3098<1> One directional UDP flow with 10 packets of 64B
3099
3100
3101==== Config file command 
3102
3103./cfg/trex_08_5mflows.yaml
3104[source,python]
3105----
3106- port_limit: 4
3107  version: 2
3108  interfaces: ['05:00.0', '05:00.1', '84:00.0', '84:00.1']
3109  port_info:
3110      - ip: 1.1.1.1
3111        default_gw: 2.2.2.2
3112      - ip: 3.3.3.3
3113        default_gw: 4.4.4.4
3114
3115      - ip: 4.4.4.4
3116        default_gw: 3.3.3.3
3117      - ip: 2.2.2.2
3118        default_gw: 1.1.1.1
3119
3120  platform:
3121      master_thread_id: 0
3122      latency_thread_id: 15
3123      dual_if:
3124        - socket: 0
3125          threads: [1,2,3,4,5,6,7]
3126
3127        - socket: 1
3128          threads: [8,9,10,11,12,13,14]
3129  memory    :                                          
3130        dp_flows    : 1048576                    <1>
3131----
3132<1> add memory section with more flows 
3133
3134==== Traffic command 
3135
3136.command 
3137[source,bash]
3138----
3139$sudo ./t-rex-64 -f cap2/cur_flow_single.yaml -m 30000 -c 7 -d 40 -l 1000 --active-flows 5000000 -p --cfg cfg/trex_08_5mflows.yaml
3140----
3141
3142The number of active flows can be change using `--active-flows` CLI. in this example it is set to 5M flows
3143
3144
3145==== Script to get performance per active number of flows 
3146
3147[source,python]
3148----
3149
3150def minimal_stateful_test(server,csv_file,a_active_flows):
3151
3152    trex_client = CTRexClient(server)                                   <1>
3153
3154    trex_client.start_trex(                                             <2>
3155            c = 7,
3156            m = 30000,
3157            f = 'cap2/cur_flow_single.yaml',
3158            d = 30,
3159            l = 1000,
3160            p=True,
3161            cfg = "cfg/trex_08_5mflows.yaml",
3162            active_flows=a_active_flows,
3163            nc=True
3164            )
3165
3166    result = trex_client.sample_to_run_finish()                         <3>
3167
3168    active_flows=result.get_value_list('trex-global.data.m_active_flows')
3169    cpu_utl=result.get_value_list('trex-global.data.m_cpu_util')
3170    pps=result.get_value_list('trex-global.data.m_tx_pps')
3171    queue_full=result.get_value_list('trex-global.data.m_total_queue_full')
3172    if queue_full[-1]>10000:
3173        print("WARNING QUEU WAS FULL");
3174    tuple=(active_flows[-5],cpu_utl[-5],pps[-5],queue_full[-1])         <4>
3175    file_writer = csv.writer(test_file)
3176    file_writer.writerow(tuple);
3177    
3178
3179
3180if __name__ == '__main__':
3181    test_file = open('tw_2_layers.csv', 'wb');
3182    parser = argparse.ArgumentParser(description="Example for TRex Stateful, assuming server daemon is running.")
3183
3184    parser.add_argument('-s', '--server',
3185                        dest='server',
3186                        help='Remote trex address',
3187                        default='127.0.0.1',
3188                        type = str)
3189    args = parser.parse_args()
3190
3191    max_flows=8000000;
3192    min_flows=100;
3193    active_flow=min_flows;
3194    num_point=10
3195    factor=math.exp(math.log(max_flows/min_flows,math.e)/num_point);
3196    for i in range(num_point+1):
3197        print("=====================",i,math.floor(active_flow))
3198        minimal_stateful_test(args.server,test_file,math.floor(active_flow))      
3199        active_flow=active_flow*factor
3200
3201    test_file.close();
3202----
3203<1> connect 
3204<2> Start with different active_flows
3205<3> wait for the results 
3206<4> get the results and save to csv file
3207
3208This script iterate between 100 to 8M active flows and save the results to csv file.
3209
3210==== The results v2.12 vs v2.14
3211
3212.MPPS/core 
3213image:images/tw1_0.png[title="results",align="center"]
3214
3215.MPPS/core 
3216image:images/tw0_0_chart.png[title="results",align="center",width=800]
3217
3218* TW0 - v2.14 default configuration 
3219* PQ  - v2.12 default configuration 
3220
3221* To run the same script on v2.12 (that does not support `active_flows` directive) a patch was introduced.
3222
3223*Observation*::
3224  * TW works better (up to 250%) in case of 25-100K flows 
3225  * TW scale better with active-flows
3226
3227==== Tunning 
3228
3229let's add another modes called *TW1*, in this mode the scheduler is tune to have more buckets (more memory)
3230
3231.TW1 cap2/cur_flow_single_tw_8.yaml
3232[source,python]
3233----
3234- duration : 0.1
3235  generator :  
3236          distribution : "seq"
3237          clients_start : "16.0.0.1"
3238          clients_end   : "16.0.0.255"
3239          servers_start : "48.0.0.1"
3240          servers_end   : "48.0.255.255"
3241          clients_per_gb : 201
3242          min_clients    : 101
3243          dual_port_mask : "1.0.0.0" 
3244  tw :                            
3245     buckets : 16384                    <1>
3246     levels  : 2                        <2>
3247     bucket_time_usec : 20.0
3248  cap_info : 
3249     - name: cap2/udp_10_pkts.pcap
3250       cps : 100
3251       ipg : 200
3252       rtt : 200
3253       w   : 1
3254----
3255<1> more buckets  
3256<2> less levels 
3257
3258
3259in *TW2* mode we have the same template, duplicated one with short IPG and another one with high IPG 
326010% of the new flows will be with long IPG
3261
3262.TW2 cap2/cur_flow.yaml
3263[source,python]
3264----
3265- duration : 0.1
3266  generator :  
3267          distribution : "seq"
3268          clients_start : "16.0.0.1"
3269          clients_end   : "16.0.0.255"
3270          servers_start : "48.0.0.1"
3271          servers_end   : "48.0.255.255"
3272          clients_per_gb : 201
3273          min_clients    : 101
3274          dual_port_mask : "1.0.0.0" 
3275          tcp_aging      : 0
3276          udp_aging      : 0
3277  mac        : [0x0,0x0,0x0,0x1,0x0,0x00]
3278  #cap_ipg    : true
3279  cap_info : 
3280     - name: cap2/udp_10_pkts.pcap
3281       cps : 10
3282       ipg : 100000
3283       rtt : 100000
3284       w   : 1
3285     - name: cap2/udp_10_pkts.pcap   
3286       cps : 90
3287       ipg : 2
3288       rtt : 2
3289       w   : 1
3290----
3291
3292==== Full results 
3293
3294
3295* PQ - v2.12 default configuration 
3296* TW0 - v2.14 default configuration 
3297* TW1 - v2.14 more buckets 16K
3298* TW2 - v2.14 two templates 
3299
3300.MPPS/core Comparison
3301image:images/tw1.png[title="results",align="center",width=800]
3302
3303.MPPS/core
3304image:images/tw1_tbl.png[title="results",align="center"]
3305
3306.Factor relative to v2.12 results 
3307image:images/tw2.png[title="results",align="center",width=800]
3308
3309.Extrapolation Total GbE per UCS with average packet size of 600B 
3310image:images/tw3.png[title="results",align="center",width=800]
3311
3312Observation:
3313
3314* TW2 (two flows) almost does not have a performance impact
3315* TW1 (more buckets) improve the performance up to a point
3316* TW is general is better than PQ
3317
3318
3319
3320