trex_book.asciidoc revision 4f6782cd
1TRex 
2====
3:author: hhaim 
4:email: <hhaim@cisco.com> 
5:revnumber: 2.1
6:quotes.++:
7:numbered:
8:web_server_url: http://trex-tgn.cisco.com/trex
9:local_web_server_url: csi-wiki-01:8181/trex
10:toclevels: 4
11
12include::trex_ga.asciidoc[]
13
14
15== Introduction
16
17=== A word on traffic generators
18
19Traditionally, routers have been tested using commercial traffic generators, while performance
20typically has been measured using packets per second (PPS) metrics. As router functionality and
21services became more complex, stateful traffic generators now need to provide more realistic traffic scenarios.
22
23Advantages of realistic traffic generators:
24
25* Accurate performance metrics.
26* Discovering bottlenecks in realistic traffic scenarios.
27
28==== Current Challenges:
29
30* *Cost*: Commercial stateful traffic generators are very expensive.
31* *Scale*: Bandwidth does not scale up well with feature complexity.
32* *Standardization*: Lack of standardization of traffic patterns and methodologies.
33* *Flexibility*: Commercial tools do not allow agility when flexibility and changes are needed.
34
35==== Implications 
36
37* High capital expenditure (capex) spent by different teams.
38* Testing in low scale and extrapolation became a common practice. This is non-ideal and fails to indicate bottlenecks that appear in real-world scenarios.
39* Teams use different benchmark methodologies, so results are not standardized.
40* Delays in development and testing due to dependence on testing tool features.
41* Resource and effort investment in developing different ad hoc tools and test methodologies.
42
43=== Overview of TRex
44
45TRex addresses these problems through an innovative and extendable software implementation and by leveraging standard and open software and x86/UCS hardware.
46
47* Generates and analyzes L4-7 traffic. In one package, provides capabilities of commercial L7 tools.
48* Stateful traffic generator based on pre-processing and smart replay of real traffic templates.
49* Generates and *amplifies* both client and server side traffic.
50* Customized functionality can be added.
51* Scales to 200Gb/sec for one UCS (using Intel 40Gb/sec NICs).
52* Low cost.
53* Self-contained package that can be easily installed and deployed.
54* Virtual interface support enables TRex to be used in a fully virtual environment without physical NICs. Example use cases:
55** Amazon AWS
56** Cisco LaaS
57// Which LaaS is this? Location as a service? Linux?
58** TRex on your laptop
59
60
61
62.TRex Hardware 
63[options="header",cols="1^,1^"]
64|=================
65|Cisco UCS Platform | Intel NIC 
66| image:images/ucs200_2.png[title="generator"] | image:images/Intel520.png[title="generator"]  
67|=================
68
69=== Purpose of this guide
70
71This guide explains the use of TRex internals and the use of TRex together with Cisco ASR1000 Series routers. The examples illustrate novel traffic generation techniques made possible by TRex. 
72
73== Download and installation
74
75=== Hardware recommendations
76
77TRex operates in a Linux application environment, interacting with Linux kernel modules. 
78TRex curretly works on x86 architecture and can operate well on Cisco UCS hardware. The following platforms have been tested and are recommended for operating TRex. 
79
80[NOTE]
81=====================================
82 A high-end UCS platform is not required for operating TRex in its current version, but may be required for future versions.
83=====================================
84
85[NOTE]
86=====================================
87 Not all supported DPDK interfaces are supported by TRex
88=====================================
89 
90
91.Preferred UCS hardware
92[options="header",cols="1,3"]
93|=================
94| UCS Type | Comments 
95| UCS C220 M3/M4  | *Preferred Low-End*. Supports up to 40Gb/sec with 540-D2. With newer Intel NIC (recommended), supports 80Gb/sec with 1RU. See table below describing components.
96| UCS C200| Early UCS model.
97| UCS C210 M2 | Supports up to 40Gb/sec PCIe3.0.
98| UCS C240 M3/M4 | *Preferred, High-End* Supports up to 200Gb/sec. 6x XL710 NICS (PCIex8) or 2xFM10K (PCIex16). See table below describing components.
99| UCS C260M2 | Supports up to 30Gb/sec (limited by V2 PCIe).
100|=================
101
102.Low-End UCS C220 M4 - Internal components
103[options="header",cols="1,2",width="60%"]
104|=================
105| Components |  Details 
106| CPU  | 2x E5-2620 @ 2.0 GHz. 
107| CPU Configuration | 2-Socket CPU configurations (also works with 1 CPU).
108| Memory | 2x4 banks f.or each CPU. Total of 32GB in 8 banks.	 
109| RAID | No RAID.
110|=================
111
112.High-End C240 M4 - Internal components 
113[options="header",cols="1,2",width="60%"]
114|=================
115| Components |  Details 
116| CPU  | 2x E5-2667 @ 3.20 GHz.
117| PCIe | 1x Riser PCI expansion card option A PID UCSC-PCI-1A-240M4 enables 2 PCIex16.
118| CPU Configuration | 2-Socket CPU configurations (also works with 1 CPU).
119| Memory | 2x4 banks for each CPU. Total of 32GB in 8 banks. 
120| RAID | No RAID.
121| Riser 1/2 | both left and right should support x16 PCIe. Right (Riser1) should be from option A x16 and Left (Riser2) should be x16. need to order both  
122|=================
123 
124.Supported NICs
125[options="header",cols="1,1,4",width="90%"]
126|=================
127| Chipset              | Bandwidth  (Gb/sec)  |  Example
128| Intel I350           | 1   | Intel 4x1GE 350-T4 NIC
129| Intel 82599          | 10  |   Cisco part ID:N2XX-AIPCI01 Intel x520-D2, Intel X520 Dual Port 10Gb SFP+ Adapter
130| Intel X710           | 10  | Cisco part ID:UCSC-PCIE-IQ10GF link:https://en.wikipedia.org/wiki/Small_form-factor_pluggable_transceiver[SFP+], *Preferred* support per stream stats in hardware link:http://www.silicom-usa.com/PE310G4i71L_Quad_Port_Fiber_SFP+_10_Gigabit_Ethernet_PCI_Express_Server_Adapter_49[Silicom PE310G4i71L]
131| Intel XL710          | 40  | Cisco part ID:UCSC-PCIE-ID40GF, link:https://en.wikipedia.org/wiki/QSFP[QSFP+] (copper/optical)
132| Intel FM10420        | 25/100 | QSFP28,  by Silicom link:http://www.silicom-usa.com/100_Gigabit_Dual_Port_Fiber_Ethernet_PCI_Express_PE3100G2DQiR_96[Silicom PE3100G2DQiR_96] (*in development*) 
133| Mellanox ConnectX-4  | 25/40/50/56/100 | QSFP28, link:http://www.mellanox.com/page/products_dyn?product_family=201&[ConnectX-4] link:http://www.mellanox.com/related-docs/prod_adapter_cards/PB_ConnectX-4_VPI_Card.pdf[ConnectX-4-brief] (copper/optical) supported from v2.11 more details xref:connectx_support[TRex Support]
134| Mellanox ConnectX-5  | 25/40/50/56/100 | Not supported yet 
135| Cisco 1300 series    | 40              | QSFP+, VIC 1380,  VIC 1385, VIC 1387 see more xref:ciscovic_support[TRex Support]
136| VMXNET / +
137VMXNET3 (see notes) | VMware paravirtualized  | Connect using VMware vSwitch
138| E1000    | paravirtualized  | VMware/KVM/VirtualBox 
139| Virtio | paravirtualized  | KVM
140|=================
141
142// in table above, is it correct to list "paravirtualized" as chipset? Also, what is QSFP28? It does not appear on the lined URL. Clarify: is Intel X710 the preferred NIC?
143
144.SFP+ support 
145[options="header",cols="2,1,1,1",width="90%"]
146|=================
147| link:https://en.wikipedia.org/wiki/Small_form-factor_pluggable_transceiver[SFP+]  | Intel Ethernet Converged X710-DAX |  Silicom link:http://www.silicom-usa.com/PE310G4i71L_Quad_Port_Fiber_SFP+_10_Gigabit_Ethernet_PCI_Express_Server_Adapter_49[PE310G4i71L] (Open optic) | 82599EB 10-Gigabit
148| link:http://www.cisco.com/c/en/us/products/collateral/interfaces-modules/transceiver-modules/data_sheet_c78-455693.html[Cisco SFP-10G-SR] | Does not work     | [green]*works* | [green]*works*
149| link:http://www.cisco.com/c/en/us/products/collateral/interfaces-modules/transceiver-modules/data_sheet_c78-455693.html[Cisco SFP-10G-LR] | Does not work     | [green]*works* | [green]*works*
150| link:http://www.cisco.com/c/en/us/products/collateral/interfaces-modules/transceiver-modules/data_sheet_c78-455693.html[Cisco SFP-H10GB-CU1M]| [green]*works* | [green]*works* | [green]*works*
151| link:http://www.cisco.com/c/en/us/products/collateral/interfaces-modules/transceiver-modules/data_sheet_c78-455693.html[Cisco SFP-10G-AOC1M] | [green]*works* | [green]*works* | [green]*works*
152|=================
153
154[NOTE]
155=====================================
156 Intel X710 NIC (example: FH X710DA4FHBLK) operates *only* with Intel SFP+. For open optic, use the link:http://www.silicom-usa.com/PE310G4i71L_Quad_Port_Fiber_SFP+_10_Gigabit_Ethernet_PCI_Express_Server_Adapter_49[Silicom PE310G4i71L] NIC.
157=====================================
158
159// clarify above table and note
160
161.XL710 NIC base QSFP+ support 
162[options="header",cols="1,1,1",width="90%"]
163|=================
164| link:https://en.wikipedia.org/wiki/QSFP[QSFP+]             |  Intel Ethernet Converged XL710-QDAX | Silicom link:http://www.silicom-usa.com/Dual_Port_Fiber_40_Gigabit_Ethernet_PCI_Express_Server_Adapter_PE340G2Qi71_83[PE340G2Qi71] Open optic 
165| QSFP+ SR4 optics  |  APPROVED OPTICS [green]*works*,  Cisco QSFP-40G-SR4-S does *not* work   | Cisco QSFP-40G-SR4-S [green]*works* 
166| QSFP+ LR-4 Optics |   APPROVED OPTICS [green]*works*, Cisco QSFP-40G-LR4-S does *not* work   | Cisco QSFP-40G-LR4-S [green]*works* 
167| QSFP Active Optical Cables (AoC) | Cisco QSFP-H40G-AOC [green]*works*  | Cisco QSFP-H40G-AOC [green]*works* 
168| QSFP+ Intel Ethernet Modular Optics |    N/A            |  N/A
169| QSFP+ DA twin-ax cables | N/A  | N/A  
170| Active QSFP+ Copper Cables | Cisco QSFP-4SFP10G-CU [green]*works*  | Cisco QSFP-4SFP10G-CU [green]*works*
171|=================
172
173[NOTE]
174=====================================
175 For Intel XL710 NICs, Cisco SR4/LR QSFP+ does not operate. Use Silicom with Open Optic.
176=====================================
177
178
179.ConnectX-4 NIC base QSFP28 support (100gb)
180[options="header",cols="1,2",width="90%"]
181|=================
182| link:https://en.wikipedia.org/wiki/QSFP[QSFP28]             |  ConnectX-4
183| QSFP28 SR4 optics  |  N/A
184| QSFP28 LR-4 Optics |  N/A 
185| QSFP28 (AoC)       | Cisco QSFP-100G-AOCxM  [green]*works*  
186| QSFP28 DA twin-ax cables | Cisco QSFP-100G-CUxM [green]*works*  
187|=================
188
189.Cisco VIC NIC base QSFP+ support 
190[options="header",cols="1,2",width="90%"]
191|=================
192| link:https://en.wikipedia.org/wiki/QSFP[QSFP+]             |  Intel Ethernet Converged XL710-QDAX 
193| QSFP+ SR4 optics  |  N/A
194| QSFP+ LR-4 Optics |   N/A
195| QSFP Active Optical Cables (AoC) | Cisco QSFP-H40G-AOC [green]*works*  
196| QSFP+ Intel Ethernet Modular Optics |    N/A            
197| QSFP+ DA twin-ax cables | N/A  | N/A  
198| Active QSFP+ Copper Cables | N/A
199|=================
200
201
202// clarify above table and note. let's discuss.
203.FM10K QSFP28 support 
204[options="header",cols="1,1",width="70%"]
205|=================
206| QSFP28             | Example 
207| todo  |  todo
208|=================
209
210// do we want to show "todo"? maybe "pending"
211
212
213[IMPORTANT]
214=====================================
215* Intel SFP+ 10Gb/sec is the only one supported by default on the standard Linux driver. TRex also supports Cisco 10Gb/sec SFP+. 
216// above, replace "only one" with "only mode"?
217* For operating high speed throughput (example: several Intel XL710 40Gb/sec), use different link:https://en.wikipedia.org/wiki/Non-uniform_memory_access[NUMA] nodes for different NICs. +
218    To verify NUMA and NIC topology: `lstopo (yum install hwloc)` +
219    To display CPU info, including NUMA node: `lscpu` +
220    NUMA usage xref:numa-example[example]
221* For Intel XL710 NICs, verify that the NVM is v5.04 . xref:xl710-firmware[Info].
222**  `> sudo ./t-rex-64 -f cap2/dns.yaml -d 0 *-v 6* --nc | grep NVM` +
223    `PMD:  FW 5.0 API 1.5 NVM 05.00.04  eetrack 800013fc` 
224=====================================
225
226// above, maybe rename the bullet points "NIC usage notes"? should we create a subsection for NICs? Maybe it would be under "2.1 Hardware recommendations" as a subsection.
227
228
229.Sample order for recommended low-end Cisco UCSC-C220-M3S with 4x10Gb ports
230[options="header",cols="1,1",width="70%"]
231|=================
232| Component | Quantity 
233| UCSC-C220-M3S    |  1
234| UCS-CPU-E5-2650  |  2
235| UCS-MR-1X041RY-A |  8
236| A03-D500GC3      |  1
237| N2XX-AIPCI01     |  2
238| UCSC-PSU-650W    |  1
239| SFS-250V-10A-IS  |  1
240| UCSC-CMA1        |  1
241| UCSC-HS-C220M3   |  2
242| N20-BBLKD        |  7
243| UCSC-PSU-BLKP    |  1
244| UCSC-RAIL1       |  1
245|=================
246
247// should table above say "low-end Cisco UCS C220 M3S" instead of "low-end USCS-C220-M3S"?
248
249NOTE: Purchase the 10Gb/sec SFP+ separately. Cisco would be fine with TRex (but not for plain Linux driver).
250// does note above mean "TRex operates with 10Gb/sec SFP+ components, but plain Linux does not provide drivers."? if so, how does purchasing separately solve this? where do they get drivers?
251
252=== Installing OS 
253
254==== Supported versions
255
256Supported Linux versions:
257
258* Fedora 20-23, 64-bit kernel (not 32-bit)
259* Ubuntu 14.04.1 LTS, 64-bit kernel (not 32-bit)
260* Ubuntu 16.xx LTS, 64-bit kernel (not 32-bit) 
261
262NOTE: Additional OS version may be supported by compiling the necessary drivers.
263
264To check whether a kernel is 64-bit, verify that the ouput of the following command is `x86_64`.
265
266[source,bash]
267----
268$uname -m
269x86_64 
270----
271
272
273==== Download Linux
274
275ISO images for supported Linux releases can be downloaded from:
276
277.Supported Linux ISO image links
278[options="header",cols="1^,2^",width="50%"]
279|======================================
280| Distribution | SHA256 Checksum
281| link:http://archives.fedoraproject.org/pub/archive/fedora/linux/releases/20/Fedora/x86_64/iso/Fedora-20-x86_64-DVD.iso[Fedora 20]
282    | link:http://archives.fedoraproject.org/pub/archive/fedora/linux/releases/20/Fedora/x86_64/iso/Fedora-20-x86_64-CHECKSUM[Fedora 20 CHECKSUM] 
283| link:http://fedora-mirror01.rbc.ru/pub/fedora/linux/releases/21/Server/x86_64/iso/Fedora-Server-DVD-x86_64-21.iso[Fedora 21]
284    | link:http://fedora-mirror01.rbc.ru/pub/fedora/linux/releases/21/Server/x86_64/iso/Fedora-Server-21-x86_64-CHECKSUM[Fedora 21 CHECKSUM] 
285| link:http://old-releases.ubuntu.com/releases/14.04.1/ubuntu-14.04-desktop-amd64.iso[Ubuntu 14.04.1]
286    | http://old-releases.ubuntu.com/releases/14.04.1/SHA256SUMS[Ubuntu 14.04* CHECKSUMs]
287| link:http://releases.ubuntu.com/16.04.1/ubuntu-16.04.1-server-amd64.iso[Ubuntu 16.04.1]
288    | http://releases.ubuntu.com/16.04.1/SHA256SUMS[Ubuntu 16.04* CHECKSUMs]
289
290|======================================
291
292For Fedora downloads...
293
294* Select a mirror close to your location: +
295https://admin.fedoraproject.org/mirrormanager/mirrors/Fedora +
296Choose: "Fedora Linux http" -> releases -> <version number> -> Server -> x86_64 -> iso -> Fedora-Server-DVD-x86_64-<version number>.iso
297
298* Verify the checksum of the downloaded file matches the linked checksum values with the `sha256sum` command. Example:
299
300[source,bash]
301----
302$sha256sum Fedora-18-x86_64-DVD.iso
30391c5f0aca391acf76a047e284144f90d66d3d5f5dcd26b01f368a43236832c03 #<1>
304----
305<1> Should be equal to the link:https://en.wikipedia.org/wiki/SHA-2[SHA-256] values described in the linked checksum files. 
306
307
308==== Install Linux 
309
310Ask your lab admin to install the Linux using CIMC, assign an IP, and set the DNS. Request the sudo or super user password to enable you to ping and SSH.
311
312xref:fedora21_example[Example of installing Fedora 21 Server]
313
314[NOTE]
315=====================================
316 * To use TRex, you should have sudo on the machine or the root password.
317 * Upgrading the linux Kernel using `yum upgrade` requires building the TRex drivers. 
318 * In Ubuntu 16, auto-updater is enabled by default. It's advised to turn it off as with update of Kernel need to compile again the DPDK .ko file. +
319Command to remove it: +
320 > sudo apt-get remove unattended-upgrades
321=====================================
322
323==== Verify Intel NIC installation
324
325Use `lspci` to verify the NIC installation. 
326
327Example 4x 10Gb/sec TRex configuration (see output below):
328
329* I350 management port
330
331* 4x Intel Ethernet Converged Network Adapter model x520-D2 (82599 chipset)
332
333[source,bash]
334----
335$[root@trex]lspci | grep Ethernet
33601:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)                #<1>
33701:00.1 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)                #<2>
33803:00.0 Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01) #<3>
33903:00.1 Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01)
34082:00.0 Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01)
34182:00.1 Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01)
342----
343<1> Management port 
344<2> CIMC port
345<3> 10Gb/sec traffic ports (Intel 82599EB) 
346
347=== Obtaining the TRex package
348
349Connect using `ssh` to the TRex machine and execute the commands described below.
350
351NOTE: Prerequisite: *$WEB_URL* is *{web_server_url}* or *{local_web_server_url}* (Cisco internal)
352
353Latest release:
354[source,bash]
355----
356$mkdir trex
357$cd trex
358$wget --no-cache $WEB_URL/release/latest 
359$tar -xzvf latest 
360----
361
362
363Bleeding edge version:
364[source,bash]
365----
366$wget --no-cache $WEB_URL/release/be_latest 
367----
368
369To obtain a specific version, do the following: 
370[source,bash]
371----
372$wget --no-cache $WEB_URL/release/vX.XX.tar.gz #<1>
373----
374
375<1> X.XX = Version number
376
377== First time Running
378
379=== Configuring for loopback
380
381Before connecting TRex to your DUT, it is strongly advised to verify that TRex and the NICs work correctly in loopback. +
382To get best performance, it is advised to loopback interfaces on the same NUMA (controlled by the same physical processor). If you do not know how to check this, you can ignore this advice for now. +
383
384[NOTE]
385=====================================================================
386If you are using 10Gbs NIC based on Intel 520-D2 NICs, and you loopback ports on the same NIC, using SFP+, it might not sync, and you will fail to get link up. +
387We checked many types of SFP+ (Intel/Cisco/SR/LR) and it worked for us. +
388If you still encounter link issues, you can either try to loopback interfaces from different NICs, or use link:http://www.fiberopticshare.com/tag/cisco-10g-twinax[Cisco twinax copper cable].
389=====================================================================
390
391.Loopback example
392image:images/loopback_example.png[title="Loopback example"] 
393
394==== Identify the ports 
395
396[source,bash]
397----
398 $>sudo ./dpdk_setup_ports.py -s
399
400 Network devices using DPDK-compatible driver
401 ============================================
402
403 Network devices using kernel driver
404 ===================================
405 0000:03:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection' drv= unused=ixgb #<1>
406 0000:03:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection' drv= unused=ixgb
407 0000:13:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection' drv= unused=ixgb
408 0000:13:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection' drv= unused=ixgb
409 0000:02:00.0 '82545EM Gigabit Ethernet Controller (Copper)' if=eth2 drv=e1000 unused=igb_uio *Active* #<2>
410
411 Other network devices
412 =====================
413 <none>
414----
415
416<1> If you did not run any DPDK application, you will see list of interfaces binded to the kernel, or not binded at all.
417<2> Interface marked as 'active' is the one used by your ssh connection. *Never* put it in TRex config file.
418
419Choose ports to use and follow the instructions in the next section to create configuration file.
420
421==== Creating minimum configuration file
422
423Default configuration file name is: `/etc/trex_cfg.yaml`.
424
425You can copy basic configuration file from cfg folder
426
427[source,bash]
428----
429$cp  cfg/simple_cfg.yaml /etc/trex_cfg.yaml
430----
431
432Then, edit the configuration file and put your interface's and IP addresses details.
433
434Example:
435
436[source,bash]
437----
438<none>
439- port_limit      : 2
440  version         : 2
441#List of interfaces. Change to suit your setup. Use ./dpdk_setup_ports.py -s to see available options
442interfaces    : ["03:00.0", "03:00.1"]  #<1>
443 port_info       :  # Port IPs. Change to suit your needs. In case of loopback, you can leave as is.
444          - ip         : 1.1.1.1
445            default_gw : 2.2.2.2
446          - ip         : 2.2.2.2
447            default_gw : 1.1.1.1
448----
449<1> You need to edit this line to match the interfaces you are using.
450Notice that all NICs you are using should have the same type. You cannot mix different NIC types in one config file. For more info, see link:http://trex-tgn.cisco.com/youtrack/issue/trex-201[trex-201].
451
452You can find xref:trex_config[here] full list of configuration file options.
453
454=== Script for creating config file
455
456To help starting with basic configuration file that suits your needs, there a script that can automate this process.
457The script helps you getting started, and you can then edit the file and add advanced options from xref:trex_config[here]
458if needed. +
459There are two ways to run the script. Interactively (script will pormpt you for parameters), or providing all parameters
460using command line options.
461
462==== Interactive mode
463
464[source,bash]
465----
466sudo ./dpdk_setup_ports.py -i
467----
468
469You will see a list of available interfaces with their related information +
470Just follow the instructions to get basic config file.
471
472==== Specifying input arguments using command line options
473
474First, run this command to see the list of all interfaces and their related information:
475
476[source,bash]
477----
478sudo ./dpdk_setup_ports.py -t
479----
480
481* In case of *Loopback* and/or only *L1-L2 Switches* on the way, you do not need to provide IPs or destination MACs. +
482The script Will assume the following interface connections: 0&#8596;1, 2&#8596;3 etc. +
483Just run:
484
485[source,bash]
486----
487sudo ./dpdk_setup_ports.py -c <TRex interface 0> <TRex interface 1> ...
488----
489
490* In case of *Router* (or other next hop device, such as *L3 Switch*), you should specify the TRex IPs and default gateways, or
491MACs of the router as described below.
492
493.Additional arguments to creating script (dpdk_setup_ports.py -c)
494[options="header",cols="2,5,3",width="100%"]
495|=================
496| Arg | Description | Example
497| -c  | Create a configuration file by specified interfaces (PCI address or Linux names: eth1 etc.) | -c 03:00.1 eth1 eth4 84:00.0
498| --dump | Dump created config to screen. |
499| -o | Output the config to this file. | -o /etc/trex_cfg.yaml
500| --dest-macs | Destination MACs to be used per each interface. Specify this option if you want MAC based config instead of IP based one. You must not set it together with --ip and --def_gw | --dest-macs 11:11:11:11:11:11 22:22:22:22:22:22
501| --ip | List of IPs to use for each interface. If this option and --dest-macs is not specified, script assumes loopback connections (0&#8596;1, 2&#8596;3 etc.) | --ip 1.2.3.4 5.6.7.8
502|--def-gw | List of default gateways to use for each interface. If --ip given, you must provide --def_gw as well | --def-gw 3.4.5.6 7.8.9.10
503| --ci | Cores include: White list of cores to use. Make sure there is enough for each NUMA. | --ci 0 2 4 5 6
504| --ce | Cores exclude: Black list of cores to exclude. Make sure there will be enough for each NUMA. | --ci 10 11 12
505| --no-ht | No HyperThreading: Use only one thread of each Core in created config yaml. |
506| --prefix | Advanced option: prefix to be used in TRex config in case of parallel instances. | --prefix first_instance
507| --zmq-pub-port | Advanced option: ZMQ Publisher port to be used in TRex config in case of parallel instances. | --zmq-pub-port 4000
508| --zmq-rpc-port | Advanced option: ZMQ RPC port to be used in TRex config in case of parallel instances. | --zmq-rpc-port
509| --ignore-numa | Advanced option: Ignore NUMAs for config creation. Use this option only if you have to, as it might reduce performance. For example, if you have pair of interfaces at different NUMAs |
510|=================
511
512=== Configuring ESXi for running TRex
513
514To get best performance, it is advised to run TRex on bare metal hardware, and not use any kind of VM.
515Bandwidth on VM might be limited, and IPv6 might not be fully supported.
516Having said that, there are sometimes benefits for running on VM. +
517These include: +
518    * Virtual NICs can be used to bridge between TRex and NICs not supported by TRex. +
519    * If you already have VM installed, and do not require high performance. +
520
5211. Click the host machine, enter Configuration -> Networking.
522
523a. One of the NICs should be connected to the main vSwitch network to get an "outside" connection, for the TRex client and ssh: +
524image:images/vSwitch_main.png[title="vSwitch_main"]
525
526b. Other NICs that are used for TRex traffic should be in distinguish vSwitch: +
527image:images/vSwitch_loopback.png[title="vSwitch_loopback"]
528
5292. Right-click guest machine -> Edit settings -> Ensure the NICs are set to their networks: +
530image:images/vSwitch_networks.png[title="vSwitch_networks"]
531
532[NOTE]
533=====================================================================
534Before version 2.10, the following command did not function as expected:
535[subs="quotes"]
536....
537sudo ./t-rex-64 -f cap2/dns.yaml *--lm 1 --lo* -l 1000 -d 100
538....
539The vSwitch did not "know" where to route the packet. Was solved on version 2.10 when TRex started to support ARP.
540=====================================================================
541
542* Pass-through is the way to use directly the NICs from host machine inside the VM. Has no limitations except the NIC/hardware itself. The only difference via bare-metal OS is occasional spikes of latency (~10ms). Passthrough settings cannot be saved to OVA.
543
5441. Click on the host machine. Enter Configuration -> Advanced settings -> Edit. Mark the desired NICs. Reboot the ESXi to apply. +
545image:images/passthrough_marking.png[title="passthrough_marking"]
546
5472. Right click on guest machine. Edit settings -> Add -> *PCI device* -> Choose the NICs one by one. +
548image:images/passthrough_adding.png[title="passthrough_adding"]
549
550=== Configuring for running with router (or other L3 device) as DUT
551
552You can follow link:trex_config_guide.html[this] presentation for an example of how to configure router as DUT.
553
554=== Running TRex
555
556When all is set, use the following command to start basic TRex run for 10 seconds
557(it will use the default config file name /etc/trex_cfg.yaml):
558[source,bash]
559----
560$sudo ./t-rex-64 -f cap2/dns.yaml -c 4 -m 1 -d 10  -l 1000
561----
562
563If successful, the output will be similar to the following:
564
565[source,python]
566----
567$ sudo ./t-rex-64 -f cap2/dns.yaml -d 10 -l 1000
568Starting  TRex 2.09 please wait  ...
569zmq publisher at: tcp://*:4500 
570 number of ports found : 4
571  port : 0 
572  ------------
573  link         :  link : Link Up - speed 10000 Mbps - full-duplex      <1>
574  promiscuous  : 0 
575  port : 1 
576  ------------
577  link         :  link : Link Up - speed 10000 Mbps - full-duplex
578  promiscuous  : 0 
579  port : 2 
580  ------------
581  link         :  link : Link Up - speed 10000 Mbps - full-duplex
582  promiscuous  : 0 
583  port : 3 
584  ------------
585  link         :  link : Link Up - speed 10000 Mbps - full-duplex
586  promiscuous  : 0 
587
588
589 -Per port stats table 
590      ports |               0 |               1 |               2 |               3 
591 -------------------------------------------------------------------------------------
592   opackets |            1003 |            1003 |            1002 |            1002 
593     obytes |           66213 |           66229 |           66132 |           66132 
594   ipackets |            1003 |            1003 |            1002 |            1002 
595     ibytes |           66225 |           66209 |           66132 |           66132 
596    ierrors |               0 |               0 |               0 |               0 
597    oerrors |               0 |               0 |               0 |               0 
598      Tx Bw |     217.09 Kbps |     217.14 Kbps |     216.83 Kbps |     216.83 Kbps 
599
600 -Global stats enabled 
601 Cpu Utilization : 0.0  % <2>  29.7 Gb/core <3>
602 Platform_factor : 1.0    
603 Total-Tx        :     867.89 Kbps                                             <4>
604 Total-Rx        :     867.86 Kbps                                             <5>
605 Total-PPS       :       1.64 Kpps  
606 Total-CPS       :       0.50  cps  
607
608 Expected-PPS    :       2.00  pps   <6>
609 Expected-CPS    :       1.00  cps   <7>
610 Expected-BPS    :       1.36 Kbps   <8>
611
612 Active-flows    :        0 <9> Clients :      510   Socket-util  : 0.0000 %
613 Open-flows      :        1 <10> Servers :      254   Socket   :        1  Socket/Clients :  0.0
614 drop-rate       :       0.00  bps   <11>
615 current time    : 5.3 sec  
616 test duration   : 94.7 sec  
617
618 -Latency stats enabled 
619 Cpu Utilization : 0.2 %  <12>
620 if|   tx_ok , rx_ok  , rx   ,error,    average   ,   max         , Jitter ,  max window 
621   |         ,        , check,     , latency(usec),latency (usec) ,(usec)  ,             
622 --------------------------------------------------------------------------------------------------
623 0 |     1002,    1002,         0,   0,         51  ,      69,       0      |   0  69  67    <13>
624 1 |     1002,    1002,         0,   0,         53  ,     196,       0      |   0  196  53
625 2 |     1002,    1002,         0,   0,         54  ,      71,       0      |   0  71  69 
626 3 |     1002,    1002,         0,   0,         53  ,     193,       0      |   0  193  52 
627----
628<1> Link must be up for TRex to work.
629<2> Average CPU utilization of transmitters threads. For best results it should be lower than 80%.
630<3> Gb/sec generated per core of DP. Higher is better.
631<4> Total Tx must be the same as Rx at the end of the run
632<5> Total Rx must be the same as Tx at the end of the run
633<6> Expected number of packets per second (calculated without latency packets).
634<7> Expected number of connections per second (calculated without latency packets).
635<8> Expected number of bits per second (calculated without latency packets).
636<9> Number of TRex active "flows". Could be different than the number of router flows, due to aging issues. Usualy the TRex number of active flows is much lower than that of the router because the router ages flows slower.
637<10> Total number of TRex flows opened since startup (including active ones, and ones already closed).
638<11> Drop rate.
639<12> Rx and latency thread CPU utilization.
640<13> Tx_ok on port 0 should equal Rx_ok on port 1, and vice versa.
641
642More statistics information:
643
644*socket*::  Same as the active flows.  
645
646*Socket/Clients*:: Average of active flows per client, calculated as active_flows/#clients.
647
648*Socket-util*:: Estimation of number of L4 ports (sockets) used per client IP. This is approximately (100*active_flows/#clients)/64K, calculated as (average active flows per client*100/64K). Utilization of more than 50% means that TRex is generating too many flows per single client, and that more clients must be added in the generator config.
649// clarify above, especially the formula
650
651*Max window*:: Momentary maximum latency for a time window of 500 msec. There are few numbers shown per port.
652 The newest number (last 500msec) is on the right. Oldest on the left. This can help identifying spikes of high latency clearing after some time. Maximum latency is the total maximum over the entire test duration. To best understand this,
653 run TRex with latency option (-l) and watch the results with this section in mind.
654
655*Platform_factor*:: There are cases in which we duplicate the traffic using splitter/switch and we would like all numbers displayed by TRex to be multiplied by this factor, so that TRex counters will match the DUT counters.
656
657WARNING: If you don't see rx packets, revisit your MAC address configuration.
658
659include::trex_book_basic.asciidoc[]
660
661== Advanced features
662
663=== VLAN Trunk support 
664
665anchor:trex_vlan[]
666
667The VLAN Trunk TRex feature attempts to solve the router port bandwidth limitation when the traffic profile is asymmetric. Example: Asymmetric SFR profile.
668This feature converts asymmetric traffic to symmetric, from the port perspective, using router sub-interfaces.
669This requires TRex to send the traffic on two VLANs, as described below.
670
671.YAML format 
672[source,python]
673----
674  vlan       : { enable : 1  ,  vlan0 : 100 , vlan1 : 200 }
675----
676
677
678.Example
679[source,python]
680----
681- duration : 0.1
682  vlan       : { enable : 1  ,  vlan0 : 100 , vlan1 : 200 }   <1>
683----
684<1> Enable VLAN feature, vlan0==100 , vlan1==200
685        
686*Problem definition:*::
687
688Scenario: TRex with two ports and an SFR traffic profile.
689
690.Without VLAN/sub interfaces 
691[source,python]
692----
6930 ( client) -> [  ] - 1 ( server)
694----
695Without VLAN support the traffic is asymmetric. 10% of the traffic is sent from port 0 (client side), 90% is from port 1 (server). Port 1 become the bottlneck (10Gb/s limit) before port 0.
696
697.With VLAN/sub interfaces 
698[source,python]
699----
700port 0 ( client VLAN0) <->  |  | <-> port 1 ( server-VLAN0)
701port 0 ( server VLAN1) <->  |  | <-> port 1 ( client-VLAN1)
702----
703
704In this case both ports have the same amount of traffic.
705
706*Router configuation:*::
707[source,python]
708----
709        !
710        interface TenGigabitEthernet1/0/0      <1>
711         mac-address 0000.0001.0000   
712         mtu 4000
713         no ip address
714         load-interval 30
715        !
716        i
717        interface TenGigabitEthernet1/0/0.100
718         encapsulation dot1Q 100               <2> 
719         ip address 11.77.11.1 255.255.255.0
720         ip nbar protocol-discovery
721         ip policy route-map vlan_100_p1_to_p2 <3> 
722        !
723        interface TenGigabitEthernet1/0/0.200
724         encapsulation dot1Q 200               <4>
725         ip address 11.88.11.1 255.255.255.0
726         ip nbar protocol-discovery
727         ip policy route-map vlan_200_p1_to_p2 <5> 
728        !
729        interface TenGigabitEthernet1/1/0
730         mac-address 0000.0001.0000
731         mtu 4000
732         no ip address
733         load-interval 30
734        !
735        interface TenGigabitEthernet1/1/0.100
736         encapsulation dot1Q 100
737         ip address 22.77.11.1 255.255.255.0
738         ip nbar protocol-discovery
739         ip policy route-map vlan_100_p2_to_p1
740        !
741        interface TenGigabitEthernet1/1/0.200
742         encapsulation dot1Q 200
743         ip address 22.88.11.1 255.255.255.0
744         ip nbar protocol-discovery
745         ip policy route-map vlan_200_p2_to_p1
746        !
747        
748        arp 11.77.11.12 0000.0001.0000 ARPA      <6>
749        arp 22.77.11.12 0000.0001.0000 ARPA
750        
751        route-map vlan_100_p1_to_p2 permit 10    <7>
752         set ip next-hop 22.77.11.12
753        !
754        route-map vlan_100_p2_to_p1 permit 10
755         set ip next-hop 11.77.11.12
756        !
757        
758        route-map vlan_200_p1_to_p2 permit 10
759         set ip next-hop 22.88.11.12
760        !
761        route-map vlan_200_p2_to_p1 permit 10
762         set ip next-hop 11.88.11.12
763        !
764----
765<1> Disable the IP on the main port it is important.
766// above, clarify what's important
767<2> Enable VLAN1
768<3> PBR configuration
769<4> Enable VLAN2  
770<5> PBR configuration
771<6> TRex destination port MAC address
772<7> PBR configuration rules
773
774=== Static source MAC address setting  
775
776With this feature, TRex replaces the source MAC address with the client IP address.
777
778 Note: This feature was requested by the Cisco ISG group. 
779
780
781*YAML:*::
782[source,python]
783----
784 mac_override_by_ip : true
785----
786
787.Example
788[source,python]
789----
790- duration : 0.1
791 ..
792  mac_override_by_ip : true <1>
793----
794<1> In this case, the client side MAC address looks like this:
795SRC_MAC = IPV4(IP) + 00:00  
796
797=== IPv6 support 
798
799Support for IPv6 includes:
800
8011. Support for pcap files containing IPv6 packets
8022. Ability to generate IPv6 traffic from pcap files containing IPv4 packets
803The following command line option enables this feature: `--ipv6`
804The keywords (`src_ipv6` and `dst_ipv6`) specify the most significant 96 bits of the IPv6 address - for example:
805
806[source,python]
807----
808      src_ipv6 : [0xFE80,0x0232,0x1002,0x0051,0x0000,0x0000]
809      dst_ipv6 : [0x2001,0x0DB8,0x0003,0x0004,0x0000,0x0000]
810----
811      
812The IPv6 address is formed by placing what would typically be the IPv4
813address into the least significant 32 bits and copying the value provided
814in the src_ipv6/dst_ipv6 keywords into the most signficant 96 bits.
815If src_ipv6 and dst_ipv6 are not specified, the default
816is to form IPv4-compatible addresses (most signifcant 96 bits are zero).
817
818There is support for all plugins.
819  
820*Example:*::
821[source,bash]
822----
823$sudo ./t-rex-64 -f cap2l/sfr_delay_10_1g.yaml -c 4 -p -l 100 -d 100000 -m 30  --ipv6
824----
825
826*Limitations:*::
827
828* TRex cannot generate both IPv4 and IPv6 traffic.
829* The `--ipv6` switch must be specified even when using pcap file containing only IPv6 packets.
830
831
832*Router configuration:*::
833
834[source,python]
835----
836interface TenGigabitEthernet1/0/0
837 mac-address 0000.0001.0000
838 mtu 4000
839 ip address 11.11.11.11 255.255.255.0
840 ip policy route-map p1_to_p2
841 load-interval 30
842 ipv6 enable   ==> IPv6
843 ipv6 address 2001:DB8:1111:2222::1/64                  <1>
844 ipv6 policy route-map ipv6_p1_to_p2                    <2>
845!
846
847
848ipv6 unicast-routing                                    <3>
849
850ipv6 neighbor 3001::2 TenGigabitEthernet0/1/0 0000.0002.0002   <4>
851ipv6 neighbor 2001::2 TenGigabitEthernet0/0/0 0000.0003.0002
852
853route-map ipv6_p1_to_p2 permit 10                              <5>
854 set ipv6 next-hop 2001::2
855!
856route-map ipv6_p2_to_p1 permit 10
857 set ipv6 next-hop 3001::2
858!
859
860
861asr1k(config)#ipv6 route 4000::/64 2001::2                 
862asr1k(config)#ipv6 route 5000::/64 3001::2 
863----
864<1> Enable IPv6 
865<2> Add pbr 
866<3> Enable IPv6 routing 
867<4> MAC address setting. Should be TRex MAC.
868<5> PBR configuraion
869
870
871=== Client clustering configuration
872TRex supports testing complex topologies, with more than one DUT, using a feature called "client clustering".
873This feature allows specifying the distribution of clients TRex emulates.
874
875Let's look at the following topology:
876
877.Topology Example 
878image:images/topology.png[title="Client Clustering",width=850]
879
880We have two clusters of DUTs.
881Using config file, you can partition TRex emulated clients to groups, and define
882how they will be spread between the DUT clusters.
883
884Group configuration includes:
885
886* IP start range.
887* IP end range.
888* Initiator side configuration. - These are the parameters affecting packets sent from client side.
889* Responder side configuration. - These are the parameters affecting packets sent from server side.
890
891[NOTE]
892It is important to understand that this is *complimentary* to the client generator
893configured per profile - it only defines how the clients will be spread between clusters.
894
895Let's look at an example.
896
897We have a profile defining client generator.
898
899[source,bash]
900----
901$cat cap2/dns.yaml 
902- duration : 10.0
903  generator :  
904          distribution : "seq"           
905          clients_start : "16.0.0.1"
906          clients_end   : "16.0.0.255"   
907          servers_start : "48.0.0.1"
908          servers_end   : "48.0.0.255"   
909          dual_port_mask : "1.0.0.0" 
910  cap_info : 
911     - name: cap2/dns.pcap
912       cps : 1.0          
913       ipg : 10000        
914       rtt : 10000        
915       w   : 1            
916----
917
918We want to create two clusters with 4 and 3 devices respectively.
919We also want to send *80%* of the traffic to the upper cluster and *20%* to the lower cluster.
920We can specify to which DUT the packet will be sent by MAC address or IP. We will present a MAC
921based example, and then see how to change to be IP based.
922
923We will create the following cluster configuration file.
924
925[source,bash]
926----
927#
928# Client configuration example file
929# The file must contain the following fields
930#
931# 'vlan'   - if the entire configuration uses VLAN,
932#            each client group must include vlan
933#            configuration
934#
935# 'groups' - each client group must contain range of IPs
936#            and initiator and responder section
937#            'count' represents the number of different DUTs
938#            in the group.
939#
940
941# 'true' means each group must contain VLAN configuration. 'false' means no VLAN config allowed.
942vlan: true
943
944groups:
945
946-    ip_start  : 16.0.0.1
947     ip_end    : 16.0.0.204
948     initiator :
949                 vlan    : 100
950                 dst_mac : "00:00:00:01:00:00"
951     responder :
952                 vlan    : 200
953                 dst_mac : "00:00:00:02:00:00"
954
955     count     : 4
956
957-    ip_start  : 16.0.0.205
958     ip_end    : 16.0.0.255
959     initiator :
960                 vlan    : 101
961                 dst_mac : "00:00:01:00:00:00"
962
963     responder:
964                 vlan    : 201
965                 dst_mac : "00:00:02:00:00:00"
966
967     count     : 3
968
969----
970
971The above configuration will divide the generator range of 255 clients to two clusters. The range
972of IPs in all groups in the client config file together, must cover the entire range of client IPs
973from the traffic profile file.
974
975MACs will be allocated incrementally, with a wrap around after count addresses.
976
977e.g.
978
979*Initiator side: (packets with source in 16.x.x.x net)*
980
981* 16.0.0.1 -> 48.x.x.x - dst_mac: 00:00:00:01:00:00  vlan: 100 
982* 16.0.0.2 -> 48.x.x.x - dst_mac: 00:00:00:01:00:01  vlan: 100 
983* 16.0.0.3 -> 48.x.x.x - dst_mac: 00:00:00:01:00:02  vlan: 100 
984* 16.0.0.4 -> 48.x.x.x - dst_mac: 00:00:00:01:00:03  vlan: 100 
985* 16.0.0.5 -> 48.x.x.x - dst_mac: 00:00:00:01:00:00  vlan: 100 
986* 16.0.0.6 -> 48.x.x.x - dst_mac: 00:00:00:01:00:01  vlan: 100 
987
988*responder side: (packets with source in 48.x.x.x net)* 
989
990* 48.x.x.x -> 16.0.0.1  - dst_mac(from responder) : "00:00:00:02:00:00" , vlan:200
991* 48.x.x.x -> 16.0.0.2  - dst_mac(from responder) : "00:00:00:02:00:01" , vlan:200
992
993and so on. +
994This means that the DUT MACs of each cluster has to be changed to be sequential. Other option is to
995specify instead of ``dst_mac'', ip address, using ``next_hop''. +
996For example, config file first group will look like:
997
998----
999-    ip_start  : 16.0.0.1
1000     ip_end    : 16.0.0.204
1001     initiator :
1002                 vlan     : 100
1003                 next_hop : 1.1.1.1
1004                 src_ip   : 1.1.1.100
1005     responder :
1006                 vlan     : 200
1007                 next_hop : 2.2.2.1
1008                 src_ip   : 2.2.2.100
1009
1010     count     : 4
1011----
1012
1013In this case, TRex will try to resolve using ARP requests the addresses
10141.1.1.1, 1.1.1.2, 1.1.1.3, 1.1.1.4 (and the range 2.2.2.1-2.2.2.4). If not all IPs are resolved,
1015TRex will exit with an error message. ``src_ip'' will be used for sending gratitues ARP, and
1016for filling relevant fields in ARP request. If no ``src_ip'' given, TRex will look for source
1017IP in the relevant port section in the platform config file (/etc/trex_cfg.yaml). If none is found, TRex
1018will exit with an error message. +
1019If client config file is given, the ``dest_mac'' and ``default_gw'' parameters from the platform config
1020file are ignored.
1021
1022
1023[NOTE]
1024It is important to understand that the ip to MAC coupling (both with MAC based config or IP based)
1025is done at the beginning and never changes. Meaning, for example, for the MAC case, packets
1026with source IP 16.0.0.2 will always have VLAN 100 and dst MAC 00:00:00:01:00:01.
1027Packets with destination IP 16.0.0.2 will always have VLAN 200 and dst MAC "00:00:00:02:00:01.
1028This way, you can predict exactly which packet (and how many packets) will go to each DUT.
1029
1030*Usage:*
1031
1032[source,bash]
1033----
1034sudo ./t-rex-64 -f cap2/dns.yaml --client_cfg my_cfg.yaml
1035----
1036
1037=== NAT support 
1038
1039TRex can learn dynamic NAT/PAT translation. To enable this feature add `--learn-mode <mode>` to the command line.
1040To learn the NAT translation, TRex must embed information describing the flow a packet belongs to, in the first
1041packet of each flow. This can be done in two different methods, depending on the chosen <mode>.
1042
1043*mode 1:*::
1044
1045Flow info is embedded in the ACK of the first TCP SYN.
1046In this mode, there is a limitation that bidirectional UDP templates (for example, DNS) are not supported. 
1047This mode was developed for testing NAT with firewalls (which usually do not work with mode 2).
1048In this mode, TRex also learn and compensate for TCP sequence number randomization that might be done by the DUT.
1049TRex can learn and compensate for seq num randomization in both directions of the connection.
1050
1051*mode 2:*::
1052
1053Flow info is added in a special IPv4 option header (8 bytes long 0x10 id). The option is added only to the first packet in the flow.
1054This mode does not work with DUTs that drop packets with IP options (for example, Cisco ASA firewall).
1055
1056*mode 3:*::
1057
1058This is like mode 1, with the only change being that TRex does not learn the seq num randomization in the server->client direction.
1059This mode can give much better connections per second performance than mode 1 (still, for all existing firewalls, mode 1 cps rate is more than enough).
1060
1061==== Examples
1062
1063*simple HTTP traffic*
1064
1065[source,bash]
1066----
1067$sudo ./t-rex-64 -f cap2/http_simple.yaml -c 4  -l 1000 -d 100000 -m 30  --learn-mode 1
1068----
1069
1070*SFR traffic without bundling/ALG support*
1071
1072[source,bash]
1073----
1074$sudo ./t-rex-64 -f avl/sfr_delay_10_1g_no_bundling.yaml -c 4  -l 1000 -d 100000 -m 10  --learn-mode 2
1075----
1076
1077*NAT terminal counters:*::
1078
1079[source,python]
1080----
1081-Global stats enabled 
1082 Cpu Utilization : 0.6  %  33.4 Gb/core 
1083 Platform_factor : 1.0
1084 Total-Tx        :       3.77 Gbps   NAT time out    :      917 <1> (0 in wait for syn+ack) <5>
1085 Total-Rx        :       3.77 Gbps   NAT aged flow id:        0 <2>
1086 Total-PPS       :     505.72 Kpps   Total NAT active:      163 <3> (12 waiting for syn) <6>
1087 Total-CPS       :      13.43 Kcps   Total NAT opened:    82677 <4>
1088----
1089<1> Number of connections for which TRex had to send the next packet in the flow, but did not learn the NAT translation yet. Should be 0. Usually, value different than 0 is seen if the DUT drops the flow (probably because it can't handle the number of connections)
1090<2> Number of flows for which when we got the translation info, flow was aged out already. Non 0 value here should be very rare. Can occur only when there is huge latency in the DUT input/output queue.
1091<3> Number of flows for which we sent the first packet, but did not learn the NAT translation yet. Value seen depends on the connection per second rate and round trip time.
1092<4> Total number of translations over the lifetime of the TRex instance. May be different from the total number of flows if template is uni-directional (and consequently does not need translation).
1093<5> Out of the timed out flows, how many were timed out while waiting to learn the TCP seq num randomization of the server->client from the SYN+ACK packet (Seen only in --learn-mode 1)
1094<6> Out of the active NAT sessions, how many are waiting to learn the client->server translation from the SYN packet (others are waiting for SYN+ACK from server) (Seen only in --learn-mode 1)
1095
1096*Configuration for Cisco ASR1000 Series:*::
1097
1098This feature was tested with the following configuration and  sfr_delay_10_1g_no_bundling. yaml traffic profile.
1099Client address range is 16.0.0.1 to 16.0.0.255 
1100
1101[source,python]
1102----
1103interface TenGigabitEthernet1/0/0            <1>
1104 mac-address 0000.0001.0000
1105 mtu 4000
1106 ip address 11.11.11.11 255.255.255.0
1107 ip policy route-map p1_to_p2
1108 ip nat inside                               <2>
1109 load-interval 30
1110!
1111
1112interface TenGigabitEthernet1/1/0
1113 mac-address 0000.0001.0000
1114 mtu 4000
1115 ip address 11.11.11.11 255.255.255.0
1116 ip policy route-map p1_to_p2
1117 ip nat outside                              <3>
1118 load-interval 30
1119
1120ip  nat pool my 200.0.0.0 200.0.0.255 netmask 255.255.255.0  <4>
1121
1122ip nat inside source list 7 pool my overload 
1123access-list 7 permit 16.0.0.0 0.0.0.255                      <5>
1124
1125ip nat inside source list 8 pool my overload                 <6>
1126access-list 8 permit 17.0.0.0 0.0.0.255                      
1127----
1128<1> Must be connected to TRex Client port (router inside port)
1129<2> NAT inside 
1130<3> NAT outside
1131<4> Pool of outside address with overload
1132<5> Match TRex YAML client range
1133<6> In case of dual port TRex
1134
1135// verify 1 and 5 above; rephrased
1136
1137
1138*Limitations:*::
1139
1140. The IPv6-IPv6 NAT feature does not exist on routers, so this feature can work only with IPv4.
1141. Does not support NAT64. 
1142. Bundling/plugin is not fully supported. Consequently, sfr_delay_10.yaml does not work. Use sfr_delay_10_no_bundling.yaml instead.
1143
1144[NOTE]
1145=====================================================================
1146* `--learn-verify` is a TRex debug mechanism for testing the TRex learn mechanism.
1147* Need to run it when DUT is configured without NAT. It will verify that the inside_ip==outside_ip and inside_port==outside_port.
1148=====================================================================
1149
1150=== Flow order/latency verification 
1151
1152In normal mode (without this feature enabled), received traffic is not checked by software. Hardware (Intel NIC) testing for dropped packets occurs at the end of the test. The only exception is the Latency/Jitter packets.
1153This is one reason that with TRex, you *cannot* check features that terminate traffic (for example TCP Proxy).
1154To enable this feature, add `--rx-check <sample>` to the command line options, where <sample> is the sample rate. 
1155The number of flows that will be sent to the software for verification is (1/(sample_rate). For 40Gb/sec traffic you can use a sample rate of 1/128. Watch for Rx CPU% utilization.
1156
1157[NOTE]
1158============
1159This feature changes the TTL of the sampled flows to 255 and expects to receive packets with TTL 254 or 255 (one routing hop). If you have more than one hop in your setup, use `--hops` to change it to a higher value. More than one hop is possible if there are number of routers betwean TRex client side and TRex server side.
1160============
1161
1162This feature ensures that:
1163
1164* Packets get out of DUT in order (from each flow perspective).
1165* There are no packet drops (no need to wait for the end of the test). Without this flag, you must wait for the end of the test in order to identify packet drops, because there is always a difference between TX and Rx, due to RTT.
1166
1167
1168.Full example 
1169[source,bash]
1170----
1171$sudo ./t-rex-64 -f avl/sfr_delay_10_1g.yaml -c 4 -p -l 100 -d 100000 -m 30  --rx-check 128
1172----
1173
1174[source,python]
1175----
1176Cpu Utilization : 0.1 %                                                                       <1>
1177 if|   tx_ok , rx_ok  , rx   ,error,    average   ,   max         , Jitter ,  max window 
1178   |         ,        , check,     , latency(usec),latency (usec) ,(usec)  ,             
1179 --------------------------------------------------------------------------------
1180 0 |     1002,    1002,      2501,   0,         61  ,      70,       3      |  60
1181 1 |     1002,    1002,      2012,   0,         56  ,      63,       2      |  50
1182 2 |     1002,    1002,      2322,   0,         66  ,      74,       5      |  68
1183 3 |     1002,    1002,      1727,   0,         58  ,      68,       2      |  52
1184
1185 Rx Check stats enabled                                                                       <2>
1186 -------------------------------------------------------------------------------------------
1187 rx check:  avg/max/jitter latency,       94  ,     744,       49      |  252  287  309       <3>
1188 
1189 active flows: <6>      10, fif: <5>     308,  drop:        0, errors:        0                <4>
1190 -------------------------------------------------------------------------------------------
1191----
1192<1> CPU% of the Rx thread. If it is too high, *increase* the sample rate.
1193<2> Rx Check section. For more detailed info, press 'r' during the test or at the end of the test.
1194<3> Average latency, max latency, jitter on the template flows in microseconds. This is usually *higher* than the latency check packet because the feature works more on this packet.
1195<4> Drop counters and errors counter should be zero. If not, press 'r' to see the full report or view the report at the end of the test.
1196<5> fif - First in flow. Number of new flows handled by the Rx thread.
1197<6> active flows - number of active flows handled by rx thread 
1198
1199.Press R to Display Full Report
1200[source,python]
1201----
1202 m_total_rx                              : 2 
1203 m_lookup                                : 2 
1204 m_found                                 : 1 
1205 m_fif                                   : 1 
1206 m_add                                   : 1 
1207 m_remove                                : 1 
1208 m_active                                : 0 
1209                                                        <1>
1210 0  0  0  0  1041  0  0  0  0  0  0  0  0  min_delta  : 10 usec 
1211 cnt        : 2 
1212 high_cnt   : 2 
1213 max_d_time : 1041 usec
1214 sliding_average    : 1 usec                            <2>
1215 precent    : 100.0 %
1216 histogram 
1217 -----------
1218 h[1000]  :  2 
1219 tempate_id_ 0 , errors:       0,  jitter: 61           <3>
1220 tempate_id_ 1 , errors:       0,  jitter: 0 
1221 tempate_id_ 2 , errors:       0,  jitter: 0 
1222 tempate_id_ 3 , errors:       0,  jitter: 0 
1223 tempate_id_ 4 , errors:       0,  jitter: 0 
1224 tempate_id_ 5 , errors:       0,  jitter: 0 
1225 tempate_id_ 6 , errors:       0,  jitter: 0 
1226 tempate_id_ 7 , errors:       0,  jitter: 0 
1227 tempate_id_ 8 , errors:       0,  jitter: 0 
1228 tempate_id_ 9 , errors:       0,  jitter: 0 
1229 tempate_id_10 , errors:       0,  jitter: 0 
1230 tempate_id_11 , errors:       0,  jitter: 0 
1231 tempate_id_12 , errors:       0,  jitter: 0 
1232 tempate_id_13 , errors:       0,  jitter: 0 
1233 tempate_id_14 , errors:       0,  jitter: 0 
1234 tempate_id_15 , errors:       0,  jitter: 0 
1235 ager :
1236 m_st_alloc                                 : 1 
1237 m_st_free                                  : 0 
1238 m_st_start                                 : 2 
1239 m_st_stop                                  : 1 
1240 m_st_handle                                : 0 
1241----
1242<1> Errors, if any, shown here
1243<2> Low pass filter on the active average of latency events 
1244<3> Error per template info
1245
1246// IGNORE: this line added to help rendition. Without this line, the "Notes and Limitations" section below does not appear.
1247
1248*Notes and Limitations:*::
1249
1250** To receive the packets TRex does the following:
1251*** Changes the TTL to 0xff and expects 0xFF (loopback) or oxFE (route). (Use `--hop` to configure this value.)
1252*** Adds 24 bytes of metadata as ipv4/ipv6 option header.
1253// clarify "ipv4/ipv6 option header" above
1254
1255== Reference
1256
1257=== Traffic YAML (parameter of -f option)
1258
1259==== Global Traffic YAML section 
1260
1261[source,python]
1262----
1263- duration : 10.0                          <1>
1264  generator :                              <2>
1265          distribution : "seq"           
1266          clients_start : "16.0.0.1"     
1267          clients_end   : "16.0.0.255"   
1268          servers_start : "48.0.0.1"     
1269          servers_end   : "48.0.0.255"   
1270          clients_per_gb : 201
1271          min_clients    : 101
1272          dual_port_mask : "1.0.0.0" 
1273          tcp_aging      : 1
1274          udp_aging      : 1
1275  mac        : [0x00,0x00,0x00,0x01,0x00,0x00] <3>
1276  cap_ipg    : true                            <4>
1277  cap_ipg_min    : 30                          <5>
1278  cap_override_ipg    : 200                    <6>
1279  vlan       : { enable : 1  ,  vlan0 : 100 , vlan1 : 200 } <7>
1280  mac_override_by_ip : true  <8>
1281----
1282<1> Test duration (seconds). Can be overridden using the `-d` option.
1283<2> See the link:trex_manual.html#_clients_servers_ip_allocation_scheme[generator] section.
1284// what does note 2 mean? see somewhere else? isn't this simply the generator section?
1285<3> Default source/destination MAC address. The configuration YAML can override this.
1286<4> true (default) indicates that the IPG is taken from the cap file (also taking into account cap_ipg_min and cap_override_ipg if they exist). false indicates that IPG is taken from per template section.
1287<5> The following two options can set the min ipg in microseconds: (if (pkt_ipg<cap_ipg_min) { pkt_ipg=cap_override_ipg} )
1288<6> Value to override (microseconds), as described in note above.
1289<7> Enable vlan feature. See xref:trex_vlan[trex_vlan section] for info.
1290<8> Enable MAC address replacement by client IP.
1291
1292==== Per template section 
1293// clarify "per template"  
1294
1295[source,python]
1296----
1297     - name: cap2/dns.pcap <1>
1298       cps : 10.0          <2>
1299       ipg : 10000         <3>
1300       rtt : 10000         <4>
1301       w   : 1             <5>
1302       server_addr : "48.0.0.7"    <6>
1303       one_app_server : true       <7>
1304       
1305----
1306<1> The name of the template pcap file. Can be relative path from the t-rex-64 image directory, or an absolute path. The pcap file should include only one flow. (Exception: in case of plug-ins).
1307<2> Connection per second. This is the value that will be used if specifying -m 1 from command line (giving -m x will multiply this 
1308<3> If the global section of the YAML file includes `cap_ipg    : false`, this line sets the inter-packet gap in microseconds. 
1309<4> Should be set to the same value as ipg (microseconds). 
1310<5> Default value: w=1. This indicates to the IP generator how to generate the flows. If w=2, two flows from the same template will be generated in a burst (more for HTTP that has burst of flows).
1311<6> If `one_app_server` is set to true, then all templates will use the same server.
1312<7> If the same server address is required, set this value to true.
1313
1314
1315
1316=== Configuration YAML (parameter of --cfg option)
1317
1318anchor:trex_config[]
1319
1320The configuration file, in YAML format, configures TRex behavior, including:
1321 
1322- IP address or MAC address for each port (source and destination).
1323- Masked interfaces, to ensure that TRex does not try to use the management ports as traffic ports.
1324- Changing the zmq/telnet TCP port.
1325
1326You specify which config file to use by adding --cfg <file name> to the command line arguments. +
1327If no --cfg given, the default `/etc/trex_cfg.yaml` is used. +
1328Configuration file examples can be found in the `$TREX_ROOT/scripts/cfg` folder.
1329
1330==== Basic Configurations
1331
1332[source,python]
1333----
1334     - port_limit    : 2    #mandatory <1>
1335       version       : 2    #mandatory <2>
1336       interfaces    : ["03:00.0", "03:00.1"]   #mandatory <3>
1337       #enable_zmq_pub  : true #optional <4>
1338       #zmq_pub_port    : 4500 #optional <5>
1339       #prefix          : setup1 #optional <6>
1340       #limit_memory    : 1024 #optional <7>
1341       c               : 4 #optional <8>
1342       port_bandwidth_gb : 10 #optional <9>
1343       port_info       :  # set eh mac addr  mandatory
1344            - default_gw : 1.1.1.1   # port 0 <10>
1345              dest_mac   : '00:00:00:01:00:00' # Either default_gw or dest_mac is mandatory <10>
1346              src_mac    : '00:00:00:02:00:00' # optional <11>
1347              ip         : 2.2.2.2 # optional <12>
1348              vlan       : 15 # optional <13>
1349            - dest_mac   : '00:00:00:03:00:00'  # port 1
1350              src_mac    : '00:00:00:04:00:00'
1351            - dest_mac   : '00:00:00:05:00:00'  # port 2
1352              src_mac    : '00:00:00:06:00:00'
1353            - dest_mac   :   [0x0,0x0,0x0,0x7,0x0,0x01]  # port 3 <14>
1354              src_mac    :   [0x0,0x0,0x0,0x8,0x0,0x02] # <14>
1355----
1356<1>  Number of ports. Should be equal to the number of interfaces listed in 3. - mandatory
1357<2>  Must be set to 2. - mandatory
1358<3>  List of interfaces to use. Run `sudo ./dpdk_setup_ports.py --show` to see the list you can choose from. - mandatory
1359<4>  Enable the ZMQ publisher for stats data, default is true. 
1360<5>  ZMQ port number. Default value is good. If running two TRex instances on the same machine, each should be given distinct number. Otherwise, can remove this line.
1361<6>  If running two TRex instances on the same machine, each should be given distinct name. Otherwise, can remove this line. ( Passed to DPDK as --file-prefix arg)
1362<7>  Limit the amount of packet memory used. (Passed to dpdk as -m arg)
1363<8> Number of threads (cores) TRex will use per interface pair ( Can be overridden by -c command line option )
1364<9> The bandwidth of each interface in Gbs. In this example we have 10Gbs interfaces. For VM, put 1. Used to tune the amount of memory allocated by TRex.
1365<10> TRex need to know the destination MAC address to use on each port. You can specify this in one of two ways: +
1366Specify dest_mac directly. +
1367Specify default_gw (since version 2.10). In this case (only if no dest_mac given), TRex will issue ARP request to this IP, and will use
1368the result as dest MAC. If no dest_mac given, and no ARP response received, TRex will exit.
1369
1370<11> Source MAC to use when sending packets from this interface. If not given (since version 2.10), MAC address of the port will be used.
1371<12> If given (since version 2.10), TRex will issue gratitues ARP for the ip + src MAC pair on appropriate port. In stateful mode,
1372gratitues ARP for each ip will be sent every 120 seconds (Can be changed using --arp-refresh-period argument).
1373<13> If given, gratitues ARP and ARP request will be sent using the given VLAN tag.
1374<14> Old MAC address format. New format is supported since version v2.09.
1375
1376[NOTE]
1377=========================================================================================
1378If you use version earlier than 2.10, or choose to omit the ``ip''
1379and have mac based configuration, be aware that TRex will not send any
1380gratitues ARP and will not answer ARP requests. In this case, you must configure static
1381ARP entries pointing to TRex port on your DUT. For an example config, you can look
1382xref:trex_config[here].
1383=========================================================================================
1384
1385To find out which interfaces (NIC ports) can be used, perform the following:
1386
1387[source,bash]
1388----
1389 $>sudo ./dpdk_setup_ports.py --show
1390
1391 Network devices using DPDK-compatible driver
1392 ============================================
1393
1394 Network devices using kernel driver
1395 ===================================
1396 0000:02:00.0 '82545EM Gigabit Ethernet Controller' if=eth2 drv=e1000 unused=igb_uio *Active* #<1>
1397 0000:03:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection' drv= unused=ixgb #<2>
1398 0000:03:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection' drv= unused=ixgb
1399 0000:13:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection' drv= unused=ixgb
1400 0000:13:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection' drv= unused=ixgb
1401
1402 Other network devices
1403 =====================
1404 <none>
1405----
1406<1> We see that 02:00.0 is active (our management port).
1407<2> All other NIC ports (03:00.0, 03:00.1, 13:00.0, 13:00.1) can be used.
1408
1409minimum configuration file is:
1410
1411[source,bash]
1412----
1413<none>
1414- port_limit    : 4         
1415  version       : 2
1416  interfaces    : ["03:00.0","03:00.1","13:00.1","13:00.0"]
1417----
1418
1419==== Memory section configuration  
1420
1421The memory section is optional. It is used when there is a need to tune the amount of memory used by TRex packet manager.
1422Default values (from the TRex source code), are usually good for most users. Unless you have some unusual needs, you can
1423eliminate this section.
1424
1425[source,python]
1426----
1427        - port_limit      : 2                                                                           
1428          version       : 2                                                                             
1429          interfaces    : ["03:00.0","03:00.1"]                                                         
1430          memory    :                                           <1>
1431             mbuf_64     : 16380                                <2>
1432             mbuf_128    : 8190
1433             mbuf_256    : 8190
1434             mbuf_512    : 8190
1435             mbuf_1024   : 8190
1436             mbuf_2048   : 4096
1437             traffic_mbuf_64     : 16380                        <3>
1438             traffic_mbuf_128    : 8190
1439             traffic_mbuf_256    : 8190
1440             traffic_mbuf_512    : 8190
1441             traffic_mbuf_1024   : 8190
1442             traffic_mbuf_2048   : 4096
1443             dp_flows    : 1048576                              <4>
1444             global_flows : 10240                               <5>
1445----
1446<1> Memory section header
1447<2> Numbers of memory buffers allocated for packets in transit, per port pair. Numbers are specified per packet size.
1448<3> Numbers of memory buffers allocated for holding the part of the packet which is remained unchanged per template.
1449You should increase numbers here, only if you have very large amount of templates.
1450<4> Number of TRex flow objects allocated (To get best performance they are allocated upfront, and not dynamically).
1451If you expect more concurrent flows than the default (1048576), enlarge this.
1452<5> Number objects TRex allocates for holding NAT ``in transit'' connections. In stateful mode, TRex learn NAT
1453translation by looking at the address changes done by the DUT to the first packet of each flow. So, these are the
1454number of flows for which TRex sent the first flow packet, but did not learn the translation yet. Again, default
1455here (10240) should be good. Increase only if you use NAT and see issues.
1456
1457
1458==== Platform section configuration  
1459
1460The platform section is optional. It is used to tune the performance and allocate the cores to the right NUMA  
1461a configuration file now has the folowing struct to support multi instance 
1462
1463[source,python]
1464----
1465- version       : 2
1466  interfaces    : ["03:00.0","03:00.1"]
1467  port_limit    : 2 
1468....
1469  platform :                                                    <1>
1470        master_thread_id  : 0                                   <2>
1471        latency_thread_id : 5                                   <3>
1472        dual_if   :                                             <4>
1473             - socket   : 0                                     <5>
1474               threads  : [1,2,3,4]                             <6>
1475----
1476<1> Platform section header.
1477<2> Hardware thread_id for control thread.
1478<3> Hardware thread_id for RX thread.
1479<4> ``dual_if'' section defines info for interface pairs (according to the order in ``interfaces'' list).
1480each section, starting with ``- socket'' defines info for different interface pair.
1481<5> The NUMA node from which memory will be allocated for use by the interface pair.
1482<6> Hardware threads to be used for sending packets for the interface pair. Threads are pinned to cores, so specifying threads
1483actually determines the hardware cores.
1484
1485*Real example:* anchor:numa-example[]
1486
1487We connected 2 Intel XL710 NICs close to each other on the motherboard. They shared the same NUMA:
1488
1489image:images/same_numa.png[title="2_NICSs_same_NUMA"]
1490
1491CPU utilization was very high ~100%, with c=2 and c=4 the results were same.
1492
1493Then, we moved the cards to different NUMAs:
1494
1495image:images/different_numa.png[title="2_NICSs_different_NUMAs"]
1496
1497*+*
1498We added configuration to the /etc/trex_cfg.yaml:
1499
1500[source,python]
1501  platform :
1502        master_thread_id  : 0
1503        latency_thread_id : 8
1504        dual_if   :
1505             - socket   : 0
1506               threads  : [1, 2, 3, 4, 5, 6, 7]
1507             - socket   : 1
1508               threads  : [9, 10, 11, 12, 13, 14, 15]
1509
1510This gave best results: with *\~98 Gb/s* TX BW and c=7, CPU utilization became *~21%*! (40% with c=4)
1511
1512=== Command line options 
1513
1514anchor:cml-line[]
1515
1516*--allow-coredump*::
1517Allow creation of core dump.
1518
1519*--arp-refresh-period <num>*::
1520Period in seconds between sending of gratuitous ARP for our addresses. Value of 0 means ``never send``.
1521
1522*-c <num>*::
1523Number of hardware threads to use per interface pair. Use at least 4 for TRex 40Gbs. +
1524TRex uses 2 threads for inner needs. Rest of the threads can be used. Maximum number here, can be number of free threads
1525divided by number of interface pairs. +
1526For virtual NICs on VM, we always use one thread per interface pair.
1527
1528*--cfg <file name>*::
1529TRex configuration file to use. See relevant manual section for all config file options.
1530
1531*--checksum-offload*::
1532Enable IP, TCP and UDP tx checksum offloading, using DPDK. This requires all used interfaces to support this.
1533
1534*--client_cfg <file>*::
1535YAML file describing clients configuration. Look link:trex_manual.html#_client_clustering_configuration[here] for details.
1536
1537*-d <num>*::
1538Duration of the test in seconds.
1539
1540*-e*::
1541  Same as `-p`, but change the src/dst IP according to the port. Using this, you will get all the packets of the
1542  same flow from the same port, and with the same src/dst IP. +
1543  It will not work good with NBAR as it expects all clients ip to be sent from same direction.
1544
1545*-f <yaml file>*::
1546Specify traffic YAML configuration file to use. Mandatory option for stateful mode.
1547
1548*--hops <num>*::
1549   Provide number of hops in the setup (default is one hop). Relevant only if the Rx check is enabled.
1550   Look link:trex_manual.html#_flow_order_latency_verification[here] for details.
1551
1552*--iom <mode>*::
1553        I/O mode. Possible values: 0 (silent), 1 (normal), 2 (short).
1554
1555*--ipv6*::
1556       Convert templates to IPv6 mode.
1557
1558*-k <num>*::
1559   Run ``warm up'' traffic for num seconds before starting the test. This is needed if TRex is connected to switch running
1560   spanning tree. You want the switch to see traffic from all relevant source MAC addresses before starting to send real
1561   data. Traffic sent is the same used for the latency test (-l option) +
1562   Current limitation (holds for TRex version 1.82): does not work properly on VM.
1563
1564*-l <rate>*::
1565    In parallel to the test, run latency check, sending packets at rate/sec from each interface.
1566
1567*--learn-mode <mode>*::
1568    Learn the dynamic NAT translation. Look link:trex_manual.html#_nat_support[here] for details.
1569
1570*--learn-verify*::
1571   Used for testing the NAT learning mechanism. Do the learning as if DUT is doing NAT, but verify that packets
1572   are not actually changed.
1573
1574*--limit-ports <port num>*::
1575   Limit the number of ports used. Overrides the ``port_limit'' from config file.
1576
1577*--lm <hex bit mask>*::
1578Mask specifying which ports will send traffic. For example, 0x1 - Only port 0 will send. 0x4 - only port 2 will send.
1579This can be used to verify port connectivity. You can send packets from one port, and look at counters on the DUT.
1580
1581*--lo*::
1582   Latency only - Send only latency packets. Do not send packets from the templates/pcap files.
1583
1584*-m <num>*::
1585   Rate multiplier. TRex will multiply the CPS rate of each template by num.
1586
1587*--nc*::
1588    If set, will terminate exacly at the end of the specified duration.
1589    This provides faster, more accurate TRex termination.
1590    By default (without this option), TRex waits for all flows to terminate gracefully. In case of a very long flow, termination might prolong. 
1591
1592*--no-flow-control-change*::
1593  Prevents TRex from changing flow control. By default (without this option), TRex disables flow control at startup for all cards, except for the Intel XL710 40G card.
1594
1595*--no-key*:: Daemon mode, don't get input from keyboard.
1596
1597*--no-watchdog*:: Disable watchdog.
1598    
1599*-p*::
1600Send all packets of the same flow from the same direction. For each flow, TRex will randomly choose between client port and
1601server port, and send all the packets from this port. src/dst IPs keep their values as if packets are sent from two ports.
1602Meaning, we get on the same port packets from client to server, and from server to client. +
1603If you are using this with a router, you can not relay on routing rules to pass traffic to TRex, you must configure policy
1604based routes to pass all traffic from one DUT port to the other. +
1605
1606*-pm <num>*::
1607   Platform factor. If the setup includes splitter, you can multiply all statistic number displayed by TRex by this factor, so that they will match the DUT counters.
1608  
1609*-pubd*::
1610  Disable ZMQ monitor's publishers.
1611
1612*--rx-check <sample rate>*::
1613        Enable Rx check module. Using this, each thread randomly samples 1/sample_rate of the flows and checks packet order, latency, and additional statistics for the sampled flows.
1614        Note: This feature works on the RX thread.
1615
1616*-v <verbosity level>*::
1617   Show debug info. Value of 1 shows debug info on startup. Value of 3, shows debug info during run at some cases. Might slow down operation.
1618
1619*--vlan*:: Relevant only for stateless mode with Intel 82599 10G NIC.
1620   When configuring flow stat and latency per stream rules, assume all streams uses VLAN.
1621
1622*-w <num seconds>*::
1623   Wait additional time between NICs initialization and sending traffic. Can be useful if DUT needs extra setup time. Default is 1 second.
1624
1625ifndef::backend-docbook[]
1626
1627
1628endif::backend-docbook[]
1629
1630== Appendix
1631
1632=== Simulator 
1633 
1634The TRex simulator is a linux application (no DPDK needed) that can run on any Linux (it can also run on TRex machine itself).
1635you can create output pcap file from input of traffic YAML.
1636
1637====  Simulator 
1638
1639
1640[source,bash]
1641----
1642
1643$./bp-sim-64-debug -f avl/sfr_delay_10_1g.yaml -v 1
1644
1645 -- loading cap file avl/delay_10_http_get_0.pcap 
1646 -- loading cap file avl/delay_10_http_post_0.pcap 
1647 -- loading cap file avl/delay_10_https_0.pcap 
1648 -- loading cap file avl/delay_10_http_browsing_0.pcap 
1649 -- loading cap file avl/delay_10_exchange_0.pcap 
1650 -- loading cap file avl/delay_10_mail_pop_0.pcap 
1651 -- loading cap file avl/delay_10_mail_pop_1.pcap 
1652 -- loading cap file avl/delay_10_mail_pop_2.pcap 
1653 -- loading cap file avl/delay_10_oracle_0.pcap 
1654 -- loading cap file avl/delay_10_rtp_160k_full.pcap 
1655 -- loading cap file avl/delay_10_rtp_250k_full.pcap 
1656 -- loading cap file avl/delay_10_smtp_0.pcap 
1657 -- loading cap file avl/delay_10_smtp_1.pcap 
1658 -- loading cap file avl/delay_10_smtp_2.pcap 
1659 -- loading cap file avl/delay_10_video_call_0.pcap 
1660 -- loading cap file avl/delay_10_sip_video_call_full.pcap 
1661 -- loading cap file avl/delay_10_citrix_0.pcap 
1662 -- loading cap file avl/delay_10_dns_0.pcap 
1663 id,name                                    , tps, cps,f-pkts,f-bytes, duration,   Mb/sec,   MB/sec,   c-flows,  PPS,total-Mbytes-duration,errors,flows    #<2>
1664 00, avl/delay_10_http_get_0.pcap             ,404.52,404.52,    44 ,   37830 ,   0.17 , 122.42 ,   15.30 ,         67 , 17799 ,       2 , 0 , 1 
1665 01, avl/delay_10_http_post_0.pcap            ,404.52,404.52,    54 ,   48468 ,   0.21 , 156.85 ,   19.61 ,         85 , 21844 ,       2 , 0 , 1 
1666 02, avl/delay_10_https_0.pcap                ,130.87,130.87,    96 ,   91619 ,   0.22 ,  95.92 ,   11.99 ,         29 , 12564 ,       1 , 0 , 1 
1667 03, avl/delay_10_http_browsing_0.pcap        ,709.89,709.89,    37 ,   34425 ,   0.13 , 195.50 ,   24.44 ,         94 , 26266 ,       2 , 0 , 1 
1668 04, avl/delay_10_exchange_0.pcap             ,253.81,253.81,    43 ,    9848 ,   1.57 ,  20.00 ,    2.50 ,        400 , 10914 ,       0 , 0 , 1 
1669 05, avl/delay_10_mail_pop_0.pcap             ,4.76,4.76,    20 ,    5603 ,   0.17 ,   0.21 ,    0.03 ,          1 ,    95 ,       0 , 0 , 1 
1670 06, avl/delay_10_mail_pop_1.pcap             ,4.76,4.76,   114 ,  101517 ,   0.25 ,   3.86 ,    0.48 ,          1 ,   543 ,       0 , 0 , 1 
1671 07, avl/delay_10_mail_pop_2.pcap             ,4.76,4.76,    30 ,   15630 ,   0.19 ,   0.60 ,    0.07 ,          1 ,   143 ,       0 , 0 , 1 
1672 08, avl/delay_10_oracle_0.pcap               ,79.32,79.32,   302 ,   56131 ,   6.86 ,  35.62 ,    4.45 ,        544 , 23954 ,       0 , 0 , 1 
1673 09, avl/delay_10_rtp_160k_full.pcap          ,2.78,8.33,  1354 , 1232757 ,  61.24 ,  27.38 ,    3.42 ,        170 ,  3759 ,       0 , 0 , 3 
1674 10, avl/delay_10_rtp_250k_full.pcap          ,1.98,5.95,  2069 , 1922000 ,  61.38 ,  30.48 ,    3.81 ,        122 ,  4101 ,       0 , 0 , 3 
1675 11, avl/delay_10_smtp_0.pcap                 ,7.34,7.34,    22 ,    5618 ,   0.19 ,   0.33 ,    0.04 ,          1 ,   161 ,       0 , 0 , 1 
1676 12, avl/delay_10_smtp_1.pcap                 ,7.34,7.34,    35 ,   18344 ,   0.21 ,   1.08 ,    0.13 ,          2 ,   257 ,       0 , 0 , 1 
1677 13, avl/delay_10_smtp_2.pcap                 ,7.34,7.34,   110 ,   96544 ,   0.27 ,   5.67 ,    0.71 ,          2 ,   807 ,       0 , 0 , 1 
1678 14, avl/delay_10_video_call_0.pcap           ,11.90,11.90,  2325 , 2532577 ,  36.56 , 241.05 ,   30.13 ,        435 , 27662 ,       3 , 0 , 1 
1679 15, avl/delay_10_sip_video_call_full.pcap    ,29.35,58.69,  1651 ,  120315 ,  24.56 ,  28.25 ,    3.53 ,        721 , 48452 ,       0 , 0 , 2 
1680 16, avl/delay_10_citrix_0.pcap               ,43.62,43.62,   272 ,   84553 ,   6.23 ,  29.51 ,    3.69 ,        272 , 11866 ,       0 , 0 , 1 
1681 17, avl/delay_10_dns_0.pcap                  ,1975.02,1975.02,     2 ,     162 ,   0.01 ,   2.56 ,    0.32 ,         22 ,  3950 ,       0 , 0 , 1 
1682
1683 00, sum                                      ,4083.86,93928.84,  8580 , 6413941 ,   0.00 , 997.28 ,  124.66 ,       2966 , 215136 ,      12 , 0 , 23 
1684 Memory usage 
1685 size_64        : 1687 
1686 size_128       : 222 
1687 size_256       : 798 
1688 size_512       : 1028 
1689 size_1024      : 86 
1690 size_2048      : 4086 
1691 Total    :       8.89 Mbytes  159% util #<1>
1692
1693----
1694<1> the memory usage of the templates 
1695<2> CSV for all the templates 
1696
1697
1698=== firmware update to XL710/X710 
1699anchor:xl710-firmware[]
1700 
1701To upgrade the firmware  follow  this
1702
1703==== Download the driver 
1704
1705*Download driver i40e from link:https://downloadcenter.intel.com/download/24411/Network-Adapter-Driver-for-PCI-E-40-Gigabit-Network-Connections-under-Linux-[here]
1706*Build the kernel module
1707
1708[source,bash]
1709----
1710$tar -xvzf i40e-1.3.47
1711$cd i40e-1.3.47/src
1712$make 
1713$sudo insmod  i40e.ko
1714----
1715
1716
1717====  Bind the NIC to Linux
1718
1719In this stage we bind the NIC to Linux (take it from DPDK)
1720
1721[source,bash]
1722----
1723$sudo ./dpdk_nic_bind.py --status # show the ports 
1724
1725Network devices using DPDK-compatible driver
1726============================================
17270000:02:00.0 'Device 1583' drv=igb_uio unused=      #<1>
17280000:02:00.1 'Device 1583' drv=igb_uio unused=      #<2>
17290000:87:00.0 'Device 1583' drv=igb_uio unused=
17300000:87:00.1 'Device 1583' drv=igb_uio unused=
1731
1732$sudo dpdk_nic_bind.py -u 02:00.0  02:00.1          #<3> 
1733
1734$sudo dpdk_nic_bind.py -b i40e 02:00.0 02:00.1      #<4>
1735
1736$ethtool -i p1p2                                    #<5>  
1737
1738driver: i40e
1739version: 1.3.47
1740firmware-version: 4.24 0x800013fc 0.0.0             #<6>
1741bus-info: 0000:02:00.1
1742supports-statistics: yes
1743supports-test: yes
1744supports-eeprom-access: yes
1745supports-register-dump: yes
1746supports-priv-flags: yes
1747
1748   
1749$ethtool -S p1p2 
1750$lspci -s 02:00.0 -vvv                              #<7>
1751
1752
1753----
1754<1> XL710 ports that need to unbind from DPDK
1755<2> XL710 ports that need to unbind from DPDK
1756<3> Unbind from DPDK using this command
1757<4> Bind to linux to i40e driver 
1758<5> Show firmware version throw linux driver 
1759<6> Firmare version
1760<7> More info 
1761
1762
1763====  Upgrade 
1764
1765Download NVMUpdatePackage.zip from Intel site link:http://downloadcenter.intel.com/download/24769/NVM-Update-Utility-for-Intel-Ethernet-Converged-Network-Adapter-XL710-X710-Series[here]  
1766It includes the utility `nvmupdate64e`
1767
1768Run this:
1769
1770[source,bash]
1771----
1772$sudo ./nvmupdate64e  
1773----
1774
1775You might need a power cycle and to run this command a few times to get the latest firmware  
1776
1777====  QSFP+ support for XL710
1778
1779see link:https://www.google.co.il/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&cad=rja&uact=8&ved=0ahUKEwjJhPSH3b3LAhUp7nIKHSkACUYQFggaMAA&url=http%3A%2F%2Fwww.intel.co.id%2Fcontent%2Fdam%2Fwww%2Fpublic%2Fus%2Fen%2Fdocuments%2Frelease-notes%2Fxl710-ethernet-controller-feature-matrix.pdf&usg=AFQjCNFhwozfz-XuKGMOy9_MJDbetw15Og&sig2=ce7YU9F9Et6xf6KvqSFBxg&bvm=bv.116636494,d.bGs[QSFP+ support] for QSFP+ support and Firmware requirement for XL710
1780
1781
1782=== TRex with ASA 5585
1783
1784When running TRex aginst ASA 5585, you have to notice following things:
1785
1786* ASA can't forward ipv4 options, so there is a need to use --learn-mode 1 (or 3) in case of NAT. In this mode, bidirectional UDP flows are not supported.
1787--learn-mode 1 support TCP sequence number randomization in both sides of the connection (client to server and server client). For this to work, TRex must learn
1788the translation of packets from both sides, so this mode reduce the amount of connections per second TRex can generate (The number is still high enough to test
1789any existing firewall). If you need higher cps rate, you can use --learn-mode 3. This mode handles sequence number randomization on client->server side only.
1790* Latency should be tested using ICMP with `--l-pkt-mode 2`
1791
1792====  ASA 5585 sample configuration
1793
1794[source,bash]
1795----
1796ciscoasa# show running-config 
1797: Saved
1798
1799: 
1800: Serial Number: JAD194801KX
1801: Hardware:   ASA5585-SSP-10, 6144 MB RAM, CPU Xeon 5500 series 2000 MHz, 1 CPU (4 cores)
1802:
1803ASA Version 9.5(2) 
1804!
1805hostname ciscoasa
1806enable password 8Ry2YjIyt7RRXU24 encrypted
1807passwd 2KFQnbNIdI.2KYOU encrypted
1808names
1809!
1810interface Management0/0
1811 management-only
1812 nameif management
1813 security-level 100
1814 ip address 10.56.216.106 255.255.255.0 
1815!             
1816interface TenGigabitEthernet0/8
1817 nameif inside
1818 security-level 100
1819 ip address 15.0.0.1 255.255.255.0 
1820!             
1821interface TenGigabitEthernet0/9
1822 nameif outside
1823 security-level 0
1824 ip address 40.0.0.1 255.255.255.0 
1825!             
1826boot system disk0:/asa952-smp-k8.bin
1827ftp mode passive
1828pager lines 24
1829logging asdm informational
1830mtu management 1500
1831mtu inside 9000
1832mtu outside 9000
1833no failover   
1834no monitor-interface service-module 
1835icmp unreachable rate-limit 1 burst-size 1
1836no asdm history enable
1837arp outside 40.0.0.2 90e2.baae.87d1 
1838arp inside 15.0.0.2 90e2.baae.87d0 
1839arp timeout 14400
1840no arp permit-nonconnected
1841route management 0.0.0.0 0.0.0.0 10.56.216.1 1
1842route inside 16.0.0.0 255.0.0.0 15.0.0.2 1
1843route outside 48.0.0.0 255.0.0.0 40.0.0.2 1
1844timeout xlate 3:00:00
1845timeout pat-xlate 0:00:30
1846timeout conn 1:00:00 half-closed 0:10:00 udp 0:02:00 sctp 0:02:00 icmp 0:00:02
1847timeout sunrpc 0:10:00 h323 0:05:00 h225 1:00:00 mgcp 0:05:00 mgcp-pat 0:05:00
1848timeout sip 0:30:00 sip_media 0:02:00 sip-invite 0:03:00 sip-disconnect 0:02:00
1849timeout sip-provisional-media 0:02:00 uauth 0:05:00 absolute
1850timeout tcp-proxy-reassembly 0:01:00
1851timeout floating-conn 0:00:00
1852user-identity default-domain LOCAL
1853http server enable
1854http 192.168.1.0 255.255.255.0 management
1855no snmp-server location
1856no snmp-server contact
1857crypto ipsec security-association pmtu-aging infinite
1858crypto ca trustpool policy
1859telnet 0.0.0.0 0.0.0.0 management
1860telnet timeout 5
1861ssh stricthostkeycheck
1862ssh timeout 5 
1863ssh key-exchange group dh-group1-sha1
1864console timeout 0
1865!             
1866tls-proxy maximum-session 1000
1867!             
1868threat-detection basic-threat
1869threat-detection statistics access-list
1870no threat-detection statistics tcp-intercept
1871dynamic-access-policy-record DfltAccessPolicy
1872!             
1873class-map icmp-class
1874 match default-inspection-traffic
1875class-map inspection_default
1876 match default-inspection-traffic
1877!             
1878!             
1879policy-map type inspect dns preset_dns_map
1880 parameters   
1881  message-length maximum client auto
1882  message-length maximum 512
1883policy-map icmp_policy
1884 class icmp-class
1885  inspect icmp 
1886policy-map global_policy
1887 class inspection_default
1888  inspect dns preset_dns_map 
1889  inspect ftp 
1890  inspect h323 h225 
1891  inspect h323 ras 
1892  inspect rsh 
1893  inspect rtsp 
1894  inspect esmtp 
1895  inspect sqlnet 
1896  inspect skinny  
1897  inspect sunrpc 
1898  inspect xdmcp 
1899  inspect sip  
1900  inspect netbios 
1901  inspect tftp 
1902  inspect ip-options 
1903!             
1904service-policy global_policy global
1905service-policy icmp_policy interface outside
1906prompt hostname context 
1907!             
1908jumbo-frame reservation
1909!             
1910no call-home reporting anonymous
1911: end         
1912ciscoasa# 
1913----
1914
1915====  TRex commands example 
1916
1917Using these commands the configuration is:
1918
19191. NAT learn mode (TCP-ACK)
19202. Delay of 1 second at start up (-k 1). It was added because ASA drops the first packets.
19213. Latency is configured to ICMP reply mode (--l-pkt-mode 2).
1922
1923
1924*Simple HTTP:*::
1925[source,bash]
1926----
1927$sudo ./t-rex-64 -f cap2/http_simple.yaml -d 1000 -l 1000 --l-pkt-mode 2 -m 1000  --learn-mode 1 -k 1
1928----
1929
1930This is more realistic traffic for enterprise (we removed from SFR file the bidirectional UDP traffic templates, which (as described above), are not supported in this mode).
1931
1932*Enterprise profile:*::
1933[source,bash]
1934----
1935$sudo ./t-rex-64 -f avl/sfr_delay_10_1g_asa_nat.yaml -d 1000 -l 1000 --l-pkt-mode 2 -m 4 --learn-mode 1 -k 1
1936----
1937
1938The TRex output
1939
1940[source,bash]
1941----
1942-Per port stats table 
1943      ports |               0 |               1 
1944 -----------------------------------------------------------------------------------------
1945   opackets |       106347896 |       118369678 
1946     obytes |     33508291818 |    118433748567 
1947   ipackets |       118378757 |       106338782 
1948     ibytes |    118434305375 |     33507698915 
1949    ierrors |               0 |               0 
1950    oerrors |               0 |               0 
1951      Tx Bw |     656.26 Mbps |       2.27 Gbps 
1952
1953-Global stats enabled 
1954 Cpu Utilization : 18.4  %  31.7 Gb/core 
1955 Platform_factor : 1.0  
1956 Total-Tx        :       2.92 Gbps   NAT time out    :        0 #<1> (0 in wait for syn+ack) #<1>
1957 Total-Rx        :       2.92 Gbps   NAT aged flow id:        0 #<1>
1958 Total-PPS       :     542.29 Kpps   Total NAT active:      163  (12 waiting for syn)
1959 Total-CPS       :       8.30 Kcps   Nat_learn_errors:        0
1960
1961 Expected-PPS    :     539.85 Kpps
1962 Expected-CPS    :       8.29 Kcps  
1963 Expected-BPS    :       2.90 Gbps  
1964
1965 Active-flows    :     7860  Clients :      255   Socket-util : 0.0489 %    
1966 Open-flows      :  3481234  Servers :     5375   Socket :     7860 Socket/Clients :  30.8 
1967 drop-rate       :       0.00  bps   #<1>
1968 current time    : 425.1 sec  
1969 test duration   : 574.9 sec  
1970
1971-Latency stats enabled 
1972 Cpu Utilization : 0.3 %  
1973 if|   tx_ok , rx_ok  , rx   ,error,    average   ,   max         , Jitter ,  max window 
1974   |         ,        , check,     , latency(usec),latency (usec) ,(usec)  ,             
1975 ---------------------------------------------------------------------------------------------------------------- 
1976 0 |   420510,  420495,         0,   1,         58  ,    1555,      14      |  240  257  258  258  219  930  732  896  830  472  190  207  729 
1977 1 |   420496,  420509,         0,   1,         51  ,    1551,      13      |  234  253  257  258  214  926  727  893  826  468  187  204  724
1978----
1979<1>  These counters should be zero 
1980
1981anchor:fedora21_example[]
1982
1983=== Fedora 21 Server installation
1984
1985Download the .iso file from link above, boot with it using Hypervisor or CIMC console. +
1986Troubleshooting -> install in basic graphics mode
1987
1988* In packages selection, choose:
1989
1990** C Development Tools and Libraries
1991
1992** Development Tools
1993
1994** System Tools
1995
1996* Set Ethernet configuration if needed
1997
1998* Use default hard-drive partitions, reclaim space if needed
1999
2000* After installation, edit file /etc/selinux/config +
2001set: +
2002SELINUX=disabled
2003
2004* Run: +
2005systemctl disable firewalld 
2006
2007* Edit file /etc/yum.repos.d/fedora-updates.repo +
2008set everywhere: +
2009enabled=0
2010
2011* Reboot
2012
2013=== Configure Linux host as network emulator
2014
2015There are lots of Linux tutorials on the web, so this will not be full tutorial, only highlighting some key points. Commands
2016were checked on Ubuntu system.
2017
2018For this example:
2019
20201. TRex Client side network is 16.0.0.x 
20212. TRex Server side network is 48.0.0.x 
20223. Linux Client side network eth0 is configured with IPv4 as 172.168.0.1 
20234. Linux Server side network eth1 is configured with IPv4 as 10.0.0.1 
2024
2025[source,bash]
2026----
2027
2028  TRex-0 (16.0.0.1->48.0.0.1 )   <-->
2029
2030                ( 172.168.0.1/255.255.0.0)-eth0 [linux] -( 10.0.0.1/255.255.0.0)-eth1 
2031
2032                <--> TRex-1 (16.0.0.1<-48.0.0.1)
2033  
2034----
2035
2036
2037==== Enable forwarding
2038One time (will be discarded after reboot): +
2039
2040[source,bash]
2041----
2042echo 1 > /proc/sys/net/ipv4/ip_forward
2043----
2044To make this permanent, add the following line to the file /etc/sysctl.conf: +
2045----
2046net.ipv4.ip_forward=1
2047----
2048
2049==== Add static routes
2050Example if for the default TRex networks, 48.0.0.0 and 16.0.0.0.
2051
2052Routing all traffic from 48.0.0.0 to the gateway 10.0.0.100
2053[source,bash]
2054----
2055route add -net 48.0.0.0 netmask 255.255.0.0 gw 10.0.0.100
2056----
2057
2058Routing all traffic from 16.0.0.0 to the gateway 172.168.0.100
2059[source,bash]
2060----
2061route add -net 16.0.0.0 netmask 255.255.0.0 gw 172.168.0.100
2062----
2063If you use stateless mode, and decide to add route only in one direction, remember to disable reverse path check. +
2064For example, to disable on all interfaces:
2065[source,bash]
2066----
2067for i in /proc/sys/net/ipv4/conf/*/rp_filter ; do
2068  echo 0 > $i 
2069done
2070----
2071
2072Alternatively, you can edit /etc/network/interfaces, and add something like this for both ports connected to TRex.
2073This will take effect, only after restarting networking (rebooting the machine in an alternative also).
2074----
2075auto eth1
2076iface eth1 inet static
2077address 16.0.0.100
2078netmask 255.0.0.0
2079network 16.0.0.0
2080broadcast 16.255.255.255
2081... same for 48.0.0.0
2082----
2083
2084==== Add static ARP entries
2085[source,bash]
2086----
2087sudo arp -s 10.0.0.100 <Second TRex port MAC>
2088sudo arp -s 172.168.0.100 <TRex side the NICs are not visible to ifconfig, run:
2089----
2090
2091
2092=== Mellanox ConnectX-4 support 
2093
2094anchor:connectx_support[]
2095
2096Mellanox ConnectX-4 adapter family supports 100/56/40/25/10 Gb/s Ethernet speeds. 
2097Its DPDK support is a bit different from Intel DPDK support, more information can be found link:http://dpdk.org/doc/guides/nics/mlx5.html[DPDK support].
2098Intel NICs does not require kernel drivers (except dpdk igb_uio which already supported) while ConnectX-4 works on top of Infinibad API (verbs) and require a kernel modules/user space libs. 
2099This means that it is required to install OFED package to be able to work with the NIC.
2100Installing OFED is the simplest way to make it work (trying to install part of the package can work too but didn't work for us).
2101The advantage of this model that you can control it throw Linux driver (ethtol can still work, you will be able to ifconfig it).
2102The disadvantage is the OFED dependency.
2103
2104==== Installation  
2105
2106==== Install Linux 
2107
2108The following distro were tested with TRex and OFED, others might work
2109
2110* Ubuntu 14.04.3 LTS (GNU/Linux 3.19.0-25-generic x86_64) 
2111
2112
2113The following distro were  tested and did *not* work for us
2114
2115* Fedora 21 (3.17.4-301.fc21.x86_64)  
2116
2117==== Install OFED 
2118
2119The information was taken from  link:http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers[Install OFED]
2120
2121.Download 3.4-1 OFED tar for your distro  
2122link::http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers[download]
2123
2124[IMPORTANT]
2125=====================================
2126it must be version *MLNX_OFED_LINUX-3.4-1*
2127=====================================
2128
2129[IMPORTANT]
2130=====================================
2131Make sure you have an internet connection without firewalls for HTTPS/HTTP - required by yum/apt-get
2132=====================================
2133
2134.Verify md5
2135[source,bash]
2136----
2137$md5sum MLNX_OFED_LINUX-3.4-1.0.0.0-ubuntu14.04-x86_64.tgz
2138b3c17dc0ea64fd1f0892d7b7ba7e45f3  MLNX_OFED_LINUX-3.4-1.0.0.0-ubuntu14.04-x86_64.tgz
2139----
2140
2141.Open the tar
2142[source,bash]
2143----
2144$tar -xvzf MLNX_OFED_LINUX-3.4-1.0.0.0-ubuntu14.04-x86_64.tgz
2145$cd MLNX_OFED_LINUX-3.4-1.0.0.0-ubuntu14.04-x86_64
2146----
2147
2148.Run Install script
2149[source,bash]
2150----
2151$sudo ./mlnxofedinstall  
2152
2153
2154Log: /tmp/ofed.build.log
2155Logs dir: /tmp/MLNX_OFED_LINUX.10406.logs
2156
2157Below is the list of MLNX_OFED_LINUX packages that you have chosen
2158(some may have been added by the installer due to package dependencies):
2159
2160ofed-scripts
2161mlnx-ofed-kernel-utils
2162mlnx-ofed-kernel-dkms
2163iser-dkms
2164srp-dkms
2165mlnx-sdp-dkms
2166mlnx-rds-dkms
2167mlnx-nfsrdma-dkms
2168libibverbs1
2169ibverbs-utils
2170libibverbs-dev
2171libibverbs1-dbg
2172libmlx4-1
2173libmlx4-dev
2174libmlx4-1-dbg
2175libmlx5-1
2176libmlx5-dev
2177libmlx5-1-dbg
2178libibumad
2179libibumad-static
2180libibumad-devel
2181ibacm
2182ibacm-dev
2183librdmacm1
2184librdmacm-utils
2185librdmacm-dev
2186mstflint
2187ibdump
2188libibmad
2189libibmad-static
2190libibmad-devel
2191libopensm
2192opensm
2193opensm-doc
2194libopensm-devel
2195infiniband-diags
2196infiniband-diags-compat
2197mft
2198kernel-mft-dkms
2199libibcm1
2200libibcm-dev
2201perftest
2202ibutils2
2203libibdm1
2204ibutils
2205cc-mgr
2206ar-mgr
2207dump-pr
2208ibsim
2209ibsim-doc
2210knem-dkms
2211mxm
2212fca
2213sharp
2214hcoll
2215openmpi
2216mpitests
2217knem
2218rds-tools
2219libdapl2
2220dapl2-utils
2221libdapl-dev
2222srptools
2223mlnx-ethtool
2224libsdp1
2225libsdp-dev
2226sdpnetstat
2227
2228This program will install the MLNX_OFED_LINUX package on your machine.
2229Note that all other Mellanox, OEM, OFED, or Distribution IB packages will be removed.
2230Do you want to continue?[y/N]:y
2231
2232Checking SW Requirements...
2233
2234One or more required packages for installing MLNX_OFED_LINUX are missing.
2235Attempting to install the following missing packages:
2236autotools-dev tcl debhelper dkms tk8.4 libgfortran3 graphviz chrpath automake dpatch flex bison autoconf quilt m4 tcl8.4 libltdl-dev pkg-config pytho
2237bxml2 tk swig gfortran libnl1
2238
2239..
2240
2241Removing old packages...
2242Installing new packages
2243Installing ofed-scripts-3.4...
2244Installing mlnx-ofed-kernel-utils-3.4...
2245Installing mlnx-ofed-kernel-dkms-3.4...
2246
2247Removing old packages...
2248Installing new packages
2249Installing ofed-scripts-3.4...
2250Installing mlnx-ofed-kernel-utils-3.4...
2251Installing mlnx-ofed-kernel-dkms-3.4...
2252Installing iser-dkms-1.8.1...
2253Installing srp-dkms-1.6.1...
2254Installing mlnx-sdp-dkms-3.4...
2255Installing mlnx-rds-dkms-3.4...
2256Installing mlnx-nfsrdma-dkms-3.4...
2257Installing libibverbs1-1.2.1mlnx1...
2258Installing ibverbs-utils-1.2.1mlnx1...
2259Installing libibverbs-dev-1.2.1mlnx1...
2260Installing libibverbs1-dbg-1.2.1mlnx1...
2261Installing libmlx4-1-1.2.1mlnx1...
2262Installing libmlx4-dev-1.2.1mlnx1...
2263Installing libmlx4-1-dbg-1.2.1mlnx1...
2264Installing libmlx5-1-1.2.1mlnx1...
2265Installing libmlx5-dev-1.2.1mlnx1...
2266Installing libmlx5-1-dbg-1.2.1mlnx1...
2267Installing libibumad-1.3.10.2.MLNX20150406.966500d...
2268Installing libibumad-static-1.3.10.2.MLNX20150406.966500d...
2269Installing libibumad-devel-1.3.10.2.MLNX20150406.966500d...
2270Installing ibacm-1.2.1mlnx1...
2271Installing ibacm-dev-1.2.1mlnx1...
2272Installing librdmacm1-1.1.0mlnx...
2273Installing librdmacm-utils-1.1.0mlnx...
2274Installing librdmacm-dev-1.1.0mlnx...
2275Installing mstflint-4.5.0...
2276Installing ibdump-4.0.0...
2277Installing libibmad-1.3.12.MLNX20160814.4f078cc...
2278Installing libibmad-static-1.3.12.MLNX20160814.4f078cc...
2279Installing libibmad-devel-1.3.12.MLNX20160814.4f078cc...
2280Installing libopensm-4.8.0.MLNX20160906.32a95b6...
2281Installing opensm-4.8.0.MLNX20160906.32a95b6...
2282Installing opensm-doc-4.8.0.MLNX20160906.32a95b6...
2283Installing libopensm-devel-4.8.0.MLNX20160906.32a95b6...
2284Installing infiniband-diags-1.6.6.MLNX20160814.999c7b2...
2285Installing infiniband-diags-compat-1.6.6.MLNX20160814.999c7b2...
2286Installing mft-4.5.0...
2287Installing kernel-mft-dkms-4.5.0...
2288Installing libibcm1-1.0.5mlnx2...
2289Installing libibcm-dev-1.0.5mlnx2...
2290Installing perftest-3.0...
2291Installing ibutils2-2.1.1...
2292Installing libibdm1-1.5.7.1...
2293Installing ibutils-1.5.7.1...
2294Installing cc-mgr-1.0...
2295Installing ar-mgr-1.0...
2296Installing dump-pr-1.0...
2297Installing ibsim-0.6...
2298Installing ibsim-doc-0.6...
2299Installing knem-dkms-1.1.2.90mlnx1...
2300Installing mxm-3.5.220c57f...
2301Installing fca-2.5.2431...
2302Installing sharp-1.1.1.MLNX20160915.8763a35...
2303Installing hcoll-3.6.1228...
2304Installing openmpi-1.10.5a1...
2305Installing mpitests-3.2.18...
2306Installing knem-1.1.2.90mlnx1...
2307Installing rds-tools-2.0.7...
2308Installing libdapl2-2.1.9mlnx...
2309Installing dapl2-utils-2.1.9mlnx...
2310Installing libdapl-dev-2.1.9mlnx...
2311Installing srptools-1.0.3...
2312Installing mlnx-ethtool-4.2...
2313Installing libsdp1-1.1.108...
2314Installing libsdp-dev-1.1.108...
2315Installing sdpnetstat-1.60...
2316Selecting previously unselected package mlnx-fw-updater.
2317(Reading database ... 70592 files and directories currently installed.)
2318Preparing to unpack .../mlnx-fw-updater_3.4-1.0.0.0_amd64.deb ...
2319Unpacking mlnx-fw-updater (3.4-1.0.0.0) ...
2320Setting up mlnx-fw-updater (3.4-1.0.0.0) ...
2321
2322Added RUN_FW_UPDATER_ONBOOT=no to /etc/infiniband/openib.conf
2323
2324Attempting to perform Firmware update...
2325Querying Mellanox devices firmware ...
2326
2327Device #1:
2328
2329  Device Type:      ConnectX4
2330  Part Number:      MCX416A-CCA_Ax
2331  Description:      ConnectX-4 EN network interface card; 100GbE dual-port QSFP28; PCIe3.0 x16; ROHS R6
2332  PSID:             MT_2150110033
2333  PCI Device Name:  03:00.0
2334  Base GUID:        248a07030014fc60
2335  Base MAC:         0000248a0714fc60
2336  Versions:         Current        Available     
2337     FW             12.16.1006     12.17.1010    
2338     PXE            3.4.0812       3.4.0903      
2339
2340  Status:           Update required
2341
2342
2343Found 1 device(s) requiring firmware update...
2344
2345Device #1: Updating FW ... Done
2346
2347Restart needed for updates to take effect.
2348Log File: /tmp/MLNX_OFED_LINUX.16084.logs/fw_update.log
2349Please reboot your system for the changes to take effect.
2350Device (03:00.0):
2351        03:00.0 Ethernet controller: Mellanox Technologies MT27620 Family
2352        Link Width: x16
2353        PCI Link Speed: 8GT/s
2354
2355Device (03:00.1):
2356        03:00.1 Ethernet controller: Mellanox Technologies MT27620 Family
2357        Link Width: x16
2358        PCI Link Speed: 8GT/s
2359
2360Installation passed successfully
2361To load the new driver, run:
2362/etc/init.d/openibd restart
2363-----
2364
2365
2366.Reboot 
2367[source,bash]
2368----
2369$sudo  reboot 
2370----
2371
2372
2373.After reboot 
2374[source,bash]
2375----
2376$ibv_devinfo 
2377hca_id: mlx5_1
2378        transport:                      InfiniBand (0)
2379        fw_ver:                         12.17.1010             << 12.17.00
2380        node_guid:                      248a:0703:0014:fc61
2381        sys_image_guid:                 248a:0703:0014:fc60
2382        vendor_id:                      0x02c9
2383        vendor_part_id:                 4115
2384        hw_ver:                         0x0
2385        board_id:                       MT_2150110033
2386        phys_port_cnt:                  1
2387        Device ports:
2388                port:   1
2389                        state:                  PORT_DOWN (1)
2390                        max_mtu:                4096 (5)
2391                        active_mtu:             1024 (3)
2392                        sm_lid:                 0
2393                        port_lid:               0
2394                        port_lmc:               0x00
2395                        link_layer:             Ethernet
2396
2397hca_id: mlx5_0
2398        transport:                      InfiniBand (0)
2399        fw_ver:                         12.17.1010
2400        node_guid:                      248a:0703:0014:fc60
2401        sys_image_guid:                 248a:0703:0014:fc60
2402        vendor_id:                      0x02c9
2403        vendor_part_id:                 4115
2404        hw_ver:                         0x0
2405        board_id:                       MT_2150110033
2406        phys_port_cnt:                  1
2407        Device ports:
2408                port:   1
2409                        state:                  PORT_DOWN (1)
2410                        max_mtu:                4096 (5)
2411                        active_mtu:             1024 (3)
2412                        sm_lid:                 0
2413                        port_lid:               0
2414                        port_lmc:               0x00
2415                        link_layer:             Ethernet
2416
2417----
2418
2419.ibdev2netdev
2420[source,bash]
2421-----
2422$ibdev2netdev
2423mlx5_0 port 1 ==> eth6 (Down)
2424mlx5_1 port 1 ==> eth7 (Down)
2425-----
2426
2427
2428==== TRex specific implementation details 
2429
2430TRex uses flow director filter to steer specific packets to specific queues. 
2431To support that we change IPv4.TOS/Ipv6.TC  LSB to *1* to be steered. So latency packets will have this bit turn on (not only for ConnectX-4)
2432Watch out, In case DUT will clear this bit (change the TOS with LSB==0, e.g. 0x3->0x2) packets won't be forward to TRex.
2433
2434==== Which NIC to buy?
2435
2436NIC with two ports will work better from performance prospective, so it is better to have MCX456A-ECAT(two 100gb port) and *not* the  MCX455A-ECAT (one 100gb port).
2437
2438==== Limitation/Issues  
2439
2440* Stateless per stream statistic is not supported yet
2441* link:https://trex-tgn.cisco.com/youtrack/issue/trex-260[64B performance issue]
2442* link:https://trex-tgn.cisco.com/youtrack/issue/trex-261[Latency issue]
2443* link:https://trex-tgn.cisco.com/youtrack/issue/trex-262[Statful RX out of order]
2444* link:https://trex-tgn.cisco.com/youtrack/issue/trex-273[Fedora 21 & OFED 3.4.1]
2445
2446
2447==== Performance Cycles/Packet ConnectX-4 vs Intel XL710
2448
2449For version TRex v2.11, this is the comparison results between XL710 and ConnectX-4 for various scenarios
2450
2451.Stateless MPPS/Core [Perlimeniary]
2452image:images/xl710_vs_mlx5_64b.png[title="Stateless 64B"] 
2453
2454.Stateless Gb/Core [Perlimeniary]
2455image:images/xl710_vs_mlx5_var_size.png[title="Stateless variable size packet"] 
2456
2457*Comments*::
2458 
24591. For Stateless 64B profiles ConnectX-4 cost 50-90% more cycles per packet (it is actually even more because there is the TRex scheduler overhead) - it means that in the worst case scenario you will need x2 CPU for the same total MPPS
24602. For Stateless/Stateful 256B profiles, ConnectX-4 cost half of the cycles per packets. ConnectX-4 probably can handle in a better way chained mbuf (scatter gather).
24613. In Average Stateful senario ConnectX-4 will be slightly better.
24624. MLX5 can reach ~90MPPS while XL710 limited to 35MPPS 
2463
2464
2465[NOTE]
2466=====================================
2467There is a task to automate the production of this reports
2468=====================================
2469
2470==== Troubleshooting
2471
2472* Before running TRex make sure the commands    `ibv_devinfo` and  `ibdev2netdev` present the NICS
2473* `ifconfig` should work too, you need to be able to ping from those ports
2474* run TRex server with '-v 7' for example `$sudo ./t-rex-64 -i -v 7`
2475
2476
2477=== Cisco VIC support 
2478
2479anchor:ciscovic_support[]
2480
2481* Supported from TRex version v2.11 
2482* Only 1300 series Cisco adapter 
2483* Firmware version 2.0(13) for UCS C-series servers, GA in Febuary 2017. 
2484* Firmware version 3.1(2) for blade servers supports more filtering capabilities. 
2485* The feature can be enabled via Cisco CIMC or USCM with the 'advanced filters' radio button.  When enabled, the these additional flow director modes are available:
2486        RTE_ETH_FLOW_NONFRAG_IPV4_OTHER
2487        RTE_ETH_FLOW_NONFRAG_IPV4_SCTP
2488        RTE_ETH_FLOW_NONFRAG_IPV6_UDP
2489        RTE_ETH_FLOW_NONFRAG_IPV6_TCP
2490        RTE_ETH_FLOW_NONFRAG_IPV6_SCTP
2491        RTE_ETH_FLOW_NONFRAG_IPV6_OTHER
2492
2493
2494==== Limitation/Issues  
2495
2496* Stateless per stream statistic is not supported yet
2497* link:https://trex-tgn.cisco.com/youtrack/issue/trex-265[flow-director] 
2498* link:https://trex-tgn.cisco.com/youtrack/issue/trex-272[QSFP+ issue]
2499
2500
2501
2502