xref: /csit/
NameDateSize

..13-Mar-2018256

.gitattributesH A D11-Jul-2018438

.gitignoreH A D09-Jan-2020775

.gitreviewH A D24-May-201876

docs/H10-Mar-20204 KiB

DPDK_VPP_VERH A D27-Aug-201911

INFO.yamlH A D28-Jan-20202.3 KiB

LICENSEH A D03-Jul-201911.1 KiB

NOTICEH A D03-Jul-20191.2 KiB

pylint.cfgH A D05-Dec-20199 KiB

PyPI/H18-Jun-201874

README.mdH A D16-Jan-20206.7 KiB

requirements.txtH A D16-Dec-20192 KiB

resources/H05-Aug-2019171

tests/H15-Jan-202029

topologies/H16-May-201954

tox-requirements.txtH A D27-Feb-2019808

tox.iniH A D21-Feb-20203.6 KiB

VPP_DEVICE_IMAGE_CENTOSH A D17-Oct-201928

VPP_DEVICE_IMAGE_UBUNTUH A D05-Jun-201926

VPP_DEVICE_IMAGE_UBUNTU_ARMH A D24-Jul-201930

VPP_REPO_URLH A D13-Sep-201857

VPP_STABLE_VER_CENTOSH A D27-Mar-202029

VPP_STABLE_VER_UBUNTU_BIONICH A D27-Mar-202025

README.md

1# CSIT - Continuous System Integration Testing
2
31. [Architecture](#architecture)
41. [Directory Structure](#directory-structure)
5   1. [Tests](#tests)
6   1. [Keywords](#keywords)
7   1. [Other Resources](#other-resources)
81. [Quickstart](#quick-start)
9   1. [Vagrant](#vagrant)
10   1. [Physical Testbed](#physical-testbed)
111. [Report](#report)
121. [Trending](#trending)
131. [Code Documentation](#code-documentation)
141. [Coding Guidelines](#coding-guidelines)
15
16## Architecture
17
18FD.io CSIT system design needs to meet continuously expanding requirements of
19FD.io projects including VPP, related sub-systems (e.g. plugin applications,
20DPDK drivers) and FD.io applications (e.g. DPDK applications), as well as
21growing number of compute platforms running those applications. With CSIT
22project scope and charter including both FD.io continuous testing AND
23performance trending/comparisons, those evolving requirements further amplify
24the need for CSIT framework modularity, flexibility and usability.
25
26CSIT follows a hierarchical system design with SUTs and DUTs at the bottom level
27of the hierarchy, presentation level at the top level and a number of functional
28layers in-between. The current CSIT system design including CSIT framework is
29depicted in the figure below.
30
31![csit design](docs/report/csit_framework_documentation/csit_design_picture.svg "CSIT architecture")
32
33A brief bottom-up description is provided here:
34
351. SUTs, DUTs, TGs
36   - SUTs - Systems Under Test;
37   - DUTs - Devices Under Test;
38   - TGs - Traffic Generators;
391. Level-1 libraries - Robot and Python
40   - Lowest level CSIT libraries abstracting underlying test environment, SUT,
41     DUT and TG specifics;
42   - Used commonly across multiple L2 KWs;
43   - Performance and functional tests:
44     - L1 KWs (KeyWords) are implemented as RF libraries and Python
45       libraries;
46   - Performance TG L1 KWs:
47     - All L1 KWs are implemented as Python libraries:
48       - Support for TRex only today;
49   - Performance data plane traffic profiles:
50     - TG-specific stream profiles provide full control of:
51       - Packet definition – layers, MACs, IPs, ports, combinations thereof
52         e.g. IPs and UDP ports;
53       - Stream definitions - different streams can run together, delayed,
54         one after each other;
55       - Stream profiles are independent of CSIT framework and can be used
56         in any T-rex setup, can be sent anywhere to repeat tests with
57         exactly the same setup;
58       - Easily extensible – one can create a new stream profile that meets
59         tests requirements;
60       - Same stream profile can be used for different tests with the same
61         traffic needs;
62   - Functional data plane traffic scripts:
63     - Scapy specific traffic scripts;
641. Level-2 libraries - Robot resource files
65   - Higher level CSIT libraries abstracting required functions for executing
66     tests;
67   - L2 KWs are classified into the following functional categories:
68     - Configuration, test, verification, state report;
69     - Suite setup, suite teardown;
70     - Test setup, test teardown;
711. Tests - Robot
72   - Device tests using containerized environment with SR-IOV access to a NIC;
73     - VPP;
74   - Performance tests using physical testbed environment:
75     - VPP;
76     - DPDK-Testpmd;
77     - DPDK-L3Fwd;
78   - Tools:
79     - Documentation generator;
80     - Report generator;
81     - Testbed environment setup ansible playbooks;
82     - Operational debugging scripts;
83
84## Directory Structure
85
86### Tests
87
88```
89.
90└── tests
91    ├── dpdk
92    │   ├── dpdk_scripts            # DPDK helper scripts
93    │   └── perf                    # DPDK performance tests
94    └── vpp
95        ├── device                  # VPP device tests
96        └── perf                    # VPP performance tests
97```
98
99### Keywords
100
101```
102.
103resources
104└── libraries
105    ├── bash                        # Contains a dependency of KubernetesUtils
106    │   ├── config                  # Config for KubernetesUtils dependency
107    │   ├── entry                   # Main bootstrap entry directory
108    │   ├── function                # Bootstrap function library
109    │   └── shell                   # Various functions for KubernetesUtils
110    ├── python                      # Python L1 KWs
111    └── robot                       # Robot Framework L2 KWs
112```
113
114### Other Resources
115
116```
117.
118├── docs                            # Main documentaion
119├── PyPI                            # PyPI packages provided by CSIT
120│   ├── jumpavg
121│   └── MLRsearch
122├── resources
123│   ├── templates                   # Templates (vpp_api_test, kubernetes, ...)
124│   ├── test_data                   # Robot Test configuration
125│   ├── tools
126│   │   ├── disk-image-builder      # Utilities for building (DCR, VM) images
127│   │   ├── doc_gen                 # Code documentation generator
128│   │   ├── papi                    # PAPI driver
129│   │   ├── presentation            # Report generator
130│   │   ├── scripts                 # Various tools
131│   │   ├── testbed-setup           # Physical testbed setup scripts
132│   │   ├── topology                # Helper scripts for topology manipulation
133│   │   ├── trex                    # TRex driver
134│   │   ├── vagrant                 # VPP device vagrant environment
135│   │   └── wrk                     # WRK driver
136│   ├── topology_schemas
137│   ├── traffic_profiles            # Performance tests traffic profiles
138│   │   ├── trex
139│   │   └── wrk
140│   └── traffic_scripts             # Functional tests traffic profiles
141│       ├── dhcp
142│       └── lisp
143└── topologies                      # Linux Foundation topology files
144    ├── available
145    └── enabled
146```
147
148## Quickstart
149
150### Vagrant
151
152[Vagrant environment preparation](docs/testing_in_vagrant.rst) documentaion is
153describing local VPP Device functional testing.
154
155### Physical Testbed
156
157[Physical testbed preparation](resources/tools/testbed-setup/README.rst)
158documentation is describing PXE and Ansible setup process. All the software
159requirements for running Performance Teste are part of Ansible playbooks.
160
161## Report
162
163[CSIT Report](https://docs.fd.io/csit/master/report/).
164
165## Trending
166
167[CSIT Trending](https://docs.fd.io/csit/master/trending/).
168
169## Code Documentation
170
171[CSIT Code Documentation](https://docs.fd.io/csit/master/doc/).
172
173## Coding Guidelines
174
175If you are interested in contributing, please see the
176[coding guidelines](docs/test_code_guidelines.rst).
177