pal_lld.rst revision e7ad66f3
1Presentation and Analytics
2==========================
3
4Overview
5--------
6
7The presentation and analytics layer (PAL) is the fourth layer of CSIT
8hierarchy. The model of presentation and analytics layer consists of four
9sub-layers, bottom up:
10
11 - sL1 - Data - input data to be processed:
12
13   - Static content - .rst text files, .svg static figures, and other files
14     stored in the CSIT git repository.
15   - Data to process - .xml files generated by Jenkins jobs executing tests,
16     stored as robot results files (output.xml).
17   - Specification - .yaml file with the models of report elements (tables,
18     plots, layout, ...) generated by this tool. There is also the configuration
19     of the tool and the specification of input data (jobs and builds).
20
21 - sL2 - Data processing
22
23   - The data are read from the specified input files (.xml) and stored as
24     multi-indexed `pandas.Series <https://pandas.pydata.org/pandas-docs/stable/
25     generated/pandas.Series.html>`_.
26   - This layer provides also interface to input data and filtering of the input
27     data.
28
29 - sL3 - Data presentation - This layer generates the elements specified in the
30   specification file:
31
32   - Tables: .csv files linked to static .rst files.
33   - Plots: .html files generated using plot.ly linked to static .rst files.
34
35 - sL4 - Report generation - Sphinx generates required formats and versions:
36
37   - formats: html, pdf
38   - versions: minimal, full (TODO: define the names and scope of versions)
39
40.. only:: latex
41
42    .. raw:: latex
43
44        \begin{figure}[H]
45            \centering
46                \graphicspath{{../_tmp/src/csit_framework_documentation/}}
47                \includegraphics[width=0.90\textwidth]{pal_layers}
48                \label{fig:pal_layers}
49        \end{figure}
50
51.. only:: html
52
53    .. figure:: pal_layers.svg
54        :alt: PAL Layers
55        :align: center
56
57Data
58----
59
60Report Specification
61````````````````````
62
63The report specification file defines which data is used and which outputs are
64generated. It is human readable and structured. It is easy to add / remove /
65change items. The specification includes:
66
67 - Specification of the environment.
68 - Configuration of debug mode (optional).
69 - Specification of input data (jobs, builds, files, ...).
70 - Specification of the output.
71 - What and how is generated:
72   - What: plots, tables.
73   - How: specification of all properties and parameters.
74 - .yaml format.
75
76Structure of the specification file
77'''''''''''''''''''''''''''''''''''
78
79The specification file is organized as a list of dictionaries distinguished by
80the type:
81
82::
83
84    -
85      type: "environment"
86    -
87      type: "configuration"
88    -
89      type: "debug"
90    -
91      type: "static"
92    -
93      type: "input"
94    -
95      type: "output"
96    -
97      type: "table"
98    -
99      type: "plot"
100    -
101      type: "file"
102
103Each type represents a section. The sections "environment", "debug", "static",
104"input" and "output" are listed only once in the specification; "table", "file"
105and "plot" can be there multiple times.
106
107Sections "debug", "table", "file" and "plot" are optional.
108
109Table(s), files(s) and plot(s) are referred as "elements" in this text. It is
110possible to define and implement other elements if needed.
111
112
113Section: Environment
114''''''''''''''''''''
115
116This section has the following parts:
117
118 - type: "environment" - says that this is the section "environment".
119 - configuration - configuration of the PAL.
120 - paths - paths used by the PAL.
121 - urls - urls pointing to the data sources.
122 - make-dirs - a list of the directories to be created by the PAL while
123   preparing the environment.
124 - remove-dirs - a list of the directories to be removed while cleaning the
125   environment.
126 - build-dirs - a list of the directories where the results are stored.
127
128The structure of the section "Environment" is as follows (example):
129
130::
131
132    -
133      type: "environment"
134      configuration:
135        # Debug mode:
136        # - Skip:
137        #   - Download of input data files
138        # - Do:
139        #   - Read data from given zip / xml files
140        #   - Set the configuration as it is done in normal mode
141        # If the section "type: debug" is missing, CFG[DEBUG] is set to 0.
142        CFG[DEBUG]: 0
143
144      paths:
145        # Top level directories:
146        ## Working directory
147        DIR[WORKING]: "_tmp"
148        ## Build directories
149        DIR[BUILD,HTML]: "_build"
150        DIR[BUILD,LATEX]: "_build_latex"
151
152        # Static .rst files
153        DIR[RST]: "../../../docs/report"
154
155        # Working directories
156        ## Input data files (.zip, .xml)
157        DIR[WORKING,DATA]: "{DIR[WORKING]}/data"
158        ## Static source files from git
159        DIR[WORKING,SRC]: "{DIR[WORKING]}/src"
160        DIR[WORKING,SRC,STATIC]: "{DIR[WORKING,SRC]}/_static"
161
162        # Static html content
163        DIR[STATIC]: "{DIR[BUILD,HTML]}/_static"
164        DIR[STATIC,VPP]: "{DIR[STATIC]}/vpp"
165        DIR[STATIC,DPDK]: "{DIR[STATIC]}/dpdk"
166        DIR[STATIC,ARCH]: "{DIR[STATIC]}/archive"
167
168        # Detailed test results
169        DIR[DTR]: "{DIR[WORKING,SRC]}/detailed_test_results"
170        DIR[DTR,PERF,DPDK]: "{DIR[DTR]}/dpdk_performance_results"
171        DIR[DTR,PERF,VPP]: "{DIR[DTR]}/vpp_performance_results"
172        DIR[DTR,FUNC,VPP]: "{DIR[DTR]}/vpp_functional_results"
173        DIR[DTR,FUNC,NSHSFC]: "{DIR[DTR]}/nshsfc_functional_results"
174        DIR[DTR,PERF,VPP,IMPRV]: "{DIR[WORKING,SRC]}/vpp_performance_tests/performance_improvements"
175
176        # Detailed test configurations
177        DIR[DTC]: "{DIR[WORKING,SRC]}/test_configuration"
178        DIR[DTC,PERF,VPP]: "{DIR[DTC]}/vpp_performance_configuration"
179        DIR[DTC,FUNC,VPP]: "{DIR[DTC]}/vpp_functional_configuration"
180
181        # Detailed tests operational data
182        DIR[DTO]: "{DIR[WORKING,SRC]}/test_operational_data"
183        DIR[DTO,PERF,VPP]: "{DIR[DTO]}/vpp_performance_operational_data"
184
185        # .css patch file to fix tables generated by Sphinx
186        DIR[CSS_PATCH_FILE]: "{DIR[STATIC]}/theme_overrides.css"
187        DIR[CSS_PATCH_FILE2]: "{DIR[WORKING,SRC,STATIC]}/theme_overrides.css"
188
189      urls:
190        URL[JENKINS,CSIT]: "https://jenkins.fd.io/view/csit/job"
191        URL[JENKINS,HC]: "https://jenkins.fd.io/view/hc2vpp/job"
192
193      make-dirs:
194      # List the directories which are created while preparing the environment.
195      # All directories MUST be defined in "paths" section.
196      - "DIR[WORKING,DATA]"
197      - "DIR[STATIC,VPP]"
198      - "DIR[STATIC,DPDK]"
199      - "DIR[STATIC,ARCH]"
200      - "DIR[BUILD,LATEX]"
201      - "DIR[WORKING,SRC]"
202      - "DIR[WORKING,SRC,STATIC]"
203
204      remove-dirs:
205      # List the directories which are deleted while cleaning the environment.
206      # All directories MUST be defined in "paths" section.
207      #- "DIR[BUILD,HTML]"
208
209      build-dirs:
210      # List the directories where the results (build) is stored.
211      # All directories MUST be defined in "paths" section.
212      - "DIR[BUILD,HTML]"
213      - "DIR[BUILD,LATEX]"
214
215It is possible to use defined items in the definition of other items, e.g.:
216
217::
218
219    DIR[WORKING,DATA]: "{DIR[WORKING]}/data"
220
221will be automatically changed to
222
223::
224
225    DIR[WORKING,DATA]: "_tmp/data"
226
227
228Section: Configuration
229''''''''''''''''''''''
230
231This section specifies the groups of parameters which are repeatedly used in the
232elements defined later in the specification file. It has the following parts:
233
234 - data sets - Specification of data sets used later in element's specifications
235   to define the input data.
236 - plot layouts - Specification of plot layouts used later in plots'
237   specifications to define the plot layout.
238
239The structure of the section "Configuration" is as follows (example):
240
241::
242
243    -
244      type: "configuration"
245      data-sets:
246        plot-vpp-throughput-latency:
247          csit-vpp-perf-1710-all:
248          - 11
249          - 12
250          - 13
251          - 14
252          - 15
253          - 16
254          - 17
255          - 18
256          - 19
257          - 20
258        vpp-perf-results:
259          csit-vpp-perf-1710-all:
260          - 20
261          - 23
262      plot-layouts:
263        plot-throughput:
264          xaxis:
265            autorange: True
266            autotick: False
267            fixedrange: False
268            gridcolor: "rgb(238, 238, 238)"
269            linecolor: "rgb(238, 238, 238)"
270            linewidth: 1
271            showgrid: True
272            showline: True
273            showticklabels: True
274            tickcolor: "rgb(238, 238, 238)"
275            tickmode: "linear"
276            title: "Indexed Test Cases"
277            zeroline: False
278          yaxis:
279            gridcolor: "rgb(238, 238, 238)'"
280            hoverformat: ".4s"
281            linecolor: "rgb(238, 238, 238)"
282            linewidth: 1
283            range: []
284            showgrid: True
285            showline: True
286            showticklabels: True
287            tickcolor: "rgb(238, 238, 238)"
288            title: "Packets Per Second [pps]"
289            zeroline: False
290          boxmode: "group"
291          boxgroupgap: 0.5
292          autosize: False
293          margin:
294            t: 50
295            b: 20
296            l: 50
297            r: 20
298          showlegend: True
299          legend:
300            orientation: "h"
301          width: 700
302          height: 1000
303
304The definitions from this sections are used in the elements, e.g.:
305
306::
307
308    -
309      type: "plot"
310      title: "VPP Performance 64B-1t1c-(eth|dot1q|dot1ad)-(l2xcbase|l2bdbasemaclrn)-ndrdisc"
311      algorithm: "plot_performance_box"
312      output-file-type: ".html"
313      output-file: "{DIR[STATIC,VPP]}/64B-1t1c-l2-sel1-ndrdisc"
314      data:
315        "plot-vpp-throughput-latency"
316      filter: "'64B' and ('BASE' or 'SCALE') and 'NDRDISC' and '1T1C' and ('L2BDMACSTAT' or 'L2BDMACLRN' or 'L2XCFWD') and not 'VHOST'"
317      parameters:
318      - "throughput"
319      - "parent"
320      traces:
321        hoverinfo: "x+y"
322        boxpoints: "outliers"
323        whiskerwidth: 0
324      layout:
325        title: "64B-1t1c-(eth|dot1q|dot1ad)-(l2xcbase|l2bdbasemaclrn)-ndrdisc"
326        layout:
327          "plot-throughput"
328
329
330Section: Debug mode
331'''''''''''''''''''
332
333This section is optional as it configures the debug mode. It is used if one
334does not want to download input data files and use local files instead.
335
336If the debug mode is configured, the "input" section is ignored.
337
338This section has the following parts:
339
340 - type: "debug" - says that this is the section "debug".
341 - general:
342
343   - input-format - xml or zip.
344   - extract - if "zip" is defined as the input format, this file is extracted
345     from the zip file, otherwise this parameter is ignored.
346
347 - builds - list of builds from which the data is used. Must include a job
348   name as a key and then a list of builds and their output files.
349
350The structure of the section "Debug" is as follows (example):
351
352::
353
354    -
355      type: "debug"
356      general:
357        input-format: "zip"  # zip or xml
358        extract: "robot-plugin/output.xml"  # Only for zip
359      builds:
360        # The files must be in the directory DIR[WORKING,DATA]
361        csit-dpdk-perf-1707-all:
362        -
363          build: 10
364          file: "csit-dpdk-perf-1707-all__10.xml"
365        -
366          build: 9
367          file: "csit-dpdk-perf-1707-all__9.xml"
368        csit-nsh_sfc-verify-func-1707-ubuntu1604-virl:
369        -
370          build: 2
371          file: "csit-nsh_sfc-verify-func-1707-ubuntu1604-virl-2.xml"
372        csit-vpp-functional-1707-ubuntu1604-virl:
373        -
374          build: lastSuccessfulBuild
375          file: "csit-vpp-functional-1707-ubuntu1604-virl-lastSuccessfulBuild.xml"
376        hc2vpp-csit-integration-1707-ubuntu1604:
377        -
378          build: lastSuccessfulBuild
379          file: "hc2vpp-csit-integration-1707-ubuntu1604-lastSuccessfulBuild.xml"
380        csit-vpp-perf-1707-all:
381        -
382          build: 16
383          file: "csit-vpp-perf-1707-all__16__output.xml"
384        -
385          build: 17
386          file: "csit-vpp-perf-1707-all__17__output.xml"
387
388
389Section: Static
390'''''''''''''''
391
392This section defines the static content which is stored in git and will be used
393as a source to generate the report.
394
395This section has these parts:
396
397 - type: "static" - says that this section is the "static".
398 - src-path - path to the static content.
399 - dst-path - destination path where the static content is copied and then
400   processed.
401
402::
403
404    -
405      type: "static"
406      src-path: "{DIR[RST]}"
407      dst-path: "{DIR[WORKING,SRC]}"
408
409
410Section: Input
411''''''''''''''
412
413This section defines the data used to generate elements. It is mandatory
414if the debug mode is not used.
415
416This section has the following parts:
417
418 - type: "input" - says that this section is the "input".
419 - general - parameters common to all builds:
420
421   - file-name: file to be downloaded.
422   - file-format: format of the downloaded file, ".zip" or ".xml" are supported.
423   - download-path: path to be added to url pointing to the file, e.g.:
424     "{job}/{build}/robot/report/*zip*/{filename}"; {job}, {build} and
425     {filename} are replaced by proper values defined in this section.
426   - extract: file to be extracted from downloaded zip file, e.g.: "output.xml";
427     if xml file is downloaded, this parameter is ignored.
428
429 - builds - list of jobs (keys) and numbers of builds which output data will be
430   downloaded.
431
432The structure of the section "Input" is as follows (example from 17.07 report):
433
434::
435
436    -
437      type: "input"  # Ignored in debug mode
438      general:
439        file-name: "robot-plugin.zip"
440        file-format: ".zip"
441        download-path: "{job}/{build}/robot/report/*zip*/{filename}"
442        extract: "robot-plugin/output.xml"
443      builds:
444        csit-vpp-perf-1707-all:
445        - 9
446        - 10
447        - 13
448        - 14
449        - 15
450        - 16
451        - 17
452        - 18
453        - 19
454        - 21
455        - 22
456        csit-dpdk-perf-1707-all:
457        - 1
458        - 2
459        - 3
460        - 4
461        - 5
462        - 6
463        - 7
464        - 8
465        - 9
466        - 10
467        csit-vpp-functional-1707-ubuntu1604-virl:
468        - lastSuccessfulBuild
469        hc2vpp-csit-perf-master-ubuntu1604:
470        - 8
471        - 9
472        hc2vpp-csit-integration-1707-ubuntu1604:
473        - lastSuccessfulBuild
474        csit-nsh_sfc-verify-func-1707-ubuntu1604-virl:
475        - 2
476
477
478Section: Output
479'''''''''''''''
480
481This section specifies which format(s) will be generated (html, pdf) and which
482versions will be generated for each format.
483
484This section has the following parts:
485
486 - type: "output" - says that this section is the "output".
487 - format: html or pdf.
488 - version: defined for each format separately.
489
490The structure of the section "Output" is as follows (example):
491
492::
493
494    -
495      type: "output"
496      format:
497        html:
498        - full
499        pdf:
500        - full
501        - minimal
502
503TODO: define the names of versions
504
505
506Content of "minimal" version
507~~~~~~~~~~~~~~~~~~~~~~~~~~~~
508
509TODO: define the name and content of this version
510
511
512Section: Table
513''''''''''''''
514
515This section defines a table to be generated. There can be 0 or more "table"
516sections.
517
518This section has the following parts:
519
520 - type: "table" - says that this section defines a table.
521 - title: Title of the table.
522 - algorithm: Algorithm which is used to generate the table. The other
523   parameters in this section must provide all information needed by the used
524   algorithm.
525 - template: (optional) a .csv file used as a template while generating the
526   table.
527 - output-file-ext: extension of the output file.
528 - output-file: file which the table will be written to.
529 - columns: specification of table columns:
530
531   - title: The title used in the table header.
532   - data: Specification of the data, it has two parts - command and arguments:
533
534     - command:
535
536       - template - take the data from template, arguments:
537
538         - number of column in the template.
539
540       - data - take the data from the input data, arguments:
541
542         - jobs and builds which data will be used.
543
544       - operation - performs an operation with the data already in the table,
545         arguments:
546
547         - operation to be done, e.g.: mean, stdev, relative_change (compute
548           the relative change between two columns) and display number of data
549           samples ~= number of test jobs. The operations are implemented in the
550           utils.py
551           TODO: Move from utils,py to e.g. operations.py
552         - numbers of columns which data will be used (optional).
553
554 - data: Specify the jobs and builds which data is used to generate the table.
555 - filter: filter based on tags applied on the input data, if "template" is
556   used, filtering is based on the template.
557 - parameters: Only these parameters will be put to the output data structure.
558
559The structure of the section "Table" is as follows (example of
560"table_performance_improvements"):
561
562::
563
564    -
565      type: "table"
566      title: "Performance improvements"
567      algorithm: "table_performance_improvements"
568      template: "{DIR[DTR,PERF,VPP,IMPRV]}/tmpl_performance_improvements.csv"
569      output-file-ext: ".csv"
570      output-file: "{DIR[DTR,PERF,VPP,IMPRV]}/performance_improvements"
571      columns:
572      -
573        title: "VPP Functionality"
574        data: "template 1"
575      -
576        title: "Test Name"
577        data: "template 2"
578      -
579        title: "VPP-16.09 mean [Mpps]"
580        data: "template 3"
581      -
582        title: "VPP-17.01 mean [Mpps]"
583        data: "template 4"
584      -
585        title: "VPP-17.04 mean [Mpps]"
586        data: "template 5"
587      -
588        title: "VPP-17.07 mean [Mpps]"
589        data: "data csit-vpp-perf-1707-all mean"
590      -
591        title: "VPP-17.07 stdev [Mpps]"
592        data: "data csit-vpp-perf-1707-all stdev"
593      -
594        title: "17.04 to 17.07 change [%]"
595        data: "operation relative_change 5 4"
596      data:
597        csit-vpp-perf-1707-all:
598        - 9
599        - 10
600        - 13
601        - 14
602        - 15
603        - 16
604        - 17
605        - 18
606        - 19
607        - 21
608      filter: "template"
609      parameters:
610      - "throughput"
611
612Example of "table_details" which generates "Detailed Test Results - VPP
613Performance Results":
614
615::
616
617    -
618      type: "table"
619      title: "Detailed Test Results - VPP Performance Results"
620      algorithm: "table_details"
621      output-file-ext: ".csv"
622      output-file: "{DIR[WORKING]}/vpp_performance_results"
623      columns:
624      -
625        title: "Name"
626        data: "data test_name"
627      -
628        title: "Documentation"
629        data: "data test_documentation"
630      -
631        title: "Status"
632        data: "data test_msg"
633      data:
634        csit-vpp-perf-1707-all:
635        - 17
636      filter: "all"
637      parameters:
638      - "parent"
639      - "doc"
640      - "msg"
641
642Example of "table_details" which generates "Test configuration - VPP Performance
643Test Configs":
644
645::
646
647    -
648      type: "table"
649      title: "Test configuration - VPP Performance Test Configs"
650      algorithm: "table_details"
651      output-file-ext: ".csv"
652      output-file: "{DIR[WORKING]}/vpp_test_configuration"
653      columns:
654      -
655        title: "Name"
656        data: "data name"
657      -
658        title: "VPP API Test (VAT) Commands History - Commands Used Per Test Case"
659        data: "data show-run"
660      data:
661        csit-vpp-perf-1707-all:
662        - 17
663      filter: "all"
664      parameters:
665      - "parent"
666      - "name"
667      - "show-run"
668
669
670Section: Plot
671'''''''''''''
672
673This section defines a plot to be generated. There can be 0 or more "plot"
674sections.
675
676This section has these parts:
677
678 - type: "plot" - says that this section defines a plot.
679 - title: Plot title used in the logs. Title which is displayed is in the
680   section "layout".
681 - output-file-type: format of the output file.
682 - output-file: file which the plot will be written to.
683 - algorithm: Algorithm used to generate the plot. The other parameters in this
684   section must provide all information needed by plot.ly to generate the plot.
685   For example:
686
687   - traces
688   - layout
689
690   - These parameters are transparently passed to plot.ly.
691
692 - data: Specify the jobs and numbers of builds which data is used to generate
693   the plot.
694 - filter: filter applied on the input data.
695 - parameters: Only these parameters will be put to the output data structure.
696
697The structure of the section "Plot" is as follows (example of a plot showing
698throughput in a chart box-with-whiskers):
699
700::
701
702    -
703      type: "plot"
704      title: "VPP Performance 64B-1t1c-(eth|dot1q|dot1ad)-(l2xcbase|l2bdbasemaclrn)-ndrdisc"
705      algorithm: "plot_performance_box"
706      output-file-type: ".html"
707      output-file: "{DIR[STATIC,VPP]}/64B-1t1c-l2-sel1-ndrdisc"
708      data:
709        csit-vpp-perf-1707-all:
710        - 9
711        - 10
712        - 13
713        - 14
714        - 15
715        - 16
716        - 17
717        - 18
718        - 19
719        - 21
720      # Keep this formatting, the filter is enclosed with " (quotation mark) and
721      # each tag is enclosed with ' (apostrophe).
722      filter: "'64B' and 'BASE' and 'NDRDISC' and '1T1C' and ('L2BDMACSTAT' or 'L2BDMACLRN' or 'L2XCFWD') and not 'VHOST'"
723      parameters:
724      - "throughput"
725      - "parent"
726      traces:
727        hoverinfo: "x+y"
728        boxpoints: "outliers"
729        whiskerwidth: 0
730      layout:
731        title: "64B-1t1c-(eth|dot1q|dot1ad)-(l2xcbase|l2bdbasemaclrn)-ndrdisc"
732        xaxis:
733          autorange: True
734          autotick: False
735          fixedrange: False
736          gridcolor: "rgb(238, 238, 238)"
737          linecolor: "rgb(238, 238, 238)"
738          linewidth: 1
739          showgrid: True
740          showline: True
741          showticklabels: True
742          tickcolor: "rgb(238, 238, 238)"
743          tickmode: "linear"
744          title: "Indexed Test Cases"
745          zeroline: False
746        yaxis:
747          gridcolor: "rgb(238, 238, 238)'"
748          hoverformat: ".4s"
749          linecolor: "rgb(238, 238, 238)"
750          linewidth: 1
751          range: []
752          showgrid: True
753          showline: True
754          showticklabels: True
755          tickcolor: "rgb(238, 238, 238)"
756          title: "Packets Per Second [pps]"
757          zeroline: False
758        boxmode: "group"
759        boxgroupgap: 0.5
760        autosize: False
761        margin:
762          t: 50
763          b: 20
764          l: 50
765          r: 20
766        showlegend: True
767        legend:
768          orientation: "h"
769        width: 700
770        height: 1000
771
772The structure of the section "Plot" is as follows (example of a plot showing
773latency in a box chart):
774
775::
776
777    -
778      type: "plot"
779      title: "VPP Latency 64B-1t1c-(eth|dot1q|dot1ad)-(l2xcbase|l2bdbasemaclrn)-ndrdisc"
780      algorithm: "plot_latency_box"
781      output-file-type: ".html"
782      output-file: "{DIR[STATIC,VPP]}/64B-1t1c-l2-sel1-ndrdisc-lat50"
783      data:
784        csit-vpp-perf-1707-all:
785        - 9
786        - 10
787        - 13
788        - 14
789        - 15
790        - 16
791        - 17
792        - 18
793        - 19
794        - 21
795      filter: "'64B' and 'BASE' and 'NDRDISC' and '1T1C' and ('L2BDMACSTAT' or 'L2BDMACLRN' or 'L2XCFWD') and not 'VHOST'"
796      parameters:
797      - "latency"
798      - "parent"
799      traces:
800        boxmean: False
801      layout:
802        title: "64B-1t1c-(eth|dot1q|dot1ad)-(l2xcbase|l2bdbasemaclrn)-ndrdisc"
803        xaxis:
804          autorange: True
805          autotick: False
806          fixedrange: False
807          gridcolor: "rgb(238, 238, 238)"
808          linecolor: "rgb(238, 238, 238)"
809          linewidth: 1
810          showgrid: True
811          showline: True
812          showticklabels: True
813          tickcolor: "rgb(238, 238, 238)"
814          tickmode: "linear"
815          title: "Indexed Test Cases"
816          zeroline: False
817        yaxis:
818          gridcolor: "rgb(238, 238, 238)'"
819          hoverformat: ""
820          linecolor: "rgb(238, 238, 238)"
821          linewidth: 1
822          range: []
823          showgrid: True
824          showline: True
825          showticklabels: True
826          tickcolor: "rgb(238, 238, 238)"
827          title: "Latency min/avg/max [uSec]"
828          zeroline: False
829        boxmode: "group"
830        boxgroupgap: 0.5
831        autosize: False
832        margin:
833          t: 50
834          b: 20
835          l: 50
836          r: 20
837        showlegend: True
838        legend:
839          orientation: "h"
840        width: 700
841        height: 1000
842
843The structure of the section "Plot" is as follows (example of a plot showing
844VPP HTTP server performance in a box chart with pre-defined data
845"plot-vpp-httlp-server-performance" set and  plot layout "plot-cps"):
846
847::
848
849    -
850      type: "plot"
851      title: "VPP HTTP Server Performance"
852      algorithm: "plot_http_server_performance_box"
853      output-file-type: ".html"
854      output-file: "{DIR[STATIC,VPP]}/http-server-performance-cps"
855      data:
856        "plot-vpp-httlp-server-performance"
857      # Keep this formatting, the filter is enclosed with " (quotation mark) and
858      # each tag is enclosed with ' (apostrophe).
859      filter: "'HTTP' and 'TCP_CPS'"
860      parameters:
861      - "result"
862      - "name"
863      traces:
864        hoverinfo: "x+y"
865        boxpoints: "outliers"
866        whiskerwidth: 0
867      layout:
868        title: "VPP HTTP Server Performance"
869        layout:
870          "plot-cps"
871
872
873Section: file
874'''''''''''''
875
876This section defines a file to be generated. There can be 0 or more "file"
877sections.
878
879This section has the following parts:
880
881 - type: "file" - says that this section defines a file.
882 - title: Title of the table.
883 - algorithm: Algorithm which is used to generate the file. The other
884   parameters in this section must provide all information needed by the used
885   algorithm.
886 - output-file-ext: extension of the output file.
887 - output-file: file which the file will be written to.
888 - file-header: The header of the generated .rst file.
889 - dir-tables: The directory with the tables.
890 - data: Specify the jobs and builds which data is used to generate the table.
891 - filter: filter based on tags applied on the input data, if "all" is
892   used, no filtering is done.
893 - parameters: Only these parameters will be put to the output data structure.
894 - chapters: the hierarchy of chapters in the generated file.
895 - start-level: the level of the the top-level chapter.
896
897The structure of the section "file" is as follows (example):
898
899::
900
901    -
902      type: "file"
903      title: "VPP Performance Results"
904      algorithm: "file_test_results"
905      output-file-ext: ".rst"
906      output-file: "{DIR[DTR,PERF,VPP]}/vpp_performance_results"
907      file-header: "\n.. |br| raw:: html\n\n    <br />\n\n\n.. |prein| raw:: html\n\n    <pre>\n\n\n.. |preout| raw:: html\n\n    </pre>\n\n"
908      dir-tables: "{DIR[DTR,PERF,VPP]}"
909      data:
910        csit-vpp-perf-1707-all:
911        - 22
912      filter: "all"
913      parameters:
914      - "name"
915      - "doc"
916      - "level"
917      data-start-level: 2  # 0, 1, 2, ...
918      chapters-start-level: 2  # 0, 1, 2, ...
919
920
921Static content
922``````````````
923
924 - Manually created / edited files.
925 - .rst files, static .csv files, static pictures (.svg), ...
926 - Stored in CSIT git repository.
927
928No more details about the static content in this document.
929
930
931Data to process
932```````````````
933
934The PAL processes tests results and other information produced by Jenkins jobs.
935The data are now stored as robot results in Jenkins (TODO: store the data in
936nexus) either as .zip and / or .xml files.
937
938
939Data processing
940---------------
941
942As the first step, the data are downloaded and stored locally (typically on a
943Jenkins slave). If .zip files are used, the given .xml files are extracted for
944further processing.
945
946Parsing of the .xml files is performed by a class derived from
947"robot.api.ResultVisitor", only necessary methods are overridden. All and only
948necessary data is extracted from .xml file and stored in a structured form.
949
950The parsed data are stored as the multi-indexed pandas.Series data type. Its
951structure is as follows:
952
953::
954
955    <job name>
956      <build>
957        <metadata>
958        <suites>
959        <tests>
960
961"job name", "build", "metadata", "suites", "tests" are indexes to access the
962data. For example:
963
964::
965
966    data =
967
968    job 1 name:
969      build 1:
970        metadata: metadata
971        suites: suites
972        tests: tests
973      ...
974      build N:
975        metadata: metadata
976        suites: suites
977        tests: tests
978    ...
979    job M name:
980      build 1:
981        metadata: metadata
982        suites: suites
983        tests: tests
984      ...
985      build N:
986        metadata: metadata
987        suites: suites
988        tests: tests
989
990Using indexes data["job 1 name"]["build 1"]["tests"] (e.g.:
991data["csit-vpp-perf-1704-all"]["17"]["tests"]) we get a list of all tests with
992all tests data.
993
994Data will not be accessible directly using indexes, but using getters and
995filters.
996
997**Structure of metadata:**
998
999::
1000
1001    "metadata": {
1002        "version": "VPP version",
1003        "job": "Jenkins job name"
1004        "build": "Information about the build"
1005    },
1006
1007**Structure of suites:**
1008
1009::
1010
1011    "suites": {
1012        "Suite name 1": {
1013            "doc": "Suite 1 documentation"
1014            "parent": "Suite 1 parent"
1015        }
1016        "Suite name N": {
1017            "doc": "Suite N documentation"
1018            "parent": "Suite N parent"
1019        }
1020
1021**Structure of tests:**
1022
1023Performance tests:
1024
1025::
1026
1027    "tests": {
1028        "ID": {
1029            "name": "Test name",
1030            "parent": "Name of the parent of the test",
1031            "doc": "Test documentation"
1032            "msg": "Test message"
1033            "tags": ["tag 1", "tag 2", "tag n"],
1034            "type": "PDR" | "NDR",
1035            "throughput": {
1036                "value": int,
1037                "unit": "pps" | "bps" | "percentage"
1038            },
1039            "latency": {
1040                "direction1": {
1041                    "100": {
1042                        "min": int,
1043                        "avg": int,
1044                        "max": int
1045                    },
1046                    "50": {  # Only for NDR
1047                        "min": int,
1048                        "avg": int,
1049                        "max": int
1050                    },
1051                    "10": {  # Only for NDR
1052                        "min": int,
1053                        "avg": int,
1054                        "max": int
1055                    }
1056                },
1057                "direction2": {
1058                    "100": {
1059                        "min": int,
1060                        "avg": int,
1061                        "max": int
1062                    },
1063                    "50": {  # Only for NDR
1064                        "min": int,
1065                        "avg": int,
1066                        "max": int
1067                    },
1068                    "10": {  # Only for NDR
1069                        "min": int,
1070                        "avg": int,
1071                        "max": int
1072                    }
1073                }
1074            },
1075            "lossTolerance": "lossTolerance"  # Only for PDR
1076            "vat-history": "DUT1 and DUT2 VAT History"
1077            },
1078            "show-run": "Show Run"
1079        },
1080        "ID" {
1081            # next test
1082        }
1083
1084Functional tests:
1085
1086::
1087
1088    "tests": {
1089        "ID": {
1090            "name": "Test name",
1091            "parent": "Name of the parent of the test",
1092            "doc": "Test documentation"
1093            "msg": "Test message"
1094            "tags": ["tag 1", "tag 2", "tag n"],
1095            "vat-history": "DUT1 and DUT2 VAT History"
1096            "show-run": "Show Run"
1097            "status": "PASS" | "FAIL"
1098        },
1099        "ID" {
1100            # next test
1101        }
1102    }
1103
1104Note: ID is the lowercase full path to the test.
1105
1106
1107Data filtering
1108``````````````
1109
1110The first step when generating an element is getting the data needed to
1111construct the element. The data are filtered from the processed input data.
1112
1113The data filtering is based on:
1114
1115 - job name(s).
1116 - build number(s).
1117 - tag(s).
1118 - required data - only this data is included in the output.
1119
1120WARNING: The filtering is based on tags, so be careful with tagging.
1121
1122For example, the element which specification includes:
1123
1124::
1125
1126    data:
1127      csit-vpp-perf-1707-all:
1128      - 9
1129      - 10
1130      - 13
1131      - 14
1132      - 15
1133      - 16
1134      - 17
1135      - 18
1136      - 19
1137      - 21
1138    filter:
1139      - "'64B' and 'BASE' and 'NDRDISC' and '1T1C' and ('L2BDMACSTAT' or 'L2BDMACLRN' or 'L2XCFWD') and not 'VHOST'"
1140
1141will be constructed using data from the job "csit-vpp-perf-1707-all", for all
1142listed builds and the tests with the list of tags matching the filter
1143conditions.
1144
1145The output data structure for filtered test data is:
1146
1147::
1148
1149    - job 1
1150      - build 1
1151        - test 1
1152          - parameter 1
1153          - parameter 2
1154          ...
1155          - parameter n
1156        ...
1157        - test n
1158        ...
1159      ...
1160      - build n
1161    ...
1162    - job n
1163
1164
1165Data analytics
1166``````````````
1167
1168Data analytics part implements:
1169
1170 - methods to compute statistical data from the filtered input data.
1171 - trending.
1172
1173Throughput Speedup Analysis - Multi-Core with Multi-Threading
1174'''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''
1175
1176Throughput Speedup Analysis (TSA) calculates throughput speedup ratios
1177for tested 1-, 2- and 4-core multi-threaded VPP configurations using the
1178following formula:
1179
1180::
1181
1182                                N_core_throughput
1183    N_core_throughput_speedup = -----------------
1184                                1_core_throughput
1185
1186Multi-core throughput speedup ratios are plotted in grouped bar graphs
1187for throughput tests with 64B/78B frame size, with number of cores on
1188X-axis and speedup ratio on Y-axis.
1189
1190For better comparison multiple test results' data sets are plotted per
1191each graph:
1192
1193    - graph type: grouped bars;
1194    - graph X-axis: (testcase index, number of cores);
1195    - graph Y-axis: speedup factor.
1196
1197Subset of existing performance tests is covered by TSA graphs.
1198
1199**Model for TSA:**
1200
1201::
1202
1203    -
1204      type: "plot"
1205      title: "TSA: 64B-*-(eth|dot1q|dot1ad)-(l2xcbase|l2bdbasemaclrn)-ndrdisc"
1206      algorithm: "plot_throughput_speedup_analysis"
1207      output-file-type: ".html"
1208      output-file: "{DIR[STATIC,VPP]}/10ge2p1x520-64B-l2-tsa-ndrdisc"
1209      data:
1210        "plot-throughput-speedup-analysis"
1211      filter: "'NIC_Intel-X520-DA2' and '64B' and 'BASE' and 'NDRDISC' and ('L2BDMACSTAT' or 'L2BDMACLRN' or 'L2XCFWD') and not 'VHOST'"
1212      parameters:
1213      - "throughput"
1214      - "parent"
1215      - "tags"
1216      layout:
1217        title: "64B-*-(eth|dot1q|dot1ad)-(l2xcbase|l2bdbasemaclrn)-ndrdisc"
1218        layout:
1219          "plot-throughput-speedup-analysis"
1220
1221
1222Comparison of results from two sets of the same test executions
1223'''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''
1224
1225This algorithm enables comparison of results coming from two sets of the
1226same test executions. It is used to quantify performance changes across
1227all tests after test environment changes e.g. Operating System
1228upgrades/patches, Hardware changes.
1229
1230It is assumed that each set of test executions includes multiple runs
1231of the same tests, 10 or more, to verify test results repeatibility and
1232to yield statistically meaningful results data.
1233
1234Comparison results are presented in a table with a specified number of
1235the best and the worst relative changes between the two sets. Following table
1236columns are defined:
1237
1238    - name of the test;
1239    - throughput mean values of the reference set;
1240    - throughput standard deviation  of the reference set;
1241    - throughput mean values of the set to compare;
1242    - throughput standard deviation  of the set to compare;
1243    - relative change of the mean values.
1244
1245**The model**
1246
1247The model specifies:
1248
1249    - type: "table" - means this section defines a table.
1250    - title: Title of the table.
1251    - algorithm: Algorithm which is used to generate the table. The other
1252      parameters in this section must provide all information needed by the used
1253      algorithm.
1254    - output-file-ext: Extension of the output file.
1255    - output-file: File which the table will be written to.
1256    - reference - the builds which are used as the reference for comparison.
1257    - compare - the builds which are compared to the reference.
1258    - data: Specify the sources, jobs and builds, providing data for generating
1259      the table.
1260    - filter: Filter based on tags applied on the input data, if "template" is
1261      used, filtering is based on the template.
1262    - parameters: Only these parameters will be put to the output data
1263      structure.
1264    - nr-of-tests-shown: Number of the best and the worst tests presented in the
1265      table. Use 0 (zero) to present all tests.
1266
1267*Example:*
1268
1269::
1270
1271    -
1272      type: "table"
1273      title: "Performance comparison"
1274      algorithm: "table_performance_comparison"
1275      output-file-ext: ".csv"
1276      output-file: "{DIR[DTR,PERF,VPP,IMPRV]}/vpp_performance_comparison"
1277      reference:
1278        title: "csit-vpp-perf-1801-all - 1"
1279        data:
1280          csit-vpp-perf-1801-all:
1281          - 1
1282          - 2
1283      compare:
1284        title: "csit-vpp-perf-1801-all - 2"
1285        data:
1286          csit-vpp-perf-1801-all:
1287          - 1
1288          - 2
1289      data:
1290        "vpp-perf-comparison"
1291      filter: "all"
1292      parameters:
1293      - "name"
1294      - "parent"
1295      - "throughput"
1296      nr-of-tests-shown: 20
1297
1298
1299Advanced data analytics
1300```````````````````````
1301
1302In the future advanced data analytics (ADA) will be added to analyze the
1303telemetry data collected from SUT telemetry sources and correlate it to
1304performance test results.
1305
1306:TODO:
1307
1308    - describe the concept of ADA.
1309    - add specification.
1310
1311
1312Data presentation
1313-----------------
1314
1315Generates the plots and tables according to the report models per
1316specification file. The elements are generated using algorithms and data
1317specified in their models.
1318
1319
1320Tables
1321``````
1322
1323 - tables are generated by algorithms implemented in PAL, the model includes the
1324   algorithm and all necessary information.
1325 - output format: csv
1326 - generated tables are stored in specified directories and linked to .rst
1327   files.
1328
1329
1330Plots
1331`````
1332
1333 - `plot.ly <https://plot.ly/>`_ is currently used to generate plots, the model
1334   includes the type of plot and all the necessary information to render it.
1335 - output format: html.
1336 - generated plots are stored in specified directories and linked to .rst files.
1337
1338
1339Report generation
1340-----------------
1341
1342Report is generated using Sphinx and Read_the_Docs template. PAL generates html
1343and pdf formats. It is possible to define the content of the report by
1344specifying the version (TODO: define the names and content of versions).
1345
1346
1347The process
1348```````````
1349
13501. Read the specification.
13512. Read the input data.
13523. Process the input data.
13534. For element (plot, table, file) defined in specification:
1354
1355   a. Get the data needed to construct the element using a filter.
1356   b. Generate the element.
1357   c. Store the element.
1358
13595. Generate the report.
13606. Store the report (Nexus).
1361
1362The process is model driven. The elements' models (tables, plots, files
1363and report itself) are defined in the specification file. Script reads
1364the elements' models from specification file and generates the elements.
1365
1366It is easy to add elements to be generated in the report. If a new type
1367of an element is required, only a new algorithm needs to be implemented
1368and integrated.
1369
1370
1371Continuous Performance Measurements and Trending
1372------------------------------------------------
1373
1374Performance analysis and trending execution sequence:
1375`````````````````````````````````````````````````````
1376
1377CSIT PA runs performance analysis, change detection and trending using specified
1378trend analysis metrics over the rolling window of last <N> sets of historical
1379measurement data. PA is defined as follows:
1380
1381    #. PA job triggers:
1382
1383        #. By PT job at its completion.
1384        #. Manually from Jenkins UI.
1385
1386    #. Download and parse archived historical data and the new data:
1387
1388        #. New data from latest PT job is evaluated against the rolling window
1389           of <N> sets of historical data.
1390        #. Download RF output.xml files and compressed archived data.
1391        #. Parse out the data filtering test cases listed in PA specification
1392           (part of CSIT PAL specification file).
1393
1394    #. Calculate trend metrics for the rolling window of <N> sets of historical
1395       data:
1396
1397        #. Calculate quartiles Q1, Q2, Q3.
1398        #. Trim outliers using IQR.
1399        #. Calculate TMA and TMSD.
1400        #. Calculate normal trending range per test case based on TMA and TMSD.
1401
1402    #. Evaluate new test data against trend metrics:
1403
1404        #. If within the range of (TMA +/- 3*TMSD) => Result = Pass,
1405           Reason = Normal.
1406        #. If below the range => Result = Fail, Reason = Regression.
1407        #. If above the range => Result = Pass, Reason = Progression.
1408
1409    #. Generate and publish results
1410
1411        #. Relay evaluation result to job result.
1412        #. Generate a new set of trend analysis summary graphs and drill-down
1413           graphs.
1414
1415            #. Summary graphs to include measured values with Normal,
1416               Progression and Regression markers. MM shown in the background if
1417               possible.
1418            #. Drill-down graphs to include MM, TMA and TMSD.
1419
1420        #. Publish trend analysis graphs in html format on
1421           https://docs.fd.io/csit/master/trending/.
1422
1423
1424Parameters to specify:
1425``````````````````````
1426
1427*General section - parameters common to all plots:*
1428
1429    - type: "cpta";
1430    - title: The title of this section;
1431    - output-file-type: only ".html" is supported;
1432    - output-file: path where the generated files will be stored.
1433
1434*Plots section:*
1435
1436    - plot title;
1437    - output file name;
1438    - input data for plots;
1439
1440        - job to be monitored - the Jenkins job which results are used as input
1441          data for this test;
1442        - builds used for trending plot(s) - specified by a list of build
1443          numbers or by a range of builds defined by the first and the last
1444          build number;
1445
1446    - tests to be displayed in the plot defined by a filter;
1447    - list of parameters to extract from the data;
1448    - plot layout
1449
1450*Example:*
1451
1452::
1453
1454    -
1455      type: "cpta"
1456      title: "Continuous Performance Trending and Analysis"
1457      output-file-type: ".html"
1458      output-file: "{DIR[STATIC,VPP]}/cpta"
1459      plots:
1460
1461        - title: "VPP 1T1C L2 64B Packet Throughput - Trending"
1462          output-file-name: "l2-1t1c-x520"
1463          data: "plot-performance-trending-vpp"
1464          filter: "'NIC_Intel-X520-DA2' and 'MRR' and '64B' and ('BASE' or 'SCALE') and '1T1C' and ('L2BDMACSTAT' or 'L2BDMACLRN' or 'L2XCFWD') and not 'VHOST' and not 'MEMIF'"
1465          parameters:
1466          - "result"
1467          layout: "plot-cpta-vpp"
1468
1469        - title: "DPDK 4T4C IMIX MRR Trending"
1470          output-file-name: "dpdk-imix-4t4c-xl710"
1471          data: "plot-performance-trending-dpdk"
1472          filter: "'NIC_Intel-XL710' and 'IMIX' and 'MRR' and '4T4C' and 'DPDK'"
1473          parameters:
1474          - "result"
1475          layout: "plot-cpta-dpdk"
1476
1477The Dashboard
1478`````````````
1479
1480Performance dashboard tables provide the latest VPP throughput trend, trend
1481compliance and detected anomalies, all on a per VPP test case basis.
1482The Dashboard is generated as three tables for 1t1c, 2t2c and 4t4c MRR tests.
1483
1484At first, the .csv tables are generated (only the table for 1t1c is shown):
1485
1486::
1487
1488    -
1489      type: "table"
1490      title: "Performance trending dashboard"
1491      algorithm: "table_performance_trending_dashboard"
1492      output-file-ext: ".csv"
1493      output-file: "{DIR[STATIC,VPP]}/performance-trending-dashboard-1t1c"
1494      data: "plot-performance-trending-all"
1495      filter: "'MRR' and '1T1C'"
1496      parameters:
1497      - "name"
1498      - "parent"
1499      - "result"
1500      ignore-list:
1501      - "tests.vpp.perf.l2.10ge2p1x520-eth-l2bdscale1mmaclrn-mrr.tc01-64b-1t1c-eth-l2bdscale1mmaclrn-ndrdisc"
1502      outlier-const: 1.5
1503      window: 14
1504      evaluated-window: 14
1505      long-trend-window: 180
1506
1507Then, html tables stored inside .rst files are generated:
1508
1509::
1510
1511    -
1512      type: "table"
1513      title: "HTML performance trending dashboard 1t1c"
1514      algorithm: "table_performance_trending_dashboard_html"
1515      input-file: "{DIR[STATIC,VPP]}/performance-trending-dashboard-1t1c.csv"
1516      output-file: "{DIR[STATIC,VPP]}/performance-trending-dashboard-1t1c.rst"
1517
1518Root Cause Analysis
1519-------------------
1520
1521Root Cause Analysis (RCA) by analysing archived performance results – re-analyse
1522available data for specified:
1523
1524    - range of jobs builds,
1525    - set of specific tests and
1526    - PASS/FAIL criteria to detect performance change.
1527
1528In addition, PAL generates trending plots to show performance over the specified
1529time interval.
1530
1531Root Cause Analysis - Option 1: Analysing Archived VPP Results
1532``````````````````````````````````````````````````````````````
1533
1534It can be used to speed-up the process, or when the existing data is sufficient.
1535In this case, PAL uses existing data saved in Nexus, searches for performance
1536degradations and generates plots to show performance over the specified time
1537interval for the selected tests.
1538
1539Execution Sequence
1540''''''''''''''''''
1541
1542    #. Download and parse archived historical data and the new data.
1543    #. Calculate trend metrics.
1544    #. Find regression / progression.
1545    #. Generate and publish results:
1546
1547        #. Summary graphs to include measured values with Progression and
1548           Regression markers.
1549        #. List the DUT build(s) where the anomalies were detected.
1550
1551CSIT PAL Specification
1552''''''''''''''''''''''
1553
1554    - What to test:
1555
1556        - first build (Good); specified by the Jenkins job name and the build
1557          number
1558        - last build (Bad); specified by the Jenkins job name and the build
1559          number
1560        - step (1..n).
1561
1562    - Data:
1563
1564        - tests of interest; list of tests (full name is used) which results are
1565          used
1566
1567*Example:*
1568
1569::
1570
1571    TODO
1572
1573
1574API
1575---
1576
1577List of modules, classes, methods and functions
1578```````````````````````````````````````````````
1579
1580::
1581
1582    specification_parser.py
1583
1584        class Specification
1585
1586            Methods:
1587                read_specification
1588                set_input_state
1589                set_input_file_name
1590
1591            Getters:
1592                specification
1593                environment
1594                debug
1595                is_debug
1596                input
1597                builds
1598                output
1599                tables
1600                plots
1601                files
1602                static
1603
1604
1605    input_data_parser.py
1606
1607        class InputData
1608
1609            Methods:
1610                read_data
1611                filter_data
1612
1613            Getters:
1614                data
1615                metadata
1616                suites
1617                tests
1618
1619
1620    environment.py
1621
1622        Functions:
1623            clean_environment
1624
1625        class Environment
1626
1627            Methods:
1628                set_environment
1629
1630            Getters:
1631                environment
1632
1633
1634    input_data_files.py
1635
1636        Functions:
1637            download_data_files
1638            unzip_files
1639
1640
1641    generator_tables.py
1642
1643        Functions:
1644            generate_tables
1645
1646        Functions implementing algorithms to generate particular types of
1647        tables (called by the function "generate_tables"):
1648            table_details
1649            table_performance_improvements
1650
1651
1652    generator_plots.py
1653
1654        Functions:
1655            generate_plots
1656
1657        Functions implementing algorithms to generate particular types of
1658        plots (called by the function "generate_plots"):
1659            plot_performance_box
1660            plot_latency_box
1661
1662
1663    generator_files.py
1664
1665        Functions:
1666            generate_files
1667
1668        Functions implementing algorithms to generate particular types of
1669        files (called by the function "generate_files"):
1670            file_test_results
1671
1672
1673    report.py
1674
1675        Functions:
1676            generate_report
1677
1678        Functions implementing algorithms to generate particular types of
1679        report (called by the function "generate_report"):
1680            generate_html_report
1681            generate_pdf_report
1682
1683        Other functions called by the function "generate_report":
1684            archive_input_data
1685            archive_report
1686
1687
1688PAL functional diagram
1689``````````````````````
1690
1691.. only:: latex
1692
1693    .. raw:: latex
1694
1695        \begin{figure}[H]
1696            \centering
1697                \graphicspath{{../_tmp/src/csit_framework_documentation/}}
1698                \includegraphics[width=0.90\textwidth]{pal_func_diagram}
1699                \label{fig:pal_func_diagram}
1700        \end{figure}
1701
1702.. only:: html
1703
1704    .. figure:: pal_func_diagram.svg
1705        :alt: PAL functional diagram
1706        :align: center
1707
1708
1709How to add an element
1710`````````````````````
1711
1712Element can be added by adding it's model to the specification file. If
1713the element is to be generated by an existing algorithm, only it's
1714parameters must be set.
1715
1716If a brand new type of element needs to be added, also the algorithm
1717must be implemented. Element generation algorithms are implemented in
1718the files with names starting with "generator" prefix. The name of the
1719function implementing the algorithm and the name of algorithm in the
1720specification file have to be the same.
1721