History log of /vpp/test/framework.py
Revision Date Author Comments
# 9beabd8a 02-Dec-2019 Paul Vinciguerra <pvinci@vinciconsulting.com>

tests: clean up logging

Tests currently expect the logger to be poked from run_tests.py.
The tests should run without any magic values. This change sets a default
null logger and removes the hasattr checks for the logger.

For reference, see: https://docs.python-guide.org/writing/logging/

Type: test

Change-Id: I98f953d73d12d00e74b59c94a0fb8c7a625b9c44
Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>

# 7784140f 29-Apr-2020 Dave Barach <dave@barachs.net>

misc: binary api fuzz test fixes

Add a hook to src/vlibapi/api_shared.c to fuzz (screw up) binary API
messages, e.g. by xoring random data into them before processing. We
specifically exempt client connection messages, and inband debug CLI
messages. We step over msg_id, client index, client context, and
sw_if_index. Otherwise, "make test" vectors fail too rapidly to learn

The goal is to reduce the number of crashes caused to zero. We're
fairly close with this patch.

Add vl_msg_api_max_length(void *mp), which returns the maximum
plausible length for a binary API message.

Use it to hardern vl_api_from_api_to_new_vec(...) which takes an
additional argument - message pointer - so it can verify that
astr->length is sane. If it's not sane, return a u8 *vector of the
form "insane astr->length nnnn\0".

Verify array lengths in vl_api_dhcp6_send_client_message_t_handler(...)
and vl_api_dhcp6_pd_send_client_message_t_handler(...).

Add a fairly effective binary API fuzz hook to the unittest plugin,
and modify the "make test" framework.py to pass "api-fuzz { on|off }"
to enable API fuzzing: "make API_FUZZ=on TEST=xxx test-debug" or similar

Type: improvement

Signed-off-by: Dave Barach <dave@barachs.net>
Change-Id: I0157267652a163c01553d5267620f719cc6c3bde

# e64e5fff 28-Apr-2020 Paul Vinciguerra <pvinci@vinciconsulting.com>

tests: implement ipaddress convenience methods

Add vpp specific properties to ip addresses for use in the api.
.vapi_af -- returns [ADDRESS_IP4, ADDRESS_IP6]
.vapi_af_name -- returns the string ['ip4', 'ip6']

Update tests to demonstrate usage.

Type: feature

Change-Id: I43447a1522769d99f89debdc714c51700068d771
Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>

# 4830e4f7 10-Mar-2020 Ray Kinsella <mdr@ashroe.eu>

vlib: startup multi-arch variant configuration

Support for startup node multi-arch variant selection through startup.conf.
This is to facilitate unit, functional testing and benchmarking of non-default
multi-arch variant node code path. Also added parameters to make test, to
specific using multi-arch variants in unit testing.

Type: improvement

Signed-off-by: Ray Kinsella <mdr@ashroe.eu>
Change-Id: I94fd332bb629683b7a7dd770ee9f615a9a424060

# 19718003 11-Mar-2020 Dave Barach <dave@barachs.net>

vppinfra: refactor clib_timebase_t

Add a clib_time_t * argument to clib_timebase_init(...), to encourage
client code to share the vlib_main_t's clib_time_t object.

Display the current day / date in GMT via the "show time" debug CLI.

Fix the test framework so it processes the new "show time" output format.

Type: refactor

Signed-off-by: Dave Barach <dave@barachs.net>
Change-Id: I5e52d57eb164b7cdb6355362d520df6928491711

# d498c9eb 10-Mar-2020 Dave Barach <dave@barachs.net>

tests: add running_gcov_tests to framework.py

Mark a few code coverage tests appropriately:

@unittest.skipUnless(running_gcov_tests, "part of code coverage tests")
def test_my_coverage_unittest(self):

Almost exactly like "make EXTENDED_TESTS=yes test".

Type: feature
Signed-off-by: Dave Barach <dave@barachs.net>
Change-Id: Ic6d0a097a608ba93442b3d88252f66f8e6805b97

# 6e6ad64a 04-Feb-2020 Ole Troan <ot@cisco.com>

tests: support python 3.8

Make test framework python3 version independence.

Type: fix
Signed-off-by: Ole Troan <ot@cisco.com>
Change-Id: I1ef1eb77b6c1f422ebc4dad0818f87c8e587b34b

# 48bdbcd8 05-Dec-2019 Paul Vinciguerra <pvinci@vinciconsulting.com>

tests: fix worker thread initialization

from threading.thread __init__:

This constructor should always be called with keyword arguments.

If a subclass overrides the constructor, it must make sure to invoke
the base class constructor (Thread.__init__()) before doing anything
else to the thread.

Type: test
Change-Id: Ifa89202e97053a4baf19e9a0ca0913430d5087a3
Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>

# 4ed25985 25-Dec-2019 Dave Barach <dave@barachs.net>

tests: configure 32 mb physmem

Absolutely nothing good happens when we force the kernel to briefly
map and then unmap 16gb as vpp starts. Effect exacerbated when
TEST_JOBS = 20...40, and so forth.

Type: test

Signed-off-by: Dave Barach <dave@barachs.net>
Change-Id: Id8e3ce1763cad3a0891d5d6c8d2c1e3e610682d7

# 4a56f4e4 23-Dec-2019 Neale Ranns <nranns@cisco.com>

ipsec: Test and fix IPSec worker hand-off

Type: fix

Change-Id: I5cb9a3845ddbc5f4de4eb4e9c481f606fe5cec9a
Signed-off-by: Neale Ranns <nranns@cisco.com>

# c701e571 19-Dec-2019 Paul Vinciguerra <pvinci@vinciconsulting.com>

tests: don't prompt to launch gdb for sanity test case

Type: test

Change-Id: I4c54121b76b341381a819cee928c3c2455a83503
Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>

# e090f4db 29-Nov-2019 Paul Vinciguerra <pvinci@vinciconsulting.com>

papi: lazily initialize stats client

remove wait-loop on stats socket from test framework.

Type: refactor

Change-Id: I5bb95a7c597707a87f9d9a471215c4b4af1a2280
Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>

# 66cea092 05-Dec-2019 Klement Sekera <ksekera@cisco.com>

tests: fix wrong checksum error message

This change fixes the error message, which would previously report
mismatch on IPv6 layer instead of TCP layer.

Type: fix

Change-Id: Ie78779685362a420d155f928b3cdb341bc3fb8bc
Signed-off-by: Klement Sekera <ksekera@cisco.com>

# fd574087 27-Nov-2019 Ole Troan <ot@cisco.com>

papi: add call stats

Type: feature
Change-Id: Ic6d44122d3e62e09402e3f1946f7e57e9b5e7c5f
Signed-off-by: Ole Troan <ot@cisco.com>

# 8b2fffd9 02-Dec-2019 Paul Vinciguerra <pvinci@vinciconsulting.com>

tests: Revert "Python Tests: Make VppTestCase unit testable."

Type: test
This reverts commit 087c811b63aa6b0575149749e188bb37e7fb0513.

Change-Id: Ica12597f27d5a75c33c12c2d27a227dc4866adcb
Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>

# 0c629323 28-Nov-2019 Ole Troan <ot@cisco.com>

tests: add test run time.

Type: feature
Signed-off-by: Ole Troan <ot@cisco.com>
Change-Id: Id7b30b5fe63e68ae88f3a42aa77d2614ccda724d

# 1043fd38 03-Dec-2019 Paul Vinciguerra <pvinci@vinciconsulting.com>

tests: fix exception handling around vapi.connect()/.disconnect()

vapi.connect() only raises subclasses of vpp_papi.VPPIOError.
vapi.disconnect() doesn't raise any exceptions, removed uneeded try block.

Type: test

Change-Id: Ide6e6db298b205f14382065862e358757619a5ba
Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>

# bfd7d294 27-Oct-2019 Paul Vinciguerra <pvinci@vinciconsulting.com>

vlib: add 'wait' cli command

When running exec scripts, there can be a need to wait between statements.

Type: feature

Change-Id: I2a45b390697e09fc222358c9354f28e3368a06ba
Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>

# 1a7ed5e3 21-Nov-2019 Benoît Ganne <bganne@cisco.com>

tests: display stdout and stderr when output caching is disabled

Type: fix

Change-Id: I4286ba58bb15dd67c1199f76166bedf3b4e0344c
Signed-off-by: Benoît Ganne <bganne@cisco.com>

# ead1e536 31-Oct-2019 Renato Botelho do Couto <renato@netgate.com>

misc: Fix python scripts shebang line

Type: fix

Since CentOS 8, RPM build script doesn't accept '#!/usr/bin/env python'
as a valid shebang line. It requires scripts to explicitly chose
between python2 or python3.

Change all to use python3 as suggested by Paul Vinciguerra.

Depends-On: https://gerrit.fd.io/r/23170

Signed-off-by: Renato Botelho do Couto <renato@netgate.com>
Change-Id: Ie72af9f60fd0609e07f05b70f8d96e738b2754d1

# 45a95dd7 05-Nov-2019 Klement Sekera <ksekera@cisco.com>

tests: support setting random seed

Log the random seed used when running tests and provide means to re-use
it in a later run.

Type: feature

Change-Id: I18d2a36ee802b901d4cca5577df41cec07f09cc0
Signed-off-by: Klement Sekera <ksekera@cisco.com>

# b31d3935 31-Oct-2019 Andrew Yourtchenko <ayourtch@gmail.com>

tests: Revert "tests: add sending SIGQUIT to vpp quit sequence"

This reverts commit 4465fe1e30556b40c9ef8a1a155aa1665102dd22.

That commit makes *every* test to send SIGQUIT to VPP when it is done,
thereby triggering the coredump if there was none.

It is most surely not what we want.

Type: test

Change-Id: I3cc4c96f44c05527d586142f8feccc2f93c3e536
Signed-off-by: Andrew Yourtchenko <ayourtch@gmail.com>

# 4465fe1e 27-Oct-2019 Paul Vinciguerra <pvinci@vinciconsulting.com>

tests: add sending SIGQUIT to vpp quit sequence

As suggested in the docs by:

Type: test

Change-Id: I10caf16220e2507351636c1b26e57e19bcf8a542
Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>

# 97ea2f4e 29-Oct-2019 Dave Wallace <dwallacelf@gmail.com>

tests: decode worker stdout/stderr output

Type: fix

Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
Change-Id: I309254e9e99920fbeaa50eea503c1c2a9470abfe

# 63cb8827 13-Oct-2019 Andrew Yourtchenko <ayourtch@gmail.com>

tests: make RA tests run on VPP time

the IPv6 RA tests take timing into the account,
but the time inside VPP may go slightly differently compared
to the time inside the driving python thread,
if the machine running the tests is heavily loaded.
Make a sleep function which sleeps "on VPP time" and use it.

Change-Id: I3b34b0164f6e0ec7a619b92ee308089a4a8935e3
Type: test
Signed-off-by: Andrew Yourtchenko <ayourtch@gmail.com>

# 2456433d 21-Oct-2019 Dave Wallace <dwallacelf@gmail.com>

tests: DEBUG=gdb[server]-all to debug worker(s)

Type: test

Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
Change-Id: I0264451632c1ce780b38a2c15a7e34350fc6d521

# 586d3edf 21-Oct-2019 Andrew Yourtchenko <ayourtch@gmail.com>

tests: add PID of VPP under test to API dump filename for the test

Some testclasses (e.g. IPSec) have methods with the same names,
this causes a race when running in parallel to save the API and
move it into the test's directory - the name of the file has
only the test method name as a discriminator.

Saving two traces from two VPP instances "succeeds", in that
it silently overwrites the file. But only one mv operation
works - the other gives an error "file not found" and fails the test.

Solution: add the PID of VPP under test to the API dump filename,
this way the filenames become guaranteed unique and the race is avoided.

Change-Id: I1bb47ed73369fea0f1bfbf907588a51e2cabfb7f
Type: test
Signed-off-by: Andrew Yourtchenko <ayourtch@gmail.com>

# 45ec5708 16-Oct-2019 Ole Troan <ot@cisco.com>

vppinfra: test support python3

Type: fix
Signed-off-by: Ole Troan <ot@cisco.com>
Change-Id: Id1794f38fcb776ded9b769141f5f47d7be75f247

# 6ed154f7 15-Oct-2019 Ole Troan <ot@cisco.com>

tests: cli wrapper should return string

Python3 fixes.

Type: fix
Signed-off-by: Ole Troan <ot@cisco.com>
Change-Id: I648b2142d45dfab9146a02eeb1b12de11103ff9f
Signed-off-by: Ole Troan <ot@cisco.com>

# 8d829f6c 13-Oct-2019 Andrew Yourtchenko <ayourtch@gmail.com>

tests: make pg_start() wait until pg completes

A sizable number of tests call pg_start() to get the packets flowing and then
immediately expect to have the entirety of the packets gone through.
This works on powerful and unstressed hardware, but fails in beautifully random
ways under load.

This also necessitates the complicated logic of remembering the "zombie captures",
then sleeping for some time before cleaning them up....

The solution is simple: in pg_start(), start the generators, wait till they
all finish, clean up, done.

Signed-off-by: Andrew Yourtchenko <ayourtch@gmail.com>
Change-Id: I930e51b7aae39c9841d22dd905a4d13a465a672b
Type: test

# 4f05a8e4 13-Oct-2019 Andrew Yourtchenko <ayourtch@gmail.com>

tests: log error which happens during connect and wait longer for stats socket

Intermittently, a test would start VPP, but no testcases would execute.
This would be more probable apparent during the high load or if there
is another testcase dumping the core at that moment.
Adding the logging to the connection revealed it was the stats socket
connection erroring with error -2. Increasing the deadline
from 3 seconds to 5 minutes has eliminated this error.

Change-Id: I40bd7e642abb9e2aef0238c612e4c34781de5db2
Signed-off-by: Andrew Yourtchenko <ayourtch@gmail.com>
Type: test

# 3a350702 02-Sep-2019 Klement Sekera <ksekera@cisco.com>

tests: fix centos detection

Type: fix
Fixes: defde0f870

Change-Id: Ib81e3610bd0fc22901a902dc76692f1951ead2b8
Signed-off-by: Klement Sekera <ksekera@cisco.com>

# 630ab584 19-Jul-2019 Klement Sekera <ksekera@cisco.com>

ip: reassembly: send packet out on correct worker

Note which worker received fragment with offset zero and use this worker
to send out the reassembled packet.

Type: fix
Change-Id: I1d3cee16788db3b230682525239c0100d51dc380
Signed-off-by: Klement Sekera <ksekera@cisco.com>

# 00625a64 31-Jul-2019 Neale Ranns <nranns@cisco.com>

tests: Split IPSec ESP into parameterized tests per engine

Type: feature

Change-Id: Icb1bd3fce768aebf8919c63a104f771ca7fa1d6f
Signed-off-by: Neale Ranns <nranns@cisco.com>

# 22e9cfd7 23-Jul-2019 Mohsin Kazmi <sykazmi@cisco.com>

pg: add GSO support

Type: feature

Change-Id: I72676495a85fbecc946aa266a75234cce70c3a5e
Signed-off-by: Mohsin Kazmi <sykazmi@cisco.com>

# f7457521 13-Jul-2019 Paul Vinciguerra <pvinci@vinciconsulting.com>

tests: fix error in VppDiedError exception

Discovered running test-debug job in CI.

- fix missing paren () around format value.
Type: test

Change-Id: Iebddd3035a435f8ad1cb1d6fa4e8e8c2d4ddaf96
Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>

# bfb9fe34 09-Jul-2019 Paul Vinciguerra <pvinci@vinciconsulting.com>

tests: remove unused class attribute in VppTestCase

Identified in post-merge review.

Type: test

Change-Id: I46e19285479437561a43975ba9b5cb68f478736c
Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>

# 0cbc71d7 03-Jul-2019 Paul Vinciguerra <pvinci@vinciconsulting.com>

tests: treat all truthy env vars the same way

Introduce a new class, that returns the truthiness of a env var.
Since an environment variable is just a string, it would normally
be true if not unset. The new class returns true when the env var is
set to a string that would be considered true.

Type: test
Depends-on: https://gerrit.fd.io/r/20484

Change-Id: I90ef010156f6fec246bde5c0e208ced1869b180f
Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>

# 32dcd3b2 08-Jul-2019 Dave Barach <dave@barachs.net>

vppinfra: allocate bihash virtual space on demand

Reduces the vpp image virtual size by multiple gigabytes

Add a "show bihash" command which displays configured and current
virtual space in use by bihash tables.

Modify the .py test framework to call "show bihash" on test tear-down

Type: refactor

Change-Id: Ifc1b7e2c43d29bbef645f6802fa29ff8ef09940c
Signed-off-by: Dave Barach <dave@barachs.net>

# 063366eb 30-Jun-2019 Paul Vinciguerra <pvinci@vinciconsulting.com>

tests: Have worker return immediately on bad executable

No reason to waste CI cycles if we know the test will not run.

17:55:11 ==============================================================================
17:55:11 VPP Object Model Test
17:55:11 ==============================================================================
17:55:11 Exception in thread Thread-4:
17:55:11 Traceback (most recent call last):
17:55:11 File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
17:55:11 self.run()
17:55:11 File "/w/workspace/vpp-beta-verify-master-ubuntu1804/test/framework.py", line 1475, in run
17:55:11 stdout=subprocess.PIPE, stderr=subprocess.PIPE)
17:55:11 File "/w/workspace/vpp-beta-verify-master-ubuntu1804/test/run/venv/local/lib/python2.7/site-packages/subprocess32.py", line 617, in __init__
17:55:11 restore_signals, start_new_session)
17:55:11 File "/w/workspace/vpp-beta-verify-master-ubuntu1804/test/run/venv/local/lib/python2.7/site-packages/subprocess32.py", line 1415, in _execute_child
17:55:11 raise child_exception_type(errno_num, err_msg)
17:55:11 OSError: [Errno 2] No such file or directory: '/w/workspace/vpp-beta-verify-master-ubuntu1804/test/build/vom_test/vom_test'
17:55:11 17:55:11,328 Timeout! Worker did not finish in 120s
17:55:11 run C++ VOM tests ERROR [ temp dir used by test case: /tmp/vpp-unittest-VOMTestCase-vpMcWF ]

Type: fix

Change-Id: I3d8252807e98a09a8abd70de8a22517151f9d786
Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>

# fea82607 27-Jun-2019 Paul Vinciguerra <pvinci@vinciconsulting.com>

tests: handle TypeErrors in calls to VppDiedError

Type: test

Change-Id: I869768ed25efa3350f025fba329466db69805803
Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>

# 5dd6a7b2 19-Jun-2019 Paul Vinciguerra <pvinci@vinciconsulting.com>

tests: if vpp is dead, specify the test name in the output

'Exception: VPP is dead when setting up the test'
is more helpful with the test info.

Type: test

Change-Id: I6a262fdcf2ecb33aae17bef3d19745053a0bade8
Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>

# 496b0dee 20-Jun-2019 Paul Vinciguerra <pvinci@vinciconsulting.com>

tests: refactor VppDiedError.

- Move Exception into same module as TestCase.
- Move the error reporting logic inside the error.
- Allows testing of the returncode and signal_name for tests to consume.
- Fix the signal reporting code:
VppDiedError: VPP subprocess died unexpectedly with returncode -6 [unknown].
displays as:
VppDiedError: VPP subprocess died unexpectedly with return code: -6 [SIGABRT].

Type: test

Change-Id: I8488ab318a596c9b737308829cedfb7e96e57302
Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>

# 3a9f11e6 19-Jun-2019 Paul Vinciguerra <pvinci@vinciconsulting.com>

tests: add sudo to gdb commands

Prepend sudo to the gdb command line.

Type: test

Change-Id: I09013c3ca512e26de0c46c02f643b21064ba499c
Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>

# 499ea648 15-Mar-2019 Paul Vinciguerra <pvinci@vinciconsulting.com>

tests: framework gracefully handle 'VppTransportShmemIOError'

Traceback (most recent call last):
File "/vpp/test/framework.py", line 593, in tearDown
self.logger.info(self.vapi.ppcli("api trace save %s" % api_trace))
File "/vpp/test/vpp_papi_provider.py", line 413, in ppcli
return cli + "\n" + str(self.cli(cli))
File "/vpp/test/vpp_papi_provider.py", line 402, in cli
r = self.papi.cli_inband(cmd=cli)
File "/vpp/src/vpp-api/python/vpp_papi/vpp_papi.py", line 100, in __call__
return self._func(**kwargs)
File "/vpp/src/vpp-api/python/vpp_papi/vpp_papi.py", line 414, in f
return self._call_vpp(i, msg, multipart, **kwargs)
File "/vpp/src/vpp-api/python/vpp_papi/vpp_papi.py", line 634, in _call_vpp
msg = self.transport.read()
File "/vpp/src/vpp-api/python/vpp_papi/vpp_transport_shmem.py", line 120, in read
raise VppTransportShmemIOError(rv, 'vac_read failed')
VppTransportShmemIOError: [Errno -1] vac_read failed

Change-Id: I767e48c4d03081eb5df6a8aa67da7e192d25e4cc
Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>

# 6aa58b73 16-May-2019 Klement Sekera <ksekera@cisco.com>

make test: fix packet counter assert

Change-Id: Ic6e6ac0629bbf07f7dad1803a85aabfdccbc2035
Signed-off-by: Klement Sekera <ksekera@cisco.com>

# 233e4681 16-May-2019 Ole Troan <ot@cisco.com>

stats: support multiple works for error counters

The current code only allowed access to the main thread error counters.
That is not so useful for a multi worker instance.
No return a vector indexed by thread of counter_t values.

Type: fix

Change-Id: Ie322c8889c0c8175e1116e71de04a2cf453b9ed7
Signed-off-by: Ole Troan <ot@cisco.com>

# 3a343d42 16-May-2019 Klement Sekera <ksekera@cisco.com>

reassembly: prevent long chain attack

limit max # of fragments to 3 per packet by default
add API option to configure the limit at runtime

Change-Id: Ie4b9507bf5c6095b9a5925972b37fe0032f4f9e8
Signed-off-by: Klement Sekera <ksekera@cisco.com>

# 7d31ab2a 08-May-2019 Dave Barach <dave@barachs.net>

add mactime plugin unit / code coverage tests

The unit and code coverage tests are boring. The rest of the patch
involves test and packet-generator infra cleanups.

Teach the "make test-xxx" family of targets to set the api test plugin
path correctly, to make "binary-api <api-message-name> <args>" debug
CLI commands work correctly in the "make test"
environment. Unfortunately involves both the top-level and test

Add a minor pg cli feature, a CLI to manually set

Consider the case where one configures an interface with both a
device-input and an output feature. To test the output feature using
the pg, it's necessary to inject packets into the interface output
node with both b->sw_if_index[VLIB_TX] and b->sw_if_index[VLIB_RX] set
correctly. For example:

packet-generator new {
name tx
limit 15
size 128-128
interface local0 # rx: device input feature not configured on local0
tx-interface loop0 # tx: output node requires b->sw_if_index[VLIB_TX]
node loop0-output
data {
hex 0x01005e7ffffa000dead0000008000102030405060708090a0b0c0d0e0f0102030405

Fix a longstanding bug in the packet generator stream setup. Remove
kludges which set b->sw_if_index[VLIB_TX] to ~0 [in multiple places]
instead of using the stream value s->sw_if_index[VLIB_TX], and setting
THAT datum correctly.

Change-Id: I1097a18e8db73661ded6b822c1d718f7e5cf36ed
Signed-off-by: Dave Barach <dave@barachs.net>

# cc991497 07-May-2019 Alexandre Poirrier <apoirrie@cisco.com>

make test: add python3 compatibility in extend_packet

Change-Id: I342ff8e44e0d1ebd9f2c8cc386372b2f5a731080
Signed-off-by: Alexandre Poirrier <apoirrie@cisco.com>

# 3e9b7a21 10-Apr-2019 Dave Wallace <dwallacelf@gmail.com>

make test: Fix truncated core file

- Wait for coredump to complete before killing VPP.

Change-Id: Ib7b96b339f796f9ef7e0eadff81d181464b8a03a
Signed-off-by: Dave Wallace <dwallacelf@gmail.com>

# ad64687e 06-May-2019 Dave Barach <dave@barachs.net>

gcov / test framework: sigterm not sigkill

Otherwise, gcov data vanishes without a trace.

Add a __gcov_flush() call to the unix signal handler, under #ifdef

Change-Id: I2726e671b26dfbe7fae88f46a8207bb2b5106884
Signed-off-by: Dave Barach <dave@barachs.net>

# 4ff09ae3 15-Apr-2019 Ole Troan <ot@cisco.com>

API: Python and Unix domain socket improvement

Handle the case where buffer overflows.
Then SOCK_SEQPACKET assumption that multiple API messages
are not returned by recv() is broken. Use SOCK_STREAM for
API exchanges instead.

Add support for running tests over sockets.
make test SOCKET=1

Change-Id: Ibe5fd69b1bf617de4c7ba6cce0a7c2b3f97a2821
Signed-off-by: Ole Troan <ot@cisco.com>

# 90cf21b5 13-Mar-2019 Paul Vinciguerra <pvinci@vinciconsulting.com>

Tests: Refactor tearDown show command logging, add lifecycle markers.

This change adds a consistent interface for adding test-specific show commands to

It also adds log markers for the execution of setUp[Class], tearDown[Class]
in the logs.

Change-Id: I7d42e396e594a59e866a7d55dac0af25548e657a
Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>

# 2e1c8967 10-Apr-2019 Ole Troan <ot@cisco.com>

API: Fix shared memory only action handlers.

Some API action handlers called vl_msg_ai_send_shmem()
directly. That breaks Unix domain socket API transport.

A couple (bond / vhost) also tried to send a sw_interface_event
directly, but did not send the message to all that had
registred interest. That scheme never worked correctly.
Refactored and improved the interface event code.

Change-Id: Idb90edfd8703c6ae593b36b4eeb4d3ed7da5c808
Signed-off-by: Ole Troan <ot@cisco.com>

# d7603d97 28-Mar-2019 Neale Ranns <nranns@cisco.com>

IPSEC: tunnel fragmentation

Change-Id: I63741a22bc82f5f861e1c0f26a93b5569cc52061
Signed-off-by: Neale Ranns <nranns@cisco.com>

# 192b13f9 15-Mar-2019 Neale Ranns <nranns@cisco.com>

BVI Interface

a new dedicated BVI interface as opposed to [re]using a loopback.

- removes ambiguity over the purpose of a loopback interface
- TX node dedicated to BVI only functions.

Change-Id: I749d6b38440d450ac5b909a28053c75ec9df946a
Signed-off-by: Neale Ranns <nranns@cisco.com>

# 22364e49 13-Mar-2019 Paul Vinciguerra <pvinci@vinciconsulting.com>

Tests: post-merge review comments for https://gerrit.fd.io/r/#/c/18081/

Change-Id: I2195b3264b5db34d869facf669f3e71a04fe3ff9
Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>

# a7427ec6 10-Mar-2019 Paul Vinciguerra <pvinci@vinciconsulting.com>

VPP-1508: Use scapy.compat to manage packet level library differences.

Change-Id: Icdf6abc9e53d33b26fd1d531c7dda6be0bb9cb55
Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>

# 0f6602cb 10-Mar-2019 Paul Vinciguerra <pvinci@vinciconsulting.com>

Tests: fix time.sleep(0) # yield. Reduce sleep related log messages.

Reduce the incidence of:
20:04:23,606 unexpected time.sleep() result - slept for 2.187967e-03s instead of ~6.837845e-04s!

Change-Id: Ic576fda7f75e571c9019111588a6a936ee2cf5c2
Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>

# eaea421e 06-Mar-2019 Paul Vinciguerra <pvinci@vinciconsulting.com>

Tests: Refactor payload_to_info()

All callers of payload_to_info were required to wrap payload with str().
Refactor to call scapy's payload.load for raw payloads or specify the
specific fieldname.

Change-Id: I1c80599d4df8dc129dbb8274733afaad406d5bcf
Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>

# ea2450fa 06-Mar-2019 Paul Vinciguerra <pvinci@vinciconsulting.com>

VPP-1508: Python3 tests. Explicitly specify string formatting.

.format() doesn't understand what to do with binary values by default.
Specify that we expect coersion to string.

Change-Id: Ic4b697f70852124dc85ab231f76b7934d0d71f4d
Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>

# 38a4ec73 28-Nov-2018 Paul Vinciguerra <pvinci@vinciconsulting.com>

make test: Add exception handling around subprocess.

This cleans up exception catching to identify oserrors. By raising the
specific exception closer to the offending call, we get additional stack history
and can add clearer error logging to assist in troubleshooting.

Change-Id: I592e4d46844b822a816485bf54910f8daed92088
Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>

# eb414435 20-Feb-2019 Paul Vinciguerra <pvinci@vinciconsulting.com>

TESTS: refactor framework send_and_expect.

Create a common pg_send method for cases when results are just discarded.

Change-Id: I786960d2d7bbb96dcb407f6e59aa96951b7b19e7
Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>

# 8d991d92 25-Jan-2019 Paul Vinciguerra <pvinci@vinciconsulting.com>

VTL Cleanup: Fix missing calls to setUpClass/tearDownClass, fix numerous TypeErrors.

* TypeError: assertIsNotNone() got an unexpected keyword argument 'msg'

* Correct missing calls to setUpClass/tearDownClass.
If you want the setUpClass and tearDownClass on base classes called then you must call up to them yourself.
The implementations in TestCase are empty.

Cleans up issues in parallel test mode:
Testcase name: VCL Thru Host Stack Bidir Nsock
FAILURE: test_vcl.VCLThruHostStackBidirNsock.test_vcl_thru_host_stack_bi_dir_nsock [test_vcl.VCLThruHostStackBidirNsock.test_vcl_thru_host_stack_bi_dir_nsock]
Testcase name: Bidirectional Forwarding Detection (BFD) (changing auth)
ERROR: test_bfd.BFDAuthOnOffTestCase.test_auth_change_key_delayed [test_bfd.BFDAuthOnOffTestCase.test_auth_change_key_delayed]
ERROR: test_bfd.BFDAuthOnOffTestCase.test_auth_change_key_immediate [test_bfd.BFDAuthOnOffTestCase.test_auth_change_key_immediate]
ERROR: test_bfd.BFDAuthOnOffTestCase.test_auth_off_delayed [test_bfd.BFDAuthOnOffTestCase.test_auth_off_delayed]
ERROR: test_bfd.BFDAuthOnOffTestCase.test_auth_off_immediate [test_bfd.BFDAuthOnOffTestCase.test_auth_off_immediate]
ERROR: test_bfd.BFDAuthOnOffTestCase.test_auth_on_delayed [test_bfd.BFDAuthOnOffTestCase.test_auth_on_delayed]
ERROR: test_bfd.BFDAuthOnOffTestCase.test_auth_on_immediate [test_bfd.BFDAuthOnOffTestCase.test_auth_on_immediate]
Testcase name: Punt Socket for IPv4
ERROR: test_punt.TestIP4PuntSocket.test_punt_socket_dump [test_punt.TestIP4PuntSocket.test_punt_socket_dump]
ERROR: test_punt.TestIP4PuntSocket.test_punt_socket_traffic_multi_port_multi_sockets [test_punt.TestIP4PuntSocket.test_punt_socket_traffic_multi_port_multi_sockets]
ERROR: test_punt.TestIP4PuntSocket.test_punt_socket_traffic_multi_ports_single_socket [test_punt.TestIP4PuntSocket.test_punt_socket_traffic_multi_ports_single_socket]
ERROR: test_punt.TestIP4PuntSocket.test_punt_socket_traffic_single_port_single_socket [test_punt.TestIP4PuntSocket.test_punt_socket_traffic_single_port_single_socket]
Testcase name: Bidirectional Forwarding Detection (BFD) (IPv6)
ERROR: test_bfd.BFD6TestCase.test_echo [test_bfd.BFD6TestCase.test_echo]
ERROR: test_bfd.BFD6TestCase.test_echo_looped_back [test_bfd.BFD6TestCase.test_echo_looped_back]
ERROR: test_bfd.BFD6TestCase.test_intf_deleted [test_bfd.BFD6TestCase.test_intf_deleted]
ERROR: test_bfd.BFD6TestCase.test_session_up [test_bfd.BFD6TestCase.test_session_up]
ERROR: test_bfd.BFD6TestCase.test_session_up_by_ip [test_bfd.BFD6TestCase.test_session_up_by_ip]
Testcase name: Bidirectional Forwarding Detection (BFD) (CLI)
ERROR: test_bfd.BFDCLITestCase.test_add_mod_del_bfd_udp [test_bfd.BFDCLITestCase.test_add_mod_del_bfd_udp]
ERROR: test_bfd.BFDCLITestCase.test_add_mod_del_bfd_udp6 [test_bfd.BFDCLITestCase.test_add_mod_del_bfd_udp6]
ERROR: test_bfd.BFDCLITestCase.test_add_mod_del_bfd_udp6_auth [test_bfd.BFDCLITestCase.test_add_mod_del_bfd_udp6_auth]
ERROR: test_bfd.BFDCLITestCase.test_add_mod_del_bfd_udp_auth [test_bfd.BFDCLITestCase.test_add_mod_del_bfd_udp_auth]
ERROR: test_bfd.BFDCLITestCase.test_admin_up_down [test_bfd.BFDCLITestCase.test_admin_up_down]
ERROR: test_bfd.BFDCLITestCase.test_auth_on_off [test_bfd.BFDCLITestCase.test_auth_on_off]
ERROR: test_bfd.BFDCLITestCase.test_auth_on_off_delayed [test_bfd.BFDCLITestCase.test_auth_on_off_delayed]
ERROR: test_bfd.BFDCLITestCase.test_set_del_meticulous_sha1_key [test_bfd.BFDCLITestCase.test_set_del_meticulous_sha1_key]
ERROR: test_bfd.BFDCLITestCase.test_set_del_sha1_key [test_bfd.BFDCLITestCase.test_set_del_sha1_key]
ERROR: test_bfd.BFDCLITestCase.test_set_del_udp_echo_source [test_bfd.BFDCLITestCase.test_set_del_udp_echo_source]
ERROR: test_bfd.BFDCLITestCase.test_show [test_bfd.BFDCLITestCase.test_show]
Testcase name: VAPI test
ERROR: test_vapi.VAPITestCase.test_vapi_c [test_vapi.VAPITestCase.test_vapi_c]
Testcase name: Container integration extended testcases
ERROR: test_container.ContainerIntegrationTestCase.test_0010_basic_conn_test [test_container.ContainerIntegrationTestCase.test_0010_basic_conn_test]
ERROR: test_container.ContainerIntegrationTestCase.test_0011_basic_conn_test [test_container.ContainerIntegrationTestCase.test_0011_basic_conn_test]
ERROR: test_container.ContainerIntegrationTestCase.test_0050_loopback_prepare_test [test_container.ContainerIntegrationTestCase.test_0050_loopback_prepare_test]
ERROR: test_container.ContainerIntegrationTestCase.test_0110_basic_conn_test [test_container.ContainerIntegrationTestCase.test_0110_basic_conn_test]
ERROR: test_container.ContainerIntegrationTestCase.test_0111_basic_conn_test [test_container.ContainerIntegrationTestCase.test_0111_basic_conn_test]
ERROR: test_container.ContainerIntegrationTestCase.test_0200_basic_conn_test [test_container.ContainerIntegrationTestCase.test_0200_basic_conn_test]
ERROR: test_container.ContainerIntegrationTestCase.test_0210_basic_conn_test [test_container.ContainerIntegrationTestCase.test_0210_basic_conn_test]
ERROR: test_container.ContainerIntegrationTestCase.test_0211_basic_conn_test [test_container.ContainerIntegrationTestCase.test_0211_basic_conn_test]
ERROR: test_container.ContainerIntegrationTestCase.test_0300_unconfigure_commands [test_container.ContainerIntegrationTestCase.test_0300_unconfigure_commands]
ERROR: test_container.ContainerIntegrationTestCase.test_0410_spoof_test [test_container.ContainerIntegrationTestCase.test_0410_spoof_test]
ERROR: test_container.ContainerIntegrationTestCase.test_0411_spoof_test [test_container.ContainerIntegrationTestCase.test_0411_spoof_test]
Testcase name: Re-enable IPFIX
ERROR: test_flowprobe.ReenableIPFIX.test_0011 [test_flowprobe.ReenableIPFIX.test_0011]
Testcase name: VXLAN over IPv6 Test Case
ERROR: setUpClass [setUpClass (test_vxlan6.TestVxlan6)]
Testcase name: JVPP Core Test Case
ERROR: test_jvpp.TestJVpp.test_vpp_acl_callback_api [test_jvpp.TestJVpp.test_vpp_acl_callback_api]
ERROR: test_jvpp.TestJVpp.test_vpp_acl_future_api [test_jvpp.TestJVpp.test_vpp_acl_future_api]
ERROR: test_jvpp.TestJVpp.test_vpp_core_callback_api [test_jvpp.TestJVpp.test_vpp_core_callback_api]
ERROR: test_jvpp.TestJVpp.test_vpp_core_future_api [test_jvpp.TestJVpp.test_vpp_core_future_api]
ERROR: test_jvpp.TestJVpp.test_vpp_ioamexport_callback_api [test_jvpp.TestJVpp.test_vpp_ioamexport_callback_api]
ERROR: test_jvpp.TestJVpp.test_vpp_ioamexport_future_api [test_jvpp.TestJVpp.test_vpp_ioamexport_future_api]
ERROR: test_jvpp.TestJVpp.test_vpp_ioampot_callback_api [test_jvpp.TestJVpp.test_vpp_ioampot_callback_api]
ERROR: test_jvpp.TestJVpp.test_vpp_ioampot_future_api [test_jvpp.TestJVpp.test_vpp_ioampot_future_api]
ERROR: test_jvpp.TestJVpp.test_vpp_ioamtrace_callback_api [test_jvpp.TestJVpp.test_vpp_ioamtrace_callback_api]
ERROR: test_jvpp.TestJVpp.test_vpp_ioamtrace_future_api [test_jvpp.TestJVpp.test_vpp_ioamtrace_future_api]
ERROR: test_jvpp.TestJVpp.test_vpp_snat_callback_api [test_jvpp.TestJVpp.test_vpp_snat_callback_api]
ERROR: test_jvpp.TestJVpp.test_vpp_snat_future_api [test_jvpp.TestJVpp.test_vpp_snat_future_api]
Testcase name: LDP Cut Thru Tests
FAILURE: test_vcl.LDPCutThruTestCase.test_ldp_cut_thru_iperf3 [test_vcl.LDPCutThruTestCase.test_ldp_cut_thru_iperf3]
Testcase name: ACL plugin connection-oriented extended testcases
ERROR: test_acl_plugin_conns.ACLPluginConnTestCase.test_0000_conn_prepare_test [test_acl_plugin_conns.ACLPluginConnTestCase.test_0000_conn_prepare_test]
ERROR: test_acl_plugin_conns.ACLPluginConnTestCase.test_0001_basic_conn_test [test_acl_plugin_conns.ACLPluginConnTestCase.test_0001_basic_conn_test]
ERROR: test_acl_plugin_conns.ACLPluginConnTestCase.test_0002_basic_conn_test [test_acl_plugin_conns.ACLPluginConnTestCase.test_0002_basic_conn_test]
ERROR: test_acl_plugin_conns.ACLPluginConnTestCase.test_0005_clear_conn_test [test_acl_plugin_conns.ACLPluginConnTestCase.test_0005_clear_conn_test]
ERROR: test_acl_plugin_conns.ACLPluginConnTestCase.test_0006_clear_conn_test [test_acl_plugin_conns.ACLPluginConnTestCase.test_0006_clear_conn_test]
ERROR: test_acl_plugin_conns.ACLPluginConnTestCase.test_0011_active_conn_test [test_acl_plugin_conns.ACLPluginConnTestCase.test_0011_active_conn_test]
ERROR: test_acl_plugin_conns.ACLPluginConnTestCase.test_0012_active_conn_test [test_acl_plugin_conns.ACLPluginConnTestCase.test_0012_active_conn_test]
ERROR: test_acl_plugin_conns.ACLPluginConnTestCase.test_1001_basic_conn_test [test_acl_plugin_conns.ACLPluginConnTestCase.test_1001_basic_conn_test]
ERROR: test_acl_plugin_conns.ACLPluginConnTestCase.test_1002_basic_conn_test [test_acl_plugin_conns.ACLPluginConnTestCase.test_1002_basic_conn_test]
ERROR: test_acl_plugin_conns.ACLPluginConnTestCase.test_1005_clear_conn_test [test_acl_plugin_conns.ACLPluginConnTestCase.test_1005_clear_conn_test]
ERROR: test_acl_plugin_conns.ACLPluginConnTestCase.test_1006_clear_conn_test [test_acl_plugin_conns.ACLPluginConnTestCase.test_1006_clear_conn_test]
ERROR: test_acl_plugin_conns.ACLPluginConnTestCase.test_1011_active_conn_test [test_acl_plugin_conns.ACLPluginConnTestCase.test_1011_active_conn_test]
ERROR: test_acl_plugin_conns.ACLPluginConnTestCase.test_1012_active_conn_test [test_acl_plugin_conns.ACLPluginConnTestCase.test_1012_active_conn_test]
ERROR: test_acl_plugin_conns.ACLPluginConnTestCase.test_2000_prepare_for_tcp_test [test_acl_plugin_conns.ACLPluginConnTestCase.test_2000_prepare_for_tcp_test]
ERROR: test_acl_plugin_conns.ACLPluginConnTestCase.test_2001_tcp_transient_conn_test [test_acl_plugin_conns.ACLPluginConnTestCase.test_2001_tcp_transient_conn_test]
ERROR: test_acl_plugin_conns.ACLPluginConnTestCase.test_2002_tcp_transient_conn_test [test_acl_plugin_conns.ACLPluginConnTestCase.test_2002_tcp_transient_conn_test]
ERROR: test_acl_plugin_conns.ACLPluginConnTestCase.test_2003_tcp_transient_conn_test [test_acl_plugin_conns.ACLPluginConnTestCase.test_2003_tcp_transient_conn_test]
ERROR: test_acl_plugin_conns.ACLPluginConnTestCase.test_2004_tcp_transient_conn_test [test_acl_plugin_conns.ACLPluginConnTestCase.test_2004_tcp_transient_conn_test]
ERROR: test_acl_plugin_conns.ACLPluginConnTestCase.test_2005_tcp_transient_teardown_conn_test [test_acl_plugin_conns.ACLPluginConnTestCase.test_2005_tcp_transient_teardown_conn_test]
ERROR: test_acl_plugin_conns.ACLPluginConnTestCase.test_2006_tcp_transient_teardown_conn_test [test_acl_plugin_conns.ACLPluginConnTestCase.test_2006_tcp_transient_teardown_conn_test]
ERROR: test_acl_plugin_conns.ACLPluginConnTestCase.test_3001_tcp_transient_conn_test [test_acl_plugin_conns.ACLPluginConnTestCase.test_3001_tcp_transient_conn_test]
ERROR: test_acl_plugin_conns.ACLPluginConnTestCase.test_3002_tcp_transient_conn_test [test_acl_plugin_conns.ACLPluginConnTestCase.test_3002_tcp_transient_conn_test]
ERROR: test_acl_plugin_conns.ACLPluginConnTestCase.test_3003_tcp_transient_conn_test [test_acl_plugin_conns.ACLPluginConnTestCase.test_3003_tcp_transient_conn_test]
ERROR: test_acl_plugin_conns.ACLPluginConnTestCase.test_3004_tcp_transient_conn_test [test_acl_plugin_conns.ACLPluginConnTestCase.test_3004_tcp_transient_conn_test]
ERROR: test_acl_plugin_conns.ACLPluginConnTestCase.test_3005_tcp_transient_teardown_conn_test [test_acl_plugin_conns.ACLPluginConnTestCase.test_3005_tcp_transient_teardown_conn_test]
ERROR: test_acl_plugin_conns.ACLPluginConnTestCase.test_3006_tcp_transient_teardown_conn_test [test_acl_plugin_conns.ACLPluginConnTestCase.test_3006_tcp_transient_teardown_conn_test]
Testcase name: LDP Thru Host Stack Echo
FAILURE: test_vcl.LDPThruHostStackEcho.test_ldp_thru_host_stack_echo [test_vcl.LDPThruHostStackEcho.test_ldp_thru_host_stack_echo]
Testcase name: Bidirectional Forwarding Detection (BFD) - API
ERROR: test_bfd.BFDAPITestCase.test_activate_auth [test_bfd.BFDAPITestCase.test_activate_auth]
ERROR: test_bfd.BFDAPITestCase.test_add_auth_nonexistent_key [test_bfd.BFDAPITestCase.test_add_auth_nonexistent_key]
ERROR: test_bfd.BFDAPITestCase.test_add_bfd [test_bfd.BFDAPITestCase.test_add_bfd]
ERROR: test_bfd.BFDAPITestCase.test_add_bfd6 [test_bfd.BFDAPITestCase.test_add_bfd6]
ERROR: test_bfd.BFDAPITestCase.test_add_bfd_sha1 [test_bfd.BFDAPITestCase.test_add_bfd_sha1]
ERROR: test_bfd.BFDAPITestCase.test_add_sha1_keys [test_bfd.BFDAPITestCase.test_add_sha1_keys]
ERROR: test_bfd.BFDAPITestCase.test_change_key [test_bfd.BFDAPITestCase.test_change_key]
ERROR: test_bfd.BFDAPITestCase.test_deactivate_auth [test_bfd.BFDAPITestCase.test_deactivate_auth]
ERROR: test_bfd.BFDAPITestCase.test_double_add [test_bfd.BFDAPITestCase.test_double_add]
ERROR: test_bfd.BFDAPITestCase.test_double_add_sha1 [test_bfd.BFDAPITestCase.test_double_add_sha1]
ERROR: test_bfd.BFDAPITestCase.test_mod_bfd [test_bfd.BFDAPITestCase.test_mod_bfd]
ERROR: test_bfd.BFDAPITestCase.test_set_del_udp_echo_source [test_bfd.BFDAPITestCase.test_set_del_udp_echo_source]
ERROR: test_bfd.BFDAPITestCase.test_shared_sha1_key [test_bfd.BFDAPITestCase.test_shared_sha1_key]
Testcase name: LDP Thru Host Stack Iperf
FAILURE: test_vcl.LDPThruHostStackIperf.test_ldp_thru_host_stack_iperf3 [test_vcl.LDPThruHostStackIperf.test_ldp_thru_host_stack_iperf3]
Testcase name: Bidirectional Forwarding Detection (BFD) (SHA1 auth)
ERROR: test_bfd.BFDSHA1TestCase.test_session_up [test_bfd.BFDSHA1TestCase.test_session_up]
Testcase name: Punt Socket for IPv6
ERROR: test_punt.TestIP6PuntSocket.test_punt_socket_dump [test_punt.TestIP6PuntSocket.test_punt_socket_dump]
ERROR: test_punt.TestIP6PuntSocket.test_punt_socket_traffic_multi_port_multi_sockets [test_punt.TestIP6PuntSocket.test_punt_socket_traffic_multi_port_multi_sockets]
ERROR: test_punt.TestIP6PuntSocket.test_punt_socket_traffic_multi_ports_single_socket [test_punt.TestIP6PuntSocket.test_punt_socket_traffic_multi_ports_single_socket]
ERROR: test_punt.TestIP6PuntSocket.test_punt_socket_traffic_single_port_single_socket [test_punt.TestIP6PuntSocket.test_punt_socket_traffic_single_port_single_socket]
Testcase name: Disable Flowprobe feature
ERROR: test_flowprobe.DisableFP.test_0001 [test_flowprobe.DisableFP.test_0001]
Testcase name: Disable IPFIX
ERROR: test_flowprobe.DisableIPFIX.test_0001 [test_flowprobe.DisableIPFIX.test_0001]
Testcase name: VPP Object Model Test
ERROR: test_vom.VOMTestCase.test_vom_cpp [test_vom.VOMTestCase.test_vom_cpp]
Testcase name: BFD-FIB interactions (IPv6)
ERROR: test_bfd.BFDFIBTestCase.test_session_with_fib [test_bfd.BFDFIBTestCase.test_session_with_fib]
Testcase name: Bidirectional Forwarding Detection (BFD)
ERROR: test_bfd.BFD4TestCase.test_echo [test_bfd.BFD4TestCase.test_echo]
ERROR: test_bfd.BFD4TestCase.test_echo_looped_back [test_bfd.BFD4TestCase.test_echo_looped_back]
ERROR: test_bfd.BFD4TestCase.test_intf_deleted [test_bfd.BFD4TestCase.test_intf_deleted]
ERROR: test_bfd.BFD4TestCase.test_session_down [test_bfd.BFD4TestCase.test_session_down]
ERROR: test_bfd.BFD4TestCase.test_session_up [test_bfd.BFD4TestCase.test_session_up]
ERROR: test_bfd.BFD4TestCase.test_session_up_by_ip [test_bfd.BFD4TestCase.test_session_up_by_ip]
Testcase name: VXLAN-GPE Test Case
ERROR: test_vxlan_gpe.TestVxlanGpe.test_decap [test_vxlan_gpe.TestVxlanGpe.test_decap]
ERROR: test_vxlan_gpe.TestVxlanGpe.test_encap [test_vxlan_gpe.TestVxlanGpe.test_encap]
ERROR: test_vxlan_gpe.TestVxlanGpe.test_ucast_flood [test_vxlan_gpe.TestVxlanGpe.test_ucast_flood]
Testcase name: Re-enable Flowprobe feature
ERROR: test_flowprobe.ReenableFP.test_0001 [test_flowprobe.ReenableFP.test_0001]
Testcase name: LDP IPv6 Cut Thru Tests
FAILURE: test_vcl.LDPIpv6CutThruTestCase.test_ldp_ipv6_cut_thru_echo [test_vcl.LDPIpv6CutThruTestCase.test_ldp_ipv6_cut_thru_echo]
VXLAN over IPv6 Test Case

0 attempt(s) left.

Change-Id: Id202ed6ee7f57670f34ec87380244c568b509416
Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>

# 0ede47aa 29-Jan-2019 Klement Sekera <ksekera@cisco.com>

make test: fix logging

Change-Id: I3867d50746f0f424c8e94929171363e5b2e4d470
Signed-off-by: Klement Sekera <ksekera@cisco.com>

# 1ec06fff 16-Jan-2019 Paul Vinciguerra <pvinci@vinciconsulting.com>

VTL: VppTestResult: Must implement empty constructor.

Subclasses of (unittest.TestResult|unittest.TextTestResult) must implement an empty constructor.

The standard python library unittests depend on the empty constructor.

Change-Id: I6d1bd2e7d6996d85a9da66cb7d40af0ed3c30e2b
Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>

# 9889402b 14-Jan-2019 Paul Vinciguerra <pvinci@vinciconsulting.com>

VTL: Add **kwargs to VppTestRunner constructor.

Subclasses should accept **kwargs to ensure compatibility as the
interface changes.


Change-Id: Ia64ffd88899d207a5422f066700d2f7f9c77622e
Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>

# d91fa61e 15-Jan-2019 Klement Sekera <ksekera@cisco.com>

make test: raise packet tracing limit to 1000

Change-Id: I4309cefe13ee861342e7962c7652919a47748656
Signed-off-by: Klement Sekera <ksekera@cisco.com>

# dd3c5d25 14-Jan-2019 Paul Vinciguerra <pvinci@vinciconsulting.com>

VTL: Allow running simple unittest.TestCases.

It came to my attention that Ole added a simple test in:
https://gerrit.fd.io/r/#/c/16381/ and the framework forced him
to launch an instance of VPP to test the formatting of a mac address.

This change allows the test framework to run standard unittest.TestCases
without the need to spawn a VPP instance.

Change-Id: I56651ab27c4c6bf920081a526f168a743d643201
Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>

# 8aeb2207 08-Jan-2019 Paul Vinciguerra <pvinci@vinciconsulting.com>

test framework. Remove unused code.

Change-Id: I1bd14d5aff4918b58aef18fa0a9264e033989f6e
Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>

# a45dc07c 21-Dec-2018 Ole Troan <ot@cisco.com>

Revert "tests: Rework vpp config generation."

This reverts commit 919efad2671993d4c6d5a0dba8eeb99d5c60edf1.

Change-Id: I5ec4fa08a03f508dac684dc1ddbd7e6b85c49b97
Signed-off-by: Ole Troan <ot@cisco.com>

# 087c811b 15-Dec-2018 Paul Vinciguerra <pvinci@vinciconsulting.com>

Python Tests: Make VppTestCase unit testable.

* This allows for VppTestCase to instantiated with an empty constructor.

Change-Id: Ib71d3bfb4529e113f9b73c39de9b95f12ee9cfb4
Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>

# 919efad2 18-Dec-2018 Paul Vinciguerra <pvinci@vinciconsulting.com>

tests: Rework vpp config generation.

* Allows test cases to configure the VPP runtime config
during fixture setup.

* Sample use in a TestCase:
def setUpConstants(cls):
tempdir = cls.tempdir
cls.config.add('punt', 'socket', '%s/socket_punt' % cls.tempdir)
super(TestPuntSocket, cls).setUpConstants()
# enable/disabe a plugin via:
#cls.config.add_plugin('dpdk_plugin.so', 'disable')

* Supports the following config stanzas:
# currently don't support dynamic keys
# 'heapsize'

Change-Id: I44f276487267d26eaa46f87e730bdd861003b234
Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>

# 852f5ef9 15-Dec-2018 Paul Vinciguerra <pvinci@vinciconsulting.com>

VPP-1508 python3 tests: raw_input

'raw_input' does not exist in python3.

Change-Id: I95d916db2834fab83e5e791f35103a4513e610b8
Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>

# defde0f8 06-Dec-2018 Paul Vinciguerra <pvinci@vinciconsulting.com>

Tests: Cleanup @skip decorator.

The runnning environment is static as of module load time,
so only evalute the conditions once at module load time.

Track-by: VPP-1518

Change-Id: I73b0d17ae1ff90789e70307f168d43921829aec8
Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>

# b6d92d81 02-Dec-2018 Klement Sekera <ksekera@cisco.com>

make test: add option for adding extra vpp config

Change-Id: Ief316756635794dadf2f0f4190aa20745df5e53c
Signed-off-by: Klement Sekera <ksekera@cisco.com>

# 14d7e907 10-Dec-2018 Klement Sekera <ksekera@cisco.com>

VPP-1522: harden reassembly code

Change-Id: Ib5a20bff7d8a340ecf50bcd4a023d6bf36382ba3
Signed-off-by: Klement Sekera <ksekera@cisco.com>

# 413f4a5b 28-Nov-2018 Ole Troan <ot@cisco.com>

API: Use string type instead of u8.

The new string type is modelled after string in proto3.
It is always variable length.

Change-Id: I64884067e28a80072c8dac31b7c7c82d6e306051
Signed-off-by: Ole Troan <ot@cisco.com>
Signed-off-by: Michal Cmarada <mcmarada@cisco.com>
Signed-off-by: Ole Troan <ot@cisco.com>

# 6919b0de 09-Dec-2018 Paul Vinciguerra <pvinci@vinciconsulting.com>

test/framework.py: change logger to return 'e' format.

* When the framework logs an unexpected sleep time,
display the units in the same format. Typically,
it has been returning a float and an 'e'.
ex. unexpected time.sleep() result - slept for 0.000107049942017s instead of ~4.88758087158e-05s!

* Defer logger interpolation.

Change-Id: I543ad6d41c7f263d61615341437f80973c0017de
Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>

# 7f99183a 06-Dec-2018 Ole Troan <ot@cisco.com>

Test framework: StringIO fixes for Python3

Add 2/3 support to binarytomac and mactobinary and move to vpp_mac.py

Change-Id: I3dc7e4a24486aee22140c781aae7e44e58935877
Signed-off-by: Ole Troan <ot@cisco.com>

# e88865d7 28-Nov-2018 Pavel Kotucek <pavel.kotucek@pantheon.tech>

VPP-1506: dump local punts and registered punt sockets

Change-Id: If7835e9b80ec9402404bfc8d271eb11a10ef992b
Signed-off-by: Pavel Kotucek <pavel.kotucek@pantheon.tech>

# 68ebc83e 29-Nov-2018 juraj.linkes <juraj.linkes@pantheon.tech>

Skip failing ARM testcases for CI

There are a few tests that fail on ARM and thus we can't enable testing
in CI. Skip the failing tests until they're fixed so that we catch new
failures in CI (when we enable testing for the remaining tests).

Change-Id: Ie896ef5c449ef965029633e38d317a8d5ac26352
Signed-off-by: juraj.linkes <juraj.linkes@pantheon.tech>

# b8c72a4a 08-Nov-2018 Klement Sekera <ksekera@cisco.com>

make test: create virtualenv under /test/

instead of using build-root, use /test/venv directory for virtualenv
similarly, don't pollute build-root with test-built binaries

Change-Id: I1e63c04037eaee718b27b34ef16c9eb0252afa53
Signed-off-by: Klement Sekera <ksekera@cisco.com>

# abec0129 16-Nov-2018 juraj.linkes <juraj.linkes@pantheon.tech>

Split non-parallel testsuite

Split one big suite into smaller suites when not running tests in
parallel. This results in all tests being executed in one iteration.

Change-Id: I0d3d357a95d9cc596b606d5911a5819e8ffdeee5
Signed-off-by: juraj.linkes <juraj.linkes@pantheon.tech>

# 162989e0 26-Nov-2018 Ole Trøan <otroan@employees.org>

Revert "VPP-1508 python3 tests: raw_input"

This reverts commit 72f0004ac5f6e201cbe042593d76df6f2491d743.

Reason for revert: Traceback (most recent call last):
File "/home/ksekera/vpp/test/test_ipsec_ah.py", line 36, in
super(TemplateIpsecAh, cls).setUpClass()
File "/home/ksekera/vpp/test/template_ipsec.py", line 105, in
super(TemplateIpsec, cls).setUpClass()
File "/home/ksekera/vpp/test/framework.py", line 459, in setUpClass
File "/home/ksekera/vpp/test/framework.py", line 475, in quit
six.input("When done debugging, press ENTER to kill the "
AttributeError: 'module' object has no attribute 'input'

Change-Id: Idf0bbfea231730b37bae5dcb4557a0f82ab1b810
Signed-off-by: Ole Troan <ot@cisco.com>

# 86ebba6b 21-Nov-2018 Paul Vinciguerra <pvinci@vinciconsulting.com>

Fix swapped decorators in VppTestCase.

Change-Id: I39b5a8e368f0e0b5c83203141d01f22d909b6f9d
Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>

# 72f0004a 25-Nov-2018 Paul Vinciguerra <pvinci@vinciconsulting.com>

VPP-1508 python3 tests: raw_input

Raw input does not exist in python3 use six.input.

Change-Id: Ie461696ff869057bcc3969a7571602dde570f5b8
Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>

# 61e63bf4 25-Nov-2018 Paul Vinciguerra <pvinci@vinciconsulting.com>

make test: Fix too wide subprocess exceptions

When a command fails, CalledProcessError is raised. testing with except: masks other failures.

Change-Id: I7e3a6739411cb6a4c13e96dd123aff9159213fea
Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>

# ccfead6e 21-Nov-2018 juraj.linkes <juraj.linkes@pantheon.tech>

Fix test framework keepalive

The pipe used for sending keepalive messages was being added to an
instance of KeepAliveReporter and then used by the class itself. This
worked in the past but doesn't anymore. Fix the issue by adding the pipe
to the class instead of an instance.

Change-Id: If7cdca2de23ca78448e80569b155e9e29e81ff94
Signed-off-by: juraj.linkes <juraj.linkes@pantheon.tech>

# 3d9b92a0 21-Nov-2018 juraj.linkes <juraj.linkes@pantheon.tech>

Fix test logging propagation

When running tests with child processes the logs from child processes
would propagate to root logger, potentially resulting in some logs
being emitted twice. Fix this by disabling log propagation to parent
loggers in child processes.

Change-Id: I31eb265c2b7f7bceff627043956a67d6def3da2b
Signed-off-by: juraj.linkes <juraj.linkes@pantheon.tech>

# 5ba9159a 22-Nov-2018 Ole Trøan <otroan@employees.org>

Revert "Fix instance method call in test/framework.py"

This reverts commit c32023b9fd6970ed1cac1b3c7f36233b536d9968.

Change-Id: Ic934d223b10028093b0262e28515bde3ae1ccb71
Signed-off-by: Ole Troan <ot@cisco.com>

# c32023b9 21-Nov-2018 Paul Vinciguerra <pvinci@vinciconsulting.com>

Fix instance method call in test/framework.py

Instance method breaking encapsulation by calling a global fn.

Change-Id: Ifde2a207951143764aed75f1b191aed0bac83704
Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>

# c7b03fe8 18-Nov-2018 Paul Vinciguerra <pvinci@vinciconsulting.com>

VPP-1498: test/framework.py: AttributeError

Traceback if .send_keep_alive is called before pipe is configured.

Traceback (most recent call last):
File "/vpp/test/test_span.py", line 27, in setUpClass
super(TestSpan, cls).setUpClass()
File "/vpp/test/framework.py", line 411, in setUpClass
cls.reporter.send_keep_alive(cls, 'setUpClass')
File "/vpp/test/framework.py", line 172, in send_keep_alive
if self.pipe is None:
File "/vpp/test/framework.py", line 160, in pipe
return self._pipe
AttributeError: 'KeepAliveReporter' object has no attribute '_pipe'

Change-Id: I561d2748441702478a84fbb4580a4d2667d70ffd
Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>

# dd4b2bb3 15-Nov-2018 Paul Vinciguerra <pvinci@vinciconsulting.com>

VPP-1495: Fix intermittent stat sock test failures.

Loosen polling loop while waiting for stats socket.

Classifier IP6 UDP proto Test Case
connect: No such file or directory
ERROR: setUpClass (test_ip4.TestIPLoadBalance)
Traceback (most recent call last):
File "/vpp/test/framework.py", line 424, in setUpClass
cls.statistics = VPPStats(socketname=cls.stats_sock)
File "build/bdist.linux-x86_64/egg/vpp_papi/vpp_stats.py", line 120, in _init_
raise IOError()

Change-Id: I80523122f96eafc41b3ebbf27bee73f6c637d781
Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>

# dfb5f2af 09-Nov-2018 juraj.linkes <juraj.linkes@pantheon.tech>

Fix logging into multiple files

When running tests in one process, only one logger was used and each
testcase added its own file handler, which resulted in logs appearing in
multiple files. Fix this by restoring the creation of new loggers
for each testcase and only reuse the stream handler from parent process.

Change-Id: I5b8471e041dc769128fddb433d33812bfcb5ecf6
Signed-off-by: juraj.linkes <juraj.linkes@pantheon.tech>

# f37c3ba9 08-Nov-2018 Klement Sekera <ksekera@cisco.com>

make test: p2p ethernet refactor

Change-Id: I48b4b3eaa84e4174e4d0305594675bf983184fa0
Signed-off-by: Klement Sekera <ksekera@cisco.com>

# 93cc3ee3 10-Oct-2018 Neale Ranns <neale.ranns@cisco.com>

GBP Endpoint Learning

Learning GBP endpoints over vxlan-gbp tunnels

Change-Id: I1db9fda5a16802d9ad8b4efd4e475614f3b21502
Signed-off-by: Neale Ranns <neale.ranns@cisco.com>

# 40dd73bc 21-Sep-2018 juraj.linkes <juraj.linkes@pantheon.tech>

Setup, teardown, DEBUG=core, FAILFAST fixes

- Fixed a bug when an error occuring in tearDownClass would not result in
test being marked as failed
- Improved test results reporting in cases when an error occurs in setUpClass
and tearDownClass
- Fixed DEBUG=core when the core is produced in setUpClass or
- Reworked DEBUG=core to always be handled after all tests have been
- Fixed FAILFAST=1 for parallel test runs

Change-Id: I3e9cd3b97ba6fa802fa0aa2dd7678ff82eee09ec
Signed-off-by: juraj.linkes <juraj.linkes@pantheon.tech>

# 99743917 08-Oct-2018 Jakub Grajciar <jgrajcia@cisco.com>

Test FW: Use unittest temp dir as unix runtime dir

Change-Id: I5273d5f3f59cc3c43da0a15bb0c4a4056098adcf
Signed-off-by: Jakub Grajciar <jgrajcia@cisco.com>

# 611864f4 26-Sep-2018 Klement Sekera <ksekera@cisco.com>

ipsec: add missing ipv6 ah code & ipv6 tests

Change-Id: I89e90193ded1beb6cb0950c15737f9467efac1c3
Signed-off-by: Klement Sekera <ksekera@cisco.com>

# f94c63ea 26-Sep-2018 Ole Troan <ot@cisco.com>

stats: Split stat_segment and stats code in preparation for deprecation.

Split the stat_segment.c code from stats.c.
Rename stats.[ch] to prepare for removing (19.01?)
In addition stats.api can be removed.

Since the stats aggregation for the stat segment does not use the API, that
part is now done on the main thread. (Old stats aggregator is also left in place).

Change-Id: I9867429f4fc547b1a7ab7f88bc4f3625428d681b
Signed-off-by: Ole Troan <ot@cisco.com>

# cae64f89 19-Sep-2018 juraj.linkes <juraj.linkes@pantheon.tech>

VPP-1421: Reworked results gathering

The previous version sent the whole VPPTestResult object through pipe,
which uses pickle to transfer objects. Pickle does not support sending
any arbitrary objects and was causing issues. Now just a tuple of
(test_id, result) is sent.

Change-Id: I3a3a9e6f1b9ac9b05889babfc1f7560c7ac4471c
Signed-off-by: juraj.linkes <juraj.linkes@pantheon.tech>

# 721872e0 05-Sep-2018 juraj.linkes <juraj.linkes@pantheon.tech>

Fix test summary and retries

There was an issue when tests crashed and weren't properly retried.

Change-Id: Id5ef828ecc9a8dc0f08c50183721db06e162e6c3
Signed-off-by: juraj.linkes <juraj.linkes@pantheon.tech>

# 573c7af1 05-Sep-2018 Ole Troan <ot@cisco.com>

STATS: Update Prometheus port

Update temporary port allocation with fixed allocated from:

Change-Id: I99a7e069fb95d00884458aeacaba06e4713fbb76
Signed-off-by: Ole Troan <ot@cisco.com>

# 73202107 30-Aug-2018 Ole Troan <ot@cisco.com>

STATS: Python binding to access VPP statistics and counters.

from vpp_papi.vpp_stats import VPPStats
s = VPPStats(socketname='/var/run/stats.sock')
c = s.ls('/if/rx')
counters = s.dump(c)

Change-Id: I203ebe60b0c9ee5742aadc737c0f29051757959d
Signed-off-by: Ole Troan <ot@cisco.com>

# 5e2c54d0 30-Aug-2018 juraj.linkes <juraj.linkes@pantheon.tech>

Fix hanging test runner when child process dies

When fixing the test summary after a test run doesn't finish properly I
introduced a bug where child process which died would leave the whole
run hanging. This patch fixed the bug while still having the correct
test summary.

Change-Id: I206b1a7dab4032d24cbc50667b8dd0bdcebb67a6
Signed-off-by: juraj.linkes <juraj.linkes@pantheon.tech>

# 0219b8df 24-Aug-2018 juraj.linkes <juraj.linkes@pantheon.tech>

Fix logging to file and summary for crashed tests

Logging was not being logged to log.txt
Crashed tests were not reported properly when running tests in one

Change-Id: Ica2d703d88351cc8a94bad86764ae4376dc79590
Signed-off-by: juraj.linkes <juraj.linkes@pantheon.tech>

# 184870ac 16-Jul-2018 juraj.linkes <juraj.linkes@pantheon.tech>

CSIT-1139: Implement parallel test execution

The implementation of parallel test execution in VPP Test Framework.
- VPPTestCase test methods are grouped together
- tests are running in separate processes
- VPP instances spawned by tests are assigned to different cores
- output from these processes is redirected through pipes and printed
out testcase by testcase
- TEST_JOBS env var is used to specify the number of parallel processes
- improved test summary
- a bit of code cleanup

Change-Id: I9ca93904d9fe2c3daf980500c64a8611838ae28c
Signed-off-by: juraj.linkes <juraj.linkes@pantheon.tech>

# 8b5dc4fd 23-Jul-2018 Dave Barach <dave@barachs.net>

Create a unit-test plugin

Move the tcp unit test to the plugin
Add a bihash unit test and a "make test" program to call it
Adjust framework.py to load the plugin, which is disabled by default

Change-Id: Ic229d386a56a9d28dbd54974f231149053ca8f93
Signed-off-by: Dave Barach <dave@barachs.net>

# 16782368 23-Jul-2018 Neale Ranns <nranns@cisco.com>

test frame work pump thread exit: set flag then wake

Change-Id: Id95b8a7ad1bf550f615c50d77b16a530cc8fc2c4
Signed-off-by: Neale Ranns <nranns@cisco.com>

# 947ea622 08-Jun-2018 Neale Ranns <nranns@cisco.com>

IGMP improvements

- Enable/Disable an interface for IGMP
- improve logging
- refactor common code
- no orphaned timers
- IGMP state changes in main thread only
- Large groups split over multiple state-change reports
- SSM range configuration API.
- more tests

Change-Id: If5674f1044e7e97274a711f47807c9ba689d7b9a
Signed-off-by: Neale Ranns <nranns@cisco.com>

# 4c5422ee 22-Jun-2018 Klement Sekera <ksekera@cisco.com>

make test: fix RETRIES when setUpClass throws

This change adds handling of special case when setUpClass throws. In
this case TestResults receives a mock object called _ErrorHolder.
By parsing its description, we find test class name and use it to lookup
the test class in test suite to be able to add it to the list of failures
for re-running.

Change-Id: I656f21e38aa450fc567cdcbcf6e586967f947a64
Signed-off-by: Klement Sekera <ksekera@cisco.com>

# b9ef2739 24-Jun-2018 Klement Sekera <ksekera@cisco.com>

Revert "Revert "make test: fix broken interfaces""

This reverts commit c8efa29b6f9a91381897b54f1147daf922ed7164.

Change-Id: I1d5c5773d5f86a63073e255336bd9de628e26179
Signed-off-by: Klement Sekera <ksekera@cisco.com>

# 31da2e30 24-Jun-2018 Klement Sekera <ksekera@cisco.com>

Revert "Revert "ipsec: VPP-1316 calculate IP/TCP/UDP inner checksums""

This reverts commit e0d2bd6bd7fc59c0c6ac48195d7f825dc99bfd91.

Change-Id: If491e16f9ea66b2493a6a7c7f3c684ed585f8f51
Signed-off-by: Klement Sekera <ksekera@cisco.com>

# e0d2bd6b 22-Jun-2018 Ole Troan <ot@cisco.com>

Revert "ipsec: VPP-1316 calculate IP/TCP/UDP inner checksums"

This reverts commit a98346f664aae148d26a8e158008b773d73db96f.

Change-Id: Iee5b3a5ddff0e8fd3a30fe5973cee24de434fe12
Signed-off-by: Ole Troan <ot@cisco.com>

# c8efa29b 22-Jun-2018 Ole Troan <ot@cisco.com>

Revert "make test: fix broken interfaces"

This reverts commit d5c60b96a3fd93916fc4af5c8d6d25625c28242e.

Change-Id: I3632b9c3f76c615aee897f28f76d094e7031e689
Signed-off-by: Ole Troan <ot@cisco.com>

# d5c60b96 22-Jun-2018 Klement Sekera <ksekera@cisco.com>

make test: fix broken interfaces

Change-Id: I2e092774f81503e04b53cc6c6b5d357fe3fc52ab
Signed-off-by: Klement Sekera <ksekera@cisco.com>

# a98346f6 16-May-2018 Klement Sekera <ksekera@cisco.com>

ipsec: VPP-1316 calculate IP/TCP/UDP inner checksums

Calculate IP/TCP/UDP checksums in software before adding authentication.

Change-Id: I3e121cb00aeba667764f39ade8d62170f18f8b6b
Signed-off-by: Klement Sekera <ksekera@cisco.com>

# d81ae418 16-May-2018 Klement Sekera <ksekera@cisco.com>

make test: unify packet checksum verifications

Change-Id: If9cc7c5e32ebecff398fd38b39e8f485754a4ad4
Signed-off-by: Klement Sekera <ksekera@cisco.com>

# 07a38572 21-Jan-2018 Damjan Marion <damarion@cisco.com>

Add logging support

Change-Id: Ieac9cf50156dbbb4962411e900d59256441915ef
Signed-off-by: Damjan Marion <damarion@cisco.com>
Signed-off-by: Klement Sekera <ksekera@cisco.com>

# e1ace19b 23-Mar-2018 Klement Sekera <ksekera@cisco.com>

make test: fix DEBUG=gdb/gdbserver options

Change-Id: I5a7fa3b1c247ad5611907db27835724dcd31f575
Signed-off-by: Klement Sekera <ksekera@cisco.com>

# 13a83ef4 20-Mar-2018 Klement Sekera <ksekera@cisco.com>

make test: code cleanup

Change-Id: Ic689de569e5b6e6209d16d6acdb13c489daca1f5
Signed-off-by: Klement Sekera <ksekera@cisco.com>

# 4c53313c 22-Feb-2018 Klement Sekera <ksekera@cisco.com>

reassembly: feature/concurrency

This change makes ip reassembly an interface feature, while adding
concurrency support. Due to this, punt is no longer needed to test

Change-Id: I467669514ec33283ce935be0f1dd08f07684f0c7
Signed-off-by: Klement Sekera <ksekera@cisco.com>

# 05742265 14-Mar-2018 Klement Sekera <ksekera@cisco.com>

make test: early core detection, code cleanup

Change-Id: I503d69f902f1c8d3107fb3f2b9f31b0083ebb747
Signed-off-by: Klement Sekera <ksekera@cisco.com>

# 42996c0f 26-Feb-2018 Dave Wallace <dwallacelf@gmail.com>

make test: refactor vcl test cases

- Reduce replicated code in test cases
- Configure separate namespace secrets for thru hoststack
test case to validate namespace secret functionality.
- Pass per-instance environment variables to Worker class
init function.

Change-Id: I3cd5d4538f105cbfb09671c4d761541b40714b8f
Signed-off-by: Dave Wallace <dwallacelf@gmail.com>

# de0203ed 22-Feb-2018 Klement Sekera <ksekera@cisco.com>

make test: tidy worker subprocess output

Change-Id: I362765a67762a59775863af12b712abb47445b3a
Signed-off-by: Klement Sekera <ksekera@cisco.com>

# cfcf2f47 16-Feb-2018 Dave Wallace <dwallacelf@gmail.com>

make test: Add VPP VCL cut-thru test.

Change-Id: Id3ec196bfeb90b141123adee97f15d9712351680
Signed-off-by: Dave Wallace <dwallacelf@gmail.com>

# ebbaf55e 17-Feb-2018 Klement Sekera <ksekera@cisco.com>

make test: add test framework debugging

New option TEST_DEBUG=1 turns on test framework debugging, which
currently consists of printing difference in allocated objects/memory
and also creates reference graphs for any unfreed VppPapiProvider/VPP
objects - these take a lot of memory and thus should be freed regularly.

Change-Id: I29db0c1341009d4b5c5df9222d14f3095883fd0f
Signed-off-by: Klement Sekera <ksekera@cisco.com>

# 5ba5837b 13-Feb-2018 Dave Wallace <dwallacelf@gmail.com>

make-test: use api custom-dump to reduce log size.

- Reduce log size by using "api trace custom-dump" instead
of "api trace dump".
- Fix custom-dump output of cli_inband_t api message to include
cli command being executed.

New output:
DBGvpp# api trace custom-dump /tmp/test
index: 2
handle: 0x301d8e10
SCRIPT: memclnt_create name vpp_api_test
SCRIPT: sw_interface_dump all
SCRIPT: control_ping
SCRIPT: exec show run

Old output:
DBGvpp# api trace dump /tmp/test
---------- trace 0 -----------
index: 33554432
handle: 0x108e1d3000000000
---------- trace 1 -----------
name: vpp_api_test
input_queue: 0x808e1d3000000000
context: 0
ctx_quota: 0
---------- trace 2 -----------
_vl_msg_id: 61
client_index: 33554432
context: 0
name_filter_valid: 0
---------- trace 3 -----------
_vl_msg_id: 712
client_index: 33554432
context: 0
---------- trace 4 -----------
_vl_msg_id: 715
client_index: 33554432
context: 0
length: 9

Change-Id: If740c861649a3a59b8cc7a777c23c3cf94b8ff87
Signed-off-by: Dave Wallace <dwallacelf@gmail.com>

# 96867baa 02-Feb-2018 Klement Sekera <ksekera@cisco.com>

make test: use random seed

This fixes a constant setting of random seed forgotten from testing.

Change-Id: Ie3c4db8bb2b4b73ba33de1ffc02cb563391fd31c
Signed-off-by: Klement Sekera <ksekera@cisco.com>

# 75e7d130 20-Sep-2017 Klement Sekera <ksekera@cisco.com>

IPv4/6 reassembly

Change-Id: Ic5dcadd13c88b8a5e7896dab82404509c081614a
Signed-off-by: Klement Sekera <ksekera@cisco.com>

# 52fae862 08-Jan-2018 Neale Ranns <nranns@cisco.com>

test: consolidate the multiple versions of send_and_*

Change-Id: I7fa7d0ebf73dab8264a2e5ddbd412600d78ead05
Signed-off-by: Neale Ranns <nranns@cisco.com>

# e1783999 07-Nov-2017 Klement Sekera <ksekera@cisco.com>

make test: fix logic for CACHE_OUTPUT option

Change-Id: I9938c0154c860913e27cecb18ec68f247943a2e6
Signed-off-by: Klement Sekera <ksekera@cisco.com>

# 6a6f4f7f 09-Nov-2017 Klement Sekera <ksekera@cisco.com>

make test: automatically seed random generator

Change-Id: I286b61d41cc2b557de3eb8801ff95c643f680acd
Signed-off-by: Klement Sekera <ksekera@cisco.com>

# a3d933c8 06-Nov-2017 Klement Sekera <ksekera@cisco.com>

make test: add CACHE_OUTPUT option

This option allows one to toggle whether captured VPP stdout/stderr
output is printed inline (each line logged immediately after being
collected). Default setting is 0/no (behaviour same as before).

This allows one to see the output in case of vpp crashing during test
run even if test framework gets stuck waiting for shared memory mutex
(which crashed vpp can no longer unlock).

Complete stdout/stderr output is always printed as a block to the
logger after test finishes.

Change-Id: If3645a9b85562dba26ed9c87daa27ea5ef34f2e5
Signed-off-by: Klement Sekera <ksekera@cisco.com>

# 812ed39f 16-Oct-2017 Neale Ranns <neale.ranns@cisco.com>

VPP Object Model (VOM)

The VOM is a C++ library for use by clients/agents of VPP for programming
state. It uses the binary APIs to do so. Various other common client side
functions are also provided. Please see om.hpp for a more detailed description.

Change-Id: Ib756bfe99817093815a9e26ccf464aa5583fc523
Signed-off-by: Neale Ranns <neale.ranns@cisco.com>
Co-authored-by: Mohsin Kazmi <sykazmi@cisco.com>

# df2b980d 05-Oct-2017 Klement Sekera <ksekera@cisco.com>

make test: add RETRIES option

Change-Id: Ibe31e932bc997f0101a8947e01df90a90d1f100f
Signed-off-by: Klement Sekera <ksekera@cisco.com>

# e2efd12b 01-Oct-2017 Dave Wallace <dwallacelf@gmail.com>

make test: Copy api_post_mortem.$$ file tmp test dir for archiving.

Change-Id: I4baf89ef383dbc2f309081a6b56b13ebcb8fc2df
Signed-off-by: Dave Wallace <dwallacelf@gmail.com>

# 981fadf9 30-Sep-2017 Dave Wallace <dwallacelf@gmail.com>

make test: Create link to failed test dir on timeout.

- Also change default coredump configuration from
"coredump-size unlimited" to "full-coredump"

Change-Id: Iefedc2636f2d9696b7575b34e91dd7be49f601fa
Signed-off-by: Dave Wallace <dwallacelf@gmail.com>

# d3e671e0 28-Sep-2017 Klement Sekera <ksekera@cisco.com>

make test: fix/disable VAPI tests on centos

Workaround old `check' library on centos.
Disable building/running of C++ VAPI test as centos's gcc can't compile
our C++ code (vapi.hpp) due to bug.

Change-Id: If9da9c7f1bc076f7cdfb9bd3016dfe60a08afa36
Signed-off-by: Klement Sekera <ksekera@cisco.com>

# f413bef1 15-Aug-2017 Klement Sekera <ksekera@cisco.com>

make test: collect symlinks to failed tests

Compress files in temporary directories of failed tests and symlink
the directories under /tmp/vpp-failed-unittests location - preparation
for jenkins archivation. Automatically cleanup the directory at start
of test run.

The compression is performed only when environment variable
COMPRESS_FAILED_TEST_LOGS is set to one of "yes", "y", "1".
This is set in verify target, but left unset by default, so when
invoking make test by hand, files won't be compressed.

Change-Id: I84c8f1c6aa79aa9c0b753357022b1f195f17a283
Signed-off-by: Klement Sekera <ksekera@cisco.com>

# 3f6ff19a 11-Aug-2017 Klement Sekera <ksekera@cisco.com>

make test: fix broken DEBUG=gdb* options

Change-Id: I5d80982eeab78a629760f567eda3b1539d96e3a8
Signed-off-by: Klement Sekera <ksekera@cisco.com>

# 909a6a1e 08-Aug-2017 Klement Sekera <ksekera@cisco.com>

make test: detect hung tests

Run tests in a forked process with a set of pipes to communicate
keep-alives and overall result. This allows us to detect when e.g.
vpp dies mid-API call causing the test to hang waiting for response
(which will never come since vpp died).

Support setting a (per test case) TIMEOUT make test option to set timeout,
with a default timeout of 120 seconds.

Example - fail the test suite if any test-case fails to finish within

make TIMEOUT=300 test

Change-Id: I0d04f26a7232968f4bf043adf5d5b508f5018717
Signed-off-by: Klement Sekera <ksekera@cisco.com>

# 60c1223a 18-Jul-2017 Klement Sekera <ksekera@cisco.com>

make test: improve console output messages

Remove the word derp and replace it with a proper, more gramatically
correct message.

Change-Id: I04fd44cc67dace1a31ca48fc8ce67b246162ba79
Signed-off-by: Klement Sekera <ksekera@cisco.com>

# 3658adca 07-Jun-2017 Klement Sekera <ksekera@cisco.com>

make test: improve debugability

Introduce faulthandler to print stack trace to stderr on python crash.
Don't disable automatic garbage collection to decrease the chances of
running out of memory.

Change-Id: I6927a5f6ea9569735d084d4ed3d258950a400d74
Signed-off-by: Klement Sekera <ksekera@cisco.com>

# 3cfa558a 19-Apr-2017 Klement Sekera <ksekera@cisco.com>

make test: improve bfd reliability

Change-Id: Iaf446a2d7d8e595c3379fb9ed61a954351c17b90
Signed-off-by: Klement Sekera <ksekera@cisco.com>

# 027dbd52 11-Apr-2017 Klement Sekera <ksekera@cisco.com>

make test: don't rely on cPython GC to close fds

This code improvement allows running in pypy (and other interpreters)
without exhausting file descriptors.

Change-Id: Icb692a0fe1343c12cbbb15af6c58753420e74330
Signed-off-by: Klement Sekera <ksekera@cisco.com>

# 3747c75a 10-Apr-2017 Klement Sekera <ksekera@cisco.com>

make test: automatic "vpp finishes startup" check

Add code which checks if vpp doesn't crash/exit immediately after
startup to aid debugging stuff like mistyped graph node name or so.
Refuse to run tests if the vpp is unable to start, complain loudly
and print vpp's stderr at critical log level if this happens to make
spotting these problems in jenkins easy.

Change-Id: I40d3fbd05c822c0534713bae6bef05ecfb0e0c1d
Signed-off-by: Klement Sekera <ksekera@cisco.com>

# 88fc83eb 05-Apr-2017 Neale Ranns <nranns@cisco.com>

BFD-FIB interactions

- single-hop BFD: attach a delegate to the appropriate adjacency
- multi-hop BFD [not supported yet]: attach a delegate to the FIB entry.

adjacency/fib_entry state tracks the BFD session state. when the state is down the object does not contribute forwarding hence and hence dependent objects will not use it.
For example, if a route is ECMP via two adjacencies and one of them is BFD down, then only the other is used to forward (i.e. we don't drop half the traffic).

Change-Id: I0ef53e20e73b067001a132cd0a3045408811a822
Signed-off-by: Neale Ranns <nranns@cisco.com>

# 9a69a609 26-Mar-2017 Neale Ranns <nranns@cisco.com>

Sub-net broadcast addresses for IPv4

Change-Id: Ib2189d01e8bc61de57404159690fb70f89c47277
Signed-off-by: Neale Ranns <nranns@cisco.com>

# 6abbc288 24-Mar-2017 Klement Sekera <ksekera@cisco.com>

make test: fix broken plugin paths

Change-Id: I25a6882ec503fc5bb3694411fbdc2eb1f1e1fafc
Signed-off-by: Klement Sekera <ksekera@cisco.com>

# 47e275bb 21-Mar-2017 Klement Sekera <ksekera@cisco.com>

make test: support out-of-tree plugins


env EXTERN_PLUGINS=/path/to/plugins make test

Change-Id: I8eece726dfafeff1cffd921c1e18cd3eb7eb64ed
Signed-off-by: Klement Sekera <ksekera@cisco.com>

# 59dda065 02-Mar-2017 Pavel Kotucek <pkotucek@cisco.com>

ACL plugin rejects ICMP messages (VPP-624)

Change-Id: I95113a277b94cce5ff332fcf9f57ec6f385acec0
Signed-off-by: Pavel Kotucek <pkotucek@cisco.com>

# 374e2c5f 09-Mar-2017 Damjan Marion <damarion@cisco.com>

Retire vpp_lite

vpp_lite platform is not needed anymore as same efect can be
achieved with following startup.conf config:

plugins {
plugin dpdk_plugin.so { disable }

Change-Id: I690ea8ceb1c6e1fe32e01e7da54e9958019a93bf
Signed-off-by: Damjan Marion <damarion@cisco.com>

# 87134937 07-Mar-2017 Klement Sekera <ksekera@cisco.com>

make test: split into basic and extended tests

Implement plumbing to allow decorating tests as extended, e.g.:

@unittest.skipUnless(running_extended_tests(), "part of extended tests")

both methods and classes can be decorated this way.

Change make test and make test-debug to run only non-extended tests.
Introduce make test-all and make test-all-debug to run the full suite.
Run full suite as part of make verify.
Decorate most BFD tests as extended.

Change-Id: I3bc64f59e9fe238f7f767d7e043dc165d03e9dfa
Signed-off-by: Klement Sekera <ksekera@cisco.com>

# 1b686409 02-Mar-2017 Klement Sekera <ksekera@cisco.com>

make test: reset object registry if vpp dies

Change-Id: If0e30837e07a21f3912676f5147cb242d3d2b235
Signed-off-by: Klement Sekera <ksekera@cisco.com>

# 80a7f0a8 02-Mar-2017 Klement Sekera <ksekera@cisco.com>

make test: tell vpp to set coredump size

Unless overridden by COREDUMP_SIZE env variable, tell VPP to set
coredump size to unlimited, otherwise use $COREDUMP_SIZE as
the argument.

Change-Id: Ia2a6508207c66a171b33d272c820b1deb4a83e82
Signed-off-by: Klement Sekera <ksekera@cisco.com>

# 239790fd 16-Feb-2017 Klement Sekera <ksekera@cisco.com>

BFD: echo function

Change-Id: Ib1e301d62b687d4e42434239e7cd412065c28da0
Signed-off-by: Klement Sekera <ksekera@cisco.com>

# 90c55724 16-Feb-2017 Dave Wallace <dwallacelf@gmail.com>

make test: save + dump VPP api trace log; VPP-640

Change-Id: I20aacc927f2b04f42b0a7220c4283560b4d2a359
Signed-off-by: Dave Wallace <dwallacelf@gmail.com>

# aeeac3bf 14-Feb-2017 Klement Sekera <ksekera@cisco.com>

BFD: loop back echo packets

Change-Id: I772b63ac25ebfccaff9ab9d8d0b1445e85f21df7
Signed-off-by: Klement Sekera <ksekera@cisco.com>

# acb9b8e8 14-Feb-2017 Klement Sekera <ksekera@cisco.com>

make test: improve stability

Disable automatic garbage collection and run it manually before
running each test case to minimize stalls. Improve vpp subprocess
cleanup. Reduce helper thread count to one and properly clean that
thread once it's not needed.

Change-Id: I3ea78ed9628552b5ef3ff29cc7bcf2d3fc42f2c3
Signed-off-by: Klement Sekera <ksekera@cisco.com>

# b91017a3 09-Feb-2017 Klement Sekera <ksekera@cisco.com>

make test: work around scapy truncated packets

Under stress, it's possible to hit a race condition, when the packet
header is fully written to pcap, but not all packet data - yet.
Scapy is stupid enough to:
1. not detect and report this error, truncating the packet instead
2. continue munching more data from wrong offset
The work around is to scan the file ahead, parse the packet header,
figure out how much data we need, wait for the file to be big
enough, then restore the file position back to where it was
and finally let scapy parse the packet.

Change-Id: I9fc71d3ebdc62ecab6c90b90f177d0eaeb09b8bb
Signed-off-by: Klement Sekera <ksekera@cisco.com>

# a57a9709 02-Feb-2017 Klement Sekera <ksekera@cisco.com>

BFD: modify session parameters

Change-Id: I666e5c0cc71a3693640960c93cdd1907f84fbe23
Signed-off-by: Klement Sekera <ksekera@cisco.com>

# 104543fa 03-Feb-2017 Klement Sekera <ksekera@cisco.com>

make test: improve test filtering

Implement fine-grained test filtering by supporting more complicated
filters beside the original file name suffix filter.

Change-Id: If5a166d08cffe8c58cc6cf174e6df861c34dbaa6
Signed-off-by: Klement Sekera <ksekera@cisco.com>

# b17dd960 09-Jan-2017 Klement Sekera <ksekera@cisco.com>

BFD: SHA1 authentication

Add authentication support to BFD feature. Out of three existing
authentication types, implement SHA1 (sole RFC requirement). Simple
password is insecure and MD5 is discouraged by the RFC, so ignore
Add/change APIs to allow configuring BFD authentication keys
and their usage with BFD sessions.

Change-Id: Ifb0fb5b19c2e72196d84c1cde919bd4c074ea415
Signed-off-by: Klement Sekera <ksekera@cisco.com>

# 7a161da3 17-Jan-2017 Klement Sekera <ksekera@cisco.com>

make test: fix text output when non-interactive

Output everything to stdout instead of mixing stdout and stderr.

Change-Id: I3146760e068fda7f1ffaaf36683d5153dcda4561
Signed-off-by: Klement Sekera <ksekera@cisco.com>

# 52e84f3c 13-Jan-2017 Klement Sekera <ksekera@cisco.com>

make test: improve test results printing

Accomodate longer test descriptions by increasing the headings

Change-Id: I51c90b8d91feaa83b78972d0802d3054f8510f36
Signed-off-by: Klement Sekera <ksekera@cisco.com>

# da505f60 03-Jan-2017 Klement Sekera <ksekera@cisco.com>

make test: improve documentation and PEP8 compliance

Change-Id: Ib4f0353aab6112fcc3c3d8f0bcbed5bc4b567b9b
Signed-off-by: Klement Sekera <ksekera@cisco.com>

# 10db26f7 11-Jan-2017 Klement Sekera <ksekera@cisco.com>

BFD: fix bfd_udp_add API

Fix reporting of bs_index in the return message. Enhance test suite
to cover this case.

Change-Id: I37d35b850818bc1a05abe67ca919c22aeac978b6
Signed-off-by: Klement Sekera <ksekera@cisco.com>

# 8d8a1da5 22-Dec-2016 Matej Klotton <mklotton@cisco.com>

make test: Loopback interface CRUD test

Change-Id: I0581da7a682bfe4dd6520ecf1b2ea6bd8c20b1b3
Signed-off-by: Matej Klotton <mklotton@cisco.com>

# dab231a1 21-Dec-2016 Klement Sekera <ksekera@cisco.com>

make test: improve handling of packet captures

Perform accounting of expected packets based on created packet infos.
Use this accounting info to automatically expect (and verify) the
correct number of packets to be captured. Automatically retry the read
of the capture file if scapy raises an exception while doing so to
handle rare cases when capture file is read while only partially
written during busy wait. Don't fail assert_nothing_captured if only
junk packets arrived.

Change-Id: I16ec2e9410ef510d313ec16b7e13c57d0b2a63f5
Signed-off-by: Klement Sekera <ksekera@cisco.com>

# 6c7440ca 23-Dec-2016 Klement Sekera <ksekera@cisco.com>

make test: add log.txt

Always create a log.txt under test's temporary directory containing
V=2 output for debugging purposes.

Change-Id: If149c8126f42f80724ee17b63c411ddec6645f85
Signed-off-by: Klement Sekera <ksekera@cisco.com>

# 9225dee9 12-Dec-2016 Klement Sekera <ksekera@cisco.com>

make test: improve robustness and performance

Introduce an API which asserts empty capture for interface.
Throw exception in old API if the capture does not exist, thus
making it clear if the test expects packets to arrive or not.
Improve performance by not doing sleeps after starting the packet
generator, rather lazily deleting captures when needed.
Fix wrong usage of packet.show() in various tests.

Change-Id: I456cb23316eef99b3f35f80344fe595c4db9a21c
Signed-off-by: Klement Sekera <ksekera@cisco.com>

# 16a14cdb 07-Dec-2016 Matej Klotton <mklotton@cisco.com>

make test: FIB add/update/delete - ip4 routes


Change-Id: Idb4c5bd7a234bc975f3380ece58c0e8d4bfdafd9
Signed-off-by: Matej Klotton <mklotton@cisco.com>

# e4504c63 08-Dec-2016 Klement Sekera <ksekera@cisco.com>

BFD: fix timing in the main loop

Properly wakeup the main process loop only when necessary to avoid
missed events. Improve BFD unit test reliability and speed. Add
timestamps to logs and replace Queue with more efficient deque.

Change-Id: I01255a02057d3146917b43b669ccdc74f31ef0c8
Signed-off-by: Klement Sekera <ksekera@cisco.com>

# 0c1519b5 08-Dec-2016 Klement Sekera <ksekera@cisco.com>

BFD: improve ip header handling, fix assert

Instead of storing a pointer, store an offset within vlib_buffer marking
the start of the ip header (similar to storing start of ethernet
header). Handle negative timeout in the bfd process main loop.

Change-Id: I05a1ff3ac41da5bdc3b2ac6f9e03b3241994688b
Signed-off-by: Klement Sekera <ksekera@cisco.com>

# 0e3c0de1 29-Sep-2016 Klement Sekera <ksekera@cisco.com>

BFD: basic asynchronous session up/down

This is a work-in-progress basic BFD session handling. Only
asynchronous mode is supported at the moment. Setting the session flags
doesn't work.

Change-Id: Idba27f721b5c35be5a66a6d202a63d23ff7ecf6f
Signed-off-by: Klement Sekera <ksekera@cisco.com>

# 7bb873a4 18-Nov-2016 Klement Sekera <ksekera@cisco.com>

make test: fix missing log/packet messages

Change-Id: Idb3119792943664748c4abc3829ad723f4156dfe
Signed-off-by: Klement Sekera <ksekera@cisco.com>

# 0529a743 02-Dec-2016 Klement Sekera <ksekera@cisco.com>

make test: handle exceptions from VppPapiProvider.__init__

Change-Id: I3c0372d5d09d554ad4ebe0adfa97c111571f628a
Signed-off-by: Klement Sekera <ksekera@cisco.com>

# 085f5c00 24-Nov-2016 Klement Sekera <ksekera@cisco.com>

make test: detect early vpp crash

If VPP process dies right at start, do a quick detection instead of being
stuck in the connect timeout (60s).

Change-Id: I41675181635fb81a6a7d93fbf652480a16bf78a0
Signed-off-by: Klement Sekera <ksekera@cisco.com>

# 177bbdcd 15-Nov-2016 Neale Ranns <nranns@cisco.com>

GRE tests and fixes

Change-Id: I234240e9bdd4b69ad64a17b1449ae1e81c0edaca
Signed-off-by: Neale Ranns <nranns@cisco.com>

# 86d87c40 11-Nov-2016 Matej Klotton <mklotton@cisco.com>

Update test documentation.

- update IRB, IPv4, ipv6 doc
- revert 778c2765c8ea5c6628f6d668847f0b9ae06dbf3d

Change-Id: I9af5ed9329ce5fe01392cf28d5bf321cfc647e48
Signed-off-by: Matej Klotton <mklotton@cisco.com>

# 49c0fcab 26-Oct-2016 Jan <jgelety@cisco.com>

Add single-loop test variant to L2BD and L2XC tests

- create single-loop version of tests
- update doc strings

- add possibility to log CLI string for logging level "info"

Change-Id: Ibc9e5650b8a33d2ed168a8440c2ae03227be60bb
Signed-off-by: Jan <jgelety@cisco.com>

# 0178d523 04-Nov-2016 Matej Klotton <mklotton@cisco.com>

Add IRB test

- JIRA: CSIT-255
- create loopback interfaces
- move pg-interface specific arp and neighbor discovery from vpp_interface to vpp_pg_interface
- base configuration of IRB tests
- IP test scenario

Change-Id: I9945a188163652a4e22325877aef008c4d029557
Signed-off-by: Matej Klotton <mklotton@cisco.com>

# 931be3ac 03-Nov-2016 Klement Sekera <ksekera@cisco.com>

Test framework: improve gdbserver handling

Produce a user-friendly message if gdbserver is not available, instead
of cryptic exception.

Change-Id: Ia0d99e0488d2ee6e8af764b466dae2639f17ea55
Signed-off-by: Klement Sekera <ksekera@cisco.com>

# 01bbbe91 02-Nov-2016 Klement Sekera <ksekera@cisco.com>

Improve debug-cli in test framework

Opening debug-cli only makes sense if there is time for user to use it,
so either the run must be debugged or stepped. Only open the debug-cli
in these cases.

Change-Id: Ied276071797a549880d730cda43c59230a412efe
Signed-off-by: Klement Sekera <ksekera@cisco.com>

# 8fe8cc21 01-Nov-2016 Neale Ranns <nranns@cisco.com>

MPLS Exp-null Tests

Add some 'make test' unit tests for MPLS explicit NULL label handling.
Fix the stacking of the MPLS load-balance result form the lookup onto the IPx lookup object.

Change-Id: I890d1221b8e3dea99bcc714ed9d0154a5f602c52
Signed-off-by: Neale Ranns <nranns@cisco.com>

# 277b89c9 28-Oct-2016 Klement Sekera <ksekera@cisco.com>

add vpp debugging support to test framework
improve test documentation

Change-Id: Ia9678aa2532ecb4cb33736aedb4a31aa3f2a3f93
Signed-off-by: Klement Sekera <ksekera@cisco.com>

# f62ae128 11-Oct-2016 Klement Sekera <ksekera@cisco.com>

refactor test framework

Change-Id: I31da3b1857b6399f9899276a2d99cdd19436296c
Signed-off-by: Klement Sekera <ksekera@cisco.com>
Signed-off-by: Matej Klotton <mklotton@cisco.com>
Signed-off-by: Jan Gelety <jgelety@cisco.com>
Signed-off-by: Juraj Sloboda <jsloboda@cisco.com>

# 749294df 25-Oct-2016 Peter Ginchev <pginchev@cisco.com>

Disable colored output for tests, when not tty

Change-Id: I73f01bd3a8e7caa00c75b845b9e61d3cb0f34877
Signed-off-by: Peter Ginchev <pginchev@cisco.com>

# cd8e318a 07-Oct-2016 Pierre Pfister <ppfister@cisco.com>

Test Infra: Add plugin support

This patch adds plugin path to vpp commandline arguments
when 'make test' is run.
Hence, test cases can test plugins.

Change-Id: Ib90efa1f62e03b45b84533c49c7a5d040aa8cddf
Signed-off-by: Pierre Pfister <ppfister@cisco.com>

# b80f9d1b 05-Oct-2016 Klement Sekera <ksekera@cisco.com>

don't require root privileges when running the tests

Change-Id: Ib67bf1a898e3c1e4038698f1cb068ae9099d7921
Signed-off-by: Klement Sekera <ksekera@cisco.com>

# f56b77a0 03-Oct-2016 Damjan Marion <damarion@cisco.com>

test: new test infrastructure

Change-Id: I73ca19c431743f6b39669c583d9222a6559346ef
Signed-off-by: Jan Gelety <jgelety@cisco.com>
Signed-off-by: Juraj Sloboda <jsloboda@cisco.com>
Signed-off-by: Stefan Kobza <skobza@cisco.com>
Signed-off-by: Matej Klotton <mklotton@cisco.com>
Signed-off-by: Maciek Konstantynowicz <mkonstan@cisco.com>
Signed-off-by: Damjan Marion <damarion@cisco.com>