Tempest Testing¶

Tempest basics in Networking projects¶

Tempest is the integration test suite of Openstack, for details see Tempest Testing Project.

Tempest makes it possible to add project-specific plugins, and for networking this is neutron-tempest-plugin.

neutron-tempest-plugin covers API and scenario tests not just for core Neutron functionality, but for stadium projects as well. For reference please read Testing Neutron's related sections

API Tests¶

API tests (neutron-tempest-plugin/neutron_tempest_plugin/api/) are intended to ensure the function and stability of the Neutron API. As much as possible, changes to this path should not be made at the same time as changes to the code to limit the potential for introducing backwards-incompatible changes, although the same patch that introduces a new API should include an API test.

Since API tests target a deployed Neutron daemon that is not test-managed, they should not depend on controlling the runtime configuration of the target daemon. API tests should be black-box - no assumptions should be made about implementation. Only the contract defined by Neutron’s REST API should be validated, and all interaction with the daemon should be via a REST client.

The neutron-tempest-plugin/neutron_tempest_plugin directory was copied from the Tempest project around the Kilo timeframe. At the time, there was an overlap of tests between the Tempest and Neutron repositories. This overlap was then eliminated by carving out a subset of resources that belong to Tempest, with the rest in Neutron.

API tests that belong to Tempest deal with a subset of Neutron’s resources:

Port
Network
Subnet
Security Group
Router
Floating IP

These resources were chosen for their ubiquity. They are found in most Neutron deployments regardless of plugin, and are directly involved in the networking and security of an instance. Together, they form the bare minimum needed by Neutron.

This is excluding extensions to these resources (For example: Extra DHCP options to subnets, or snat_gateway mode to routers) that are not mandatory in the majority of cases.

Tests for other resources should be contributed to the neutron-tempest-plugin repository. Scenario tests should be similarly split up between Tempest and Neutron according to the API they’re targeting.

To create an API test, the testing class must at least inherit from neutron_tempest_plugin.api.base.BaseNetworkTest base class. As some of tests may require certain extensions to be enabled, the base class provides required_extensions class attribute which can be used by subclasses to define a list of required extensions for a particular test class.

Scenario Tests¶

Scenario tests (neutron-tempest-plugin/neutron_tempest_plugin/scenario), like API tests, use the Tempest test infrastructure and have the same requirements. Guidelines for writing a good scenario test may be found at the Tempest developer guide: https://docs.openstack.org/tempest/latest/field_guide/scenario.html

Scenario tests, like API tests, are split between the Tempest and Neutron repositories according to the Neutron API the test is targeting.

Some scenario tests require advanced Glance images (for example, Ubuntu or CentOS) in order to pass. Those tests are skipped by default. To enable them, include the following in tempest.conf:

[compute]
image_ref = <uuid of advanced image>
[neutron_plugin_options]
default_image_is_advanced = True

To use the advanced image only for the tests that really need it and cirros for the rest to keep test execution as fast as possible:

[compute]
image_ref = <uuid of cirros image>
[neutron_plugin_options]
advanced_image_ref = <uuid of advanced image>
advanced_image_flavor_ref = <suitable flavor for the advance image>
advanced_image_ssh_user = <username for the advanced image>

Specific test requirements for advanced images are:

test_trunk requires 802.11q kernel module loaded.
test_metadata requires capabilty to run curl for IPv6 addresses.
test_multicast needs to execute python script on the VM to open multicast sockets.
test_mtu requires ping to be able to send packets with specific mtu.

Zuul basics & job structure¶

Zuul is the gating system behind Openstack, for details see: Zuul - A Project Gating System.

Zuul job definitions are in yaml, ansible in the depths. The job definitions can be inherited. Networking projects job definitions parents are coming from devstack zuul job config and from tempest and defined in neutron-tempest-plugin zuul.d folder and in neutron zuul.d folder .

Where to look¶

Debugging zuul results¶

Tempest executed with different configurations, for details check this page Tempest jobs running in Neutron CI

When zuul reports back job results to a review it gives links to the results as well.

The logs can be checked online if you select Logs tab on the logs page.

job-output.txt is the full log which contains not just test execution logs, but devstack console output.
test_results.html is the clickable html test report.
controller and compute (in case of multinode job) are a dictionary tree containing the relevant files (configuration files, logs etc) created in the job. For example under controller/logs/etc/neutron/ you can check how Neutron services were configured, or in the file controller/logs/tempest_conf.txt you can check tempest configuration file.
services’ log files are the in files controller/logs/screen-`*`.txt, so for example neutron l2 agent logs are in the file controller/logs/screen-q-agt.txt.

Downloading logs¶

There is a possibility to download all logs related to a job.

If you choose this on the zuul logs page select Artifacts tab on the logs page and click on Download all logs. This will download a script download-logs.sh, which when executed downloads all the logs for the job under /tmp/:

$ chmod +x download-logs.sh
$ ./download-logs.sh
2020-12-07T18:12:09+01:00 | Querying https://zuul.opendev.org/api/tenant/openstack/build/8caed05f5ba441b4be2b061d1d421e4e for manifest
2020-12-07T18:12:11+01:00 | Saving logs to /tmp/zuul-logs.c8ZhLM
2020-12-07T18:12:11+01:00 | Getting logs from https://3612101d6c142bf9c77a-c96c299047b55dcdeaefef8e344ceab6.ssl.cf1.rackcdn.com/694539/11/check/tempest-slow-py3/8caed05/
2020-12-07T18:12:11+01:00 |   compute1/logs/apache/access_log.txt                                              [ 0001/0337 ]
...

$ ls /tmp/zuul-logs.c8ZhLM/
compute1
controller

Executing tempest locally¶

For executing tempest locally you need a working devstack, to make it worse if you have to debug a test executed in a multinode job you need a multinode setup as well.

For devstack documentation please refer to this page: DevStack

To have tempest installed and have a proper configuration file for it in your local.conf file enable tempest as service:

ENABLED_SERVICES+=tempest

enable_service tempest

To use specific config options for tempest you can add those as well to local.conf:

[[test-config|/opt/stack/tempest/etc/tempest.conf]]
[network-feature-enabled]
qos_placement_physnet=physnet1

To make devstack setup neutron and neutron-tempest-plugin as well enable their devstack plugin:

enable_plugin neutron https://opendev.org/openstack/neutron
enable_plugin neutron-tempest-plugin https://opendev.org/openstack/neutron-tempest-plugin

If you need a special image for the tests you can set that too in local.conf:

IMAGE_URLS="http://download.cirros-cloud.net/0.3.4/cirros-0.3.4-i386-disk.img,https://cloud-images.ubuntu.com/releases/bionic/release/ubuntu-18.04-server-cloudimg-amd64.img"
ADVANCED_IMAGE_NAME=ubuntu-18.04-server-cloudimg-amd64
ADVANCED_INSTANCE_TYPE=ds512M
ADVANCED_INSTANCE_USER=ubuntu

If devstack succeeds you can find tempest and neutron-tempest-plugin under /opt/stack/ directory (with all other project folders which are set to be installed from git).

Tempest’s configuration file is under /opt/stack/tempest/etc/ folder, you can check there if everything is as expected.

You can check if neutron-tempest-plugin is known as a tempest plugin by tempest:

$ tempest list-plugins
+---------------------------------+------------------------------------------------------------+
|               Name              |                         EntryPoint                         |
+---------------------------------+------------------------------------------------------------+
|          neutron_tests          |     neutron_tempest_plugin.plugin:NeutronTempestPlugin     |
+---------------------------------+------------------------------------------------------------+

To execute a given test or group of tests you can use a regex, or you can use the idempotent id of a test or the tag associated with the test:

tempest run --config etc/tempest.conf --regex tempest.scenario
tempest run --config etc/tempest.conf --regex neutron_tempest_plugin.scenario
tempest run --config etc/tempest.conf smoke
tempest run --config etc/tempest.conf ab40fc48-ca8d-41a0-b2a3-f6679c847bfe

Tempest Testing