Overview of Testing¶
The Senlin project currently has five different types of testing facilities in place for developers to perform different kinds of tests:
Unit Tests: These are source code level testing that verifies the classes and methods behave as implemented. Once implemented, these tests are also used to guarantee that code behavior won’t change accidentally by other patches.
API Tests: These tests treat the senlin-api and the senlin-engine as black boxes. The test cases focus more on the API surface rather than how each API is implemented. Once implemented, these tests help ensure that the user-visible service interface don’t change without a good reason.
Functional Tests: These tests also treat the senlin-api and the senlin-engine as block boxes. They focus more on the user perceivable service behavior. Most tests are anticipated to test a particular “story” and verify that the senlin-engine always behave consistently.
Integration Tests: These are the tests that integrate senlin with other OpenStack services and verify the senlin service can perform its operations correctly when interacting with other services.
Stress Tests: These are tests for measuring the performance of the senlin-api and senlin-engine under different workloads.
The senlin server is shipped with two collections of “cloud backends”: one for
interacting with a real OpenStack deployment, the other for running complex
tests including api tests, functional tests, stress tests. The first cloud
backend is referred to as ‘openstack’ and the second is referred to as
‘openstack_test’. While the openstack cloud backend contains full featured
drivers for senlin to talk to the OpenStack services supported, the
openstack_test backend contains some “dummy” drivers that return fake
responses for service requests. The openstack_test driver is located at
senlin/tests/drivers subdirectory. It is provided to facilitate tests
on the senlin service itself without involving any other OpenStack services.
Several types of tests can benefit from these “dummy” drivers because 1) they
can save developers a lot time on debugging complex issues when interacting
with other OpenStack services, and 2) they make running those types of tests
much easier and quicker.
Note that “Integration Tests” are designed for senlin to interact with real services so we should use the openstack backend rather than the openstack_test backend.
To configure the backend to use before running tests, you can check the
[DEFAULT] section in the configuration file
[DEFAULT] cloud_backend = openstack_test # use this for api, functional tests; # or 'openstack' for production environment # and integration tests.
All unit tests are to be placed in the
Test cases are organized by the targeted subsystems/modules. Each subsystem
directory must contain a separate blank __init__.py for tests discovery to
An example directory structure:
senlin `- tests `- unit |-- db | |-- __init__.py | |-- test_cluster_api.py | `-- test_node_api.py |-- engine | |-- __init__.py | |-- test_clusters.py | `-- test_nodes.py |-- __init__.py `-- test_utils.py
Writing a Unit Test¶
The os-testr software (see: https://pypi.org/project/os-testr/) is used to find and run tests, parallelize their runs, and record timing/results.
If new dependencies are introduced upon the development of a test, the test-requirements.txt file needs to be updated so that the virtual environment will be able to successfully execute all tests.
The test-requirements.txt file needs to be synchronized with the openstack/global-requirements project. Developers should try avoid introducing additional package dependencies unless forced to.
Running Unit Tests¶
Senlin uses tox for running unit tests, as practiced by many other OpenStack projects:
This by default will run unit tests suite with Python 2.7 and PEP8/HACKING style checks. To run only one type of tests you can explicitly provide tox with the test environment to use:
$ tox -e py27 # test suite on python 2.7 $ tox -e pep8 # run full source code checker
To run only a subset of tests, you can provide tox with a regex argument:
$ tox -e py27 -- -r ClusterTest
To use debugger like pdb during test run, you have to run tests directly with other, non-concurrent test runner instead of testr. That also presumes that you have a virtual env with all senlin dependencies installed and configured.
A more convenient way to run specific test is to name the unit test directly, as shown below:
$ python -m testtools.run senlin.tests.unit.db.test_cluster_api
This command, however, is not using dependent packages in a particular virtual environment as the tox command does. It is using the system-wide Python package repository when running the tests.
Senlin API test cases are written based on the tempest framework (see: tempest_overview). Test cases are developed using the Tempest Plugin Interface (see: tempest_plugin ).
Writing an API Test Case¶
API tests are hosted in the senlin-tempest-plugin project. When new APIs are added
or existing APIs are changed, an API test case should be added to the
senlin_tempest_plugin/tests/api sub-directory, based on the resources impacted
by the change.
Each test case should derive from the class
test cases should be separated from negative ones. We don’t encourage combining
more than one test case into a single method, unless there is an obvious reason.
To improve the readability of the test cases, Senlin has provided a utility
module which can be leveraged -
Running API Tests¶
Senlin API tests use fake OpenStack drivers to improve the throughput of test execution. This is because in API tests, we don’t care about the details in how senlin-engine is interacting with other services. We care more about the APIs succeeds in an expected way or fails in a predictable manner.
Although the senlin engine is talking to fake drivers, the test cases still
need to communicate to the senlin API service as it would in a real
deployment. That means you will have to export your OpenStack credentials
before running the tests. For example, you will source the
when using a devstack environment:
$ . $HOME/devstack/openrc
This will ensure you have environment variables such as
OS_USERNAME properly set and exported. The next step is to enter the
tempest directory and run the tests there:
$ cd /opt/stack/tempest $ nosetests -v -- senlin
To run a single test case, you can specify the test case name. For example:
$ cd /opt/stack/tempest $ nosetests -v -- \ senlin_tempest_plugin.tests.api.clusters.test_cluster_create
If you prefer running API tests in a virtual environment, you can simply use the following command:
$ cd /opt/stack/senlin $ tox -e api
Similar to the API tests, senlin functional tests are also developed based on the tempest framework. Test cases are written using the Tempest Plugin Interface (see: tempest_plugin).
Writing Functional Tests¶
Functional tests are hosted in the senlin-tempest-plugin project. There are current
a limited collection of functional test cases which can be
senlin_tempest_plugin/tests/functional/ subdirectory. In future,
we may add more test cases when needed. The above subdirectory will remain the
home of newly added functional tests.
When writing functional tests, it is highly desirable that each test case is designed for a specific use case or story line.
Running Functional Tests¶
Similar to API tests, you will need to export your OpenStack credentials before running any functional tests.
The most straight forward way to run functional tests is to use the virtual
environment defined in the
tox.ini file, that is:
$ cd /opt/stack/senlin $ tox -e functional
If you prefer running a particular functional test case, you can do the following as well:
$ cd /opt/stack/senlin $ python -m testtools.run senlin_tempest_plugin.tests.functional.test_cluster_basic
Integration tests are basically another flavor of functional tests. The only difference from functional tests is that integration tests use real device drivers so the senlin-engine is talking to real services.
Writing Integration Tests¶
Integration tests are hosted in the senlin-tempest-plugin project. Integration tests are designed to be run at Gerrit gate to ensure that changes to senlin code won’t break its interactions with other (backend) services. Since OpenStack gate infrastructure is a shared resource pool for all OpenStack projects, we are supposed to be very careful when adding new test cases. The test cases added are supposed to focus more on the interaction between senlin and other services than other things.
All integration test cases are to be placed under the subdirectory
senlin_tempest_plugin/tests/integration. Test cases are expected to be
organized into a small number of story lines that can exercise as many
interactions between senlin and backend services as possible.
Each “story line” should be organized into a separate class module that
inherits from the
BaseSenlinIntegrationTest class which can be found at
senlin_tempest_plugin/tests/integration/base.py file. Each test case should
be annotated with a
decorators.attr annotator and an idempotent ID as shown
from tempest.lib import decorators from senlin.tests.tempest.integration import base class MyIntegrationTest(base.BaseSenlinIntegrationTest): @decorators.attr(type=['integration']) @decorators.idempotent_id('<A UUID for the test case>') def test_a_sad_story(self): # Test logic goes here # ...
Running Integration Tests¶
The integration tests are designed to be executed at Gerrit gate. However, you can still run them locally in your development environment, i.e. a devstack installation.
To run integration tests, you will need to configure tempest accounts by
/etc/tempest/accounts.yaml file. For each entry of the
tempest account, you will need to provide values for
password at least. For example:
- username: 'demo' tenant_name: 'demo' password: 'secretee'
After this is configured, you can run a specific test case using the following command:
$ cd /opt/stack/senlin $ python -m testtools.run \ senlin_tempest_plugin.tests.integration.test_nova_server_cluster
Writing Stress Test Cases¶
Running Stress Tests¶