Testing¶

Adding tests to a new role¶

Each of the role tests is in its tests/ folder.

This folder contains at least the following files:

test.yml (“super” playbook acting as test router to sub-playbooks)
<role name>-overrides.yml. This var file is automatically loaded by our shell script in our tests repository.
inventory. A static inventory for role testing. It’s possible some roles have multiple inventories. See for example the neutron role with its lxb_inventory.
group_vars and host_vars. These folders will hold override the necessary files for testing. For example, this is where you override the IP addresses, IP ranges, and ansible connection details.
ansible-role-requirements.yml. This should be fairly straightforward: this file contains all the roles to clone before running your role. The roles’ relative playbooks will have to be listed in the test.yml file. However, keep in mind to NOT re-invent the wheel. For example, if your role needs keystone, you don’t need to create your own keystone install playbook, because we have a generic keystone install playbook in the tests repository.
Only add a zuul.d folder when your role is imported into the openstack-ansible namespace.

Extending tests of an existing role¶

Modify the tox.ini to add your new scenario. If required, you can override the inventory, and/or the variable files.
Add a new non-voting job in zuul.d/jobs.yaml, and wire it in the project tests file zuul.d/project.yaml.

Improve testing with tempest¶

Once the initial convergence is working and the services are running, the role development should focus on implementing some level of functional testing.

Ideally, the functional tests for an OpenStack role should make use of Tempest to execute the functional tests. The ideal tests to execute are scenario tests as they test the functions that the service is expected to do in a production deployment. In the absence of any scenario tests for the service a fallback option is to implement the smoke tests instead.

If no tempest is provided, some other functional testing should be done. For APIs, you can probably check the HTTP response codes, with specially crafted requests.

Running tests locally¶

Linting¶

Python coding conventions are tested using PEP8, with the following convention exceptions:

F403 - ‘from ansible.module_utils.basic import *’

Testing may be done locally by executing:

./run_tests.sh pep8

Bash coding conventions are tested using Bashate, with the following convention exceptions:

E003: Indent not multiple of 4. We prefer to use multiples of 2 instead.
E006: Line longer than 79 columns. As many scripts are deployed as templates
and use jinja templating, this is very difficult to achieve. It is still considered a preference and should be a goal to improve readability, within reason.
E040: Syntax error determined using bash -n. As many scripts are deployed
as templates and use jinja templating, this will often fail. This test is reasonably safely ignored as the syntax error will be identified when executing the resulting script.

Testing may be done locally by executing:

./run_tests.sh bashate

Ansible is lint tested using ansible-lint.

Testing may be done locally by executing:

./run_tests.sh ansible-lint

Ansible playbook syntax is tested using ansible-playbook.

Testing may be done locally by executing:

./run_tests.sh ansible-syntax

A consolidated set of all lint tests may be done locally by executing:

./run_tests.sh linters

Documentation building¶

Documentation is developed in reStructuredText (RST) and compiled into HTML using Sphinx.

Documentation may be built locally by executing:

./run_tests.sh docs

The OpenStack-Ansible integrated repo also has an extra documentation building process, to build the deployment guide.

This guide may be built locally by executing:

./run_tests.sh deploy-guide

Release notes building¶

Release notes are generated using the the reno tool and compiled into HTML using Sphinx.

Release notes may be built locally by executing:

./run_tests.sh releasenotes

Note

The releasenotes build argument only tests committed changes. Ensure your local changes are committed before running the releasenotes build.

Roles functional or scenario testing¶

To run a functional test of the role, execute:

./run_tests.sh functional

Testing a new role with an AIO¶

Include your role on the deploy host. See also Adding new or overriding roles in your OpenStack-Ansible installation.
Perform any other host preparation (such as the tasks performed by the bootstrap-aio.yml playbook). This includes any preparation tasks that are particular to your service.
Generate files to include your service in the Ansible inventory using env.d and conf.d files for use on your deploy host.

Hint

You can follow examples from other roles, making the appropriate modifications being sure that group labels in env.d and conf.d files are consistent.

Hint

A description of how these work can be found in Understanding host groups (conf.d structure) and Understanding container groups (env.d structure).
Generate secrets, if any, as described in the Configure Service Credentials. You can append your keys to an existing user_secrets.yml file or add a new file to the openstack_deploy directory to contain them. Provide overrides for any other variables you will need at this time as well, either in user_variables.yml or another file.

See also our Overriding default configuration page.

Any secrets required for the role to work must be noted in the etc/openstack_deploy/user_secrets.yml file for reuse by other users.
If your service is installed from source or relies on python packages which need to be installed from source, specify a repository for the source code of each requirement by adding a file to your deploy host under inventory/group_vars/<service_group>/source_git.yml in the OpenStack-Ansible source repository and following the pattern of files currently in that directory. You could also simply add an entry to an existing file there.
Make any required adjustments to the load balancer configuration (e.g. modify inventory/group_vars/all/haproxy.yml in the OpenStack-Ansible source repository on your deploy host) so that your service can be reached through a load balancer, if appropriate, and be sure to run the haproxy-install.yml play later so your changes will be applied. Please note, you can also use haproxy_extra_services variable if you don’t want to provide your service as default for everyone.
Put together a service install playbook file for your role. This can also be modeled from any existing service playbook that has similar dependencies to your service (database, messaging, storage drivers, container mount points, etc.). A common place to keep playbook files in a Galaxy role is in an examples directory off the root of the role. If the playbook is meant for installing an OpenStack service, name it os-<service>-install.yml and target it at the appropriate group defined in the service env.d file. It is crucial that the implementation of the service is optional and that the deployer must opt-in to the deployment through the population of a host in the applicable host group. If the host group has no hosts, Ansible skips the playbook’s tasks automatically.
Any variables needed by other roles to connect to the new role, or by the new role to connect to other roles, should be implemented in inventory/group_vars. The group vars are essentially the glue which playbooks use to ensure that all roles are given the appropriate information. When group vars are implemented it should be a minimum set to achieve the goal of integrating the new role into the integrated build.
Documentation must be added in the role to describe how to implement the new service in an integrated environement. This content must adhere to the Documentation and Release Note Guidelines. Until the role has integrated functional testing implemented (see also the Role development maturity paragraph), the documentation must make it clear that the service inclusion in OpenStack-Ansible is experimental and is not fully tested by OpenStack-Ansible in an integrated build. Alternatively, an user story can be created.
A feature release note must be added to announce the new service availability and to refer to the role documentation for further details. This content must adhere to the Documentation and Release Note Guidelines.
It must be possible to execute a functional, integrated test which executes a deployment in the same way as a production environment. The test must execute a set of functional tests using Tempest. This is the required last step before a service can remove the experimental warning from the documentation.

Hint

If you adhere to the pattern of isolating your role’s extra deployment requirements (secrets and var files, HAProxy yml fragments, repo_package files, etc.) in their own files it makes it easy for you to automate these additional steps when testing your role.

Integrated repo functional or scenario testing¶

To test the integrated repo, follow the Deployment Guide

Alternatively, you can check the aio guide, or even run the gate wrapper script, named scripts/gate-check-commit.sh, described below.

The OpenStack Infrastructure automated tests¶

There should be no difference between running tests in the openstack infrastructure, versus running locally.

The tests in the openstack infrastructure are triggered by jobs defined in each repo zuul.d folder.

Role development maturity¶

A role may be fully mature, even if it is not integrated in the openstack-ansible repository. The maturity depends on its testing levels.

A role can be in one of the four maturity levels:

Complete
Incubated
Unmaintained
Retired

Here are a series of rules that define maturity levels:

A role can be retired at any time if it is not relevant anymore.
A role can be Incubated for maximum 2 cycles.
An Incubated role that passes functional testing will be upgraded to the Complete status, and cannot return in Incubated status.
An Incubated role that didn’t implement functional testing in the six month timeframe will become Unmaintained.
A role in Complete status can be downgraded to Unmaintained. status, according to the maturity downgrade procedure.

Maturity downgrade procedure¶

If a role has failed periodics or gate test for two weeks, a bug should be filed, and a message to the mailing list will be sent, referencing the bug.

The next community meeting should discuss about role deprecation, and if no contributor comes forward to fix the role, periodic testing will be turned off, and the role will move to an unmaintained state.

Maturity Matrix¶

All of the OpenStack-Ansible roles do not have the same level of maturity and testing.

Here is a dashboard of the current status of the roles:

Active roles

Role name	Created during cycle	Maturity level	Included into openstack-ansible by default	Supports Ubuntu	Supports CentOS
ansible-hardening	Liberty	Complete	✔	✔	✔
apt_package_pinning	Mitaka	Complete	✔	✔	✘
ceph_client	Newton	Complete	✔	✔	✔
galera_client	Mitaka	Complete	✔	✔	✔
galera_server	Mitaka	Complete	✔	✔	✔
haproxy_server	Newton	Complete	✔	✔	✔
lxc_container_create	Mitaka	Complete	✔	✔	✔
lxc_hosts	Mitaka	Complete	✔	✔	✔
memcached_server	Mitaka	Complete	✔	✔	✔
openstack_hosts	Mitaka	Complete	✔	✔	✔
openstack_openrc	Mitaka	Complete	✔	✔	✔
os_adjutant	Ussuri	Incubated	✔	✔	✔
os_aodh	Mitaka	Complete	✔	✔	✔
os_barbican	Ocata	Complete	✔	✔	✔
os_ceilometer	Mitaka	Complete	✔	✔	✔
os_cinder	Mitaka	Complete	✔	✔	✔
os_cloudkitty	Ocata	Incubated	✘	✔	✔
os_designate	Mitaka	Complete	✔	✔	✔
os_glance	Mitaka	Complete	✔	✔	✔
os_gnocchi	Newton	Complete	✔	✔	✔
os_heat	Mitaka	Complete	✔	✔	✔
os_horizon	Mitaka	Complete	✔	✔	✔
os_ironic	Mitaka	Complete	✔	✔	✔
os_keystone	Mitaka	Complete	✔	✔	✔
os_magnum	Newton	Complete	✔	✔	✔
os_manila	Stein	Incubated	✔	✘	✘
os_masakari	Stein	Complete	✔	✔	✔
os_murano	Train	Incubated	✔	✔	✔
os_neutron	Mitaka	Complete	✔	✔	✔
os_nova	Mitaka	Complete	✔	✔	✔
os_octavia	Pike	Complete	✔	✔	✔
os_rally	Newton	Complete	✔	✔	✔
os_sahara	Newton	Complete	✔	✔	✔
os_senlin	Ussuri	Inclubated	✔	✔	✔
os_swift	Mitaka	Complete	✔	✔	✔
os_tacker	Queens	Complete	✔	✔	✘
os_tempest	Mitaka	Complete	✔	✔	✔
os_trove	Ocata	Complete	✔	✔	✘
os_zun	Rocky	Complete	✔	✔	✔
pki	Wallaby	Complete	✔	✔	✔
plugins	Mitaka	Complete	✔	✔	✔
rabbitmq_server	Mitaka	Complete	✔	✔	✔
repo_server	Mitaka	Complete	✔	✔	✔
uwsgi	Train	Complete	✔	✔	✔
zookeeper	Zed	Complete	✔	✔	✔

Retired roles

Role name	Created during cycle	Retired during cycle
openstack-ansible-security	Liberty	Pike
pip_lock_down	Liberty	Newton
repo_build	Mitaka	Ussuri
rsyslog_client	Mitaka	Zed
rsyslog_server	Mitaka	Zed
pip_install	Mitaka	Ussuri
os_almanach	Queens	Train
os_congress	Unknown	Ussuri
os_monasca	Ocata	Train
os_monasca-agent	Ocata	Train
os_monasca-ui	Ocata	Train
os_molteniron	Pike	Train
os_searchlight	Queens	Train
os_watcher	Queens	Train
os_panko	Rocky	Xena
os_zaqar	Queens	Train

Testing

Testing¶

Adding tests to a new role¶

Extending tests of an existing role¶

Improve testing with tempest¶

Running tests locally¶

Linting¶

Documentation building¶

Release notes building¶

Roles functional or scenario testing¶

Testing a new role with an AIO¶

Integrated repo functional or scenario testing¶

The OpenStack Infrastructure automated tests¶

Role development maturity¶

Maturity downgrade procedure¶

Maturity Matrix¶

Active roles

Retired roles

openstack-ansible 28.1.0.dev98

Page Contents