Testing (QA and CI)

OpenStack projects have robust automated testing. In general, we believe something not tested is broken. OpenStack is an extremely complex suite of software largely designed to interact with other software that can be operated in a variety of configurations. Manual localized testing is unlikely to be sufficient in this situation. By default, all the testing is done with current versions of dependencies managed centrally in the upper-constraints.txt file within the openstack/requirements repository. Testing lower bounds of dependencies using a lower-constraints.txt file is optional for projects.

Types of Tests

OpenStack projects employ a number of approaches to testing:

Unit tests

These tests are generally designed to validate a single method or class and verify that the code operates as designed across all inputs. They may employ mocks or fakes to simplify interaction with components outside the immediate area of the test.

Most projects have a large number of unit tests and expect to have near-complete code coverage with those tests. Changes to project code should generally include changes to add testing as well.

Functional tests

Functional tests validate a project in operation (e.g., if the project is an API server, the server will actually be running), though interactions with external components are kept to a minimum. This may be accomplished by using fake drivers for interactions with other components, or simpler implementations of those components.

Functional tests are expected to catch most actual operational bugs in an environment that is easier to understand and debug than a full integration test environment.

Integration tests

Integration testing includes the project and all related components operating in as realistic manner as possible. The goal is to identify bugs that are only likely to appear in production, before they are found in production. Often, when a bug is found in integration testing, it may indicate a gap in functional testing.

Style checks

When large numbers of developers work together on a project’s codebase, it is often useful to have a set style for code. Style checks (such as pep8 for Python) are an automated way to ensure a consistent code style from all contributors.

Automated Test Systems

OpenStack projects can generally be tested locally by developers before submitting changes. However, not all developers have access to sufficient test resources to run all tests, and moreover, project reviewers can not assume that a developer has performed all of the relevant testing. Therefore, OpenStack projects can define a set of test jobs that automatically run on every change to the project. These are run by a tool called Zuul and results appear in the Gerrit code review system under the name Jenkins.

In order to facilitate projects interacting with the automated test system in a standardized way, the Technical Committee has adopted the Consistent Testing Interface which describes what facilities a project must provide for both developers and the automated systems to run tests on the project.

Project Gating

OpenStack projects do not permit anyone to directly merge code to a source code repository. Instead, any member of a core reviewer team may approve a change for inclusion, but the actual process of merging is completely automated. After approval, a change is automatically run through tests again, and only if the change passes all of the tests, is it merged.

This process ensures that the main branch of development is always working (at least as well as can be determined by its testing infrastructure). This means that a developer can, at any point, check out a copy of the repository and begin work on it without worrying about whether it actually functions.

This is also an important part of the egalitarian structure of our community. With no member of the project able to override the results of the automated test system, no one is tempted to merge a change on their own authority under the perhaps mistaken impression that they know better than the test system whether a change functions correctly.

For more information on the automated testing infrastructure itself, including how to configure and use it, see the OpenDev Manual.

How to Handle Test Failures

If Zuul reports a test failure on a patch, the first step should be identifying what went wrong. You will be tempted to just recheck the patch to see if it fails again, but please DO NOT DO THAT. CI test resources are a very scarce resource (and becoming more so all the time), so please be extremely sparing when asking the system to re-run tests.

Note

Please do not EVER simply recheck without a reason. Even if that reason is “I don’t know”, please indicate that you at least attempted to determine the reason for the failure.

It is important that before you request a recheck, you adhere to the following guidelines:

  1. First, you should examine the logs of the jobs that failed. Look for the reason why the job failed, be it failed tests, or a setup failure, such as a failed devstack run, or job timeout. You should always begin this process suspecting that the failure is a result of the proposed patch itself, but with an eye to the problem being unrelated. Try to determine the most obvious cause for the failure, and do not ignore failures in multiple voting jobs.

  2. If the failure is likely caused by the proposed patch, you should try whenever possible to reproduce the failure locally. This will allow you to revise the change and re-submit with a higher likelihood of subsequently getting a passing run.

  3. If the failure appears to be totally unrelated to the patch at hand, look for some indication of what went wrong. Only after you have done this should you ask Zuul to re-run the tests. To do this, comment on the patch the recheck command and a reason. Examples of this are:

    recheck nova timed out waiting for glance

    recheck glance lost connection to mysql

    recheck cinder failed to detach volume

  4. The gold standard for recheck commands is recheck bug #XXXXXXX, which directly references a known problem that is being worked. Doing this helps add heat to that bug and enables stats tracking so the community knows what bugs are blocking the most people in the CI system.

  5. In some cases, it may be entirely unclear why something failed. In this case, you may need to recheck with a reason of “Not sure what failed, rechecking to get another data point.”

  6. If a recheck results in a similar failure on the subsequent run, it would be best to reach out (via the mailing list or IRC) to the project team responsible for the service you think is failing and look for some guidance on whether or not the issue is known and being worked, as it may be that a patch for the problem is proposed but not merged, which you can Depends-On to move forward.

  7. Especially if the same failure occurs more than once and is not yet reported, it is highly recommended that you open a bug against the project (or projects) affected and use that for your recheck.

Suggestions For Determining Causes of Failure

This is more art than science, but here are some ideas:

  • First examine the job-output.txt file to see if the job failed while running tests, or earlier when setup was running.

  • If it looks like a test failure, the testr_results.html file is usually very helpful for looking at individual failures.

  • If a test failed, try to identify which services are being used in that test. Quickly skim the logs for those services looking for ERROR lines and especially tracebacks that seem to line up with the test failure. For example, if the test is a compute failure to attach a volume, it would be good to look at n-api, n-cpu, c-api, and c-vol logs as Nova and Cinder are both involved in that process.

  • Test failures in tempest-based jobs generally print out resource IDs, such as instance or volume UUIDs. Use these to search the relevant logs for errors and warnings related to a resource that was involved in the test failure.

  • Looking at the timestamps of test failures can also help locate relevant lines in the service logs.

Checking status of other job results

Each Zuul CI job results is sent to the Opensearch service. This service can be very useful specially when the Zuul job status is Failure. Checking whether a given error has not occured in another project or it has not appeared regularly recently will allow for faster problem recognition or notification of the OpenStack community about the problem.

To check the Opensearch service, you need to login with credentials that are described below:

Automatic Test Failure Identification

OpenStack project integration tests have logs from running services automatically uploaded to a logstash-based processing system. An automated system finds signatures related to known bugs and leaves code review comments about them. Additionally, aggregate statistics about the number of times a known bug has caused a test failure are reported.

If your project participates in an integration test with other OpenStack projects, you are welcome to add definitions for your project’s logs and signatures for known bugs to the log processing system.

For more information on how to deal with integration test failures, see the Developer’s Guide.

Zuul v3 native jobs

At the beginning of the Victoria cycle an important amount of jobs of “legacy” jobs is still present. The legacy jobs are scripts automatically converted from the old Jenkins jobs and contain a lot duplicated code shell code. One of the community goals for the cycle deals with porting them to Zuul v3.

A detailed porting guide, originally part of the OpenDev documentation, has been imported here:

In addition to it, other documentation sources are:

It is worth noting that, while the goal focuses on the fixing the jobs used in Victoria (and future releases), backporting the converted jobs to the older branches where they are still in use is recommended for maintenance reasons in the long run.