Zuul v3 Migration Guide

Zuul v3 Migration Guide

This is a temporary section of the Infra Manual to assist in the conversion to Zuul v3. Some of the content herein will only be relevant before and shortly after we move from Zuul v2 to v3.

What is Zuul v3?

Zuul v3 is the third major version of the project gating system developed for use by the OpenStack project as part of its software development process. It includes several major new features and backwards incompatible changes from previous versions.

It was first described in the Zuul v3 spec.

In short, the major new features of interest to OpenStack developers are:

  • In-repo configuration
  • Native support for multi-node jobs
  • Ansible job content
  • Integration with more systems

We’re pretty excited about Zuul v3, and we think it’s going to improve the devolpment process for all OpenStack developers. But we also know that not everyone needs to know everything about Zuul v3 in order for this to work. The sections below provide increasing amounts of information about Zuul v3. Please at least read the first section, and then continue reading as long as subsequent sections remain relevant to the way you work.

What’s the Minimum I Need to Know?

You have stuff to do, and most of it doesn’t involve the CI system, so this will be short.

The name of the CI system will be changing

For varied historical reasons, the name OpenStack’s CI system used to report to Gerrit has been Jenkins, even 5 years after it actually became Zuul doing the reporting and 1 year after we stopped using Jenkins altogether. We’re finally changing it to Zuul. If you see a comment from Jenkins, it’s Zuul v2. If you see a comment from Zuul, it’s Zuul v3.

Job names will be changing

In Zuul v2, almost every project has a unique python27 job. For example, gate-nova-python27. In v3, we will have a single python27 job that can be used for every project. So when Zuul reports on your changes, the job name will now be openstack-py27 rather than gate-project-python27.

For details about job names, see Consistent Naming for Jobs with Zuul v3.

Most jobs will be migrated automatically

The jobs covered by the Consistent Testing Interface will all be migrated automatically and you should not need to do anything. Most devstack jobs should be migrated as well. If your project uses only these jobs, you shouldn’t need to do anything; we’ll handle it for you.

If you have custom jobs for your project, you or someone from your project should keep reading this document.

My Project Has Customized Jobs, Tell Me More

If you’ve read this far, you may have a passing familiarity with the project-config repo and you have created some jobs of your own, or customized how jobs are run on your project.

As mentioned earlier, we’re going to try to automatically migrate all of the jobs from v2 to v3. However, some jobs may benefit from further manual tweaks. This section and the one following should give you the information needed to understand how to make those.

How Jobs Are Defined in Zuul v3

In Zuul v2, jobs were defined in Jenkins and Zuul merely instructed Jenkins to run them. This split between job definition and execution produced the often confusing dual configuration in the project-config repository, where we were required to define a job in jenkins/jobs and then separately tell Zuul to run it in zuul/layout.yaml.

Zuul v3 is responsible for choosing when to run which jobs, and running them; jobs only need to be added to one system.

All aspects of Zuul relating to jobs are configured with YAML files similar to the Zuul v2 layout. See the Zuul User Guide for more information on how jobs are configured.

Where Jobs Are Defined in Zuul v3

Zuul v3 loads its configuration directly from git repos. This lets us accomplish a number of things we have long desired: instantaneous reconfiguration and in-repo configuration.

Zuul starts by loading the configuration in the project-config repository. This contains all of the pipeline definitions and some very basic job definitions. Zuul looks for its configuration in files named zuul.yaml or .zuul.yaml, or in directories named zuul.d or .zuul.d. Then it loads configuration from the zuul-jobs repository. This repository contains job definitions intended to be used by any Zuul installation, including, but not limited to, OpenStack’s Zuul. Then it loads jobs from the openstack-zuul-jobs repository which is where we keep most of the OpenStack-specific jobs. Finally, it loads jobs defined in all of the repositories in the system. This means that any repo can define its own jobs. And in most cases, changes to those jobs will be self-testing, as Zuul will dynamically change its configuration in response to proposed changes.

This is very powerful, but there are some limitations. See the sections of the Zuul User Guide about Security Contexts and Configuration Loading for more details.

Note that all OpenStack projects share a single namespace for job names, so we have established some guidelines detailed in Consistent Naming for Jobs with Zuul v3 for how to name jobs. Adhere to these so that we may avoid collisions between jobs defined in various repositories.

Zuul jobs are documented in their own repositories. Here are links to the documentation for the repositories mentioned above:

How Jobs Are Selected to Run in Zuul v3

TODO:

  • variants
  • selectors
  • pipeline-project definition

I Write Jobs From Scratch, How Does Zuul v3 Actually Work?

TODO:

  • inheritance
  • ansible (and how it’s optional)
  • playbooks
  • roles
Creative Commons Attribution 3.0 License

Except where otherwise noted, this document is licensed under Creative Commons Attribution 3.0 License. See all OpenStack Legal Documents.