Continuous Integration (CI) & Testing

Kolla-Ansible uses Zuul for continuous integration. Similar to testing performed using devstack, Kolla-Ansible is capable of integrating and testing pre-merged dependencies from many other projects.

Debugging with ARA in CI

Frequently, the need arises to obtain more verbose ansible logging in CI. ARA is an ansible plugin that collects a large amount of execution information and can render it into a browser friendly format.

This plugin is not enabled by default because there is a per-task overhead. However, it’s possible to trigger it when trying to debug a failing job.

By adding the text #ara to the git commit message of the review, the CI jobs will enable the plugin and generate a sqlite database containing comprehensive logging. It’s possible to render an HTML version of this by using #ara_verbose. Generating the HTML is not very efficient, however, and consumes a large amount of logging resources.

Please note that git usually strips lines beginning with # from the commit message. This can be avoided by preceding the string with a space.

CI coverage matrix

Zuul job templates live in zuul.d and are applied in zuul.d/project.yaml. Pipelines always run tox/python style+unit checks, docs/release builds, and requirements checks; the matrix below shows the scenario jobs in check/gate.

Scenario matrix

Scenario

Disabled

Voting

Tempest

Core

Upgrade

SLURP

Multinode

TLS

Ceph

Cinder

Octavia

Valkey

Notes

AIO core

–

✅

✅

✅

✅

✅

–

✅

–

–

–

–

Core OpenStack (Keystone/Glance/Neutron/Nova/Heat) with Horizon; RabbitMQ/MariaDB/Memcached; Fluentd.

Bifrost

–

–

–

–

–

–

–

–

–

–

–

–

Bifrost baremetal provisioning standalone.

Cells

–

✅

✅

–

✅

–

✅

–

–

–

–

–

Nova Cells v2 + ProxySQL; Horizon/Heat on; dashboard sanity.

Cephadm

–

✅

✅

✅

✅

✅

✅

✅

✅

–

✅

✅

External Ceph (cinder/glance/nova) + RGW; RabbitMQ tuned.

Container engine migration

❌

–

✅

✅

–

–

✅

✅

–

–

–

–

Container engine migration playbook; Tempest smoke; currently disabled (failing).

Federation

–

–

✅

–

✅

–

✅

✅

–

–

–

–

Keystone federation behind httpd.

HAProxy FQDN

–

–

✅

–

–

–

✅

✅

–

–

–

–

HAProxy VIP/FQDN handling.

Ironic

–

–

✅

✅

–

–

–

–

–

–

–

–

Ironic + Tenks; Prometheus on; DNSMasq/iSCSI helpers.

IPv6

❌

–

✅

–

✅

–

–

–

–

–

–

–

IPv6-only deployment; currently disabled (disk pressure).

Kayobe

–

–

–

–

–

–

–

–

–

–

–

–

Kayobe overcloud smoke (non-voting).

KVM

–

–

–

✅

–

–

✅

–

–

–

–

–

Nested virt compute (KVM).

Let’s Encrypt

–

✅

✅

–

✅

–

✅

–

–

–

–

–

Pebble ACME end-to-end, dashboard; DB/Rabbit TLS.

Magnum

–

–

✅

–

–

–

–

–

–

–

–

–

Magnum + Designate + Trove.

MariaDB

–

✅

–

✅

✅

–

✅

–

–

–

–

–

Galera + ProxySQL, Fluentd, HAProxy, Keepalived.

Masakari

–

–

✅

–

–

–

–

–

–

–

–

✅

Masakari + Pacemaker.

NFV

–

–

✅

–

✅

–

–

–

–

–

✅

✅

Tacker + Neutron SFC, Barbican, Mistral, Aodh.

Octavia

–

–

✅

–

–

–

–

–

✅

✅

–

–

Octavia/Amphora.

OpenBao

–

–

–

–

–

–

–

–

–

–

–

–

OpenBao-backed password management (password round-trip only; no deploy).

OVN

–

✅

✅

✅

✅

–

✅

✅

–

✅

–

–

Neutron ML2/OVN + OVN Octavia provider.

Prometheus + OpenSearch

–

–

–

✅

–

✅

✅

–

–

–

–

–

Fluentd, OpenSearch, Grafana, Prometheus, RabbitMQ/DB core.

Skyline

❌

–

✅

–

–

✅

–

–

–

–

–

–

Skyline dashboard; currently disabled (SSO issues).

Telemetry

–

–

✅

–

–

✅

–

–

–

–

✅

✅

Aodh, Ceilometer, Gnocchi.

Zun

–

✅

✅

–

✅

✅

✅

✅

–

–

–

–

Zun + Kuryr + Cinder LVM; etcd; dashboard.

Legend: ✅ = covered, ❌ = disabled, – = not covered. Core means the standard Keystone/Glance/Neutron/Nova/Heat stack is deployed; Tempest indicates smoke tests run; SLURP = Skip Level Upgrade Release Process.