Train Series Release Notes


New Features

  • Adds Cyrus SASL packages necessary for the DIGEST-MD5 and SCRAM-SHA-256 mechanisms. These can be used for libvirt SASL authentication. LP#1964013

Critical Issues

  • CentOS Linux 8 (non-Stream) support has been dropped, since repositories have been removed from CentOS mirrors - see announcement.

Security Issues

  • Adds mitigation for Apache Log4j 2 Remote Code Execution (RCE) vulnerabilities CVE-2021-44228 and CVE-2021-45046 to Apache Storm.

Other Notes

  • CentOS images (only source, not binary) are now buildable using CentOS Stream 8 as base.


Upgrade Notes

  • RabbitMQ and Erlang packages are now installed from (and PPA for Ubuntu) since is getting shut down May 1st, 2021.

  • Ubuntu based images use APT mirrors now. May affect builds done behind http proxies.

Bug Fixes

  • Fix support for kolla install in ~/.local. LP#1930544

  • Fixes issues arising from the lack of Debian updates repo being enabled. LP#1931544

  • Fixes Mistral source images to respect upper-constraints.

Other Notes

  • Debian images enable the Debian updates repo now. This is aligned with the base Debian image.


Upgrade Notes

  • Kolla now no longer supports CentOS 8.2 and below. This is to support CentOS 8.3 without extra workarounds (please see the fixes section for more details). The promise is to support the latest CentOS 8 release which is 8.3 now.

  • Almanach and Dragonflow images are no longer available for Debian/Ubuntu.

  • The networking-hyperv package is no longer installed in the neutron-server source image.

  • The kuryr-libnetwork image is no longer available.

  • helm-repository image is now unbuildable due to the chart repository being gone. The image was deprecated and is not known to be used by any tooling.

Bug Fixes

  • Fixes mixed RabbitMQ and Erlang package sources on CentOS 8 (Train only). Those could lead to RabbitMQ cluster instability in certain circumstances. LP#1884034

  • Fixes the FC Cinder backend usage in Nova. LP#1884484

  • Logstash 6 introduced in Centos 8 in Train release comes with log4j2 configuration that does not remove old compressed logs after rotation Log rotation config backported from Logstash 7 - Combination of Size Based and Time Based policies. Deletion occurs after 30 days or 3000 MB log files size - whichever comes first.

  • Fixes MariaDB incremental backup failure when full backup was not created the same day. LP#1897948

  • Fixes builds on CentOS 8.3 failing due to renamed repos. Notice Kolla now no longer supports CentOS 8.2 and below. LP#1907213

  • Fixes an issue with the kolla_set_configs --check command when the source is a directory. LP#1890567

  • Fixes an issue with the Masakari dashboard where policies were not loaded correctly.

  • Fixes the masakari-monitors image on CentOS 8.

  • nova-compute uses daxio to cleanup vpmem backend device on instance delete. If the daxio binary is missing in the nova-compute container instance delete fails. daxio is provided in centos via daxio, in ubuntu via the pmdk-tools package.

  • Fixes an issue which can block the Monasca Fluentd output plugin. LP#1889065


New Features

  • Adds Elasticsearch Curator for managing aggregated log data.

Upgrade Notes

  • The Logstash image has been upgraded from Logstash 2 to Logstash 6 for Centos 7 and Centos 8 only.

Bug Fixes

  • Drop systemd support from nsswitch.conf on RHEL-based distros. This avoids unneeded systemd nss lookups inside containers and it also avoids possible selinux denials when a container bind mounts /run and makes the dbus socket available inside the container only to be denied by selinux on the host.

  • Fixes an issue with loading Storm and Monasca Thresh when using Centos8 containers.

  • Fixes a bug in Monasca Agent Statsd which causes it to fail under Python 3.


New Features

  • Adding ndctl package to nova-compute to expose NVDIMM namespaces to guests. The package is needed to manage PMEM namespaces.


New Features

  • Adds collectd-dpdk_telemetry and collectd-logparser packages to collectd RHEL8-based image. dpdk_telemetry plugin collects DPDK ethernet device metrics via dpdk_telemetry library. Logparser is plugin for filtering and parsing log messages.

  • Adds support for CentOS 8 as a base container image. This is the only major version of CentOS supported from the Ussuri release. The Train release supports both CentOS 7 and 8 images, and provides a route for migration.

  • Adds new elasticsearch6 and kibana6 images for CentOS 7 and 8 only. These images are used to provide compatibility between versions in CentOS 7 and 8 images. These images will only be available in the Train release.

Known Issues

  • AArch64 images using CentOS 8 as base system are not supported in Train. This may be fixed later in release cycle as we need CentOS 8.2 release.

Upgrade Notes

  • The following images are supported by CentOS 7 but lack suitable packages in CentOS 8, and are not supported for CentOS 8: cyborg-agent, hacluster-pcs, nova-spicehtml5proxy.

  • The following images are supported by CentOS 7 but are not supported for CentOS 8 as they have been dropped in Ussuri: almanach-*, ceph-*, dind, dragonflow-*, helm-repository, kube*, mongodb, opendaylight, sensu.

  • Support for the SCSI target daemon (tgtd) has been removed for CentOS/RHEL 8. In CentOS/RHEL 7 and beyond LIO kernel subsystem can be used instead of the tgtd daemon. The tgtd image is no longer available for CentOS/RHEL 8.

  • Changes the behaviour of the --skip-existing and --skip-parents flags. Previously these were not applied if no regular expression or profile argument was provided to kolla-build, but now they are.

  • The trickle package is no longer available for CentOS 8, and has been removed from the CentOS 8 Freezer images.

  • Adds a new rabbitmq-3.7.24 image for CentOS 7 only. This image is used to provide compatibility between RabbitMQ versions in CentOS 7 and 8 images. This image will only be available in the Train release.

  • Removes the ceph and process-checks plugins from the sensu-client image. These plugins have a dependency on version 0.6.3 of the Ruby gem english, which has been “yanked” from

Bug Fixes

  • Adds openssh-clients to ironic conductor container build to enable ansible deploy interface to function properly.

  • Adds python3-systemd package to ironic-conductor source based container to allow the Ansible deploy interface to function correctly. Fixes bug #1861427

  • Fix inability to run UEFI-based images/instances by installing UEFI packages also in nova-libvirt image which is not based on nova-base. LP#1814552

  • Keystone bootstrap could produce invalid json. LP#1866017

  • Fixes the MAX_NUMBER variable usage when running the database online migrations for cinder.

  • Fixes Glance inability to use Cinder NFS backend for images by including NFS client components in the Glance API image. LP#1868574

  • Adds missing vitrage-persistor image, required by Vitrage deployments for storing data. LP#1869319

  • Fix kolla_toolbox_pip_virtualenv_packages customisation. LP#1865119

  • Fixes an issue with Cyborg and Monasca APIs in Debian and Ubuntu source type images. LP#1873421

  • Fixes an issue with the --skip-existing and --skip-parents flags which could cause images to not build. LP#1867614.


Bug Fixes

  • Fix bug which caused Keystone Fernet key distribution to fail on Python 3 systems, by adapting script to work on Python 3. LP#1859047

  • Fixes an issue with keystone bootstrap where an error message emitted by the keystone-manage bootstrap command is hidden. See bug 1855701 for details.

  • Converts deprecated command rally-manage db to rally db. LP#1856693

  • Fixes swift-object-expirer for Debian and Ubuntu binary images. LP#1859607



The Kolla 9.0.0 release is the first release in the Train cycle. Highlights include new images for the Masakari instance High Availability service and Qinling which provides Function as a Service. Ubuntu and Debian source images are now using Python 3.

New Features

  • Adds HAcluster images. These images contain services supporting High Availability such as Corosync, Pacemaker, Pacemaker Remote and PCS.

    HAcluster will not handle any OpenStack control plane resources, it will be used as third party for OpenStack Masakari for example to handle instance failover following a Nova compute crash.

  • Adds Qinling images. Qinling is an OpenStack project to provide “Function as a Service”. This project aims to provide a platform to support serverless functions.

  • Adds configration option use_dumb_init, with default value of True. This can be use to avoid the of dumb-init as the container entrypoint, using kolla_start directly instead. This option can also be disabled via the kolla-build --nouse-dumb-init CLI argument.

  • Adds Masakari images. Masakari provides Instances High Availability Service for OpenStack clouds by automatically recovering failed Instances.

  • Improves the skipped images feature, allowing filtering based on used image distribution, installation type and processor architecture.

Upgrade Notes

  • Moves the ENTRYPOINT statement outside of the dumb_init_installation Jinja block in the base image. Overriding this block to install dumb-init by another method no longer requires repeating the ENTRYPOINT statement. Users wishing to avoid the use of dumb-init altogether can now use the use_dumb_init configuration option.

  • The fluentd image no longer includes the kubernetes_metadata_filter plugin. It is not used by Kolla downstream projects (Kolla-Ansible, TripleO, OpenStack-Helm). It can be installed by customizing fluentd_packages.

  • The xtrabackup image has been removed because XtraBackup is no longer compatible with the versions of MariaDB shipped with Kolla images. Mariabackup should be used instead.

  • The crane image which was deprecated in the Stein cycle has been removed.

  • The nova-consoleauth image has been removed. This service has been deprecated in nova since Rocky and has not been used by other nova services since.

  • The nova-placement-api image was renamed to placement-api in the Stein release, and has now been removed.

  • Removes support for building OracleLinux container images.

  • The tripleo-ui container is no longer built as the project has been retired.

  • In Ubuntu images, MariaDB has been upgraded from 10.1 to 10.3. As usual, ensure that all data has been backed up prior to upgrading.

  • en_US.UTF-8 is set as the default locale (LANG) for images. This affects both build- and run-time. Distributions supported by Kolla default to UTF-8 locale in installs so this change should provide a more expected experience. It makes images Unicode-friendly.

Deprecation Notes

  • The Almanach images are deprecated and will be removed in the Ussuri cycle. This includes almanach-api and almanach-collector. These are not used by Kolla downstream projects.

  • The dind image is deprecated and will be removed in the Ussuri cycle. It is not used by Kolla downstream projects (Kolla-Ansible, TripleO, OpenStack-Helm). It has not seen recent usage and the upstream project seems no longer active.

  • The Dragonflow images are deprecated and will be removed in the Ussuri cycle. This includes dragonflow-controller, dragonflow-metadata, and dragonflow-publisher-service. These are not used by Kolla downstream projects.

  • Kubernetes-related images are deprecated and will be removed in the Ussuri cycle. They are not used by Kolla downstream projects (Kolla-Ansible, TripleO, OpenStack-Helm). They were used in the Kolla-Kubernetes project which was retired in the Rocky cycle. The deprecated images include: helm-repository, kube-*, kubernetes-entrypoint and kubetoolbox. The install_kubectl macro is deprecated as well and, along with it, the magnum-conductor image’s kubectl command because it is of limited usefulness being pinned to an old version. Magnum end-users use an externally-provided kubectl. Please note this deprecation does not affect Magnum nor Qinling support.

Security Issues

  • In prior versions of InfluxDB, including 1.3.x, InfluxDB incorrectly ignored tag names starting with a leading underscore. In Monasca this broke tenant isolation because queries containing where _tenant_id = ‘some_id’ where not scoped to the tenant_id. Upgrading to InfluxDB 1.7.x solves this issue.

Bug Fixes

  • Fixes unavailability of an etcd3-compatible tooz coordination driver in Ubuntu binary images by installing python3-etcd3gw. See bug 1852086 for details.