Zed Series Release Notes


Bug Fixes

  • The Nova API container extended startup script has been updated to only sync the local Nova cell. This resolves an error that would occur when the Nova database password changes. More details can be found on this bug report.


Bug Fixes

  • Fixes missing way for kolla-toolbox to be built offline. LP#2020761

  • Adds missing grafana-opensearch-datasource plugin to the list of plugins in the docker image.

  • Fix container restart conditions to be tolerant of missing optional source and/or destination files. For more details, see the following bug report


Bug Fixes

  • Fixes an issue where the script kolla_ensure_openvswitch_configured in the openvswitch-db-server image would ignore errors encountered while configuring bridges and ports. LP#1999778



  • Support for binary images got removed in Zed. Users are requested to migrate to source based images.

  • Rocky Linux 9 is now supported as a base container image.

New Features

  • Updates Alertmanager version to 0.24.0.

  • Adds support for TPM emulation in Nova (via “swtpm”).

  • Adds OpenSearch and OpenSearch Dashboards images.

  • Updates the OpenStack exporter for Prometheus to version 1.6.0.

  • Adds prometheus-ovn-exporter image.

  • Quiet mode (enabled with --quiet argument) can be combined with --logs-dir option now. Console output will be quiet as expected while building output will be stored in separate log files.

  • Added an --repos-yaml argument to allow user to provide own file with definitions of external package repositories. Useful for those building in offline environments with set of internal mirrors.

Upgrade Notes

  • Change kolla_version label to git sha-1 hash if images are built with kolla from git repository.

  • To fix CVE-2022-38060, support for KOLLA_CONFIG and KOLLA_CONFIG_FILE environment variables in kolla-built containers has been dropped. Now, only the single trusted path of /var/lib/kolla/config_files/config.json will be utilised for loading container config. We believe this is a reasonable tradeoff as these environment variables were not used by any known downstream and potential users in the wild can easily adapt as this does not limit the functionality per se, only making it stricter as to where the config can come from.

  • Prometheus services were updated to following versions:
    • blackbox_exporter -> 0.22.0

    • elasticsearch_exporter -> 1.5.0

    • haproxy_exporter -> 0.13.0

    • memcached_exporter_version -> 0.10.0

    • mysqld_exporter -> 0.14.0

    • node_exporter -> 1.4.0

    • prometheus -> 2.38.0

    • prometheus_cadvisor -> 0.45.0

    • prometheus_libvirt_exporter -> 2.3.2

    • prometheus_msteams -> 1.5.1

    • prometheus_mtail -> v3.0.0-rc50

  • Default base distribution has been changed from CentOS Stream to Rocky Linux.

  • Removes images for monasca, kafka, storm and zookeeper, since support for them has been dropped in Kolla-Ansible in Zed release. Prometheus + Grafana + Fluentd + OpenSearch remain as the primary monitoring, logging and alerting stack in Kolla.

  • elasticsearch, kibana and logstash images have been dropped. Zed release brings in support for opensearch and opensearch-dashboards but there’s no equivalent for logstash.

  • Python 3.6 & 3.7 support has been dropped. The minimum version of Python now supported is Python 3.8.

  • The qdrouterd image has been dropped.

  • etcd is now installed from the upstream binaries published to github rather than via the OS package manager. This aligns the etcd version across all distributions for compatibility.

  • Kolla Build no longer prepends the base (distro) name to image names. Instead, the user is able to choose any prefix they wish via the image_name_prefix setting.

  • The updated OpenStack exporter for Prometheus uses the latest Nova API microversion by default, resulting in changes to existing metrics. To keep existing behaviour, set prometheus_openstack_exporter_compute_api_version to 2.1.

  • RabbitMQ version has been updated to 3.10 (together with Erlang to 25).

  • The Debian and Ubuntu images use rabbitmq from cloudsmith and erlang from Team RabbitMQ PPA now. Operators might want to mirror/proxy those new sources as they provides the correct set of packages unlike the previous combination.

  • Ansible in kolla-toolbox container has been upgraded to 2.13 version.

Deprecation Notes

  • The hacluster-pcs image has been deprecated for removal in the Antelope release.

  • Use of install_type argument is now deprecated. We no longer support other values than source therefore handling of argument was dropped. Please update your scripts as it will be removed in Antelope cycle.

Security Issues

  • Fixes a hypothetical security issue related to privilege escalation via rootwrap/privsep. A potential vulnerable service could previously allow writes to its rootwrap/privsep config and thus allow for more commands to be run with root privileges via rootwrap/privsep. For a succesful attack, this would also require the service to allow to run arbitrary commands via rootwrap/privsep. Thus far, no such vulnerabilities have been reported and thus this fix is simply strengthening the container images against such an issue in the future. LP#1874298

  • Fixes CVE-2022-38060, a sudo privilege escalation vulnerability. LP#1985784

Bug Fixes

  • The apt-get update command by default didn’t fail on erroneous source repositories, it show the warning ‘W: Some index files failed to download. They have been ignored, or old ones used instead.’ and continue to work. This causes some containers (eg. rabbitmq, kolla-toolbox) successfully built, but makes them inconsistent because the official Ubuntu repository contains packages with the same names. Now we use apt-get -eany update command to stop building with an error in such cases.

  • Fixes CentOS builds of Skydive SEGV on startup. Skydive versions prior to 0.28.0 panic on newer versions of libc. This especially affects Centos 8. LP#1940862

  • Fixes an issue building images that use a source with a type of git, when using a git that includes the fix for CVE-2022-24765 (2.35.2 or later). By default, this includes the gnocchi-base image, but may include other images with a non-default configuration. LP#837710

  • Fixes the Debian and Ubuntu images to use rabbitmq from cloudsmith and erlang from Team RabbitMQ PPA so that the images are still buildable and use proper versions.

  • Fixes an issue with Swift deployment via Kolla Ansible caused by the fix to CVE-2022-38060. The kolla-toolbox container now have its own sudoers secure_path configuration which allows the necessary binaries to execute.

Other Notes

  • Added ‘–retry 5’ to curlrc to improve curl downloads during image builds.