Train Series Release Notes


New Features

  • Adds the ipa_build_upper_constraints_file variable to select the upper constraints file used to install diskimage-builder in the virtual environment used for building IPA images. This allows you to install a newer release than the one allowed by the default constraints.

  • Adds support for using a CentOS 8.3 seed VM image.

Upgrade Notes

  • Updates the stackhpc.os-images role to version 1.10.2. This allows compatibility with versions of diskimage-builder that can be installed when customising ipa_build_upper_constraints_file. You must run kayobe control host upgrade to update this role in order to successfully build CentOS 8 IPA images.

Bug Fixes

  • Fixes an issue when using the --limit argument with a host pattern including commas. See story 2008255 for details.

  • Fixes calls to kolla-ansible when arguments to --kolla-limit contain special characters such as ~ or &.

  • Fixes an issue with copying Swift ring files. See story 2007297 for details.

  • Fixes issue with deleting swift ring temporary files. See story 2008354 for details.

  • Fixes building CentOS 7 deployment (IPA) images by setting the DIB_INSTALLTYPE_pip_and_virtualenv environment variable to package by default.


Bug Fixes

  • Fixes an issue with Python setup when venv is not used. See story 2008378 for details.

  • Fixes an issue with the kayobe seed service deploy command on CentOS 8.3 release. See story 2008430 for details.


New Features

  • Provides project and domain default variables for Monasca. Defaults can be overridden from the Monasca config file etc/kayobe/monasca.yml.

  • Updates the default image for the seed VM to CentOS 8.2.

Upgrade Notes

Deprecation Notes

  • Removes support for Neutron LBaaS as Kolla Ansible dropped support in the Train release.

Bug Fixes

  • Fixes the Monasca install type causing issues pulling container images. See story 2007597 for details.

  • Adds support for a custom Barbican configuration file (barbican.conf), as only extended configuration stored under a barbican folder was supported.

  • Fix an issue when dnf proxy is set and python3-pip package fails during host configure.

  • Fixes a number of issues with using Kayobe on CentOS 8.3.

  • Fixes an issue when configuring the external API network interface on controllers without an IP address which would cause kayobe overcloud host configure to fail on a host with an active virtual IP address. See story 2007736 for details.

  • Fixes an issue where disabling SELinux would fail on systems without SELinux installed. See story 2007704.

  • Fixes issues when network interfaces are configured without IP addresses. See story 2007900 for details.

  • Fixes generation of pip configuration when using a pip proxy without a local mirror.

  • Changes the value of OS_ENDPOINT_TYPE, OS_MANILA_ENDPOINT_TYPE, and OS_MISTRAL_ENDPOINT_TYPE in to use publicURL, which should fix issues with legacy CLI tools on hosts without access to the internal API. See story 2007950 for details.

  • Removes the ifcfg-eth0 network interface configuration file when not required, using a new version of the MichaelRigaert.interfaces Ansible role. This file ships with CentOS 8 cloud images and can prevent the network service from starting successfully when no eth0 interface is present or being configured. See story 2007913 for details.

  • Fix the implementation of the kayobe seed hypervisor host package update command, which was incomplete. See story 2008458 for details.

  • Fixes an issue with seed service deployment where modifying the overcloud image fails on a CentOS 8.1 host with a CentOS 8.2 bifrost_deploy container, by updating the default image for the seed VM to CentOS 8.2. See story 2007942 for details.


New Features

  • Adds a kayobe overcloud service stop command. This can be used to stop containerised services running on overcloud hosts.

  • Adds support for CentOS 8 as a host Operating System and base container image. This is the only major version of CentOS supported from the Ussuri release. The Train release supports both CentOS 7 and 8 hosts, and provides a route for migration.

  • Adds support for configuration of DNF repositories on CentOS 8. Variables have been added in a new configuration file, dnf.yml. Backwards compatibility with the Yum configuration variables is provided.

  • Adds support for applying regular package updates on CentOS 8 via DNF Automatic. Variables have been added in a new configuration file, dnf.yml. Backwards compatibility with the Yum-cron configuration variables is provided.

  • Adds a seed_vm_interfaces variable which defines the network interfaces to which the seed VM is attached.

  • Adds a variable for controlling the tag applied to built container images - kolla_tag. This separates the configuration of the tag for image building from that used for deployment (kolla_openstack_release). The default for kolla_tag is kolla_openstack_release.

Known Issues

  • Fixes an issue where provisioning a seed VM would fail when the Ansible control host and the seed hypervisor are different hosts. See story 2007530 for more details.

Upgrade Notes

  • Some images were supported by CentOS 7 but lack suitable packages in CentOS 8, and are no longer supported for CentOS. See Kolla release notes for details.

  • Support for configuring an NTP daemon on the seed and overcloud hosts is no longer present for CentOS 8, as appropriate packages are not available. Instead, Kolla Ansible is configured to deploy the chrony container on CentOS 8 overcloud hosts by default. Note that for CentOS 8 during the Train release, the standard kolla_enable_chrony variable in ${KAYOBE_CONFIG_PATH}/kolla.yml has no effect. Instead, chrony may be disabled by setting enable_chrony to false in ${KAYOBE_CONFIG_PATH}/kolla/globals.yml.

  • The default order of network interfaces in the seed VM is now sorted alphabetically based on their Kayobe network name. This may require the seed’s network interface names to be changed in configuration if the seed VM is recreated. See story 2007259 for details.

Deprecation Notes

  • The Yum configuration variables in yum.yml are deprecated and will be removed in a future release. Adapt any configuration overrides to use the new DNF variables in dnf.yml instead.

  • The yum-cron configuration variables in yum-cron.yml are deprecated and will be removed in a future release. Adapt any configuration overrides to use the new DNF automatic variables in dnf.yml.

Bug Fixes

  • Fixes an issue where chronyd would be enabled as a systemd service in addition to ntpd. This causes issues in deployments where the NTP servers have been customized, as chronyd would win the race on startup, but its configuration file would not have been configured by Kayobe. See story 2005272 for more details.

  • Fixes an issue where it was not possible to load dashboards into the Monasca Grafana fork when the default Monasca control plane OpenStack project name is used from Kolla Ansible.

  • Fix an issue where the StackHPC iDRAC role would break when configuring RAID when used with a recent release of the python-dracclient module.

  • Fixes concurrency issues while adding SSH keys to the known hosts file by performing the action serially. See story 2007628 for details.

  • Fixes issues running the following commands:

    • kayobe baremetal compute inspect

    • kayobe baremetal compute manage

    • kayobe baremetal compute provide

    See story 2007797 for details.

  • Fixes a package conflict while provisioning a seed VM on a CentOS 8 seed hypervisor with coreutils-single already installed. See story 2007612 for details.

  • Fixes failure to configure Docker devicemapper storage when the default value of docker_storage_driver is used.

  • Fixes an issue where the default value of public_net_name included a trailing newline. See story 2007654.

  • Fixes an issue with idempotency of Ironic Inspector rule creation. See story 2007399 for details.

  • Fixes a bug where introspection data save would fail. See Story 2007326 for more details.

  • Fixes an issue with seed VMs with multiple network interfaces where interfaces could come up in a different order if the VM is recreated. The interfaces are now created in alphabetical order of their Kayobe network name by default. See story 2007259 for details.

  • Fixes an issue seen when libselinux-python is not installed on the Ansible control host. See story 2007703 for details.

  • Improves error message seen when discovering SSH known hosts for a host without an IP address defined in ${KAYOBE_CONFIG_PATH}/network-allocation.yml.

  • Fixes an issue where OS_CACERT variable in openrc files would be set to the path of a non-existent file. New openrc files can be generated with the kayobe control host bootstrap command. See story 2007516 for more details.

  • Fixes an issue where host configuration would fail if ntp_service_enabled is set to false or kolla_enable_chrony is set to true. See story 2007384 for details.



This Kayobe release is the first as part of an official OpenStack project. Kayobe is now a deliverable of the Kolla project.

New Features

  • Adds support for custom Elasticsearch configuration.

  • Adds support for passing through group vars to Kolla Ansible.

  • Adds support for configuration of Arista switches running EOS 4.15 or later. This is integrated with the kayobe physical network configure command.

  • Adds new variables, kolla_bifrost_dib_elements_extra and kolla_bifrost_dib_env_vars_extra, in ${KAYOBE_CONFIG_PATH}/bifrost.yml for setting additional diskimage-builder elements and environment variables for the disk image generated by Bifrost. Default elements and environment variables are now respectively defined by kolla_bifrost_dib_elements_default and kolla_bifrost_dib_env_vars_default.

  • Added the KAYOBE_DATA_FILES_PATH environment variable to override the auto-detection of the data files path. See the documentation for more details.

  • Adds support for encryption of internal API traffic. This can be done via the following variables:

    • kolla_enable_tls_internal

    • kolla_internal_tls_cert

    • kolla_internal_fqdn_cacert

  • Adds support for installing packages in locally built Ironic Python Agent (IPA) images via the ipa_build_dib_packages variable. See story 2006855 for details.

  • Adds a new configuration variable, pip_upper_constraints_file, which is used to configure the file or URL containing the python upper version contraints. Its default value is{{ openstack_release }}.

  • Adds a new kayobe overcloud service deploy containers command. This is similar to kayobe overcloud service deploy, but only deploys new containers if necessary, and skips service registration, bootstrapping and configuration.

Upgrade Notes

  • Updates the minimum supported version of Ansible from 2.5 to 2.6, and the maximum supported version from 2.7 to 2.8. This is true for both Kayobe and Kolla Ansible.

  • It is now possible to set additional diskimage-builder elements and environment variables for the disk image generated by Bifrost, while keeping defaults set by Kayobe. If your configuration overrides kolla_bifrost_dib_elements or kolla_bifrost_dib_env_vars while still including default values set in ansible/group_vars/all/bifrost, you can move your additions to kolla_bifrost_dib_elements_extra and kolla_bifrost_dib_env_vars_extra instead.

  • The default IPA images now use CentOS as a base image. This is because CoreOS based images are no longer supported or built by the upstream Ironic Python Agent project as of the Train release. Please see the Ironic Python Agent release notes for more details.

  • Uses the new Kolla Ansible variable docker_custom_config to populate Docker’s daemon.json configuration file.

  • The database backup support in Kolla Ansible has been modified to use Mariabackup rather than Xtrabackup.

  • Removes transitional support for stopping the chrony container when kolla_enable_chrony is false. See story 2006949 for details.

  • The default value of kolla_upper_constraints_file has been changed to {{ pip_upper_constraints_file }}.

Deprecation Notes

  • The variable kolla_tls_cert has been deprecated in favour of kolla_external_tls_cert. Support for using kolla_tls_cert will be removed in a future release.

  • The kolla_enable_xtrabackup variable is deprecated in favour of kolla_enable_mariabackup.

Bug Fixes

  • Fixes an issue with kayobe overcloud post configure when Ironic is disabled, but ironic_serial_console_autoenable is set to true. See story 2006662 for details.

  • Fixes the detection of kayobe’s data files on Ubuntu when not installing into a virtualenv, see story 2005510 for details.

  • Fixes an issue when specifying multiple regular expressions to the kayobe seed container image build and kayobe overcloud container image build commands. See story 2006475 for details.

  • Fixes an issue with hardware inspection of bare metal compute nodes configured to use UEFI. See story 2006214 for details.

  • Fixes an issue with building IPA images when no additional packages are specified. See story 2007069 for details.

  • Fixes an issue where kayobe overcloud post configure would use the public OpenStack API interface, which might not be accessible from the control hosts. See story 2006814 for details.

  • Fixes an issue with provisioning the seed VM using the stackhpc.libvirt-vm role at version 1.8.0. See story 2007063 for details.

  • Modifies provisioning and cleaning networks in multi-tenant ironic environments to be non-shared. Flat networks remain shared. To apply the change to an existing environment, run kayobe overcloud post configure. See story 2006409 for details.

  • Fixes an issue with virtual environments on remote hosts, which may over the course of time become stale and incompatible with Kayobe or other software. This was fixed by installing the latest version of packages allowed by OpenStack upper constraints. See story 2005923 for details.

  • Fixes an issue seen during overcloud provisioning due to Bifrost renaming the IPA kernel file. See story 2007068 for details.