Ussuri Series Release Notes


New Features

  • Adds support for inspection of L3-routed Ironic networks via DHCP-relay.

Bug Fixes

  • Fixes a failure to detect the Kayobe installation prefix when lib is present multiple times in the installation path. See story 2009721 for details.


New Features

  • Adds support for configuring the filter and gather_subset arguments for the setup module via kayobe_ansible_setup_filter and kayobe_ansible_setup_gather_subset respectively. These can be used to reduce the number of facts, which can have a significant effect on performance of Ansible.

  • Adds a new command, kayobe overcloud facts gather, to gather Ansible facts for overcloud hosts. This may be useful for populating a fact cache.

  • Adds support for the metalink option in custom DNF repositories configured with dnf_custom_repos in dnf.yml.

Upgrade Notes

  • Updates all references to Ansible facts within Kayobe from using individual fact variables to using the items in the ansible_facts dictionary. This allows users to disable fact variable injection in their Ansible configuration, which may provide some performance improvement. Check for facts referenced in local configuration files, and update to use ansible_facts before disabling fact variable injection.

  • Removes the workaround for bogus name server entries in some CentOS 7 images, together with the overcloud_host_image_workaround_resolv_enabled variable.

Bug Fixes

  • Filter out 25 Gigabit Ethernet interface names in the Ironic inspector rule setting node names from interface LLDP switch port descriptions.


New Features

  • Adds the ipa_build_upper_constraints_file variable to select the upper constraints file used to install diskimage-builder in the virtual environment used for building IPA images. This allows you to install a newer release than the one allowed by the default constraints.

  • Updates the default image for the seed VM to CentOS 8.3.

Upgrade Notes

  • Updates the stackhpc.os-images role to version 1.10.2, for compatibility with the version of diskimage-builder installed by Victoria upper constraints. You must run kayobe control host upgrade to update this role in order to successfully build IPA images.

Bug Fixes

  • Fixes an issue when using the --limit argument with a host pattern including commas. See story 2008255 for details.

  • Fixes an issue with Python setup when venv is not used. See story 2008378 for details.

  • Fixes an issue where custom SSH arguments, such as when setting ansible_ssh_extra_args, were not being respected when rebooting a host to disable selinux.

  • Fixes calls to kolla-ansible when arguments to --kolla-limit contain special characters such as ~ or &.

  • Fixes an issue with copying Swift ring files. See story 2007297 for details.

  • Fixes issue with deleting swift ring temporary files. See story 2008354 for details.

  • Fixes an issue with the kayobe seed service deploy command on CentOS 8.3 release. See story 2008430 for details.

  • Fixes building deployment images (IPA) with recent versions of ironic-python-agent-builder, by using Victoria upper constraints to install a newer version of diskimage-builder into the virtual environment used for building IPA images. This can be changed using the ipa_build_upper_constraints_file variable in ${KAYOBE_CONFIG_PATH}/ipa.yml.


New Features

  • Updates the default image for the seed VM to CentOS 8.2.

Upgrade Notes

  • Kayobe configures Bifrost to use the trusted zone of firewalld, ensuring that all services running on the seed host are accessible. Deployments with stricter firewall policies can select another zone by setting the kolla_bifrost_firewalld_internal_zone variable in ${KAYOBE_CONFIG_PATH}/bifrost.yml. To avoid loss of connectivity to the seed host, ensure that firewalld is already configured on the seed host before deploying seed services.

Bug Fixes

  • Adds support for a custom Barbican configuration file (barbican.conf), as only extended configuration stored under a barbican folder was supported.

  • Fixes loss of connectivity to the seed host after deploying seed services, when using a shared provisioning and admin network. This was caused by Bifrost configuring firewalld to only allow Ironic traffic. Kayobe now configures Bifrost to use the trusted zone, which allows all traffic.

  • Fix an issue when dnf proxy is set and python3-pip package fails during host configure.

  • Fixes a number of issues with using Kayobe on CentOS 8.3.

  • Fixes issues when network interfaces are configured without IP addresses. See story 2007900 for details.

  • Fixes generation of pip configuration when using a pip proxy without a local mirror.

  • Changes the value of OS_ENDPOINT_TYPE, OS_MANILA_ENDPOINT_TYPE, and OS_MISTRAL_ENDPOINT_TYPE in to use publicURL, which should fix issues with legacy CLI tools on hosts without access to the internal API. See story 2007950 for details.

  • Removes the ifcfg-eth0 network interface configuration file when not required, using a new version of the MichaelRigaert.interfaces Ansible role. This file ships with CentOS 8 cloud images and can prevent the network service from starting successfully when no eth0 interface is present or being configured. See story 2007913 for details.

  • Fix the implementation of the kayobe seed hypervisor host package update command, which was incomplete. See story 2008458 for details.

  • Fixes an issue with seed service deployment where modifying the overcloud image fails on a CentOS 8.1 host with a CentOS 8.2 bifrost_deploy container, by updating the default image for the seed VM to CentOS 8.2. See story 2007942 for details.



The Kayobe 8.0.0 release is the first release in the Ussuri cycle. Notable changes include:

  • all playbooks and scripts now use Python 3 and support for Python 2 has been dropped

  • CentOS 8 is now supported as a host operating system and container image, and support for CentOS 7 has been dropped

  • supports running custom playbooks before and after a Kayobe command

  • Ceph deployment support has been dropped

New Features

  • Adds an experimental mechanism to automatically run custom playbooks before and after kayobe commands. Please see the Custom Ansible Playbooks section in the documentation for more details.

  • Provides project and domain default variables for Monasca. Defaults can be overridden from the Monasca configuration file ${KAYOBE_CONFIG_PATH}/monasca.yml.

  • Adds a kayobe overcloud service stop command. This can be used to stop containerised services running on overcloud hosts.

  • Adds the variable, docker_registry_datadir_volume. This allows you to configure the name or path of the docker volume used for the docker registry.

  • Adds support for CentOS 8 as a host Operating System and base container image. This is the only major version of CentOS supported from the Ussuri release. The Train release supports both CentOS 7 and 8 hosts, and provides a route for migration.

  • Adds support for providing custom Ansible configuration files via Kayobe configuration. For Kayobe the file should be located at ${KAYOBE_CONFIG_PATH}/ansible.cfg. For Kolla Ansible, it may be located either at ${KAYOBE_CONFIG_PATH}/kolla/ansible.cfg or ${KAYOBE_CONFIG_PATH}/ansible.cfg. A file specified via the ANSIBLE_CONFIG environment variable overrides these.

  • Adds support for passing custom TLS certificates to Kolla Ansible via ${KAYOBE_CONFIG_PATH}/kolla/certificates/. See story 2007679 for details.

  • New feature to deploy user-defined containers on seed node with pre and post scripts has been added to Kayobe.

  • Adds support for configuration of DNF repositories on CentOS 8. Variables have been added in a new configuration file, dnf.yml. Backwards compatibility with the Yum configuration variables is provided.

  • Adds support for applying regular package updates on CentOS 8 via DNF Automatic. Variables have been added in a new configuration file, dnf.yml. Backwards compatibility with the Yum-cron configuration variables is provided.

  • Adds a seed_vm_interfaces variable which defines the network interfaces to which the seed VM is attached.

  • Adds a new variable, kolla_enable_openstack_core, which can be set a default value for whether the default OpenStack services are enabled. This includes Glance, Heat, Horizon, Ironic, Keystone, Neutron and Nova. It is true by default.

  • Adds a variable for controlling the tag applied to built container images - kolla_tag. This separates the configuration of the tag for image building from that used for deployment (kolla_openstack_release). The default for kolla_tag is kolla_openstack_release.

  • Adds support for configuring encrypted block devices using dm-crypt. Encryption is applied before LVM configuration and after software raid, which allows creating LVM volumes on top of encrypted block devices. See story 2007555 for details.

  • Introduces a new option - pip_proxy - to configure Pip package installation via a user-defined http(s) proxy. This is set on a per-user basis, and by default this is for the same users as pip_local_mirror feature.

  • Adds support for plugging the Open vSwitch provider bridge directly into a an Ethernet interface. Previously it was necessary to define a Linux bridge, into which Kayobe would plug a virtual Ethernet pair. The use of a direct connection may provide improved performance, or allow additional hardware offloading. See story 2007364 for details.

Known Issues

  • Fixes an issue where the default inspector rule setting node names from LLDP switch port descriptions would fail to filter out interface names on Ruckus switches.

  • Fixes an issue where provisioning a seed VM would fail when the Ansible control host and the seed hypervisor are different hosts. See story 2007530 for more details.

Upgrade Notes

  • Updates the minimum supported version of Ansible from 2.6 to 2.8, and the maximum supported version from 2.8 to 2.9. This is true for both Kayobe and Kolla Ansible.

  • Avoids unnecessary fact gathering using the setup module. This should improve the performance of environments using fact caching and the Ansible smart fact gathering policy. See story 2007492 for details.

  • The kolla-ansible bootstrap-servers command is used by Kayobe during the kayobe seed host configure and kayobe overcloud host configure tasks. In previous releases it was executed as the Kayobe Ansible user (kayobe_ansible_user) and using the remote Kayobe Python interpreter (ansible_python_interpreter) since it was responsible for creation of the Kolla Ansible user account (kolla_ansible_user) and Python virtual environment (kolla_ansible_target_venv). This mix of environments causes problems for Ansible fact caching. To avoid this issue, Kayobe is now responsible for creation of the Kolla Ansible user and Python virtual environment, and kolla-ansible bootstrap-servers is run using the normal Kolla Ansible user and remote Python interpreter.

    Previously it was possible to avoid creation of the user account during kolla-ansible bootstrap-servers by setting create_kolla_user to false in ${KAYOBE_CONFIG_PATH}/kolla/globals.yml. The same may now be achieved by setting kolla_ansible_create_user to false in ${KAYOBE_CONFIG_PATH}/kolla.yml.

  • CentOS 7 is no longer supported as a host Operating System or base container image. CentOS users should migrate to CentOS 8. The Train release supports both CentOS 7 and 8 images, and provides a route for migration.

  • Some images were supported by CentOS 7 but lack suitable packages in CentOS 8, and are no longer supported for CentOS. See Kolla release notes for details.

  • Support for configuring an NTP daemon on the seed, seed hypervisor and overcloud hosts is no longer present, as appropriate packages are not available for CentOS 8. Instead, Kolla Ansible is configured to deploy the chrony container on overcloud hosts by default. This may be disabled by setting kolla_enable_chrony to false. There is no support for running a chrony container on the seed or seed hypervisor hosts.

  • Reverts to use the Kolla Ansible default value for kolla_external_fqdn_cert and kolla_internal_fqdn_cert when kolla_external_tls_cert and kolla_internal_tls_cert are respectively not set. This allows for the standard Kolla Ansible configuration approach of dropping these certificates into the $KAYOBE_CONFIG_PATH/kolla/certificates directory, rather than defining them as variables. This can be useful if using the kolla-ansible certificates command to generate certificates for testing.

  • The default LVM configuration is now empty for all hosts, unless they are configured to use the devicemapper Docker storage driver (which is the default, but is expected to change in a future release).

    Note that while the default LVM configuration existed primarily for the devicemapper driver, it also included a docker-volumes logical volume mounted at /var/lib/docker/volumes for Docker volumes. If the docker-volumes volume is required on a host which is not configured to use the Docker devicemapper storage driver, the following variables may be used to enable it: compute_lvm_group_data_enabled, controller_lvm_group_data_enabled, seed_lvm_group_data_enabled, storage_lvm_group_data_enabled.

  • Drops support for Kolla Ceph deployment. Kayobe follows upstream decision of Kolla and Kolla Ansible. Please use other means of Ceph deployment. Please note Ceph backends will still work if using external Ceph cluster.

  • Python 2.7 support has been dropped. Last release of kayobe to support py2.7 is OpenStack Train. The minimum version of Python now supported by kayobe is Python 3.6.

  • The default order of network interfaces in the seed VM is now sorted alphabetically based on their Kayobe network name. This may require the seed’s network interface names to be changed in configuration if the seed VM is recreated. See story 2007259 for details.

  • Changes Ironic Python Agent (IPA) image builds to use IPA builder. The following variables have been added to configure the IPA builder source location: ipa_builder_source_url, ipa_builder_source_version. See story 2007070 for details.

  • Adds new variables to allow extension of the list of Disk Image Builder (DIB) git repositories available when building Ironic Python Agent (IPA) images: ipa_build_dib_git_elements_default and ipa_build_dib_git_elements_extra. The existing ipa_build_dib_git_elements variable is a concatenation of these. By default there is now one git repository configured for IPA builder, and use of ipa_build_dib_git_elements should typically be replaced with ipa_build_dib_git_elements_extra to avoid losing this default.

  • The file extension used for the Ironic Python Agent (IPA) kernel image has changed from vmlinuz to kernel, in line with upstream changes in Ironic. This affects both Bifrost running on the seed and Ironic running in the overcloud for bare metal compute. If building images locally, this should be done prior to upgrading services.

  • Modifies the default value of kolla_ansible_become to false. This means that Kolla Ansible will no longer use privilege escalation for all tasks, and will only use it where necessary.

  • Removes the workaround for using a tagged VLAN as the admin network interface on overcloud hosts. This was necessary for CentOS 7.5 which shipped an old version of cloud-init. The variables overcloud_host_image_workaround_cloud_init_enabled and overcloud_host_image_workaround_cloud_init_repo have been removed.

  • Timezone configuration has been moved from the ntp.yml configuration file to time.yml. The Ansible tag applied to the play has been renamed from ntp to timezone.

Deprecation Notes

  • The Yum configuration variables in yum.yml are deprecated and will be removed in a future release. Adapt any configuration overrides to use the new DNF variables in dnf.yml instead.

  • The yum-cron configuration variables in yum-cron.yml are deprecated and will be removed in a future release. Adapt any configuration overrides to use the new DNF automatic variables in dnf.yml.

Bug Fixes

  • Fixes the Monasca install type causing issues pulling container images. See story 2007597 for details.

  • Blacklists Ansible 2.9.8 to avoid an issue with the Ansible fileglob plugin. See story 2007659.

  • Fixes an issue where it was not possible to load dashboards into the Monasca Grafana fork when the default Monasca control plane OpenStack project name is used from Kolla Ansible.

  • Fix an issue where the StackHPC iDRAC role would break when configuring RAID when used with a recent release of the python-dracclient module.

  • Fixes concurrency issues while adding SSH keys to the known hosts file by performing the action serially. See story 2007628 for details.

  • Fixes an issue when configuring the external API network interface on controllers without an IP address which would cause kayobe overcloud host configure to fail on a host with an active virtual IP address. See story 2007736 for details.

  • Fixes issues running the following commands:

    • kayobe baremetal compute inspect

    • kayobe baremetal compute manage

    • kayobe baremetal compute provide

    See story 2007797 for details.

  • Fixes a package conflict while provisioning a seed VM on a CentOS 8 seed hypervisor with coreutils-single already installed. See story 2007612 for details.

  • Fixes failure to configure Docker devicemapper storage when the default value of docker_storage_driver is used.

  • Fixes an issue where the default value of public_net_name included a trailing newline. See story 2007654.

  • Fixes an issue where disabling SELinux would fail on systems without SELinux installed. See story 2007704.

  • Fixes an issue with idempotency of Ironic Inspector rule creation. See story 2007399 for details.

  • Fixes a bug where introspection data save would fail. See Story 2007326 for more details.

  • Fixes an issue with building IPA images when no additional packages are specified. See story 2007069 for details.

  • Fixes an issue with seed VMs with multiple network interfaces where interfaces could come up in a different order if the VM is recreated. The interfaces are now created in alphabetical order of their Kayobe network name by default. See story 2007259 for details.

  • Fixes an issue with provisioning the seed VM using the stackhpc.libvirt-vm role at version 1.8.0. See story 2007063 for details.

  • Improves error message seen when discovering SSH known hosts for a host without an IP address defined in ${KAYOBE_CONFIG_PATH}/network-allocation.yml.

  • Kayobe now updates openrc files when using the kayobe overcloud service upgrade command.

  • Fixes an issue where OS_CACERT variable in openrc files would be set to the path of a non-existent file. New openrc files can be generated with the kayobe control host bootstrap command. See story 2007516 for more details.

  • Removes the seed_hypervisor_libvirt_pool_capacity variable, which was passed to Libvirt through the libvirt-host role when creating a storage pool. This information is actually ignored by Libvirt, which gathers capacity and other metadata from the storage backend. See story 2007381 for details.

  • Fixes an issue seen during overcloud provisioning due to Bifrost renaming the IPA kernel file. See story 2007068 for details.