Zed Series Release Notes

13.6.0-13

New Features

• Adds support for custom RabbitMQ configuration.

Bug Fixes

• Fixes an issue where Dell OS6 and Dell OS9 switch configuration was not applied correctly. LP#2061102.

• Fixes default Ubuntu Apt keyrings location to the recommended /etc/apt/keyrings.

• Fixes an issue with overcloud service destroy where it failed to remove the inspection store docker volume. See LP#2050092.

• Changes the default cloud image for seed and infra VMs to use Rocky Linux 9.3, to fix boot failures seen with newer cloud images which require UEFI boot mode.

13.6.0

Bug Fixes

• Fixes the wipe-disks role which was failing on supported host operating systems due to a change in the output format of lsblk -J in util-linux version 2.37. LP#2051859

13.5.0

Deprecation Notes

• Support for the devicemapper Docker storage driver is deprecated following its removal from Docker Engine 25.0. Support will be fully removed in the Caracal 16.0.0 release. Operators using devicemapper should ensure that a compatible version of Docker Engine is installed (i.e. release 24.x or below).

13.4.0

Upgrade Notes

• If the admin network does not have a gateway defined and seed_enable_snat is false, which is the default, overcloud hosts will not have a default gateway immediately after provisioning anymore. A default gateway on another network can still be applied during the host configuration step.

• Introduces a new variable kolla_ansible_extra_custom_passwords to avoid the need to combine kolla_ansible_default_custom_passwords and kolla_ansible_custom_passwords when adding or overriding passwords.

Bug Fixes

• Fixes an issue where local configuration generation would be skipped when running in check mode. This would lead to Kolla Ansible checking with stale configuration. See story 2010526 for details.

• Fixes an issue where kayobe configuration dump would fail when variables are encrypted using Ansible Vault. Encrypted variables are now sanitised in the dump output. LP#2031390

• Fixes slow fact gathering in some environments by not configuring the seed host as the initial default gateway for overcloud hosts when seed_enable_snat is false, which is the default. LP#2039461

• Fixes an issue where the Kolla Ansible variable kolla_admin_openrc_cacert was not set to the value of kolla_internal_fqdn_cacert.

• Disables configuration of SELinux by Kolla Ansible, which could revert configuration set by Kayobe.

• Fixes gateway assignment when seed SNAT is disabled. In this circumstance Bifrost was generating ConfigDrive data with the default gateway unset even when one is available on the admin network.

• Fixes a bug where NetworkManager would overwrite resolv.conf when resolv_is_managed is set to True. LP#2044537

• When determining whether or not a host needs bootstrapping, we attempt to connect to the host using ansible_user, if the login fails, we then assume that the host needs bootstrapping. In previous releases we used a manually crafted ssh command. This did not respect any customisations to the SSH arguments made through Ansible configuration. We now use the raw module so that these customisations are used when connecting to the host. One possible use case is to configure a jump host between the control host and the target hosts. If bootstrapping was needed, hosts will now show as unreachable in the summary stats at the end of the run. This can safely be ignored.

• Fixes an issue when user forgot to combine kolla_ansible_custom_passwords, kolla_ansible_default_custom_passwords and own dictionary with custom passwords in configuration files. Now kolla_ansible_extra_custom_passwords should provide only user custom passwords to add or override in kolla/passwords.yml.

13.3.0

New Features

• The Spanning Tree Protocol (STP) can now be configured on bridge interfaces. Enable or disable STP by setting the bridge_stp attribute for a network. Note that STP is not set by default on Ubuntu, but it is disabled on Rocky Linux 9 for compatibility with network scripts, as NetworkManager enables STP on all bridges by default.

• Attempts to log in to the kolla docker registry can be skipped by setting deploy_containers_registry_attempt_login to false.

  This is required for deployments using a non-standard registry deployed on the seed during the deploy-container step, since it takes place after the registry login attempt.

Upgrade Notes

• For Rocky Linux 9, Kayobe now disables STP on a bridge by default. This action will cause the bridge interface to restart during the host configuration process.

• Adds an introspection rule to update the location of the deployment kernel registered in existing Ironic nodes. Nodes discovered on a deployment running the Train release or earlier may still be using the ipa.vmlinuz kernel, which stays unchanged when deployment images get updated. If only default introspection rules are in use, existing nodes may be updated from the Bifrost container with the following command:

  OS_CLOUD=bifrost baremetal introspection reprocess $NODE_UUID_OR_NAME

  If non-default rules are used, reprocessing may revert any customisation done by the operator. In this case, a more cautious approach is to update the deployment kernel location manually:

  OS_CLOUD=bifrost baremetal node set --driver-info deploy_kernel=<http://url/to/ipa.kernel> $NODE_UUID_OR_NAME

  If the kolla_bifrost_inspector_rules list is customised, the rule inspector_rule_legacy_deploy_kernel should be added to it.

Bug Fixes

• Fixes failure to run kayobe overcloud deprovision after Bifrost is redeployed. LP#2038889

• Improves performance of Bifrost operations by preventing unnecessary requests to the Ironic API.

• Fixes detection of data file path when using editable installations with a recent pip.

• Fixes the regression in configuring additional route options on CentOS / Rocky.

• Fixed issue of seed containers being unable to use password protected registry by adding docker login function to kayobe deploy-containers role.

• Adds a workaround to avoid NetworkManager setting the MTU of bridge VLAN interfaces to an incorrect value. LP#2039947

• Fixes conflicts between NetworkManager nmconnection files generated by cloud-init and those generated by Kayobe by upgrading the MichaelRigart.interfaces role to version 1.14.4. LP#2039975

13.2.0

New Features

• Adds support for custom Multipathd configuration.

• Since Kolla containers can built with user provided repos.yaml Kayobe can override the file with their own content. The override files can be ${KAYOBE_CONFIG_PATH}/kolla/repos.yaml (default Kolla filename) or ${KAYOBE_CONFIG_PATH}/kolla/repos.yml. Multiple Environments supported.

Upgrade Notes

• Modifies the default value of kolla_ansible_venv_python to /usr/bin/python3. Using operating system python to create kolla-ansible venv fixes corner cases when using older venvs created with virtualenv command.

Bug Fixes

• Fixes download of roles from Ansible Galaxy following the renaming of the mrlesmithjr.manage_lvm role. LP#2023502

• Fixes an issue where generation of passwords.yml for Kolla Ansible could fail if the directory containing the file does not exist. This is typical in a multiple environment setup, when creating a new environment. See story 2010293 for details.

• Fixes an issue with systemd-networkd configuration on Ubuntu with multiple VLAN interfaces. See story 2009013 for details.

• Fixes repositories files names in Rocky Linux 9. Distributions moved to lowercase names with RHEL 9 release.

• Fixes various issues when applying network configuration on Rocky 9 hosts. See bugs: 2016970 and 2016971.

• Installs ncclient dependency for Juniper switch configuration when using Ansible check mode.

13.1.0

New Features

• Adds support for configuring arbitrarily named VLAN interfaces using systemd-networkd. See story 2010266 for details.

Bug Fixes

• Synchronises the default value kolla_tag with the container image tagging scheme expected by Kolla Ansible. This ensures images are built with tags such as zed-ubuntu-jammy instead of zed.

13.0.0

Prelude

Ubuntu Jammy Jellyfish (22.04) LTS and Rocky Linux 9 are now supported as a host Operating System and base container image.

New Features

• Adds the --skip-hooks argument to ignore hooks for the execution of a command. See story 2009241 for details.

• Adds support for configuring a firewall via firewalld on Ubuntu. See story 2010160 for details.

• Adds support for configuring Dell OS10 Switches using the dellemc.os10 Ansible collection. This is integrated with the kayobe physical network configure command.

• Adds support for installing additional build host dependencies when building IPA and overcloud host images via ipa_build_dib_host_packages_extra and overcloud_dib_host_packages_extra.

• Adds support for specifying a custom playbook when running Kolla Ansible commands via a --kolla-playbook argument. For example:

  kayobe overcloud service deploy --kolla-playbook /path/to/playbook.yml
  

  This may be used to specify a playbook that replaces or extends the default site.yml playbook, and needs to execute in the Kolla Ansible context.

• Adds support for copying $KAYOBE_CONFIG_PATH/kolla/config/nova_compute to Kolla configuration. This folder can contain a Nova release file which can configure the vendor or product strings used by Nova.

• Roles, collections and plugins included with Kayobe configuration are now accessible to all Kayobe playbook executions.

• Adds functionality to configure desired SELinux state (in addition to disabling SELinux previously).

• Adds support for Rocky Linux 9 as a host Operating System and base container image. CentOS Stream 8 is not supported anymore.

• Adds support for copying the Bifrost clouds.yaml file and optionally a TLS CA certificate from the Bifrost container to the seed host. This makes it possible to enable authentication and TLS for Bifrost services.

• Kayobe now configures SELinux on the seed hypervisor. The default is to set SELinux to permissive.

• Adds support for specifying SNAT source and destination filters. This is useful if forwarded packets need to exit on a different interface depending on the source or destination IP address or port.

• Adds the --add-known-hosts option to control host bootstrap. This will add SSH known hosts entries for each host. This should provide a way around the issues described in story 2001670.

• Adds support for the ANSIBLE_VAULT_PASSWORD_FILE environment variable as a source for the Ansible Vault password. See story 2006766 for details.

• Adds support for configuring swap files and devices on seed, seed hypervisor, overcloud and infra VM hosts during host configure commands.

• Adds support for Ubuntu Jammy Jellyfish (22.04) LTS as a host and container Operating System for seed, seed hypervisor and overcloud hosts.

Upgrade Notes

• Updates the maximum supported version of Ansible from 5.x (ansible-core 2.12) to 6.x (ansible-core 2.13). The minimum supported version is updated from 4.x to 5.x. This is true for both Kayobe and Kolla Ansible.

• Changes the Kayobe playbook group variables in ansible/group_vars/ to be inventory group variables in ansible/inventory/group_vars. This has two important consequences:

  1. Inventory group variables have a lower precedence than playbook group variables. This means that these variables can now be overridden by group variables in the Kayobe configuration inventory.

  2. The new inventory group variables are automatically used by all Kayobe commands, and do not need to be in the same directory as the playbook being executed. This means that the previous workaround for custom playbooks involving symlinking to the group_vars directory from the directory containing the custom playbook is no longer necessary.

• Removes the kolla_ironic_default_boot_option variable and the inspector_rule_local_boot inspector rule, since Ironic has removed support for defining a boot option configuration. The Set local boot capability rule should be removed from Bifrost and Ironic Inspector by the operator.

• Starting with Yoga, Ironic has changed the default PXE from plain PXE to iPXE. Kayobe follows this upstream decision but allows users to revert to the previous default of plain PXE. For details, please refer to Kolla Ansible’s documentation.

• Removes the kolla_install_type variable. This is due to removal of support for binary images from the Kolla project.

• Overcloud host images are now built via DIB by default, rather than Bifrost. The old behaviour may be obtained by setting overcloud_dib_build_host_images to false.

• Removes support for configuring Grafana with kayobe overcloud post configure. See the Kolla Ansible documentation for an alternative method of loading dashboards.

• Changes the environment used during Kayobe playbook execution to include Kayobe’s collections, roles and plugins in the Ansible lookup paths. This allows custom playbooks to use these items, without the requirement to symlink into the Kayobe installation. Existing symlinks may be removed.

• Removes support for deploying Monasca and its dependencies (Kafka, Storm and Zookeeper).

• The disable-selinux role has been renamed to selinux and so have been the related variables. If you set one of them, adapt your configuration:

  • disable_selinux_do_reboot becomes selinux_do_reboot

  • disable_selinux_reboot_timeout becomes selinux_reboot_timeout

• Kayobe now sets SELinux to permissive by default (compared to disabled previously). This may require a reboot, which will only be triggered if selinux_do_reboot is set to true. If you want to retain previous behaviour, set selinux_state to disabled.

• Elasticsearch has been replaced with OpenSearch. Any custom Kayobe configuration should be moved from ${KAYOBE_CONFIG_PATH}/kolla/config/elasticsearch to ${KAYOBE_CONFIG_PATH}/kolla/config/opensearch.

• The default value of os_distribution was changed to rocky. CentOS Stream 8 is not supported anymore.

• Enables authentication by default in Bifrost.

• Updates the stackhpc.os-images role to version 0.16.0. This new release separates configuration of upper constraints for diskimage-builder (DIB) from those used by the OpenStack SDK and client. This allows operators to use a newer version of DIB while keeping compatible versions of the OpenStack SDK and client. This is configured with the following variables:

  • ipa_build_dib_upper_constraints_file in ipa.yml

  • overcloud_dib_dib_upper_constraints_file in overcloud-dib.yml

  The variables are empty by default in order to allow for Rocky Linux 9 image builds.

Security Issues

• Fixes an issue where any passwords in kolla_ansible_custom_passwords were exposed in Ansible logs. When using verbosity level 3 (-vvv), they were also exposed in Ansible output.

Bug Fixes

• Ironic inspection through Bifrost now work even if DHCP-relay is used. The dhcp-range in dnsmasq.conf is now correctly configured with its network mask.

• Adds missing Ansible group following the addition of support in Kolla Ansible for forwarding Prometheus alerts to Microsoft Teams.

• Fixes an issue with undefined kolla_enable_hacluster variable.

• Fixes an issue where a host configure with --wipe-disks would wipe block devices that were mounted. See story 2010367 for details.

• Fixes an error when generating passwords.yml if an unencrypted file exists but a password has been supplied.

• Fixes an issue where hacluster images are not built when the service is enabled.

• Fixes an issue where a custom playbook using become_user could fail when setting permissions on temporary files. The acl package is now installed on all systems by default.

• Fixes an issue where any passwords in kolla_ansible_custom_passwords were exposed in Ansible logs. When using verbosity level 3 (-vvv), they were also exposed in Ansible output.

• Fixes an issue with nclu-switch command ordering, when description was applied first to a non-existent (virtual) interface. See story 2010279 for details.

• Fixes an issue where the MTU defined in Kayobe was not applied to Ironic provisioning and cleaning networks in Neutron.

• Configures SELinux to permissive on the seed hypervisor, which fixes permission issues when provisioning seed or infra VMs.

• Fixes failures to run kayobe overcloud bios raid configure by upgrading the stackhpc.drac role to version 1.1.6.