Yoga Series Release Notes

14.6.0-7

Bug Fixes

  • Fixes an issue with ironic-inspector using the wrong option to configure the interface used to communicate with the Ironic API. LP#1995246

14.6.0

Upgrade Notes

  • image_upload_use_cinder_backend = True is no longer set on the Cinder’s default Ceph RBD backend, the common upstream default is now used (False currently). See also LP#1991516

Bug Fixes

  • image_upload_use_cinder_backend = True is no longer set on the Cinder’s default Ceph RBD backend. Related ERRORs and WARNINGs in Cinder and Glance logs are prevented. LP#1991516

  • Fixes Keystone OIDC failing to validate JWT because of missing key on Azure auth-oidc endpoint. Adds new variable containing JWKS uri that delivers missing keys. LP#1990375

  • Fixes Ironic API healthchecks when backend TLS encryption is enabled. LP#1990819

  • Removes the dhcp-sequential-ip configuration option from ironic_dnsmasq to avoid a race condition offering the same IP address to multiple hosts being inspected at the same time.

  • Fixes an issue with ironic-neutron-agent using the wrong option to configure the interface used to communicate with the Ironic API. LP#1990675

14.5.0

New Features

  • Adds support for deploying prometheus-msteams, which can be used to forward Prometheus Alertmanager notifications to Microsoft Teams. It is enabled by setting enable_prometheus_msteams to true.

Bug Fixes

  • Fixes an issue with AlertManager external Web URL being unconfigurable. A new variable prometheus_alertmanager_external_url has been introduced that users can use to set web.external-url to public.

  • Under circumstances of extended disruption to the Fluentd-ElasticSearch central logging pipeline, it is possible to generate a sufficient buffer of unsent log data that takes longer than the default Fluentd request timeout (default 5 seconds) to transfer the buffer. The default request timeout value is raised to 60s, and made configurable using new parameter fluentd_elasticsearch_request_timeout. LP#1983031

14.4.0

Security Issues

  • Kolla Ansible used to run Ironic’s tftpd as an (unprivileged) root user. Now, it will explicitly use the nobody user.

Bug Fixes

  • Fixes 1982777. Set multipathd user_friendly_names to “no” to make os-brick able to resize volumes online. Adds ability to override multipathd config. LP#1982777

  • Fixed bug #1987982 This bug caused the database log_bin_trust_function_creators variable not to be set back to “OFF” after a keystone upgrade.

  • If ironic_enabled_notification_topics is set to true, ironic_notification_level is set to info in order to ensure that Ironic actually sends out notifications.

    See bug 1969826 for details.

14.3.0

New Features

  • Adds variables to configure whether monitoring services should be exposed externally:

    • enable_grafana_external

    • enable_kibana_external

    • enable_prometheus_alertmanager_external

Bug Fixes

  • Fixes an issue where Ironic Inspector could be configured without authentication in a multi-region environment in a region without a local Keystone service.

14.2.0

New Features

  • Adds support for configuring the Openstack Compute API microversion used by the OpenStack exporter for Prometheus using the prometheus_openstack_exporter_compute_api_version variable. The default value is 2.1 to keep metrics unchanged when using recent exporter releases.

Bug Fixes

  • Fixes the issue of exponential growth of /run/openvswitch mounts when kolla-toolbox container is restarted. LP#1979295

  • Fixes an issue with recovering multi-node MariaDB Galera cluster.

  • Increases prometheus_openstack_exporter_timeout to 45 seconds to reduce the odds of scrape failures on deployments with large number of OpenStack resources. LP#1976629

14.1.0

New Features

  • New switches added to control deployment of the Masakari monitors. The deployment of each type of monitors can be controlled individually via enable_masakari_instancemonitor and enable_masakari_hostmonitor. By default, both are set to true when the deployment of the Masakari is enabled via enable_masakari.

Bug Fixes

  • Fixes an issue with Masakari instance monitor when libvirt SASL is enabled. libvirt SASL was enabled by default in a recent change to Kolla Ansible. LP#1965754

  • The prometheus openstack exporters are now behind haproxy, providing a unique time series in the prometheus database. Also ensures that only one exporter queries the openstack APIs at any given time interval. With the previous behavior each openstack exporter was scraped at the same time. This caused each exporter to query the openstack APIs simultaneously introducing unneccesary load and duplicate time series in the prometheus database due to the instance label being unique for each exporter. LP#1972818

14.0.0

New Features

  • Adds support for the VMware NSX Policy plugin

  • Adds support for openEuler 20.03-LTS-SP2 as a host OS distribution.

  • Deploys and configures a prometheus-libvirt-exporter image as part of the Prometheus monitoring stack.

  • Adds support for the VMware FCD as Cinder volumes.

  • Adds a tls_connect module to the Prometheus blackbox exporter. This can be used to test connectivity of TLS servers.

  • Adds the ability to use Prometheus as the metrics database for Ceilometer.

    Adapts Ceilometer configurations so metrics can be pushed to a Prometheus Pushgateway. LP#1964135

  • Adds new variables to be used by the common role, cron_logrotate_log_minsize and cron_logrotate_log_maxsize. They allow to configure global logrotate’s minsize and maxsize options.

  • Allow to disable Designate Sink service (and notifications to/from it) by setting designate_enable_notifications_sink to no.

  • Introduce nova_enable_external_metadata that defaults to no to control if external facing metadata haproxy frontend should be configured.

  • With this release, kolla-ansible no longer creates admin endpoints for any service other than Keystone. Make sure that you only reference public or internal endpoints in your applications and configurations.

  • Allows the use of variables in ceph configuration and keyring files. This includes but is not limited to ansible lookup expressions. LP#1959565

  • Implements the HAProxy Admin Socket. Allows operators to set the flag haproxy_socket_level_admin (default: “no”) which adds level admin to socket that gets created at /var/lib/kolla/haproxy/haproxy.sock inside the HAProxy container. This allows operators to interact with HAProxy, including but not limited to disabling backend servers for controlled maintenance operations. bug 1960215.

  • horizon deployment now supports custom themes.

  • Implements container healthchecks for ironic-neutron-agent service. See blueprint

  • Implements container healthchecks for neutron-bgp-dragent service. See blueprint

  • Implements container healthchecks for solum services. See blueprint

  • Implements container healthchecks for storm services. See blueprint

  • Implements container healthchecks for zookeeper services. See blueprint

  • Support for both PXE and iPXE enabled in Ironic at the same time.

  • Adds support for running a libvirt daemon on the host, rather than in a container. This is done by setting enable_nova_libvirt_container to false. Currently this is only supported for fresh deployments without an existing nova_libvirt container.

  • Adds support for libvirt SASL authentication. It is enabled by default. LP#1964013

  • Adds support to the kolla-ansible certificates command for generating certificates for libvirt TLS, when libvirt_tls is true. The same certificate and key are used for the libvirt client and server.

    The certificates use the same root CA as the other generated certificates, and are written to {{ node_custom_config }}/nova/nova-libvirt/, ready to be picked up by nova-libvirt and nova-compute.

  • Adds a new variable to be used by the common role, cron_logrotate_schedule. This allows to configure how often the cron runs for logrotate.

  • Adds an SSH key for Neutron server which can be used for passwordless public key authentication in external systems (e.g. for networking-generic-switch managed switches).

  • Adds a kolla-ansible nova-libvirt-cleanup command, which may be used to clean up the nova_libvirt container. This may be useful if switching to a host libvirt daemon.

  • Keystone OIDC integration now uses memcached for the caching backend if enable_memcached is True. This can be disabled by setting keystone_oidc_enable_memcached to False.

  • Adds functionality to enable HW offload in OpenvSwitch using openvswitch_hw_offload variable.

  • Adds variables to define extra command-line parameters to be passed to Prometheus exporters:

    • prometheus_blackbox_exporter_cmdline_extras

    • prometheus_elasticsearch_exporter_cmdline_extras

    • prometheus_haproxy_exporter_cmdline_extras

    • prometheus_memcached_exporter_cmdline_extras

    • prometheus_mysqld_exporter_cmdline_extras

    • prometheus_node_exporter_cmdline_extras

    • prometheus_openstack_exporter_cmdline_extras

  • Add enable_prometheus_etcd_integration configuration parameter which can be used to configure Prometheus to scrape etcd metrics endpoints. The default value of enable_prometheus_etcd_integration is set to the combined values of enable_prometheus and enable_etcd.

  • Adds “manila_cephfs_filesystem_name” variable to support multi-fs Ceph Pacific+ deloyments.

  • Adds support for Rocky Linux 8 as Host OS.

  • Adds support for configuring a Vendordata file for Nova. This allows users to pass through arbitrary data to instances.

Known Issues

  • Existing fluentd log rotation failed to delete old haproxy, swift, glance-tls-proxy and neutron-tls-proxy logs. These will not be deleted by the new logrotate config and will have to be removed manually.

Upgrade Notes

  • Minimum supported Ansible version is now 4 (ansible-core 2.11) and maximum supported is 5 (ansible-core 2.12).

  • Restores upstream default value for max_allowed_request_size_in_bytes in barbican.conf. It was set to 1000000 bytes instead of the upstream default of 25000 bytes.

  • RabbitMQ’s Prometheus plugin is no longer enabled by default if Prometheus is not deployed. If external Prometheus is used, you need to turn on rabbitmq_enable_prometheus_plugin to get old behaviour.

  • External Nova metadata service is now disabled by default. It can be enabled by setting nova_enable_external_metadata to yes.

  • With this release, kolla-ansible no longer creates admin endpoints for any service other than Keystone. Existing endpoints will not be removed automatically, if you want to clean up your existing cloud, you can use a command like:

    openstack endpoint list --interface admin -f value | \
    awk '!/keystone/ {print $1}' | xargs openstack endpoint delete
    
  • enable_host_ntp variable is dropped per the deprecation process.

  • Support for deploying vmtp has been dropped per the mailing list notice. The vmtp project is no longer buildable, is outside of the OpenStack namespace and looks plain abandoned. See the mailing list notice

  • fluentd_binary and fluentd_version variables are no longer in use as Kolla Ansible supports a single fluentd version across all supported Kolla image flavours.

  • Starting with Yoga, Ironic has changed the default PXE from plain PXE to iPXE. Kolla Ansible follows this upstream decision but allows users to revert to the previous default of plain PXE. For details, please refer to Kolla Ansible’s documentation.

  • The bootloader used to boot Ironic nodes in UEFI boot mode during inspection when iPXE is enabled has been changed from ipxe.efi to snponly.efi. This is in line with the default UEFI iPXE bootloader used in Ironic since the Xena release. The bootloader may be changed via ironic_dnsmasq_uefi_ipxe_boot_file.

  • Ironic volumes related to PXE (TFTP) and iPXE & direct deploy (HTTP) are refactored to share a common parent path at /var/lib/ironic. This is done to support both PXE and iPXE at the same time. Operators doing advanced customisations might need to review the relevant defaults section.

  • Upgrades of Ironic will now wait for nodes in wait states to change their state. This is to improve the user experience by avoiding breaking processes being waited on. This can be disabled by setting ironic_upgrade_skip_wait_check to yes.

  • Ironic containers related to PXE (TFTP) and iPXE & direct deploy (HTTP) are renamed to better reflect their role: ironic_pxe is now ironic_tftp, while ironic_ipxe is now ironic_http. Operators doing advanced customisations might need to review the relevant defaults section. Additionally, their respective host groups have changed analogously: ironic-pxe is now ironic-tftp, and ironic-ipxe is now ironic-http.

  • ironic.conf now sets [pxe]\kernel_append_params instead of [pxe]\pxe_append_params which has been deprecated. Please override the new config option if you are overriding the old one.

  • The addition of libvirt SASL authentication requires a new password in passwords.yml, libvirt_sasl_password. This may be generated using the existing kolla-genpwd and kolla-mergepwd tooling.

  • The addition of libvirt SASL authentication requires both the nova_libvirt and nova_compute containers to be updated simultaneously, using new images with the necessary Cyrus SASL dependencies, as well as configuration containing the SASL credentials.

  • If both Designate and Neutron are enabled, Neutron now uses the subnet_dns_publish_fixed_ip instead of the simpler dns extension in order to support more features in the DNS integration. Override via the neutron_extension_drivers config option if this is not suitable for your deployment.

  • It is no longer possible to override the removal of the Monasca Log Metrics service and it will be removed automatically if it hasn’t already been removed in the Wallaby release. It is up to the operator to remove any associated docker volumes.

  • The policy for classic transient mirrored queues in RabbitMQ has been removed from the RabbitMQ configuration. The policy will be removed automatically during upgrade of the RabbitMQ service.

  • The wsrep-notify.sh script has been removed (following deprecation in Wallaby).

  • update the default value of node_custom_config to {{ node_config }}/config, when specified using –configdir

Deprecation Notes

  • enable_ironic_ipxe is deprecated in favour of ironic_dnsmasq_serve_ipxe which reflects the effect better. enable_ironic_ipxe will be removed in Zed.

  • enable_ironic_pxe_uefi is deprecated and will be removed in Zed. This variable is not documented and results in a broken PXE setup for Ironic Inspector. The recommended way to support EFI/UEFI deployments in Ironic Inspector is to stay with the recommended default of iPXE in Ironic Inspector (see docs on ironic_dnsmasq_serve_ipxe).

  • The storage_interface variable is deprecated and will be removed in the next release as it was causing confusion. The variable only sets the default for swift_storage_interface which we now recommend to set directly instead.

Security Issues

  • Explicitly removes the net.ipv4.ip_forward sysctl from /etc/sysctl.conf on hosts with Neutron L3 Agent. In the absence of another source for this sysctl, it should revert to the default of 0 after the next reboot. This is a follow up to a previous change which stopped setting the sysctl, but leaves existing systems with the original value of 1 set.

    A deployer looking to more aggressively change the value may set neutron_l3_agent_host_ipv4_ip_forward to 0 using a Yoga release of Kolla Ansible. This option will be removed in future. Any deployments still relying on the previous value may set neutron_l3_agent_host_ipv4_ip_forward to 1. LP#1945453

  • Fixes an issue where the default configuration of libvirt did not use authentication for the API exposed over TCP on the internal API network. This allowed anyone with access to the internal API network read-write access to libvirt. While the internal API network is typically trusted, other services on this network generally at least require authentication.

    SASL authentication is now enabled for libvirt by default. Kolla Ansible supports libvirt TLS since the Train release, and this is recommended to provide a higher level of security. LP#1964013

  • Adds mitigation for the Apache Log4j2 Remote Code Execution (RCE) Vulnerability in Elasticsearch - CVE-2021-44228.

Bug Fixes

  • Fixes an issue with an OIDC authentication flow requiring unnecessary action from the user. Redirecting to the target IdP page now happens automatically. LP#930055

  • Removes custom value of max_allowed_secret_in_bytes in barbican.conf. The default maximum size in Barbican was doubled to avoid issues with some certificates. LP #1957795

  • Fixes deploy Zun with Cinder Ceph support. Adds support for zun to access cinder volumes when external ceph is configured for cinder. LP#1848934

  • Fixed the deployment failure of outward_rabbitmq by resolving port conflicts by customizing RabbitMQ’s prometheus.tcp.port. LP #1885106

  • Use Volume V3 API in OpenStack exporter. Volume V2 API has been removed since OpenStack Wallaby. LP#1938194

  • Adds the node parameter when using the rabbitmq_user Ansible module. LP#1946506

  • Fixes an issue with multinode MariaDB deployments which could fail the playbook execution on WSREP check due to the new behaviour of Galera 4. LP#1947485.

  • Fixes an issue with single node MariaDB deployments with HAProxy disabled. See bug 1947534 for details.

  • Fixes the generation of wsrep_cluster_address in galera.cnf when --limit is used while deploying MariaDB nodes. LP#1947589

  • Fixes the copy job for grafana custom home dashboard file. The copy job for the grafana home dashboard file needs to run priviliged, otherwise permission denied error occurs. LP#1947710

  • Fixes an error in placement role which prevents to deploy the placement service when custom policy file is used. LP#1948835

  • Fixes missing current Ansible version in the error message. LP#1948979

  • Fix octavia role doesn’t set the amphora network’s gateway_ip LP#1949260

  • Fixes Octavia’s “Connection refused” errors by adding ovn_sb_connection to octavia.conf. LP#195011

  • Ironic API and Ironic Inspector API use separate policy files. Ironic role was updated to be able to handle both policies separately. LP#1952948

  • Only run configure ovn in ovsdb task on ovn-controller hosts The task will fail on hosts (like controller nodes) without tunnel interface LP#1953367

  • Continue to run all actions if one action failed in Elasticsearch curator. LP#1954720

  • Fixes Placement no logrotate configuration LP#1954723

  • Fixes Nova resize failing when migration_interface is customised. LP#1956976

  • Fixes unable to connect to zun console when kolla_enable_tls_external is true. Access to console of any zun container fails when kolla_enable_tls_external is true. This fix sets the protocol for wsproxy base_url in zun.conf according to the value of kolla_enable_tls_external LP#1957117

  • Fixes Register Identity Providers in OpenStack task which was missing an = in the openstack command causing the task to fail to register an IDP with Keystone. LP#1959022

  • Fixes Glance with Cinder iSCSI backend failing due to lack of lock_path setting. LP#1959663

  • Fixes logrotate config missing for openvswitch and prometheus services. LP#1961795

  • Fixes an issue with Ironic’s PXE components not getting updated on upgrade. LP#1963752

  • Adds Fluentd configurations to allow matching OpenvSwitch logs. LP#1965815

  • Fixes an issue where the Nova API logs were written to files ending with -wsgi.log which affected the processing of these logs in the Fluentd pipeline. LP#1950185

  • Fixes configuration of the Prometheus HTTP API URL when using the Prometheus collector in CloudKitty. LP#1961615

  • Fixes an issue with Prometheus scraping when targets’ Ansible inventory hostnames (inventory_hostname) do not resolve to reachable IP addresses. Reverts to the previous behaviour of using IP addresses to communicate with targets. The side effect of this is that targets instances will again be labelled using IP addresses rather than hostnames. LP#1955563

  • Fix the apache’s wsgi configuration for the aodh service in Debuntu binary flavours. LP#1953059

  • Fixes the baremetal role to avoid an error “Unable to remove “libvirtd”. Now the symlink /etc/apparmor.d/disable/usr.sbin.libvirtd is created by the role. LP#1960302

  • Existing fluentd log rotation failed to delete old haproxy, swift, glance-tls-proxy and neutron-tls-proxy logs. Standardise rotation and deletion of logs using logrotate.

  • Fixes an issue with setting up OIDC based Keystone federation against IDP that has a different response type than id_token. This can now be set using a new variable keystone_federation_oidc_response_type. LP#1959781

  • adds back the option to configure the rabbitmq clustering interface via kolla LP#1900160 <https://bugs.launchpad.net/kolla-ansible/+bug/1900160>

  • On slower nodes, the initial grafana startup could experience a timeout failure when the migrations for setting up the database took longer than expected. This has been fixed by increasing the default timeout. The timeout settings can be changed via new parameters grafana_start_first_node_delay and grafana_start_first_node_retries for the grafana role. LP#1769962

  • Fixes an issue seen when using Jinja2 3.1.0.

  • Fixes the configuration option setting the type of endpoint used by Neutron to send requests to Placement. LP#1960503

  • Fixes a configuration issue with Node Exporter causing all file system metrics of a host to be identical. LP#1961438

  • Fixes an issue where a failure of any Nova compute service to register itself would cause only the host querying the nova API to fail. Now, only hosts that fail to register will fail the Kolla Ansible run. Alternatively, to fail all hosts in a cell when any compute service fails to register, set nova_compute_registration_fatal to true. LP#1940119

  • Fixes an issue where RabbitMQ was configured to mirror classic transient queues for all services. According to the RabbitMQ documentation this is not a supported configuration, and contributed to numerous bug reports. LP#1954925

  • Removes “fix_cephfs_owner.yaml” which related to pre-wallaby Manila’s use of subfolders. Post-wallaby Manila now uses cephfs volumes instead, as such this file is no longer required. LP#1938285 LP#1935784

  • Removes use of “cephfs_enable_snapshots” in Manila config as this option was removed from Manila in the Wallaby release.

  • Fixes an issue with Cinder upgrade where Cinder services would remain pinned to the previous release’s RPC & object versions. LP#1954932

Other Notes

  • The container ironic-dnsmasq now creates the dnsmasq.log just as the container neutron-dhcp-agent. For both log files verbosity can be increased globally via openstack_logging_debug or per service via ironic_logging_debug or neutron_logging_debug variables.