Victoria Series Release Notes

11.0.0-43

New Features

  • OVN deployment will now configure external_ids:ovn-chassis-mac-mappings to make DVR work on VLAN tenant networks.

Bug Fixes

  • Fixes an issue with kolla-ansible bootstrap-servers if Zun is enabled where Zun-specific configuration for Docker was applied to all nodes. LP#1914378

  • Fix the issue when Swift deployed with S3 Token Middleware enabled. Fixes LP#1862765

  • Fixes the Northbound and Southbound database socket paths in OVN.

  • Fixes an issue preventing prechecks from succeeding when “non-native” NTP daemon was used, such as ntpd` as opposed to ``systemd-timesyncd on a Debian/Ubuntu system or to chronyd on a CentOS/RHEL system. LP#1922721

  • Fixed an issue when Docker was configured after startup on Debian/Ubuntu, which resulted in iptables rules being created - before they were disabled. LP#1923203

  • Fixes an issue with Octavia SSH key copying if user disabled Octavia auto configuration. LP##1927727

  • Fixes potential issue with Alertmanger in non-HA deployments. In this scenario, peer gossip protocol is now disabled and Alertmanager won’t try to form a cluster with non-existing other instances. LP#1926463

  • Fixes some configuration issues around Barbican logging. LP#1891343

  • Fixes some configuration issues around Cinder logging. LP#1916752

  • Fix cyborg api doesn’t listen on api interface. change host to host_ip in cyborg.conf. See the cyborg documentation

  • Fix the wrong configuration of the ovs-dpdk service. this breaks the deployment of kolla-ansible. For more details please see bug 1908850.

  • Fixes an issue with executing kolla-ansible when installed via pip install --user. LP#1915527

  • Fixes an issue where masakari.conf was generated for the masakari-instancemonitor service but not used.

  • Fixes an issue where masakari-monitors.conf was generated for the masakari-api and masakari-engine services but not used.

  • Uses a consistent variable name for container dimensions for masakari-instancemonitor - masakari_instancemonitor_dimensions. The old name of masakari_monitors_dimensions is still supported.

  • Fixes LP#1892376 by updating deprecated syntax in the Monasca Elasticsearch template.

  • Removes whitespace around equal signs in zookeeper.cfg which were preventing the zkCleanup.sh script from running correctly.

11.0.0

New Features

  • Adds ability to provide a custom elasticsearch config.

  • Adds configuration parameter kolla_httpd_keep_alive to configure the keep-alive timeout for services that use httpd to handle HTTP requests. The default value is 60 seconds, overriding the httpd default of 5 seconds.

  • Adds support for multiple globals files. The main globals.yml file still exists. In addition to that, operators can now create a globals.d directory (next to globals.yml), where they can place any number of *.yml files, for example for specific services they want to add.

  • With the boolean parameter enable_neutron_trunk it is now possible to enable or disable the Neutron service plugin trunk.

  • add support of octavia dev mod.

  • Add “etcd_enable_tls” configuration parameter which can be used to enable TLS encryption for the etcd service. The default value of “etcd_enable_tls” is set by the value of “kolla_enable_tls_backend”.

  • Adds support for automatic registration resources required for Octavia. This includes a management network and subnet, security groups, flavor and SSH key. The amphora image must be uploaded manually currently. BP#automatic-deploy-of-octavia

  • Adds support for automatic generation of certificates for Octavia via the kolla-ansible octavia-certificates command.

  • Kolla Ansible checks now that the local Ansible Python environment is coherent, i.e. used Ansible can see Kolla Ansible. LP#1856346

  • Makes it possible to set HAProxy’s timeout http-keep-alive via haproxy_http_keep_alive_timeout. The default is 10s, like the previous default that applied via haproxy_http_request_timeout. LP#1892622

  • Adds kolla_sysctl_conf_path variable that allows to customise the path to sysctl.conf that will be modified by Kolla Ansible plays. The default is /etc/sysctl.conf as it was before.

  • Adds a new flag, docker_disable_default_iptables_rules, which defaults to no. Docker is manipulating iptables rules by default to provide network isolation, and this might cause problems if the host already has an iptables based firewall. A common problem is that Docker sets the default policy of the FORWARD chain in the filter to DROP. Setting docker_disable_default_iptables_rules to yes will disable Docker’s iptables manipulation. This feature will be enabled by default from the Victoria 11.0.0 release.

  • Adds configuration options to enable backend TLS encryption from HAProxy to the Nova, Ironic, and Neutron services. When used in conjunction with enabling TLS for service API endpoints, network communcation will be encrypted end to end, from client through HAProxy to the backend service.

  • Adds support for completing the http-01 challenge of ACME (e.g. as provided by Let’s Encrypt - letsencrypt.org) using an external ACME client (e.g. certbot). The relevant variable is acme_client_servers. Please read the docs for more info on this integration.

  • Improves performance of the common role by generating all fluentd configuration in a single file.

  • Self-signed TLS certificates can be used to test TLS in a development OpenStack environment. The kolla-ansible certificates command will generate the required self-signed TLS certificates. This command has been updated to first create a self-signed root certificate authority. The command then generates the internal and external facing certificates and signs them using the root CA. If backend TLS is enabled, the command will generate the backend certificate and sign it with the root CA.

  • (internal/advanced) Adds support for with_frontend and with_backend to haproxy service definitions. These new fields preserve the old logic by defaulting to true but can be set to false to make the selected service not configure the respective “end”. This requires haproxy_service_template to be set to haproxy_single_service_split.cfg.j2 which is the new default.

  • Implements container healthchecks for core OpenStack services. Docker healthchecks are periodically called scripts that check health of a running service that expose health information in docker ps output and trigger a health_status event. Healthchecks are now enabled by default and can be disabled by setting enable_container_healthchecks to no in globals.yml.

  • Improves performance of the common role by generating all logrotate configuration in a single file.

  • Enable Galera node state checking by using clustercheck script that is used by HAProxy to define node up/down state.

  • Kolla started building neutron-mlnx-agent in Ussuri cycle. Now those containers can be deployed too via Kolla-Ansible. Note that neutron-mlnx-agent image is also used to deploy neutron-eswitchd container.

  • Adds support for TLS encryption of RabbitMQ client-server communication. See blueprint for details.

  • Extracts the common role into a separate play. This provides a performance benefit at scale, since the role dependency mechanism used previously had a overhead. This change allows the only common role to be executed by specifying the common tag.

  • Adds a mechanism to customize skydive.conf.

  • Allows to skip and unset sysctl variables controlled by Kolla Ansible plays using KOLLA_SKIP and KOLLA_UNSET values.

  • Adds timesync prechecks which run when containerised chrony is not enabled to ensure the host has its system clock synchronized.

Known Issues

  • Since Ussuri, there is a bug in how Ceph (RBD) is handled with Cinder: the backend_host option is missing from the generated configuration for external Ceph. The symptoms are that volumes become unmanageable until extra admin action is taken. This does not affect the data plane - running virtual machines are not affected.

    There is a related issue regarding active-active cinder-volume services (single-host cinder-volume not affected), which is that they should not have been configured with backend_host in the first place but with cluster and proper coordination instead. Some users might have customised their config already to address this issue.

    The Kolla team is investigating the best way to address this for all its users. In the meantime, please ensure that, before upgrading to Ussuri, the backend_host option is set to its previous value (the default was rbd:volumes) via a config override.

    For more details please refer to the referenced bug. Do note this issue affects both new deployments and upgrades. LP#1904062

Upgrade Notes

  • Adds configuration parameter kolla_httpd_keep_alive to configure the keep-alive timeout for services that use httpd to handle HTTP requests. The default value is 60 seconds, overriding the httpd default of 5 seconds.

  • Resources required for Octavia are now registered automatically by default. Octavia users upgrading from Ussuri should set octavia_auto_configure to no in globals.yml to avoid registering conflicting resources.

  • When deploying Monasca with Logstash 6, any custom Logstash 2 configuration for Monasca will need to be updated to work with Logstash 6. Please consult the documentation.

  • HAProxy’s timeout http-keep-alive is now set via haproxy_http_keep_alive_timeout. The default is 10s, like the previous default that applied via haproxy_http_request_timeout, but the new variable does not follow the old one so might need customising if the former was customised for keep-alive purposes. Do note the two values do not have to have the same value and often do not. Please read the linked record for background. LP#1892622

  • No longer uses option http-tunnel for Neutron Server in HAProxy. Please amend manually if you relied on the quirky behaviour. LP#1892686

  • The default migration_interface is moved from network_interface to api_interface, which is treaded as internal and security network plane in most case.

  • Replaced kolla_external_fqdn_cacert and kolla_internal_fqdn_cacert with kolla_admin_openrc_cacert, which by default is not set. OS_CACERT is now set to the value of kolla_admin_openrc_cacert in the generated admin-openrc.sh file.

  • The default of haproxy_service_template changed to haproxy_single_service_split.cfg.j2. This template allows more flexibility in service config. The previously-default haproxy_single_service_listen.cfg.j2 is now deprecated for removal. No action needs to be taken unless one relied on replacing the previous default template contents with one’s own (under the same name).

  • Changes the default value of kibana_elasticsearch_ssl_verify from false to true. LP#1885110

  • mariadb role uses now mariadb-server image by default (compared to mariadb previously), since mariadb image will be deprecated in the Kolla Victoria release and removed in Wallaby.

  • Enabling multipathd will now configure Nova to use it.

  • The Prometheus OpenStack exporter now uses internal endpoints to communicate with OpenStack services, to match the configuration of other services deployed by Kolla Ansible. Using public endpoints can be retained by setting the prometheus_openstack_exporter_endpoint_type variable to public.

  • The congress project is no longer maintained. This has been retired since Victoria and has not been used by other OpenStack services since.

  • Support for deploying with Hyper-V integrations has been removed.

  • Customizing Neutron Linux bridge and Open vSwitch Agents config via ml2_conf.ini is removed. The config has been split out for these agents into linuxbridge_agent.ini and openvswitch_agent.ini respectively. The old behaviour was deprecated in Ussuri for removal in Victoria.

  • Support for deploying with mongodb integrations has been removed.

  • The neutron-fwaas project is no longer maintained. This has been retired and will be removed in the Victoria cycle.

  • Support mongodb for panko as backend has been removed.

  • Support for deploying with XenAPI integrations has been removed.

  • Kolla-Ansible now requires Ansible 2.9. Ansible 2.8 is not supported anymore.

  • The common role is now executed in a separate play. This introduces a few small changes in behaviour:

    • the common role is now run for all hosts at the beginning, rather than prior to their first enabled service

    • hosts must be in the necessary group for each of the common services (cron, fluentd, kolla-logs, kolla-toolbox) in order to have that service deployed

    • if tags are specified for another service e.g. nova, the common role will not automatically run for matching hosts. The common tag must be specified explicitly

  • Apache ZooKeeper will now be automatically deployed whenever Apache Storm is enabled.

  • The default value of REST_API_REQUIRED_SETTINGS was synchronized with Horizon. You may want to review settings exposed by the updated configuration.

  • Adds support for Ubuntu Focal 20.04 as a host operating system. Ubuntu users upgrading from Ussuri should first upgrade OpenStack containers to Victoria, which uses the Ubuntu Focal 20.04 base container image. Hosts should then be upgraded to Ubuntu Focal 20.04.

  • The Monasca Log API has been removed. All logs now go to the unified Monasca API when Monasca is enabled. Any custom Fluentd configuration and inventory files will need to be updated. Any monasca_log_api containers will be removed automatically.

Deprecation Notes

  • The previous default of haproxy_service_template (haproxy_single_service_listen.cfg.j2) is now deprecated for removal as haproxy_single_service_split.cfg.j2 takes it place to allow more flexibility in service config.

  • The variable kolla_internal_address is deprecated. This variable is used only as the default value for kolla_internal_vip_address, and is not documented.

  • VMware integration, which was deprecated in the Ussuri release, is no longer deprecated. Nova has reversed the deprecation of its VMware driver, and the Kolla community has shown interest in VMware.

Security Issues

  • The admin-openrc.sh file generated by kolla-ansible post-deploy was previously created with root:root ownership and 644 permissions. This would allow anyone with access to the same directory to read the file, including the admin credentials. The ownership of admin-openrc.sh is now set to the user executing kolla-ansible, and the file is assigned a mode of 600. This change can be applied by running kolla-ansible post-deploy.

Bug Fixes

  • Add support to use bifrost-deploy behind proxy. It uses existing container_proxy variable.

  • Fixes handling of /dev/kvm permissions to be more robust against host-level actions. LP#1681461

  • Fixes Kibana deployment with the new E*K stack (6+). LP#1799689

  • IPv6 fully-routed topology (/128 addressing) is now allowed (where applicable). LP#1848941

  • Fix prometheus-openstack-exporter to use CA certificate.

  • Makes haproxy and keepalived restarts during Kolla-Ansible actions more robust, especially in multinode scenarios (HA).

  • Removing chrony package and AppArmor profile from docker host if containerized chrony is enabled. LP#1882513

  • When deploying Elasticsearch 6, Logstash 2 was deployed by default which is not compatible with Elasticsearch 6. Logstash 6 is now deployed by default.

  • Fixes an issue when using ip addresses instead of hostnames in Ansible inventory. OpenvSwitch role sets system-id based on inventory_hostname, which in case of ip addresses in is first ip octet. Such a deployment would result in multiple OVN chassis with duplicate name e.g. “10” connecting to OVN Southbound database - which spawns high numbers of create/delete events in Encap database table - leading to near 100% CPU usage of OVN/OVS/Neutron processes.

  • Fixes an issue with Manila deployment starting openvswitch and neutron-openvswitch-agent containers when enable_manila_backend_generic was set to False. LP#1884939

  • Fixes the Elasticsearch Curator cron schedule run. LP#1885732

  • Fixes an incorrect configuration for nova-conductor when a custom Nova policy was applied, preventing the nova_conductor container from starting successfully. LP#1886170

  • Fix Castellan (Barbican client) when used with enabled TLS. LP#1886615

  • Fixes an incorrect Ceph keyring file configuration in gnocchi.conf, which prevented Gnocchi from connecting to Ceph. LP#1886711

  • Fixes --configdir parameter to apply to default passwords.yml location. LP#1887180

  • fluentd is now logging to /var/log/kolla/fluentd/fluentd.log instead of stdout. LP#1888852

  • Add functionality to the glance role to add extra config file for image property protection and interoperable image import LP#1889272

  • Fixes deploy-containers action missing for the Masakari role. LP#1889611

  • No longer uses option http-tunnel for Neutron Server in HAProxy. This mode was long not recommended and recently deprecated entirely. LP#1892686

  • Fixes some Neutron subprojects not using the rolling upgrade scheme. SFC forcibly used the legacy scheme and dynamic routing was not migrated at all. LP#1894380

  • An issue has been fixed when keystone container would be stuck in restart loop with a message that fernet key is stale. LP#1895723

  • Fixes haproxy_single_service_split template to work with default for mode (http). LP#1896591

  • Fixed invalid fernet cron file path on Debian/Ubuntu from /var/spool/cron/crontabs/root/fernet-cron to /var/spool/cron/crontabs/root. LP#1898765

  • Add with_first_found on placement for placement-api wsgi configuration to allow overwrite from users. LP#1898766

  • OVN will no longer schedule SNAT routers on compute nodes when neutron_ovn_distributed_fip is enabled. LP#1901960

  • RabbitMQ services are now restarted serially to avoid a split brain. LP#1904702

  • Fixes LP#1906796 by adding notice and note loglevels to monasca log-metrics drop configuration

  • Fixes Swift’s stop action. It will no longer try to start swift-object-updater container again. LP#1906944

  • Fixes an issue with the kolla-ansible prechecks command with Docker 20.10. LP#1907436

  • Fixes an issue with kolla-ansible mariadb_recovery when the mariadb container does not exist on one or more hosts. LP#1907658

  • fix deploy freezer failed when use kolla_dev_mod LP#1888242

  • Add missing “become: true” on some VMWare related tasks. Fixed on Copying VMware vCenter CA file and Copying over nsx.ini.

  • fix deploy nova failed when use kolla_dev_mod.

  • In line with clients for other services used by Magnum, Cinder and Octavia also use endpoint_type = internalURL. In the same tune, these services also use the globally defined openstack_region_name.

  • Fixes issues with some CloudKitty commands trying to connect to an external TLS endpoint using HTTP. LP#1888544

  • Fixes an issue where Docker may fail to start if iptables is not installed. LP#1899060

  • The admin-openrc.sh file generated by kolla-ansible post-deploy was previously created with root:root ownership and 644 permissions. This would allow anyone with access to the same directory to read the file, including the admin credentials. The ownership of admin-openrc.sh is now set to the user executing kolla-ansible, and the file is assigned a mode of 600. This change can be applied by running kolla-ansible post-deploy.

  • Fixes an issue with Cinder upgrades that would cause online schema migration to fail. LP#1880753

  • Fix the configuration of the etcd service so that its protocol is independant of the value of the internal_protocol parameter. The etcd service is not load balanced by HAProxy, so there is no proxy layer to do TLS termination when internal_protocol is configured to be https.

  • Fixes an issue during deleting evacuated instances with encrypted block devices. LP#1891462

  • Fixes LP#1885885 where the default chunk size in the Monasca Fluentd output plugin increased from 8MB to 256MB for file buffering which exceeded the limit allowed by the Monasca Log / Unified API.

  • Fixes an issue where Keystone Fernet key rotation may fail due to permission denied error if the Keystone rotation happens before the Keystone container starts. LP#1888512

  • Fixes an issue with Keystone startup when Fernet key rotation does not occur within the configured interval. This may happen due to one of the Keystone hosts being down at the scheduled time of rotation, or due to uneven intervals between cron jobs. LP#1895723

  • Reverts the arp_responder option setting to the default (‘False’) for the LinuxBridge agent, as this is known to cause problems with l2_population as well as other issues such as not being fully compatible with the allowed-address-pairs extension. LP#1892776

  • Fixes an issue with the Neutron Linux bridge ML2 driver where the firewall driver configuration was not applied. LP#1889455

  • Fixes an issue with Masakari and internal TLS where CA certificates were not copied into containers, and the path to the CA file was not configured. Depends on masakari bug 1873736 being fixed. LP#1888655

  • Fixes an issue where Grafana instances would race to bootstrap the Grafana DB. See LP#1888681.

  • Fixes LP#1892210 where the number of open connections to Memcached from neutron-server would grow over time until reaching the maximum set by memcached_connection_limit (5000 by default), at which point the Memcached instance would stop working.

  • Adds a new variable fluentd_elasticsearch_cacert, which defaults to the value of openstack_cacert. If set, this will be used to set the path of the CA certificate bundle used by Fluentd when communicating with Elasticsearch. LP#1885109

  • An issue where when Kafka default topic creation was used to create a Kafka topic, no redundant replicas were created in a multi-node cluster. LP#1888522. This affects Monasca which uses Kafka, and was previously masked by the legacy Kafka client used by Monasca which has since been upgraded in Ussuri. Monasca users with multi-node Kafka clusters should consultant the Kafka documentation to increase the number of replicas.

  • Improves error reporting in kolla-genpwd and kolla-mergepwd when input files are not in the expected format. LP#1880220.

  • Fixes an issue where the br_netfilter kernel module was not loaded on compute hosts. LP#1886796

  • Fixes Magnum trust operations in multi-region deployments.

  • The Prometheus OpenStack exporter now uses internal endpoints to communicate with OpenStack services, to match the configuration of other services deployed by Kolla Ansible.

  • Prevents adding a new Keystone host to an existing cluster when not targeting all Keystone hosts (e.g. due to --limit or --serial arguments), to avoid overwriting existing Fernet keys. LP#1891364

  • Reduce the use of SQLAlchemy connection pooling, to improve service reliability during a failover of the controller with the internal VIP. LP#1896635

  • No longer configures the Prometheus OpenStack exporter to use the prometheus Docker volume, which was never required.

  • Deploys Apache ZooKeeper if Apache Storm is enabled explicitly. ZooKeeper would only be deployed if Apache Kafka was also enabled, which is often done implicitly by enabling Monasca.

Other Notes

  • Add trove-guestagent.conf for trove