Zed Series Release Notes

11.0.0-6

Bug Fixes

  • Fix an authentication error with Barbican when creating a TERMINATED_HTTPS listener with application credential tokens or trust IDs.

  • Fixed backwards compatibility issue with the feature that preserves HAProxy server states between reloads. HAProxy version 1.5 or below do not support this feature, so Octavia will not to activate it on amphorae with those versions.

11.0.0

New Features

  • Configuration of the amphora’s timezone is now possible using new configuration setting “amp_timezone” in the controller_worker options group.

  • Octavia now supports oslo.message notifications for loadbalancer create, delete, and update operations.

  • A new configuration option failover_threshold can be set to limit the number of amphorae simultaneously pending failover before halting the automatic failover process. This should help prevent unwanted mass failover events that can happen in cases like network interruption to an AZ or the database becoming read-only. This feature is not enabled by default, and it should be configured carefully based on the size of the environment. For example, with 100 amphorae a good threshold might be 20 or 30, or a value greater than the typical number of amphorae that would be expected on a single host.

  • It is now possible to create a loadbalancer with more than one VIP. There is a new structure additional_vips in the create body, which allows a subnet, and optionally an IP, to be specified. All VIP subnets must be part of the same network.

Known Issues

  • When using a distribution with a recent SELinux release such as CentOS 8 Stream, PING health-monitor does not work as shell_exec_t calls are denied by SELinux.

  • Fixed configuration issue which allowed authenticated and authorized users to inject code into HAProxy configuration using API requests. Octavia API no longer accepts unencoded whitespace characters in url_path values in update requests for healthmonitors.

Upgrade Notes

  • A new option is provided in the oslo_messaging namespace to disable event_notifications.

  • The default for the output file has been changed in diskimage-create.sh. It is now amphora-x64-haproxy.qcow2 instead of amphora-x64-haproxy.

  • The fix that updates the Netfilter Conntrack Sysfs variables requires rebuilding the amphora image in order to be effective.

  • Update Python base version from 3.6 to 3.8. As per Openstack Python runtime versions policy Python 3.8 will be the the minimum Python version in the Zed release cycle.

  • To support multi-VIP loadbalancers, a new amphora image must be built. It is safe to upload the new image before the upgrade, as it is fully backwards compatible.

  • diskimage-create defaults now to distribution release 9 when selecting RHEL as base OS and to release 9-stream when selecting CentOS as base OS.

Deprecation Notes

  • The ‘amphorav1’ provider is deprecated and will be removed in a future release. Use the ‘amphora’ provider (an alias for ‘amphorav2’) instead.

Bug Fixes

  • In order to avoid hitting the Neutron API hard when batch update with creating many new members, we cache the subnet validation results in batch update members API call. We also change to validate new members only during batch update members since subnet ID is immutable.

  • diskimage-create.sh used $AMP_OUTPUTFILENAME.$AMP_IMAGETYPE for constructing the image file path when checking the file size, which was not correct and caused an “No such file or directory” error.

  • Ensure that the provided rsyslog configuration file is used by rsyslog in the amphora by restarting the service when using the amphorav1 provider, it fixes the log offloading feature on distributions that start rsyslog before cloud-init.

  • Fix an issue that may have occurred when running the amphorav2 with persistence, the ComputeActiveWait was incorrectly executed twice on different controllers.

  • Fix disabled UDP pools. Disabled UDP pools were marked as “OFFLINE” but the requests were still forwarded to the members of the pool.

  • Fix the shutdown of the driver-agent, the process might have been stuck while waiting for threads to finish. Systemd would have killed the process after a timeout, but some children processes might have leaked on the controllers.

  • Enable required SELinux booleans for CentOS or RHEL amphora image.

  • Fix a bug that prevented the provisioning_state of a health-monitor to be set to ERROR when an error occurred while creating, updating or deleting a health-monitor.

  • Fixes listener creation failure when protocol used is PROXY or PROXYV2 which are pool protocol and not listener protocol.

  • Fix update listener certs failed. The fix ensures that an existing certificate gets overwritten properly.

  • Netfilter Conntrack Sysfs variables net.netfilter.nf_conntrack_max and nf_conntrack_expect_max get set to sensible values on the amphora now. Previously, kernel default values were used which were much too low for the configured net.netfilter.nf_conntrack_buckets value. As a result packets could get dropped because the conntrack table got filled too quickly. Note that this affects only UDP and SCTP protocol listeners. Connection tracking is disabled for TCP-based connections on the amphora including HTTP(S).

  • Now the [nova] service_name parameter is effectively used to find the nova endpoint in keystone catalog. The parameter had no effect before it was fixed.

  • Fix PING health-monitors with recent haproxy releases (>=2.2), haproxy now requires an additional “insecure-fork-wanted” option to authorize the Octavia PING healthcheck.

  • Fix a bug when adding a member on a subnet that belongs to a network with multiple subnets, an incorrect subnet may have been plugged in the amphora.

  • Fix a bug when deleting the last member plugged on a network, the port that was no longer used was not deleted.

  • Fix a bug when updating a load balancer with a QoS policy after a failover, Octavia attempted to update the VRRP ports of the deleted amphorae, moving the provisioning status of the load balancer to ERROR.

  • Fix a potential race condition when updating a resource in the amphorav2 worker. The worker was not waiting for the resource to be set to PENDING_UPDATE, so the resource may have been updated with old data from the database, resulting in a no-op update.

  • Fix the rescheduling of taskflow tasks that have been resumed after being interrupted.

  • Fixed issue with SELinux and the lvs-masquerade.sh script on the amphora. The script already runs with root permissions, so the use of sudo inside the script is unneeded.

  • Fix an issue when Octavia performs a failover of an ACTIVE-STANDBY load balancer that has both amphorae missing. Some tasks in the controller took too much time to timeout because the timeout value defined in [haproxy_amphora].active_connection_max_retries and [haproxy_amphora].active_connection_rety_interval was not used.

  • Fix a serialization issue when using TLSContainer with amphorav2 driver with persistence, a list of bytes type in the data model was not correctly converted to serializable data.

  • Fixed “Could not retrieve certificate” error when updating/deleting the client_ca_tls_container_ref field of a listener after a CA/CRL was deleted.

  • Fix a python3 error that prevented to use the [controller_worker]/user_data_config_drive option when building amphorae.

  • Fixed validations in L7 rule and session cookie APIs in order to prevent authenticated and authorized users to inject code into HAProxy configuration. CR and LF (\r and \n) are no longer allowed in L7 rule keys and values. The session persistence cookie names must follow the rules described in https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Set-Cookie.

  • Fix load balancers stuck in PENDING_UPDATE issues for some API calls (POST /l7rule, PUT /pool) when a provider denied the call.

  • The Octavia API returned an unhelpful message when a constraint failed while creating an object in the DB. The error now contains the name and the value of the parameter that breaks the constraints.

  • Validate that the creation of L7 policies is compatible with the protocol of the listener in the Amphora driver. L7 policies are allowed for Terminated HTTPS or HTTP protocol listeners, but not for HTTPS, TCP or UDP protocols listeners.

Other Notes

  • The netaddr python module has been removed as an Octavia requirement. It has been replaced with the python standard library ‘ipaddress’ module.

  • Admin documentation page has been added to explain the available events, the notification format, and how to disable event notifications.

  • The string representation of data base model objects has been improved. Calling str() on them will return a certain subset of fields and calling repr() on them will return all fields. This is helpful for debugging, but it may also change some of the log messages that Octavia emits.