Victoria Series Release Notes


Bug Fixes

  • Fixed an issue with batch member updates, that don’t have any changes, not properly rolling back the update.

  • Fixed an issue that an amphorav2 LB cannot be reached after loadbalancer failover. The LB security group was not set in the amphora port.

  • Fixes an issue where provider drivers may not decrement the load balancer objects quota on delete.

  • Fix an issue with the rsyslog configuration file in the Amphora when the log offloading feature and the local log storage feature are both disabled.

  • Some IPv6 UDP members were incorrectly marked in ERROR status, because of a formatting issue while generating the health message in the amphora.

  • Fixed an issue with the lo interface in the amphora-haproxy network namespace. The lo interface was down and prevented haproxy to communicate with other haproxy processes (for persistent stick tables) on configuration change. It delayed old haproxy worker cleanup and increased the memory consumption usage after reloading the configuration.

  • Fix load balancers that use customized host_routes in the VIP or the member subnets in amphorav2.

  • Fix weighted round-robin for UDP listeners with keepalived and lvs. The algorithm must be specified as ‘wrr’ in order for weighted round-robin to work correctly, but was being set to ‘rr’.

  • Fixed the healthcheck endpoint always querying the backends by caching results for a configurable time. The default is five seconds.


Bug Fixes

  • Fixes an issue with load balancer failover, when the VIP subnet is out of IP addresses, that could lead to the VIP being deallocated.

  • Fix default value override for timeout values for listeners. Changing the default timeouts in the configuration file wasn’t correctly applied in the default listener parameters.

  • Fix nf_conntrack_buckets sysctl in the Amphora, its value was incorrectly set.

  • Fixed an issue were updating a CRL or client certificate on a pool would cause the pool to go into ERROR.

  • Fixed a bug where pools with PROXYV2 will go into ERROR.

  • Fix an issue when updating tls_versions and tls_ciphers in Pools with empty (None) values, unsetting theses parameters now resets their values to the default values.

  • Add a validation step in the Octavia Amphora driver to ensure that the port_security_enabled parameter is set on the VIP network.


Bug Fixes

  • Fixed an issue that could cause load balancers, with multiple amphora in a failed state, to be unable to complete a failover.

  • Fix an incorrect operating_status with empty UDP pools. A UDP pool without any member is now ONLINE instead of OFFLINE.

  • Add missing cloud-utils-growpart RPM to Red Hat based amphora images.

  • Add missing cronie RPM to Red Hat based amphora images.


New Features

  • Added support for proxy protocol version 2.

  • Added HTTP/2 over TLS support via ALPN protocol negotiation to the amphora provider driver. Feature available in amphora images with HAProxy 2.0 or newer.

  • Added the ability to delete amphora that are not in use.

  • Operators can now use the ‘amp_image_tag’ Octavia flavor capability when using the amphora provider driver. This allows custom amphora images to be used per-load balancer. If this is not defined in an Octavia flavor, the amp_image_tag Octavia configuration file setting will continue to be used.

  • Introduced an image driver interface. Supported drivers are noop and Glance.

  • Add l7policy and l7rule to octavia quota.

  • Added support for TLS extension Application Layer Protocol Negotiation (ALPN) to TLS-terminated HTTPS load balancers. A new parameter alpn_protocols was added to the Listener API.

  • Octavia provider drivers can now offer HTTP/2 over TLS (protocol negotiation via ALPN) to clients.

  • Added support for nftables to the devstack plugin and the amphora.

  • Add support for SCTP protocol. SCTP support has been added in the Octavia API for listener, pool, and health-monitor resources.

  • Added a new configuration setting ([task_flow]/jobboard_enabled) to enable/disable jobboard functionality in the amphorav2 provider. When disabled, the amphorav2 provider behaves similarly to the amphora v1 provider and does not require extra dependencies. The default setting is jobboard disabled while jobboard remains an experimental feature.

  • Added minimum_tls_version to octavia.conf. Listeners, pools, and the defaults for either will be blocked from using any lower TLS versions. By default, there is no minumum version.

  • Add a new configuration option to define the default connection_limit for new listeners that use the Amphora provider. The option is [haproxy_amphora].default_connection_limit and its default value is 50,000. This value is used when creating or setting a listener with -1 as connection_limit parameter, or when unsetting connection_limit parameter.

  • TLS-enabled pools can now be configured to use only specified versions of TLS. Default TLS versions for new pools can be set with default_pool_tls_versions in octavia.conf. Existing pools will continue to use the old defaults.

  • Loadbalancer statistics can now be reported to multiple backend locations simply by specifying multiple statistics drivers in config.

  • Added tls_cipher_prohibit_list to octavia.conf. Listeners, pools, and the default values for either will be blocked from using any of these ciphers. By default, no ciphers are prohibited.

  • HTTPS-terminated listeners can now be configured to use only specified versions of TLS. Default TLS versions for new listeners can be set with default_listener_tls_versions in octavia.conf. Existing listeners will continue to use the old defaults.

Upgrade Notes

  • When the amphora provider driver is enabled, operators need to set option [controller_worker]/image_driver. The default image driver is image_glance_driver. For testing could be used image_noop_driver.

  • An amphora image update is recommended to pick up a workaround to an HAProxy issue where it would fail to reload on configuration change should the local peer name start with “-x”.

  • The failover improvements do not require an updated amphora image, but updating existing amphora will minimize the failover outage time for standalone amphora on subsequent failovers.

  • The option [controller_worker]/amp_ssh_access_allowed has been deprecated since Queens release and is now removed. This option was superseded by [controller_worker]/amp_ssh_key_name option.

  • The option [controller_worker]/amp_image_id has been deprecated since Mitaka release and is now removed. This option was superseded by [controller_worker]/amp_image_tag option.

  • The internal interface for loadbalancer statistics collection has moved. When upgrading, see deprecation notes for the stats_update_driver config option, as it will need to be moved and renamed.

  • The default drivers have been switched to live from noop drivers for the most part. Volume and distributor remain set to noop drivers as those are experimental features. Operators do not need to make configuration changes.

  • HTTPS-terminated listeners will now only allow TLS1.2 and TLS1.3 by default. If no TLS versions are specified at listener create time, the listener will only accept TLS1.2 and TLS1.3 connections. Previously TLS listeners would accept any TLS version. Existing listeners will not be changed.

Deprecation Notes

  • Spares pool support is deprecated, pending removal in the X release. Use of the spares pool was originally recommended to increase provisioning speed, but since Nova’s server groups do not support adding existing VMs, Octavia cannot support use of the spares pool with the Active-Standby topology. Since this is our recommended topology for production deployments, and speed is less essential in development/testing environments (the only place we could recommend the use of Single topology), the overhead of maintaining spares pool support exceeds its theoretical usefulness.

  • Terminology such as blacklist has been replaced with more inclusive words, such as prohibit list wherever possible.

    The configuration option tls_cipher_blacklist has been deprecated and replaced with tls_cipher_prohibit_list. It will be removed in a future release.

  • The deprecated option status_update_threads has been removed, health_update_threads and stats_update_threads should be used instead.

  • The option health_manager.health_update_driver has been deprecated as it was never really used, so the driver layer was removed. The option health_manager.stats_update_driver was moved and renamed to controller_worker.statistics_drivers (note it is now plural). It can now contain a list of multiple drivers for handling statistics.

Security Issues

  • If you are using the admin_or_owner-policy.yaml policy override file you should upgrade your API processes to include the unscoped token fix. The default policies are not affected by this issue.

Bug Fixes

  • Fixed an issue when a loadbalancer is disabled, Octavia Health Manager keeps failovering the amphorae

  • Fixed an issue where members added to TLS-enabled pools would go to ERROR provisioning status.

  • Fixed an issue with failing over an amphora if the pair amphora in an active/standby pair had a missing VRRP port in neutron.

  • Fixed an issue where setting of SNI containers were not being applied on listener update API calls.

  • Fixed an Octavia API validation on listener update where SNI containers could be set on non-TERMINATED_HTTPS listeners.

  • Fixed an issue where some columns could not be used for sort keys in API list calls.

  • Fixed an issue in the CADF audit map file for failover actions that could cause keystonemiddleware to raise an exception.

  • Fix an issue when the barbican service enable TLS, we create the listerner failed.

  • Fix operational status for disabled UDP listeners. The operating status of disabled UDP listeners is now OFFLINE instead of ONLINE, the behavior is now similary to the behavior of HTTP/HTTPS/TCP/… listeners.

  • Fixed an issue where clearing listener TLS versions resulted in a server-side error.

  • Fixed an issue where when clearing listener TLS versions and ciphers would not apply the default values per defined in the API configuration settings.

  • Fixed an issue where amphora load balancers fail to create when Nova anti-affinity is enabled and topology is SINGLE.

  • Workaround an HAProxy issue where it would fail to reload on configuration change should the local peer name start with “-x”.

  • Fixed an issue where listener “insert_headers” parameter was accepted for protocols that do not support header insertion.

  • Fixed an issue where TLS-enabled pools would fail to provision.

  • Fixed an issue where UDP only load balancers would not bring up the VIP address.

  • Fix a potential invalid DOWN operating status for members of a UDP pool. A race condition could have occured when building the first heartbeat message after adding a new member in a pool, this recently added member could have been seen as DOWN.

  • Fixes an issue when using the admin_or_owner-policy.yaml policy override file and unscoped tokens.

  • With haproxy 1.8.x releases, haproxy consumes much more memory in the amphorae because of pre-allocated data structures. This amount of memory depends on the maxconn parameters in its configuration file (which is related to the connection_limit parameter in the Octavia API). In the Amphora provider, the default connection_limit value -1 is now converted to a maxconn of 50,000. It was previously 1,000,000 but that value triggered some memory allocation issues when quickly performing multiple configuration updates in a load balancer.

  • Significantly improved the reliability and performance of amphora and load balancer failovers. This is especially true when the Nova service is experiencing failures.

Other Notes

  • Though the current HAProxy version 1.8 used in some distributions support HTTP/2, we highly recommend using HAProxy version 2.0 or newer in the amphora image when using HTTP/2.