Victoria Series Release Notes

13.5.0-31

New Features

  • Added new heat role specific parameter option ‘DdpPackage’ to select the required DDP Package.

  • Added new heat role specific param OVNAvailabilityZone to set availability-zones for ovn. This param replace seting availability-zones throught OVNCMSOptions

  • New config options for Neutron logging service plugin configuration were added. There are options added for L3 Agent: NeutronL3AgentLoggingRateLimit, NeutronL3AgentLoggingBurstLimit, NeutronL3AgentLoggingLocalOutputLogBase, for OVS agent: NeutronOVSAgentLoggingRateLimit, NeutronOVSAgentLoggingBurstLimit, NeutronOVSAgentLoggingLocalOutputLogBase and for ML2/OVN backend: NeutronOVNLoggingRateLimit, NeutronOVNLoggingBurstLimit, NeutronOVNLoggingLocalOutputLogBase.

  • With conditional monitoring enabled in OVN, southbound ovsdb-serve takes lot of time in handling the monitoring and sending the updates to all its connected clients. Its takes lot of CPU. With monitor-all option, all ovn-controllers do not enable conditional monitoring there by reducing the load on the Southbound ovsdb-server.

  • A heat parameter IronicPowerStateChangeTimeout has been added which sets the number of seconds to wait for power operations to complete, i.e., so that a baremetal node is in the desired power state. If timed out, the power operation is considered a failure. The default is 60 seconds, which is the same as the current Ironic default.

13.5.0

New Features

  • Added OVN DBs clustering support. In this service model, a clustered database runs across multiple hosts in multi-active mode.

  • To help operators protect their workload, they can now enable the KernelArgsDeferReboot role parameter. This will prevent the tripleo-kernel ansible module from automatically rebooting nodes even if KernelArgs were changed unexpectedly.

Upgrade Notes

  • Upgrades from OVN non-HA and OVN DBs pacemaker to OVN DBs clustered are currently not supported.

Security Issues

  • The OVN database servers in an OVN DBs clustering and TLS-everywhere deployment will listen on all IP addresses (0.0.0.0). This is a caveat that can only be addressed once RHBZ 1952038 is fixed.

13.4.0

New Features

  • The following parameters add support for mounting Cinder’s image conversion directory on an external NFS share.

    • CinderImageConversionNfsShare

    • CinderImageConversionNfsOptions

  • The new MemcacheUseAdvancedPool parameter is added which enables usage of advanced poll for memcached connections in keystone middleware. This parameter is set to true by default to avoind bursting connections in some services like neutron.

  • Add support for OVS DPDK pmd auto balance parameters. This feature adds 3 new role specific THT parameters to set pmd-auto-lb-load-threshold, pmd-auto-lb-improvement-threshold, and pmd-auto-lb-rebal-interval in OVS through OvsPmdLoadThreshold, OvsPmdImprovementThreshold and OvsPmdRebalInterval respectively.

  • Introduce new parameter to configure OVS PMD Auto Load Balance for OVS DPDK

Bug Fixes

  • InternalTLSNbdCAFile, InternalTLSVncCAFile and InternalTLSQemuCAFile do not point to the default IPA ca.crt file and instead are requested to be loaded to component specific CA files (even if they are the same). This can lead to a race where the CA cert is not being written by certmonger in time. Ib868465c20d97c62cbcb214bfc62d949bd6efc62 already changed the default to use the IPA system cacert file ‘/etc/ipa/ca.crt’ per default starting with the wallaby release using the ansible role. This change backports to also use the IPA system cacert file ‘/etc/ipa/ca.crt’ to previous release when managing the certs via puppet-tripleo.

  • Bug #1915800: Add support for ports filtering in XtremIO driver.

Other Notes

  • These parameters can now be set per-role - DnfStreams, UpgradeInitCommand, UpgradeLeappCommandOptions, UpgradeLeappDevelSkip, UpgradeLeappToRemove, UpgradeLeappToInstall

13.3.0

New Features

  • Added new options for deploying Barbican with PKCS#11 backends: BarbicanPkcs11CryptoTokenLabels and BarbicanPkcs11CryptoOsLockingOk

  • The OS::TripleO::{{role.name}}::PreNetworkConfig resource has been restored. This resource can be used to implement any configuration steps executed before network configurations are applied.

  • QemuDefaultTLSVerify will allow operators to enable or disable TLS client certificate verification. Enabling this option will reject any client who does not have a certificate signed by the CA in /etc/pki/qemu/ca-cert.pem. The default is true and matches libvirt’s. We will want to disable this by default in train.

  • Adding ptp parameters for timemaster service configuration on overcloud compute node.Timemaster will use already present chrony parameters. PTPMessageTransport, PTPInterfaces are added new.

Deprecation Notes

  • The BarbicanPkcs11CryptoTokenLabel option has been deprecated and replaced with the BarbicanPkcs11CryptoTokenLabels option.

Bug Fixes

  • Now ExtraConfigPre resource and NodeExtraConfig resource are executed after network configurations are applied in nodes. This is consitent with the previous version with heat software deployment mechanism instead of config-download.

  • Previously access to the sshd running by the nova-migration-target container is only limited via the sshd_config. While login is not possible from other networks, the service is reachable via all networks. This change limits the access to the NovaLibvirt and NovaApi networks which are used for cold and live-migration.

  • Nova vnc configuration right now uses NovaVncProxyNetwork, NovaLibvirtNetwork and NovaApiNetwork to configure the different components (novnc proxy, nova-compute and libvirt) for vnc. If one of the networks get changed from internal_api, the service configuration between libvirt, nova-compute and novnc proxy gets inconsistent and the console is broken. This changed to just use NovaLibvirtNetwork for configuring the vnc endpoints and removes NovaVncProxyNetwork completely.

13.2.0

New Features

  • The new parameter GlanceCinderMountPointBase has been added which will be used for mounting NFS volumes on glance nodes. When glance uses cinder as store and cinder backend is NFS, this parameter must be set to match cinder’s mount point.

  • Added MemcachedMaxConnections setting with a default of 8192 maximum connections in order to allow an operator to override that value in environments where memcached is heavily sollicited.

  • The logic to configure the connection from barbican to nShield HSMs has been augmented to parse a nshield_hsms parameter, which allows the specification of multiple HSMs. The underlying ansible role (ansible-role-thales-hsm) will configure the HSMs in load sharing mode to provide HA.

  • New CinderRpcResponseTimeout and CinderApiWsgiTimeout parameters provide a means for configuring Cinder’s RPC response and WSGI connection timeouts, respectively.

  • Add posibilities to configure ovn dbs monitor interval in tht by OVNDBSPacemakerMonitorInterval (default 30s). Under load, this can create extra stress and since the timeout has already been bumped, it makes sense to bump this interval to a higher value as a trade off between detecting a failure and stressing the service.

  • When a node has hugepages enabled, we can help with live migrations by enabling NovaLiveMigrationPermitPostCopy and NovaLiveMigrationPermitAutoConverge. These flags are automatically enabled if hugepages are detected, but operators can override these settings.

  • Add NovaLibvirtMaxQueues role parameter to set [libvirt]/max_queues in nova.conf of the compute. Default 0 corresponds to not set meaning the legacy limits based on the reported kernel major version will be used.

Known Issues

  • Cell_v2 discovery has been moved from the nova-compute|nova-ironic containers as this requires nova api database credentials which must not be configured for the nova-compute service. As a result scale-up deployments which explicitly omit the Controller nodes will need to make alternative arrangements to run cell_v2 discovery. Either the nova-manage command can be run manually after scale-up, or an additional helper node using the NovaManage role can be deployed that will be used for this task instead of a Controller node. See Bug: 1786961 and Bug: 1871482.

Deprecation Notes

  • Some parameters within ThalesVars have been deprecated. These are - thales_hsm_ip_address and thales_hsm_config_location. See environments/barbican-backend-pkcs11-thales.yaml for details.

Bug Fixes

  • When deploying a spine-and-leaf (L3 routed architecture) with TLS enabled for internal endpoints the deployment would fail because some roles are not connected to the network mapped to the service in ServiceNetMap. To fix this issue a role specific parameter {{role.name}}ServiceNetMap is introduced (defaults to: {}). The role specific ServiceNetMap parameter allow the operator to override one or more service network mappings per-role. For example:

    ComputeLeaf2ServiceNetMap:
      NovaLibvirtNetwork: internal_api_leaf2
    

    The role specific {{role.name}}ServiceNetMap override is merged with the global ServiceNetMap when it’s passed as a value to the {{role.name}}ServiceChain resources, and the {{role.name}} resource groups so that the correct network for this role is mapped to the service.

    Closes bug: 1904482.

  • Do not relabel Swift files on every container (re-)start. These will be relabeled already in step 3 preventing additional delays.

13.1.0

New Features

  • The NovaApiMaxLimit parameter allows the operator to set Nova API max_limit using a Heat parameter in their templates.

Bug Fixes

  • Certificates get merged into the containers using kolla_config mechanism. If a certificate changes, or e.g. UseTLSTransportForNbd gets disabled and enabled at a later point the containers running the qemu process miss the required certificates and live migration fails. This change moves to use bind mount for the certificates and in case of UseTLSTransportForNbd ans creates the required certificates even if UseTLSTransportForNbd is set to False. With this UseTLSTransportForNbd can be enabled/disabled as the required bind mounts/certificates are already present.

  • https://review.opendev.org/q/I8df21d5d171976cbb8670dc5aef744b5fae657b2 introduced THT parameters to set libvirt/cpu_mode. The patch sets the NovaLibvirtCPUMode wrong to ‘none’ string which results in puppet-nova not to handle the default cases correct and sets libvirt/cpu_mode to none which results in ‘qemu64’ CPU model, which is highly buggy and undesirable for production usage. This changes the default to the recommended CPU mode ‘host-model’, for various benefits documented elsewhere.

  • When using RHSM Service (deployment/rhsm/rhsm-baremetal-ansible.yaml) based registration of the overcloud nodes and enabling the KSM using NovaComputeEnableKsm=True the overcloud deployment will fail because the RHSM registration and the ksm task run as host_prep task. The handling of enable/disable ksm is now handled in deploy step 1.

  • In case of cellv2 multicell environment nova-metadata is the only httpd managed service on the cell controller role. In case of tls-everywhere it is required that the cell controller host has ther needed metadata to be able to request the HTTP certificates. Otherwise the getcert request fails with “Insufficient ‘add’ privilege to add the entry ‘krbprincipalname=HTTP/cell1-cellcontrol-0….’”

13.0.0

Prelude

This change deprecates the novajoin and the composable service that enables TLS-Everywhere using novajoin. Instead, TLS Everywhere will be implemented using the tripleo-ipa ansible module.

New Features

  • Add parameter NovaAllowResizeToSameHost to allow instances to resize to the host they are currently on. Normally the source host is excluded.

  • To isolate LVM volumes created by compute guests, within Cinder volumes, from the LVM volumes created/managed by the host itself, a new task has been introduced to create an allowlist and denylist of devices which should be accessible (or not) to the host, configured in lvm.conf using the global_filter key. The allowlist is generated gathering the list of existing in-use physical disks (or partitions) and appending to it any user provided device passed via LVMFilterAllowlist parameter. The denylist is configured via LVMFilterDenylist and defaults to [‘.*’], which means it blocks any device not explicitly allowed. Both the list parameters can be specified per-role. The feature is, by default, disabled and can be enabled passing LVMFilterEnabled: true; when disabled the existing lvm.conf won’t be touched and a version of it which includes the global_filter will be left, for debugging, in /tmp/tripleo_lvmfilter.conf.

  • A new multipathd-container-ansible.yaml heat template replaces the multipathd-container.yaml template. The new template adds support for the following new parameters. * MultipathdSkipKpartx * MultipathdCustomConfigFile

  • Add parameters NovaLibvirtCPUMode, NovaLibvirtCPUModels and NovaLibvirtCPUModelExtraFlags to allow configuration of CPU related parameters libvirt/cpu_mode, libvirt/cpu_models and libvirt/cpu_model_extra_flags respectively.

  • Add GlanceSparseUploadEnabled to enable sparse image upload for rbd and filesystem glance backends. By default GlanceSparseUploadEnabled is disabled, it needs to be enabled while using sparse image upload for rbd and file stores.

  • Add a Heat parameter HorizonSessionTimeout to expose Puppet variable horizon::session_timeout, so operators can configure session timeout of horizon in seconds. It currently defaults to 1800 seconds.

  • The parameter NetworkDeletionPolicy (defaults to: delete) was added to enable control of the deletion_policy for overcloud network resources (networks, segments and subnets). By setting the parameter to retain the physical resources will not be deleted if removed from the heat templates, or the stack is deleted.

  • This changes network config generation with ansible, using the provided jinja templates for roles, the default. To keep using old heat templates for network config with``OS::TripleO::{{role.name}}::Net::SoftwareConfig`` mappings, use NetworkConfigWithAnsible: false parameter_defaults in an environment file.

  • A new parameter, NovaEnableVTPM, can be used to enable vTPM support in Nova.

  • The new SshServerOptionsOverrides parameter has been added. This parameter can be used to override a part of sshd_config, which is defined by the SshServerOptions.

Upgrade Notes

  • Cinder’s legacy “volume” service and its associated endpoints are automatically removed from the keystone catalog. The “volume” service is associated with Cinder’s v1 API, which was removed in Queens.

  • When upgrading from the multipathd-container.yaml template to the new multipathd-container-ansible.yaml template, bear in mind the new MultipathdSkipKpartx parameter will configure the corresponding skip_kpartx setting in /etc/multipath.conf.

  • This change deprecates novajoin and the service that depends on novajoin to enable TLS-Everywhere. From now on, TLS-Everywhere will be set up using the tripleo-ansible ansible module instead.

  • When the undercloud is upgraded, for TLS Everywhere systems, a new composable service will run to remove the novajoin containers.

  • A pre-upgrade validation has been written to ensure that some necessary permissions and ACIs have been added to the IPA server. As these changes require admin privileges, they cannot be automated in THT.

  • The environments/ssl/enable-internal-tls.j2.yaml file has been modified to automatically point to the new service that implements TLS-Everywhere using tripleo-ansible. Assuming you are adding this environment file to your templates (which is typically the case when setting up TLS-Everywhere) no other changes are required.

  • Now NotificationDriver is set to noop by default, as legacy telemetry services are disabled by default. Explicitly set NotificationDriver parameter to notifications from each services.

  • Network config generation with ansible is the default now. Please make necessary changes in the custom environment files and migrate the custom network config temmplates used. To keep using the old heat network config templates use NetworkConfigWithAnsible: false parameter_defaults in an environment file.

  • Use of the role name in jinja2 tripleo heat templates has been replaced with the use of role tags. Users of custom role data should update the tags for their custom roles data file adding the relevant tags. The following tag should be added, depending on the role:

    • Compute roles: add the compute tag.

    • HciCeph roles: add the compute tag.

    • DistributedCompute roles: add the compute tag.

    • Ceph roles: add the ceph and storage tag.

    • ObjectStorage roles: add the storage tag.

    • BlockStorage roles: add the storage tag.

    • ComputeOvsDpdk roles: add the ovsdpdk tag.

Deprecation Notes

  • The multipathd-container.yaml template is deprecated in favor of a new multipathd-container-ansible.yaml template. The new template is backward compatible with the old template, but see the features and upgrade notes for additional details.

  • This change deprecates novajoin, the service that deploys it on the undercloud, and the corresponding service that implements TLS-Everywhere using novajoin. TLS everywhere will be implemented from now on using the tripleo-ipa ansible module instead.

  • These services are novajoin-container-puppet.yaml and ipaclient-baremetal-ansible.yaml

  • On undercloud upgrade, a new composable service will remove the novajoin and novajoin-notifier containers from the undercloud.

  • Mistral services are deprecated and will be removed in a next release.

  • The old way of generating network configs with heat templates has been deprecated.

  • Use of the role name in jinja2 tripleo heat templates has been replaced with the use of role tags. By using tags the role name become truly customizable. To keep backward compatiblity tags will be added automatically to roles based on the role name until the next release.

Bug Fixes

  • As per launchpad bug 1855704, the lvmfilter task aims at hiding to the host the LVM2 volumes created by compute guests in Cinder volumes or Glance images.

  • When using the Shared File Systems service (manila), you may now use the Heat template parameter “ManilaEnabledShareProtocols” to configure the NAS protocols that users may use. If not set, the value is inferred per the storage backends that have been enabled.

  • The keystone catalog is automatically updated to remove any entries associated with Cinder’s v1 API “volume” service. This fixes bug 1897761.

  • Fixed the Octavia OctaviaTenantLogFacility setting default to 0 to align it with the project default.

12.7.0

New Features

  • Add parameter NovaVGPUTypesDeviceAddressesMapping provide mapping for multiple vgpu types and corresponding device addresses.

Bug Fixes

  • Ansible GroupVars incorrectly keept a single subnet prefix per-network. This caused a problem when multiple subnets using different subnet prefixes where defined. Resulting in the wrong subnet prefix being referenced in the NetworkConfig for roles.

    AnsibleHostVars stores networks subnet prefixes instead. See bug: 1895899.

12.6.0

New Features

  • Adds a new ContainerNovaLibvirtPidsLimit parameter in order to set the PIDs limit for nova_libvirt container. Defaults to 65536, set to 0 for unlimited.

  • This change updates the multiple-nics and multiple-nics-vlans templates so that an external bridge is created if either the role uses the External network or the “external_bridge” tag is set in the role definition. This is done instead of checking if the role name is “Controller”. This change also assigns the “external_bridge” tag to the Controller as well as the Compute roles so that both roles can access the Neutron external bridge for floating IPs or SNAT by default so that OVN can use DVR.

  • The following parameters were added to support configuration of gnocchi nfs backend.

    • GnocchiNfsEnabled

    • GnocchiNfsShare

    • GnocchiNfsOptions

  • The MariaDB tuning parameter for Innodb_buffer_pool_size can now be set via a new TripleO Heat Template parameter ‘MysqlInnodbBufferPoolSize’. By default this is undefined.

  • The following parameters are now role specific:

    • IronicConductorGroup

    • IronicCleaningNetwork

    • IronicProvisioningNetwork

    • IronicRescuingNetwork

    This allow operators to deploy the Baremetal service in a configuration with multiple conductor groups utilizing different networks for provisioning, cleaning and rescuing per conductor group or conductor.

  • The nova-ironic setting for ‘max_concurrent_builds’ can now be set via the use of a new TripleO Heat templates parameter ‘IronicMaxConcurrentBuilds’. It is set to the service default of 10 by default in TripleO Heat templates.

  • Add the NovaImageCacheTTL to the nova compute service. This exposes the remove_unused_original_minimum_age_seconds from nova.conf which controls the time (in seconds) that nova compute should continue caching an image once it is no longer used by and instances on the host. Defaults to 86400 (24hrs)

  • Floating IP port forwarding is now supported under ML2/OVN. A more detailed explanation can be found in bug 1877447.

  • Adding support for PowerFlex cinder driver.

  • Support for PowerStore backend cinder driver. Supports both iSCSI and FC protocols and support deploying one or multiple cinder PowerStore storage backends.

  • When SwiftRawDisks is set, try to mount the disks using uuids instead of paths. This makes mounts more stable, eg. if a kernel gets updates and device orders are changed.

Upgrade Notes

  • The “external_bridge” tag is now used for the Compute node. An external network bridge is required on the compute nodes in order to host floating IPs when using DVR. OVN deploys with DVR by default.

  • The CIDR for the StorageNFS network in the sample network_data_ganesha.yaml file has been modified to provide more usable IPs for the corresponding Neutron overcloud StorageNFS provider network. Since the CIDR of an existing network cannot be modified, deployments with existing StorageNFS networks should be sure to customize the StorageNFS network definition to use the same CIDR as that in their existing deployment in order to avoid a heat resource failure when updating or upgrading the overcloud.

  • Operators utilizing the firstboot/os-net-config-mappings.yaml should remove the resource registry entry. The template was replaced by a module in ansible. The data format and parameter NetConfigDataLookup is still used.

  • The deprecated``ManilaCephFSDataPoolPGNum`` and ManilaCephFSMetadataPoolPGNum parameters have been removed. As with the other Ceph pools, use the CephPoolDefaultPgNum parameter for the default value, or CephPools to override pg_num for specific pools.

Deprecation Notes

  • The firstboot template firstboot/os-net-config-mappings.yaml that was used to create an os-net-config mapping file has been deprecated and replaced by an ansible module. The parameter NetConfigDataLookup remains with no data format change for compatibility.

  • As the fast forward upgrade workflow to skip multiple releases now relies on the very same upgrade_tasks, there is no need to mantain the fast_forward_upgrade_tasks, as well as any of its references.

  • Resource OS::TripleO::Services::CinderBackendDellEMCVxFlexOS is deprecated. Use the new resource OS::TripleO::Services::CinderBackendDellEMCPowerFlex.

Bug Fixes

  • Designate producers will no longer be deployed in standalone mode and produce duplicates as they are now configured to coordinate via redis.

  • Fixed issue in the sample network_data_ganesha.yaml file where the IPv4 allocation range for the StorageNFS network occupies almost the whole of its CIDR. If network_data_ganesha.yaml is used without modification in a customer deployment then there are too few IPs left over in its CIDR for use by the corresponding overcloud Neutron StorageNFS provider network for its overcloud DHCP service. (See bug: #1889682)

  • Previously, HorizonDebug and Debug parameters change the value of horizon::django_debug. However, those parameters didn’t set DEBUG log level to horizon logger components. By this change, if those are true, horizon::log_level is set to ‘DEBUG’.

  • Fix Swift ring synchronization to ensure every node on the overcloud has the same copy to start with. This is especially required when replacing nodes or using manually modifed rings.

12.5.0

New Features

  • Add new BarbicanClient tripleo service for configuring DCN/Edge nodes to access a barbican service running in the control plane. The client service is disabled by default, and can be enabled by including the environments/services/barbican-edge.yaml environment file when deploying a DCN/Edge stack.

  • Added new PublicTLSCAFile parameter, that is used to set the ca cert in clouds.yaml for keystone public endpoint. This defaults to empty string (‘’) assuming that the certs are already trusted.

  • Add GlanceImagePrefetcherInterval parameter to run periodic job which fetches the queued images for caching in cache directory, when image cache is enabled.

  • Add boolean parameter NovaSchedulerQueryPlacementForAvailabilityZone that sets scheduler/query_placement_for_availability_zone parameter. It allows the scheduler to look up a host aggregate with metadata key of availability zone set to the value provided by incoming request, and request result from placement be limited to that aggregate. Default value for NovaSchedulerQueryPlacementForAvailabilityZone is false.

  • Adds the “OctaviaLogOffload” setting to enable amphora log offloading.

  • Added TCP Segmentation Offload (TSO) support for ovs-dpdk.

  • Added support for VxFlexOS cinder block storage backend driver

  • A new Heat parameter ‘ZaqarWsTimeout’ exposes the Puppet variable ‘tripleo::haproxy::zaqar_ws_timeout_tunnel’. This allows operators to configure the Mistral API timeout. It currently defaults to four hours.

Deprecation Notes

  • Usage of the option NeutronFirewallDriver which was used to set firewall_driver config option in the Neutron server’s config is now deprecated. Firewall driver should be set per agent in the agent’s config. It can be done using NeutronOVSFirewallDriver option. Option in the Neutron server was in there just for backward compatybility reasons but since Newton release all Neutron agents are reporting to the server what firewall driver is used so there is no need to keep this legacy, server side option anymore.

  • Paunch was deprecated in Ussuri and is now being retired, to be fully replaced by the new tripleo-ansible role, tripleo_container_manage.

  • Support for Neutron FUJITSU plugin has been removed.

  • Resource OS::TripleO::Services::CinderBackendScaleIO is no longer supported. Use the new resource OS::TripleO::Services::CinderBackendDellEMCVxFlexOS.

  • NeutronSriovNumVFs parameter to configure the VFs for SR-IOV is removed. Instead, use the network interface types sriov_pf in the nic configs to configure the VFs for SR-IOV devices.

Bug Fixes

  • Ensure the barbican Key Manager settings are configured on DCN/Edge nodes when the barbican service is deployed in the control plane. See bug 1886070.

Other Notes

  • Ensure we’re using only /run location, since /var/run is a symlink to /run

  • The ValidateNtp has been removed from the all nodes validation configuration. During the time sync configuration we already do a check to ensure the ntp servers are available. If they are not we will fail with an appropriate message. The ValidateNtp option came from a time before we could fail in a more explicit way.