Zed Series Release Notes



The OVN changed support for NAT rules including a new column and auto-discovery logic to know about logical router gateway ports for NAT on a Logical Router.

New Features

  • A new OVN driver Northbound DB column has been added to allow configuring gateway port for NAT rule. If the OVN backend supports the gateway_port column in the Northbound DB NAT table, the gateway port uuid will be configured to any floating IP to prevent North/South traffic issues. Previously created FIP rules will be updated only once during the maintenance task to include the gateway_port reference (if OVN backend supports it). In case all FIP entries are already configured no maintenance action will be performed.


New Features

  • Remote address group support was added to the iptables-based firewall drivers (IptablesFirewallDriver and OVSHybridIptablesFirewallDriver), Previously it was only available in the OVSFirewallDriver. For more information, see bug 2058138.

Known Issues

  • The fix of bug 2048785 only fixes newly created trunk parent ports. If the fix of already existing trunks is needed, then either delete and re-create the affected trunks or set tpt ports’ vlan_mode and tag manually: ovs-vsctl set Port tpt-... vlan_mode=access tag=0

Bug Fixes

  • The config option agent_down_time is now limited to a maximum value of 2147483, as neutron-server will fail to start if it is configured higher. See bug 2028724 for more information.

  • [bug 2036423] Now it is not possible to delete a subnet gateway IP if that subnet has a router interface; the subnet gateway IP modification was already forbidden.

  • When synchronizing the OVN databases, either when running the migration command or during startup, the code responsible for synchronization will only clean up segment-to-host mappings for hosts with agent_type OVN Controller agent. Before, the synchronization would clean up (delete) segment-to-host mappings for non-OVN hosts. Fixes bug: 2040172.

Other Notes

  • Added extension subnetpool-prefix-ops to the ML2/OVN mechanism driver.


Known Issues

  • When using ML2/OVN, during an upgrade procedure, the OVS system-id stored value can be changed. The ovn-controller service will create the “Chassis” and “Chassis_Private” registers based on this OVS system-id. If the ovn-controller process is not gracefully stopped, that could lead to the existence of duplicated “Chassis” and “Chassis_Private” registers in the OVN Southbound database.

Bug Fixes

  • [bug 2022914] Neutron-API supports using relays as the southbound connection in a ML2/OVN setup. Before the maintenance worker of the API required a leader_only connection, which was removed.

  • Fixed the scenario where the DHCP agent is deployed in conjunction with the OVN metadata agent in order to serve metadata for baremetal nodes. In this scenario, the DHCP agent would not set the route needed for the OVN metadata agent service resulting in baremetal nodes not being able to query the metadata service. For more information see bug 1982569.

  • For OVN versions v22.09.0 and above, the mcast_flood_reports option is now set to false on all ports except “localnet” types. In the past, this option was set to true as a workaround for a bug in core OVN multicast implementation.

  • Now the ML2/OVN trunk driver prevents a trunk creation if the parent port is already bound. In the same way, if a parent port being used in a trunk is bound, the trunk cannot be deleted.

  • During the port bulk creation, if an IPAM allocation fails (for example, if the IP address is outside of the subnet CIDR), the other IPAM allocations already created are deleted before raising the exception. Fixes bug 2039550.

  • A new OVN maintenance method remove_duplicated_chassis_registers is added. This method will periodically check the OVN Southbound “Chassis” and “Chassis_Private” tables looking for duplicated registers. The older ones (based on the “Chassis_Private.nb_cfg_timestamp” value) will be removed when more than one register has the same hostname, that should be unique.

Other Notes

  • The external_mac entry in the NAT table is used to distribute/centralize the traffic to the FIPs. When there is an external_mac set the traffic is distributed (DVR). When it is empty it is centralized through the gateway port (no DVR). Upon port status transition to down, the external_mac was removed regardless of DVR being enabled or not, leading to centralize the FIP traffic for DVR – though it was for down ports that won’t accept traffic anyway.

  • Adds a maintenance task that runs once a day and is responsible for cleaning up Hash Ring nodes that haven’t been updated in 5 days or more. See LP #2033281 for more information.

  • Added the missing extension uplink-status-propagation to the ML2/OVN mechanism driver. This extension is used by the ML2/SR-IOV mechanism driver, that could be loaded with ML2/OVN. Now it is possible to create ports with the “uplink-status-propagation” flag defined.


Known Issues

  • The high availability of metadata service on isolated networks is limited or non-existent. IPv4 metadata is redundant when the DHCP agent managing it is redundant, but recovery is tied to the renewal of the DHCP lease, making most recoveries very slow. IPv6 metadata is not redundant at all as the IPv6 metadata address can only be configured in a single place at a time as it is link-local. Multiple agents trying to configure it will generate an IPv6 duplicate address detection failure.

    Administrators may observe the IPv6 metadata address in “dadfailed” state in the DHCP namespace for this reason, which is only an indication it is not highly available. Until a redesign is made to the isolated metadata service there is not a better deployment option. See bug 1953165 for information.

  • The redirect-type=bridged option is only used if all the tenant networks connected to the router are of type VLAN or FLAT. In this case their traffic will be distributed. However, if there is a mix of VLAN/FLAT and geneve networks connected to the same router, the redirect-type option is not set, and therefore the traffic for the VLAN/FLAT networks will also be centralized but not tunneled.

Bug Fixes

  • 1986003 Fixed an issue with concurrent requests to activate the same port binding where one of the requests returned a 500 Internal Server Error. With the fix one request will return successfully and the other will return a 409 Conflict (Binding already active). This fixes errors in nova live-migrations where those concurrent requests might be sent. Nova handles the 409/Conflict response gracefully.

  • Fix an issue in the OVN driver where network metadata could become unavailable if the metadata port was ever deleted, even if accidental. To re-create the port, a user can now disable, then enable, DHCP for one of the subnets associated with the network using the Neutron API. This will try and create the port, similar to what happens in the DHCP agent for ML2/OVS. For more information, see bug 2015377.

  • [bug 2003455] As part of a previous commit (https://review.opendev.org/c/openstack/neutron/+/875644) the redirect-type=bridged option was set in all the router gateway ports (cr-lrp ovn ports). However this was breaking the N/S traffic for geneve tenant networks connected to the provider networks through those routers with the redirect-type option enabled. To fix this we ensure that the redirect-type option is only set if all the networks connected to the router are of VLAN or FLAT type, otherwise we fall back to the default option. This also means that if there is a mix of VLAN and geneve tenant networks connected to the same router, the VLAN traffic will be centralized (but not tunneled). If the traffic for the VLAN/FLAT needs to be distributed, then it should use a different router.


New Features

  • Address scope is now added to all OVN LSP port registers in the northbound. Northd then writes the address scope from the northbound to the southbound so it can be used there by the ovn-bgp-agent.

Known Issues

  • Until the OVN bug (https://bugzilla.redhat.com/show_bug.cgi?id=2162756) is fixed, setting the “reside-on-redirect-chassis” to true for the logical router port associated to vlan provider network is needed. This workaround makes the traffic centrallized, but not tunneled, through the node with the gateway port, thus avoiding MTU issues.

Upgrade Notes

  • The default value for the metadata_workers configuration option has changed to 0 for the ML2/OVN driver. Since [OVN] Allow to execute “MetadataProxyHandler” in a local thread, the OVN metadata proxy handler can be spawned in the same process of the OVN metadata agent, in a local thread. That reduces the number of OVN SB database connections to one.

Bug Fixes

  • [bug 2003455] It is added an extra checking to ensure the “reside-on-redirect-chassis” is set to true for the logical router port associated to vlan provider network despite having the “ovn_distributed_floating_ip” enabled or not. This is needed as there is an OVN bug (https://bugzilla.redhat.com/show_bug.cgi?id=2162756) making it not work as expected. Until that is fixed, we need these workaround that makes the traffic centrallized, but not tunneled, through the node with the gateway port, thus avoiding MTU issues.

  • Normalise OVN agent heartbeat timestamp format to match other agent types. This fixes parsing of GET /v2.0/agents for some clients, such as gophercloud.

  • Neutron can record full connection using log-related feature introduced in OVN 21.12. For more info see bug LP#<https://bugs.launchpad.net/neutron/+bug/2003706>

Other Notes

  • Since OVN 20.06, the “Chassis” register configuration is stored in the “other_config” field and replicated into “external_ids”. This replication is stopped in OVN 22.09. The ML2/OVN plugin tries to retrieve the “Chassis” configuration from the “other_config” field first; if this field does not exist (in OVN versions before 20.06), the plugin will use “external_ids” field instead. Neutron will be compatible with the different OVN versions (with and without “other_config” field).



Introduce the experimental features framework.

New Features

  • Some Neutron features are not supported due to lack of resources or technical expertise to maintain them. As they arise, those features will be marked as experimental by the Neutron core team. Deployers will be able to continue using experimental features by explicitly enabling them in the ‘experimental’ section of neutron.conf. The ML2 linuxbridge driver is the first feature to be marked as experimental. To continue using it, deployers have to set to True the ‘linuxbridge’ option in the ‘experimental’ section of neutron.conf.

  • Add support for port ranges in the port forwarding rules. The supported ranges are N:M with N <= M. Also, the ranges of internal and external ports relation must be: internal range = external range or internal range = 1.

  • After the port is considered as provisioned, the Nova port binding update could have not been received, leaving the port as not bound. Now the port provisioning method has an active wait that will retry several times, waiting for the port binding update. If received, the port status will be set as active if the admin state flag is set.

  • Support for IPv6 NDP proxy has been added. Read the related specification for more details.

  • Support for baremetal provisioning using OVN’s built-in DHCP server has been added for IPv4.

  • Added support for QoS minimum bandwidth rules (egress only) in ML2/OVN. OVN supports setting these rule types in the logical switch ports since release 22.06.0.

  • OVN mechanism driver refuses to bind a port to a dead agent.

  • Core OVN now can set the destination host on the logical switch port during a live migration. That allows to prepare the destination host earlier, achieving a quicker live migration and a lower downtime during the switch between hosts. Neutron includes this information in the port options.

  • Added support for router gateway IP QoS in OVN backend. The L3 OVN router plugin now can apply router QoS policy rules on the router gateway port.

  • Ovn configuration items “ovn_nb_connection” and “ovn_sb_connection” can set multiple addresses separated by commas. Setting NB/SB “connection” inactivity probe can also work well, if multiple connection be specified.

  • Added a new configuration variable, in [OVS] section, to control the OVS OpenFlow rule processing operations when using the OVS native firewall driver (securitygroup.firewall_driver=openvswitch):

    • openflow_processed_per_port: by default “False”. If enabled, all OpenFlow rules associated to a port will be processed at once, in a single transaction. If disabled, the flows will be processed in batches of “AGENT_RES_PROCESSING_STEP=100” number of OpenFlow rules.

  • If uplink-status-propagation extension is enabled, all existing ports before enabling it will have the flag “propagate_uplink_status” enabled by default. This is aligned with the aim of an administrator that enables this extension. Now only new ports can be created with this flag disabled.

  • Gateway IP QoS network inheritance is now available for OVN L3 plugin QoS extension. If the router external network (gateway network) has a QoS policy associated, the gateway IP port will inherit the network QoS policy.

  • QoS rule type list accepts two filter flags:

    • all_supported: if True, the listing call will print all QoS rule types supported by at least one loaded mechanism driver.

    • all_rules: if True, the listing call will print all QoS rule types supported by the Neutron server.

    Both filter flags are exclusive and not required.

  • Enabled DbQuotaDriverNull as production ready database quota driver. This driver does not have access to the database and will return empty values to the request queries. This driver can be used to override the Neutron quota engine.

  • A new script to remove the duplicated port bindings was added. This script will list all ml2_port_bindings records in the database, finding those ones with the same port ID. Then the script removes those ones with status=INACTIVE. This script is useful to remove those leftovers that remain in the database after a failed live migration. It is important to remark that this script should not be executed during any live migration process.

  • Add use_random_fully setting to allow an operator to disable the iptables random-fully property on an iptable rules.

Known Issues

  • If the use_random_fully setting is disabled, it will prevent random fully from being used and if there’re 2 guests in different networks using the same source_ip and source_port and they try to reach the same dest_ip and dest_port, packets might be dropped in the kernel do to the racy tuple generation . Disabling this setting should only be done if source_port is really important such as in network firewall ACLs and that the source_ip are never repeating within the platform.

Upgrade Notes

  • Previously deprecated configuration option allow_overlapping_ips is now removed.

  • Python 3.6 & 3.7 support has been dropped. The minimum version of Python now supported is Python 3.8.

  • A new configuration option called [ovn]/disable_ovn_dhcp_for_baremetal_ports has been added to ML2/OVN for IPv4. Since PXE booting nodes can be very sensitive depending on the hardware and some operators may prefer to use a fully-fledged DHCP server instead of OVN’s DHCP server this option allows for disabling OVN’s built-in DHCP server for baremetal ports (vnic type “baremetal”) when set to True. It defaults to False.

Deprecation Notes

  • The ML2 linuxbridge agent has been marked as experimental due to lack of resources to maintain it. To continue using it, deployers have to set to True the ‘linuxbridge’ option in the ‘experimental’ section of neutron.conf

Bug Fixes

  • 1942329 Port binding logic for direct-physical ports has been extended to allow providing the MAC address of the physical device via the binding profile. If it is provided then Neutron overwrites the value of the device_mac_address field of the port object in the database with the value from the active binding profile. If there are ports bound before the nova side of this fix is depolyed then the VM using the port needs to be moved or the port needs to be detached and re-attached to force nova to provide the MAC address of the direct-physical port in the port binding.

  • Forbid the creation of a duplicate NDP proxy entry on the same router, since the IP address of a router is unique and an IPv6 address only needs one NDP proxy.

  • Fixes an issue in the ML2/OVN driver where the network segment tag was not being updated in the OVN Northbound database. For more information, see bug 1944708.

Other Notes

  • The OVN migration performs validation by default. This validation means an instance is spawned and is tested by simple ping after the migration is finished. Also it tries to create new workload post migration. This is useful for very simple scenarios when migration is tested but is not really useful in production since likely the production envrionments already have running workloads. It makes more sense to require the validation explicitly rather than implicitly run it as the migration is mostly intended for production. The VALIDATE_MIGRATION now defaults to False and needs to be changed to True if validation upon request.

  • From now on, gateway interface will be kept up on all nodes where HA router is hosted, regardless of their state (active or standby). For more information see bug 1952907.

  • OVN driver reverted to using stateful NAT for floating IP implementation. The previous switch to stateless didn’t materialize the expected performance benefits and instead introduced problems with potential hardware offloading.