Current Series Release Notes

30.0.0.0b1-51

Known Issues

  • Live migrating an instance with an InfiniBand port needs [compute] live_migration_wait_for_vif_plug = False in Nova’s configuration on the compute hosts that can receive it. Nova otherwise waits for a network-vif-plugged event that nothing sends for an InfiniBand port before the guest is copied, and the migration fails when [DEFAULT] vif_plugging_timeout expires. See the live migration page of the documentation.

Upgrade Notes

  • neutron-server now refuses to start if [sdn] bind_normal_ports is enabled and [sdn] bind_normal_ports_physnets is empty. Previously, with [sdn] physical_networks set to *, the default, it started but the mlnx_sdn_assist mechanism driver bound no normal ports. Set [sdn] bind_normal_ports_physnets to the InfiniBand physical networks, or disable [sdn] bind_normal_ports.

  • Ports on VLAN networks of mlnx_infiniband fail to bind unless mlnx_sdn_assist is enabled, with [sdn] sync_enabled and the network’s physical network in [sdn] physical_networks. If the subnet manager’s partition configuration matches the networks’ segmentation IDs by other means, or the virtual functions are ConnectX-3 ones, set the new [mlnx] allow_unenforced_pkeys option to True.

  • Support for NVIDIA (Mellanox) NEO as the SDN controller of mlnx_sdn_assist has been removed. NEO has reached end of life; see EOL notice LCR-000668. UFM is the only supported controller.

  • mlnx_sdn_assist no longer binds bare metal ports without a DHCP client ID, such as Ethernet ports described only by their local link information, on any network. Deployments that relied on it to bind them, including on flat networks without an SDN controller, must enable another mechanism driver that does, such as baremetal from networking-baremetal.

  • The mlnx_sdn_assist mechanism driver now sends a bare metal node’s InfiniBand port to the SDN controller when its binding is committed, rather than when its host is set.

  • Requests that failed because the SDN controller was unreachable are also retried with the longer delays, so after a controller outage some of them can take up to five minutes to be sent once the controller is back. Later operations on the same networks and ports wait for them.

  • When [sdn] sync_enabled is true, neutron-server now refuses to start if the networking-mlnx database schema is out of date, and names the command to run: neutron-db-manage --subproject networking-mlnx upgrade head. It checks that the sdn_journal.state column accepts every SDN journal state. Deployments that missed the migration adding the monitoring state could not record it, so the journal row of every network or port operation that started a UFM job was sent to UFM again and again and never deleted, the sdn_journal table kept growing, and later operations on the same networks and ports were held behind those rows. These rows were sent again without counting a retry, so [sdn] retry_count did not limit them. After the schema is upgraded, they are sent to UFM once more, then completed and deleted as usual.

  • A database migration adds a seqnum column, and a unique constraint on object_uuid and seqnum, to the sdn_journal table. Run neutron-db-manage --subproject networking-mlnx upgrade head.

  • Projects can no longer set the client-id DHCP option of direct or baremetal ports on networks that mlnx_sdn_assist syncs.

  • Instances with InfiniBand ports on VLAN networks now fail to boot if the SDN controller doesn’t add their ports to the partition key. This relies on the Nova defaults [DEFAULT] vif_plugging_is_fatal = True and a non-zero vif_plugging_timeout. If the controller can’t be reached, boots fail when the timeout expires. Check that the controller is reachable and that the sdn_journal table has no rows stuck in the pending or monitoring state before upgrading.

  • The mlnx_sdn_assist mechanism driver now sends an instance’s direct port to the SDN controller when its binding is committed, rather than each time ML2 tries to bind it, so failed binding attempts no longer send the port. When Nova live migrates an instance, the port is now sent for the destination host when its binding is activated.

  • When [sdn] sync_enabled is True and [sdn] physical_networks doesn’t contain *, creating a network with segments both on synced and on unsynced physical networks, or adding such a segment to a network, now fails.

  • Failed mlnx_sdn_assist journal entries are now deleted after [sdn] completed_rows_retention, like completed ones.

  • mlnx_sdn_assist now also sends the SDN controller the creation of networks without a segmentation ID: flat networks, and networks with several segments. It already sent their updates, deletion and ports, but not their creation, since 2016, for a reason that wasn’t recorded. A controller that rejects such a creation now logs a failed journal row for each one; nothing else waits on it.

  • Networks created before the upgrade aren’t sent again. A controller that checks a port’s network, such as the alternative fabric manager, rejects their ports until it learns of the network. Updating the network, for example its description, sends it, which the alternative fabric manager takes as its creation.

  • mlnx_sdn_assist now only sends the SDN controller ports that have a DHCP client ID, by which UFM identifies an InfiniBand port, once they are bound to a network that it syncs. Other ports, such as Ethernet ports and networks’ DHCP and router ports, are no longer sent, and if earlier releases sent any of them, they aren’t deleted from the controller either.

  • The DHCP client ID of a port bound to a network that mlnx_sdn_assist syncs can no longer be changed, added or removed; unbind the port first. Previously the port stayed up while the controller moved it. This includes Ironic changing the client-id of an active node’s port, which now fails.

Security Issues

  • On ConnectX-3 adapters, when a port’s partition key is not in the physical function’s pkey table, eswitchd now removes the virtual function from its partition and reports a failure, so the port isn’t reported up. Previously it did nothing, so a virtual function configured again without being plugged again kept the partition key it had before.

  • eswitchd now disables the link of a virtual function when it is plugged for a port, and only enables it when neutron-mlnx-agent asks with the port’s MAC address and the virtual function has the GUID derived from it. A virtual function left with another port’s GUID, for example after Nova ignored a failed ebrctl add-port, can no longer reach the fabric. Failures to enable or disable a link are now reported to the agent, rather than logged and reported as successful.

  • neutron-mlnx-agent now sets the link of every assigned virtual function on each polling iteration, rather than only when it thinks the state changed, so a link that Nova disabled by plugging the virtual function again is enabled again. When it can’t get its ports from the Neutron server, it disables every link except those it enabled for the same port, including after it restarts. It also disables the link of a virtual function that is no longer assigned.

  • neutron-mlnx-agent now enables the link of an InfiniBand virtual function only while its port is ACTIVE, administratively up and bound to the agent’s host, and only if the virtual function has the port’s GUID. ML2 makes a port ACTIVE only once nothing holds it, which includes the SDN controller adding it to its partition key. The agent disables the link when any of this stops being true. So an instance can’t reach the fabric outside its partition, even where Nova runs it without waiting for its port, for example when it attaches an interface, migrates or resumes an instance, or when vif_plugging_is_fatal is False. Instances get network access once their ports are ACTIVE, within one agent polling interval.

  • neutron-mlnx-agent no longer reports a port up when it fails to configure it, so the port can’t become ACTIVE, and it carries on with the other ports.

  • mlnx_infiniband now treats a VLAN segment’s partition key as enforced only if mlnx_sdn_assist syncs the physical networks of all the network’s segments, which is when it sends the network’s ports to the SDN controller. Previously a port on a network with segments on several physical networks could be bound while the SDN driver ignored it, and so was never added to its partition key.

  • The mlnx_infiniband mechanism driver no longer binds a port to a VLAN segment unless the mlnx_sdn_assist mechanism driver syncs the segment’s physical network to an SDN controller, which adds the port to the segment’s partition key. Previously the port was bound, and its virtual function used the default partition, whatever the segmentation ID. It also no longer binds a port if another port with the same MAC address, on another network, is bound as ib_hostdev, since both virtual functions would have the same GUID.

  • A bare metal node’s InfiniBand port (VNIC type baremetal with a DHCP client ID) on a VLAN network now stays DOWN until the SDN controller’s job that adds it to the network’s partition key has completed, and is set to ERROR if the controller fails. Previously mlnx_sdn_assist made the port ACTIVE as soon as it was bound. Ironic waits for its ports to be ACTIVE before it powers a node on into the tenant network, when [neutron] fail_on_port_binding_failure is True, its default, so the deployment now fails instead. A port in ERROR fails the deployment when Ironic’s wait times out, which is set by [agent] neutron_agent_max_attempts and [agent] neutron_agent_status_retry_interval.

  • mlnx_sdn_assist now rejects a DHCP client ID set or changed by a project on any port of a network it syncs, not only on direct and baremetal ports, since a port’s VNIC type can change later. It accepts client IDs from users with the service role as well as from administrators. A port with more than one client ID option can’t be bound, nor can a direct InfiniBand port whose client ID isn’t the one derived from its MAC address, from which its virtual function’s GUID is derived. The deletion of every direct port is now sent to the SDN controller, whatever its device owner.

  • Failed deletions of ports are no longer removed from the sdn_journal table after [sdn] completed_rows_retention, so ports with the same GUID keep being set to ERROR until an operator has checked that the GUID was removed from its partition key and deleted the row.

  • mlnx_sdn_assist now rejects a request that sets or changes the DHCP client ID of a direct or baremetal port unless it comes from an administrator. The SDN controller takes an InfiniBand port’s GUID from its client ID, so a project that set it could have had another port’s GUID added to its network’s partition key. Nova, Ironic and mlnx_infiniband set client IDs as administrators. A bare metal node’s InfiniBand port is no longer sent when it is created, only when its binding is committed.

  • Every direct InfiniBand port is now held DOWN until the SDN controller has added it to its partition key, whatever its device owner. Previously only instances’ ports were.

  • When the mlnx_sdn_assist mechanism driver syncs to an SDN controller, an instance’s InfiniBand port (VIF type ib_hostdev) on a VLAN network now stays DOWN until the controller’s job that adds the port to the network’s partition key has completed. If the controller rejects the port, its job fails, it doesn’t implement the request, it doesn’t return a job, or the retries run out, the port is set to ERROR. Nova then fails the boot, at the latest when vif_plugging_timeout expires. Previously the port could become ACTIVE, and the instance could start, while the port was still in the default partition, where it can reach other tenants’ ports.

  • The mlnx_sdn_assist journal now sends a port’s POST to the SDN controller only once the deletion of any other port with the same GUID has finished. A bare metal node’s provisioning and tenant ports have the same DHCP client ID, and ports with the same MAC address on different networks give virtual functions the same GUID. Previously a tenant port could be added to its partition key, and made ACTIVE, while the controller was still removing, or had failed to remove, the same GUID from another partition. If the controller failed to delete the other port, an instance’s or bare metal node’s port that is held DOWN is now set to ERROR instead of being sent.

  • mlnx_sdn_assist now refuses a segment that would give a network segments both on physical networks in [sdn] physical_networks and on others. It only sends a network’s ports to the SDN controller while all the network’s segments are on synced physical networks, so adding a segment on an unsynced physical network to a synced network stopped its ports from being sent, including their deletion, which left their GUIDs in the network’s partition key. Networks that already mix them aren’t synced; check for them after upgrading, and for ports whose GUIDs were left in partition keys.

Bug Fixes

  • Fixed a race in neutron-mlnx-agent where a port update received from the Neutron server while the agent was scanning ports could be lost, so the port’s binding was not refreshed until the next update or resync.

  • The mlnx_infiniband mechanism driver no longer replaces a client-id DHCP option set by the user when it binds a port, and keeps the port’s other extra DHCP options in the port it passes to other mechanism drivers and returns from the API.

  • If [sdn] bind_normal_ports was enabled, [sdn] physical_networks listed physical networks and [sdn] bind_normal_ports_physnets was empty, neutron-server failed to start with an unhelpful TypeError. It now reports that [sdn] bind_normal_ports_physnets must be set.

  • The mlnx_infiniband mechanism driver no longer adds the InfiniBand client-id DHCP option to every instance port, including ports on other networks bound by other mechanism drivers such as OVN. It now adds it only when a live Mellanox agent on the port’s host can bind the port’s network.

  • On ConnectX-3 adapters, releasing a virtual function now sets its partition key to the invalid one, without logging an error.

  • eswitchd no longer exits when it receives a message that is not a JSON object, such as a stray connection to its ZeroMQ port. It now logs an error and replies with a failure.

  • After a live migration, neutron-mlnx-agent now reports the port of the virtual function that the instance has on the destination host, rather than one it had there before, so the port becomes ACTIVE. The agent also releases removed virtual functions before configuring new ones, so a virtual function given to another port within one polling interval keeps the new port’s configuration.

  • Ports bound by the mlnx_infiniband mechanism driver now stay DOWN until neutron-mlnx-agent reports that it has configured the virtual function. Previously the driver never added the L2 agent provisioning block when a port was bound to a host, so the port could become ACTIVE, and Nova could start the instance, before the agent had configured the virtual function.

  • neutron-mlnx-agent no longer stops processing ports when it fails to get the virtual functions from eswitchd, for example when eswitchd reports an error. It now tries again after the polling interval, and keeps the port updates it received meanwhile. Previously its main loop stopped while the agent was still reported as alive.

  • When the SDN controller can’t be reached, or doesn’t answer in time, mlnx_sdn_assist’s journal now handles it as a connection error, and retries the request. Previously the error ended the journal’s whole sync run, and a request being sent was left in the processing state until the cleanup reset it.

  • A port’s POST now only waits for deletions of ports with the same GUID that were recorded no later than it, so that operations recorded during a controller outage can’t wait for each other forever.

  • When a UFM job fails, the SDN journal now counts sending the request again against [sdn] retry_count, as it does for other failures, and marks the row failed when the count is reached. Previously the request was sent again indefinitely, whatever retry_count was set to.

  • The SDN journal now waits longer before each new attempt to send a request that has failed: at least one [sdn] sync_timeout before the first retry, doubling with each further retry up to five minutes, or one [sdn] sync_timeout if that is longer. Previously a request that kept failing, for example because its controller job kept failing, was sent again on every sync. With the default [sdn] retry_count of -1 such a request is still retried indefinitely, but at most every five minutes, or every [sdn] sync_timeout if that is longer.

  • The mlnx_sdn_assist journal now orders the operations on a network or port by a sequence number recorded with each operation, rather than by their creation time, which only has a resolution of a second. Operations recorded in the same second, such as the DELETE and POST of a port that moves to another host, or two POST operations of a port that is bound twice in a second, are now sent in the order they were recorded. Previously a port could be left DOWN, or out of its partition key, in these cases. Operations recorded before the upgrade are sent first, in the old order.

  • The mlnx_sdn_assist mechanism driver now waits for the SDN controller to finish a job before sending operations that depend on it. For example, it used to delete a network while UFM was still running the job that removes the network’s ports, which UFM rejected with Network ID cannot be removed because it is associated to one or more hosts. Similarly, a port could be sent before UFM had finished creating its network. Dependent operations now wait for as long as the job runs. If the job status can’t be read, the request is sent again, up to [sdn] retry_count times.

  • The SDN journal no longer updates a row that is waiting for another operation, such as a network deletion waiting for its ports’ deletions, on every sync. With row-based binary logging and the default binlog_row_image=FULL, each update logged the whole row, including the network or port data it holds, so waiting rows added to the database’s binary log volume for as long as they waited.

  • When mlnx_sdn_assist’s journal can’t reach the SDN controller to check a job, it now keeps monitoring the job and checks it again later. Previously it sent the request again, which started another job on the controller although the first one may have been fine. When the controller answers a job check with an error, for example because it no longer knows the job, the request is still sent again, and this now counts towards [sdn] retry_count.

  • The mlnx_sdn_assist journal now locks a port’s binding while it releases the port, or sets it to ERROR, after the SDN controller’s job for the port has finished. Previously, if the port was rebound at the same moment, the result of the job for the old binding could release the new binding before the controller had added it to the partition key.

  • When the SDN controller accepts a create or update request but returns no job to monitor, mlnx_sdn_assist’s journal now records the request as done. Previously the request stayed in the processing state until the cleanup reset it, and was then sent again. A deletion without a job is sent again, counting towards [sdn] retry_count, since nothing shows it was done. A request that holds an InfiniBand port DOWN until it is in its partition key still fails, and sets the port to ERROR.

  • The mlnx_sdn_assist mechanism driver now recognises a port’s DHCP client ID when the option is set by its number, 61, as Ironic does. Neutron stores option names as strings and the driver only matched the integer, so it never found the client ID of such bare metal InfiniBand ports and did not send them to UFM unless they had local link information.

  • When a port moves to another host, the mlnx_sdn_assist journal now sends the DELETE of the port’s old binding to the SDN controller before the POST of its new one, even when both were recorded in the same second. Previously the two could be sent in either order, and the DELETE could remove the port from its partition key after the POST had added it.

  • The mlnx_sdn_assist mechanism driver no longer retries a create or update request that the SDN controller rejects with HTTP 400 or 422, such as UFM’s Missing attribute(s): client-id for a port without an InfiniBand client ID. These were treated as connection errors, so with the default [sdn] retry_count of -1 the request was retried forever, and each retry stopped the rest of the journal from being synced until the next sync interval. The journal entry is now marked as failed and the error is logged. To send a port again, fix it and update it. To send a network again, delete and recreate it. Rejected deletions are still retried, because the controller can reject them only until the resources that use them are removed.

  • When the mlnx_sdn_assist journal fails to release a port after the SDN controller’s job for it has completed, for example because of a database deadlock, it now tries again on its next run. Previously it marked the journal row as completed first, so the port stayed DOWN.

  • The SDN controller no longer gets ports, and updates, of flat networks and networks with several segments that it was never told were created.

  • A request that would bind a direct port (VIF type ib_hostdev) on a VLAN network that mlnx_sdn_assist syncs now fails if the port has no DHCP client ID, since the controller can’t add it to its partition key, and the port stays unbound. Previously the controller rejected the port and set it to ERROR.

  • mlnx_sdn_assist no longer asks the controller to delete or update ports that it never sent, such as a port whose binding failed, and now deletes the old port when a port update makes ML2 bind it again on the same host.

  • When a port is unbound, the mlnx_infiniband mechanism driver now removes the client-id DHCP option only if it has the value the driver generates, so a client-id set by the user is kept. Ports that were wrongly given the driver’s client-id have it removed when they are next unbound.

  • The mlnx_infiniband mechanism driver no longer deletes all of a port’s extra DHCP options when the port is unbound. Only the client-id option that the driver added is removed.

  • The mlnx_infiniband mechanism driver now removes the client-id DHCP option from a port when it is unbound, for example when its instance is deleted. Previously the option was left on the port.

Other Notes

  • neutron-mlnx-agent now asks the Neutron server for the direct ports bound to its host on every polling iteration while virtual functions are attached, and handles binding_activate notifications for live migrated instances.

30.0.0.0b1

Upgrade Notes

  • The minimum versions of networking-mlnx’s requirements now match neutron 29.0.0. This raises the minimum versions of neutron-lib to 5.0.0, oslo.config to 10.2.0, oslo.privsep to 2.3.0, pbr to 6.0.0, pyroute2 to 0.7.3 and SQLAlchemy to 1.4.23.

  • Following the lead from Neutron, support for the eventlet library has been dropped. Although not a user facing change, users should be on the lookout for multi-threading related bugs now that networking-mlnx uses native threads.

  • The neutron package is no longer a required dependency. It has been moved to an optional extra called neutron. This allows installing networking-mlnx without replacing or upgrading an existing Neutron installation.

    For installations where Neutron is not already present, install with the extra, which requires neutron 29.0.0 or later:

    pip install 'networking-mlnx[neutron]'

    Existing deployments that install neutron separately (e.g. via distribution packages, Kolla images or devstack) are not affected.

  • networking-mlnx itself no longer requires python-openstackclient, python-neutronclient, Babel, defusedxml or netaddr, which it does not use. Install python-openstackclient separately if you relied on networking-mlnx to provide the openstack command.

  • The libraries that networking-mlnx imports directly are now declared as requirements: alembic, oslo.db, oslo.i18n, oslo.log, oslo.messaging, oslo.serialization, oslo.service (with its threading extra), oslo.utils and requests.

Bug Fixes

  • Fixed a race in neutron-mlnx-agent where its state report thread and its main loop could send requests to eswitchd on the same ZeroMQ socket at the same time. The main loop then stopped with ZMQError: Operation cannot be accomplished in current state while the agent was still reported as alive, so ports were never bound and instances with InfiniBand ports failed to boot with a VIF plugging timeout. Requests are now serialised, and the socket is replaced after a failed request.

Other Notes