Queens Series (6.1.0 - 7.2.x) Release Notes


Security Issues

  • Fixes insufficient input filtering when looking up a node by information from the introspection data. It could potentially allow SQL injections via the /v1/continue API endpoint. See story 2005678 for details.

Bug Fixes

  • Fix starting inspection of node having IPv6 BMC address. Inspection could not be initiated because v6 address was being considered as a hostname. Thus resolving incorrect hostname ended up with blocking error.


Bug Fixes

  • Allows the set-attribute introspection rule action to accept None as value for a property.


Bug Fixes

  • A new rootwrap filter is now included to allow control of the systemd dnsmasq service used by ironic-inspector. This fixes a permission issue when systemctl commands are used as dnsmasq_start_command and dnsmasq_stop_command in the configuration for the dnsmasq pxe filter. See bug 2002818.


    The filter uses the systemd service name used by the RDO distrubution (openstack-ironic-inspector-dnsmasq.service).

  • Fixes issue that can result in introspection failure when a network switch sends incomplete information for LLDP switch_id or port_id. The validation expects these fields when a port is updated, this fix now handles the validation exception.

  • The dnsmasq PXE filter no longer whitelists the MAC addresses of ports deleted from the Bare Metal service. Instead they are blacklisted unless introspection is active or the node_not_found_hook is set in the configuration. This ensures that no previously enrolled node accidentally boot the inspection image when no node introspection is active. Bug #2001979.


New Features

  • Adds wildcard ignore entry to dnsmasq PXE filter. When node introspection is active, or if node_not_found_hook is set in the configuration the ignore is removed from the wildcard entry. This ensures that unknown nodes do not accidentally boot into the introspection image when no node introspection is active.

    This brings dnsmasq PXE filter driver feature parity with the iptables PXE filter driver, which uses a firewall rule to block any DHCP request on the interface where Ironic Inspector’s DHCP server is listening.


Upgrade Notes

  • Adds dependency on the retrying python library.

Bug Fixes

  • Fixes bug in which the switch_id field in a port’s local_link_connection can be set to a non-MAC address if the processed LLDP has a value other than a MAC address for ChassisID. The bare metal API requires the switch_id field to be a MAC address, and will return an error otherwise. See bug 1748022 for details.

  • Ironic introspection no longer tries to access the Identity service if the auth_strategy option is set to noauth and the auth_type option is not set to none.

  • The periodic PXE filter update task now retries fetching port list from the Bare Metal service 5 times (with 1 second delay) before giving up. This ensures that a temporary networking glitch will not result in the ironic-inspector service stopping.


Deprecation Notes

  • Several configuration options related to ironic API access are deprecated and will be removed in the Rocky release. These include:

    • [ironic]/os_region - use [ironic]/region_name option instead

    • [ironic]/auth_strategy - set [ironic]/auth_type option to none to access ironic API in noauth mode

    • [ironic]/ironic_url - use [ironic]/endpoint_override option to set specific ironic API endpoint address if discovery of ironic API endpoint is not desired or impossible (for example in standalone mode)

    • [ironic]/os_service_type - use [ironic]/service_type option

    • [ironic]/os_endpoint_type - use [ironic]/valid_interfaces option to set ironic endpoint types that will be attempted to be used

  • Several configuration options related to swift API access are deprecated and will be removed in Rocky release. These include:

    • [swift]/os_service_type - use [swift]/service_type option

    • [swift]/os_endpoint_type - use [swift]/valid_interfaces option

    • [swift]/os_region - use [swift]region_name option

Other Notes


New Features

  • Introduces the dnsmasq PXE filter driver. This driver takes advantage of the inotify facility to reconfigure the dnsmasq service in real time to implement a caching black-/white-list of port MAC addresses.

Upgrade Notes

  • A new state aborting was introduced to distinguish between the node introspection abort precondition (being able to perform the state transition from the waiting state) from the activities necessary to abort an ongoing node introspection (power-off, set finished timestamp etc.)

  • Handling of local_gb property was moved from the scheduler hook to root_disk_selection.

Bug Fixes

  • The node_info.finished(<transition>, error=<error>) now updates node state together with other status attributes in a single DB transaction.

Other Notes


New Features

  • The PXE filter drivers mechanism is now enabled. The firewall-based filtering was re-implemented as the iptables PXE filter driver.

  • Adds an API access policy enforcment based on oslo.policy rules. Similar to other OpenStack services, operators now can configure fine-grained access policies using policy.yaml file. See policy.yaml.sample in the code tree for the list of available policies and their default rules. This file can also be generated from the code tree with the following command:

    tox -egenpolicy

    See the oslo.policy package documentation for more information on using and configuring API access policies.

Upgrade Notes

  • Due to the choice of default values for API access policies rules, some API parts of the ironic-inspector service will become available to wider range of users after upgrade:

    • general access to the whole API is by default granted to a user with either admin, administrator or baremetal_admin role (previously it allowed access only to a user with admin role)

    • listing of current introspection statuses and showing a given introspection is by default also allowed to a user with the baremetal_observer role

    If these access policies are not appropriate for your deployment, override them in a policy.json file in the ironic-inspector configuration directory (usually /etc/ironic-inspector).

    See the oslo.policy package documentation for more information on using and configuring API access policies.

Deprecation Notes

  • The firewall-specific configuration options were moved from the firewall to the iptables group. All options in the iptables group are now deprecated.

  • The generic firewall options firewall_update_period and manage_firewall were moved under the pxe_filter group as sync_period and driver=iptables/noop respectively.

Bug Fixes

  • The older ipmi_address field in the introspection data no longer has priority over the newer bmc_address inventory field during lookup. This fixes lookup based on MAC addresses, when the BMC address is reported as for any reason (see bug 1714944).

  • Should the iptables PXE filter encounter an unexpected exception in the periodic sync call, the exception will be logged and the filter driver will be reset in order to make subsequent sync calls fail (and propagate the failure, exiting the ironic-inspector process eventually).

Other Notes

  • Allows a periodic task to shut down an ironic-inspector process upon a failure.


New Features

  • Querying ironic-inspector rules API now also returns the invert and multiple attributes of the associated conditions.

  • Add disabled option to add_ports, so discovered nodes can be created without creating ports.

  • Add a check from the link_local_connection plugin to use data stored by the lldp_basic; this avoids parsing the LLDP packets twice.

  • Adds node state to the GET /v1/introspection/<node UUID or name> and GET /v1/introspection API response data.

  • Processing hooks can now define dependencies on other processing hooks. ironic-inspector start up fails when required hooks are not enabled before the hook that requires them.

  • Update pxe_enabled field on ports. It is set to True for the PXE-booting port and False for the remaining ports. Both newly discovered and existing ports are affected.

Upgrade Notes

  • Experimental setting IPMI credentials support was removed from all versions of the API. The current ironic-inspector API version was bumped to 1.12 to mark this change.

  • The default API version was synchronized with the current API version again after removal of the IPMI credentials setting.

  • Ports creating logic was moved from core processing code to the validate_interfaces processing hook. This may affect deployments that disable this hook or replace it with something else. Also make sure to place this hook before any hooks expecting ports to be created.

  • Bare metal API version 1.19 is now required.

  • Removes deprecated configuration options: introspection_delay_drivers from the default section and log_bmc_address from the processing section.

  • Support for rollback actions in introspection rules was removed.

  • Old status records are no longer removed by default. They are still removed if a node is removed from Ironic.

Deprecation Notes

  • The node_status_keep_time configuration option is deprecated. Now that we can remove status information about nodes removed from ironic, this option does not make much sense, and may be confusing

Bug Fixes

  • Timeout in an active state led to an undefined transition error. This is fixed and an introspection finishes now with Timeout error.

  • and an empty string in the bmc_address inventory field are now correctly treated as missing BMC address.

  • For postgreSQL, the database migration command ironic-inspector-dbsync upgrade always failed (with enum NODE_STATE does not exist). This is fixed and the migration now works.

  • Do not fail the whole introspection due to a value formatting error during introspection rules rollback. See bug 1686942 for an example and detailed investigation.


Bug Fixes

  • The POST /v1/introspection/<Node ID>/data/unprocessed API updates the started_at time when ironic inspector begins processing the node.

  • Exception CalledProcessError is raised when running iptables cmd on start up. The issue is caused by eventlet bug, see: https://github.com/eventlet/eventlet/issues/357 The issue affects ironic-inspector only if it manages firewall - configured with manage_firewall = True configuration option.

  • Wrong provision state name ‘inspectfail’ in ironic-inspector valid states for node inspection. This issue leads to state inconsistency between ironic and ironic-inspector. For example, if ironic inspection timeout is lower than ironic-inspector’s, and inspection timeout occurs, ironic will transition node into ‘inspect failed’ provision state. In such case when node inspection finishes without errors the node will be in ‘inspect failed’ provision state with inspection in ‘finished’ state.


New Features

  • Extend the introspection status returned from GET@/v1/introspection/<Node Id> to contain the uuid, started_at and finished_at fields.

  • Add a plugin to parse raw LLDP Basic Management, 802.1, and 802.3 TLVs and store the data in Swift.

  • Add an API endpoint for listing introspection statuses. Operators can use this to get the status for all running or previously run introspection processing.

  • Introduce a new configuration option api_max_limit that defines the maximum number of items per page when API results are paginated.

  • InfiniBand interface discovery is now supported through introspection. The ironic-inspector will add the client-id to the corresponding ironic port that represents the InfiniBand interface. The ironic-inspector should be configured with a list of interfaces firewall.ethoib_interfaces to indicate which Ethernet Over InfiniBand Interfaces are used for DHCP.

  • Node introspection state is now kept in a dedicated database column. The introspection is now using a finite state machine. The state isn’t exposed to the user yet.

  • Adds support for using operators with the root device hints mechanism. The supported operators are =, ==, !=, >=, <=, >, <, s==, s!=, s>=, s>, s<=, s<, <in>, <all-in> and <or>.

  • Looking up nodes during introspection or discovery now supports multiple attributes matching. For example, two nodes can use the same bmc_address and still can be distinguished by MAC addresses.

  • Avoid failing introspection on diskless nodes. The node property local_gb == 0 is set in that case.

Known Issues

  • Due to the nature of the NodeInfo.state attribute (being updated independently from the rest of the node_info attributes) if a (DB) connection was lost before the Node.state column was updated, Node.finished_at and Node.error columns may not be in sync with the Node.state column.

Upgrade Notes

  • Add a new dependency, pytz.

  • A database migration is required to change some columns from Float to DateTime type. This may take some time based on the number of introspection statuses in DB.

  • Removed previously deprecated authentication options from “ironic”, “swift”, and “keystone_authtoken” sections.

  • Removed long deprecated support for “discoverd” section in configuration file.

  • The default value for the configuration option “introspection_delay_drivers” was changed to .*, which means that by default “introspection_delay” is now applied to all drivers. Set “introspection_delay” to 0 to disable the delay.

  • Node.state and Node.version_id database columns are introduced.

  • The introspection state column defaults to the state finished unless the introspection error column value on a node row isn’t null, then node state is set to error.

  • Uniqueness of a node bmc_address isn’t enforced any more.

  • The primary key of the attributes table is relaxed from the attributes.name, attributes.value column pair to a new column attributes.uuid.

Deprecation Notes

  • The configuration option “log_bmc_address” is deprecated.

  • The configuration option “introspection_delay_drivers” is deprecated.

Bug Fixes

  • Change database columns started_at and finished_at to type DateTime from type Float so that timestamps fit into these columns correctly.

  • Fix bug where periodic clean up failed with DBDeadlock if introspection timed out.

  • Ensure the configuration options firewall.firewall_update_period and clean_up_period are applied to the periodic_clean_up and periodic_update tasks after the config file is read.

  • LLC hook now formats the chassis ID and port ID MAC addresses into Unix format as expected by ironic.

  • LLC hook ensures that correct port information is passed to the patch_port function

  • LLC hook no longer assumes all inspected ports are added to ironic

  • Loopback BMC addresses (useful e.g. with virtualbmc) are no longer used for lookup.

  • Introspection fails on nodes with the same IPMI address but different IPMI ports.

Other Notes

  • Default API version is temporary pinned to 1.8 (before deprecating setting IPMI credentials). It will be reset to the latest version again when support for setting IPMI credentials is removed.


New Features

  • Adds new processing hook pci_devices for setting node capabilities based on PCI devices present on a node and rules in the [pci_devices] aliases configuration option. Requires “pci-devices” collector to be enabled in IPA.

Bug Fixes

  • Use only single quotes for strings inside SQL statements. Fixes a crash when PostgreSQL is used as a database backend.

  • Set the node to the error state when it failed get data from swift.


New Features

  • Added GenericLocalLinkConnectionHook processing plugin to process LLDP data returned during inspection and set port ID and switch ID in an Ironic node’s port local link connection information using that data.

  • Add configuration option processing.power_off defaulting to True, which allows to leave nodes powered on after introspection.

Bug Fixes

  • Fix setting non string ‘value’ field for rule’s actions. As non string value is obviously not a formatted value, add the check to avoid AttributeError exception.



Starting with this release only ironic-python-agent (IPA) is supported as an introspection ramdisk.

New Features

  • Added a new “capabilities” processing hook detecting the CPU and boot mode capabilities (the latter disabled by default).

  • File name for stored ramdisk logs can now be customized via “ramdisk_logs_filename_format” option.

Upgrade Notes

  • The default file name for stored ramdisk logs was change to contain only node UUID (if known) and the current date time. A proper “.tar.gz” extension is now appended.

  • API “POST /v1/rules” returns 201 response code instead of 200 on creating success. API version was bumped to 1.6. API less than 1.6 continues to return 200.

  • Default API version was changed from minimum to maximum which Inspector can support.

  • Support for the old bash-based ramdisk was removed. Please switch to IPA before upgrading.

  • Removed the deprecated “root_device_hint” alias for the “raid_device” hook.

Bug Fixes

  • Fixed “/v1/continue” to return HTTP 500 on unexpected exceptions, not HTTP 400.

  • Fix response return code for rule creating endpoint, it returns 201 now instead of 200 on success.

  • The “size” root device hint is now always converted to an integer for consistency with IPA.


New Features

  • Ironic-Inspector is now using keystoneauth and proper auth_plugins instead of keystoneclient for communicating with Ironic and Swift. It allows to finely tune authentification for each service independently. For each service, the keystone session is created and reused, minimizing the number of authentification requests to Keystone.

  • Add support for using Ironic node names in API instead of UUIDs. Note that using node names in the introspection status API will require a call to Ironic to be made by the service.

  • Introduced API “POST /v1/introspection/UUID/data/unprocessed” for reapplying the introspection over stored data.

Upgrade Notes

  • Operators are advised to specify a proper keystoneauth plugin and its appropriate settings in [ironic] and [swift] config sections. Backward compatibility with previous authentification options is included. Using authentification informaiton for Ironic and Swift from [keystone_authtoken] config section is no longer supported.

  • Handling ramdisk logs was moved out of the “ramdisk_error” plugin, so disabling it will no longer disable handling ramdisk logs. As before, you can set “ramdisk_logs_dir” option to an empty value (the default) to disable storing ramdisk logs.

Deprecation Notes

  • Most of current authentification options for either Ironic or Swift are deprecated and will be removed in a future release. Please configure the keystoneauth auth plugin authentification instead.

Bug Fixes

  • Fixes a problem which caused an unhandled TypeError exception to bubble up when inspector was attempting to convert some eDeploy data to integer.

  • Fixed a regression in the firewall code, which causes re-running introspection for an already inspected node to fail.

  • Fixed the “is-empty” condition to return True on missing values.

  • The lookup procedure now uses all valid MAC’s, not only the MAC(s) that will be used for creating port(s).

  • The “enroll” node_not_found_hook now uses all valid MAC’s to check node existence, not only the MAC(s) that will be used for creating port(s).

  • The ramdisk logs are now stored on all preprocessing errors, not only ones reported by the ramdisk itself. This required moving the ramdisk logs handling from the “ramdisk_error” plugin to the generic processing code.


New Features

Bug Fixes

  • Don’t fail on finish power off if node in ‘enroll’ state. Nodes in ‘enroll’ state are not expected to have power credentials.


New Features

  • Introduced API “POST /v1/introspection/<UUID>/abort” for aborting the introspection process.

  • New condition plugins “contains” and “matches” allow to match value against regular expressions.

  • Added new condition plugin “is-empty”, which allows to match empty string, list, dictionary or None.

  • Add a new node_not_found hook - enroll, which allows automatically discover Ironic’s node.

  • Conditions now support comparing fields from node info;

  • Introspection rules conditions got a new generic “invert” parameter that inverts the result of the condition.

Upgrade Notes

  • Switch required Ironic API version to ‘1.11’, which supports ‘enroll’ state.

  • Minimum possible value for the “max_concurrency” setting is now 2.

  • Removed deprecated support for passing “node_patches” and “ports_patches” arguments to processing hooks.

  • Ramdisk logs are no longer part of data stored to Swift and returned by the API.

  • Introspection rules actions ‘set-attribute’, ‘set-capability’ and ‘extend-attribute’ no longer have the opposite effect on nodes that do not match a rule.

Deprecation Notes

  • The rollback actions for introspection rules are deprecated. No in-tree actions are using them, 3rdpart should stop using them as soon as possible.

  • Using the root_device_hint alias for the raid_device plugin is deprecated.

Bug Fixes

  • Fixed extra_hardware plugin connection to Swift.

  • Only issue iptables calls when list of active MAC’s changes.

  • Dropped rollback actions from ‘set-attribute’, ‘set-capability’ and ‘extend-attribute’ introspection rules actions, as they were confusing, completely undocumented and broke some real world use cases (e.g. setting driver field).

  • Introspection rules (e.g. set-attribute action) now accept ‘path’ field without leading forward slash as Ironic cli does.

Other Notes

  • Switched to Futurist library for asynchronous tasks.

  • Log level for error when node was not found in Inspector cache was changed from error to info level. It was done because not_found_hook may handle this case, so this wouldn’t be error anymore.



Starting with this release, ironic-python-agent becomes the default introspection ramdisk, with the old bash-based ramdisk being deprecated.

New Features

  • Inspector no longer requires old-style “local_gb”, “memory_mb”, “cpus” and “cpu_arch” fields from the introspection ramdisk. They are still supported, though, for compatibility with the old ramdisk.

Upgrade Notes

  • Removed support for introspecting nodes in maintenance mode, deprecated in the liberty cycle. Use “inspecting”, “manageable” or “enroll” states instead.

  • The root_disk_selection processing hook will now error out if root device hints are specified on ironic node, but ironic-python-agent is not used as an introspection ramdisk.

Deprecation Notes

  • Using old bash-based ramdisk is deprecated, please switch to ironic-python-agent as soon as possible.

Bug Fixes

  • The data processing API endpoint now validates that data received from the ramdisk is actually a JSON object instead of failing the internal error later (issue https://bugs.launchpad.net/bugs/1525876).

Other Notes

  • Make debug-level logging more compact by removing newlines from firewall logging and disabling some 3rdparty debug messages by default.

  • Improve logging for ramdisk logs collection.

  • Logging during processing is now more consistent in terms of how it identifies the node. Now we try to prefix the log message with node UUID, BMC address and PXE MAC address (if available). Logging BMC addresses can be disabled via new “log_bmc_address” option in the “processing” section.



This release includes automatic docs generation via Sphinx.

Security Issues

Bug Fixes

  • Log a warning when add_ports is set to pxe, but no PXE MAC is returned from the ramdisk.

  • Acquire a lock on a node UUID when handling it.

Other Notes

  • IPA (ironic-python-agent) is now fully supported in the devstack plugin and will become the default ramdisk in the next release.

  • Allow autogeneration of database migrations.

  • Introduced new docs generation via Sphinx and ReST.

    • Separate doc folder includes source and build

    • Integration with tox as docs target

    • makefile for manual building

    • Openstack Theme support