Zed Series (20.2.0 - 21.1.x) Release Notes¶
21.1.2-7¶
Upgrade Notes¶
When upgrading Ironic to address the
qemu-img
image conversion security issues, theironic-python-agent
ramdisks will also need to be upgraded.
As a result of security fixes to address
qemu-img
image conversion security issues, a new configuration parameter has been added to Ironic,[conductor]permitted_image_formats
with a default value of “raw,qcow2,iso”. Raw and qcow2 format disk images are the image formats the Ironic community has consistently stated as what is supported and expected for use with Ironic. These formats also match the formats which the Ironic community tests. Operators who leverage other disk image formats, may need to modify this setting further.
Security Issues¶
Ironic now checks the supplied image format value against the detected format of the image file, and will prevent deployments should the values mismatch. If being used with Glance and a mismatch in metadata is identified, it will require images to be re-uploaded with a new image ID to represent corrected metadata. This is the result of CVE-2024-44082 tracked as bug 2071740.
Ironic always inspects the supplied user image content for safety prior to deployment of a node should the image pass through the conductor, even if the image is supplied in
raw
format. This is utilized to identify the format of the image and the overall safety of the image, such that source images with unknown or unsafe feature usage are explicitly rejected. This can be disabled by setting[conductor]disable_deep_image_inspection
toTrue
. This is the result of CVE-2024-44082 tracked as bug 2071740.
Ironic also inspect images which would normally be provided as a URL for direct download by the
ironic-python-agent
ramdisk. This is enabled by default and increases the overall network traffic and disk space utilization of the conductor. This level of inspection can be disabled by setting[conductor]conductor_always_validates_images
toFalse
. Doing so is not advisable as Zed release and earlierironic-python-agent
ramdisks will not be made available due to backport regression risk. This is the result of CVE-2024-44082 tracked as bug 2071740.
Ironic now explicitly enforces a list of permitted image types for deployment via the
[conductor]permitted_image_formats
setting, which defaults to “raw”, “qcow2”, and “iso”. While the project has classically always declared permissible images as “qcow2” and “raw”, it was previously possible to supply other image formats known toqemu-img
, and the utility would attempt to convert the images. The “iso” support is required for “boot from ISO” ramdisk support.
Ironic now explicitly passes the source input format to executions of
qemu-img
to limit the permitted qemu disk image drivers which may evaluate an image to prevent any mismatched format attacks againstqemu-img
.
The
ansible
deploy interface example playbooks now supply an input format to execution ofqemu-img
. If you are using customized playbooks, please add “-f {{ ironic.image.disk_format }}” to your invocations ofqemu-img
. If you do not do so,qemu-img
will automatically try and guess which can lead to known security issues with the incorrect source format driver.
Operators who have implemented any custom deployment drivers or additional functionality like machine snapshot, should review their downstream code to ensure they are properly invoking
qemu-img
. If there are any questions or concerns, please reach out to the Ironic project developers.
Operators are reminded that they should utilize cleaning in their environments. Disabling any security features such as cleaning or image inspection are at your own risk. Should you have any issues with security related features, please don’t hesitate to open a bug with the project.
The
[conductor]disable_deep_image_inspection
setting is conveyed to theironic-python-agent
ramdisks automatically, and will prevent those operating ramdisks from performing deep inspection of images before they are written.
The
[conductor]permitted_image_formats
setting is conveyed to theironic-python-agent
ramdisks automatically. Should a need arise to explicitly permit an additional format, that should take place in the Ironic service configuration.
An issue in Ironic has been resolved where image checksums would not be checked prior to the conversion of an image to a
raw
format image from another image format.With default settings, this normally would not take place, however the
image_download_source
option, which is available to be set at anode
level for a single deployment, by default for that baremetal node in all cases, or via the[agent]image_download_source
configuration option when set tolocal
. By default, this setting ishttp
.This was in concert with the
[DEFAULT]force_raw_images
when set toTrue
, which caused Ironic to download and convert the file.In a fully integrated context of Ironic’s use in a larger OpenStack deployment, where images are coming from the Glance image service, the previous pattern was not problematic. The overall issue was introduced as a result of the capability to supply, cache, and convert a disk image provided as a URL by an authenticated user.
Ironic will now validate the user supplied checksum prior to image conversion on the conductor. This can be disabled using the
[conductor]disable_file_checksum
configuration option.
Bug Fixes¶
Fixes multiple issues in the handling of images as it relates to the execution of the
qemu-img
utility, which is used for image format conversion, where a malicious user could craft a disk image to potentially extract information from anironic-conductor
process’s operating environment.Ironic now explicitly enforces a list of approved image formats as a
[conductor]permitted_image_formats
list, which mirrors the image formats the Ironic project has historically tested and expressed as known working. Testing is not based upon file extension, but upon content fingerprinting of the disk image files. This is tracked as CVE-2024-44082 via bug 2071740.
Fixes a security issue where Ironic would fail to checksum disk image files it downloads when Ironic had been requested to download and convert the image to a raw image format. This required the
image_download_source
to be explicitly set tolocal
, which is not the default.This fix can be disabled by setting
[conductor]disable_file_checksum
toTrue
, however this option will be removed in new major Ironic releases.As a result of this, parity has been introduced to align Ironic to Ironic-Python-Agent’s support for checksums used by
standalone
users of Ironic. This includes support for remote checksum files to be supplied by URL, in order to prevent breaking existing users which may have inadvertently been leveraging the prior code path. This support can be disabled by setting[conductor]disable_support_for_checksum_files
toTrue
.
21.1.2¶
Upgrade Notes¶
Adds
sha256
,sha384
andsha512
as supported SNMPv3 authentication protocols to iRMC driver.
ipxe
boot_interface will be preferred toirmc-pxe
by default with theirmc
hardware type during the new baremetal node creation. This change doesn’t affect to the existing baremetal nodes. Operators who want to continue to useirmc-pxe
as default boot interface need to specify it in[DEFAULT]default_boot_interface
setting inironic.conf
.
Bug Fixes¶
Fixes bug in iRMC driver, where irmc power_interface sets and updates irmc_ipmi_succeed flag which is used by rest of iRMC driver code to deal with iRMC firmware’s IPMI incompatibility but ipmitool power_interface doesn’t set nor update irmc_ipmi_succeed flag and rest of iRMC driver code fail to handle iRMC firmware’s IPMI incompatibility correctly.
Fixes an issue where a System Scoped user could not trigger a node into a
manageable
state with cleaning enabled, as the Neutron client would attempt to utilize their user’s token to create the Neutron port for the cleaning operation, as designed. This is because with requests made in thesystem
scope, there is no associated project and the request fails.Ironic now checks if the request has been made with a
system
scope, and if so it utilizes the internal credential configuration to communicate with Neutron.
Fixes the default boot interface order for the
irmc
hardware type where previously it would preferirmc-pxe
overipxe
. This created inconsistencies for operators using multiple hardware types, where both interfaces were enabled in the deployment. Because the fix is backported on this point of releasae of Zed, operators who will update their environment beyond this release have to check the related upgrade note and take necessary action even though their update will be done in Zed release.
Fixes issues with Lenovo hardware where the system firmware may display a blue “Boot Option Restoration” screen after the agent writes an image to the host in UEFI boot mode, requiring manual intervention before the deployed node boots. This issue is rooted in multiple changes being made to the underlying NVRAM configuration of the node. Lenovo engineers have suggested to only change the UEFI NVRAM and not perform any further changes via the BMC to configure the next boot. Ironic now does such on Lenovo hardware. More information and background on this issue can be discovered in bug 2053064.
21.1.1¶
Upgrade Notes¶
When Ironic operator uses irmc driver against Fujitsu server which runs iRMC version S6 2.00 or later, operator may need to set Redfish parameters in
driver_info
so this fix can operate properly or operator should enable IPMI over LAN through BMC settings, if possible.
Bug Fixes¶
Fixes Ironic integration with Cinder because of changes which resulted as part of the recent Security related fix in bug 2004555. The work in Ironic to track this fix was logged in bug 2019892. Ironic now sends a service token to Cinder, which allows for access restrictions added as part of the original CVE-2023-2088 fix to be appropriately bypassed. Ironic was not vulnerable, but the restrictions added as a result did impact Ironic’s usage. This is because Ironic volume attachments are not on a shared “compute node”, but instead mapped to the physical machines and Ironic handles the attachment life-cycle after initial attachment.
When aborting cleaning, the
last_error
field is no longer initially empty. It is now populated on the state transition toclean failed
.
When cleaning or deployment fails, the
last_error
field is no longer temporary set toNone
while the power off action is running.
Fixes
Invalid cross-device link
in some cases when usingfile://
image URLs.
Fixes an issue where if selinux is enabled and enforcing, and the published image is a hardlink, the source selinux context is preserved, causing access denied when retrieving the image using hardlink URL.
Fixes bug of iRMC driver in parse_driver_info where, if FIPS is enabled, SNMP version is always required to be version 3 even though iRMC driver’s xxx_interface doesn’t use SNMP actually.
Fixes a firmware incompatibility issue with iRMC versions S6 2.00 and later now doesn’t support IPMI over LAN by default. To deal with this problem, irmc driver first tries IPMI operation then, if IPMI operation fails, it tries Redfish API of Fujitsu server. The operator must set Redfish parameters in the
driver_info
if iRMC disable or doesn’t support IPMI over LAN.
Fixes
'NoneType' object is not iterable
in conductor logs forredfish
andidrac-redfish
RAID clean and deploy steps. The message should no longer appear. For affected nodes re-create the node or deleteraid_configs
entry fromdriver_internal_info
field.
Fixes an issue in the online upgrade logic where database models for Node Traits and BIOS Settings resulted in an error when performing the online data migration. This was because these tables were originally created as extensions of the Nodes database table, and the schema of the database was slightly different enough to result in an error if there was data to migrate in these tables upon upgrade, which would have occured if an early BIOS Setting adopter had data in the database prior to upgrading to the Yoga release of Ironic.
The online upgrade parameter now subsitutes an alternate primary key name name when applicable.
Fixes an issue where an agent token could be inadvertently orphaned if a node is already in the target power state when we attempt to turn the node off.
Fixes scope classification check with the “self_owned_node” policy check where it was limited to check execution with only project scoped, so system scoped users who ticked the policy endpoint would basically get an incorrect error.
Fixes the bug where provisioning a Redfish managed node fails if the BMC doesn’t support EthernetInterfaces attribute, even if MAC address information is provided manually. This is done by handling of MissingAttributeError sushy exception in get_mac_addresses() method. This fix is needed to successfully provision machines such as Cisco UCSB and UCSX.
Modify iRMC driver to use ironic.conf [deploy] default_boot_mode to determine default boot_mode.
No longer re-calculates checksums for images that are already raw. Previously, it would cause significant delays in deploying raw images.
The per-node
external_http_url
setting in the driver info is now used for a boot ISO. Previously this setting was only used for a config floppy.
Fixes an issue where an agent token was being orphaned if a baremetal node timed out during cleaning operations, leading to issues where the node would not be able to establish a new token with Ironic upon future in some cases. We now always wipe the token in this case.
21.1.0¶
Prelude¶
The Ironic team hereby announces the release of the Zed version of Ironic. This version, 21.1.0, represents the collaboration of Ironic’s contributors during the Zed release cycle, which first saw the release of Ironic 20.2.0, and Ironic 21.1.0. These versions saw improvements in functionality to better support infrastructure operators from the configuration of individual nodes, to support a greater separation of duties, and ultimately Self-Service Bare Metal as a Service, or “SSBMaaS”. Along with these features, these releases have seen numerous bug fixes. We sincerely hope you enjoy it!
New Features¶
Adds
raritan_pdu2
,servertech_sentry3
,servertech_sentry4
, andvertivgest_pdu
snmp drivers to support additional PDU models.
Adds an automatic switch to
url
for the kickstart template when the source is a URL path as opposed to astage2
ramdisk.
Adds a concurrency limiter for number of nodes in states related to Cleaning and Provisioning operations across the ironic deployment. These settings default to a maximum number of concurrent deployments to
250
and a maximum number of concurrent deletes and cleaning operations to50
. These settings can be tuned using[conductor]max_concurrent_deploy
and[conductor]max_concurrent_clean
, respectively. The defaults should generally be good for most operators in most cases. Large scale operators should evaluate the defaults and tune appropriately as this feature cannot be disabled, as it is a security mechanism.
Adds new clean steps
create_csr
andadd_https_certificate
toilo
andilo5
hardware types which allows users to create Certificate Signing Request(CSR) and adds signed HTTPS certificate to the iLO.
The
[dhcp]dhcp_provider
configuration option can now be set todnsmasq
as an alternative tonone
for standalone deployments. This enables the same node-specific DHCP capabilities as theneutron
provider. See the[dnsmasq]
section for configuration options.
Provides vendor passthru methods for
ilo
andilo5
hardware types to create, delete and get subscriptions for BMC events. These methods are supported forHPE ProLiant Gen10
andHPE ProLiant Gen10 Plus
servers.
Adds the capability for a project scoped
admin
user to be able to create nodes in Ironic, which are then manageable by the project scopedadmin
user. Effectively, this is self service Bare Metal as a Service, however more advanced fields such as drivers, chassies, are not available to these users. This is controlled through an auto-population of the Nodeowner
field, and can be controlled through the[api]project_admin_can_manage_own_nodes
setting, which defaults toTrue
, and the new policybaremetal:node:create:self_owned_node
.
Adds the capability for a project scoped
admin
user to be able to delete nodes from Ironic which their project owns. This can be contolled through the[api]project_admin_can_manage_own_nodes
setting, which defaults toTrue
, as well as thebaremetal:node:delete:self_owned_node
policy.
Upgrade Notes¶
Large scale operators should be aware that a new feature, referred to as “Concurrent Action Limit” was introduced as a security mechanism to provide a means to limit attackers, or faulty scripts, from potentially causing irreperable harm to an environment. This feature cannot be disabled, and operators are encouraged to tune the new settings
[conductor]max_concurrent_deploy
and[conductor]max_concurrent_clean
to match the needs of their environment.
Operators who are upgrading should be aware that a bug was discovered with the automatic selection of
boot_interface
for users of theilo
andilo5
hardware types. This was an inconsistency, resulting inpxe
being selected instead ofipxe
if both boot interfaces were enabled. Depending on the local configuration, this may, or may not have happened and will remain static on preexisting baremetal nodes. Some users may have been relying upon this incorrect behavior by having misalligned defaults by trying to use thepxe
interface foripxe
. Users wishing to continue this usage as it was previously will need to explicitly set aboot_interface
value to eitherpxe
orilo-ipxe
by default, depending on the local configuration. Most operators have leveraged the default examples, and thus did not explicitly encounter this condition. Operators explicitly wishing to usepxe
boot interfaces with theipxe
templates and defaults set to override the defaults forironic.conf
will need to either continue to leverage default override configurations in theirironic.conf
file.
Security Issues¶
This release contains an improvement which, by default, allows users to create and delete baremetal nodes inside their own project. This can be disabled using the
[api]project_admin_can_manage_own_nodes
setting.
Bug Fixes¶
Fixes detecting of allowable values for a BIOS settings enumeration in the
redfish
BIOS interface when onlyValueDisplayName
is provided.
Adds a configuration option,
[anaconda]insecure_heartbeat
to allow for TLS certificate validation to be disabled in theanaconda
deployment interface, which is needed for continious integration to be able to be performed without substantial substrate image customization. This option is not advised for any production usage.
Fixes an issue where image information retrieval would fail when a path was supplied when using the
anaconda
deploy interface, as HTTPHEAD
requests on a URL path have noContent-Length
. We now consider if a path is used prior to attempting to collect additional configuration data from what is normally expected to be Glance.
Fixes an issue where the fallback to a default kickstart template value would result in error indicating “Scheme-less image href is not a UUID”. This was becaues the handling code falling back to the default did not explicitly indicate it was a file URL before saving the value.
Fixes an issue where cleaning operations could fail in such a way that was not easily recoverable when pre-cleaning network interface configuration was validated, yet contained invalid configuration. Now Ironic properly captures the error and exits from cleaning in a state which allows for cleaning to be retried.
Fixes
idrac-redfish
RAIDdelete_configuration
step to convert PERC 9 and PERC 10 controllers to RAID mode if it is not already set.
Fixes the default boot interface order for the
ilo
hardware type where previously it would preferpxe
overipxe
. This created inconsistencies for operators using multiple hardware types, where both interfaces were enabled in the deployment.
Fixes API error messages with jsonschema>=4.8. A possible root cause is now detected for generic schema errors.
Fixes an issue where the Redfish session cache would continue using an old session when a password for a Redfish BMC was changed. Now the old session will not be found in this case, and a new session will be created with the latest credential information available.
Other Notes¶
The maximum disk erasure concurrency setting,
[deploy]disk_erasure_concurrency
has been incremed to 4. Previously, this was kept at 1 in order to maintain continuity of experience, but operators have not reported any issues with an increased concurrency, and as such we feel comfortable upstream enabling concurrent disk erasure/cleaning. This setting applies to theerase_devices
clean step.