Current Series Release Notes

15.2.0-54

New Features

  • Adds ilo-uefi-https boot interface to ilo5 hardware type. This boot interface levereges the iLO UEFI firmware capability to boot from given HTTPS URLs hosted securely over HTTPS webserver with standard/custom certificates.

  • Adds an ability to accept a custom TLS certificate in the heartbeat API.

  • Adds a configuration option webserver_verify_ca to support custom certificates to validate URLs hosted on a HTTPS webserver.

  • Using the network_data property on the node field, operators can now apply network configuration to be embedded in Redfish based Virtual Media based deployment ramdisks which include networking configuration enabling the deployment to operate without the use of DHCP. See Redfish driver documentation for more information.

  • file:// images are now supported in the direct deploy interface.

  • Adds a new possible value for image_download_source: local. When used, even http:// images are downloaded, converted to RAW if needed and served from the conductor’s HTTP server. This feature targets primarily nodes with low RAM.

  • Adds a new configuration option [ilo]verify_ca and a new driver_info parameter ilo_verify_ca to enhance certificate verification for hardware type ilo and ilo5 which can take directory and bolean values apart from file.

  • Adds functionality to perform out-of-band one button secure erase operation for iLO5 based HPE Proliant servers as a management clean step one_button_secure_erase for ilo5 hardware type.

  • The image_download_source configuration option can now also be set per node in the instance_info or driver_info (the former having the highest priority).

  • Allows configuring IPMI cipher suite via the new driver_info parameter ipmi_cipher_suite.

  • Adds driver_internal_info field to the node-related notification baremetal.node.provision_set.*, new payload version 1.16.

  • A new configuration option [agent]require_tls allows rejecting ramdisk callback URLs that don’t use the https:// schema.

Upgrade Notes

  • The do_one_button_secure_erase interface of ‘proliantutils’ library has been enhanced to support out-of-band one button secure erase operation for ilo5 hardware type. To leverage this feature, the ‘proliantutils’ library needs to be upgraded to version ‘2.10.0’.

  • The default value of the configuration option [agent]image_download_source has been changed to http to simplify transition from the iscsi deploy interface. Set it to swift explicitly to maintain the previous behavior.

  • Support for token-less agents has been removed as the token-less agent support was deprecated in the Ussuri development cycle. The ironic-python-agent must be updated to 6.1.0 or higher to support communicating with the Ironic deployment after upgrade. This will generally require deployment, cleaning, and rescue kernels and ramdisks to be updated. If this is not done, actions such as cleaning and deployment will time out as the agent will be unable to record heartbeats with Ironic. For more information, please see the agent token documentation.

  • The redfish-virtual-media boot interface is now the last in the list of priorities from the redfish hardware type. This means that new nodes will be created with ipxe or pxe boot if they are enabled. The reason for this change is limited support for pure Redfish virtual media from hardware vendors.

    To use virtual media with Redfish, please provide an explicit boot_interface parameter when creating nodes. If you enable only the redfish hardware type, you can also set the default_boot_interface configuration option to redfish-virtual-media.

Deprecation Notes

  • The [ilo]ca_file configuration option is deprecated for removal, please use [ilo]verify_ca instead which can take directory and boolean values apart from file for certificate verification.

  • With the switch from neutronclient to openstacksdk the [neutron]/retries option has been deprecated, use [neutron]/status_code_retries and [neutron]/status_code_retry_delay instead.

Security Issues

  • Ramdisks supporting agent token are now globally required by Ironic. As this is a core security mechanism, it cannot be disabled and support for the [DEFAULT]require_agent_token configuration parameter has been removed as tokens are now always required by Ironic. For more information, please see the agent token documentation.

Bug Fixes

  • Add the suffix “.iso” for the ISO images generated by Ironic, as some hardwares can not load the ISO images if they are not suffixed with “.iso”. It is also recommended to name the files with proper extensions.

  • Fixes the deployment failure with Ussuri (and older) ramdisks that happens when another IPA command runs after prepare_image.

  • Fixes an issue with the ansible deployment interface where automatic root deviec selection would accidently choose the system CD-ROM device, which was likely to occur when the ansible deployment interface was used with virtual media boot. The ansible deployment interface now ignores all Ramdisks, Loopbacks, CD-ROMs, and floppy disk devices.

  • Fixes an issue that caused in-band deploy steps inserted before write_image to be skipped when fast-track is used.

  • Makes sure in-band deploy and clean steps are not cached across reboots.

  • Fixes an issue with agent token handling where the agent has not been upgraded resulting in an AgentAPIError, when the token is not required. The conductor now retries without sending an agent token.

  • Agent heartbeat implementation will no longer retry attempts to acquire an exclusive lock. This is done to avoid several attempts to get a lock in a busy environment with shorter heartbeat period.

  • Fixes wiping the agent secret token on manual power off or reboot. Also makes sure to remove the agent URL since it may potentially change.

  • Fixes cleaning and managed inspection not respecting the default_boot_mode configuration option.

  • Fixes cleaning and managed inspection not following the standard boot mode handling logic, particularly, not trying to assert the requested boot mode if the driver allows it.

Other Notes

  • Communication with neutron is now using openstacksdk, removing the dependency on neutronclient.

15.2.0

New Features

  • Adds inband deploy step flash_firmware_sum to the management interface of the ilo and ilo5 hardware types. The required minimum version for the proliantutils library is 2.9.5.

  • Adds functionality to the ipxe boot interface to support use of an instance_info\boot_iso value with the ramdisk deployment interface.

  • Adds functionality to allow a user to supply a node instance_info/boot_iso parameter on machines utilizing the redfish-virtual-media boot interface. When combined with the ramdisk deployment interface, this allows an instance to boot into a user supplied ISO image.

  • The new experimental agent power interface allows limited provisioning operations on nodes without BMC credentials. See story 2007771 for details.

  • The agent RAID interface now supports building RAID as a deploy step apply_configuration.

  • Adds raid configuration validation to deploy step apply_configuration of agent RAID interface. Also, a post deploy hook has been added to this deploy step to update root device hint.

  • Adds a new driver_info parameter agent_verify_ca that allows specifying a file with certificates to use when accessing IPA. Set to False to disable certificate validation.

  • The deploy deploy step of the direct deploy interface has been split into three deploy steps:

    • deploy itself (priority 100) boots the deploy ramdisk

    • write_image (priority 80) downloads the user image from inside the ramdisk and writes it to the disk.

    • prepare_instance_boot (priority 60) prepares the boot device and writes the bootloader (if needed).

    Priorities 81 to 99 to be used for in-band deploy steps that run before the image is written. Priorities 61 to 79 can be used for in-band deploy steps that modify the written image before the bootloader is installed.

  • Provides a new option [DEFAULT]hash_ring_algorithm that specifies which cryptographic algorithm to use when building the hash ring. Set to something other than md5 when using ironic on a system in FIPS mode.

  • Adds support for boot mode retrieval and setting with the ilo and ilo5 hardware types.

  • Adds support for running custom in-band deploy steps when provisioning. Step priorities from 41 to 59 can be used for steps that run after the image is written and the bootloader is installed.

  • Adds the capability for an operator to set a configuration setting which tells the ironic-python-agent it is okay to skip read-only block devices when performing an erase_devices cleaning operation. This requires ironic-python-agent version 6.0.0 or greater and can be set using the [deploy]erase_skip_read_only configuration option.

  • The deploy deploy step of the iscsi deploy interface has been split into three deploy steps:

    • deploy itself (priority 100) boots the deploy ramdisk

    • write_image (priority 80) writes the image to the disk exposed via iSCSI.

    • prepare_instance_boot (priority 60) prepares the boot device and writes the bootloader (if needed).

    Priorities 81 to 99 to be used for in-band deploy steps that run before the image is written. Priorities 61 to 79 can be used for in-band deploy steps that modify the written image before the bootloader is installed.

  • The deploy deploy step of the ansible deploy interface has been split into two deploy steps:

    • deploy itself (priority 100) boots the deploy ramdisk

    • write_image (priority 80) writes the image to the disk and configures the bootloader.

    Priorities 81 to 99 to be used for in-band deploy steps that run before the image is written.

  • Adds network_data property to the node, a dictionary that represents the node static network configuration. The Ironic API performs formal JSON validation of node network_data content against user-supplied JSON schema at driver validation step.

  • Allow port lists to be filtered by project. Doing so checks the specified project against the port’s node’s owner and lessee.

Deprecation Notes

  • Running the whole deployment process as a monolithic deploy.deploy deploy step is now deprecated. In a future release this step will only be used to prepare deployment and starting the agent, and special handling will be removed. All third party deploy interfaces must be updated to provide real deploy steps instead and set the has_decomposed_deploy_steps attribute to True on the deploy interface level.

  • The configuration options [json_rpc]http_basic_username and [json_rpc]http_basic_password have been deprecated in favour of the more generic [json_rpc]username and [json_rpc]password.

Bug Fixes

  • Fixes RAID apply_configuration deploy step for idrac-wsman where deployment failed with TypeError. See story 2007963.

  • Fixes deployment hanging on an invalid in-band deploy step in a deploy templates.

  • Allows deleting nodes with a broken driver unless they require stopping serial console.

  • Fixes updating driver fields for nodes with a broken driver. This is required to be able to set maintenance for such nodes.

  • Fixes json_rpc client connections always using HTTP even if use_ssl was set to True.

  • When Ironic is doing IPMI retries the configured min_command_interval should be used instead of a default value of 1, which may be too short for some BMCs.

  • Fixes missing agent RAID compatibility for the ilo5 and idrac hardware type preventing software RAID for working with them.

  • Fixes an issue where ironic-conductor initialization could return a NodeNotLocked error for requests requiring locks when the conductor was starting. This was due to the conductor removing locks after beginning accepting new work. The lock removal has been moved to after the Database connectivity has been established but before the RPC bus is initialized.

  • Fixes the conductor so the power sync operations are not asserted for nodes in the adopt failed state.

  • Fixes the issue that port auto allocation for the socat console failed to correctly identify the availablility of ports under IPv6 networks.

  • Removes stale agent tokens when rebooting nodes using API. This prevents lookup failures for nodes that get rebooted between fast-track operations.

  • Removes stale agent token on rescue and unrescue operations. Previously it would cause subsequent rescue operations to fail.

  • Fixes the preservation of potentially incorrect power state information when adoption process fails. Power state is now wiped as part of the failure handling process instead of being preserved.

Other Notes

  • The proliantutils library version 2.9.5 enables ssacli based in-band deploy step apply_configuration of agent RAID interface for ilo and ilo5 hardware types.

  • Support for iPXE booting a ISO medium will only work if the ramdisk loaded by the bootloader contains all artifacts required for the booting operating system to load. This is a limitation of iPXE and x86 systems architecture, as the memory allocated for the rest of the ISO disk image in memory is freed by the booting kernel.

  • As part of the agent deploy interfaces refactoring, breaking changes will be made to implementations of AgentDeploy and ISCSIDeploy. Third party deploy interfaces must be updated to inherit HeartbeatMixin, AgentBaseMixin or AgentDeployMixin from ironic.drivers.modules.agent_base instead since their API is considered more stable.

  • Starting in ironic-python-agent 6.0.0, metadata erasure of read-only devices is skipped by default.

  • A new method supports_power_sync has been added to PowerInterface. If it returns False, the conductor will not try to assert power state for the node, merely recording the returned state instead.

  • The base agent deploy interface code now correctly handles power interfaces that do not support the power on action but support reboot.

15.1.0

New Features

  • Adds raid interface for ibmc driver which includes delete_configuration and create_configuration steps.

  • Enable Basic HTTP authentication middleware.

    Having noauth as the only option for standalone ironic causes constraints on how the API is exposed on the network. Having some kind of authentication layer behind a TLS deployment eases these constraints.

    When the config option auth_strategy is set to http_basic then non-public API calls require a valid HTTP Basic authentication header to be set. The config option http_basic_auth_user_file defaults to /etc/ironic/htpasswd and points to a file which supports the Apache htpasswd syntax[1]. This file is read for every request, so no service restart is required when changes are made.

    Like the noauth auth strategy, the http_basic auth strategy is intended for standalone deployments of ironic, and integration with other OpenStack services cannot depend on a service catalog.

    The only password digest supported is bcrypt, and the bcrypt python library is used for password checks since it supports $2y$ prefixed bcrypt passwords as generated by the Apache htpasswd utility.

    To try HTTP basic authentication, the following can be done:

    • Set /etc/ironic/ironic.conf DEFAULT auth_strategy to http_basic

    • Populate the htpasswd file with entries, for example: htpasswd -nbB myName myPassword >> /etc/ironic/htpassw

    • Make basic authenticated HTTP requests, for example: curl --user myName:myPassword http://localhost:6385/v1/drivers

    [1] https://httpd.apache.org/docs/current/misc/password_encryptions.html

  • Adds a new [ipmi]use_ipmitool_retries option. When set to True and timing is supported by ipmitool, the number of retries and command interval will be passed to ipmitool so that ipmitool will do the retries. When set to False, ironic will do the retries. Default is True.

  • Adds an ability to generate network boot templates even for nodes that use local boot via the new [pxe]enable_netboot_fallback option. This is required to work around the situation where switching boot devices does not work reliably.

  • Adds the ability for Ironic to attach a node to a specific port or portgroup. This is accomplished by having the node vif_attach API accept a port_uuid or portgroup_uuid key within vif_info. If one is specified, then Ironic will attempt to attach to the specified port/portgroup. Specifying both returns an error.

Known Issues

  • Some BMCs do not support the Channel Cipher Suites command that newer versions of ipmitool use. These versions of ipmitool will resend this command for each ipmitool retry, resulting in long response times. Setting [ipmi]use_ipmitool_retries to false will avoid this situation by implementing retries on the ironic level.

  • The SNMP hardware type cannot change boot devices and thus may fail to deploy nodes with local boot. To work around this problem, set [pxe]enable_netboot_fallback to True.

  • Some redfish-enabled hardware is known not to support persistent boot device setting that is used by the Bare Metal service for deployed instances. The redfish hardware type tries to work around this problem, but rebooting such an instance in-band may cause it to boot incorrectly. A predictable boot order should be configured in the node’s boot firmware to avoid issues and at least metadata cleaning must be enabled. See this mailing list thread for technical details.

Upgrade Notes

  • The [conductor]api_url was deprecated and removed, use [service_catalog]endpoint_override instead if required to use a specific ironic api url.

  • The [cinder]url was removed, use [cinder]endpoint_override instead.

  • The [DEFAULT]fatal_exception_format_errors was removed, use [ironic_lib]fatal_exception_format_errors instead.

  • Operators upgrading from earlier versions using PXE should explicitly set [pxe]ipxe_bootfile_name, [pxe]uefi_ipxe_bootfile_name, and possibly [pxe]ipxe_bootfile_name_by_arch settings, as well as a iPXE specific [pxe]ipxe_config_template override, if required.

    Setting the [pxe]ipxe_config_template to no value will result in the [pxe]pxe_config_template being used. The default value points to the supplied standard iPXE template, so only highly customized operators may have to tune this setting.

  • Updates required ibmcclient version for ibmc drivers to 0.2.2.

  • A permission setting has been added for redfish-virtual-media boot interface, which allows for explicit file permission setting when the driver is used. The default for the new [redfish]file_permission setting is ``0u644, or 644 if manually changed using chmod on the command line. Operators may need to check /httpboot/redfish folder permissions if using redfish-virtual-media if they were running the conductor with a specific umask to work around the permission setting defect.

Bug Fixes

  • Instead of increasing timeout when running long synchronous tasks on ironic-python-agent, ironic now runs them asynchronously and polls the agent until completion. It is no longer necessary to account for long-running tasks when setting [agent]command_timeout.

  • Fixes a rare issue where agent successfully powers off a node after deployment, but ironic never learns about it and does another reboot.

  • Fixes deployment in fast-track mode by keeping the required internal fields (agent_url and agent_secret_token) intact when starting and finishing deployment and cleaning.

  • Fixes deleting nodes with maintenance mode on and an allocation present. Previously it caused an internal server error. See story 2007823 for details.

  • Change the default for use_ipmitool_retries to False so that Ironic will do the retries by default. This is needed for certain BMCs that don’t support the Cipher Suites command and ipmitool retries take an excessively long time. See story 2007632 for additional information.

  • Cleans up nodes stuck in the deleting state on conductor restart.

  • Fixes fast-track deployments with the direct deploy interface that used to hang previously.

  • Fixes periodic task initialization options to prevent a negative number. If [conductor]clean_callback_timeout, [conductor]inspect_wait_timeout or [conductor]inspect_wait_timeout have a negative value an error will be triggered.

  • Ironic now does not try to allocate the space needed for instance image conversion to raw format if it is already raw.

  • Addresses the lack of an ability to explicitly set different bootloaders for iPXE and PXE based boot operations via their respective ipxe and pxe boot interfaces.

  • Fixes a bug in “fast track” where Ironic would delete the agent token upon exiting cleaning steps. However, if we are in fast track mode, we can preserve the token and continue operations with the agent as it is not powered off during fast track operations.

  • Fixes a workaround for hardware that does not support persistent boot device setting with the redfish or idrac-redfish management interface implementation. When such situation is detected, ironic falls back to one-time boot device setting, restoring it on every reboot or power on.

    For more information, see story 2007733.

  • Fixes the virtual disks creation by changing PERC H740P controller mode from Enhanced HBA to RAID in delete_configuration clean step. PERC H740P controllers supports RAID mode and Enhanced HBA mode. When the controller is in Enhanced HBA, it creates single disk RAID0 virtual disks of NON-RAID physical disks. Hence the request for VD creation with supported RAID fails due to no available physical disk. This patch converts the PERC H740P RAID controllers to RAID mode if enhanced HBA mode found enabled See bug bug 2007711 for more details

  • Fixes fast track deployment preceeded by managed inspection by providing the ironic API URL to the ramdisk so that it can heartbeat.

  • Fixes the JSON RPC backend potentially hanging on inability to connect to a conductor. The default timeout is now 120 seconds. The timeout and the number of retries can be adjusted via the configuration options [json_rpc]timeout and [json_rpc]connect_retries accordingly.

  • Fixes logic that is applied to port deletions to also consider the presence of a VIF attachment record, which should be removed before attempting to delete the node. Failure to do so can result in erroneous records in the Networking Service.

  • No longer tries to set local_gb to MAX when building RAID with the root disk using MAX for its size.

  • To provide a workaround for incorrect boot order problems on some hardware, the redfish hardware type now supports the noop management interface, similarly to IPMI and SNMP.

  • Rebooting a node with the redfish power interface is now implemented via a power off request followed by power on to avoid returning success when a node stays powered on after the reboot request.

  • Provides a workaround for hardware that does not support persistent boot device setting with the redfish hardware type. When such situation is detected, ironic will fall back to one-time boot device setting, restoring it on every reboot.

  • Fixes an issue where the folder /httpboot/redfish was being created with incorrect permissions.

  • If the disk format of the image is provided in the instance_info, skip the memory check if it is set to raw and raw image streaming is enabled. That allows to stream raw images provided as URL and not through Glance.

Other Notes

  • Ramdisk logs are now collected during cleaning the same way as during deployment.

  • The following configuration options can now be reloaded without restarting ironic:

    From [agent]: memory_consumed_by_agent, stream_raw_images, deploy_logs_*, image_download_source, command_timeout and neutron_agent_poll_interval.

    From [api]: max_limit, public_endpoint and ramdisk_heartbeat_timeout.

    From [conductor]: heartbeat_timeout, force_power_state_during_sync, automated_clean, soft_power_off_timeout, power_state_change_timeout, rescue_password_hash_algorithm and require_rescue_password_hashed.

    From [DEFAULT]: default_resource_class, force_raw_images, parallel_image_downloads, default_portgroup_mode and require_agent_token.

    From [deploy]: enable_ata_secure_erase, erase_devices_priority, erase_devices_metadata_priority, shred_random_overwrite_iterations, shred_final_overwrite_with_zeros, continue_if_disk_secure_erase_fails, disk_erasure_concurrency, power_off_after_deploy_failure, default_boot_option, default_boot_mode, configdrive_use_object_store, fast_track, and fast_track_timeout.

    From [ipmi]: kill_on_timeout, disable_boot_timeout, command_retry_interval, min_command_interval, debug and additional_retryable_ipmi_errors.

    From [iscsi]: portal_port, conv_flags and verify_attempts.

    From [neutron]: port_setup_delay, *_network, *_network_security_groups, request_timeout, add_all_ports and dhcpv6_stateful_address_count.

    From [nova]: send_power_notifications.

    From [pxe]: pxe_append_params, default_ephemeral_format, pxe_config_template, uefi_pxe_config_template, pxe_config_template_by_arch, ip_version and ipxe_use_swift.

    From [redfish]: use_swift, swift_container, swift_object_expiry_timeout and kernel_append_params.