Security Overview¶
While the Bare Metal service is intended to be a secure application, it is important to understand what it does and does not cover today.
Deployers must properly evaluate their use case and take the appropriate actions to secure their environment(s). This document is intended to provide an overview of what risks an operator of the Bare Metal service should be aware of. It is not intended as a How-To guide for securing a data center or an OpenStack deployment.
Image Checksums¶
Ironic has long provided a capacity to supply and check a checksum for disk images being deployed. However, one aspect which Ironic has not asserted is “Why?” in terms of “Is it for security?” or “Is it for data integrity?”.
The answer is both to ensure a higher level of security with remote image files, and provide faster feedback should a image being transferred happens to be corrupted.
Normally checksums are verified by the ironic-python-agent
OR the
deployment interface responsible for overall deployment operation. That being
said, not every deployment interface relies on disk images which have
checksums, and those deployment interfaces are for specific use cases which
Ironic users leverage, outside of the “general” use case capabilities provided
by the direct
deployment interface.
Note
Use of the node instance_info/image_checksum
field is discouraged
for integrated OpenStack Users as usage of the matching Glance Image
Service field is also deprecated. That being said, Ironic retains this
feature by popular demand while also enabling also retain simplified
operator interaction.
The newer field values supported by Glance are also specifically
supported by Ironic as instance_info/image_os_hash_value
for
checksum values and instance_info/image_os_hash_algo
field for
the checksum algorithm.
Warning
Setting a checksum value to a URL is supported, however doing this is
making a “tradeoff” with security as the remote checksum can change.
Conductor support this functionality can be disabled using the
conductor.disable_support_for_checksum_files
setting.
REST API: user roles and policy settings¶
Warning
This information is presently in flux as of the Wallaby release with the
implementation of Secure RBAC
where system
and project
scoped requests are able to be parsed and default access controls support
a delineation of roles and responsibilities through the roles.
Please see Secure RBAC.
Beginning with the Newton (6.1.0) release, the Bare Metal service allows operators significant control over API access:
Access may be restricted to each method (GET, PUT, etc) for each REST resource. Defaults are provided with the release and defined in code.
Access may be divided between an “administrative” role with full access and “observer” role with read-only access. By default, these roles are assigned the names
baremetal_admin
andbaremetal_observer
, respectively.By default, passwords and instance secrets are hidden in
driver_info
andinstance_info
, respectively. In case of debugging or diagnosing, the behavior can be overridden by changing the policy file. To allow password indriver_info
unmasked for users with administrative privileges, apply following changes to policy configuration file:"show_password": "rule:is_admin"
And restart the Bare Metal API service to take effect. Please check Policies for more details.
Prior to the Newton (6.1.0) release, the Bare Metal service only supported two policy options:
API access may be secured by a simple policy rule: users with administrative privileges may access all API resources, whereas users without administrative privileges may only access public API resources.
Passwords contained in the
driver_info
field may be hidden from all API responses with theshow_password
policy setting. This defaults to always hide passwords, regardless of the user’s role. You can override it with policy configuration as described above.
Multi-tenancy¶
There are two aspects of multitenancy to consider when evaluating a deployment of the Bare Metal Service: interactions between tenants on the network, and actions one tenant can take on a machine that will affect the next tenant.
Network Interactions¶
Interactions between tenants’ workloads running simultaneously on separate servers include, but are not limited to: IP spoofing, packet sniffing, and network man-in-the-middle attacks.
By default, the Bare Metal service provisions all nodes on a “flat” network, and does not take any precautions to avoid or prevent interaction between tenants. This can be addressed by integration with the OpenStack Identity, Compute, and Networking services, so as to provide tenant-network isolation. Additional documentation on network multi-tenancy is available.
Lingering Effects¶
Interactions between tenants placed sequentially on the same server include, but are not limited to: changes in BIOS settings, modifications to firmware, or files left on disk or peripheral storage devices (if these devices are not erased between uses).
By default, the Bare Metal service will erase (clean) the local disk drives during the “cleaning” phase, after deleting an instance. It does not reset BIOS or reflash firmware or peripheral devices. This can be addressed through customizing the utility ramdisk used during the “cleaning” phase. See details in the Firmware security section.
Firmware security¶
When the Bare Metal service deploys an operating system image to a server, that image is run natively on the server without virtualization. Any user with administrative access to the deployed instance has administrative access to the underlying hardware.
Most servers’ default settings do not prevent a privileged local user from gaining direct access to hardware devices. Such a user could modify device or firmware settings, and potentially flash new firmware to the device, before deleting their instance and allowing the server to be allocated to another user.
If the [conductor]/automated_clean
configuration option is enabled (and
the [deploy]/erase_devices_priority
configuration option is not zero),
the Bare Metal service will securely erase all local disk devices within a
machine during instance deletion. However, the service does not ship with
any code that will validate the integrity of, or make any modifications to,
system or device firmware or firmware settings.
Operators are encouraged to write their own hardware manager plugins for the
ironic-python-agent
ramdisk. This should include custom clean steps
that would be run during the Node cleaning process, as part of Node
de-provisioning. The clean steps
would perform the specific actions necessary within that environment to ensure
the integrity of each server’s firmware.
Ideally, an operator would work with their hardware vendor to ensure that proper firmware security measures are put in place ahead of time. This could include:
installing signed firmware for BIOS and peripheral devices
using a TPM (Trusted Platform Module) to validate signatures at boot time
booting machines in UEFI secure boot mode, rather than BIOS mode, to validate kernel signatures
disabling local (in-band) access from the host OS to the management controller (BMC)
disabling modifications to boot settings from the host OS
Additional references:
UEFI secure boot mode¶
Some hardware types support turning UEFI secure boot dynamically when deploying an instance. Currently these are iLO driver, iRMC driver and Redfish driver.
Support for the UEFI secure boot is declared by adding the secure_boot
capability in the capabilities
parameter in the properties
field of
a node. secure_boot
is a boolean parameter and takes value as true
or
false
.
To enable secure_boot
on a node add it to capabilities
:
baremetal node set <node> --property capabilities='secure_boot:true'
Alternatively use Hardware Inspection to automatically populate the secure boot capability.
Warning
UEFI secure boot only works in UEFI boot mode, see Boot mode support for how to turn it on and off.
Compatible images¶
Use element ubuntu-signed
or fedora
to build signed deploy ISO and
user images with diskimage-builder.
The below command creates files named cloud-image-boot.iso, cloud-image.initrd, cloud-image.vmlinuz and cloud-image.qcow2 in the current working directory:
cd <path-to-diskimage-builder>
./bin/disk-image-create -o cloud-image ubuntu-signed baremetal iso
Ensure the public key of the signed image is loaded into bare metal to deploy signed images.
Enabling with OpenStack Compute¶
Nodes having secure_boot
set to true
may be requested by adding an
extra_spec
to the nova flavor:
openstack flavor set <flavor> --property capabilities:secure_boot="true"
openstack server create --flavor <flavor> --image <image> instance-1
If capabilities
is used in extra_spec
as above, nova scheduler
(ComputeCapabilitiesFilter
) will match only ironic nodes which have
the secure_boot
set appropriately in properties/capabilities
. It will
filter out rest of the nodes.
The above facility for matching in nova can be used in heterogeneous
environments where there is a mix of machines supporting and not supporting
UEFI secure boot, and operator wants to provide a choice to the user
regarding secure boot. If the flavor doesn’t contain secure_boot
then
nova scheduler will not consider secure boot mode as a placement criteria,
hence user may get a secure boot capable machine that matches with user
specified flavors but deployment would not use its secure boot capability.
Secure boot deploy would happen only when it is explicitly specified through
flavor.
Enabling standalone¶
To request secure boot for an instance in standalone mode (without OpenStack
Compute), you need to add the capability directly to the node’s
instance_info
:
baremetal node set <node> --instance-info capabilities='{"secure_boot": "true"}'
Other considerations¶
Internal networks¶
Access to networks which the Bare Metal service uses internally should be prohibited from outside. These networks are the ones used for management (with the nodes’ BMC controllers), provisioning, cleaning (if used) and rescuing (if used).
This can be done with physical or logical network isolation, traffic filtering, etc.
Management interface technologies¶
Some nodes support more than one management interface technology (vendor and IPMI for example). If you use only one modern technology for out-of-band node access, it is recommended that you disable IPMI since the IPMI protocol is not secure. If IPMI is enabled, in most cases a local OS administrator is able to work in-band with IPMI settings without specifying any credentials, as this is a DCMI specification requirement.
Tenant network isolation¶
If you use tenant network isolation, services (TFTP or HTTP) that handle the nodes’ boot files should serve requests only from the internal networks that are used for the nodes being deployed and cleaned.
TFTP protocol does not support per-user access control at all.
For HTTP, there is no generic and safe way to transfer credentials to the node.
Also, tenant network isolation is not intended to work with network-booting a node by default, once the node has been provisioned.
API endpoints for RAM disk use¶
There are two (unauthorized) endpoints in the Bare Metal API that are intended for use by the ironic-python-agent RAM disk. They are not intended for public use.
These endpoints can potentially cause security issues. Access to these endpoints from external or untrusted networks should be prohibited. An easy way to do this is to:
set up two groups of API services: one for external requests, the second for deploy RAM disks’ requests.
to disable unauthorized access to these endpoints in the (first) API services group that serves external requests, the following lines should be added to the policy.yaml file:
# Send heartbeats from IPA ramdisk "baremetal:node:ipa_heartbeat": "rule:is_admin" # Access IPA ramdisk functions "baremetal:driver:ipa_lookup": "rule:is_admin"
Disk Images¶
Ironic relies upon the qemu-img
tool to convert images from a supplied
disk image format, to a raw
format in order to write the contents of a
disk image to the remote device.
By default, only qcow2
format is supported for this operation, however there
have been reports other formats work when so enabled using the
[conductor]permitted_image_formats
configuration option.
Ironic takes several steps by default.
Ironic checks and compares supplied metadata with a remote authoritative source, such as the Glance Image Service, if available.
Ironic attempts to “fingerprint” the file type based upon available metadata and file structure. A file format which is not known to the image format inspection code may be evaluated as “raw”, which means the image would not be passed through
qemu-img
. When in doubt, use araw
image which you can verify is in the desirable and expected state.The image then has a set of safety and sanity checks executed which look for unknown or unsafe feature usage in the base format which could permit an attacker to potentially leverage functionality in
qemu-img
which should not be utilized. This check, by default, occurs only through images which transverse through the conductor.Ironic then checks if the fingerprint values and metadata values match. If they do not match, the requested image is rejected and the operation fails.
The image is then provided to the
ironic-python-agent
.
Images which are considered “pass-through”, as in they are supplied by an
API user as a URL, or are translated to a temporary URL via available
service configuration, are supplied as a URL to the
ironic-python-agent
.
Ironic can be configured to intercept this interaction and have the conductor
download and inspect these items before the ironic-python-agent
will do so,
however this can increase the temporary disk utilization of the Conductor
along with network traffic to facilitate the transfer. This check is disabled
by default, but can be enabled using the
[conductor]conductor_always_validates_images
configuration option.
An option exists which forces all files to be served from the conductor, and
thus force image inspection before involvement of the ironic-python-agent
is the use of the [agent]image_download_source
configuration parameter
when set to local
which proxies all disk images through the conductor.
This setting is also available in the node driver_info
and
instance_info
fields.
Mitigating Factors to disk images¶
In a fully integrated OpenStack context, Ironic requires images to be set to “public” in the Image Service.
A direct API user with sufficient elevated access rights can submit a URL
for the baremetal node instance_info
dictionary field with an
image_source
key value set to a URL. To do so explicitly requires
elevated (trusted) access rights of a System scoped Member,
or Project scoped Owner-Member, or a Project scoped Lessee-Admin via
the baremetal:node:update_instance_info
policy permission rule.
Before the Wallaby release of OpenStack, this was restricted to
admin
and baremetal_admin
roles and remains similarly restrictive
in the newer “Secure RBAC” model.
>>>>>>> 8491abb92 (Harden all image handling and conversion code)