OSSN-0090: Best practices when configuring Glance with COW backends

Summary

When deploying Glance in a popular configuration where Glance shares a common storage backend with Nova and/or Cinder, it is possible to open some known attack vectors by which malicious data modification can occur. This note reviews the known issues and suggests a Glance deployment configuration that can mitigate such attacks.

Affected Services / Software

Glance, all supported releases (Queens through Zed)

Discussion

This note applies to you if you are operating a glance-api service with the show_multiple_locations option set to True (the default value is False) or if your end-user-facing glance-api has the show_image_direct_url option set to True (default value is False).

Our recommendation is that the image locations and direct_url fields never be displayed to end users in a cloud. This can be accomplished by running two glance-api services:

  • A “user-facing” glance-api that is accessible to end users and which appears in users’ service catalogs.

  • An “internal-only-facing” glance-api that is accessible only to those services that require access to the direct_url or image location fields, and which is protected by firewalls from access by end users. (Nova, Cinder, and Ironic all have configuration options to specify the Glance API endpoint each service uses.)

This dual glance-api deployment was suggested in “Known Issues” in Glance release notes in the Rocky through Ussuri releases, but it seems that the idea has not received sufficient attention. Hence this security note.

The attack vector that becomes available when image locations are exposed to users was originally outlined in OSSN-0065, though that note was not clear about the attack surface or mitigation, and contained some forward-looking statements that were not fulfilled. The attack vector is:

A malicious user could create an image in Glance, set an additional location on that image pointing to an altered image, then delete the original location, so that consumers of the original image would unwittingly be using the malicious image. Note, however, that this attack vector cannot change the original image’s checksum, and it is limited to images that are owned by the attacker.

OSSN-0065 suggests that this is only an issue when users do not checksum their image data. It neglects the fact that in some popular deployment configurations in which Nova creates a root disk snapshot, data is never uploaded to Glance, but instead a snapshot is created directly in the backend and Nova creates a Glance image record with “size” 0 and an empty os_hash_value, making it impossible to compare the hash of downloaded image data to the value maintained by Glance. Thus, if the backend store is itself compromised and image data is replaced directly in the backend, the substitution will not be detected.

Further, when Nova is configured to use the same storage for ephemeral disks that is used as a Glance image store, Nova efficiently creates a server root disk directly in the backend without checksumming the image data. This is an intentional design choice to optimize storage space and host resources, but an implication is that even if the image record has a recorded hash, it is not being checked at the point of image consumption.

Similarly, when using a shared backend, or a cinder glance_store, Cinder will efficiently clone a volume created from an image directly in the backend without checksumming the image data. Again, this is done intentionally in order to optimize resources, but it is important to be aware of the security tradeoff being made by this configuration.

OSSN-0065 suggested that the attack vector of substituting image data by modifying the image locations could be addressed by using policies, but that has turned out not to be the case. The only way currently to mitigate this vector is to deploy glance-api in a dual configuration as described above, namely with an internal-only-facing glance-api used by Nova and Cinder (that has show_multiple_locations enabled), and an end- user-facing glance-api (that has show_multiple_locations disabled).

So far the focus has been on show_multiple_locations. When that setting is disabled in Glance, it is not possible to manipulate the locations via the OpenStack Images API. Keep in mind, however, that in any Glance/Nova/Cinder configuration where Nova and/or Cinder do copy-on- write directly in the image store, image data transfer takes place outside Glance’s image data download path, and the os_hash_value is not checked.

This brings us to the show_image_direct_url option, which includes a direct_url field in the image-show response that can be used by various OpenStack services to consume images directly from the storage backend. Exposing the direct_url to end users leaks information about the storage backend. The direct_url, being read-only, may appear innocuous, but its use by services is usually to perform some kind of optimized image data access that most likely does not include computing a hash of the image data.

We therefore recommend that OpenStack services that require exposure of the direct_url image property be similarly configured to use an internal-only-facing glance-api. End users who wish to download an image do not require access to the direct_url image property because they can simply use the image-data-download API call.

Contacts / References

Author: Brian Rosmaita, Red Hat