As of the Newton release, keystone encrypts all credentials stored in the
sql backend. Credentials are encrypted with the same mechanism used
to encrypt Fernet tokens,
fernet. Keystone provides only one type of
credential encryption but the encryption provider is pluggable in the event
you wish to supply a custom implementation.
This document details how credential encryption works, how to migrate existing credentials in a deployment, and how to manage encryption keys for credentials.
Configuring credential encryption¶
The configuration for credential encryption is straightforward. There are only two configuration options needed:
[credential] provider = fernet key_repository = /etc/keystone/credential-keys/
[credential] provider defaults to the only option supplied by keystone,
fernet. There is no reason to change this option unless you wish to provide
a custom credential encryption implementation. The
key_repository location is a requirement of using
fernet but will default
/etc/keystone/credential-keys/ directory. Both
[fernet_tokens] key_repository define locations for
keys used to encrypt things. One holds the keys to encrypt and decrypt
credentials and the other holds keys to encrypt and decrypt tokens. It is
imperative that these repositories are managed separately and they must not
share keys. Meaning they cannot share the same directory path. The
[credential] key_repository is only allowed to have three keys. This is not
configurable and allows for credentials to be re-encrypted periodically with a
new encryption key for the sake of security.
How credential encryption works¶
The implementation of this feature did not change any existing credential API contracts. All changes are transparent to the user unless you’re inspecting the credential backend directly.
When creating a credential, keystone will encrypt the
blob attribute before
persisting it to the backend. Keystone will also store a hash of the key that
was used to encrypt the information in that credential. Since Fernet is used to
encrypt credentials, a key repository consists of multiple keys. Keeping track
of which key was used to encrypt each credential is an important part of
encryption key management. Why this is important is detailed later in the
Encryption key management section.
When updating an existing credential’s
blob attribute, keystone will encrypt
blob and update the key hash.
When listing or showing credentials, all
blob attributes are decrypted in
the response. Neither the cipher text, nor the hash of the key used to encrypt
blob are exposed through the API. Furthermore, the key is only used
internally to keystone.
Encrypting existing credentials¶
When upgrading a Mitaka deployment to Newton, three database migrations will ensure all credentials are encrypted. The process is as follows:
An additive schema change is made to create the new
key_hashcolumns in the existing
keystone-manage db_sync --expand.
A data migration will loop through all existing credentials, encrypt each
bloband store the result in the new
encrypted_blobcolumn. The hash of the key used is also written to the
key_hashcolumn for that specific credential. This step is done using
keystone-manage db_sync --migrate.
A contractive schema will remove the
blobcolumn that held the plain text representations of the credential using
keystone-manage db_sync --contract. This should only be done after all nodes in the deployment are running Newton. If any Mitaka nodes are running after the database is contracted, they won’t be able to read credentials since they are looking for the
blobcolumn that no longer exists.
You may also use
keystone-manage db_sync --check in order to check the
current status of your rolling upgrades.
If performing a rolling upgrade, please note that a limited service outage will
take affect during this migration. When the migration is in place, credentials
will become read-only until the database is contracted. After the contract
phase is complete, credentials will be writeable to the backend. A
[credential] key_repository location must be specified through
configuration and bootstrapped with keys using
credential_setup prior to migrating any existing credentials. If a new key
repository isn’t setup using
keystone-manage credential_setup keystone will
assume a null key to encrypt and decrypt credentials until a proper key
repository is present. The null key is a key consisting of all null bytes and
its only purpose is to ease the upgrade process from Mitaka to Newton. It is
highly recommended that the null key isn’t used. It is no more secure than
storing credentials in plain text. If the null key is used, you should migrate
to a proper key repository using
keystone-manage credential_setup and
Encryption key management¶
Key management of
[credential] key_repository is handled with three
keystone-manage credential_setup will populate
key_repository with new encryption keys. This must be done in order for
proper credential encryption to work, with the exception of the null key. This
step should only be done once.
keystone-manage credential_rotate will create and rotate a new encryption
key in the
[credential] key_repository. This will only be done if all
credential key hashes match the hash of the current primary key. If any
credential has been encrypted with an older key, or secondary key, the rotation
will fail. Failing the rotation is necessary to prevent overrotation, which
would leave some credentials indecipherable since the key used to encrypt it
no longer exists. If this step fails, it is possible to forcibly re-key all
credentials using the same primary key with
keystone-manage credential_migrate will check the backend for credentials
whose key hash doesn’t match the hash of the current primary key. Any
credentials with a key hash mismatching the current primary key will be
re-encrypted with the current primary key. The new cipher text and key hash
will be updated in the backend.