Password Rotation

This guide describes how to change the internal secrets from passwords.yml used by Kolla-Ansible. It does not cover every possible passwords.yml variable, only the most common ones.

Warning

Always back up your passwords.yml file before making any changes. Otherwise, it is easy to make unrecoverable mistakes.

Warning

This guide relies on recent changes to Kolla and Kolla-Ansible. You may encounter errors if applying this guide to older deployments. It is recommended that you update your containers and kolla-ansible to the latest available versions before proceeding.

Regenerating secrets

Passwords can be quickly re-generated using kolla-genpwd.

Assuming an existing /etc/kolla/passwords.yml file, make a backup:

cp /etc/kolla/passwords.yml ./passwords.yml.bak

Edit the passwords.yml file to remove the password strings for any secrets that need to be regenerated i.e. change foo: "bar" to foo:.

Regenerate the removed passwords:

kolla-genpwd -p /etc/kolla/passwords.yml

Applying regenerated secrets

The majority of the secrets can be applied by simply reconfiguring services with kolla-ansible reconfigure. Below is a list of secrets that can be applied this way.

  • *_keystone_password

  • *_database_password (excluding nova_database_password)

  • *_ssh_key (excluding kolla_ssh_key)

  • keystone_admin_password

  • designate_rndc_key

  • keepalived_password

  • libvirt_sasl_password

  • metadata_secret

  • opensearch_dashboards_password

  • osprofiler_secret

  • prometheus_alertmanager_password

  • qdrouterd_password

  • redis_master_password

It is possible to change more secrets however some require manual steps. The manual steps vary depending on the secret. They are listed below in the order they should be applied if they are to be changed at the same time. Once all manual steps are complete, reconfigure services (kolla-ansible reconfigure).

For simplicity, this guide assumes Docker is being used. The same commands should also work for Podman deployments by replacing instances of docker with podman in all relevant commands.

Kolla SSH key

There is currently no mechanism within Kolla-Ansible to rotate kolla_ssh_key. It is however a relatively simple task to perform using a standard Ansible playbook, or can be performed by hand on smaller deployments.

Horizon Secret Key

The Horizon secret key (horizon_secret_key) is unique because it explicitly supports rotation. In reality, it is a Django secret key, and is used for cryptographic signing e.g. generating password recovery links. To minimise user impact, it is possible to set two secret keys at once. The new one will be used for generating new artifacts, while the old one will still be accepted for existing artifacts.

Take note of the old password, generate a new one, and take note of it as well.

Add it to the passwords.yml file, along with the old secret, in this exact format (including quotes in the middle):

horizon_secret_key: newsecret' 'oldsecret

It is important to remember to remove the old key and reconfigure services again, after all old artifacts have expired e.g. after approximately one to two weeks.

Grafana Admin Password

The Grafana admin password (grafana_admin_password) must be rotated manually.

  1. Generate a new Grafana Admin password.

  2. Replace the old password in passwords.yml.

  3. Exec into any Grafana container:

    docker exec -it grafana bash

  4. Run the password reset command, then enter the new password:

    grafana-cli admin reset-admin-password --password-from-stdin

Database Password

The database administrator password (database_password) must be rotated manually.

  1. Generate a new database password.

  2. Replace the old password in passwords.yml, take note of both the old and new passwords.

  3. SSH to a host running a MariaDB container.

  4. Exec into the MariaDB container:

    docker exec -it mariadb bash

  5. Log in to the database. You will be prompted for the password. Use the old value of database_password:

    mysql --batch -uroot -p

  6. Check the current state of the root user:

    SELECT Host,User,Password FROM mysql.user WHERE User='root';

  7. Update the password for the root user:

    SET PASSWORD FOR 'root'@'%' = PASSWORD('newpassword');

  8. Check that the password hash has changed in the user list:

    SELECT Host,User,Password FROM mysql.user WHERE User='root';

  9. If there are any remaining root users with the old password e.g. root@localhost, change the password for them too.

Nova Database Password

The nova database admin user password (nova_database_password) must be rotated manually.

Warning

From this point onward, API service may be disrupted.

  1. Generate a new Nova database password.

  2. Replace the old password in passwords.yml.

  3. Exec into the nova_conductor container:

    docker exec -it nova_conductor bash

  4. List the cells:

    nova-manage cell_v2 list_cells --verbose

  5. Find the entry for cell0, copy the Database Connection value, replace the password in the string with the new value, and update it with the following command:

    nova-manage cell_v2 update_cell --cell_uuid 00000000-0000-0000-0000-000000000000 --database_connection "CONNECTION WITH NEW PASSWORD HERE" --transport-url "none:///"

    (If the cell_uuid for cell0 is not 00000000-0000-0000-0000-000000000000, change the above command accordingly)

Heat Domain Admin Password

The keystone password for the heat domain admin service user (heat_domain_admin_password) must be rotated manually.

It can be changed by an administrator just like any other standard OpenStack user password. Generate a new password, replace the old password in passwords.yml, then apply the change manually:

openstack user set --password <password> heat_domain_admin --domain heat_user_domain

RabbitMQ Secrets

RabbitMQ uses two main secrets. An Erlang cookie for cluster membership (rabbitmq_cluster_cookie), and a RabbitMQ management user password (rabbitmq_password). There is currently no documented process for seamlessly rotating these secrets. Many OpenStack services use RabbitMQ for communication and reconfiguring them with the new credentials can take some time, resulting in a relatively long API outage.

It is recommended that you stop all services, then stop and destroy the RabbitMQ containers and volumes. Because the RabbitMQ containers are destroyed, kolla-ansible deploy should be used to restart services rather than kolla-ansible reconfigure. Detailed steps are listed below:

  1. Generate a new rabbitmq_cluster_cookie and rabbitmq_password.

  2. Replace the old values in passwords.yml.

  3. Stop OpenStack services:

    kolla-ansible -i inventory stop

  4. On each node running RabbitMQ, destroy its containers and volumes:

    docker stop rabbitmq
docker rm rabbitmq
docker volume rm rabbitmq

  5. Redeploy services:

    kolla-ansible -i inventory deploy

Post-redeploy changes

Once services have been redeployed, the existing Memcached data should be flushed. The old Memcached password will no longer be used so any data stored using it will be inaccessible.

The instructions below must be run from a host that has access to the network the Memcached containers are using. If you are not sure, run them from a host that is running Memcached.

  1. Install a telnet client:

    apt/dnf install telnet

  2. Check the config for the IP and port used by Memcached (on every host running Memcached):

    sudo grep command /etc/kolla/memcached/config.json

    The IP and port will be printed after -l and -p respectively

  3. For each container start a Telnet session, clear all data, then exit:

    telnet <ip> <port>
flush_all
quit

Known out-of-scope secrets

Below is a list of passwords that are known to be outside the scope of this guide.

  • docker_registry_password - kolla-ansible cannot manage docker registries.