Stage 5: Upgrade And Maintain

The final stage in the Scaling Journey is Upgrade and Maintain.

Once you have successfully scaled out, how to do you effectively upgrade and maintain your deployment going forward? This stage is about establishing regular operations to keep your deployment up to date and benefit from the latest bugfixes and operating system improvements. This page aims to help answer those questions.


Q: Scaled out my Cloud… now how can I upgrade my OpenStack Infrastructure?

A: This is a simple question that has a very complex answer… It depends! on your setup. Because you are running a Large Scale infrastructure I’m assuming that all your OpenStack projects are running on independent servers. In this case you can look at each OpenStack project separately which makes the planning and testing much easier. An OpenStack project upgrade usually requires a small downtime (for the database upgrade and data migrations). However, there are projects that didn’t change the database version during several versions, for example Glance, and in these cases you can do a live upgrade. Also, there are projects that allow online data migrations to avoid long upgrades interventions, for example Nova.

Definitely there are projects much more challenging to upgrade than others. For example, Keystone, Glance, Cinder are relatively easy to upgrade because they only have few components. To upgrade these projects, at CERN usually we install new nodes with the upgraded version and the upgrade is basically replacing the nodes in the Load Balancer. If database migrations are required, they are done before enabling the new nodes.

Don’t forget to test the functionality and all the procedures in a test infrastructure before attempting an upgrade!

For projects with a lot of components, like Nova, upgrades are definitely more challenging. Depending on the number of servers for each component, replacing them may not be always feasible. At CERN we usually upgrade the Nova components in place. In a Large Scale infrastructure, with several Cells, upgrading Nova involves upgrading several nodes and several database migrations (remember, each Cell has one database). We usually perform data migrations during the upgrade window. The upgrade of nova-compute happens when the infrastructure is already open to the users again. Upgrading this component is usually transparent to users and we do it per cell. Remember to test and review the “upgrade levels” configuration for each cell.

Again, upgrades are always challenging! Make sure you test and practice all the procedures before attempting the upgrade in a Large Scale production infrastructure.

Q: Can I run different versions of the OpenStack projects in the same Infrastructure?

A: Yes, you can!

But in this case it’s better if each OpenStack project runs on different servers to avoid library conflicts. For example you can have Keystone and Glance running Victoria release and Nova still in Stein.

Q: How can I upgrade the base Operating System of my OpenStack Infrastructure?

A: Of course, this depends on your base Operating System. At CERN we use CentOS. We have automatic upgrades enabled and every week the nodes receive the normal packages upgrades (CERN stages packages upgrades per week and they are tested meanwhile in the QA infrastructure). We usually stage minor Operating System upgrades per Cell (for example from CentOS 7.8 to 7.9). This allows us to identify issues (unfortunately, it is difficult to detect everything in QA especially when having several hardware types) and to contain them per Cell.

Major Operating System upgrades are more challenging (for example from CentOS 7 to CentOS 8). The OpenStack control plane can be easily upgraded replacing the nodes. For the compute nodes it can be very disruptive. In place upgrades usually are not recommended and live migration between different versions may not work.

Q: MySQL upgrades for OpenStack projects?

A: At CERN when we deployed the Infrastructure almost 8 years ago, we were running MySQL 5.6. We just reached the EOL (05/02/2021) of this MySQL version.

Upgrading the databases to 5.7 didn’t pose any issue for all the OpenStack projects. Make sure you backup your data! Just in case.

Also, dumping the data from MySQL 5.6 and moving it to MySQL 5.7 we didn’t identify any issue. Currently we are in the process to move to MySQL 8. For now we didn’t identify any issue when moving to data from different OpenStack projects running on MySQL 5.7 to MySQL 8.

Q: RabbitMQ clusters upgrades?

A: I won’t recommend having automatic upgrades enabled for RabbitMQ nodes, especially if running in a cluster configuration. Treat your RabbitMQ clusters as your pets!

Any issue in a RabbitMQ cluster can impact the infrastructure, especially if we are talking about a Large Scale infrastructure.