Rocky Series Release Notes

Bug Fixes

  • Newer releases of CentOS ship a version of libnss that depends on the existance of /dev/random and /dev/urandom in the operating system in order to run. This causes a problem during the cache preparation process which runs inside chroot that does not contain this, resulting in errors with the following message.

    error: Failed to initialize NSS library

    This has been resolved by introducing a /dev/random and /dev/urandom inside the chroot-ed environment.

  • With the release of CentOS 7.6, deployments were breaking and becoming very slow when we restart dbus in order to catch some PolicyKit changes. However, those changes were never actaully used so they were happening for no reason. We no longer make any modifications to the systemd-machined configuration and/or PolicyKit to maintain upstream compatibility.

New Features

  • The option lxc_hosts_container_image_url has been added allowing deployers to define their base image url to whatever it needs to be removing the requirement for operators to maintain an internal LXC index in the event they want to host a private repository.

  • The option lxc_hosts_container_image_download_legacy has been added allowing a deployer to enable the use of the legacy lxc image repository. This option is a Boolean and has a default of false.

  • The variable lxc_user_defined_container has been added to the lxc_hosts role allowing deployers to define the variable file loaded when preparing a base container image. This option defaults to using a base image most closely associated with the underlying OS however should a deployer need, this option can be used to customize the base container image for a given host.

  • An option to disable the machinectl quota system has been changed. The variable lxc_host_machine_quota_disabled is a Boolean with a default of false. When this option is set to true it will disable the machinectl quota system.

  • The options lxc_host_machine_qgroup_space_limit and lxc_host_machine_qgroup_compression_limit have been added allowing a deployer to set qgroup limits as they see fit. The default value for these options is “none” which is effectively unlimited. These options accept any nominal size value followed by the single letter type, example 64G. These options are only effective when the option lxc_host_machine_quota_disabled is set to false.

Deprecation Notes

  • The variable lxc_image_cache_server_mirrors has been deprecated in the “lxc_hosts” role. This option has been replaced by the static variable lxc_hosts_container_image_url. This variable will continue to function as a single element list allowing existing automation to function when in legacy image mode but should not be considered in use by default.

  • The variable lxc_image_cache_server has been deprecated in the lxc_hosts role. This option has been replaced by the static variable lxc_hosts_container_image_url.

  • The option cache_prep_commands from lxc_cache_map has been removed. This option has been converted to a template file within the lxc_hosts role. In order to set specific cache commands within the template it is recommended that deployers set lxc_cache_prep_pre_commands or lxc_cache_prep_post_commands. If the entire prep script needs to be overridden deployers can set lxc_cache_prep_template to the full local path of the prep template and the role will use this script irrespective of the base container type.

Other Notes

  • The use of is no longer required. While the images provided by that build system are perfectly functional they have been less than optimal in a lot ways for a very long time. The lxc_hosts role will now pull a base image from the upstream distro being deployed. If a deployer wishes to continue using the images from they are welcome to but it is no longer forced.

Security Issues

  • The PermitRootLogin in sshd_config changed from ‘yes’ to ‘prohibit-password’ in the containers. By default there is no password set in the containers but the ssh pub key from the deployment host is injected in the targets nodes authorized_keys.

New Features

  • The lxcbr0 bridge now allows NetworkManager to control it, which allows for networks to start in the correct order when the system boots. In addition, the NetworkManager-wait-online.service is enabled to ensure that all services that require networking to function, such as keepalived, will only start when network configuration is complete. These changes are only applied if a deployer is actively using NetworkManager in their environment.

Other Notes

  • CentOS deployments require a special COPR repository for modern LXC packages. The COPR repository is not mirrored at this time and this causes failed gate tests and production deployments.

    The role now syncs the LXC packages down from COPR to each host and builds a local LXC package repository in /opt/thm-lxc2.0. This greatly reduces the amount of times that packages must be downloaded from the COPR server during deployments, which will reduce failures until the packages can be hosted with a more reliable source.

    In addition, this should speed up playbook runs since yum can check a locally-hosted repository instead of a remote repository with availability and performance challenges.

New Features

  • The maximum amount of time to wait until forcibly failing the LXC cache preparation process is now configurable using the lxc_cache_prep_timeout variable. The value is specified in seconds, with the default being 20 minutes.

New Features

  • The lxc_cache_distro_packages has been moved to the role defaults from vars to enable easier overriding of the container cache package list.

  • A new LXC container template has been added which will allow us to better manage containers on the host machines we support. The new template uses the machinectl command to create container rootfs using the existing cache. This in-turn will provide easier management of container images, faster build times, and the ability to instantly clone a container (or a given variant) without impacting a containers state. This new lxc container create template, and the features it provides, will only impact new containers created allowing deployers to safely adopt this change in any existing environment.

  • Deployers can set lxc_hosts_opensuse_mirror_url to use their preferred mirror for the openSUSE repositories. They can also set the lxc_hosts_opensuse_mirror_obs_url if they want to set a different mirror for the OBS repositories. If they want to use the same mirror in both cases then they can leave the latter variable to its default value. The full list of mirrors and their capabilities can be obtained at

Bug Fixes

  • In Ubuntu the dnsmasq package actually includes init scripts and service configuration which conflict with LXC and are best not included. The actual dependent package is dnsmasq-base. The package list has been adjusted and a task added to remove the dnsmasq package and purge the related configuration files from all LXC hosts.

New Features

  • Add support for Ubuntu on IBM z Systems (s390x).

New Features

  • The COPR repository for installing LXC on CentOS 7 is now set to a higher priority than the default to ensure that LXC packages always come from the COPR repository.

  • LXC on CentOS is now installed via package from a COPR repository rather than installed from the upstream source.

  • The variable lxc_net_manage_iptables has been added. This variable can be overridden by deployers if system wide iptables rules are already in place or managed by deployers chioce.

New Features

  • The variable lxc_image_cache_server_mirrors has been added to the “lxc_hosts” role. This is a list type variable and gives deployers the ability to specify multiple lxc-image mirrors at the same time.

Deprecation Notes

  • The variable lxc_image_cache_server has been deprecated in the “lxc_hosts” role. By default this value will pull the first item out of lxc_image_cache_server_mirrors list which is only done for compatibility (legacy) purposes. The default string type variable, lxc_image_cache_server, will be removed from the “lxc_hosts” role in the in “R” release.

New Features

  • IPv6 support has been added for the LXC bridge network. This can be configured using lxc_net6_address, lxc_net6_netmask, and lxc_net6_nat.

New Features

  • The container cache preparation process now allows copy-on-write to be set as the lxc_container_backing_method when the lxc_container_backing_store is set to lvm. When this is set a base container will be created using a name of the form <linux-distribution>-distribution-release>-<host-cpu-architecture>. The container will be stopped as it is not used for anything except to be a backing store for all other containers which will be based on a snapshot of the base container.

  • When using copy-on-write backing stores for containers, the base container name may be set using the variable lxc_container_base_name which defaults to <linux-distribution>-distribution-release>-<host-cpu-architecture>.

  • The lxc_hosts role can now make use of a primary and secondary gpg keyserver for gpg validation of the downloaded cache. Setting the servers to use can be done using the lxc_image_cache_primary_keyserver and lxc_image_cache_secondary_keyserver variables.

  • The lxc_hosts role now supports the ability to configure whether apt/yum tasks install the latest available package, or just ensure that the package is present. The default action is to ensure that the latest package is present. The action taken may be changed to only ensure that the package is present by setting lxc_hosts_package_state to present.

Upgrade Notes

  • The variable lxc_apt_packages has been renamed to lxc_hosts_distro_packages.

  • The lxc_hosts role always checks whether the latest package is installed when executed. If a deployer wishes to change the check to only validate the presence of the package, the option lxc_hosts_package_state should be set to present.

New Features

  • The container cache preparation process now allows overlayfs to be set as the lxc_container_backing_store. When this is set a base container will be created using a name of the form <linux-distribution>-distribution-release>-<host-cpu-architecture>. The container will be stopped as it is not used for anything except to be a backing store for all other containers which will be based on a snapshot of the base container. The overlayfs backing store is not recommended to be used for production unless the host kernel version is 3.18 or higher.

Upgrade Notes

  • Hosts running LXC on Ubuntu 14.04 will now need to enable the “trusty-backports” repository. The backports repo on Ubuntu 14.04 is now required to ensure LXC is updated to the latest stable version.

New Features

  • The lxc_host cache prep has been updated to use the LXC download template. This removes the last remaining dependency the project has on the rpc-trusty-container.tgz image.

  • The lxc_host role will build lxc cache using the download template built from images found here. These images are upstream builds from the greater LXC/D community.

  • The lxc_host role introduces support for CentOS 7 and Ubuntu 16.04 container types.

  • Support had been added to allow the functional tests to pass when deploying on ppc64le architecture using the Ubuntu distributions.

Upgrade Notes

  • The ca-certificates package has been included in the LXC container build process in order to prevent issues related to trying to connect to public websites which make use of newer certificates than exist in the base CA certificate store.

  • The LXC container cache preparation process now copies package repository configuration from the host instead of implementing its own configuration. The following variables are therefore unnecessary and have been removed:

    • lxc_container_template_main_apt_repo

    • lxc_container_template_security_apt_repo

    • lxc_container_template_apt_components

  • The LXC container cache preparation process now copies DNS resolution configuration from the host instead of implementing its own configuration. The lxc_cache_resolvers variable is therefore unnecessary and has been removed.

  • The lxc_host role no longer uses the distro specific lxc container create template.

  • The following variable changes have been made in the lxc_host role:

    • lxc_container_user_password: Removed because the default lxc container user is no longer created by the lxc container template.

    • lxc_container_template_options: This option was renamed to lxc_cache_download_template_options. The deprecation filter was not used because the values provided from this option have been fundamentally changed and potentially old overrides will cause problems.

    • lxc_container_base_delete: Removed because the cache will be refreshed upon role execution.

    • lxc_cache_validate_certs: Removed because the Ansible get_url module is no longer used.

    • lxc_container_caches: Removed because the container create process will build a cached image based on the host OS.

Bug Fixes

  • The check to validate whether an appropriate ssh public key is available to copy into the container cache has been corrected to check the deployment host, not the LXC host.