Liberty Series Release Notes

12.2.5

Bug Fixes

  • Fix run-upgrade.sh so that it correctly calls nova-flavor-migration.yml and no longer fails due to the non-existent playbook nova-extra-migrations.yml.

12.2.4

New Features

  • AIDE is configured to skip the entire /var directory when it does the database initialization and when it performs checks. This reduces disk I/O and allows these jobs to complete faster.

    This also allows the initialization to become a blocking process and Ansible will wait for the initialization to complete prior to running the next task.

  • Although the STIG requires martian packets to be logged, the logging is now disabled by default. The logs can quickly fill up a syslog server or make a physical console unusable.

    Deployers that need this logging enabled will need to set the following Ansible variable:

    security_sysctl_enable_martian_logging: yes

Upgrade Notes

  • The upgrade playbook nova-flavor-migration.yml will perform a migration of nova flavor data. This will need to be completed prior to upgrading to Liberty. It is recommended that Kilo be deployed from the eol-kilo tag prior to upgrading to Liberty to ensure that this task is completed successfully.

    This upgrade task is related to bug 1594584.

  • All of the discretionary access control (DAC) auditing is now disabled by default. This reduces the amount of logs generated during deployments and minor upgrades. The following variables are now set to no:

    security_audit_DAC_chmod: no
security_audit_DAC_chown: no
security_audit_DAC_lchown: no
security_audit_DAC_fchmod: no
security_audit_DAC_fchmodat: no
security_audit_DAC_fchown: no
security_audit_DAC_fchownat: no
security_audit_DAC_fremovexattr: no
security_audit_DAC_lremovexattr: no
security_audit_DAC_fsetxattr: no
security_audit_DAC_lsetxattr: no
security_audit_DAC_setxattr: no

  • New overrides are provided to allow for better customization around logfile retention and rate limiting for UDP/TCP sockets. rsyslog_server_logrotation_window defaults to 14 days rsyslog_server_ratelimit_interval defaults to 0 seconds rsyslog_server_ratelimit_burst defaults to 10000

  • The rsyslog.conf is now using v7+ style configuration settings

Bug Fixes

  • The /run directory is excluded from AIDE checks since the files and directories there are only temporary and often change when services start and stop.

  • AIDE initialization is now always run on subsequent playbook runs when initialize_aide is set to yes. The initialization will be skipped if AIDE isn’t installed or if the AIDE database already exists.

    See bug 1616281 for more details.

  • The auditd rules for auditing V-38568 (filesystem mounts) were incorrectly labeled in the auditd logs with the key of export-V-38568. They are now correctly logged with the key filesystem_mount-V-38568.

12.2.2

Known Issues

  • For OpenStack-Ansible Liberty releases earlier than 12.2.2 the default container apt source used was https://mirror.rackspace.com/ubuntu. This mirror seems to sometimes have broken package indexes or missing packages. The default package source has therefore been changed to make use of http://archive.ubuntu.com/ubuntu for packages and http://security.ubuntu.com/ubuntu for security packages.

Upgrade Notes

  • The default container apt sources have been changed from using https://mirror.rackspace.com/ubuntu to http://archive.ubuntu.com/ubuntu for packages and http://security.ubuntu.com/ubuntu for security packages. This is to resolve issues with unavailable packages during the install process due to incomplete mirror updates.

Bug Fixes

  • The --compact flag has been removed from xtrabackup options. This had been shown to cause crashes in some SST situations

12.2.0

New Features

  • A new variable has been added to allow a deployer to control the restart of containers via the handler. This new option is lxc_container_allow_restarts and has a default of true. If a deployer wishes to disable the auto-restart functionality they can set this value to false and automatic container restarts that are not absolutely required will be disabled.

  • The py_pkgs lookup plugin now has strict ordering for requirement files discovered. These files are used to add additional requirements to the python packages discovered. The order is defined by the constant, REQUIREMENTS_FILE_TYPES which contains the following entries, ‘test-requirements.txt’, ‘dev-requirements.txt’, ‘requirements.txt’, ‘global-requirements.txt’, ‘global-requirement-pins.txt’. The items in this list are arranged from least to most priority.

  • The repo_build role now provides the ability to override the upper-constraints applied which are sourced from OpenStack and from the global-requirements-pins.txt file. The variable repo_build_upper_constraints_overrides can be populated with a list of upper constraints. This list will take the highest precedence in the constraints process, with the exception of the pins set in the git source SHAs.

Upgrade Notes

  • During a kilo to liberty upgrade, container and service restarts for the mariadb/galera cluster were being triggered multiple times and causing the cluster to become unstable and often unrecoverable. This situation has been improved immensely, and we now have tight control such that restarts of the galera containers only need to happen once, and are done so in a controlled, predictable and repeatable way.

Bug Fixes

12.1.0

Upgrade Notes

  • Cleanup tasks are added to remove the nova console git directories /usr/share/novnc and /usr/share/spice-html5, prior to cloning these inside the nova vnc and spice console playbooks. This is necessary to guarantee that local modifications do not break git clone operations, especially during upgrades.

Bug Fixes

  • The upgrade step to remove legacy MariaDB apt sources was failing due to the destruction of the repo containers. This issue has now been fixed by skipping the repo containers in this step.

12.0.16

New Features

  • The audit rules added by the security role now have key fields that make it easier to link the audit log entry to the audit rule that caused it to appear.

  • Apache MPM tunable support has been added to the os-keystone role in order to allow MPM thread tuning. Default values reflect the current Ubuntu default settings:

    keystone_httpd_mpm_backend: event
keystone_httpd_mpm_start_servers: 2
keystone_httpd_mpm_min_spare_threads: 25
keystone_httpd_mpm_max_spare_threads: 75
keystone_httpd_mpm_thread_limit: 64
keystone_httpd_mpm_thread_child: 25
keystone_httpd_mpm_max_requests: 150
keystone_httpd_mpm_max_conn_child: 0

Upgrade Notes

  • During the upgrade from Kilo to Liberty, this change deletes the repo containers and recreates them to fix an upgrade issue with dependencies.

Bug Fixes

  • The role previously did not restart the audit daemon after generating a new rules file. The bug has been fixed and the audit daemon will be restarted after any audit rule changes.

  • The dictionary-based variables in defaults/main.yml are now individual variables. The dictionary-based variables could not be changed as the documentation instructed. Instead it was required to override the entire dictionary. Deployers must use the new variable names to enable or disable the security configuration changes applied by the security role. For more information, see Launchpad Bug 1577944.

  • Failed access logging is now disabled by default and can be enabled by changing security_audit_failed_access to yes. The rsyslog daemon checks for the existence of log files regularly and this audit rule was triggered very frequently, which led to very large audit logs.

  • The security role previously set the permissions on all audit log files in /var/log/audit to 0400, but this prevents the audit daemon from writing to the active log file. This will prevent auditd from starting or restarting cleanly.

    The task now removes any permissions that are not allowed by the STIG. Any log files that meet or exceed the STIG requirements will not be modified.

  • When the security role was run in Ansible’s check mode and a tag was provided, the check_mode variable was not being set. Any tasks which depend on that variable would fail. This bug is fixed and the check_mode variable is now set properly on every playbook run.

  • The security role now handles ssh_config files that contain Match stanzas. A marker is added to the configuration file and any new configuration items will be added below that marker. In addition, the configuration file is validated for each change to the ssh configuration file.

12.0.15

New Features

  • Deployers can now blacklist certain Nova extensions by providing a list of such extensions in horizon_nova_extensions_blacklist variable, for example:

    horizon_nova_extensions_blacklist:
  - "SimpleTenantUsage"

  • The ability to support MultiStrOps has been added to the config_template action plugin. This change updates the parser to use the set() type to determine if values within a given key are to be rendered as MultiStrOps. If an override is used in an INI config file the set type is defined using the standard yaml construct of “?” as the item marker.

    # Example Override Entries
Section:
  typical_list_things:
    - 1
    - 2
  multistrops_things:
    ? a
    ? b

    # Example Rendered Config:
[Section]
typical_list_things = 1,2
multistrops_things = a
multistrops_things = b

  • Added horizon_apache_custom_log_format tunable to the os-horizon role for changing CustomLog format. Default is “combined”.

  • Added keystone_apache_custom_log_format tunable for changing CustomLog format. Default is “combined”.

Upgrade Notes

  • The Kilo upgrade playbook glance-db-storage-url-fix.yml to Liberty will migrate all existing Swift backed Glance images inside the image_locations database table from a Keystone v2 API URL to a v3 URL. This will force the Swift client to operate against a v3 Keystone URL. A backup of the old image_locations table is stored inside a new database table image_locations_keystone_v3_mig_pre_liberty and can be safely removed after a successfull upgrade to Liberty.

    This upgrade task is related to bug 1582279.

Bug Fixes

  • Previously, the ansible_managed var was being used to insert a header into the swift.conf that contained date/time information. This meant that swift.conf across different nodes did not have the same MD5SUM, causing swift-recon --md5 to break. We now insert a piece of static text instead to resolve this issue.

  • The /var/lib/libvirt/qemu/save directory is now a symlink to {{ nova_system_home_folder }}/save to resolve an issue where the default location used by the libvirt managed save command can result with the root partitions on compute nodes becoming full when nova image-create is run on large instances.

12.0.14

New Features

  • The openstack-ansible-memcached_server role includes a new override,`memcached_connections` which is automatically calculated from the number of memcached connection limit plus additional 1k to configure the OS nofile limit. Without proper nofile limit configuration, memcached will crash in order to support higher parallel connection TCP/Memcache counts.

Known Issues

  • Ceilometer does not support V3 endpoints in Liberty, which are the flavor created by OSA. To deploy Ceilometer some endpoints in the Keystone service catalog must be removed and replaced with V2 endpoints. This is neccessary, for example, to use the Swift pollster to collect metrics for Swift storage use the Swift endpoint. For detailed instructions on the steps for these changes to the service catalog see the OpenStack Liberty Install Guide <http://docs.openstack.org/liberty>.

Upgrade Notes

  • A new nova admin endpoint will be registered with the suffix /v2.1/%(tenant_id)s. The nova admin endpoint with the suffix /v2/%(tenant_id)s may be manually removed.

Bug Fixes

  • The nova admin endpoint is now correctly registered as /v2.1/%(tenant_id)s instead of /v2/%(tenant_id)s.

12.0.13

New Features

  • Allow the fallocate_reserve option to be set (in bytes) for Swift, to help prevent disks from filling up and prevent a situation where Swift is unable to remove objects due to a lack of disk space. The fallocate_reserve value to is set to a default of 10GB.

Security Issues

  • A sudoers entry is added to the repo_servers to allow the nginx user to stop and start NGINX from the init script. This ensures that the repo sync process can shut off NGINX while synchronizing data from master to slaves.

Bug Fixes

  • Containers might fail to retrieve packages from the repo server when connecting to a slave repo server that has not finished synchronizing. For more information, see https://bugs.launchpad.net/openstack-ansible/+bug/1543146. This is addressed by adding pre and post hooks into lsyncd to connect to the slave repo servers and disable NGINX for the duration for the sync.

12.0.12

Known Issues

  • Paramiko version 2.0 Python requires the Python cryptography library. New system packages must be installed for this library. For OpenStack-Ansible versions <12.0.12, <11.2.15, <13.0.2 the system packages must be installed on the deployment host manually by executing apt-get install -y build-essential libssl-dev libffi-dev.

Bug Fixes

  • The XFS filesystem is excluded from the daily mlocate crond job in order to conserve disk IO for large IOPS bursts due to updatedb/mlocate file indexing.

12.0.11

Upgrade Notes

  • The MariaDB wait_timeout setting is decreased to 1h to match the SQL Alchemy pool recycle timeout, in order to prevent unnecessary database session buildups.

12.0.10

New Features

  • The haproxy-install.yml playbook will now be run as a part of setup-infrastructure.yml.

  • LBaaS v2 is available for deployment in addition to LBaaS v1. Both versions are mutually exclusive and cannot be running at the the same time. Deployers will need to re-create any existing load balancers if they switch between LBaaS versions. Switching to LBaaS v2 will stop any existing LBaaS v1 load balancers.

  • New rabbitmq-server role override rabbitmq_async_threads defaults to 128 threads for IO operations inside the RabbitMQ erlang VM. This setting doubled the threads for IO operations.

  • New rabbitmq-server role override rabbitmq_process_limit defaults to 1048576 for number of concurrent processes inside the erlang VM. Each network connection and file handle does need its own process inside erlang.

  • Services deploy into virtual environments by default when the service relies on Python. Find the virtualenv for each service under /openstack/venvs/ on the host or in the container where the service is deployed. Disable the use of virtualenv by overriding the service-specific variable (for example cinder_venv_enabled) which defaults to True.

Known Issues

  • Depending on when the initial Kilo deployment was done it is possible the repository servers have a pip.conf locking down the environment which limits the packages available to to install. If this file is present it will cause build failures as the repository server attempts to build Liberty packages.

  • Services deploy into virtual environments by default when the service relies on Python. On upgrade any Python packages installed on the host or container are not upgraded with the release unless the virtualenv for that service is disabled. There might be older and possibly broken packages left on the system outside of the virtualenv, which can cause confusion for those who attempt to use Python-based tools or services without using the virtualenv. These left over packages can be manually removed at the operator’s discretion.

Upgrade Notes

  • Existing LBaaS v1 load balancers and agents will not be altered by the new OpenStack-Ansible release.

  • When upgrading from early Kilo versions of OpenStack-Ansible, the RabbitMQ minor version may need to be upgraded during the upgrade process. This is noted in both the manual steps and the run-upgrade.sh script.

  • To fix this issue the pip.conf file needs to be removed from all repository servers. The upgrade playbook repo-server-pip-conf-removal.yml will remove the pip.conf file from the repository servers if it’s found.

12.0.9

Known Issues

  • For OpenStack-Ansible Liberty versions <12.0.9 and Kilo versions <11.2.12 the package pywbem will fail to build due to the update to v0.8.0 including new requirements which are not met by the repo server. This issue has been resolved in 12.0.9. A workaround for this is to set pywbem<0.8.0 in the file global-requirement-pins.txt.

  • For OpenStack-Ansible Liberty versions >12.0.7,<12.0.9 the wheel version pinned in OpenStack-Ansible (0.29.0) is higher that the OpenStack upper-constraint (0.26.0). This causes an issue where the repo-server install may fail because it cannot find a version of wheel to install that meets the requirements of <0.26.0 and ==0.29.0. A workaround for this issue is to change the wheel package pin in the following files wheel==0.26.0.

    • playbooks/inventory/group_vars/hosts.yml

    • requirements.txt

Deprecation Notes

  • The repo-clone-mirror.yml file is being deprecated, effective immediately. The playbook itself has been changed to no longer sync and will be removed from the tree in the Mitaka branch.

12.0.8

New Features

  • Ubuntu has 4 different ‘components’ - main, universe, multiverse and restricted:

    • Main: Officially supported software.

    • Restricted: Supported software that is not available under a completely free license.

    • Universe: Community maintained software, i.e. not officially supported software.

    • Multiverse: Software that is not free.

    The default apt sources configuration uses all components. If deployers wish to change this to reduce the components configured then the variable lxc_container_template_apt_components may be set in /etc/openstack_deploy/user_variables.yml with the full list of desired components.

  • A new variable called lxc_container_cache_files has been implemented which contains a list of dictionaries that specify files on the deployment host which should be copied into the LXC container cache and what attributes to assign to the copied file.

Known Issues

  • There is a bug in the version of keepalived which ships with Ubuntu 14.04 which results in all backup nodes having the same priority. This causes the automatic failover to fail when more than two keepalived hosts are configured. To work around this issue it is recommended that deployers limit the number of keepalived hosts to no more than two, or that each host is configured with different priorities.

  • Neutron currently does not support enabling the port_security extension driver cleanly for existing networks. If networks are created and the plugin is enabled afterwards, VMs connected to those networks will not start. See bug https://bugs.launchpad.net/neutron/+bug/1509312

Upgrade Notes

  • During the upgrade process new secrets, such as passwords and keys, will be generated and added to /etc/openstack_deploy/user_secrets.yml. Existing values will not be changed.

  • The signing_dir configuration option has changed from /var/cache/heat to /var/lib/heat/cache/heat. This only applies to heat deployments that use PKI tokens.

  • When upgrading from Kilo to Liberty, the port_security extension driver will not be configured due to the known issues with enabling it after creating networks.

  • Some variables names have been changed to reflect upstream design decisions (such as Nova’s default API version), or to provide clarity. These require updating in /etc/openstack_deploy/user_*.yml for any overrides to continue to work. See the upgrade documentation <http://docs.openstack.org/developer/openstack-ansible/liberty/upgrade-guide/scripts.html#migrate-openstack-vars-py> for details.

Deprecation Notes

  • The Nova 2.1 variables (nova_v21_<variable>), Heat name variables ( heat_project_domain_name, heat_user_domain) and Galera SST Method (galera_sst_method) variables have changed. See the upgrade documentation <http://docs.openstack.org/developer/openstack-ansible/liberty/upgrade-guide/scripts.html#migrate-openstack-vars-py> for details.

Bug Fixes

  • The addition of multi-domain LDAP configuration support left behind a configuration file for the default domain that causes problems with Keystone. This file will automatically be removed if the deployer is not using the Default domain with an LDAP back end. (Bug 1547542)

12.0.7

New Features

  • Keystone’s v3 API is now the default for all services.

  • MariaDB version 10.x is now the default in OpenStack-Ansible.

  • The percona-xtrabackup repository is now enabled in OpenStack-Ansible and it allows deployers to install and use Percona’s XtraBackup project to perform online backups of data stored in MariaDB.

  • Deployers how have the option to set the the wsrep method via the galera_wsrep_sst_method.

  • Deployers can specify the authentication credentials to be used with wsrep by configuring galera_wsrep_sst_auth_user and galera_wsrep_sst_auth_password.

  • The Galera installation process has been optimized and takes less time to complete.

  • Each service using RabbitMQ now has a separate vhost and user.

Upgrade Notes

  • The ceilometer alarming functionality has been moved into aodh. The ceilometer_alarm_notifier and ceilometer_alarm_evaluator entries are removed from the /etc/openstack_deploy/env.d/ceilometer.yml file.

  • aodh.yml and haproxy.yml will be copied into /etc/openstack_deploy/env.d. LBaaS agent information will be added to /etc/openstack_deploy/env.d/neutron.yml.

  • When Glance is configured to use a swift store backend, it will use Keystone v3 authentication by default via the glance_swift_store_auth_version variable.

  • Two new options were added for handling authentication with Swift storage backends - glance_swift_store_user_domain and glance_swift_store_project_domain. Both are set to default and can be adjusted if deployers use a different Keystone domain to authenticate to swift.

  • The Keystone configuration has been updated for liberty. Several variables that may appear in the user_config.yml file may need to be updated. Those variables include:

    • keystone_identity_driver

    • keystone_token_driver

    • keystone_token_provider

    • keystone_revocation_driver

    • keystone_assignment_driver

    • keystone_resource_driver

    • keystone_ldap_identity_driver

    Deployers should review the defaults provided in playbooks/os_keystone/defaults/main.yml and adjust any variables in user_variables.yml if they exist there.

  • Deployers can optionally remove the Keystone v2 endpoints from the database. Those endpoints will not be removed by the upgrade process.

  • The max connections setting for Galera is now determined automatically by taking the number of vCPUs available and multiplying it by 100. Deployers may override this default via the galera_max_connections variable.

  • The upstream MariaDB init script has replaced the custom init script that was provided by OpenStack-Ansible in previous versions.

  • The galera_upgrade variable is now provided to allow the MariaDB role to update existing installs.

  • The neutron_driver_network_scheduler variable default has changed from ChanceScheduler to WeightScheduler to match the new Neutron defaults.

  • The neutron_driver_quota variable default has changed slightly to match the new upstream driver paths.

  • The LinuxBridge configuration that was in plugins/ml2/ml2_conf.ini is now found in plugins/ml2/linuxbridge_agent.ini.

  • Two Neutron variables have been deprecated and are now removed from OpenStack-Ansible - neutron_l3_router_delete_namespaces and neutron_dhcp_delete_namespaces.

  • The Nova project has set the v2.1 API as the default and those configuration variables have changed. Variables that began with nova_v21_* in the Kilo release are now renamed to nova_*. All new Liberty deployments will have only the v2.1 API registered in the service catalog.

  • The S3, v3, and EC2 API’s have been deprecated by the Nova project in the liberty release. Those variables have been removed. They include variables that begin with nova_s3_*, nova_ec2_*, and nova_v3_*.

  • The variables beginning with openstack_host_systat_ in the openstack_hosts role have been renamed to openstack_host_sysstat_. This was done to better reflect their dependency to sysstat.

  • Each service using RabbitMQ now has a separate vhost and user. The shared / vhost is cleaned up so that it contains only the default data. The shared user ‘openstack’ is removed.

  • Nova now utilizes version 2 of the Cinder API. Tempest is now configured to use the v2 Cinder API as well.

  • The upgrade process will backup and re-configure the /etc/openstack_deploy directory. This includes inserting new environment details, updating changed variable names, and generating newly added secrets.

Security Issues

  • The glance_digest_algorithm has changed from sha1 to sha256 and this improves integrity verification of stored images.

Bug Fixes

  • The python packages pip, setuptools and wheel are now all pinned on a per-tag basis. The pins are updated along with every OpenStack Service update. This is done to ensure a consistent build experience with the latest available packages at the time the tag is released. A deployer may override the pins by adding a list of required pins using the pip_packages variable in user_variables.yml.

12.0.6

New Features

  • Keystone can now be configured for multiple LDAP or Active Directory identity back-ends. Configuration of this feature is documented in the Keystone Configuration section of the Install Guide.

Upgrade Notes

  • The first tier of the keystone_ldap dictionary variable now relates to the Keystone Domain name. An existing keystone_ldap configuration entry can be converted by renaming the ldap key to the domain name ‘Default’. Note that the domain name entry is case-sensitive.

  • The keystone_ldap_identity_driver variable has been removed. The driver for an LDAP back-end in Keystone now simply uses the value ‘ldap’. There are no other back-end options for Keystone at this time.