Role - tripleo_container_manage

Role Documentation

Welcome to the “tripleo_container_manage” role documentation.

Role Defaults

This section highlights all of the defaults and variables set within the “tripleo_container_manage” role.

# All variables intended for modification should place placed in this file.
tripleo_container_manage_hide_sensitive_logs: '{{ hide_sensitive_logs | default(true)
  }}'
tripleo_container_manage_debug: '{{ ((ansible_verbosity | int) >= 2) | bool }}'
tripleo_container_manage_clean_orphans: true

# All variables within this role should have a prefix of "tripleo_container_manage"
tripleo_container_manage_check_puppet_config: false
tripleo_container_manage_cli: podman
tripleo_container_manage_concurrency: 1
tripleo_container_manage_config: /var/lib/tripleo-config/
tripleo_container_manage_config_id: tripleo
tripleo_container_manage_config_overrides: {}
tripleo_container_manage_config_patterns: '*.json'
# Some containers where Puppet is run, can take up to 10 minutes to finish
# in slow environments.
tripleo_container_manage_create_retries: 120
# Default delay is 5s so 120 retries makes a timeout of 10 minutes which is
# what we have observed a necessary value for nova and neutron db-sync execs.
tripleo_container_manage_exec_retries: 120
tripleo_container_manage_healthcheck_disabled: false
tripleo_container_manage_log_path: /var/log/containers/stdouts
tripleo_container_manage_systemd_teardown: true

Molecule Scenarios

Molecule is being used to test the “tripleo_container_manage” role. The following section highlights the drivers in service and provides an example playbook showing how the role is leveraged.

Scenario: default

Molecule Inventory

hosts:
  all:
    hosts:
      instance:
        ansible_host: localhost
        ansible_connection: local
        ansible_distribution: centos9

Example default playbook

- name: Create fedora container from /tmp/container-configs with old healthcheck
  become: true
  hosts: all
  gather_facts: false
  vars:
    tripleo_container_manage_config: /tmp/container-configs
    tripleo_container_manage_healthcheck_disabled: true
    tripleo_container_manage_debug: true
    tripleo_container_manage_config_patterns: fedora.json
  tasks:
  - include_role:
      name: tripleo_container_manage
  post_tasks:
  - name: Verify that Fedora container was created correctly and manually create old
      healthcheck for migration testing
    when:
    - not ansible_check_mode|bool
    block:
        # Reproduce what was done before to create and enable healthchecks
    - name: Enable and start systemd timers
      systemd:
        state: started
        name: tripleo_fedora_healthcheck.timer
        enabled: true
        daemon_reload: false
    - name: Add systemd requires for healthchecks
      command: systemctl add-requires tripleo_fedora.service tripleo_fedora_healthcheck.timer
        # Check that migration is ready to be tested
    - name: Check for fedora container
      command: podman container exists fedora
    - name: Check if tripleo_fedora systemd healthcheck service is active
      command: systemctl is-active --quiet tripleo_fedora_healthcheck.timer
      register: tripleo_fedora_healthcheck_active_result
    - name: Assert that tripleo_fedora systemd healthcheck service is active
      assert:
        that:
        - tripleo_fedora_healthcheck_active_result.rc == 0
        fail_msg: tripleo_fedora systemd healthcheck service is not active
        success_msg: tripleo_fedora systemd healthcheck service is active

- name: Create all containers from /tmp/container-configs
  become: true
  hosts: all
  gather_facts: false
  vars:
    tripleo_container_manage_config: /tmp/container-configs
    tripleo_container_manage_debug: true
    tripleo_container_manage_config_patterns: '*.json'
  tasks:
  - include_role:
      name: tripleo_container_manage
  post_tasks:
  - name: Verify that Fedora container was created correctly
    when:
    - not ansible_check_mode|bool
    block:
    - name: Check for fedora container
      command: podman container exists fedora
    - name: Gather facts about fedora container
      containers.podman.podman_container_info:
        name: fedora
      register: fedora_infos
    - name: Assert that fedora container has the right image
      assert:
        that:
        - "'fedora:latest' in fedora_infos.containers.0.ImageName"
        fail_msg: fedora container has wrong image
        success_msg: fedora container has the right image
    - name: Check if tripleo_fedora systemd service is active
      command: systemctl is-active --quiet tripleo_fedora
      register: tripleo_fedora_active_result
    - name: Assert that tripleo_fedora systemd service is active
      assert:
        that:
        - tripleo_fedora_active_result.rc == 0
        fail_msg: tripleo_fedora systemd service is not active
        success_msg: tripleo_fedora systemd service is active
    - name: Check if tripleo_fedora healthcheck is active and healthy
      assert:
        that:
        - "'healthy' in fedora_infos.containers.0.State.Health.Status"
        fail_msg: fedora container healthcheck is not healthy
        success_msg: fedora container healthcheck is healthy
    - name: Verify that Fedora systemd healthcheck container was removed correctly
      command: systemctl is-active --quiet tripleo_fedora_healthcheck.timer
      register: tripleo_fedora_healthcheck_active_result
      failed_when:
      - tripleo_fedora_healthcheck_active_result.rc == 0
  - name: Verify that Fedora bis container was created correctly
    block:
    - name: Check for fedora_bis container
      command: podman container exists fedora_bis
    - name: Gather facts about fedora_bis container
      containers.podman.podman_container_info:
        name: fedora_bis
      register: fedora_bis_infos
    - name: Assert that fedora_bis container has the right image
      assert:
        that:
        - "'fedora:latest' in fedora_bis_infos.containers.0.ImageName"
        fail_msg: fedora_bis container has wrong image
        success_msg: fedora_bis container has the right image
  - name: Verify that Fedora three container was created correctly
    block:
    - name: Check for fedora_three container
      command: podman container exists fedora_three
    - name: Gather facts about fedora_three container
      containers.podman.podman_container_info:
        name: fedora_three
      register: fedora_three_infos
    - name: Assert that fedora_three container has the right image
      assert:
        that:
        - "'fedora:latest' in fedora_three_infos.containers.0.ImageName"
        fail_msg: fedora_three container has wrong image
        success_msg: fedora_three container has the right image

- name: Test idempotency on fedora container
  become: true
  hosts: all
  gather_facts: false
  vars:
    tripleo_container_manage_config: /tmp/container-configs
    tripleo_container_manage_debug: true
    tripleo_container_manage_config_patterns: '*.json'
  tasks:
  - name: Gather facts about fedora container before new run
    containers.podman.podman_container_info:
      name: fedora
    register: fedora_infos_old
    when:
    - not ansible_check_mode|bool
  - include_role:
      name: tripleo_container_manage
  - name: Gather facts about fedora container after new run
    containers.podman.podman_container_info:
      name: fedora
    register: fedora_infos_new
    when:
    - not ansible_check_mode|bool
  post_tasks:
  - name: Assert that fedora container has not been re-created
    assert:
      that:
      - fedora_infos_new['containers'][0]['Id'] == fedora_infos_old['containers'][0]['Id']
      fail_msg: fedora container was wrongly re-created
      success_msg: fedora container was not re-created
    when:
    - not ansible_check_mode|bool

- name: Test systemd state on fedora container after a manual stop
  become: true
  hosts: all
  gather_facts: false
  vars:
    tripleo_container_manage_config: /tmp/container-configs
    tripleo_container_manage_debug: true
    tripleo_container_manage_config_patterns: '*.json'
  tasks:
  - name: Stop systemd service for tripleo_fedora in a manual stop
    systemd:
      name: tripleo_fedora.service
      state: stopped
      enabled: false
      daemon_reload: true
      # https://github.com/ansible/ansible/pull/68136
    ignore_errors: '{{ ansible_check_mode }}'
  - include_role:
      name: tripleo_container_manage
  post_tasks:
  - name: Check if tripleo_fedora systemd service is active after a manual stop
    command: systemctl is-active --quiet tripleo_fedora
    register: tripleo_fedora_active_result
  - name: Assert that tripleo_fedora systemd service is active after a manual stop
    when:
    - not ansible_check_mode|bool
    assert:
      that:
      - tripleo_fedora_active_result.rc == 0
      fail_msg: tripleo_fedora systemd service is not active after a manual stop
      success_msg: tripleo_fedora systemd service is active after a manual stop

- name: Manage only one container
  become: true
  hosts: all
  gather_facts: false
  vars:
    tripleo_container_manage_config: /tmp/container-configs
    tripleo_container_manage_debug: true
    tripleo_container_manage_config_patterns: fedora.json
    tripleo_container_manage_clean_orphans: false
    tripleo_container_manage_config_overrides:
      fedora:
        image: fedora:rawhide
  tasks:
  - include_role:
      name: tripleo_container_manage
  post_tasks:
  - name: Verify that all containers still exist
    when:
    - not ansible_check_mode|bool
    block:
    - name: Check for fedora container
      command: podman container exists fedora
    - name: Gather facts about fedora container
      containers.podman.podman_container_info:
        name: fedora
      register: fedora_infos
    - name: Assert that fedora container has the right image
      assert:
        that:
        - "'fedora:rawhide' in fedora_infos.containers.0.ImageName"
        fail_msg: fedora container has wrong image {{ fedora_infos.containers }}
        success_msg: fedora container has the right image
    - name: Check if tripleo_fedora systemd service is active
      command: systemctl is-active --quiet tripleo_fedora
      register: tripleo_fedora_active_result
    - name: Assert that tripleo_fedora systemd service is active
      assert:
        that:
        - tripleo_fedora_active_result.rc == 0
        fail_msg: tripleo_fedora systemd service is not active
        success_msg: tripleo_fedora systemd service is active
    - name: Check if tripleo_fedora healthcheck is active and healthy
      assert:
        that:
        - "'healthy' in fedora_infos.containers.0.State.Health.Status"
        fail_msg: fedora container healthcheck is not healthy
        success_msg: fedora container healthcheck is healthy
    - name: Check for fedora_bis container
      command: podman container exists fedora_bis
    - name: Check for fedora_three container
      command: podman container exists fedora_three

- name: Manage a wrong container (user error)
  become: true
  hosts: all
  gather_facts: false
  vars:
    tripleo_container_manage_config: /tmp/container-configs
    tripleo_container_manage_debug: true
    tripleo_container_manage_config_patterns: feduraaa.json
    tripleo_container_manage_clean_orphans: false
  tasks:
  - include_role:
      name: tripleo_container_manage
  post_tasks:
  - name: Verify that all containers still exist
    when:
    - not ansible_check_mode|bool
    block:
    - name: Check for fedora container
      command: podman container exists fedora
    - name: Gather facts about fedora container
      containers.podman.podman_container_info:
        name: fedora
      register: fedora_infos
    - name: Check if tripleo_fedora systemd service is active
      command: systemctl is-active --quiet tripleo_fedora
      register: tripleo_fedora_active_result
    - name: Assert that tripleo_fedora systemd service is active
      assert:
        that:
        - tripleo_fedora_active_result.rc == 0
        fail_msg: tripleo_fedora systemd service is not active
        success_msg: tripleo_fedora systemd service is active
    - name: Check if tripleo_fedora healthcheck is active and healthy
      assert:
        that:
        - "'healthy' in fedora_infos.containers.0.State.Health.Status"
        fail_msg: fedora container healthcheck is not healthy
        success_msg: fedora container healthcheck is healthy
    - name: Check for fedora_bis container
      command: podman container exists fedora_bis
    - name: Check for fedora_three container
      command: podman container exists fedora_three

- name: Test a container removal
  become: true
  hosts: all
  gather_facts: false
  vars:
    tripleo_container_manage_config: /tmp/container-configs
    tripleo_container_manage_debug: true
    tripleo_container_manage_config_patterns: fedora_*.json
  tasks:
  - name: Remove fedora container config
    file:
      path: /tmp/container-configs/fedora.json
      state: absent
  - include_role:
      name: tripleo_container_manage
  post_tasks:
  - name: Verify that all containers still exist
    when:
    - not ansible_check_mode|bool
    block:
    - name: Check that fedora container was removed
      command: podman container exists fedora
      register: container_exist
      failed_when: container_exist.rc == 0
    - name: Check if tripleo_fedora systemd service is still active
      command: systemctl is-active --quiet tripleo_fedora
      register: tripleo_fedora_active_result
      failed_when: tripleo_fedora_active_result.rc == 0
    - name: Check for fedora_bis container
      command: podman container exists fedora_bis
    - name: Check for fedora_three container
      command: podman container exists fedora_three

- name: Test a container update
  become: true
  hosts: all
  gather_facts: false
  vars:
    tripleo_container_manage_config: /tmp/container-configs
    tripleo_container_manage_debug: true
    tripleo_container_manage_config_patterns: fedora_*.json
  tasks:
  - name: Modify the fedora_bis container config
    copy:
      content: |
        {
          "image": "fedora:rawhide",
          "net": "host",
          "command": "sleep 10"
        }
      dest: /tmp/container-configs/fedora_bis.json
  - include_role:
      name: tripleo_container_manage
  post_tasks:
  - name: Verify that Fedora bis container was re-created correctly
    when:
    - not ansible_check_mode|bool
    block:
    - name: Check for fedora_bis container
      command: podman container exists fedora_bis
    - name: Gather facts about fedora_bis container
      containers.podman.podman_container_info:
        name: fedora_bis
      register: fedora_bis_infos
    - name: Assert that fedora_bis container has the right image
      assert:
        that:
        - "'fedora:rawhide' in fedora_bis_infos.containers.0.ImageName"
        fail_msg: fedora_bis container has wrong image
        success_msg: fedora_bis container has the right image
  - name: Check for fedora_three container
    command: podman container exists fedora_three
    when:
    - not ansible_check_mode|bool

- name: Test a container config override
  become: true
  hosts: all
  gather_facts: false
  vars:
    tripleo_container_manage_config: /tmp/container-configs
    tripleo_container_manage_debug: true
    tripleo_container_manage_config_patterns: fedora_*.json
    tripleo_container_manage_config_overrides:
      fedora_bis:
        image: fedora:latest
  tasks:
  - include_role:
      name: tripleo_container_manage
  post_tasks:
  - name: Verify that Fedora bis container was re-created correctly
    when:
    - not ansible_check_mode|bool
    block:
    - name: Check for fedora_bis container
      command: podman container exists fedora_bis
    - name: Gather facts about fedora_bis container
      containers.podman.podman_container_info:
        name: fedora_bis
      register: fedora_bis_infos
    - name: Assert that fedora_bis container has the right image
      assert:
        that:
        - "'fedora:latest' in fedora_bis_infos.containers.0.ImageName"
        fail_msg: fedora_bis container has wrong image
        success_msg: fedora_bis container has the right image
  - name: Check for fedora_three container
    command: podman container exists fedora_three
    when:
    - not ansible_check_mode|bool

Usage

Note that right now, only Podman is supported by this role. Docker support is in the roadmap though.

This Ansible role allows to do the following tasks:

• Collect container configs data, generated by TripleO Heat Templates. This data is used as a source of truth on which configuration we expect to apply with this role. It means that if a container is already managed by this role, no matter its state now, the configs data will reconfigure the container if needed.

• Manage systemd shutdown files. It takes care of cleaning up the Paunch services and files and create the TripleO Container systemd service, required for service ordering when it comes to shutdown or start a node. It also manages the netns-placeholder service.

• Delete containers that aren’t needed anymore or that will need to be re-configured. It uses a custom filter, named needs_delete() which has a set of rules which allow to determine if whether or not the container needs to be deleted. These reasons will make the containers not deleted:

  • The container is not managed by tripleo_ansible.

  • The container config_id doesn’t match with the one in input.

  Once the previous conditions checked, then these reasons will make the containers deleted:

  • The container has no config_data.

  • The container has a config_data which doesn’t match the one in input.

  Note that when a container is removed, the role also disable and remove the systemd services and healtchecks if present.

• Create containers in a specific order defined by start_order container config, where default is 0.

  • If the container is an exec, we’ll run a dedicated playbook for execs, using async so multiple execs can be run at the same time.

  • Otherwise, the podman_container is used, in async, to create the containers. If the container has a restart policy, we’ll configure the systemd service. If the container has a healthcheck script, we’ll configure the systemd healthcheck service.

  Note: tripleo_container_manage_concurrency parameter is set to 1 by default, and putting higher value than 2 can be expose issue with Podman locks.

  Here is an example of a playbook:

- name: Manage step_1 containers using tripleo-ansible
  block:
    - name: "Manage containers for step 1 with tripleo-ansible"
      include_role:
        name: tripleo_container_manage
      vars:
        tripleo_container_manage_config: "/var/lib/tripleo-config/container-startup-config/step_1"
        tripleo_container_manage_config_id: "tripleo_step1"

Roles variables

Name	Default Value	Description
tripleo_container_manage_cli	podman	Container CLI
tripleo_container_manage_concurrency	1	Number of containers managed at same time
tripleo_container_manage_config	/var/lib/tripleo-config/	Container config path
tripleo_container_manage_config_id	tripleo	Config ID
tripleo_container_manage_config_patterns	*.json	Bash REGEX to find configs
tripleo_container_manage_debug	false	Debug toggle
tripleo_container_manage_healthcheck_disable	false	Allow to disable Healthchecks
tripleo_container_manage_log_path	/var/log/containers/stdouts	Containers stdouts path
tripleo_container_manage_config_overrides	{}	Allows to override any container configuration
tripleo_container_manage_clean_orphans	true	Option to clean orphans

Healthchecks

Previously, the container healthcheck was implemented by a systemd timer which would run podman exec to determine if a given container was healthy.. Now, we are using the native healthcheck interface in Podman; which is easier to integrate and consume.

We are now using the native healthcheck interface in Podman; which is easier to integrate with and consume.

To check if a container (e.g. keystone) is healthy, run the following command:

$ sudo podman healthcheck run keystone

The return code should be 0 and “healthy” should be printed as the output. One can also use the podman inspect keystone output to figure out that the healthcheck is periodically running and healthy:

"Health": {
    "Status": "healthy",
    "FailingStreak": 0,
    "Log": [
        {
            "Start": "2020-04-14T18:48:57.272180578Z",
            "End": "2020-04-14T18:48:57.806659104Z",
            "ExitCode": 0,
            "Output": ""
        },
        (...)
    ]
}

Debug

The role allows you to perform specific actions on a given container. This can be used to:

• Run a container with a specific one-off configuration.

• Output the container commands that are run to to manage containers lifecycle.

• Output the changes that would have been made on containers by Ansible.

Note

To manage a single container, you need to know 2 things:

• At which step the container is deployed.

• The name of the generated JSON file for container config.

Here is an example of a playbook to manage HAproxy container at step 1 which overrides the image setting in one-off.

- hosts: localhost
  become: true
  tasks:
    - name: Manage step_1 containers using tripleo-ansible
      block:
        - name: "Manage HAproxy container at step 1 with tripleo-ansible"
          include_role:
            name: tripleo_container_manage
          vars:
            tripleo_container_manage_config_patterns: 'haproxy.json'
            tripleo_container_manage_config: "/var/lib/tripleo-config/container-startup-config/step_1"
            tripleo_container_manage_config_id: "tripleo_step1"
            tripleo_container_manage_clean_orphans: false
            tripleo_container_manage_config_overrides:
              haproxy:
                image: quay.io/tripleomastercentos9/centos-binary-haproxy:hotfix

If Ansible is run in check mode, no container will be removed nor created, however at the end of the playbook a list of commands will be displayed to show what would have been run. This is useful for debug purposes, as it was something that one could do with paunch debug command.

$ ansible-playbook haproxy.yaml --check

Adding the diff mode will output the changes what would have been made on containers by Ansible.

$ ansible-playbook haproxy.yaml --check --diff

The tripleo_container_manage_clean_orphans parameter is optional and can be set to false to not clean orphaned containers for a config_id. It can be used to manage a single container without impacting other running containers with same config_id.

The tripleo_container_manage_config_overrides parameter is optional and can be used to override a specific container attribute like the image or the container user. The parameter takes a dictionary where each key is the container name and its parameters that we want to override. These parameters have to exist and are the ones that define the container configuration in TripleO Heat Templates. Note that it doesn’t write down the overrides in the JSON file so if an update / upgrade is executed, the container will be re-configured with the configuration that is in the JSON file.