Reproduce CI jobs for debugging and development

Reproducing any given CI job that is executed upstream is critical for development. You will find a script called “reproducer-zuul-quickstart.sh” under the logs directory of any TripleO CI job that will assist you in recreating a TripleO Quickstart upstream job.

Each script contains the same parameters used in the job and includes any deployment or CI related settings. Any patches that were marked as dependencies will also be included in the script and execution.

This script will provision a heat stack in an OpenStack host cloud that matches the environment created by nodepool or OVB upstream. Once the environment is ready, the script prompts the user with some additional instructions that executes the same scripts and tools used upstream to bootstrap and execute the CI.

Install system dependencies

Before execute the reproduce-quickstart it’s necessary to install few dependencies on the system:

* python3-openstackclient
* python3-heatclient

If you use Python 2 change python3 by python2 in the package name.

How to execute the reproducer-quickstart script

Go to the “logs” directory of the job:

wget http://logs.openstack.org/34/653934/1/<snip>/05075b2/logs/reproducer-quickstart/reproducer-zuul-based-quickstart.tar
tar -xvf reproducer-zuul-based-quickstart.tar

Be sure you have download the OpenStack RC file. To have more information you can check the documentation here.

Source your OpenStack RC file:

source openstack_rc.sh

Execute the reproducer-quickstart.sh:

bash -x reproducer-zuul-based-quickstart.sh

Please check the script’s help command however the available options are:

Options:
-w, --workspace <dir>
                    directory where the virtualenv, inventory files, etc.
                    are created. Defaults to creating a directory in /tmp
-l, --libvirt
                    Runs a 2-node multinode job or singlenode job on a
                    single virthost using libvirt to create the nodes.
                    If a singlenode reproducer is run, two VMs will still be created.
-c, --cloud-name
                    Host cloud, specified in the clouds.yaml, to target
                    Defaults to rdo-cloud
-cp, --clouds-yaml-path
                    Full path to the clouds.yaml file
                    Defaults to /home/$USER/.config/openstack/clouds.yaml
-ok, --ovb-key-name
                    Name of the key to use in the host tenant for OVB deployments
                    Defaults to tripleo-ci-team
-f, --force-post-failure
                    Force job to fail so that the nodes will be held.
                    Temporary solution
-ug, --upstream-gerrit-user
                    Set the upstream gerrit user required to clone repos.
                    Defaults to the current $USER
-rg, --rdo-gerrit-user
                    Set the upstream RDO user required to clone RDO-related repos.
                    Defaults to the current $USER
-k, --ssh-key
                   private ssh key used to set up an access nodes.
                   Defaults to id_rsa.
-kp, --ssh-key-public
                   public ssh key used to set up an access nodes.
                   Defaults to $USER_SSH_KEY.pub
-skp, --ssh-key-path
                   Path to directory where user ssh keys are stored.
                   Defaults to /home/$USER/.ssh
-ugk, --upstream-gerrit-key
                   Set the upstream gerrit private key.
                   Defaults to the $USER_SSH_KEY.
-rgk, --rdo-gerrit-key
                   Set the upstream RDO user key.
                   Defaults to the $USER_SSH_KEY.
-e, --extra-params
                   File or/and parameters used to override default
                   parameters for playbooks. Multiple files
                   can be passed [-e @file1.yml -e @file2.yml ...]
                   and arguments [-e var=value -e var2=value2 ...]
-h, --help         print this help and exit

How does this script work

The script is generated by a jinja2 template that reads all the variables passed to the CI job and builds a custom script for each job. There are two major steps to be familiar with in this script. The first part of the script will provision the servers, networks and other infrastructure for the TripleO deployment. The second part invokes the CI tools and scripts in the same way the original CI job was executed.

Notes

If the environment was provisioned successfully you will be presented with further instructions that will prompt you to ssh to the environment and execute the upstream CI scripts. You may elect to run this automatically with the “autorun” option found in the help.

Support

The TripleO CI team will provide community support for the reproducer-quickstart.sh script. The tool is only supported in use with an OpenStack host cloud. Your results with various OpenStack providers may vary. Please open any issues or problems in launchpad with the “quickstart” tag.

The devmode.sh script is deprecated. The reproducer-zuul-based-quickstart.sh should be used for upstream development and debugging of TripleO CI.