Jenkins is a Continuous Integration system that runs tests and automates some parts of project operations. It is controlled for the most part by Zuul which determines what jobs are run when.
A large number and variety of jobs are defined in Jenkins. The configuration of all of those jobs is stored in git in the openstack-infra/project-config repository. They are defined in YAML files that are read by Jenkins Job Builder which configures the actual jobs in Jenkins.
Anyone may submit a change to the openstack-infra/project-config repository that defines a new job or alters an existing job by editing the appropriate YAML files. See Jenkins Job Builder for more information.
Because of the large number of builds that Jenkins executes, the OpenStack project favors the following approach in configuring Jenkins jobs:
- Minimal use of plugins: the more post-processing work that Jenkins needs to perform on a job, the more likely we are to run into compatibility problems among plugins, and contention for shared resources on the Jenkins master. A number of popular plugins will cause all builds of a job to be serialized even if the jobs otherwise run in parallel.
- Minimal build history: Jenkins stores build history in individual XML files on disk, and accessing a large build history can cause the Jenkins master to be unresponsive for a significant time while loading them. It also increases memory usage. Instead, we generally keep no more than a day’s worth of builds.
- Move artifacts off of Jenkins: Jenkins is not efficient at serving static information such as build artifacts (e.g., tarballs) or logs. Instead, we copy them to a static webserver which is far more efficient.
OpenStack integration testing is performed by the devstack gate test framework. This framework runs the devstack exercises and Tempest smoketests against a devstack install on single use cloud servers. The devstack gate source can be found on git.openstack.org and the Readme describes the process of using devstack gate to run your own devstack based tests.
For management of the devstack and other instances, a tool called Nodepool creates and deletes Jenkins slaves as needed in order to maintain the pool.
Jenkins is largely hidden, and has no sensitive data exposed publicly, so we use self-signed certs for Jenkins masters.
After bringing up a jenkins node (16G memory instance if you use the stock jenkins.default) with puppet, log in and configure Jenkins by hand:
Puppet takes care of the rest.
Note that jenkins talks to its slaves via ssh, the modules/openstack_project/manifests/init.pp file contains the ssh public key that puppet installs on the slaves.
Statically provisioned slaves have labels assigned by hand. E.g. centos6, and are added to a chosen Jenkins master by hand. Adding a slave is then:
Jenkins masters periodically leak threads reducing their job throughput and eventually leading to crashes. We work around this by performing weekly rolling restarts of the Jenkins masters with an ansible playbook.
If you need to perform a safe restart against a single master you can do this by running the same playbook and limiting it to a specific jenkins master
To do this:
root@puppetmaster# ansible-playbook -f1 --limit $server_fqdn \ /opt/system-config/production/playbooks/restart_jenkins_masters.yaml \ --extra-vars "user=hudson-openstack \ password=$(/opt/system-config/production/tools/hieraedit.py \ --yaml /etc/puppet/hieradata/production/fqdn/nodepool.openstack.org.yaml jenkins_api_key)"
Consider running this in screen as the worst case run time is as long as our longest running job.
Jenkins job builder may need to be run manually under certain situations. If the expected jobs are not being created in jenkins masters, running jjb manually on the masters where it failed is suggested. To do this:
user@jenkins01# sudo -H jenkins-jobs --conf /etc/jenkins_jobs/jenkins_jobs.ini \ update --delete-old /etc/jenkins_jobs/config
Consider running this in screen as the worst case run time can be of several hours.
In the case of incorrect jobs configuration caused by some jjb malfunction, all jobs will need to be regenerated. As jjb uses a local cache, to force the regeneration of all jobs, the cache needs to be ignored. To do this:
user@jenkins01# sudo -H jenkins-jobs --ignore-cache --conf \ /etc/jenkins_jobs/jenkins_jobs.ini update --delete-old /etc/jenkins_jobs/config
In order to speed up the massive job reconfiguration, it may be desired to set jenkins on shutdown mode, visiting this link:
And make Jenkins alive again after job reconfiguration finishes.