Scaling OpenStack Development with Git, Gerrit, and Jenkins

Monty Taylor <mordred@inaugust.com>
James E. Blair <corvus@inaugust.com>

Systems

https://jenkins.openstack.org
https://review.openstack.org
http://ci.openstack.org (docs)
https://jenkins.stackforge.org

Projects

nova (compute)
swift (object storge)
glance (image service)
keystone (identity service)
quantum (network service)
horizon (dashboard)
cinder (volume service)
python-novaclient
python-swiftclient
python-glanceclient
python-keystoneclient
python-quantumclient
python-cinderclient
python-openstackclient

Contributors

Individual Contributors
Commercial Entities
Number, quality, and area of contributions can change daily

Release Management

Time Based Releases
Six Month Cadence
- Tied to Ubuntu Releases
Design summits each cycle
Continuously Open Trunk
- Develop directly on master
One Month Milestone Releases
Post-release Stable Branches

Vision

Consistent Tooling
Consistent Process
Consistent Product
Multiplier Effect

Consistent Tooling

Minimize meta-development
Process divergence == wasted developer time
Lowers onboarding time
Consolidate tool development
Minimize project-specific weird build crud

Development Infrastructure Systems

Gerrit (code review, git) http://review.openstack.org
Jenkins (CI, automation) http://jenkins.openstack.org
- pre-approval testing
- pre-merge testing
- post-merge testing
- post-merge artifact management
Launchpad (bugs, blueprints, releases, translations, SSO) http://launchpad.net/openstack
Documentation servers
IRC bots
planet.openstack.org
Repository mirrors
Etherpad

Environment

Ubuntu
Python
- pep8 standards
- openstack.common
virtualenv/pip
IRC (#openstack-dev, #openstack-meeting)
devstack
gated trunk based on master

Gated Trunk

Ensures Code Quality
Protects developers
- Devs always start from working code
Protects tree
- Bad code doesn't land
Egalitarian
- Process is the same for everyone
- Process is transparent
- Process is automated

Everything Is Automated

Process Flow

Code is written and locally tested in a virtualenv
Code is submitted for code review to gerrit
Code is run through patch-uploaded automated checks
Code is peer-reviewed
Code is accepted or rejected by core team
Code is run through pre-merge automated checks
Code is merged or rejected
Code is run through post-merge automated checks

Gerrit

Developed by Google for Android
Stand-alone patch review system
Integration points: hooks, JSON queries, event-stream
Extensible review categories, default: Verified, Code-Review

Pre-merge Check

States of a Patch

Code Submitted
Code Verified
Code Reviewed
Code Accepted
Code Landed

Approved Reviews

Types of Jenkins Gerrit Triggers

Patchset uploaded
Change merged
Comment added (review state)

Git Review

External Git subcommand
Developers can easily incorporate code review into git workflow
Zero-configuration
Can be used for any project, being adopted by other projects

corvus@shiprock:~/rs/github/quantum$ git commit -a
[new-versionpy ddf1dce] Base version.py on glance.
 3 files changed, 28 insertions(+), 107 deletions(-)
 delete mode 100644 version.py

corvus@shiprock:~/rs/github/quantum$ git review
remote: Resolving deltas:   0% (0/3)
remote: 
remote: New Changes:
remote:   https://review.openstack.org/3072
remote: 
To ssh://corvus@review.openstack.org:29418/openstack/quantum.git
 * [new branch]      HEAD -> refs/for/master/bug/916018

Types of Tests

Unit tests
Integration tests
- May be able to run on virtual servers, should run on real servers
- Difficult or impossible for a developer to run

Specific Challenges/Solutions

Testing effect of merging change
Using cloud builders
Large numbers of similar projects
Disparate hardware configurations

gerrit git prep

test the result of the change, not the change
https://github.com/openstack/openstack-ci-puppet/blob/master/modules/jenkins_slave/files/slave_scripts/gerrit-git-prep.sh
For each gerrit change:
- grabs target branch
- cleans tree
- merges change to be tested

Interrelated integration testing

devstack-gate
- spin up fresh node
- run devstack
- run integration tests
Gate proposed change against current state of other projects

bottlenecking

Serialize branches across all projects
optimistic branch prediction
Run in parallel in order triggered
Assume success
Start over on failure

devstack-gate problems

Clouds suck
github sucks
pypi sucks
Ubuntu sucks
ALL NETWORK ACCESS IS FAILURE

devstack-gate solutions

Create a new node
pre-fetch all needed packages, repos
snapshot to cloud image
maintain a pool of cloud nodes
slave can only be used for one test run
set of python and shell scripts triggered by jenkins

jclouds-plugin

ec2 plugin useless to me (sorry)
Adrian Cole and Cloudsoft wrote jclouds-plugin
Provisions on any provider via jclouds
Image pre-creation and pooling features
single use slaves
Slave provisioning as build step

Templated jobs

TONS of consistent projects
Manage everything through git and code review
https://github.com/openstack/openstack-ci-puppet/tree/master/modules/jenkins_jobs
Manage jobs as yaml files in git
Jobs updated via Jenkins API
groupings of jobs supported "nova: python_jobs"

Simple example

project:
  template: 'python_jobs'

values:
  name: 'python-novaclient'
  site: 'openstack'
  disabled: 'false'

complex example

---
modules:
  - properties
  - scm
  - assignednode
  - trigger_gerrit
  - builders
  - publisher_none

main:
  name: 'gate-ceilometer-merge'
  site: 'stackforge'
  project: 'ceilometer'
  authenticatedBuild: 'true'
  disabled: 'false'

trigger:
  triggerOnPatchsetUploadedEvent: 'false'
  triggerOnChangeMergedEvent: 'false'
  triggerOnCommentAddedEvent: 'true'
  triggerOnRefUpdatedEvent: 'false'
  triggerApprovalCategory: 'APRV'
  triggerApprovalValue: 1
  failureMessage: 'This change was unable to be automatically merged with the current state of the repository. Please rebase your change and upload a new patchset.'
  projects:
    - projectCompareType: 'PLAIN'
      projectPattern: 'stackforge/ceilometer'
      branchCompareType: 'ANT'
      branchPattern: '**'

builders:
  - gerrit_git_prep

scm:
  scm: 'false'

assignednode:
  node: 'oneiric'

Scaling hardware needs

Different vendors have different combinations of configurations
http://ci.openstack.org/third_party.html#the-jenkins-gerrit-trigger-plugin-way
Run their own jenkins
Consume gerrit event stream
their jenkins votes VRFY +1/-1
Our jenkins votes VRFY +2/-2
The can run their lab - I don't have to know anything about it

Thanks!

These slides available at: https://github.com/openstack-ci/publications