OpenStack CI

• Run by OpenStack Infrastructure Team
• OpenStack Infrastructure Team is an open team
• OpenStack Infrastructure Team would love you to join
• All repos are at: https://github.com/openstack-infra

Scale

• Over 500 Active Technical Contributors (and growing)
• As many as 200 events an hour (and growing)
• 18 (integarted) projects (and growing)

Automation

• Automate Away all non-creative tasks
• Humans get repeatable processes wrong
• We don't have enough humans

Following from that

• Consistency
• Repeatability
• Availability

Consistency

• To automate 18 projects, they need to act the same
• Inteface for devs for 18 projects needs to be the same

Repeatability

• Everything must be able to run 500x a day without failing
• Everything must be able to run 500x a day without failing
• Everything must be able to run 500x a day without failing

Availability

• Once everything is automated, the projects stops if the automation does
• Access to manage needs to be egalitarian

Gated Trunk

• Ensures Code Quality
• Protects developers
  • Devs always start from working code
• Protects tree
  • Bad code doesn't land
• Egalitarian
  • Process is the same for everyone
  • Process is transparent
  • Process is automated

Gating Implemenation

• Test everything we can on every commit before it lands
• Test change when they are proposed
• Peer Code review ALL changes
• Test the resulting state of the tree before it becomes the resulting state

Test Everything

• unittests
• style tests
• integration tests

Gating Impl

• Gerrit
  • Repos
  • Code Review
  • Event Stream
• Zuul
  • Takes actions based on gerrit events
• Jenkins
  • Runs tests
• devstack-gate
  • Prepped pool of slaves

Gerrit

• Developed by Google for Android
• Stand-alone patch review system
• Integration points: hooks, JSON queries, event-stream
• Extensible review categories, default: Verified, Code-Review

devstack-gate

• Clouds can't create machines consistently enough
  • Make a pool of test machines
• Internet breaks often
  • Prefetch all network needs of devstack
• configure, run, and test result of devstack
  • Install every service in openstack using devstack
  • Test result cloud using tempest
  • Change to a project is tested against tip of all others
• Devstack does evil things to machines
  • Throw away machines after each test run

System Management

• Puppet
• Jenkins Job Builder
• jeepyb

jenkins-job-builder

• Availability of jenkins config to everyone
• Set of yaml files describing our jenkins jobs
• Macro-based for composition
• yaml files are in gerrit

projects.yaml

• Availability of project config to everyone
• Drives config of repos in gerrit and github and zuul
• Drives pypi mirror
• Source of info for pull request closer

pypi mirror

• Availability of pip commands
• Common set of requirements
• PyPI isn't available enough
• pip download reqs for all projects in projects.yaml
• turn download cache into static pypi mirror

Docmentation

• build sphinx docs on each commit
• upload to openstack.org
• optionally trigger rtfd
• why not gate on it?

Translations

• apparently some people don't speak english
• transifex for translations management
• On each commit to trunk, upload latest strings to transifex
• Periodicly, fetch translated strings from transifex and propose change to gerrit

Tag Based Releasing

• Triggered by git tag
• If tag exists, that's the version
• If tag doesn't, version is either:
  • $latesttag.$revcount.g$gitsha
  • $versionarg.a$revcount.g$gitsha
• On tag:
  • Upload sdist tarball to tarballs.o.o
  • If release tag (no letters) upload to pypi too

Things we're working on

• Jenkins Gearman Plugin
• Upstream branch tracking
• Logstash for devstack jobs

Thanks!

All repos are at: https://github.com/openstack-infra

These slides available at: https://openstack-ci.github.com/publications