OpenStack

Is open source software for building private and public clouds.

Projects

Contributors

• Unaffiliated individuals
• Commercial entities
• Nonprofit organizations
• National and local governments
• Number, quality, and area of contributions can change daily

Consistent Tooling

• Minimize meta-development
• Process divergence == wasted developer time
• Lowers onboarding time
• Consolidate tool development
• Minimize project-specific weird build crud

Developer Infrastructure

Development Environment

• Python
  • CentOS (2.6), Ubuntu LTS (2.7, 3.3, pypy)
  • PEP-8 standards
  • virtualenv/pip/tox
• Freenode IRC (#openstack-dev, #openstack-meeting)
• DevStack
• Tests run on all newly submitted changes
• Code merges are gated on tests

Project Gating

• Ensures Code Quality
• Protects developers
  • Devs always start from working code
• Protects tree
  • Bad code doesn't land
• Egalitarian
  • Process is the same for everyone
  • Process is transparent
  • Process is automated

Everything Is Automated

Process Flow

Gerrit

• Developed by Google for Android
• Stand-alone patch review system
• Integration points: hooks, JSON queries, event-stream
• Extensible review categories, default: Verified, Code-Review

Gerrit

Git Review

• External Git subcommand
• Developers can easily incorporate code review into git workflow
• Zero-configuration
• Can be used for any project, being adopted by other projects

corvus@shiprock:~/git/neutron$ git commit -a
[new-versionpy ddf1dce] Base version.py on glance.
 3 files changed, 28 insertions(+), 107 deletions(-)
 delete mode 100644 version.py

corvus@shiprock:~/git/neutron$ git review
remote: Resolving deltas:   0% (0/3)
remote: 
remote: New Changes:
remote:   https://review.openstack.org/3072
remote: 
To ssh://corvus@review.openstack.org:29418/openstack/neutron.git
 * [new branch]      HEAD -> refs/for/master/bug/916018

Interrelated Integration Testing

• devstack-gate
  • boot fresh servers
  • run DevStack
  • run integration tests
• Gate proposed change against current state of other projects

Devstack-Gate Problems

• Tests are slow
• Cloud API calls can fail
• External services are unreliable

Devstack-Gate Solutions

• Create a new node
• Pre-fetch all needed packages, repos
• Snapshot to cloud image
• Maintain a pool of cloud nodes
• Slave can only be used for one test run
• Python and shell scripts run as jobs
• Partition tests and run them in parallel

Zuul

• A general purpose trunk gating system
• Interfaces with Gerrit and Jenkins
• Flexible configuration allows for many kinds of project automation
• Allows parallel testing of serialized changes

Bottlenecking

• Serialize changes across all projects
• Speculative execution of tests
• Run in parallel in order triggered
• Assume success
• Start over on failure

Zuul Simulation

StackForge

• Organization of non-official repos in the OpenStack Infra Systems
• Self-service
• Access to all of the advanced dev/test and CI features of OpenStack

Adding your project

• It's a git commit!

# git clone git://git.openstack.org/openstack-infra/config
# cd config
# git review -s

Four main files

• modules/openstack_project/templates/review.projects.yaml.erb
• modules/openstack_project/files/gerrit/acls/stackforge/$project.config
• modules/openstack_project/files/jenkins_job_builder/config/projects.yaml
• modules/openstack_project/files/zuul/layout.yaml

review.projects.yaml.erb

• Projects registry

  - project: stackforge/wsme
    description: Web Service Made Easy (WSME)
    upstream: git://github.com/ryanpetrello/wsme.git
  

• project: stackforge/$project
• description: short description of project
• upstream: from where do we do initial clone - only used at the beginning

stackforge/$project.config

  [access "refs/heads/*"]
        label-Code-Review = -2..+2 group wsme-core
        label-Approved = +0..+1 group wsme-core
        workInProgress = group wsme-core
  [access "refs/tags/*"]
        create = group wsme-ptl
        pushTag = group wsme-ptl
  [project]
        state = active
  [receive]
        requireChangeId = true
        requireContributorAgreement = false
  [submit]
        mergeContent = true
  

• Groups are created by being referenced
• Put the user name of the initial group member in the commit message
• Core group - who can approve code
• PTL group - who can push release tags

config/projects.yaml<

• What jobs exist for your project

    - project:
        name: stevedore
        github-org: stackforge
        node: precise
        rtfd-id: 4157
        tarball-site: tarballs.openstack.org

        jobs:
          - python-jobs
          - pypi-jobs
          - hook-{name}-rtfd
  

layout.yaml

• When to run what job

  - name: stackforge/stevedore
    check:
      - gate-stevedore-pep8
      - gate-stevedore-python26
      - gate-stevedore-python27
      - gate-stevedore-python33
    gate:
      - gate-stevedore-pep8
      - gate-stevedore-python26
      - gate-stevedore-python27
      - gate-stevedore-python33
    post:
      - stevedore-branch-tarball
    pre-release:
      - stevedore-tarball
      - hook-stevedore-rtfd
    release:
      - stevedore-tarball:
        - stevedore-pypi-upload:
          - post-mirror-python26
          - post-mirror-python27
          - post-mirror-python33
      - hook-stevedore-rtfd
  

Pipelines

• check: On Upload
• gate: On Approval
• post: On Landing
• pre-release: On Tag with letters (PEP440)
• release: On Tag with only numbers and dots

Commit and submit

# git commit -a -m"Added $project to StackForge"
# git review
  

• Wait for infra-core to approve
• Approval is for syntax/sanity
• Core team is busy sometimes

Tag Based Releases

• Release artifacts are created and uploaded based on signed git tag
• Must be signed - otherwise it's ignored

# git tag -s 1.2.0
# git push gerrit 1.2.0

• In-repo versioning support in PBR
• http://docs.openstack.org/developer/pbr/semver.htm

devstack integration

• modules/openstack_project/files/jenkins_job_builder/config/

- job-template:
    name: '{pipeline}-requirements-integration-dsvm'
    node: '{node}'

    wrappers:
      - timeout:
          timeout: 120
          fail: true
      - timestamps

    builders:
      - devstack-checkout
      - shell: |
          #!/bin/bash -xe
          export PYTHONUNBUFFERED=true
          export DEVSTACK_GATE_TEMPEST=0
          export DEVSTACK_GATE_EXERCISES=0
          export DEVSTACK_GATE_REQS_INTEGRATION=1
          function gate_hook {{
              bash $GATE_SCRIPT_DIR/devstack-vm-gate.sh && \
              bash -xe $BASE/new/pbr/tools/integration.sh $PROJECTS
          }}
          export -f gate_hook
          cp devstack-gate/devstack-vm-gate-wrap.sh ./safe-devstack-vm-gate-wrap.sh
          ./safe-devstack-vm-gate-wrap.sh
      - link-logs

    publishers:
      - devstack-logs
      - console-log

Not just python

- project:
    name: gearman-plugin
    maven-group-id: org.jenkins-ci.plugins
    github-org: openstack-infra
    node: precise
    tarball-site: tarballs.openstack.org

    jobs:
      - jenkins-plugin-jobs

Extra Special Needs

• Different vendors have different combinations of configurations
• http://ci.openstack.org/third_party.html
• Run their own Jenkins
• Consume Gerrit event stream
• Their Jenkins votes VRFY +1/-1
• Our Zuul votes VRFY +2/-2
• You can run their lab - we don't have to know anything about it

Thanks!

These slides available at: http://docs.openstack.org/infra/publications/