Rally v0.1.0


Commits 355
Bug fixes 90
Dev cycle 132 days
Release date 25/September/2015


This release contains new features, new 42 plugins, 90 bug fixes, various code and API improvements.

New Features & API changes

  • Improved installation script

    • Add parameters:
      • --develop parameter to install rally in editable (develop) mode
      • --no-color to switch off output colorizing useful for automated output parsing and terminals that don't support colors.
    • Puts rally.conf under virtualenv etc/rally/ so you can have several rally installations in virtualenv
    • Many fixes related to access of different file, like: rally.conf, rally db file in case of sqlite
    • Update pip before Rally installation
    • Fix reinstallation
  • Separated Rally plugins & framework

    Now plugins are here: https://github.com/openstack/rally/tree/master/rally/plugins

    Plugins are as well separated common/* for common plugins that can be use no matter what is tested and OpenStack related plugins

  • New Rally Task framework

    • All plugins has the same Plugin base: rally.common.plugin.pluing.Plugin They have the same mechanisms for: discovering, providing information based on docstrings, and in future they will use the same deprecation/rename mechanism.

    • Some of files are moved:

      • rally/benchmark -> rally/task

        This was done to unify naming of rally task command and actually code that implements it.

      • rally/benchmark/sla/base.py -> rally/task/sla.py

      • rally/benchmark/context/base.py -> rally/task/context.py

      • rally/benchmark/scenarios/base.py -> rally/task/scenario.py

      • rally/benchmark/runners/base.py -> rally/task/runner.py

      • rally/benchmark/scenarios/utils.py -> rally/task/utils.py

      This was done to:

      • avoid doing rally.benchmark.scenarios import base as scenario_base
      • remove one level of nesting
      • simplify framework structure
    • Some of classes and methods were renamed

      • Plugin configuration:

        • context.context() -> context.configure()
        • scenario.scenario() -> scenario.configure()
        • Introduced runner.configure()
        • Introduced sla.configure()

        This resolves 3 problems:

        • Unifies configuration of different types of plugins

        • Simplifies plugin interface

        • Looks nice with new modules path:
          >>> from rally.task import scenario
          >>> @scenario.configure()
    • Atomic Actions were changed:

      • New rally.task.atomic module

        This allow us in future to reuse atomic actions in Context plugins

      • Renames:

        rally.benchmark.scenarios.base.AtomicAction -> rally.task.atomic.ActionTimer

        rally.benchmark.scenarios.base.atomic_action() -> rally.task.atomic.action_timer()

    • Context plugins decide how to map their data for scenario

      Now Context.map_for_scenario method can be override to decide how to pass context object to each iteration of scenario.

    • Samples of NEW vs OLD context, sla, scenario and runner plugins:

      • Context
      # Old
      from rally.benchmark.context import base
      @base.context(name="users", order=100)
      class YourContext(base.Context):
          def setup(self):
              # ...
          def cleanup(self):
              # ...
      # New
      from rally.task import context
      @context.configure(name="users", order=100)
      class YourContext(context.Context):
          def setup(self):
              # ...
          def cleanup(self):
              # ...
          def map_for_scenario(self):
              # Maps context object to the scenario context object
              # like context["users"] -> context["user"] and so on.
      • Scenario
      # Old Scenario
      from rally.benchmark.scenarios import base
      from rally.benchmark import validation
      class ScenarioPlugin(base.Scenario):
          def some(self):
          def _do_some_action(self):
              # ...
      # New Scenario
      from rally.task import atomic
      from rally.task import scenario
      from rally.task import validation
      # OpenStack scenario has different base now:
      # rally.plugins.openstack.scenario.OpenStackScenario
      class ScenarioPlugin(scenario.Scenario):
          def some(self):
          def _do_some_action(self):
              # ...
      • Runner
      ## Old
      from rally.benchmark.runners import base
      class SomeRunner(base.ScenarioRunner):
           __execution_type__ = "some_runner"
          def _run_scenario(self, cls, method_name, context, args)
            # Load generation
          def abort(self):
            # Method that aborts load generation
      ## New
      from rally.task import runner
      class SomeRunner(runner.ScenarioRunner):
          def _run_scenario(self, cls, method_name, context, args)
              # Load generation
          def abort(self):
              # Method that aborts load generation
      • SLA
      # Old
      from rally.benchmark import sla
      class FailureRate(sla.SLA):
          # ...
      # New
      from rally.task import sla
      class FailureRate(sla.SLA):
          # ...
  • Rally Task aborted command

    Finally you can gracefully shutdown running task by calling:

    rally task abort <task_uuid>
  • Rally CLI changes

    • [add] rally --plugin-paths specify the list of directories with plugins
    • [add] rally task report --junit - generate a JUnit report This allows users to feed reports to tools such as Jenkins.
    • [add] rally task abort - aborts running Rally task when run with the --soft key, the rally task abort command is waiting until the currently running subtask is finished, otherwise the command interrupts subtask immediately after current scenario iterations are finished.
    • [add] rally plugin show prints detailed information about plugin
    • [add] rally plugin list prints table with rally plugin names and titles
    • [add] rally verify genconfig generates tempest.conf without running it.
    • [add] rally verify install install tempest for specified deployment
    • [add] rally verify reinstall removes tempest for specified deployment
    • [add] rally verify uninstall uninstall tempest of specified deployment
    • [fix] rally verify start --no-use --no-use was always turned on
    • [remove] rally use now each command has subcommand use
    • [remove] rally info
    • [remove] rally-manage tempest now it is covered by rally verify
  • New Rally task reports

    • New code is based on OOP style which is base step to make pluggable Reports
    • Reports are now generated for only one iteration over the resulting data which resolves scalability issues when we are working with large amount of iterations.
    • New Load profiler plot that shows amount of iterations that are working in parallel
    • Failed iterations are shown as a red areas on stacked are graphic.

Non backward compatible changes

  • [remove] rally use cli command
  • [remove] rally info cli command
  • [remove] --uuid parameter from rally deployment <any>
  • [remove --deploy-id parameter from: rally task <any>, rally verify <any>, rally show <any>

Specs & Feature requests

[feature request] Explicitly specify existing users for scenarios

[feature request] Improve install script and add --uninstall and --version

[feature request] Allows specific repos & packages in install-rally.sh

[feature request] Add ability to capture logs from tested services

[feature request] Check RPC queue perfdata

[spec] Refactoring Rally cleanup

[spec] Consistent resource names


  • Scenarios:

    [new] CinderVolumes.create_volume_backup

    [new] CinderVolumes.create_and_restore_volume_backup

    [new] KeystoneBasic.add_and_remove_user_role

    [new] KeystoneBasic.create_and_delete_role

    [new] KeystoneBasic.create_add_and_list_user_roles

    [new] FuelEnvironments.list_environments

    [new] CinderVolumes.modify_volume_metadata

    [new] NovaServers.boot_and_delete_multiple_servers

    [new] NeutronLoadbalancerV1.create_and_list_pool

    [new] ManilaShares.list_shares

    [new] CeilometerEvents.create_user_and_get_event

    [new] CeilometerEvents.create_user_and_list_event_types

    [new] CeilometerEvents.create_user_and_list_events

    [new] CeilometerTraits.create_user_and_list_trait_descriptions

    [new] CeilometerTraits.create_user_and_list_traits

    [new] NeutronLoadbalancerV1.create_and_delete_pools

    [new] NeutronLoadbalancerV1.create_and_update_pools

    [new] ManilaShares.create_and_delete_share

    [new] ManilaShares.create_share_network_and_delete

    [new] ManilaShares.create_share_network_and_list

    [new] HeatStacks.create_and_delete_stack

    [new] ManilaShares.list_share_servers

    [new] HeatStacks.create_snapshot_restore_delete_stack

    [new] KeystoneBasic.create_and_delete_ec2credential

    [new] KeystoneBasic.create_and_list_ec2credentials

    [new] HeatStacks.create_stack_and_scale

    [new] ManilaShares.create_security_service_and_delete

    [new] KeystoneBasic.create_user_set_enabled_and_delete

    [new] ManilaShares.attach_security_service_to_share_network

    [new] IronicNodes.create_and_delete_node

    [new] IronicNodes.create_and_list_node

    [new] CinderVolumes.create_and_list_volume_backups

    [new] NovaNetworks.create_and_list_networks

    [new] NovaNetworks.create_and_delete_network

    [new] EC2Servers.list_servers

    [new] VMTasks.boot_runcommand_delete_custom_imagea

    [new] CinderVolumes.create_and_update_volume

  • Contexts:

    [new] ManilaQuotas

    Add context for setting up Manila quotas: shares, gigabytes, snapshots, snapshot_gigabytes, share_networks

    [new] ManilaShareNetworks

    Context for share networks that will be used in case of usage deployment with existing users. Provided share networks via context option "share_networks" will be balanced between all share creations of scenarios.

    [new] Lbaas

    Context to create LBaaS-v1 resources

    [new] ImageCommandCustomizerContext

    Allows image customization using side effects of a command execution. E.g. one can install an application to the image and use these image for 'boot_runcommand_delete' scenario afterwards.

    [new] EC2ServerGenerator

    Context that creates servers using EC2 api

    [new] ExistingNetwork

    This context lets you use existing networks that have already been created instead of creating new networks with Rally. This is useful when, for instance, you are using Neutron with a dumb router that is not capable of creating new networks on the fly.

  • SLA:

    [remove] max_failure_rate - use failure_rate instead

Bug fixes

90 bugs were fixed, the most critical are:

  • Many fixes related that fixes access of rally.conf and DB files
  • Incorrect apt-get "-yes" parameter in install_rally.sh script
  • Rally bash completion doesn't exist in a virtualenv
  • Rally show networks CLI command worked only with nova networks
  • RPS runner was not properly generating load
  • Check is dhcp_agent_scheduler support or not in network cleanup
  • NetworkContext doesn't work with Nova V2.1
  • Rally task input file was not able to use jinja2 include directive
  • Rally in docker image was not able to
  • Rally docker image didn't contain samples
  • Do not update the average duration when iteration failed


  • Add plugin reference page

    Rally Plugins Reference page page contains a full list with

  • Add maintainers section on project info page

    Rally Maintainers section contains information about core contributors of OpenStack Rally their responsibilities and contacts. This will help us to make our community more transparent and open for newbies.

  • Added who is using section in docs

  • Many small fixes