5.15. Message Queue High Availability¶
| status: | ready |
|---|---|
| version: | 1.0 |
| Abstract: | This document describes a test plan for analysing high availability of OpenStack message bus. The measurement covers message queue and oslo.messaging library. |
5.15.1. Test Plan¶
5.15.1.1. Test Environment¶
RabbitMQ is installed on 3 nodes in HA mode. Active monitoring is implemented with help of pacemaker. The test tool is executed on another host.
5.15.1.1.1. Preparation¶
Setup RabbitMQ cluster, for example by using the setup made by OpenStack Fuel.
5.15.1.1.2. Environment description¶
The environment description includes hardware specification of servers, network parameters, operation system and OpenStack deployment characteristics.
5.15.1.1.2.1. Hardware¶
This section contains list of all types of hardware nodes.
| Parameter | Value | Comments |
| model | e.g. Supermicro X9SRD-F | |
| CPU | e.g. 6 x Intel(R) Xeon(R) CPU E5-2620 v2 @ 2.10GHz |
5.15.1.1.2.2. Network¶
This section contains list of interfaces and network parameters. For complicated cases this section may include topology diagram and switch parameters.
| Parameter | Value | Comments |
| card model | e.g. Intel | |
| driver | e.g. ixgbe | |
| speed | e.g. 10G or 1G |
5.15.1.1.2.3. Software¶
This section describes installed software.
| Parameter | Value | Comments |
| OS | e.g. Ubuntu 14.04.3 | |
| oslo.messaging | e.g. 4.0.0 | |
| MQ Server | e.g. RabbitMQ 3.5.6 | |
| HA mode | e.g. Cluster |
5.15.1.2. Test Case 1: Client and Server connected to Master, Master fails¶
5.15.1.2.1. Description¶
In this test case both client and server are connected to RabbitMQ master node. The throughput is measured and at the same time RabbitMQ master process is terminated.
5.15.1.2.2. List of performance metrics¶
Test case result is time series showing message flow between client and server. It can be shown as chart and/or table. The average throughput and number of errors are calculated.
| Priority | Value | Measurement Units | Description |
|---|---|---|---|
| 1 | Throughput | msg/sec | Number of messages per second |
| 2 | Latency | ms | The latency in message processing |
5.15.1.2.3. Options¶
- The test case is executed for different types of communication:
5.15.1.3. Test Case 2: Client and Server connected to Slave 1, Slave 1 fails¶
5.15.1.3.1. Description¶
In this test case both client and server are connected to the same RabbitMQ slave node. The throughput is measured and at the same time RabbitMQ slave process is terminated.
5.15.1.3.2. List of performance metrics¶
Test case result is time series showing message flow between client and server. It can be shown as chart and/or table. The average throughput and number of errors are calculated.
| Priority | Value | Measurement Units | Description |
|---|---|---|---|
| 1 | Throughput | msg/sec | Number of messages per second |
| 2 | Latency | ms | The latency in message processing |
5.15.1.3.3. Options¶
- The test case is executed for different types of communication:
5.15.1.4. Test Case 3: Client and Server on different slaves. Client Slave fails¶
5.15.1.4.1. Description¶
In this test case client and server are connected to different RabbitMQ slave node. The throughput is measured and at the same time RabbitMQ slave process is terminated.
5.15.1.4.2. List of performance metrics¶
Test case result is time series showing message flow between client and server. It can be shown as chart and/or table. The average throughput and number of errors are calculated.
| Priority | Value | Measurement Units | Description |
|---|---|---|---|
| 1 | Throughput | msg/sec | Number of messages per second |
| 2 | Latency | ms | The latency in message processing |
5.15.1.4.3. Options¶
- The test case is executed for different types of communication:
5.15.1.5. Test Case 4: Client on Master and Server on Slave, Master fails¶
5.15.1.5.1. Description¶
In this test case client and server are connected to different RabbitMQ slave node: the client to master and server to slave. The throughput is measured and at the same time RabbitMQ slave process is terminated.
5.15.1.5.2. List of performance metrics¶
Test case result is time series showing message flow between client and server. It can be shown as chart and/or table. The average throughput and number of errors are calculated.
| Priority | Value | Measurement Units | Description |
|---|---|---|---|
| 1 | Throughput | msg/sec | Number of messages per second |
| 2 | Latency | ms | The latency in message processing |
5.15.1.5.3. Options¶
- The test case is executed for different types of communication:
5.15.2. Tools¶
This section contains tools that can be used to perform the test plan.
5.15.2.1. Performa omsimulator¶
This section describes how to perform Message Queue High Availability with Performa tool.
5.15.2.1.1. Test environment preparation¶
To perform the test plan you will need to install performa toolkit. The toolkit uses MongoDB for stats processing and reporting.
$ pip install performa
- The execution requires the following parameters to be set:
mongo-address- The host where MongoDB is installedremote-user- The user that can connect to remote host in OpenStack cloudtester-hosts- List of hosts were omsimulator will be executedrabbit-url- RabbitMQ address, it has the form of rabbit://{<user>:<password>@<host>:<port>[,]}/report- folder where to store the report
5.15.2.1.2. Execution¶
RPC CALL measurements:
$ performa --mongo-url <mongo-address> --mongo-db performa
--scenario mq/omsimulator-ha-call --remote-user <remote-user>
--vars "{tester_hosts: [<tester-hosts>], rabbit_url: \"<rabbit-url>\"}"
--book <report>
- example:
$ performa --mongo-url 172.20.9.20 --mongo-db performa --scenario mq/omsimulator-ha-call --remote-user root --vars "{rabbit_hosts: [node-123, node-111, node-58], tester_hosts: [node-144], rabbit_url: \"rabbit://nova:tGAPNtjHh8yvvkR69MooN1eD@node-58:5673,nova:tGAPNtjHh8yvvkR69MooN1eD@node-111:5673,nova:tGAPNtjHh8yvvkR69MooN1eD@node-123:5673/\"}" --book books/omsimulator-ha-call-cmsm-km
RPC CAST measurements:
$ performa --mongo-url <mongo-address> --mongo-db performa
--scenario mq/omsimulator-ha-cast --remote-user <remote-user>
--vars "{tester_hosts: [<tester-hosts>], rabbit_url: \"<rabbit-url>\"}"
--book <report>
- example:
$ performa --mongo-url 172.20.9.20 --mongo-db performa --scenario mq/omsimulator-ha-cast --remote-user root --vars "{rabbit_hosts: [node-123, node-111, node-58], tester_hosts: [node-144], rabbit_url: \"rabbit://nova:tGAPNtjHh8yvvkR69MooN1eD@node-58:5673,nova:tGAPNtjHh8yvvkR69MooN1eD@node-111:5673,nova:tGAPNtjHh8yvvkR69MooN1eD@node-123:5673/\"}" --book books/omsimulator-ha-cast-cmsm-km
NOTIFY measurements:
$ performa --mongo-url <mongo-address> --mongo-db performa
--scenario mq/omsimulator-ha-notify --remote-user <remote-user>
--vars "{tester_hosts: [<tester-hosts>], rabbit_url: \"<rabbit-url>\"}"
--book <report>
- example:
$ performa --mongo-url 172.20.9.20 --mongo-db performa --scenario mq/omsimulator-ha-notify --remote-user root --vars "{rabbit_hosts: [node-123, node-111, node-58], tester_hosts: [node-144], rabbit_url: \"rabbit://nova:tGAPNtjHh8yvvkR69MooN1eD@node-58:5673,nova:tGAPNtjHh8yvvkR69MooN1eD@node-111:5673,nova:tGAPNtjHh8yvvkR69MooN1eD@node-123:5673/\"}" --book books/omsimulator-ha-notify-cmsm-km
5.15.3. Reports¶
- Test plan execution reports: