6.5.3. OpenStack control plane performance report for 400 nodes

Abstract:This document includes OpenStack control plane performance test results for 379 nodes in fact (several more nodes were not able to be added to the cluster due to deployment issues). All tests have been performed regarding OpenStack control plane performance test plan

6.5.3.1. Environment description

Environment contains 4 types of servers:

  • rally node
  • controller node
  • compute-osd node
  • compute node
Amount of servers each role
Role Servers count Type
rally 1 1 or 2
controller 3 1 or 2
compute 291 1 or 2
compute-osd 34 3
compute-osd 49 1

6.5.3.1.1. Hardware configuration of each server

All servers have 3 types of configuration describing in table below

Description of servers hardware type 1
server vendor,model Dell PowerEdge R630
CPU vendor,model Intel,E5-2680 v3
processor_count 2
core_count 12
frequency_MHz 2500
RAM vendor,model Samsung, M393A2G40DB0-CPB
amount_MB 262144
NETWORK interface_name s eno1, eno2
vendor,model Intel,X710 Dual Port
bandwidth 10G
interface_names enp3s0f0, enp3s0f1
vendor,model Intel,X710 Dual Port
bandwidth 10G
STORAGE dev_name /dev/sda
vendor,model
raid1 - Dell, PERC H730P Mini
2 disks Intel S3610
SSD/HDD SSD
size 3,6TB

6.5.3.1.2. Network configuration of each server

All servers have same network configuration:

Network Scheme of the environment

6.5.3.1.3. Software configuration on servers with controller, compute and compute-osd roles

Services on servers by role
Role Service name
controller
horizon
keystone
nova-api
nava-scheduler
nova-cert
nova-conductor
nova-consoleauth
nova-consoleproxy
cinder-api
cinder-backup
cinder-scheduler
cinder-volume
glance-api
glance-glare
glance-registry
neutron-dhcp-agent
neutron-l3-agent
neutron-metadata-agent
neutron-openvswitch-agent
neutron-server
heat-api
heat-api-cfn
heat-api-cloudwatch
ceph-mon
rados-gw
memcached
rabbitmq_server
mysqld
galera
corosync
pacemaker
haproxy
compute
nova-compute
neutron-l3-agent
neutron-metadata-agent
neutron-openvswitch-agent
compute-osd
nova-compute
neutron-l3-agent
neutron-metadata-agent
neutron-openvswitch-agent
ceph-osd
Software version on servers with controller, compute and compute-osd roles
Software Version
OpenStack Mitaka
Ceph Hammer
Ubuntu Ubuntu 14.04.3 LTS

You can find outputs of some commands and /etc folder in the following archives:

controller-1.tar.gz
controller-2.tar.gz
controller-3.tar.gz
compute-1.tar.gz
compute-osd-1.tar.gz

6.5.3.1.4. Software configuration on servers with rally role

On this server should be installed Rally. How to do it you can find in Rally installation documentation

Software version on server with rally role
Software Version
Rally 0.5.0
Ubuntu Ubuntu 14.04.3 LTS

6.5.3.2. Testing process

Some test parameters
Name Value
Volume size to create in Cinder 1GB
Flavor to create VM from m1.tiny
Image name to create VM from cirros

  1. Create work directory on server with Rally role. In future we will call that directory as WORK_DIR

  2. Create directory “plugins” in WORK_DIR and copy to that directory nova_performance.py plugin.

  3. Create directory “scenarios” in WORK_DIR and copy to that directory boot_attach_live_migrate_and_delete_server_with_secgroups.json, create-and-delete-image.json and keystone.json scenarios.

  4. Perform tests:

    rally  --plugin-paths /home/rally/plugins task start \
keystone.json \
--task-args '{"gre_enabled": true, "compute": 375, "concurrency": 5}' \
--out=keystone.html

rally  --plugin-paths /home/rally/plugins task start \
boot_attach_live_migrate_and_delete_server_with_secgroups.json \
--task-args '{"gre_enabled": true, "compute": 375, "concurrency": 5}' \
--out=boot_attach_live_migrate_and_delete_server_with_secgroups.html

rally  --plugin-paths /home/rally/plugins task start \
create-and-delete-image.json \
--task-args '{"gre_enabled": true, "compute": 375, "concurrency": 5, "http_server_with_glance_images": "1.2.3.4"}' \
--out=create-and-delete-image.html

As a result of this part we got the following HTML file:

keystone.html
boot_attach_live_migrate_and_delete_server_with_secgroups.html
create-and-delete-image.html

6.5.3.3. Test results

All values in tables below are in seconds.

6.5.3.3.1. Cinder

Operation Mean (sec) 90%ile (sec) 50%ile (sec) Max (sec) Min (sec)
create_volume 2.619 2.754 2.593 2.988 2.453
delete_volume 2.339 2.449 2.323 2.666 2.200

6.5.3.3.2. Glance

Iterations: 400 Failures: 17

Operation Median (sec) 90%ile (sec) 95%ile (sec) Max (sec) Min (sec)
create_image 60.573 88.944 103.642 125.492 40.852
delete_image 6.428 9.804 12.221 28.287 1.072

6.5.3.3.3. Keystone

Operation Median (sec) 90%ile (sec) 95%ile (sec) Max (sec) Min (sec)
authenticate 0.195 0.22 0.231 0.988 0.086

6.5.3.3.4. Neutron

Operation Median (sec) 90%ile (sec) 95%ile (sec) Max (sec) Min (sec)
create_20_rules 5.957 6.347 6.467 8.227 5.202
create_2_security_groups 0.546 0.639 0.664 0.887 0.41
delete_2_security_groups 0.472 0.59 0.624 0.887 0.297

6.5.3.3.5. Nova

Operation Median (sec) 90%ile (sec) 95%ile (sec) Max (sec) Min (sec)
attach_volume 3.063 3.256 3.304 5.349 2.726
boot_server 12.648 13.132 13.328 15.054 11.071
delete_server 2.622 2.862 3 5.078 2.446
detach_volume 2.901 3.067 3.13 3.319 2.629
find_host_to_migrate 0.807 1.021 1.069 1.385 0.655
live_migrate 16.317 18.477 18.689 31.738 14.371