4x performance increase in Keystone inside Apache using the token creation benchmark

(Contributed by Neependra Khare, Red Hat)

Below we describe how we were able to get and verify a 4x better performance of Keystone inside Apache. To do that, we ran a Keystone token creation benchmark with Rally under different load (this benchmark scenario essentially just authenticate users with keystone to get tokens).

Goal

• Get the data about performance of token creation under different load.

• Ensure that keystone with increased public_workers/admin_workers values and under Apache works better than the default setup.

Summary

• As the concurrency increases, time to authenticate the user gets up.

• Keystone is CPU bound process and by default only one thread of keystone-all process get started. We can increase the parallelism by:

  1. increasing public_workers/admin_workers values in keystone.conf file

  2. running Keystone inside Apache

• We configured Keystone with 4 public_workers and ran Keystone inside Apache. In both cases we got up to 4x better performance as compared to default Keystone configuration.

Setup

Server : Dell PowerEdge R610

CPU make and model : Intel(R) Xeon(R) CPU X5650 @ 2.67GHz

CPU count: 24

RAM : 48 GB

Devstack - Commit#d65f7a2858fb047b20470e8fa62ddaede2787a85

Keystone - Commit#455d50e8ae360c2a7598a61d87d9d341e5d9d3ed

Keystone API - 2

To increase public_workers - Uncomment line with public_workers and set public_workers to 4. Then restart Keystone service.

To run Keystone inside Apache - Added APACHE_ENABLED_SERVICES=key in localrc file while setting up OpenStack environment with Devstack.

Results

1. Concurrency = 4

{
    "contexts": {
        "users": {
            "concurrent": 30,
            "tenants": 12,
            "users_per_tenant": 512
        }
    },
    "runner": {
        "constant": {
            "concurrency": 4,
            "times": 10000
        }
    }
}

action	min (sec)	avg (sec)	max (sec)	90 percentile	95 percentile	success	count	apache enabled keystone	public_workers
total	0.537	0.998	4.553	1.233	1.391	100.0%	10000	N	1
total	0.189	0.296	5.099	0.417	0.474	100.0%	10000	N	4
total	0.208	0.299	3.228	0.437	0.485	100.0%	10000	Y	NA

2. Concurrency = 16

{
    "contexts": {
        "users": {
            "concurrent": 30,
            "tenants": 12,
            "users_per_tenant": 512
        }
    },
    "runner": {
        "constant": {
            "concurrency": 16,
            "times": 10000
        }
    }
}

action	min (sec)	avg (sec)	max (sec)	90 percentile	95 percentile	success	count	apache enabled keystone	public_workers
total	1.036	3.905	11.254	5.258	5.700	100.0%	10000	N	1
total	0.187	1.012	5.894	1.61	1.856	100.0%	10000	N	4
total	0.515	0.970	2.076	1.113	1.192	100.0%	10000	Y	NA

3. Concurrency = 32

{
    "contexts": {
        "users": {
            "concurrent": 30,
            "tenants": 12,
            "users_per_tenant": 512
        }
    },
    "runner": {
        "constant": {
            "concurrency": 32,
            "times": 10000
        }
    }
}

action	min (sec)	avg (sec)	max (sec)	90 percentile	95 percentile	success	count	apache enabled keystone	public_workers
total	1.493	7.752	16.007	10.428	11.183	100.0%	10000	N	1
total	0.198	1.967	8.54	3.223	3.701	100.0%	10000	N	4
total	1.115	1.986	6.224	2.133	2.244	100.0%	10000	Y	NA