4. Results of the ETCD health tests

Abstract:This piece of art includes the results of the ETCD tests made basing on the ETCD health research test plan. Our goal was to research how many Kubernetes items (pods and services) could be spawned in terms of ETCD. We figured out which ETCD metrics are crucial and collected them under appropriate (pods or services) load. Corresponding dependency plots were made based on this data.

4.1. Environment description

4.1.1. Hardware configuration of each server

Description of servers hardware

server	name	node-{1..430}	node-{1..430}
	role	kubernetes cluster	kubernetes cluster
	vendor,model	Dell, R630	Lenovo, RD550-1U
	operating_system	4.4.0-36-generic Ubuntu-xenial x86_64	4.4.0-36-generic Ubuntu-xenial x86_64
CPU	vendor,model	Intel, E5-2680v3	Intel, E5-2680 v3
	processor_count	2	2
	core_count	12	12
	frequency_MHz	2500	2500
RAM	vendor,model	Hynix HMA42GR7MFR4N-TF	Samsung M393A2G40DB0-CPB
	amount_MB	262144	262144
NETWORK	interface_name	bond0	bond0
	vendor,model	Intel, X710 Dual Port	Intel, X710 Dual Port
	interfaces_count	2	2
	bandwidth	10G	10G
STORAGE	dev_name	/dev/sda	/dev/sda
	vendor,model	raid1 PERC H730P Mini 2 disks Intel S3610	raid1 MegaRAID 3108 2 disks Intel S3610
	SSD/HDD	SSD	SSD
	size	800GB	800GB

4.1.2. Network scheme and part of configuration of hardware network switches

Network scheme of the environment:

Here is the piece of switch configuration for each switch port which is a part of bond0 interface of a server:

show run int et1 interface Ethernet1

description - r02r13c33 switchport trunk native vlan 4 switchport trunk allowed vlan 4 switchport mode trunk channel-group 133 mode active lacp port-priority 16384 spanning-tree portfast

show run int po1 interface Port-Channel1

description osscr02r13c21 switchport trunk native vlan 131 switchport trunk allowed vlan 130-159 switchport mode trunk port-channel lacp fallback static port-channel lacp fallback timeout 30 mlag 1

4.1.3. Software configuration

4.1.3.1. Setting up Kubernetes cluster:

Kubernetes installation was made with Kargo and Fuel-CCP-Installer wrapper on top of the 430 nodes with preinstalled Ubuntu Xenial. You can download the content of the /etc directory of a typical K8S node here: etc_tarball

This is how Kargo sets up Kubernetes cluster:

• nodes[1-3]: ETCD nodes
• nodes[4-5]: K8S masters
• nodes[6-430]: K8S nodes (minions)

Kargo was lanched with the following parameters:

cluster_id: "1.0.0.1"
docker_options: "--insecure-registry 10.3.60.54:5000 -D"
#upstream_dns_servers: [10.3.60.52]
upstream_dns_servers: [8.8.8.8, 8.8.4.4]
nameservers: [8.8.8.8, 8.8.4.4]
kube_service_addresses: 10.224.0.0/12
kube_pods_subnet: 10.240.0.0/12
kube_network_node_prefix: 22
kube_apiserver_insecure_bind_address: "0.0.0.0"
dns_replicas: 3
dns_cpu_limit: "100m"
dns_memory_limit: "512Mi"
dns_cpu_requests: "70m"
dns_memory_requests: "70Mi"
deploy_netchecker: false
download_run_once: false
download_localhost: false

See the versions of the used software below:

Software	Version
Ubuntu	Ubuntu 16.04.1 LTS
Fuel-CCP-Installer	6b26170f70e523fb04bda8d6f15077d461fba9de
Kargo	016b7893c64fede07269c01cac31e96c8ee0d257
Kubernetes	v1.5.3
ETCD	v3.1.0 (API version 2)

4.1.3.2. Test tools

We did not use any specific tools to collect required metrics during ETCD load tests. Prometheus monitoring with Telegraf agent and Grafana visualization were utilized for these purposes as a part of our Monitoring_methodology for containerized systems. The only well-known tools that we brought into play were:

• curl: to fetch metrics for specific range of time out of Prometheus
• jq: to parse resulted json objects and strip off extra data
• paste: to combine obtained results into csv file for further analysis

Load tests were conducted with the following benchmarking tools for Kubernetes:

• MMM: suit to spawn desired number of pods on each node of the cluster
• Hoseproxy: tool that takes one pod (e.g. nginx) and spawns desired number of service endpoints for this pod

4.2. Testing process

In the first scenario with MMM we had been using “200 pods per each node” setting and, thus, spawned huge amounts of pods. Snap from the prepare-minion.py of the MMM suit:

# pods per node:
PODS = 200
def main():
# nodes range:
nodes = ["node{}".format(x+1) for x in xrange(10, 430)]

In the second scenario with Hoseproxy we launched lots of services (100 services per step in 100 parallel threads):

./hoseproxy -ns minions -src master -master http://localhost:8080 -nparallel 100 -nservices 100 -nsteps 10000

For both scenarios we were following next algorithm:

1. Launch a load test and wait until ETCD performance starts to significantly degrade or Kubernetes will not be able to spawn any more of pods or services (for the certain case).

2. Collect data series for each required ETCD metric and K8S pods/services counters within time range of the test. Save it in a text file. All resulted text files must have the same number of lines as we set the same <step=1s> parameter for each fetched metric data series.

# collecting data for a metric "etcd_server_proposals_pending":
curl -q 'http://<prometheus_server>/api/v1/query_range?query=etcd_server_proposals_pending&start=<test_start>&end=<test_end>&step=1s' \
| jq '.data.result[] | select(.metric.instance == "<instance>") | .values[] | .[1]'  | tr -d '"' > etcd_server_proposals_pending.txt

3. As the all necessary metrics are withdrawn - join all text files into a single csv file by interleaving metric values from each text file with paste.

paste -d',' k8s_services_number_of_services_total.txt process_resident_memory_bytes.txt \
etcd_debugging_snap_save_total_duration_seconds_bucket.txt etcd_disk_backend_commit_duration_seconds_bucket.txt \
etcd_http_failed_total.txt etcd_network_peer_received_bytes_total.txt etcd_network_peer_sent_bytes_total.txt \
etcd_server_proposals_pending.txt > svc_vs_etcd_29_03.csv

4. Plot resulted data in the terms of “PODS vs metrics” and “Services vs metrics” dependency.

4.3. Results

After all the tests were successfully conducted, following data was acquired:

METRICS(ETCD_vs_PODS)

METRICS(ETCD_vs_SERVICES)

4.3.1. Case 1: “ETCD vs PODS”

During this case ETCD cluster resulted in significant performance degradation. We were able to collect ETCD metrics up to ~ 40K pods and ETCD cluster nodes started to log messages like:

2017-03-29 15:50:15.586058 W | etcdserver: apply entries took too long [106.743668ms for 1 entries]
2017-03-29 15:50:15.586080 W | etcdserver: avoid queries with large range/delete range!

After that we lost ability to collect metrics (due too big API timeouts for both ETCD and K8S). When pods number reached ~80K point one of the ETCD cluster member died out with last log entries like:

2017-03-29 19:17:20.325368 E | etcdserver: publish error: etcdserver: request timed out
2017-03-29 19:17:26.850195 E | rafthttp: failed to decode raft message (rafthttp: error limit exceeded)

ETCD session leader logs for the same time:

2017-03-29 19:17:02.013860 W | rafthttp: database snapshot [index: 617915518, to: d0b191dfe9d8dd31] failed to be sent out
2017-03-29 19:17:02.669013 W | etcdserver: failed to send out heartbeat on time (exceeded the 250ms timeout for 724.9550
2017-03-29 19:17:02.669050 W | etcdserver: server is likely overloaded

It is worth to mention that ETCD instances had been running with no limitation settings on the consumption of system resources.

Grafana plots showing ETCD performance during this test:

4.3.1.1. ETCD performance(time series graphs)

Custom plots showing “metric vs pods number” dependency:

4.3.1.2. ETCD performance(number of pods dependency)

4.3.2. Case 2: “ETCD vs SERVICES”

Testing process for this case resulted in K8S stuck on the point of ~ 10K services and each new endpoint spawn took bigger and bigger amount of time. Some of the ETCD nodes reported increased latencies for their operations:

2017-03-27 14:00:55.787485 W | etcdserver: apply entries took too long [160.769033ms for 1 entries]
2017-03-27 14:00:55.787515 W | etcdserver: avoid queries with large range/delete range!

But overall ETCD performance seemed to be stable.

4.3.2.1. ETCD performance(time series graphs)

4.3.2.2. ETCD performance(number of services dependency)

4.3.3. Analysis

4.3.3.1. Case 1 (ETCD vs PODS)

As we can determine from the graphs, increase of pods number heavily affects:

1. RAM consumption

Resident memory usage grew from 2GB up to 40GB during the test ( process_resident_memory_bytes )

2. DISK IO operations

With the more requests for a change (proposal) are coming, the bigger is a queue/latency for sync operations like:

• snapshot save ( etcd_debugging_snap_save_total_duration_seconds_bucket )
• disk sync ( etcd_disk_wal_fsync_duration_seconds_bucket, etcd_disk_backend_commit_duration_seconds_bucket )
• proposal commit ( etcd_server_proposals_pending )

Also we can see an arisen number of http errors for GET queries ( etcd_http_failed_total )

4.3.3.2. Case 2 (ETCD vs SERVICES)

For this case we can infer the same dependency between ETCD and # of services, but impact is much lesser than in the case number 1. The only thing that differ this case from the previous is slightly bigger amount of peer network traffic ( etcd_network_peer_sent_bytes_total ).

4.4. Applications

4.4.1. prepare-minion.py

import os
import shlex
import subprocess

import jinja2

PODS = 200


def render(tpl_path, context):
    path, filename = os.path.split(tpl_path)
    return jinja2.Environment(
        loader=jinja2.FileSystemLoader(path or './')
    ).get_template(filename).render(context)


def create(node):
    service_name = "minion-rc-{}".format(node)
    file_name = "{}.yaml".format(service_name)

    # Create YAML file for new service
    template = render("minion-rc.yaml", {"node": node, "replicas": PODS})
    f = open(file_name, "w")
    f.write(template)
    f.close()

    f = open("nodes.txt", 'a')
    f.write("minion-{}\n".format(node))
    f.close()

    cmd = "kubectl create -f {}".format(file_name)
    args = shlex.split(cmd)

    proc = subprocess.Popen(args, stdout=subprocess.PIPE)
    (out, err) = proc.communicate()

    # Delete YAML file
    os.remove(file_name)


def main():
    # List nodes
    nodes = ["node{}".format(x+1) for x in xrange(20, 430)]
    broken_nodes = ['node16', 'node170', 'node357']
    for n in broken_nodes:
        try:
            nodes.remove(n)
        except ValueError:
            pass

    for n in nodes:
        create(n)


if __name__ == '__main__':
    main()