Current Series Release Notes¶

New Features¶

• k8s_fedora_atomic clusters are deployed with RBAC support. Along with RBAC Node authorization is added so the appropriate certificates are generated.

• This release introduces ‘federations’ endpoint to Magnum API, which allows an admin to create and manage federations of clusters through Magnum. As the feature is still under development, the endpoints are not bound to any driver yet. For more details, please refer to bp/federation-api [1].

  [1] https://review.openstack.org/#/q/topic:bp/federation-api

• Add new label ‘cert_manager_api’ enabling the kubernetes certificate manager api.

• Add new labels ‘ingress_controller’ and ‘ingress_controller_role’ enabling the deployment of a Kubernetes Ingress Controller backend for clusters. Default for ‘ingress_controller’ is ‘’ (meaning no controller deployed), with possible values being ‘traefik’. Default for ‘ingress_controller_role’ is ‘ingress’.

• In the OpenStack deployment with Octavia service enabled, the Octavia service should be used not only for master nodes high availability, but also for k8s LoadBalancer type service implementation as well.

• Update kubernetes dashboard to v1.8.3 which is compatible via kubectl proxy. Addionally, heapster is deployed as standalone deployemt and the user can enable a grafana-influx stack with the influx_grafana_dashboard_enabled label. See the kubernetes dashboard documenation for more details. https://github.com/kubernetes/dashboard/wiki

• Update k8s_fedora_atomic driver to the latest Fedora Atomic 27 release and run etcd and flanneld in system containers which are removed from the base OS.

Known Issues¶

• Adding ‘calico’ as network driver for Kubernetes so as to support network isolation between namespace with k8s network policy.

• Currently, the replicas of coreDNS pod is hardcoded as 1. It’s not a reasonable number for such a critical service. Without DNS, probably all workloads running on the k8s cluster will be broken. Now Magnum is making the coreDNS pod autoscaling based on the nodes and cores number.

Upgrade Notes¶

• Using the queens (>=2.9.0) python-magnumclient, when a user executes openstack coe cluster config, the client certificate has admin as Common Name (CN) and system:masters for Organization which are required for authorization with RBAC enabled clusters. This change in the client is backwards compatible, so old clusters (without RBAC enabled) can be reached with certificates generated by the new client. However, old magnum clients will generate certificates that will not be able to contact RBAC enabled clusters. This issue affects only k8s_fedora_atomic clusters and clients <=2.8.0, note that 2.8.0 is still a queens release but only 2.9.0 includes the relevant patch. Finally, users can always generate and sign the certificates using this [0] procedure even with old clients since only the cluster config command is affected. [0] https://docs.openstack.org/magnum/latest/user/index.html#interfacing-with-a-secure-cluster

• New clusters should be created with kube_tag=v1.9.3 or later. v1.9.3 is the default version in the queens release.

Security Issues¶

• k8s_fedora Remove cluster role from the kubernetes-dashboard account. When accessing the dashboard and skip authentication, users login with the kunernetes-dashboard service account, if that service account has the cluster role, users have admin access without authentication. Create an admin service account for this use case and others.

Bug Fixes¶

• Now user can update labels in cluster-template. Previously string is passed as a value to labels, but we know that labels can only hold dictionary values. Now we are parsing the string and storing it as dictionary for labels in cluster-template.

• Fix etcd configuration in k8s_fedora_atomic driver. Explicitly enable client and peer authentication and set trusted CA (ETCD_TRUSTED_CA_FILE, ETCD_PEER_TRUSTED_CA_FILE, ETCD_CLIENT_CERT_AUTH, ETCD_PEER_CLIENT_CERT_AUTH). Only new clusters will benefit from the fix.

• When creating a multi-master cluster, all master nodes will attempt to create kubernetes resources in the cluster at this same time, like coredns, the dashboard, calico etc. This race conditon shouldn’t be a problem when doing declarative calls instead of imperative (kubectl apply instead of create). However, due to [1], kubectl fails to apply the changes and the deployemnt scripts fail causing cluster to creation to fail in the case of Heat SoftwareDeployments. This patch passes the ResourceGroup index of every master so that resource creation will be attempted only from the first master node. [1] https://github.com/kubernetes/kubernetes/issues/44165

• Fix bug #1758672 [1] to protect kubelet in the k8s_fedora_atomic driver. Before this patch kubelet was listening to 0.0.0.0 and for clusters with floating IPs the kubelet was exposed. Also, even on clusters without fips the kubelet was exposed inside the cluster. This patch allows access to the kubelet only over https and with the appropriate roles. The apiserver and heapster have the appropriate roles to access it. Finally, all read-only ports have been closed to not expose any cluster data. The only remaining open ports without authentication are for healthz. [1] https://bugs.launchpad.net/magnum/+bug/1758672

• Strip signed certificate. Certificate (ca.crt) has to be striped for some application parsers as they might require pure base64 representation of the certificate itself, without empty characters at the beginning nor the end of file.

Known Issues¶

• Kubernetes client is incompatible with evenlet and breaks the periodic tasks. After kubernetes client 4.0.0 magnum is affected by the bug below. https://github.com/eventlet/eventlet/issues/147 Magnum has three periodic tasks, one to sync the magnum service, one to update the cluster status and one send cluster metrics The send_metrics task uses the kubernetes client for kubernetes clusters and it crashes the sync_cluster_status and send_cluster_metrics tasks. https://bugs.launchpad.net/magnum/+bug/1746510 Additionally, the kubernetes scale manager needs to be disabled to not break the scale down command completely. Note, that when magnum scales down the cluster will pick the nodes to scale randomly.

Upgrade Notes¶

• In magnum configuration, in [drivers] set send_cluster_metrics = False to to avoid collecting metrics using the kubernetes client which crashes the periodic tasks.