Queens Series Release Notes¶
6.3.0-9¶
New Features¶
Deploy kubelet in master nodes for the k8s_fedora_atomic driver. Previously it was done only for calico, now kubelet will run in all cases. Really useful, for monitoing the master nodes (eg deploy fluentd) or run the kubernetes control-plance self-hosted.
Start Kubernetes workers installation right after the master instances are created rather than waiting for all the services inside masters, which could decrease the Kubernetes cluster launch time significantly.
Use the external cloud provider in k8s_fedora_atomic. The cloud_provider_tag label can be used to select the container tag for it, together with the cloud_provider_enabled label. The cloud provider runs as a DaemonSet on all master nodes.
Upgrade Notes¶
The cloud config for kubernets has been renamed from /etc/kubernetes/kube_openstack_config to /etc/kubernetes/cloud-config as the kubelet expects this exact name when the external cloud provider is used. A copy of /etc/kubernetes/kube_openstack_config is in place for applications developed for previous versions of magnum.
6.3.0¶
New Features¶
Add ‘cloud_provider_enabled’ label for the k8s_fedora_atomic driver. Defaults to true. For specific Kubernetes versions if ‘Cinder’ is selected as a ‘volume_driver’, it is implied that the cloud provider will be enabled since they are combined.
Bug Fixes¶
Add region parameter to the Global configuration section of the Kubernetes configuration file. Setting this parameter will allow Magnum cluster to be created in the multi-regional OpenStack installation.
Fixed a bug where –live-restore was passed to Docker daemon causing the swarm init to fail. Magnum now ensures the –live-restore is not passed to the Docker daemon if it’s default in an image.
6.2.0¶
New Features¶
This allows no Floating IP to be usable with a multimaster configuration in terms of load balancers.
Bug Fixes¶
When creating a multi-master cluster, all master nodes will attempt to create Kubernetes resources in the cluster at this same time, like coredns, the dashboard, calico etc. This race condition shouldn’t be a problem when doing declarative calls instead of imperative (kubectl apply instead of create). However, due to [1], kubectl fails to apply the changes and the deployment scripts fail causing cluster to creation to fail in the case of Heat SoftwareDeployments. This patch passes the ResourceGroup index of every master so that resource creation will be attempted only from the first master node. [1] https://github.com/kubernetes/kubernetes/issues/44165
Create admin cluster role for k8s_fedora_atomic, it is defined in the configuration but it wasn’t applied.
Strip signed certificate. Certificate (ca.crt) has to be striped for some application parsers as they might require pure base64 representation of the certificate itself, without empty characters at the beginning nor the end of file.
6.1.1¶
New Features¶
k8s_fedora_atomic clusters are deployed with RBAC support. Along with RBAC Node authorization is added so the appropriate certificates are generated.
Known Issues¶
Currently, the replicas of coreDNS pod is hardcoded as 1. It’s not a reasonable number for such a critical service. Without DNS, probably all workloads running on the k8s cluster will be broken. Now Magnum is making the coreDNS pod autoscaling based on the nodes and cores number.
Upgrade Notes¶
Using the queens (>=2.9.0) python-magnumclient, when a user executes Openstack coe cluster config, the client certificate has admin as Common Name (CN) and system:masters for Organisation which are required for authorisation with RBAC enabled clusters. This change in the client is backwards compatible, so old clusters (without RBAC enabled) can be reached with certificates generated by the new client. However, old magnum clients will generate certificates that will not be able to contact RBAC enabled clusters. This issue affects only k8s_fedora_atomic clusters and clients <=2.8.0, note that 2.8.0 is still a queens release but only 2.9.0 includes the relevant patch. Finally, users can always generate and sign the certificates using this [0] procedure even with old clients since only the cluster config command is affected. [0] https://docs.openstack.org/magnum/latest/user/index.html#interfacing-with-a-secure-cluster
Security Issues¶
k8s_fedora Remove cluster role from the kubernetes-dashboard account. When accessing the dashboard and skip authentication, users login with the kunernetes-dashboard service account, if that service account has the cluster role, users have admin access without authentication. Create an admin service account for this use case and others.
Bug Fixes¶
Fix etcd configuration in k8s_fedora_atomic driver. Explicitly enable client and peer authentication and set trusted CA (ETCD_TRUSTED_CA_FILE, ETCD_PEER_TRUSTED_CA_FILE, ETCD_CLIENT_CERT_AUTH, ETCD_PEER_CLIENT_CERT_AUTH). Only new clusters will benefit from the fix.
Fix bug #1758672 [1] to protect kubelet in the k8s_fedora_atomic driver. Before this patch kubelet was listening to 0.0.0.0 and for clusters with floating IPs the kubelet was exposed. Also, even on clusters without fips the kubelet was exposed inside the cluster. This patch allows access to the kubelet only over HTTPS and with the appropriate roles. The apiserver and heapster have the appropriate roles to access it. Finally, all read-only ports have been closed to not expose any cluster data. The only remaining open ports without authentication are for healthz. [1] https://bugs.launchpad.net/magnum/+bug/1758672
6.1.0¶
New Features¶
This release introduces ‘federations’ endpoint to Magnum API, which allows an admin to create and manage federations of clusters through Magnum. As the feature is still under development, the endpoints are not bound to any driver yet. For more details, please refer to bp/federation-api [1].
[1] https://review.openstack.org/#/q/topic:bp/federation-api
Add new label ‘cert_manager_api’ enabling the Kubernetes certificate manager API.
Add new labels ‘ingress_controller’ and ‘ingress_controller_role’ enabling the deployment of a Kubernetes Ingress Controller backend for clusters. Default for ‘ingress_controller’ is ‘’ (meaning no controller deployed), with possible values being ‘traefik’. Default for ‘ingress_controller_role’ is ‘ingress’.
Update Kubernetes dashboard to v1.8.3 which is compatible via kubectl proxy. Additionally, heapster is deployed as standalone deployment and the user can enable a grafana-influx stack with the influx_grafana_dashboard_enabled label. See the Kubernetes dashboard documentation for more details. https://github.com/kubernetes/dashboard/wiki
Update k8s_fedora_atomic driver to the latest Fedora Atomic 27 release and run etcd and flanneld in system containers which are removed from the base OS.
Known Issues¶
Adding ‘calico’ as network driver for Kubernetes so as to support network isolation between namespace with k8s network policy.
Upgrade Notes¶
New clusters should be created with kube_tag=v1.9.3 or later. v1.9.3 is the default version in the Queens release.
Bug Fixes¶
Now user can update labels in cluster-template. Previously string is passed as a value to labels, but we know that labels can only hold dictionary values. Now we are parsing the string and storing it as dictionary for labels in cluster-template.
6.0.1¶
New Features¶
Prefix of all container images used in the cluster (Kubernetes components, coredns, kubernetes-dashboard, node-exporter). For example, kubernetes-apiserver is pulled from docker.io/openstackmagnum/kubernetes-apiserver, with this label it can be changed to myregistry.example.com/mycloud/kubernetes-apiserver. Similarly, all other components used in the cluster will be prefixed with this label, which assumes an operator has cloned all expected images in myregistry.example.com/mycloud.
Support passing an availability zone where all cluster nodes should be deployed, via the new availability_zone label. Both swarm_fedora_atomic_v2 and k8s_fedora_atomic_v1 support this new label.
Allow any value to be passed on the docker_storage_driver field by turning it into a StringField (was EnumField), and remove the constraints limiting the values to ‘devicemapper’ and ‘overlay’.
Now admin user can access all clusters across projects.
Magnum now support policy in code [1], which means if users didn’t modify any of policy rules, they can leave policy file (in json or yaml format) empty or just remove it all together. Because from now, Magnum keeps all default policies under magnum/common/policies module. Users can still modify/generate the policy rules they want in the policy.yaml or policy.json file which will override the default policy rules in code only if those rules show in the policy file.
[1]. https://blueprints.launchpad.net/magnum/+spec/policy-in-code
Known Issues¶
Kubernetes client is incompatible with evenlet and breaks the periodic tasks. After Kubernetes client 4.0.0 magnum is affected by the bug below. https://github.com/eventlet/eventlet/issues/147 Magnum has three periodic tasks, one to sync the magnum service, one to update the cluster status and one send cluster metrics The send_metrics task uses the Kubernetes client for Kubernetes clusters and it crashes the sync_cluster_status and send_cluster_metrics tasks. https://bugs.launchpad.net/magnum/+bug/1746510 Additionally, the Kubernetes scale manager needs to be disabled to not break the scale down command completely. Note, that when magnum scales down the cluster will pick the nodes to scale randomly.
Enhancement to support affinity policy for cluster nodes. Before this patch, There is no way to guarantee all nodes of a cluster created on different compute hosts to get high availability.
Upgrade Notes¶
In Magnum configuration, in [drivers] set send_cluster_metrics = False to to avoid collecting metrics using the Kubernetes client which crashes the periodic tasks.
Requires a db upgrade to change the docker_storage_driver field to be a string instead of an enum.
Magnum now supports policy in code, please refer to the relevant features in the release notes for more information.
Security Issues¶
Add new configuration option openstack_ca_file in the drivers section to pass the CA bundle used for the OpenStack API. Setting this file and setting verify_ca to true will result to all requests from the cluster nodes to the OpenStack APIs to be verified.
Bug Fixes¶
[bug 1663757] A configuration parameter, verify_ca, was added to magnum.conf with a default value of True and passed to the heat templates to indicate whether the cluster nodes validate the Certificate Authority when making requests to the OpenStack APIs (Keystone, Magnum, Heat). This parameter can be set to False to disable CA validation if you have self-signed certificates for the OpenStack APIs or you have your own Certificate Authority and you have not installed the Certificate Authority to all nodes.
From now on, server names are prefixed with the cluster name. The cluster name is truncated to 30 characters, (‘_’, ‘.’) are mapped to ‘-’ and non alpha-numeric characters are removed to ensure FQDN compatibility.
Other Notes¶
Default policy.json file is now removed as Magnum now generates the default policies in code. Please be aware that when using that file in your environment.
