Experimenting containerized VNFs with Kubernetes VIM

In the past, Tacker only supports creating virtual machine based VNF using Heat. This section covers how to deploy containerized VNF using Kubernetes VIM in Tacker.

Prepare Kubernetes VIM

To use Kubernetes type of VNF, firstly user must register Kubernetes VIM. Tacker supports Kubernetes authentication with two types: basic authentication (username and password) or Bearer token. User can secure the connection to Kubernetes cluster by providing SSL certificate. The following vim-config.yaml file provides necessary information to register a Kubernetes VIM.

auth_url: "https://192.168.11.110:6443"
username: "admin"
password: "admin"
project_name: "default"
ssl_ca_cert: None
type: "kubernetes"

More details about registering Kubernetes VIM, please refer 1

Sample container TOSCA templates

1. One container per VDU example

Currently, because Kubernetes does not support multiple networks such as choosing networks, connection points for applications, therefore users only deploys their applications with default networks (Pod and Service networks). In this case, user need to provide only information about VDU to create a VNF in Tacker.

The following example shows TOSCA template of containerized VNF for pure Kubernetes environment with one container per VDU.

tosca-vnfd-containerized-two-containers.yaml

tosca_definitions_version: tosca_simple_profile_for_nfv_1_0_0
description: A sample containerized VNF with two containers per VDU

metadata:
    template_name: sample-tosca-vnfd

topology_template:
  node_templates:
    VDU1:
      type: tosca.nodes.nfv.VDU.Tacker
      properties:
        namespace: default
        mapping_ports:
          - "80:80"
          - "88:88"
        service_type: NodePort
        vnfcs:
          front_end:
            num_cpus: 0.5
            mem_size: 512 MB
            image: nginx
            ports:
              - "80"
          rss_reader:
            num_cpus: 0.5
            mem_size: 512 MB
            image: nickchase/rss-php-nginx:v1
            ports:
              - "88"
  policies:
    - SP1:
        type: tosca.policies.tacker.Scaling
        targets: [VDU1]
        properties:
          min_instances: 1
          max_instances: 3
          target_cpu_utilization_percentage: 40
          default_instances: 1 # required parameter but ignored for cnf
          increment: 1         # required parameter but ignored for cnf

In “vnfcs”, there are 2 components: front_end and rss_reader. We model them as Containers 2 inside a Pod 3. To provide recover ability of these containers, we put all of containers inside a Deployment 4 object, that can warrant the number of replica with auto-healing automatically.

The following table shows details about parameter of a Container. Config is translated to ConfigMap 5, when user want to update the VNF (config), equivalent ConfigMap will be updated.

+-----------------------------------------------------------------------------------------------+
|          vnfcs          |            Example            |               Description           |
+-----------------------------------------------------------------------------------------------+
|          name           |           front_end           |  Name of container                  |
+-----------------------------------------------------------------------------------------------+
|        num_cpus         |              0.5              |  Number of CPUs                     |
+-----------------------------------------------------------------------------------------------+
|        mem_size         |            512 MB             |  Memory size                        |
+-----------------------------------------------------------------------------------------------+
|         image           |             nginx             |  Image to launch container          |
+-----------------------------------------------------------------------------------------------+
|         ports           |            - "80"             |  Exposed ports in container         |
+-----------------------------------------------------------------------------------------------+
|        command          |      ['/bin/sh','echo']       |  Command when container was started |
+-----------------------------------------------------------------------------------------------+
|         args            |          ['hello']            |  Args of command                    |
+-----------------------------------------------------------------------------------------------+
|        config           |         param0: key1          |  Set variables                      |
|                         |         param1: key2          |                                     |
+-----------------------------------------------------------------------------------------------+

In Tacker, VDU is modeled as a Service 6 in Kubernetes. Because Pods can be easily replaced by others, when the number of replica increased, workload should be shared between Pods. To do this task, we model VDU as Service, it acts as a Load balancer for Pods. Currently, we support some parameters as the following table.

+--------------------------------------------------------------------------------------------------------------------------------+
|     VDU properties      |          Example          |            Description                                                   |
+--------------------------------------------------------------------------------------------------------------------------------+
|       namespace         |          default          | Namespace in Kubernetes where all objects are deployed                   |
+--------------------------------------------------------------------------------------------------------------------------------+
|     mapping_ports       |         - "443:443"       | Published ports and target ports (container ports) of Service Kubernetes |
|                         |         - "80:8080"       |                                                                          |
+--------------------------------------------------------------------------------------------------------------------------------+
|       labels            |      "app: webserver"     | Labels which is set for Kubernetes objects, it is used as Selector to    |
|                         |                           | Service can send requests to Pods                                        |
+--------------------------------------------------------------------------------------------------------------------------------+
|     service_type        |         ClusterIP         | Set service type for Service object.                                     |
|                         |                           |                                                                          |
+--------------------------------------------------------------------------------------------------------------------------------+
|         vnfcs           |                           | Vnfcs are modeled by Containers and Deployment object. User can limit    |
|                         |                           | resource, set image, publish container ports, set commands and variables |
+--------------------------------------------------------------------------------------------------------------------------------+

User can also set scaling policy for VDU by adding the following policy. These information is translated to Horizontal Pod Autoscaler in Kubernetes. In the current scope, we just support auto-scaling with CPU utilization, more metrics will be added in the future.

policies:
  - SP1:
      type: tosca.policies.tacker.Scaling
      targets: [VDU1]
      properties:
        min_instances: 1
        max_instances: 3
        target_cpu_utilization_percentage: 40
        default_instances: 1 # required parameter but ignored for cnf
        increment: 1         # required parameter but ignored for cnf

2. Two containers per VDU example

Similar to the above example, in this scenario, we define 2 containers in VDU1.

tosca-vnfd-containerized.yaml

tosca_definitions_version: tosca_simple_profile_for_nfv_1_0_0
description: A sample containerized VNF with two containers per VDU

metadata:
    template_name: sample-tosca-vnfd

topology_template:
  node_templates:
    VDU1:
      type: tosca.nodes.nfv.VDU.Tacker
      properties:
        namespace: default
        mapping_ports:
          - "80:8080"
        labels:
          - "app: webserver"
        service_type: ClusterIP
        vnfcs:
          web_server:
              num_cpus: 0.5
              mem_size: 512 MB
              image: celebdor/kuryr-demo
              ports:
                - "8080"
              config: |
                param0: key1
                param1: key2

  policies:
    - SP1:
        type: tosca.policies.tacker.Scaling
        targets: [VDU1]
        properties:
          min_instances: 1
          max_instances: 3
          target_cpu_utilization_percentage: 40
          default_instances: 1 # required parameter but ignored for cnf
          increment: 1         # required parameter but ignored for cnf

Viewing a containerized VNF

Create sample containerized VNF

$ openstack vnf descriptor create --vnfd-file tosca-vnfd-containerized.yaml VNFD1
Created a new vnfd:
+-----------------+-------------------------------------------------------------------------------------------------------+
| Field           | Value                                                                                                 |
+-----------------+-------------------------------------------------------------------------------------------------------+
| created_at      | 2018-01-21 14:36:51.757044                                                                            |
| description     | A sample containerized VNF with one container per VDU                                                 |
| id              | fb4a0aa8-e410-4e73-abdc-d2808de155ef                                                                  |
| name            | VNFD1                                                                                                 |
| service_types   | vnfd                                                                                                  |
| template_source | onboarded                                                                                             |
| tenant_id       | 2d22508be9694091bb2f03ce27911416                                                                      |
| updated_at      |                                                                                                       |
+-----------------+-------------------------------------------------------------------------------------------------------+

$ openstack vnf create --vnfd-name VNFD1 --vim-name vim-kubernetes VNF1
Created a new vnf:
+----------------+-------------------------------------------------------------------------------------------------------+
| Field          | Value                                                                                                 |
+----------------+-------------------------------------------------------------------------------------------------------+
| created_at     | 2018-01-21 14:37:23.318018                                                                            |
| description    | A sample containerized VNF with one container per VDU                                                 |
| error_reason   |                                                                                                       |
| id             | 1faf776b-8d2b-4ee6-889d-e3b7c7310411                                                                  |
| instance_id    | default,svc-vdu1-05db44                                                                               |
| mgmt_ip_address|                                                                                                       |
| name           | VNF1                                                                                                  |
| placement_attr | {"vim_name": "vim-kubernetes"}                                                                        |
| status         | PENDING_CREATE                                                                                        |
| tenant_id      | 2d22508be9694091bb2f03ce27911416                                                                      |
| updated_at     |                                                                                                       |
| vim_id         | 791830a6-45fd-468a-bd85-e07fe24e5ce3                                                                  |
| vnfd_id        | fb4a0aa8-e410-4e73-abdc-d2808de155ef                                                                  |
+----------------+-------------------------------------------------------------------------------------------------------+

$ openstack vnf list
+--------------------------------------+------+----------------------------+--------+--------------------------------------+--------------------------------------+
| id                                   | name | mgmt_ip_address            | status | vim_id                               | vnfd_id                              |
+--------------------------------------+------+----------------------------+--------+--------------------------------------+--------------------------------------+
| 1faf776b-8d2b-4ee6-889d-e3b7c7310411 | VNF1 |                            | ACTIVE | 791830a6-45fd-468a-bd85-e07fe24e5ce3 | fb4a0aa8-e410-4e73-abdc-d2808de155ef |
+--------------------------------------+------+----------------------------+--------+--------------------------------------+--------------------------------------+

To test VNF is running in Kubernetes environment, we can check by running following commands

$ kubectl get svc
NAME              TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)   AGE
kubernetes        ClusterIP   192.168.28.129   <none>        443/TCP   5h
svc-vdu1-05db44   ClusterIP   192.168.28.187   <none>        80/TCP    12m

$ kubectl get deployment
NAME              DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
svc-vdu1-05db44   1         1         1            1           16m

$ kubectl get pod
NAME                               READY     STATUS    RESTARTS   AGE
svc-vdu1-05db44-7dcb6b955d-wkh7d   1/1       Running   0          18m

$ kubectl get hpa
NAME              REFERENCE                    TARGETS           MINPODS   MAXPODS   REPLICAS   AGE
svc-vdu1-05db44   Deployment/svc-vdu1-05db44   <unknown> / 40%   1         3         1          17m

$ kubectl get configmap
NAME              DATA      AGE
svc-vdu1-05db44   2         17m

User also can scale VNF manually, by running the following commands:

$ openstack vnf scale --scaling-policy-name SP1 --scaling-type out VNF1

$ kubectl get deployment
NAME              DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
svc-vdu1-651815   2         2         2            1           3h

$ kubectl get pods
NAME                               READY     STATUS    RESTARTS   AGE
svc-vdu1-651815-5b894b8bfb-b6mzq   2/2       Running   0          3h
svc-vdu1-651815-5b894b8bfb-b7f2c   2/2       Running   0          40s

In the same way, user also scale in VNF with scaling-type is ‘in’. The range of scaling manually is limited by ‘min_instances’ and ‘max_instances’ user provide in VNF template.

Multi-Interface for C-VNF

To use multi-interface for C-VNF, User should follow below procedure.

1. Checking kuryr.conf

After installation, user should check kuryr.conf configuration.

$ sudo cat /etc/kuryr/kuryr.conf | grep multi_vif_drivers
multi_vif_drivers = npwg_multiple_interfaces

2. Adding K8s CustomResourceDefinition

To use CustomResourceDefinition, user needs to add CRD. User can make a additional network using yaml file like below. Create yaml file like below and register it.

$ cat ./crdnetwork.yaml
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: network-attachment-definitions.k8s.cni.cncf.io
spec:
  group: k8s.cni.cncf.io
  version: v1
  scope: Namespaced
  names:
    plural: network-attachment-definitions
    singular: network-attachment-definition
    kind: NetworkAttachmentDefinition
    shortNames:
    - net-attach-def
  validation:
    openAPIV3Schema:
      properties:
        spec:
          properties:
            config:
                 type: string

Register crdnetwork.yaml

$ kubectl create -f ~/crdnetwork.yaml

Get crd list

$ kubectl get crd

NAME                                             CREATED AT
kuryrnetpolicies.openstack.org                   2019-07-31T02:23:54Z
kuryrnets.openstack.org                          2019-07-31T02:23:54Z
network-attachment-definitions.k8s.cni.cncf.io   2019-07-31T02:23:55Z

3. Adding neutron subnet id information to k8s CRD

To use neutron subnet in kubernetes, user should register neutron subnet to CRD. At first, user should create subnet.yaml file.

$ cat ./kuryr-subnetname1.yaml

apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: subnetname1
  annotations:
    openstack.org/kuryr-config: '{"subnetId": "$subnet_id"}'

After making a yaml file, user should create subnet with yaml file.

$ kubectl create -f ~/kuryr-subnetname1.yaml

After created, user can check subnet info like below.

$ kubectl get net-attach-def

NAME           AGE
k8s-multi-10   7d
k8s-multi-11   7d

Known Issues and Limitations

  • Does not support Volumes in Kubernetes

  • Horizontal Pod AutoScaler only support CPU utilization

  • Add support Kuryr-Kubernetes for making hybrid network in the future

References

1

https://opendev.org/openstack/tacker/src/branch/master/doc/source/install/kubernetes_vim_installation.rst

2

https://kubernetes.io/docs/concepts/workloads/pods/init-containers

3

https://kubernetes.io/docs/concepts/workloads/pods/pod-overview

4

https://kubernetes.io/docs/concepts/workloads/controllers/deployment

5

https://kubernetes.io/docs/tasks/configure-pod-container/configure-pod-configmap

6

https://kubernetes.io/docs/concepts/services-networking/service