Kubernetes Network Custom Resource Definition De-facto Standard v1 Support


This spec proposes an approach to support the mechanism defined in Kubernetes Network Custom Resource Definition De-facto Standard Version 1 [1], which is used to attach multiple VIFs to Pods.

Problem Description

There is always a desire for Pods to be able to be attached to multiple interfaces in NFV use cases. However CNI plugins which have implemented this functionality are using different way of defining the additional interfaces in Pods. There is no standard approach among those CNI plugins.

Therefore, the Networking Plumbing Working Group [2] drafted a spec (the NPWG spec) trying to standardize the way of attaching Pods to multiple networks.

Proposed Change

The NPWG spec defines a “Network” Custom Resource object which describes how to attach a Pod to the logical or physical network referenced by the object.

The proposed change is based on VIF-Handler And Vif Drivers Design [3]. A new VIF driver ‘npwg_multiple_interfaces’ will be created to parse the annotation of Pods and Network CRDs. The new VIF driver will be invoked by the multi-vif driver as another sub-drivers. It should return a list of VIF objects. The ‘npwg_multiple_interfaces’ should invoke other VIF driver to create the vif objects if it is necessary.

The VIFHandler then updates the Pod annotation of ‘openstack.org/kuryr-vif’ with the VIF objects. So that the Kuryr CNI can read these VIFs, and attaches each of them to Pods namespace. If any of the additional interfaces failed to be attached to the Pod, or any error happens during attachment, the CNI shall return with error.

Option in config file might look like this:


enabled_vif_drivers = npwg_multiple_interfaces

To define additional network in Pods, NPWG spec defines format of annotation. Here’s how a Pod Spec with additional networks requests might look like:

kind: Pod
  name: my-pod
  namespace: my-namespace
    k8s.v1.cni.cncf.io/networks: net-a,net-b,other-ns/net-c

Or in JSON format like:

kind: Pod
    name: my-pod
    namespace: my-namespace
      k8s.v1.cni.cncf.io/networks: |

Then the VIF driver can parse the network information defined in ‘Network’ objects. In NPWG spec, the ‘NetworkAttachmentDefinition’ object definition is very flexible. Implementations that are not CNI delegating plugins can add annotations to the Network object and use those to store non-CNI configuration. And it is up to the implementation to define the content it requires.

Here is how ‘CustomResourceDefinition’ CRD specified in the NPWG spec.

apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
  name: network-attachment-definitions.k8s.cni.cncf.io
  group: k8s.cni.cncf.io
  version: v1
  scope: Namespaced
    plural: network-attachment-definitions
    singular: network-attachment-definition
    kind: NetworkAttachmentDefinition
      - net-attach-def
              type: string

For Kuryr-kubernetes, users should define the ‘Network’ object with a Neutron subnet created previously like:

apiVersion: "kubernetes.cni.cncf.io/v1"
kind: Network
  name: a-bridge-network
    openstack.org/kuryr-config: '{
      "subnetId": "id_of_neutron_subnet_created_previously"

With information read from Pod annotation k8s.v1.cni.cncf.io/networks and ‘Network’ objects, the Neutron ports could either be created or retrieved. Then the Pod annotation openstack.org/kuryr-vif will be updated accordingly.

Here’s how openstack.org/kuryr-vif annotation with additional networks might look like:

kind: Pod
  name: my-pod
  namespace: my-namespace
    openstack.org/kuryr-vif: {
      # default interface remains intact
      "eth0": {
        ... Neutron vif object from default subnet ...
      # additional interfaces appended by driver 'npwg_multiple_interfaces'
      "eth1": {
        ... Neutron vif object ...
      "eth2": {
        ... Neutron vif object ...


Currently, Kuryr-Kubernetes has already designed a way of defining additional VIF. This spec will not change that part. Users can choose using which format they want by configuring ‘enabled_vif_drivers’.

Other end user impact

Pods always attach the default Kubernetes network as how Kuryr-Kubernetes works today, and all networks specified in the Pod annotation are sidecars.


Primary assignee: Peng Liu

Work Items

  • Implement a new NPWG spec compatible VIF driver.

  • Document the procedure of using this new VIF driver.

Possible Further Work

  • To keep on track of the subsequent releases of NPWG spec.

  • To allow defining new neutron network/subnet in ‘Network’ objects, so that kuryr can create them in Neutron first, then attach Pod to it.