Linux bridge: Provider networks

The provider networks architecture example provides layer-2 connectivity between instances and the physical network infrastructure using VLAN (802.1q) tagging. It supports one untagged (flat) network and up to 4095 tagged (VLAN) networks. The actual quantity of VLAN networks depends on the physical network infrastructure. For more information on provider networks, see Provider networks.

Prerequisites

One controller node with the following components:

  • Two network interfaces: management and provider.

  • OpenStack Networking server service and ML2 plug-in.

Two compute nodes with the following components:

  • Two network interfaces: management and provider.

  • OpenStack Networking Linux bridge layer-2 agent, DHCP agent, metadata agent, and any dependencies.

Note

Larger deployments typically deploy the DHCP and metadata agents on a subset of compute nodes to increase performance and redundancy. However, too many agents can overwhelm the message bus. Also, to further simplify any deployment, you can omit the metadata agent and use a configuration drive to provide metadata to instances.

Architecture

Provider networks using Linux bridge - overview

The following figure shows components and connectivity for one untagged (flat) network. In this particular case, the instance resides on the same compute node as the DHCP agent for the network. If the DHCP agent resides on another compute node, the latter only contains a DHCP namespace and Linux bridge with a port on the provider physical network interface.

Provider networks using Linux bridge - components and connectivity - one network

The following figure describes virtual connectivity among components for two tagged (VLAN) networks. Essentially, each network uses a separate bridge that contains a port on the VLAN sub-interface on the provider physical network interface. Similar to the single untagged network case, the DHCP agent may reside on a different compute node.

Provider networks using Linux bridge - components and connectivity - multiple networks

Note

These figures omit the controller node because it does not handle instance network traffic.

Example configuration

Use the following example configuration as a template to deploy provider networks in your environment.

Controller node

  1. Install the Networking service components that provides the neutron-server service and ML2 plug-in.

  2. In the neutron.conf file:

    • Configure common options:

      [DEFAULT]
core_plugin = ml2
auth_strategy = keystone

[database]
# ...

[keystone_authtoken]
# ...

[nova]
# ...

[agent]
# ...

      See the Installation Tutorials and Guides and Configuration Reference for your OpenStack release to obtain the appropriate additional configuration for the [DEFAULT], [database], [keystone_authtoken], [nova], and [agent] sections.

    • Disable service plug-ins because provider networks do not require any. However, this breaks portions of the dashboard that manage the Networking service. See the latest Install Tutorials and Guides for more information.

      [DEFAULT]
service_plugins =

    • Enable two DHCP agents per network so both compute nodes can provide DHCP service provider networks.

      [DEFAULT]
dhcp_agents_per_network = 2

    • If necessary, configure MTU.

  3. In the ml2_conf.ini file:

    • Configure drivers and network types:

      [ml2]
type_drivers = flat,vlan
tenant_network_types =
mechanism_drivers = linuxbridge
extension_drivers = port_security

    • Configure network mappings:

      [ml2_type_flat]
flat_networks = provider

[ml2_type_vlan]
network_vlan_ranges = provider

      Note

      The tenant_network_types option contains no value because the architecture does not support self-service networks.

      Note

      The provider value in the network_vlan_ranges option lacks VLAN ID ranges to support use of arbitrary VLAN IDs.

  4. Populate the database.

    # su -s /bin/sh -c "neutron-db-manage --config-file /etc/neutron/neutron.conf \
  --config-file /etc/neutron/plugins/ml2/ml2_conf.ini upgrade head" neutron

  5. Start the following services:

    • Server

Compute nodes

  1. Install the Networking service Linux bridge layer-2 agent.

  2. In the neutron.conf file, configure common options:

    [DEFAULT]
core_plugin = ml2
auth_strategy = keystone

[database]
# ...

[keystone_authtoken]
# ...

[nova]
# ...

[agent]
# ...

    See the Installation Tutorials and Guides and Configuration Reference for your OpenStack release to obtain the appropriate additional configuration for the [DEFAULT], [database], [keystone_authtoken], [nova], and [agent] sections.

  3. In the linuxbridge_agent.ini file, configure the Linux bridge agent:

    [linux_bridge]
physical_interface_mappings = provider:PROVIDER_INTERFACE

[vxlan]
enable_vxlan = False

[securitygroup]
firewall_driver = iptables

    Replace PROVIDER_INTERFACE with the name of the underlying interface that handles provider networks. For example, eth1.

  4. In the dhcp_agent.ini file, configure the DHCP agent:

    [DEFAULT]
interface_driver = linuxbridge
enable_isolated_metadata = True
force_metadata = True

    Note

    The force_metadata option forces the DHCP agent to provide a host route to the metadata service on 169.254.169.254 regardless of whether the subnet contains an interface on a router, thus maintaining similar and predictable metadata behavior among subnets.

  5. In the metadata_agent.ini file, configure the metadata agent:

    [DEFAULT]
nova_metadata_host = controller
metadata_proxy_shared_secret = METADATA_SECRET

    The value of METADATA_SECRET must match the value of the same option in the [neutron] section of the nova.conf file.

  6. Start the following services:

    • Linux bridge agent

    • DHCP agent

    • Metadata agent

Verify service operation

  1. Source the administrative project credentials.

  2. Verify presence and operation of the agents:

    $ openstack network agent list
+--------------------------------------+--------------------+----------+-------------------+-------+-------+---------------------------+
| ID                                   | Agent Type         | Host     | Availability Zone | Alive | State | Binary                    |
+--------------------------------------+--------------------+----------+-------------------+-------+-------+---------------------------+
| 09de6af6-c5f1-4548-8b09-18801f068c57 | Linux bridge agent | compute2 | None              | True  | UP    | neutron-linuxbridge-agent |
| 188945d1-9e70-4803-a276-df924e0788a4 | Linux bridge agent | compute1 | None              | True  | UP    | neutron-linuxbridge-agent |
| e76c440d-d5f6-4316-a674-d689630b629e | DHCP agent         | compute1 | nova              | True  | UP    | neutron-dhcp-agent        |
| e67367de-6657-11e6-86a4-931cd04404bb | DHCP agent         | compute2 | nova              | True  | UP    | neutron-dhcp-agent        |
| e8174cae-6657-11e6-89f0-534ac6d0cb5c | Metadata agent     | compute1 | None              | True  | UP    | neutron-metadata-agent    |
| ece49ec6-6657-11e6-bafb-c7560f19197d | Metadata agent     | compute2 | None              | True  | UP    | neutron-metadata-agent    |
+--------------------------------------+--------------------+----------+-------------------+-------+-------+---------------------------+

Create initial networks

The configuration supports one flat or multiple VLAN provider networks. For simplicity, the following procedure creates one flat provider network.

  1. Source the administrative project credentials.

  2. Create a flat network.

    $ openstack network create --share --provider-physical-network provider \
  --provider-network-type flat provider1
+---------------------------+-----------+-
| Field                     | Value     |
+---------------------------+-----------+
| admin_state_up            | UP        |
| mtu                       | 1500      |
| name                      | provider1 |
| port_security_enabled     | True      |
| provider:network_type     | flat      |
| provider:physical_network | provider  |
| provider:segmentation_id  | None      |
| router:external           | Internal  |
| shared                    | True      |
| status                    | ACTIVE    |
+---------------------------+-----------+

    Note

    The share option allows any project to use this network. To limit access to provider networks, see Role-Based Access Control (RBAC).

    Note

    To create a VLAN network instead of a flat network, change --provider-network-type flat to --provider-network-type vlan and add --provider-segment with a value referencing the VLAN ID.

  3. Create a IPv4 subnet on the provider network.

    $ openstack subnet create --subnet-range 203.0.113.0/24 --gateway 203.0.113.1 \
  --network provider1 --allocation-pool start=203.0.113.11,end=203.0.113.250 \
  --dns-nameserver 8.8.4.4 provider1-v4
+-------------------+----------------------------+
| Field             | Value                      |
+-------------------+----------------------------+
| allocation_pools  | 203.0.113.11-203.0.113.250 |
| cidr              | 203.0.113.0/24             |
| dns_nameservers   | 8.8.4.4                    |
| enable_dhcp       | True                       |
| gateway_ip        | 203.0.113.1                |
| ip_version        | 4                          |
| name              | provider1-v4               |
+-------------------+----------------------------+

    Important

    Enabling DHCP causes the Networking service to provide DHCP which can interfere with existing DHCP services on the physical network infrastructure. Use the --no-dhcp option to have the subnet managed by existing DHCP services.

  4. Create a IPv6 subnet on the provider network.

    $ openstack subnet create --subnet-range fd00:203:0:113::/64 --gateway fd00:203:0:113::1 \
  --ip-version 6 --ipv6-address-mode slaac --network provider1 \
  --dns-nameserver 2001:4860:4860::8844 provider1-v6
+-------------------+------------------------------------------------------+
| Field             | Value                                                |
+-------------------+------------------------------------------------------+
| allocation_pools  | fd00:203:0:113::2-fd00:203:0:113:ffff:ffff:ffff:ffff |
| cidr              | fd00:203:0:113::/64                                  |
| dns_nameservers   | 2001:4860:4860::8844                                 |
| enable_dhcp       | True                                                 |
| gateway_ip        | fd00:203:0:113::1                                    |
| ip_version        | 6                                                    |
| ipv6_address_mode | slaac                                                |
| ipv6_ra_mode      | None                                                 |
| name              | provider1-v6                                         |
+-------------------+------------------------------------------------------+

    Note

    The Networking service uses the layer-3 agent to provide router advertisement. Provider networks rely on physical network infrastructure for layer-3 services rather than the layer-3 agent. Thus, the physical network infrastructure must provide router advertisement on provider networks for proper operation of IPv6.

Verify network operation

  1. On each compute node, verify creation of the qdhcp namespace.

    # ip netns
qdhcp-8b868082-e312-4110-8627-298109d4401c

  2. Source a regular (non-administrative) project credentials.

  3. Create the appropriate security group rules to allow ping and SSH access instances using the network.

    $ openstack security group rule create --proto icmp default
+------------------+-----------+
| Field            | Value     |
+------------------+-----------+
| direction        | ingress   |
| ethertype        | IPv4      |
| protocol         | icmp      |
| remote_ip_prefix | 0.0.0.0/0 |
+------------------+-----------+

$ openstack security group rule create --ethertype IPv6 --proto ipv6-icmp default
+-----------+-----------+
| Field     | Value     |
+-----------+-----------+
| direction | ingress   |
| ethertype | IPv6      |
| protocol  | ipv6-icmp |
+-----------+-----------+

$ openstack security group rule create --proto tcp --dst-port 22 default
+------------------+-----------+
| Field            | Value     |
+------------------+-----------+
| direction        | ingress   |
| ethertype        | IPv4      |
| port_range_max   | 22        |
| port_range_min   | 22        |
| protocol         | tcp       |
| remote_ip_prefix | 0.0.0.0/0 |
+------------------+-----------+

$ openstack security group rule create --ethertype IPv6 --proto tcp --dst-port 22 default
+------------------+-----------+
| Field            | Value     |
+------------------+-----------+
| direction        | ingress   |
| ethertype        | IPv6      |
| port_range_max   | 22        |
| port_range_min   | 22        |
| protocol         | tcp       |
+------------------+-----------+

  4. Launch an instance with an interface on the provider network. For example, a CirrOS image using flavor ID 1.

    $ openstack server create --flavor 1 --image cirros \
  --nic net-id=NETWORK_ID provider-instance1

    Replace NETWORK_ID with the ID of the provider network.

  5. Determine the IPv4 and IPv6 addresses of the instance.

    $ openstack server list
+--------------------------------------+--------------------+--------+------------------------------------------------------------+--------+---------+
| ID                                   | Name               | Status | Networks                                                   | Image  | Flavor  |
+--------------------------------------+--------------------+--------+------------------------------------------------------------+--------+---------+
| 018e0ae2-b43c-4271-a78d-62653dd03285 | provider-instance1 | ACTIVE | provider1=203.0.113.13, fd00:203:0:113:f816:3eff:fe58:be4e | cirros | m1.tiny |
+--------------------------------------+--------------------+--------+------------------------------------------------------------+--------+---------+

  6. On the controller node or any host with access to the provider network, ping the IPv4 and IPv6 addresses of the instance.

    $ ping -c 4 203.0.113.13
PING 203.0.113.13 (203.0.113.13) 56(84) bytes of data.
64 bytes from 203.0.113.13: icmp_req=1 ttl=63 time=3.18 ms
64 bytes from 203.0.113.13: icmp_req=2 ttl=63 time=0.981 ms
64 bytes from 203.0.113.13: icmp_req=3 ttl=63 time=1.06 ms
64 bytes from 203.0.113.13: icmp_req=4 ttl=63 time=0.929 ms

--- 203.0.113.13 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3002ms
rtt min/avg/max/mdev = 0.929/1.539/3.183/0.951 ms

$ ping6 -c 4 fd00:203:0:113:f816:3eff:fe58:be4e
PING fd00:203:0:113:f816:3eff:fe58:be4e(fd00:203:0:113:f816:3eff:fe58:be4e) 56 data bytes
64 bytes from fd00:203:0:113:f816:3eff:fe58:be4e icmp_seq=1 ttl=64 time=1.25 ms
64 bytes from fd00:203:0:113:f816:3eff:fe58:be4e icmp_seq=2 ttl=64 time=0.683 ms
64 bytes from fd00:203:0:113:f816:3eff:fe58:be4e icmp_seq=3 ttl=64 time=0.762 ms
64 bytes from fd00:203:0:113:f816:3eff:fe58:be4e icmp_seq=4 ttl=64 time=0.486 ms

--- fd00:203:0:113:f816:3eff:fe58:be4e ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 2999ms
rtt min/avg/max/mdev = 0.486/0.796/1.253/0.282 ms

  7. Obtain access to the instance.

  8. Test IPv4 and IPv6 connectivity to the Internet or other external network.

Network traffic flow

The following sections describe the flow of network traffic in several common scenarios. North-south network traffic travels between an instance and external network such as the Internet. East-west network traffic travels between instances on the same or different networks. In all scenarios, the physical network infrastructure handles switching and routing among provider networks and external networks such as the Internet. Each case references one or more of the following components:

  • Provider network 1 (VLAN)

    • VLAN ID 101 (tagged)

    • IP address ranges 203.0.113.0/24 and fd00:203:0:113::/64

    • Gateway (via physical network infrastructure)

      • IP addresses 203.0.113.1 and fd00:203:0:113:0::1

  • Provider network 2 (VLAN)

    • VLAN ID 102 (tagged)

    • IP address range 192.0.2.0/24 and fd00:192:0:2::/64

    • Gateway

      • IP addresses 192.0.2.1 and fd00:192:0:2::1

  • Instance 1

    • IP addresses 203.0.113.101 and fd00:203:0:113:0::101

  • Instance 2

    • IP addresses 192.0.2.101 and fd00:192:0:2:0::101

North-south scenario: Instance with a fixed IP address

  • The instance resides on compute node 1 and uses provider network 1.

  • The instance sends a packet to a host on the Internet.

The following steps involve compute node 1.

  1. The instance interface (1) forwards the packet to the provider bridge instance port (2) via veth pair.

  2. Security group rules (3) on the provider bridge handle firewalling and connection tracking for the packet.

  3. The VLAN sub-interface port (4) on the provider bridge forwards the packet to the physical network interface (5).

  4. The physical network interface (5) adds VLAN tag 101 to the packet and forwards it to the physical network infrastructure switch (6).

The following steps involve the physical network infrastructure:

  1. The switch removes VLAN tag 101 from the packet and forwards it to the router (7).

  2. The router routes the packet from the provider network (8) to the external network (9) and forwards the packet to the switch (10).

  3. The switch forwards the packet to the external network (11).

  4. The external network (12) receives the packet.

Provider networks using Linux bridge - network traffic flow - north/south

Note

Return traffic follows similar steps in reverse.

East-west scenario 1: Instances on the same network

Instances on the same network communicate directly between compute nodes containing those instances.

  • Instance 1 resides on compute node 1 and uses provider network 1.

  • Instance 2 resides on compute node 2 and uses provider network 1.

  • Instance 1 sends a packet to instance 2.

The following steps involve compute node 1:

  1. The instance 1 interface (1) forwards the packet to the provider bridge instance port (2) via veth pair.

  2. Security group rules (3) on the provider bridge handle firewalling and connection tracking for the packet.

  3. The VLAN sub-interface port (4) on the provider bridge forwards the packet to the physical network interface (5).

  4. The physical network interface (5) adds VLAN tag 101 to the packet and forwards it to the physical network infrastructure switch (6).

The following steps involve the physical network infrastructure:

  1. The switch forwards the packet from compute node 1 to compute node 2 (7).

The following steps involve compute node 2:

  1. The physical network interface (8) removes VLAN tag 101 from the packet and forwards it to the VLAN sub-interface port (9) on the provider bridge.

  2. Security group rules (10) on the provider bridge handle firewalling and connection tracking for the packet.

  3. The provider bridge instance port (11) forwards the packet to the instance 2 interface (12) via veth pair.

Provider networks using Linux bridge - network traffic flow - east/west scenario 1

Note

Return traffic follows similar steps in reverse.

East-west scenario 2: Instances on different networks

Instances communicate via router on the physical network infrastructure.

  • Instance 1 resides on compute node 1 and uses provider network 1.

  • Instance 2 resides on compute node 1 and uses provider network 2.

  • Instance 1 sends a packet to instance 2.

Note

Both instances reside on the same compute node to illustrate how VLAN tagging enables multiple logical layer-2 networks to use the same physical layer-2 network.

The following steps involve the compute node:

  1. The instance 1 interface (1) forwards the packet to the provider bridge instance port (2) via veth pair.

  2. Security group rules (3) on the provider bridge handle firewalling and connection tracking for the packet.

  3. The VLAN sub-interface port (4) on the provider bridge forwards the packet to the physical network interface (5).

  4. The physical network interface (5) adds VLAN tag 101 to the packet and forwards it to the physical network infrastructure switch (6).

The following steps involve the physical network infrastructure:

  1. The switch removes VLAN tag 101 from the packet and forwards it to the router (7).

  2. The router routes the packet from provider network 1 (8) to provider network 2 (9).

  3. The router forwards the packet to the switch (10).

  4. The switch adds VLAN tag 102 to the packet and forwards it to compute node 1 (11).

The following steps involve the compute node:

  1. The physical network interface (12) removes VLAN tag 102 from the packet and forwards it to the VLAN sub-interface port (13) on the provider bridge.

  2. Security group rules (14) on the provider bridge handle firewalling and connection tracking for the packet.

  3. The provider bridge instance port (15) forwards the packet to the instance 2 interface (16) via veth pair.

Provider networks using Linux bridge - network traffic flow - east/west scenario 2

Note

Return traffic follows similar steps in reverse.