North South Networking via Direct Provider Networks

The following figure illustrates one typical networking mode, instances have two interfaces, one interface is connected to net1 for heartbeat or data replication, the other interface is connected to phy_net1 or phy_net2 to provide service. There is different physical network in different region to support service redundancy in case of region level failure.

+-----------------+       +-----------------+
|RegionOne        |       |RegionTwo        |
|                 |       |                 |
|   phy_net1      |       |   phy_net2      |
|  +--+---------+ |       |  +--+---------+ |
|     |           |       |     |           |
|     |           |       |     |           |
|  +--+--------+  |       |  +--+--------+  |
|  |           |  |       |  |           |  |
|  | Instance1 |  |       |  | Instance2 |  |
|  +------+----+  |       |  +------+----+  |
|         |       |       |         |       |
|         |       |       |         |       |
|   net1  |       |       |         |       |
|  +------+-------------------------+---+   |
|                 |       |                 |
+-----------------+       +-----------------+

How to create this network topology

Create provider network phy_net1, which will be located in RegionOne.

$ neutron --os-region-name=CentralRegion net-create --provider:network_type vlan --provider:physical_network extern --availability-zone-hint RegionOne phy_net1
+---------------------------+--------------------------------------+
| Field                     | Value                                |
+---------------------------+--------------------------------------+
| admin_state_up            | True                                 |
| availability_zone_hints   | RegionOne                            |
| id                        | b7832cbb-d399-4d5d-bcfd-d1b804506a1a |
| name                      | phy_net1                             |
| project_id                | ce444c8be6da447bb412db7d30cd7023     |
| provider:network_type     | vlan                                 |
| provider:physical_network | extern                               |
| provider:segmentation_id  | 170                                  |
| router:external           | False                                |
| shared                    | False                                |
| status                    | ACTIVE                               |
| subnets                   |                                      |
| tenant_id                 | ce444c8be6da447bb412db7d30cd7023     |
+---------------------------+--------------------------------------+

Create subnet in phy_net1.

$ neutron --os-region-name=CentralRegion subnet-create phy_net1 202.96.1.0/24
+-------------------+------------------------------------------------+
| Field             | Value                                          |
+-------------------+------------------------------------------------+
| allocation_pools  | {"start": "202.96.1.2", "end": "202.96.1.254"} |
| cidr              | 202.96.1.0/24                                  |
| created_at        | 2017-01-11T08:43:48Z                           |
| description       |                                                |
| dns_nameservers   |                                                |
| enable_dhcp       | True                                           |
| gateway_ip        | 202.96.1.1                                     |
| host_routes       |                                                |
| id                | 4941c48e-5602-40fc-a117-e84833b85ed3           |
| ip_version        | 4                                              |
| ipv6_address_mode |                                                |
| ipv6_ra_mode      |                                                |
| name              |                                                |
| network_id        | b7832cbb-d399-4d5d-bcfd-d1b804506a1a           |
| project_id        | ce444c8be6da447bb412db7d30cd7023               |
| revision_number   | 2                                              |
| subnetpool_id     |                                                |
| tenant_id         | ce444c8be6da447bb412db7d30cd7023               |
| updated_at        | 2017-01-11T08:43:48Z                           |
+-------------------+------------------------------------------------+

Create provider network phy_net2, which will be located in RegionTwo.

$ neutron --os-region-name=CentralRegion net-create --provider:network_type vlan --provider:physical_network extern --availability-zone-hint RegionTwo phy_net2
+---------------------------+--------------------------------------+
| Field                     | Value                                |
+---------------------------+--------------------------------------+
| admin_state_up            | True                                 |
| availability_zone_hints   | RegionTwo                            |
| id                        | 731293af-e68f-4677-b433-f46afd6431f3 |
| name                      | phy_net2                             |
| project_id                | ce444c8be6da447bb412db7d30cd7023     |
| provider:network_type     | vlan                                 |
| provider:physical_network | extern                               |
| provider:segmentation_id  | 168                                  |
| router:external           | False                                |
| shared                    | False                                |
| status                    | ACTIVE                               |
| subnets                   |                                      |
| tenant_id                 | ce444c8be6da447bb412db7d30cd7023     |
+---------------------------+--------------------------------------+

Create subnet in phy_net2.

$ neutron --os-region-name=CentralRegion subnet-create phy_net2 202.96.2.0/24
+-------------------+------------------------------------------------+
| Field             | Value                                          |
+-------------------+------------------------------------------------+
| allocation_pools  | {"start": "202.96.2.2", "end": "202.96.2.254"} |
| cidr              | 202.96.2.0/24                                  |
| created_at        | 2017-01-11T08:47:07Z                           |
| description       |                                                |
| dns_nameservers   |                                                |
| enable_dhcp       | True                                           |
| gateway_ip        | 202.96.2.1                                     |
| host_routes       |                                                |
| id                | f5fb4f11-4bc1-4911-bcca-b0eaccc6eaf9           |
| ip_version        | 4                                              |
| ipv6_address_mode |                                                |
| ipv6_ra_mode      |                                                |
| name              |                                                |
| network_id        | 731293af-e68f-4677-b433-f46afd6431f3           |
| project_id        | ce444c8be6da447bb412db7d30cd7023               |
| revision_number   | 2                                              |
| subnetpool_id     |                                                |
| tenant_id         | ce444c8be6da447bb412db7d30cd7023               |
| updated_at        | 2017-01-11T08:47:08Z                           |
+-------------------+------------------------------------------------+

Create net1 which will work as the L2 network across RegionOne and RegionTwo.

If net1 is vlan based cross-Neutron L2 network
$ neutron --os-region-name=CentralRegion net-create --provider:network_type vlan --provider:physical_network bridge --availability-zone-hint az1 --availability-zone-hint az2 net1
+---------------------------+--------------------------------------+
| Field                     | Value                                |
+---------------------------+--------------------------------------+
| admin_state_up            | True                                 |
| availability_zone_hints   | az1                                  |
|                           | az2                                  |
| id                        | 1897a446-bf6a-4bce-9374-6a3825ee5051 |
| name                      | net1                                 |
| project_id                | ce444c8be6da447bb412db7d30cd7023     |
| provider:network_type     | vlan                                 |
| provider:physical_network | bridge                               |
| provider:segmentation_id  | 132                                  |
| router:external           | False                                |
| shared                    | False                                |
| status                    | ACTIVE                               |
| subnets                   |                                      |
| tenant_id                 | ce444c8be6da447bb412db7d30cd7023     |
+---------------------------+--------------------------------------+

If net1 is vxlan based cross-Neutron L2 network
$ neutron --os-region-name=CentralRegion net-create --provider:network_type vxlan --availability-zone-hint az1 --availability-zone-hint az2 net1
+---------------------------+--------------------------------------+
| Field                     | Value                                |
+---------------------------+--------------------------------------+
| admin_state_up            | True                                 |
| availability_zone_hints   | az1                                  |
|                           | az2                                  |
| id                        | 0093f32c-2ecd-4888-a8c2-a6a424bddfe8 |
| name                      | net1                                 |
| project_id                | ce444c8be6da447bb412db7d30cd7023     |
| provider:network_type     | vxlan                                |
| provider:physical_network |                                      |
| provider:segmentation_id  | 1036                                 |
| router:external           | False                                |
| shared                    | False                                |
| status                    | ACTIVE                               |
| subnets                   |                                      |
| tenant_id                 | ce444c8be6da447bb412db7d30cd7023     |
+---------------------------+--------------------------------------+

Create subnet in net1.

$ neutron --os-region-name=CentralRegion subnet-create net1 10.0.1.0/24
+-------------------+--------------------------------------------+
| Field             | Value                                      |
+-------------------+--------------------------------------------+
| allocation_pools  | {"start": "10.0.1.2", "end": "10.0.1.254"} |
| cidr              | 10.0.1.0/24                                |
| created_at        | 2017-01-11T08:49:53Z                       |
| description       |                                            |
| dns_nameservers   |                                            |
| enable_dhcp       | True                                       |
| gateway_ip        | 10.0.1.1                                   |
| host_routes       |                                            |
| id                | 6a6c63b4-7f41-4a8f-9393-55cd79380e5a       |
| ip_version        | 4                                          |
| ipv6_address_mode |                                            |
| ipv6_ra_mode      |                                            |
| name              |                                            |
| network_id        | 1897a446-bf6a-4bce-9374-6a3825ee5051       |
| project_id        | ce444c8be6da447bb412db7d30cd7023           |
| revision_number   | 2                                          |
| subnetpool_id     |                                            |
| tenant_id         | ce444c8be6da447bb412db7d30cd7023           |
| updated_at        | 2017-01-11T08:49:53Z                       |
+-------------------+--------------------------------------------+

List available images in RegionOne.

$ glance --os-region-name=RegionOne image-list
+--------------------------------------+---------------------------------+
| ID                                   | Name                            |
+--------------------------------------+---------------------------------+
| 924a5078-efe5-4abf-85e8-992b7e5f6ac3 | cirros-0.3.4-x86_64-uec         |
| d3e8349d-d58d-4d17-b0ab-951c095fbbc4 | cirros-0.3.4-x86_64-uec-kernel  |
| c4cd7482-a145-4f26-9f41-a9ac17b9492c | cirros-0.3.4-x86_64-uec-ramdisk |
+--------------------------------------+---------------------------------+

List available flavors in RegionOne.

$ nova --os-region-name=RegionOne flavor-list
+----+-----------+-----------+------+-----------+------+-------+-------------+-----------+
| ID | Name      | Memory_MB | Disk | Ephemeral | Swap | VCPUs | RXTX_Factor | Is_Public |
+----+-----------+-----------+------+-----------+------+-------+-------------+-----------+
| 1  | m1.tiny   | 512       | 1    | 0         |      | 1     | 1.0         | True      |
| 2  | m1.small  | 2048      | 20   | 0         |      | 1     | 1.0         | True      |
| 3  | m1.medium | 4096      | 40   | 0         |      | 2     | 1.0         | True      |
| 4  | m1.large  | 8192      | 80   | 0         |      | 4     | 1.0         | True      |
| 5  | m1.xlarge | 16384     | 160  | 0         |      | 8     | 1.0         | True      |
| c1 | cirros256 | 256       | 0    | 0         |      | 1     | 1.0         | True      |
| d1 | ds512M    | 512       | 5    | 0         |      | 1     | 1.0         | True      |
| d2 | ds1G      | 1024      | 10   | 0         |      | 1     | 1.0         | True      |
| d3 | ds2G      | 2048      | 10   | 0         |      | 2     | 1.0         | True      |
| d4 | ds4G      | 4096      | 20   | 0         |      | 4     | 1.0         | True      |
+----+-----------+-----------+------+-----------+------+-------+-------------+-----------+

Boot instance1 in RegionOne, and connect this instance to net1 and phy_net1.

$ nova --os-region-name=RegionOne boot --flavor 1 --image 924a5078-efe5-4abf-85e8-992b7e5f6ac3 --nic net-id=1897a446-bf6a-4bce-9374-6a3825ee5051 --nic net-id=b7832cbb-d399-4d5d-bcfd-d1b804506a1a instance1
+--------------------------------------+----------------------------------------------------------------+
| Property                             | Value                                                          |
+--------------------------------------+----------------------------------------------------------------+
| OS-DCF:diskConfig                    | MANUAL                                                         |
| OS-EXT-AZ:availability_zone          |                                                                |
| OS-EXT-SRV-ATTR:host                 | -                                                              |
| OS-EXT-SRV-ATTR:hostname             | instance1                                                      |
| OS-EXT-SRV-ATTR:hypervisor_hostname  | -                                                              |
| OS-EXT-SRV-ATTR:instance_name        |                                                                |
| OS-EXT-SRV-ATTR:kernel_id            | d3e8349d-d58d-4d17-b0ab-951c095fbbc4                           |
| OS-EXT-SRV-ATTR:launch_index         | 0                                                              |
| OS-EXT-SRV-ATTR:ramdisk_id           | c4cd7482-a145-4f26-9f41-a9ac17b9492c                           |
| OS-EXT-SRV-ATTR:reservation_id       | r-eeu5hjq7                                                     |
| OS-EXT-SRV-ATTR:root_device_name     | -                                                              |
| OS-EXT-SRV-ATTR:user_data            | -                                                              |
| OS-EXT-STS:power_state               | 0                                                              |
| OS-EXT-STS:task_state                | scheduling                                                     |
| OS-EXT-STS:vm_state                  | building                                                       |
| OS-SRV-USG:launched_at               | -                                                              |
| OS-SRV-USG:terminated_at             | -                                                              |
| accessIPv4                           |                                                                |
| accessIPv6                           |                                                                |
| adminPass                            | ZB3Ve3nPS66g                                                   |
| config_drive                         |                                                                |
| created                              | 2017-01-11T10:49:32Z                                           |
| description                          | -                                                              |
| flavor                               | m1.tiny (1)                                                    |
| hostId                               |                                                                |
| host_status                          |                                                                |
| id                                   | 5fd0f616-1077-46df-bebd-b8b53d09663c                           |
| image                                | cirros-0.3.4-x86_64-uec (924a5078-efe5-4abf-85e8-992b7e5f6ac3) |
| key_name                             | -                                                              |
| locked                               | False                                                          |
| metadata                             | {}                                                             |
| name                                 | instance1                                                      |
| os-extended-volumes:volumes_attached | []                                                             |
| progress                             | 0                                                              |
| security_groups                      | default                                                        |
| status                               | BUILD                                                          |
| tags                                 | []                                                             |
| tenant_id                            | ce444c8be6da447bb412db7d30cd7023                               |
| updated                              | 2017-01-11T10:49:33Z                                           |
| user_id                              | 66d7b31664a840939f7d3f2de5e717a9                               |
+--------------------------------------+----------------------------------------------------------------+

List available images in RegionTwo.

$ glance --os-region-name=RegionTwo image-list
+--------------------------------------+---------------------------------+
| ID                                   | Name                            |
+--------------------------------------+---------------------------------+
| 1da4303c-96bf-4714-a4dc-cbd5709eda29 | cirros-0.3.4-x86_64-uec         |
| fb35d578-a984-4807-8234-f0d0ca393e89 | cirros-0.3.4-x86_64-uec-kernel  |
| a615d6df-be63-4d5a-9a05-5cf7e23a438a | cirros-0.3.4-x86_64-uec-ramdisk |
+--------------------------------------+---------------------------------+

List available flavors in RegionTwo.

$ nova --os-region-name=RegionTwo flavor-list
+----+-----------+-----------+------+-----------+------+-------+-------------+-----------+
| ID | Name      | Memory_MB | Disk | Ephemeral | Swap | VCPUs | RXTX_Factor | Is_Public |
+----+-----------+-----------+------+-----------+------+-------+-------------+-----------+
| 1  | m1.tiny   | 512       | 1    | 0         |      | 1     | 1.0         | True      |
| 2  | m1.small  | 2048      | 20   | 0         |      | 1     | 1.0         | True      |
| 3  | m1.medium | 4096      | 40   | 0         |      | 2     | 1.0         | True      |
| 4  | m1.large  | 8192      | 80   | 0         |      | 4     | 1.0         | True      |
| 5  | m1.xlarge | 16384     | 160  | 0         |      | 8     | 1.0         | True      |
| c1 | cirros256 | 256       | 0    | 0         |      | 1     | 1.0         | True      |
| d1 | ds512M    | 512       | 5    | 0         |      | 1     | 1.0         | True      |
| d2 | ds1G      | 1024      | 10   | 0         |      | 1     | 1.0         | True      |
| d3 | ds2G      | 2048      | 10   | 0         |      | 2     | 1.0         | True      |
| d4 | ds4G      | 4096      | 20   | 0         |      | 4     | 1.0         | True      |
+----+-----------+-----------+------+-----------+------+-------+-------------+-----------+

Boot instance2 in RegionTwo, and connect this instance to net1 and phy_net2.

$ nova --os-region-name=RegionTwo boot --flavor 1 --image 1da4303c-96bf-4714-a4dc-cbd5709eda29 --nic net-id=1897a446-bf6a-4bce-9374-6a3825ee5051 --nic net-id=731293af-e68f-4677-b433-f46afd6431f3 instance2
+--------------------------------------+----------------------------------------------------------------+
| Property                             | Value                                                          |
+--------------------------------------+----------------------------------------------------------------+
| OS-DCF:diskConfig                    | MANUAL                                                         |
| OS-EXT-AZ:availability_zone          |                                                                |
| OS-EXT-SRV-ATTR:host                 | -                                                              |
| OS-EXT-SRV-ATTR:hostname             | instance2                                                      |
| OS-EXT-SRV-ATTR:hypervisor_hostname  | -                                                              |
| OS-EXT-SRV-ATTR:instance_name        |                                                                |
| OS-EXT-SRV-ATTR:kernel_id            | fb35d578-a984-4807-8234-f0d0ca393e89                           |
| OS-EXT-SRV-ATTR:launch_index         | 0                                                              |
| OS-EXT-SRV-ATTR:ramdisk_id           | a615d6df-be63-4d5a-9a05-5cf7e23a438a                           |
| OS-EXT-SRV-ATTR:reservation_id       | r-m0duhg40                                                     |
| OS-EXT-SRV-ATTR:root_device_name     | -                                                              |
| OS-EXT-SRV-ATTR:user_data            | -                                                              |
| OS-EXT-STS:power_state               | 0                                                              |
| OS-EXT-STS:task_state                | scheduling                                                     |
| OS-EXT-STS:vm_state                  | building                                                       |
| OS-SRV-USG:launched_at               | -                                                              |
| OS-SRV-USG:terminated_at             | -                                                              |
| accessIPv4                           |                                                                |
| accessIPv6                           |                                                                |
| adminPass                            | M5FodqwcsTiJ                                                   |
| config_drive                         |                                                                |
| created                              | 2017-01-11T12:55:35Z                                           |
| description                          | -                                                              |
| flavor                               | m1.tiny (1)                                                    |
| hostId                               |                                                                |
| host_status                          |                                                                |
| id                                   | 010a0a24-0453-4e73-ae8d-21c7275a9df5                           |
| image                                | cirros-0.3.4-x86_64-uec (1da4303c-96bf-4714-a4dc-cbd5709eda29) |
| key_name                             | -                                                              |
| locked                               | False                                                          |
| metadata                             | {}                                                             |
| name                                 | instance2                                                      |
| os-extended-volumes:volumes_attached | []                                                             |
| progress                             | 0                                                              |
| security_groups                      | default                                                        |
| status                               | BUILD                                                          |
| tags                                 | []                                                             |
| tenant_id                            | ce444c8be6da447bb412db7d30cd7023                               |
| updated                              | 2017-01-11T12:55:35Z                                           |
| user_id                              | 66d7b31664a840939f7d3f2de5e717a9                               |
+--------------------------------------+----------------------------------------------------------------+

Make sure the instance1 is active in RegionOne.

$ nova --os-region-name=RegionOne list
+--------------------------------------+-----------+--------+------------+-------------+-------------------------------------+
| ID                                   | Name      | Status | Task State | Power State | Networks                            |
+--------------------------------------+-----------+--------+------------+-------------+-------------------------------------+
| 5fd0f616-1077-46df-bebd-b8b53d09663c | instance1 | ACTIVE | -          | Running     | net1=10.0.1.4; phy_net1=202.96.1.13 |
+--------------------------------------+-----------+--------+------------+-------------+-------------------------------------+

Make sure the instance2 is active in RegionTwo.

$ nova --os-region-name=RegionTwo list
+--------------------------------------+-----------+--------+------------+-------------+------------------------------------+
| ID                                   | Name      | Status | Task State | Power State | Networks                           |
+--------------------------------------+-----------+--------+------------+-------------+------------------------------------+
| 010a0a24-0453-4e73-ae8d-21c7275a9df5 | instance2 | ACTIVE | -          | Running     | phy_net2=202.96.2.5; net1=10.0.1.5 |
+--------------------------------------+-----------+--------+------------+-------------+------------------------------------+

Now you can ping instance2’s IP address 10.0.1.5 from instance1, or ping instance1’s IP address 10.0.1.4 from instance2.

Note: Not all images will bring up the second nic, so you can ssh into instance1 or instance2, use ifconfig -a to check whether all NICs are created, and bring up all NICs if necessary.