OpenStack on LXD

Overview

A production OpenStack deployment is typically made over a number of physical servers, using LXD containers where appropriate for control plane services.

However, the average developer probably does not have, or want to have, access to such infrastructure for day-to-day charm development.

Its possible to deploy OpenStack using the OpenStack Charms in LXD containers on a single machine; this allows for faster, localized charm development and testing.

The scenarios presented here are for development, testing and demonstration, and should not be used in place of multiple-machine, highly-available OpenStack deployment topologies and configurations more suitable for production use cases.

Host Setup

The tools in the openstack-on-lxd git repository require the use of Juju 2.x, which provides full support for the LXD local provider. The Queens version of the OpenStack clients should be used with this procedure. These tools are provided as part of the Ubuntu Cloud Archive on Ubuntu 16.04 LTS.

sudo snap install lxd
sudo snap install juju --classic

sudo add-apt-repository cloud-archive:queens -y && sudo apt update

sudo apt install zfsutils-linux squid-deb-proxy bridge-utils \
    python-novaclient python-keystoneclient python-glanceclient \
    python-neutronclient python-openstackclient curl

You’ll need a well specified machine with at least 8G of RAM and a SSD; for reference the author uses Lenovo x240 with an Intel i5 processor, 16G RAM and a 500G Samsung SSD (split into two - one partition for the OS and one partition for a ZFS pool).

For s390x, this has been validated on an LPAR with 12 CPUs, 40GB RAM, 2 ~40GB disks (one disk for the OS and one disk for the ZFS pool).

For arm64, this has been validated on a single Gigabyte R120-T33 with 48 cores, 64GB RAM, 1 240GB SSD (for the operating system), 2 1TB SAS disks (for the ZFS pool).

You’ll need to clone the repository with the bundles and configuration for the deployment:

git clone https://github.com/openstack-charmers/openstack-on-lxd

All commands in this document assume they are being made from within the local copy of this repo.

LXD

Base Configuration

This type of deployment creates numerous containers on a single host which leads to many thousands of file handles.

Some of the default system thresholds may not be high enough for this use case, potentially leading to issues such as Too many open files.

To address this, the host system should be configured according to the LXD production-setup guide, specifically the /etc/sysctl.conf bits:

echo fs.inotify.max_queued_events=1048576 | sudo tee -a /etc/sysctl.conf
echo fs.inotify.max_user_instances=1048576 | sudo tee -a /etc/sysctl.conf
echo fs.inotify.max_user_watches=1048576 | sudo tee -a /etc/sysctl.conf
echo vm.max_map_count=262144 | sudo tee -a /etc/sysctl.conf
echo vm.swappiness=1 | sudo tee -a /etc/sysctl.conf
sudo sysctl -p

In order to allow the OpenStack Cloud to function, you’ll need to reconfigure the default LXD bridge to support IPv4 networking; is also recommended that you use a fast storage backend such as ZFS on a SSD based block device. Use the lxd provided configuration tool to help do this:

sudo lxd init

The referenced config.yaml uses an apt proxy to improve installation performance. The network that you create during the lxd init procedure should accomodate that address. Also ensure that you leave a range of IP addresses free to use for floating IP addresses for OpenStack instances; The following are the values which are used in this example procedure:

Network and IP: 10.0.8.1/24 DHCP range: 10.0.8.2 -> 10.0.8.200

Also update the default profile to use Jumbo frames for all network connections into containers:

lxc profile device set default eth0 mtu 9000

This will ensure you avoid any packet fragmentation type problems with overlay networks.

Test out your configuration prior to launching an entire cloud:

lxc launch ubuntu-daily:xenial

This should result in a running container you can exec into and back out of:

lxc exec <container-name> bash
exit

Juju

Bootstrap the Juju Controller

Prior to deploying the OpenStack on LXD bundle, you’ll need to bootstrap a controller to manage your Juju models:

juju bootstrap --config config.yaml localhost lxd

Review the contents of the config.yaml prior to running this command and edit as appropriate; this configures some defaults for containers created in the model including setting up things like APT proxy to improve performance of network operations.

Juju Profile Update

Juju creates a couple of profiles for the models that it creates by default. After bootstrapping is complete, update the juju-default lxc profile:

cat lxd-profile.yaml | lxc profile edit juju-default

This will ensure that containers created by LXD for Juju have the correct permissions to run your OpenStack cloud.

Configure a PowerNV (ppc64el) Host

When deployed directly to metal, the nova-compute charm sets smt=off, as is necessary for libvirt usage. However, when nova-compute is in a container, the containment prevents ppc64_cpu from modifying the host’s smt value. It is necessary to pre-configure the host smt setting for nova-compute (libvirt + qemu) in ppc64el scenarios.

sudo ppc64_cpu --smt=off

OpenStack

Deploy

Next, deploy the OpenStack cloud using the provided bundle.

For amd64, arm64, or ppc64el Mitaka:

juju deploy bundle-xenial-mitaka.yaml

For amd64, arm64, or ppc64el Newton:

juju deploy bundle-xenial-newton.yaml

For amd64, arm64, or ppc64el Ocata:

juju deploy bundle-xenial-ocata.yaml

For amd64, arm64, or ppc64el Pike:

juju deploy bundle-xenial-pike.yaml

For amd64, arm64, or ppc64el Queens on Xenial:

juju deploy bundle-xenial-queens.yaml

For amd64, arm64, or ppc64el Queens on Bionic:

juju deploy bundle-bionic-queens.yaml

For s390x Mitaka:

juju deploy bundle-xenial-mitaka-s390x.yaml

For s390x Newton:

juju deploy bundle-xenial-newton-s390x.yaml

For s390x Ocata:

juju deploy bundle-xenial-ocata-s390x.yaml

For s390x Pike:

juju deploy bundle-xenial-pike-s390x.yaml

For s390x Queens on Xenial:

juju deploy bundle-xenial-queens-s390x.yaml

For s390x Queens on Bionic:

juju deploy bundle-bionic-queens-s390x.yaml

You can watch deployment progress using the ‘juju status’ command. This may take some time depending on the speed of your system; CPU, disk and network speed will all effect deployment time.

Using the Cloud

Check Access

Once deployment has completed (units should report a ready state in the status output), check that you can access the deployed cloud OK:

Note: While Keystone V3 can be enabled earlier than Queens, the example bundles provided herein use the default version of the Keystone API as shipped upstream. As of Queens, that default is V3, and prior to Queens the default was V2.
source openrc
openstack catalog list
openstack service list
openstack network agent list
openstack volume service list

The openstack client commands should all succeed and you should get a feel as to how the various OpenStack components are deployed in each container.

Upload an image

Before we can boot an instance, we need an image to boot in Glance.

Note: If you are using a ZFS backend for this deployment, force-raw-images must be disabled on the nova-compute charm in Pike and later. We have made this the default in our bundles - however, be aware that using this setting in a production environment is discouraged as it may have an impact on performance.

For amd64:

curl https://cloud-images.ubuntu.com/xenial/current/xenial-server-cloudimg-amd64-disk1.img | \
    openstack image create --public --container-format=bare --disk-format=qcow2 xenial

For arm64:

curl https://cloud-images.ubuntu.com/xenial/current/xenial-server-cloudimg-arm64-uefi1.img | \
    openstack image create --public --container-format=bare --disk-format=qcow2 --property hw_firmware_type=uefi xenial

For s390x:

curl https://cloud-images.ubuntu.com/xenial/current/xenial-server-cloudimg-s390x-disk1.img | \
    openstack image create --public --container-format=bare --disk-format=qcow2 xenial

For ppc64el:

curl https://cloud-images.ubuntu.com/xenial/current/xenial-server-cloudimg-ppc64el-disk1.img | \
    openstack image create --public --container-format=bare --disk-format=qcow2 xenial

Configure the network - Queens and Later

First, create the ‘external’ network which actually maps directly to the LXD bridge:

./neutron-ext-net-ksv3 --network-type flat \
    -g 10.0.8.1 -c 10.0.8.0/24 \
    -f 10.0.8.201:10.0.8.254 ext_net

and then create an internal overlay network for the instances to actually attach to:

./neutron-tenant-net-ksv3 -p admin -r provider-router \
    -N 10.0.8.1 internal 192.168.20.0/24

Configure the network - Pike and Earlier

First, create the ‘external’ network which actually maps directly to the LXD bridge:

./neutron-ext-net --network-type flat \
    -g 10.0.8.1 -c 10.0.8.0/24 \
    -f 10.0.8.201:10.0.8.254 ext_net

and then create an internal overlay network for the instances to actually attach to:

./neutron-tenant-net -t admin -r provider-router \
    -N 10.0.8.1 internal 192.168.20.0/24

Create a key-pair

Upload your local public key into the cloud so you can access instances:

openstack keypair create --public-key ~/.ssh/id_rsa.pub mykey

Create Flavors

It’s safe to skip this for Mitaka. For Newton and later, there are no pre-populated flavors. Check if flavors exist, and if not, create them:

openstack flavor list
openstack flavor create --public --ram 512 --disk 1 --ephemeral 0 --vcpus 1 m1.tiny
openstack flavor create --public --ram 1024 --disk 20 --ephemeral 40 --vcpus 1 m1.small
openstack flavor create --public --ram 2048 --disk 40 --ephemeral 40 --vcpus 2 m1.medium
openstack flavor create --public --ram 8192 --disk 40 --ephemeral 40 --vcpus 4 m1.large
openstack flavor create --public --ram 16384 --disk 80 --ephemeral 40 --vcpus 8 m1.xlarge

Boot an instance

You can now boot an instance on your cloud:

openstack server create --image xenial --flavor m1.small --key-name mykey \
   --wait --nic net-id=$(openstack network list | grep internal | awk '{ print $2 }') \
   openstack-on-lxd-ftw

Attaching a volume

First, create a volume in cinder:

openstack volume create --size 10 testvolume

then attach it to the instance we just booted in nova:

openstack server add volume openstack-on-lxd-ftw testvolume
openstack volume show testvolume

The attached volume will be accessible once you login to the instance (see below). It will need to be formatted and mounted!

Accessing your instance

In order to access the instance you just booted on the cloud, you’ll need to assign a floating IP address to the instance:

openstack floating ip create ext_net
openstack server add floating ip <uuid-of-instance> <new-floating-ip>

Permit SSH and ping quite liberally, by allowing both on all default security groups. This only needs to be done once on the deployed cloud:

for i in $(openstack security group list | awk '/default/{ print $2 }'); do \
    openstack security group rule create $i --protocol icmp --remote-ip 0.0.0.0/0; \
    openstack security group rule create $i --protocol tcp --remote-ip 0.0.0.0/0 --dst-port 22; \
done

After running these commands you should be able to access the instance from the lxd host:

ssh ubuntu@<new-floating-ip>

Access the GUIs

The method of accessing the GUI IP addresses varies, depending on where the cloud was deployed. The “public” addresses of the deployed cloud are actually 10.0.8.x addresses, sitting behind NAT.

OpenStack Dashboard

First, find the IP address of the openstack-dashboard unit by querying juju status:

juju status openstack-dashboard

Then, choose your adventure:

OpenStack-on-LXD is deployed on your local machine

The IP address of the OpenStack Dashboard will be locally accessible.

Adjust and visit the following URL from a browser your local machine:

http://<ip address of openstack-dashboard>/horizon

domain:  admin_domain
user:  admin
password:  ??????????

OpenStack-on-LXD is deployed on a remote machine

The IP address of the GUI will not be directly accessible. You can forward a tcp port across an existing ssh session, then access the dashboard on your localhost.

In your SSH session, press the ~C key combo to initiate an SSH command console on-the-fly. Adjust and issue the following command to forward localhost:10080 to openstack-dashboard:80 across that existing SSH session:

-L 10080:<ip address of openstack-dashboard>:80

Then visit the following URL from a browser on your local machine:

domain:  admin_domain
user:  admin
password:  ??????????

Juju GUI

First, find the IP address and credentials for the Juju GUI:

juju gui

Then, choose your adventure:

OpenStack-on-LXD is deployed on your local machine

The IP address of the Juju GUI will be locally accessible.

The URL provided should work directly, and it should look something like:

https://10.0.8.x:17070/gui/u/admin/default

OpenStack-on-LXD is deployed on a remote machine

The IP address of the Juju GUI will not be directly accessible. You can forward a tcp port across an existing ssh session, then access the Juju GUI on your localhost.

In your SSH session, press the ~C key combo to initiate an SSH command console on-the-fly. Adjust and issue the following command to forward localhost:10070 to juju-gui:17070 across that existing SSH session:

-L 10070:<ip address of juju-gui>:17070

Then visit the following URL from a browser on your local machine:

Switching in a dev charm

Now that you have a running OpenStack deployment on your machine, you can switch in your development changes to one of the charms in the deployment:

juju upgrade-charm --switch <path-to-your-charm> cinder

The charm will be upgraded with your local development changes; alternatively you can update the bundle.yaml to reference your local charm so that its used from the start of cloud deployment.

Known Limitations

Currently is not possible to run Cinder with iSCSI/LVM based storage under LXD; this limits use of block storage options to those that are 100% userspace, such as Ceph.