Managing Volumes

Configure Volumes for use with nova-volume

If you do not already have LVM volumes on hand, but have free drive space, you will need to create a LVM volume before proceeding. Here is a short run down of how you would create a LVM from free drive space on your system. Start off by issuing an fdisk command to your drive with the free space:

$ fdisk /dev/sda
                            

Once in fdisk, perform the following commands:

Installing and configuring the iSCSI initiator

Remember that every node will act as the iSCSI initiator while the server running nova-volumes will act as the iSCSI target. So make sure, before going further that your nodes can communicate with you nova-volumes server. If you have a firewall running on it, make sure that the port 3260 (tcp) accepts incoming connections.

First install the open-iscsi package on the initiators, so on the compute-nodes only

$ apt-get install open-iscsi
                    

Then on the target, which is in our case the cloud-controller, the iscsitarget package :

$ apt-get install iscsitarget
                        

This package could refuse to start with a "FATAL: Module iscsi_trgt not found" error.

This error is caused by the kernel which does not contain the iscsi module's source into it ; you can install the kernel modules by installing an extra package :

$ apt-get install iscsitarget-dkms
                        

(the Dynamic Kernel Module Support is a framework used for created modules with non-existent sources into the current kernel)

You have to enable it so the startut script (/etc/init.d/iscsitarget) can start the daemon:

$ sed -i 's/false/true/g' /etc/default/iscsitarget
                        

Then run on the nova-controller (iscsi target) :

$ service iscsitarget start
                            

And on the compute-nodes (iscsi initiators) :

$ service open-iscsi start
                            

Start nova-volume and create volumes

You are now ready to fire up nova-volume, and start creating volumes!

$ service nova-volume start

Once the service is started, login to your controller and ensure you’ve properly sourced your ‘novarc’ file.

One of the first things you should do is make sure that nova-volume is checking in as expected. You can do so using nova-manage:

$ nova-manage service list
                        

If you see a smiling ‘nova-volume’ in there, you are looking good. Now create a new volume:

$ nova volume-create --display_name myvolume 10
                        

--display_name sets a readable name for the volume, while the final argument refers to the size of the volume in GB.

You should get some output similar to this:

                        +----+-----------+--------------+------+-------------+--------------------------------------+
                        | ID |   Status  | Display Name | Size | Volume Type |             Attached to              |
                        +----+-----------+--------------+------+-------------+--------------------------------------+
                        | 1  | available | myvolume     | 10   | None        |                                      |
                        +----+-----------+--------------+------+-------------+--------------------------------------+
                        

You can view that status of the volumes creation using nova volume-list. Once that status is ‘available,’ it is ready to be attached to an instance:

$ nova volume-attach 857d70e4-35d5-4bf6-97ed-bf4e9a4dcf5a 1 /dev/vdb
                    

The first argument refers to the instance you will attach the volume to; The second is the volume ID; The third is the mountpoint on the compute-node that the volume will be attached to

By doing that, the compute-node which runs the instance basically performs an iSCSI connection and creates a session. You can ensure that the session has been created by running :

$ iscsiadm -m session
                    

Which should output :

root@nova-cn1:~# iscsiadm -m session
tcp: [1] 172.16.40.244:3260,1 iqn.2010-10.org.openstack:volume-1

If you do not get any errors, you can login to the instance and see if the new space is there.

KVM changes the device name, since it's not considered to be the same type of device as the instances uses as it's local one, you will find the nova-volume will be designated as "/dev/vdX" devices, while local are named "/dev/sdX".

You can check the volume attachment by running :

$ dmesg | tail
                    

You should from there see a new disk. Here is the output from fdisk -l:

Disk /dev/vda: 10.7 GB, 10737418240 bytes
16 heads, 63 sectors/track, 20805 cylinders
Units = cylinders of 1008 * 512 = 516096 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0×00000000
Disk /dev/vda doesn’t contain a valid partition table
Disk /dev/vdb: 21.5 GB, 21474836480 bytes <—–Here is our new volume!
16 heads, 63 sectors/track, 41610 cylinders
Units = cylinders of 1008 * 512 = 516096 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0×00000000
                    

Now with the space presented, let’s configure it for use:

$ fdisk /dev/vdb
                        

Your new volume has now been successfully mounted, and is ready for use! The commands are pretty self-explanatory, so play around with them and create new volumes, tear them down, attach and reattach, and so on.

ERROR "Cannot resolve host"

(nova.root): TRACE: ProcessExecutionError: Unexpected error while running command.
(nova.root): TRACE: Command: sudo iscsiadm -m discovery -t sendtargets -p ubuntu03c
(nova.root): TRACE: Exit code: 255
(nova.root): TRACE: Stdout: ''
(nova.root): TRACE: Stderr: 'iscsiadm: Cannot resolve host ubuntu03c. getaddrinfo error: [Name or service not known]\n\niscsiadm:
cannot resolve host name ubuntu03c\niscsiadm: Could not perform SendTargets discovery.\n'
(nova.root): TRACE:
                            

This error happens when the compute node is unable to resolve the nova-volume server name. You could either add a record for the server if you have a DNS server; or add it into the /etc/hosts file of the nova-compute.

ERROR "No route to host"

iscsiadm: cannot make connection to 172.29.200.37: No route to host\niscsiadm: cannot make connection to 172.29.200.37
                            

This error could be caused by several things, but it means only one thing : openiscsi is unable to establish a communication with your nova-volumes server.

The first thing you could do is running a telnet session in order to see if you are able to reach the nova-volume server. From the compute-node, run :

$ telnet $ip_of_nova_volumes  3260
                        

If the session times out, check the server firewall ; or try to ping it. You could also run a tcpdump session which may also provide extra information :

$ tcpdump -nvv -i $iscsi_interface port dest $ip_of_nova_volumes
                        

Again, try to manually run an iSCSI discovery via :

$ iscsiadm -m discovery -t st -p $ip_of_nova-volumes
                        

"Lost connectivity between nova-volumes and node-compute ; how to restore a clean state ?"

Network disconnection can happens, from an "iSCSI view", losing connectivity could be seen as a physical removal of a server's disk. If the instance runs a volume while you loose the network between them, you won't be able to detach the volume. You would encounter several errors. Here is how you could clean this :

First, from the nova-compute, close the active (but stalled) iSCSI session, refer to the volume attached to get the session, and perform the following command :

$ iscsiadm -m session -r $session_id -u
                        

Here is an iscsi -m session output :

tcp: [1] 172.16.40.244:3260,1 iqn.2010-10.org.openstack:volume-1
tcp: [2] 172.16.40.244:3260,1 iqn.2010-10.org.openstack:volume-2
tcp: [3] 172.16.40.244:3260,1 iqn.2010-10.org.openstack:volume-3
tcp: [4] 172.16.40.244:3260,1 iqn.2010-10.org.openstack:volume-4
tcp: [5] 172.16.40.244:3260,1 iqn.2010-10.org.openstack:volume-5
tcp: [6] 172.16.40.244:3260,1 iqn.2010-10.org.openstack:volume-6
tcp: [7] 172.16.40.244:3260,1 iqn.2010-10.org.openstack:volume-7
tcp: [9] 172.16.40.244:3260,1 iqn.2010-10.org.openstack:volume-9
                        

For example, to free volume 9, close the session number 9.

The cloud-controller is actually unaware of the iSCSI session closing, and will keeps the volume state as in-use:

                            +----+-----------+--------------+------+-------------+--------------------------------------+
                            | ID |   Status  | Display Name | Size | Volume Type |             Attached to              |
                            +----+-----------+--------------+------+-------------+--------------------------------------+
                            | 9  | in-use    | New Volume   | 20   | None        | 7db4cb64-7f8f-42e3-9f58-e59c9a31827d |
                            

You now have to inform the cloud-controller that the disk can be used. Nova stores the volumes info into the "volumes" table. You will have to update four fields into the database nova uses (eg. MySQL). First, conect to the database :

$ mysql -uroot -p$password nova
                        

Using the volume id, you will have to run the following sql queries

mysql> update volumes set mountpoint=NULL where id=9;
mysql> update volumes set status="available" where status "error_deleting" where id=9;
mysql> update volumes set attach_status="detached" where id=9;
mysql> update volumes set instance_id=0 where id=9;
                        

Now if you run again nova volume-list from the cloud controller, you should see an available volume now :

                            +----+-----------+--------------+------+-------------+--------------------------------------+
                            | ID |   Status  | Display Name | Size | Volume Type |             Attached to              |
                            +----+-----------+--------------+------+-------------+--------------------------------------+
                            | 9  | available  | New Volume   | 20   | None       |                                      |
                        

You can now proceed to the volume attachment again!

In order to backup our volume, we first need to create a snapshot of it. An LVM snapshot is the exact copy of a logical volume, which contains data in a frozen state. This prevents data corruption, because data will not be manipulated during the process of creating the volume itself. Remember the volumes created through a nova volume-create exist in an LVM's logical volume.

Before creating the snapshot, ensure that you have enough space to save it. As a precaution, you should have at least twice as much space as the potential snapshot size. If insufficient space is available, there is a risk that the snapshot could become corrupted.

Use the following command to obtain a list of all volumes.

$ lvdisplay
                        

In this example, we will refer to a volume called volume-00000001, which is a 10GB volume. This process can be applied to all volumes, not matter their size. At the end of the section, we will present a script that you could use to create scheduled backups. The script itself exploits what we discuss here.

First, create the snapshot; this can be achieved while the volume is attached to an instance :

$ lvcreate --size 10G --snapshot --name volume-00000001-snapshot /dev/nova-volumes/volume-00000001
                        

We indicate to LVM we want a snapshot of an already existing volume with the --snapshot configuration option. The command includes the size of the space reserved for the snapshot volume, the name of the snapshot, and the path of an already existing volume (In most cases, the path will be /dev/nova-volumes/$volume_name).

The size doesn't have to be the same as the volume of the snapshot. The size parameter designates the space that LVM will reserve for the snapshot volume. As a precaution, the size should be the same as that of the original volume, even if we know the whole space is not currently used by the snapshot.

We now have a full snapshot, and it only took few seconds !

Run lvdisplay again to verify the snapshot. You should see now your snapshot :

                  --- Logical volume ---
  LV Name                /dev/nova-volumes/volume-00000001
  VG Name                nova-volumes
  LV UUID                gI8hta-p21U-IW2q-hRN1-nTzN-UC2G-dKbdKr
  LV Write Access        read/write
  LV snapshot status     source of
                         /dev/nova-volumes/volume-00000026-snap [active]
  LV Status              available
  # open                 1
  LV Size                15,00 GiB
  Current LE             3840
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           251:13

  --- Logical volume ---
  LV Name                /dev/nova-volumes/volume-00000001-snap
  VG Name                nova-volumes
  LV UUID                HlW3Ep-g5I8-KGQb-IRvi-IRYU-lIKe-wE9zYr
  LV Write Access        read/write
  LV snapshot status     active destination for /dev/nova-volumes/volume-00000026
  LV Status              available
  # open                 0
  LV Size                15,00 GiB
  Current LE             3840
  COW-table size         10,00 GiB
  COW-table LE           2560
  Allocated to snapshot  0,00%
  Snapshot chunk size    4,00 KiB
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           251:14
            

If we want to exploit that snapshot with the tar program, we first need to mount our partition on the nova-volumes server.

kpartx is a small utility which performs table partition discoveries, and maps it. It can be used to view partitions created inside the instance. Without using the partitions created inside instances, we won' t be able to see its content and create efficient backups.

$ kpartx -av /dev/nova-volumes/volume-00000001-snapshot
                        

If no errors are displayed, it means the tools has been able to find it, and map the partition table. Note that on a Debian flavor distro, you could also use apt-get install kpartx.

You can easily check the partition table map by running the following command:

$ ls /dev/mapper/nova*
                    

You should now see a partition called nova--volumes-volume--00000001--snapshot1

If you created more than one partition on that volumes, you should have accordingly several partitions; for example. nova--volumes-volume--00000001--snapshot2, nova--volumes-volume--00000001--snapshot3 and so forth.

We can now mount our partition :

$ mount /dev/mapper/nova--volumes-volume--volume--00000001--snapshot1 /mnt
                        

If there are no errors, you have successfully mounted the partition.

You should now be able to directly access the data that were created inside the instance. If you receive a message asking you to specify a partition, or if you are unable to mount it (despite a well-specified filesystem) there could be two causes :

Allocate more space to the snapshot and try the process again.

Now that the volume has been mounted, you can create a backup of it :

$ tar --exclude={"lost+found","some/data/to/exclude"} -czf volume-00000001.tar.gz -C /mnt/ /backup/destination
                            

This command will create a tar.gz file containing the data, and data only. This ensures that you do not waste space by backing up empty sectors.

You should always have the checksum for your backup files. The checksum is a unique identifier for a file.

When you transfer that same file over the network, you can run another checksum calculation. If the checksums are different, this indicates that the file is corrupted; thus, the checksum provides a method to ensure your file has not been corrupted during its transfer.

The following command runs a checksum for our file, and saves the result to a file :

$ sha1sum volume-00000001.tar.gz > volume-00000001.checksum
                        

Be aware the sha1sum should be used carefully, since the required time for the calculation is directly proportional to the file's size.

For files larger than ~4-6 gigabytes, and depending on your CPU, the process may take a long time.

Now that we have an efficient and consistent backup, the following commands will clean up the file system.

And voila :) You can now repeat these steps for every volume you have.