Troubleshoot your nova-volume installation

This section will help if you are able to successfully create volumes with either Cinder or Nova-Volume, however you can't attach them to an instance. If you are having trouble creating volumes, go to the cinder troubleshootingsection.

If the volume attachment doesn't work, you should be able to perform different checks in order to see where the issue is. The nova-volume.log and nova-compute.log will help you to diagnosis the errors you could encounter:

nova-compute.log / nova-volume.log

ERROR "Cannot resolve host"

(nova.root): TRACE: ProcessExecutionError: Unexpected error while running command.
(nova.root): TRACE: Command: sudo iscsiadm -m discovery -t sendtargets -p ubuntu03c
(nova.root): TRACE: Exit code: 255
(nova.root): TRACE: Stdout: ''
(nova.root): TRACE: Stderr: 'iscsiadm: Cannot resolve host ubuntu03c. getaddrinfo error: [Name or service not known]\n\niscsiadm:
cannot resolve host name ubuntu03c\niscsiadm: Could not perform SendTargets discovery.\n'
(nova.root): TRACE:

This error happens when the compute node is unable to resolve the nova-volume server name. You could either add a record for the server if you have a DNS server; or add it into the /etc/hosts file of the nova-compute.

ERROR "No route to host"
```
iscsiadm: cannot make connection to 172.29.200.37: No route to host\niscsiadm: cannot make connection to 172.29.200.37
                            
```
This error could be caused by several things, but it means only one thing : openiscsi is unable to establish a communication with your nova-volumes server.
The first thing you could do is running a telnet session in order to see if you are able to reach the nova-volume server. From the compute-node, run :
```
$ telnet $ip_of_nova_volumes  3260
                        
```
If the session times out, check the server firewall; or try to ping it. You could also run a tcpdump session which may also provide extra information:
```
$ tcpdump -nvv -i $iscsi_interface port dest $ip_of_nova_volumes
                        
```
Again, try to manually run an iSCSI discovery via:
```
$ iscsiadm -m discovery -t st -p $ip_of_nova-volumes
                        
```

"Lost connectivity between nova-volumes and node-compute ; how to restore a clean state ?"

Network disconnection can happens, from an "iSCSI view", losing connectivity could be seen as a physical removal of a server's disk. If the instance runs a volume while you loose the network between them, you won't be able to detach the volume. You would encounter several errors. Here is how you could clean this :

First, from the nova-compute, close the active (but stalled) iSCSI session, refer to the volume attached to get the session, and perform the following command :

$ iscsiadm -m session -r $session_id -u

Here is an iscsi -m session output:

tcp: [1] 172.16.40.244:3260,1 iqn.2010-10.org.openstack:volume-1
tcp: [2] 172.16.40.244:3260,1 iqn.2010-10.org.openstack:volume-2
tcp: [3] 172.16.40.244:3260,1 iqn.2010-10.org.openstack:volume-3
tcp: [4] 172.16.40.244:3260,1 iqn.2010-10.org.openstack:volume-4
tcp: [5] 172.16.40.244:3260,1 iqn.2010-10.org.openstack:volume-5
tcp: [6] 172.16.40.244:3260,1 iqn.2010-10.org.openstack:volume-6
tcp: [7] 172.16.40.244:3260,1 iqn.2010-10.org.openstack:volume-7
tcp: [9] 172.16.40.244:3260,1 iqn.2010-10.org.openstack:volume-9

For example, to free volume 9, close the session number 9.

The cloud-controller is actually unaware of the iSCSI session closing, and will keeps the volume state as in-use:

                            +----+-----------+--------------+------+-------------+--------------------------------------+
                            | ID |   Status  | Display Name | Size | Volume Type |             Attached to              |
                            +----+-----------+--------------+------+-------------+--------------------------------------+
                            | 9  | in-use    | New Volume   | 20   | None        | 7db4cb64-7f8f-42e3-9f58-e59c9a31827d |

You now have to inform the cloud-controller that the disk can be used. Nova stores the volumes info into the "volumes" table. You will have to update four fields into the database nova uses (eg. MySQL). First, connect to the database:

$ mysql -uroot -p$password nova

Using the volume id, you will have to run the following sql queries:

mysql> update volumes set mountpoint=NULL where id=9;
mysql> update volumes set status="available" where status "error_deleting" where id=9;
mysql> update volumes set attach_status="detached" where id=9;
mysql> update volumes set instance_id=0 where id=9;

Now if you run again nova volume-list from the cloud controller, you should see an available volume now:

                            +----+-----------+--------------+------+-------------+--------------------------------------+
                            | ID |   Status  | Display Name | Size | Volume Type |             Attached to              |
                            +----+-----------+--------------+------+-------------+--------------------------------------+
                            | 9  | available  | New Volume   | 20   | None       |                                      |

You can now proceed to the volume attachment again!

Legal notices