If a hardware malfunction or other error causes a cloud compute node to fail, you can evacuate instances to make them available again.
To preserve user data on the server disk, configure shared storage on the target host. When you evacuate the instance, Compute detects whether shared storage is available on the target host. Also, you must validate that the current VM host is not operational. Otherwise, the evacuation fails.
There are two different ways to evacuate instances from a failed compute node. The first one using the nova evacuate command can be used to evacuate a single instance from a failed node. In some cases where the node in question hosted many instances it might be easier to use nova host-evacuate to evacuate them all in one shot.
Evacuate a single instance¶
The procedure below explains how to evacuate a single instance from a failed compute node. Please be aware that these steps describe a post failure scenario and should not be used if the instance is still up and running.
To find a host for the evacuated instance, list all hosts:
$ openstack host list
Evacuate the instance. You can use the
--password PWDoption to pass the instance password to the command. If you do not specify a password, the command generates and prints one after it finishes successfully. The following command evacuates a server from a failed host to
$ nova evacuate EVACUATED_SERVER_NAME HOST_B
The command rebuilds the instance from the original image or volume and returns a password. The command preserves the original configuration, which includes the instance ID, name, uid, IP address, and so on.
+-----------+--------------+ | Property | Value | +-----------+--------------+ | adminPass | kRAJpErnT4xZ | +-----------+--------------+
Optionally you can omit the
HOST_Bparameter and let the scheduler choose a new target host.
To preserve the user disk data on the evacuated server, deploy Compute with a shared file system. To configure your system, see Configure live migrations. The following example does not change the password.
$ nova evacuate EVACUATED_SERVER_NAME HOST_B --on-shared-storage
Starting with the 2.14 compute API version, one no longer needs to specify
--on-shared-storageeven if the server is on a compute host which is using shared storage. The compute service will automatically detect if it is running on shared storage.
Evacuate all instances¶
The procedure below explains how to evacuate all instances from a failed compute node. Please note that this method should not be used if the host still has instances up and running.
To find a host for the evacuated instances, list all hosts:
$ openstack host list
Evacuate all instances from
$ nova host-evacuate --target_host TARGET_HOST FAILED_HOST
--target_hostis optional and can be omitted to let the scheduler decide where to place the instances.
The above argument
FAILED_HOSTcan also be a pattern to search for instead of an exact hypervisor hostname but it is recommended to use a fully qualified domain name to make sure no hypervisor host is getting evacuated by mistake. As long as you are not using a pattern you might want to use the
--strictflag which got introduced in version 10.2.0 to make sure nova matches the
+------+--------+--------------+ | Name | Status | Task State | +------+--------+--------------+ | vm_1 | ACTIVE | powering-off | +------------------------------+
If the instance task state is not None, evacuation will be possible. However, depending on the ongoing operation, there may be clean up required in other services which the instance was using, such as neutron, cinder, glance, or the storage backend.