Dell EMC VMAX iSCSI and FC drivers¶

Masking view names¶

Masking views are dynamically created by the VMAX FC and iSCSI drivers using the following naming conventions. [protocol] is either I for volumes attached over iSCSI or F for volumes attached over Fiber Channel.

Initiator group names¶

For each host that is attached to VMAX volumes using the drivers, an initiator group is created or re-used (per attachment type). All initiators of the appropriate type known for that host are included in the group. At each new attach volume operation, the VMAX driver retrieves the initiators (either WWNNs or IQNs) from OpenStack and adds or updates the contents of the Initiator Group as required. Names are of the following format. [protocol] is either I for volumes attached over iSCSI or F for volumes attached over Fiber Channel.

OS-[shortHostName]-[protocol]-IG

FA port groups¶

VMAX array FA ports to be used in a new masking view are retrieved from the port group provided as the extra spec on the volume type, or chosen from the list provided in the Dell EMC configuration file.

Storage group names¶

As volumes are attached to a host, they are either added to an existing storage group (if it exists) or a new storage group is created and the volume is then added. Storage groups contain volumes created from a pool, attached to a single host, over a single connection type (iSCSI or FC). [protocol] is either I for volumes attached over iSCSI or F for volumes attached over Fiber Channel. VMAX cinder driver utilizes cascaded storage groups - a parent storage group which is associated with the masking view, which contains child storage groups for each configured SRP/slo/workload/compression-enabled or disabled/replication-enabled or disabled combination.

VMAX All Flash and Hybrid

Parent storage group:

OS-[shortHostName]-[protocol]-[portgroup_name]-SG

Child storage groups:

OS-[shortHostName]-[SRP]-[ServiceLevel/Workload]-[portgroup_name]-CD-RE

Interval and Retries¶

By default, interval and retries are 3 seconds and 200 retries respectively. These determine how long (interval) and how many times (retries) a user is willing to wait for a single Rest call, 3*200=600seconds. Depending on usage, these may need to be overriden by the user in the cinder.conf. For example, if performance is a factor, then the interval should be decreased to check the job status more frequently, and if multiple concurrent provisioning requests are issued then retries should be increased so calls will not timeout prematurely.

In the example below, the driver checks every 3 seconds for the status of the job. It will continue checking for 150 retries before it times out.

Add the following lines to the VMAX backend in the cinder.conf:

[CONF_GROUP_ISCSI]
volume_driver = cinder.volume.drivers.dell_emc.vmax.iscsi.VMAXISCSIDriver
cinder_dell_emc_config_file = /etc/cinder/cinder_dell_emc_config_CONF_GROUP_ISCSI.xml
volume_backend_name = ISCSI_backend
interval = 3
retries = 200

USE CASE 1 - Default values¶

Prerequisites - VMAX

• Host I/O Limit (MB/Sec) - No Limit
• Host I/O Limit (IO/Sec) - No Limit
• Set Dynamic Distribution - N/A

1. Create QoS Specs with the prerequisite values above:

   $ openstack volume qos create --consumer back-end \
                                 --property total_iops_sec=500 \
                                 --property total_bytes_sec=104857600 \
                                 --property DistributionType=Always \
                                 SILVER
   

2. Associate QoS specs with specified volume type:

   $ openstack volume qos associate SILVER VOLUME_TYPE
   

3. Create volume with the volume type indicated above:

   $ openstack volume create --size 1 --type VOLUME_TYPE TEST_VOLUME
   

Outcome - VMAX (storage group)

• Host I/O Limit (MB/Sec) - 100
• Host I/O Limit (IO/Sec) - 500
• Set Dynamic Distribution - Always

Outcome - Block Storage (cinder)

Volume is created against volume type and QoS is enforced with the parameters above.

USE CASE 2 - Preset limits¶

Prerequisites - VMAX

• Host I/O Limit (MB/Sec) - 2000
• Host I/O Limit (IO/Sec) - 2000
• Set Dynamic Distribution - Never

1. Create QoS specifications with the prerequisite values above. The consumer in this case use case is both for front end and back end:

   $ openstack volume qos create --consumer back-end \
                                 --property total_iops_sec=500 \
                                 --property total_bytes_sec=104857600 \
                                 --property DistributionType=Always \
                                 SILVER
   

2. Associate QoS specifications with specified volume type:

   $ openstack volume qos associate SILVER VOLUME_TYPE
   

3. Create volume with the volume type indicated above:

   $ openstack volume create --size 1 --type VOLUME_TYPE TEST_VOLUME
   

4. Attach the volume created in step 3 to an instance

   $ openstack server add volume TEST_VOLUME TEST_INSTANCE
   

Outcome - VMAX (storage group)

• Host I/O Limit (MB/Sec) - 100
• Host I/O Limit (IO/Sec) - 500
• Set Dynamic Distribution - Always

Outcome - Block Storage (cinder)

Volume is created against volume type and QoS is enforced with the parameters above.

Outcome - Hypervisor (nova)

Libvirt includes an extra xml flag within the <disk> section called iotune that is responsible for rate limitation. To confirm that, first get the OS-EXT-SRV-ATTR:instance_name value of the server instance i.e. instance-00000005. We then run the following command using the OS-EXT-SRV-ATTR:instance_name retrieved above.

$ virsh dumpxml instance-00000005 | grep -1 "total_bytes_sec\|total_iops_sec"

The outcome is shown below

<iotune>
   <total_bytes_sec>104857600</total_bytes_sec>
   <total_iops_sec>500</total_iops_sec>
</iotune>

USE CASE 3 - Preset limits¶

Prerequisites - VMAX

• Host I/O Limit (MB/Sec) - 100
• Host I/O Limit (IO/Sec) - 500
• Set Dynamic Distribution - Always

1. Create QoS specifications with the prerequisite values above:

   $ openstack volume qos create --consumer back-end \
                                 --property total_iops_sec=500 \
                                 --property total_bytes_sec=104857600 \
                                 --property DistributionType=Always \
                                 SILVER
   

2. Associate QoS specifications with specified volume type:

   $ openstack volume qos associate SILVER VOLUME_TYPE
   

3. Create volume with the volume type indicated above:

   $ openstack volume create --size 1 --type VOLUME_TYPE TEST_VOLUME
   

Outcome - VMAX (storage group)

• Host I/O Limit (MB/Sec) - 100
• Host I/O Limit (IO/Sec) - 500
• Set Dynamic Distribution - OnFailure

Outcome - Block Storage (cinder)

Volume is created against volume type and QOS is enforced with the parameters above

USE CASE 4 - Default values¶

Prerequisites - VMAX

• Host I/O Limit (MB/Sec) - No Limit
• Host I/O Limit (IO/Sec) - No Limit
• Set Dynamic Distribution - N/A

1. Create QoS specifications with the prerequisite values above:

   $ openstack volume qos create --consumer back-end \
                                 --property DistributionType=Always \
                                 SILVER
   

2. Associate QoS specifications with specified volume type:

   $ openstack volume qos associate SILVER VOLUME_TYPE
   

3. Create volume with the volume type indicated above:

   $ openstack volume create --size 1 --type VOLUME_TYPE TEST_VOLUME
   

Outcome - VMAX (storage group)

• Host I/O Limit (MB/Sec) - No Limit
• Host I/O Limit (IO/Sec) - No Limit
• Set Dynamic Distribution - N/A

Outcome - Block Storage (cinder)

Volume is created against volume type and there is no QoS change.

Multipath configuration file¶

The multipath configuration file may be edited for better management and performance. Log in as a privileged user and make the following changes to /etc/multipath.conf on the Compute (nova) node(s).

devices {
# Device attributed for EMC VMAX
    device {
            vendor "EMC"
            product "SYMMETRIX"
            path_grouping_policy multibus
            getuid_callout "/lib/udev/scsi_id --page=pre-spc3-83 --whitelisted --device=/dev/%n"
            path_selector "round-robin 0"
            path_checker tur
            features "0"
            hardware_handler "0"
            prio const
            rr_weight uniform
            no_path_retry 6
            rr_min_io 1000
            rr_min_io_rq 1
    }
}

You may need to reboot the host after installing the MPIO tools or restart iSCSI and multipath services.

On Ubuntu:

# service open-iscsi restart
# service multipath-tools restart

On openSUSE, SUSE Linux Enterprise Server, Red Hat Enterprise Linux, and CentOS:

# systemctl restart open-iscsi
# systemctl restart multipath-tools

$ lsblk
NAME                                       MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
sda                                          8:0    0     1G  0 disk
..360000970000196701868533030303235 (dm-6) 252:6    0     1G  0 mpath
sdb                                          8:16   0     1G  0 disk
..360000970000196701868533030303235 (dm-6) 252:6    0     1G  0 mpath
vda                                        253:0    0     1T  0 disk

OpenStack configurations¶

On Compute (nova) node, add the following flag in the [libvirt] section of /etc/nova/nova.conf:

iscsi_use_multipath = True

On cinder controller node, set the multipath flag to true in /etc/cinder/cinder.conf:

use_multipath_for_image_xfer = True

Restart nova-compute and cinder-volume services after the change.

Verify you have multiple initiators available on the compute node for I/O¶

1. Create a 3GB VMAX volume.

2. Create an instance from image out of native LVM storage or from VMAX storage, for example, from a bootable volume

3. Attach the 3GB volume to the new instance:

   $ multipath -ll
   mpath102 (360000970000196700531533030383039) dm-3 EMC,SYMMETRIX
   size=3G features='1 queue_if_no_path' hwhandler='0' wp=rw
   '-+- policy='round-robin 0' prio=1 status=active
   33:0:0:1 sdb 8:16 active ready running
   '- 34:0:0:1 sdc 8:32 active ready running
   

4. Use the lsblk command to see the multipath device:

   $ lsblk
   NAME                                       MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
   sdb                                          8:0    0     3G  0 disk
   ..360000970000196700531533030383039 (dm-6) 252:6    0     3G  0 mpath
   sdc                                          8:16   0     3G  0 disk
   ..360000970000196700531533030383039 (dm-6) 252:6    0     3G  0 mpath
   vda
   

Enabling WLP on Unisphere¶

1. To enable WLP on Unisphere, click on the array‣Performance‣Settings.
2. Set both the Real Time and the Root Cause Analysis.
3. Click Register.

Use case 1 - Compression disabled create, attach, detach, and delete volume¶

1. Create a new volume type called VMAX_COMPRESSION_DISABLED.
2. Set an extra spec volume_backend_name.
3. Set a new extra spec storagetype:disablecompression = True.
4. Create a new volume.
5. Check in Unisphere or symcli to see if the volume exists in storage group OS-<srp>-<servicelevel>-<workload>-CD-SG, and compression is disabled on that storage group.
6. Attach the volume to an instance. Check in Unisphere or symcli to see if the volume exists in storage group OS-<shorthostname>-<srp>-<servicelevel/workload>-<portgroup>-CD, and compression is disabled on that storage group.
7. Detach volume from instance. Check in Unisphere or symcli to see if the volume exists in storage group OS-<srp>-<servicelevel>-<workload>-CD-SG, and compression is disabled on that storage group.
8. Delete the volume. If this was the last volume in the OS-<srp>-<servicelevel>-<workload>-CD-SG storage group, it should also be deleted.

Use case 2 - Compression disabled create, delete snapshot and delete volume¶

1. Repeat steps 1-5 of Use case 1.
2. Create a snapshot. The volume should now exist in OS-<srp>-<servicelevel>-<workload>-CD-SG.
3. Delete the snapshot. The volume should be removed from OS-<srp>-<servicelevel>-<workload>-CD-SG.
4. Delete the volume. If this volume is the last volume in OS-<srp>-<servicelevel>-<workload>-CD-SG, it should also be deleted.

Use case 3 - Retype from compression disabled to compression enabled¶

1. Repeat steps 1-4 of Use case 1.
2. Create a new volume type. For example VMAX_COMPRESSION_ENABLED.
3. Set extra spec volume_backend_name as before.
4. Set the new extra spec’s compression as storagetype:disablecompression = False or DO NOT set this extra spec.
5. Retype from volume type VMAX_COMPRESSION_DISABLED to VMAX_COMPRESSION_ENABLED.
6. Check in Unisphere or symcli to see if the volume exists in storage group OS-<srp>-<servicelevel>-<workload>-SG, and compression is enabled on that storage group.

Configure the source and target arrays¶

1. Configure a synchronous SRDF group between the chosen source and target arrays for the VMAX cinder driver to use. The source array must correspond with the <Array> entry in the VMAX XML file.

2. Select both the director and the ports for the SRDF emulation to use on both sides. Bear in mind that network topology is important when choosing director endpoints. Currently, the only supported mode is Synchronous.

   Note

   If the source and target arrays are not managed by the same Unisphere server (that is, the target array is remotely connected to server - for example, if you are using embedded management), in the event of a full disaster scenario (i.e. the primary array is completely lost and all connectivity to it is gone), the Unisphere server would no longer be able to contact the target array. In this scenario, the volumes would be automatically failed over to the target array, but administrator intervention would be required to either; configure the target (remote) array as local to the current Unisphere server (if it is a stand-alone server), or enter the details to the XML file of a second Unisphere server, which is locally connected to the target array (for example, the embedded management Unisphere server of the target array), and restart the cinder volume service.

3. Enable replication in /etc/cinder/cinder.conf. To enable the replication functionality in VMAX cinder driver, it is necessary to create a replication volume-type. The corresponding back-end stanza in the cinder.conf for this volume-type must then include a replication_device parameter. This parameter defines a single replication target array and takes the form of a list of key value pairs.

   enabled_backends = VMAX_FC_REPLICATION
   [VMAX_FC_REPLICATION]
   volume_driver = cinder.volume.drivers.dell_emc.vmax_fc.VMAXFCDriver
   cinder_dell_emc_config_file = /etc/cinder/cinder_dell_emc_config_VMAX_FC_REPLICATION.xml
   volume_backend_name = VMAX_FC_REPLICATION
   replication_device = target_device_id:000197811111, remote_port_group:os-failover-pg, remote_pool:SRP_1, rdf_group_label: 28_11_07, allow_extend:False
   

   • target_device_id is a unique VMAX array serial number of the target array. For full failover functionality, the source and target VMAX arrays must be discovered and managed by the same U4V server.

   • remote_port_group is the name of a VMAX port group that has been pre-configured to expose volumes managed by this backend in the event of a failover. Make sure that this portgroup contains either all FC or all iSCSI port groups (for a given back end), as appropriate for the configured driver (iSCSI or FC).

   • remote_pool is the unique pool name for the given target array.

   • rdf_group_label is the name of a VMAX SRDF group (Synchronous) that has been pre-configured between the source and target arrays.

   • allow_extend is a flag for allowing the extension of replicated volumes. To extend a volume in an SRDF relationship, this relationship must first be broken, both the source and target volumes are then independently extended, and then the replication relationship is re-established. If not explicitly set, this flag defaults to False.

     Note

     As the SRDF link must be severed, due caution should be exercised when performing this operation. If absolutely necessary, only one source and target pair should be extended at a time. In Queens, the underlying VMAX architecture will support extending source and target volumes without having to sever links.

   Note

   Service Level and Workload: An attempt will be made to create a storage group on the target array with the same service level and workload combination as the primary. However, if this combination is unavailable on the target (for example, in a situation where the source array is a Hybrid, the target array is an All Flash, and an All Flash incompatible service level like Bronze is configured), no service level will be applied.

   Note

   The VMAX cinder drivers can support a single replication target per back-end, that is we do not support Concurrent SRDF or Cascaded SRDF. Ensure there is only a single replication_device entry per back-end stanza.

4. Create a replication-enabled volume type. Once the replication_device parameter has been entered in the VMAX backend entry in the cinder.conf, a corresponding volume type needs to be created replication_enabled property set. See above Setup VMAX drivers for details.

   $ openstack volume type set --property replication_enabled = "<is> True" \
                         VMAX_FC_REPLICATION
   

Volume replication interoperability with other features¶

Most features are supported, except for the following:

• There is no OpenStack Generic Volume Group support for replication-enabled VMAX volumes.

• Storage-assisted retype operations on replication-enabled VMAX volumes (moving from a non-replicated type to a replicated-type and vice-versa. Moving to another service level/workload combination, for example) are not supported.

• The image volume cache functionality is supported (enabled by setting image_volume_cache_enabled = True), but one of two actions must be taken when creating the cached volume:

  • The first boot volume created on a backend (which will trigger the cached volume to be created) should be the smallest necessary size. For example, if the minimum size disk to hold an image is 5GB, create the first boot volume as 5GB.
  • Alternatively, ensure that the allow_extend option in the replication_device parameter is set to True.

  This is because the initial boot volume is created at the minimum required size for the requested image, and then extended to the user specified size.

Failover host¶

In the event of a disaster, or where there is required downtime, upgrade of the primary array for example, the administrator can issue the failover host command to failover to the configured target:

$ cinder failover-host cinder_host@VMAX_FC_REPLICATION#Diamond+SRP_1+000192800111

If the primary array becomes available again, you can initiate a failback using the same command and specifying --backend_id default:

$ cinder failover-host \
  cinder_host@VMAX_FC_REPLICATION#Diamond+SRP_1+000192800111 \
  --backend_id default

Consistent group snapshot¶

For creating a consistent group snapshot, a group-spec, having the key consistent_group_snapshot_enabled set to <is> True, should be set on the group. Similarly the same key should be set on any volume type which is specified while creating the group. The VMAX plugin doesn’t support creating/managing a group which doesn’t have this group-spec set. If this key is not set on the group-spec then the generic volume group will be created/managed by cinder (not the VMAX plugin).

Storage Group Names¶

Storage groups are created on the VMAX as a result of creation of generic volume groups. These storage groups follow a different naming convention and are of the following format depending upon whether the groups have a name.

TruncatedGroupName_GroupUUID or GroupUUID

Operations¶

• Create a group type

cinder --os-volume-api-version 3.11 group-type-create GROUP_TYPE

• Show a group type

cinder --os-volume-api-version 3.11 group-type-show GROUP_TYPE

• List group types

cinder --os-volume-api-version 3.11 group-type-list

• Delete group type

cinder --os-volume-api-version 3.11 group-type-delete GROUP_TYPE

• Set/unset a group spec

cinder --os-volume-api-version 3.11 group-type-key GROUP_TYPE set consistent_group_snapshot_enabled= "<is> True"

• List group types and group specs:

cinder --os-volume-api-version 3.11 group-specs-list

• Create a group:

cinder --os-volume-api-version 3.13 group-create --name GROUP GROUP_TYPE VOLUME_TYPE1,VOLUME_TYPE2

• Show a group:

cinder --os-volume-api-version 3.13 group-show GROUP

• List all groups:

cinder --os-volume-api-version 3.13 group-list

• Create a volume and add it to a group at the time of creation:

cinder --os-volume-api-version 3.13 create --volume-type VOLUME_TYPE1 --group-id GROUP_ID 1

• Modify a group to add or remove volumes:

cinder --os-volume-api-version 3.13 group-update --add-volumes UUID1,UUID2 --remove-volumes UUID3,UUID4 GROUP

• Create a group snapshot:

cinder --os-volume-api-version 3.14 group-snapshot-create --name GROUP_SNAPSHOT GROUP

• Delete group snapshot(s):

cinder --os-volume-api-version 3.14 group-snapshot-delete GROUP_SNAPSHOT

• Create a group from a group snapshot:

$ cinder --os-volume-api-version 3.14 group-create-from-src --group-snapshot GROUP_SNAPSHOT --name GROUP

• Create a group from a source snapshot:

$ cinder --os-volume-api-version 3.14 group-create-from-src --source-group SOURCE_GROUP --name GROUP

• Delete a group

cinder --os-volume-api-version 3.13 group-delete --delete-volumes GROUP

Architecture¶

In VMAX, A volume cannot belong to two or more FAST storage groups at the same time. To get around this limitation we leverage both cascaded storage groups and a temporary non FAST storage group.

A volume can remain ‘live’ if moved between masking views that have the same initiator group and port groups which preserves the host path.

During live migration, the following steps are performed by the VMAX plugin on the volume:

1. Within the originating masking view, the volume is moved from the FAST storage group to the non-FAST storage group within the parent storage group.
2. The volume is added to the FAST storage group within the destination parent storage group of the destination masking view. At this point the volume belongs to two storage groups.
3. One of two things happens:
   • If the connection to the destination instance is successful, the volume is removed from the non-FAST storage group in the originating masking view, deleting the storage group if it contains no other volumes.
   • If the connection to the destination instance fails, the volume is removed from the destination storage group, deleting the storage group, if empty. The volume is reverted back to the original storage group.

Libvirt configuration¶

Make the following updates on all nodes, controller and compute nodes, that are involved in live migration. Update the libvirt configurations. Please refer to following link for further information: http://libvirt.org/remote.html

1. Update the libvirt configurations. Modify the /etc/libvirt/libvirtd.conf file

   before : #listen_tls = 0
   after : listen_tls = 0
   
   before : #listen_tcp = 1
   after : listen_tcp = 1
   add: auth_tcp = "none"
   

2. Modify the /etc/libvirt/qemu.conf file:

   before : #dynamic_ownership = 1
   after : dynamic_ownership = 0
   before : #security_driver = "selinux"
   after : security_driver = "none"
   before : #user = "root"
   after : user = "root"
   before : #group = "root"
   after : group = "root"
   

3. Modify the /etc/default/libvirtd file:

   before: libvirtd_opts=" -d"
   after: libvirtd_opts=" -d -l"
   

4. Restart libvirt. After you run the command below, ensure that libvirt is successfully restarted:

System configuration¶

NOVA-INST-DIR/instances/ (for example, /opt/stack/data/nova/instances) has to be mounted by shared storage. Ensure that NOVA-INST-DIR (set with state_path in the nova.conf file) is the same on all hosts.

1. Configure your DNS or /etc/hosts and ensure it is consistent across all hosts. Make sure that the three hosts can perform name resolution with each other. As a test, use the ping command to ping each host from one another.

   $ ping HostA
   $ ping HostB
   $ ping HostC
   

2. Export NOVA-INST-DIR/instances from HostA, and ensure it is readable and writable by the Compute user on HostB and HostC. Please refer to the relevant OS documentation for further details. e.g. https://help.ubuntu.com/lts/serverguide/network-file-system.html https://help.ubuntu.com/community/SettingUpNFSHowTo

3. On all compute nodes, enable the ‘execute/search’ bit on your shared directory to allow qemu to be able to use the images within the directories. On all hosts, run the following command:

   $ chmod o+x NOVA-INST-DIR/instances
   

Use case¶

For our use case shown below, we have three hosts with host names HostA, HostB and HostC. HostA is the compute node while HostB and HostC are the compute nodes. The following were also used in live migration.

• 2 gb bootable volume using the cirros image.
• Instance created using the 2gb volume above with a flavor m1.small using 2048 RAM, 20GB of Disk and 1 VCPU.

1. Create a bootable volume.

   $ openstack volume create --size 2 \
                             --image cirros-0.3.5-x86_64-disk \
                             --volume_lm_1
   

2. Launch an instance using the volume created above on HostB.

   $ openstack server create --volume volume_lm_1 \
                             --flavor m1.small \
                             --nic net-id=private \
                             --security-group default \
                             --availability-zone nova:HostB \
                             server_lm_1
   

3. Confirm on HostB has the instance created by running:

   $ openstack server show server_lm_1 | grep "hypervisor_hostname\|instance_name"
     | OS-EXT-SRV-ATTR:hypervisor_hostname | HostB
     | OS-EXT-SRV-ATTR:instance_name | instance-00000006
   

4. Confirm, through virsh using the instance_name returned in step 3 (instance-00000006), on HostB that the instance is created using:

   $ virsh list --all
   
   Id   Name                  State
   --------------------------------
   1    instance-00000006     Running
   

5. Migrate the instance from HostB to HostA with:

   $ openstack server migrate --live HostA \
                              server_lm_1
   

6. Run the command on step 3 above when the instance is back in available status. The hypervisor should be on Host A.

7. Run the command on Step 4 on Host A to confirm that the instance is created through virsh.