darrylcauldwell.com On a journey around the datacenter and public cloud.

Advanced vSphere 5.x Storage - Masking, Multipathing & Filtering

The relationship between a vSphere ESXi host and its shared storage is very important for the solution to work effectively. The storage architecture within vSphere is extensible and this extensibility is known as the Pluggable Storage Architecture often abbreviated to PSA.

The PSA is an open modular framework that coordinates the simultaneous operation of multiple multi pathing plugins (MPPs). PSA is a collection of VMkernel APIs that allow third party hardware vendors to insert code directly into the ESX storage I/O path. This allows 3rd party software developers to design their own load balancing techniques and failover mechanisms for particular storage array. The PSA coordinates the operation of the NMP and any additional 3rd party MPP.

To view what PSA plugins are loaded,

esxcli storage core plugin list

By default only NMP and MASK_PATH plugins are installed. To install a third-party PSA plugin your vendor would provide you with a vSphere Installation Bundle vib file.  An example of how to install a VIB we can see for the Netapp NFS VAAI plugin.

esxcli software vib install -n NetAppNasPlugin -d file:///NetAppNasPlugin.v20.zip
Message: The update completed successfully, but the system needs to be rebooted for the changes to be effective.
Reboot Required: true VIBs Installed:
NetApp_bootbank_NetAppNasPlugin_1.0-020
VIBs Removed:
VIBs Skipped:

As there are multiple plugins supplied and others can be plugged in there is a rule list which defines which device gets managed by which plugin. You can view existing claim rules.

esxcli storage core claimrule list

The rules are applied by rule number lowest to high,  so if you have a complex claimrule set and your devices are picking up incorrectly check the order. You will see a catch all at the bottom 65535 which assigns any without specific match to NMP .

One MP rule you will notice is listed twice with Class as being file and runtime.  The file which these rules apply is

/etc/vmware/esx.conf

Using this we can if we wish we can tell PSA to detect devices and apply the MASK_PATH plugin. You might want to consider doing this at the ESX layer if some issue with the storage which causes host to loose contact with it and the LUN (or LUNs) enter an all-paths-down condition by temporarily masking the path while the storage engineer fixes the fabric.

Obtain ESXi Device ID of the LUN you would like to mask

esxcli storage core device list

Once you have the device ID you can then obtain :C Channel :T Target :L LUN and vmhba of the device you want to mask

esxcfg-mpath -b -d {device-id}

Paths can be viewed and changed using this namespace use this to find a claim rule number not in use

esxcli storage core claimrule

Using all the above information assign the device to MASK_PATH

esxcli storage core claimrule add -r 500 -t location -A vmhba35 -C <x> -T <y> -L <z> -P MASK_PATH

Once rule is added you should be able to see it in the claimrule list but you will notice this is file state not runtime

esxcli storage core claimrule list

You now need to load the file into rules and then run the new ruleset

esxcli storage core claimrule load
esxcli storage core claimrule run

Once loaded and in running configuration you should see

esxcli storage core claimrule list

Even though the new rule is in place for new devices to pickup,  the current device still has an active claimrule,  you can either reboot,  or run a reclaim on the LUN so it picks up its new rule

esxcli storage core claiming reclaim -d {device id}

If your in lab and want to get your LUN back

esxcli storage core claimrule remove -r 500
esxcli storage core claimrule load
esxcli storage core claiming unclaim -t location -A vmhba33 -C 0 -T 0 -L 0
esxcli storage core adapter rescan -A vmhba33

You may want to mask out a whole vendor

esxcli storage core claimrule add -r 501 -t vendor -V SYNOLOGY -P MASK_PATH
esxcli storage core claimrule load
esxcli storage core claimrule run
esxcli storage core claiming reclaim -d <device id of device>

To undo mask a whole vendor

esxcli storage core claimrule remove -r 501
esxcli storage core claimrule load
esxcli storage core claiming unclaim -t vendor -v SYNOLOGY
esxcli storage core claiming unclaim -t location -A vmhba33
esxcli storage core adapter rescan -A vmhba33

Once you have correct plugin’s installed,  and claimrules in place so correct devices get picked up by correct plugin,  you might want to then configure the plugin to say how it should react for path failover and how to choose correct path.

Within the storage plugin are two submodules

  • Storage Array Type Plugin (SATP) – Used for path failover
  • Path Selection Plugin (PSP) – Used for selecting the path

An SATP plugin monitors physical path health, reports to the NMP changes in physical paths, and executes array-specific actions for activating and deactivating paths. The Path Selection Plug-ins (PSP) selects the path for I/O requests.  The flow of how these two components operate is nicely shown in this short video.

[youtube=http://www.youtube.com/watch?v=cEp69JYEx30]

The pathing policies which can be used with VMware NMP can be managed via the namespace

esxcli storage nmp psp

You can list all available PSP’s using

esxcli storage nmp psp list

VMW_PSP_MRU – Most Recently Used, on path failure recovery stays load stays on same path
MW_PSP_RR – Round Robin, rotates through available paths
VMW_PSP_FIXED – Fixed Pathing, on path failure recovery load moves back to preferred path

The SATP level path selection can be viewed

esxcli storage nmp satp list

The final thing to cover with regards how storage can be seen and managed within ESXi would be device filtering.  There are four storage filters all of which are applied by default,  these filters define what can be seen within the cli and gui.

  • VMFS Filter: filters out storage devices or LUNs that are already used by a VMFS datastore
  • RDM Filter: filters out LUNs that are already mapped as a RDMSame Host and
  • Transports Filter: filters out LUNs that can’t be used as a VMFS datastore extend.
    • Prevents you from adding LUNs as an extent not exposed to all hosts that share the original VMFS datastore.
    • Prevents you from adding LUNs as an extent that use a storage type different from the original VMFS datastore
  • Host Rescan Filter: Automatically rescans and updates VMFS datastores after you perform datastore management operations

vSphere Client -> Administration -> vCenter Server -> Settings -> Advanced Settings

To disable add the following key(s) if not already there and set it to false.

config.vpxd.filter.vmfsFilter (VMFS Filter)
config.vpxd.filter.rdmFilter (RDM Filter)
config.vpxd.filter.SameHostAndTransportsFilter (Same Host and Transports Filter)
config.vpxd.filter.hostRescanFilter (Host Rescan Filter)

Be social and share this post!