VMware ESX Multipathing for 9500V/AMS/WMS

VMware has recently changed ESX default multi-pathing behavior for modular subsystems introduced in patch ESX-1003524. This fix changes the default multipath policy of Hitachi modular storage arrays to FIXED mode since most of these arrays are either Active/Active or Pseudo Active/Active and using FIXED mode results in better performance on a Active/Active Array.

The information contained below is still appropriate for systems that do NOT contain the fix described in ESX-1003524.

By default, VMware ESX treats the 9500 and AMS subsystems as active/passive array. VMware selects the MRU parameter as the default multi-path mechanism.

On modular subsystems, while we have a concept of LUN ownership, Hitachi implementation does NOT behave as VMware expects an Active/Passive subsystem to operate. While industry standard path management software suites such as HDLM and DMP are able to detect LUN ownership assignments via INQUIRY data, VMware ESX accomplishes owner path discovery based upon the response the server receives after issuance of I/O to the non-owning controller.

In a 'true' Active/Passive subsystem, when I/O is issued to the non-owning controller, a UNIT ATTENTION will be sent to the initiator. This UNIT ATTENTION tells ESX that it should use the alternate path. In the case of the modular subsystem product, we allow access to the LUN down the non-owning path, and do NOT send a UNIT ATTENTION when I/O is received down the non-owning controller path. The modular subsystem will handle I/O down the non-owning path by using a backend interconnect between controllers. If I/O is NOT received down the owning path for 60 seconds, LUN ownership will change from the previous 'owning controller' to the new 'non-owning controller', which then becomes the owning controller path after ownership change.

As mentioned, ESX cannot determine the LUN ownership configuration of modular subsystems and utilizes MRU, or the Most Recently Used path management policy. MRU will set the active path to that which is first discovered at boot time. Usually, this path will default to the HBA which occupies the lowest PCI Bus address assignment or the HBA which is initialized first. Due to this behavior, there is a high potential for VMware to over utilize one controller during operation. Best practice for modular subsystems dictates that workload should be spread out across both controllers. By default, VMware using MRU will most likely direct I/O to only one controller, while leaving the alternate controller relatively unused.

VMware administrators have the ability to control where ESX sends I/O by utilizing the Fixed, or Preferred, path management policy. Using the Fixed policy allows the administrators to intelligently segment workload across both controllers.

In the case of failure scenarios, MRU policy will detect a communication problem down its operating path, and switch to the alternate path. However, when the original operating path is repaired, ESX will continue to operate on the alternate path, and will only switch back to the original operating path if a failure is detected on the 'alternate' path, or if the server is rebooted and ESX detects the primary path first at initialization.

In the case of a failure scenario, the Fixed policy will also detect a communication problem down its primary operating path, and switch to the alternate path. In this case though, when ESX detects that communication is possible through the Primary, or Fixed path, operation will automatically switch back from the alternate path back to primary path. This setting helps ensure that the hosts are efficiently using both controllers.

Fixed path management policy should be used in all cases involving ESX and Hitachi 9500 and WMS/AMS subsystems.