Managing Emulex HBA’s on ESXi

If you actively work with VMware vSphere, you may have heard that VMware recently confirmed that they are no longer developing their flagship platform ESX in favor of the new hotness, ESXi. This will become official when vSphere.next is released sometime late summer / early fall 2011.

This leaves many IT professionals wondering how they will translate existing VI management procedures into a world without inline management software. Those already running at least some mix of ESXi in their VMware environments have already experienced the change in what is required to keep the lights on. I would recommend that even if you are too busy putting out fires or don’t have the resources to start migrating to ESXi in advance, start engaging their vendors on the subject and at least have some idea of where they stand on support and how you would tackle issues like deployment, driver and firmware maintenance, monitoring, backup / recovery, and anything else you might be leveraging the service console.

To relate this to a current "hot topic", relate this mindset to this article on Chad’s blog that called out an issue with the Emulex 8GB HBA’s and EMC’s CX series storage processors, where FC Switch <-> SP link state could cause datastore availability issues on certain Emulex firmware revisions.

If you are running Emulex LPe12000’s @ 2.0.0.A3 with EMC CX Storage, getting to the latest firmware revision should be high on your to-do list. Unfortunately, neither Chad’s blog nor Emulex gives any guidance on how to effectively resolve the issue in the modern virtual environment (where it is possible to have 10’s, 100’s, or even 1000’s of hosts in this precarious situation).

With ESX Classic, it is simple, though time intensive, to install the Emulex HBAnywhere utility via SSH or direct SC access and update to the latest code. Simple enough with a handful of hosts, however, when you start getting into double and triple digit host counts, this can quickly become a maintenance nightmare. Scripting helps here, but it is still a pain.

With ESXi, those options are out the window. At first glance, with no Service Console and no support in vMA, vCLI, or PowerCLI to do script this remotely, this looks to be very painful. Each upgrade must be done via out-of-band management (iLO, iDRAC, KVM, etc) with a pre-boot environment (PBE). Emulex does not provide said PBE, so what is left is solutions such as a Linux LiveCD, the Windows PE solution suggested HERE, or running something similar to HP’s Firmware Maintenance DVD. However, before you go do all that, I would like to highlight a lightly published Emulex utility that makes this infinitely easier that you might expect.

Enter the Emulex OneCommand Manager for vCenter Plug-in. It provides single pane of glass management through the vSphere console and serves as a wonderful example of the management simplicity attained through CIM Provider and vSphere API integration. It gives you access to just about everything you would ever want out of an HBA management utility and works really well. What surprises me the most is that even though Emulex showed off their new vCenter plug-in at VMworld 2010, there has been very little blog coverage. Google and Bing searches for “Emulex vCenter plug-in” as of 6/19/2011 turn up only TWO useful entries (here and here)

This is a two-part solution, requiring a web service to be installed and registered with vCenter as well as a CIM provider on the ESXi host. NOTE: If you are still running ESX Classic, you can install a full blown agent within the service console to gain similar functionality, though for this particular issue, it might be easier just to upgrade the firmware while you are on the SC and skip setting up the agent.

The Emulex OCM for vCenter can help IT administrators with the following tasks:

• Manage Emulex Fibre Channel (FC), Fibre Channel over Ethernet (FCoE) and Network Interface Card (NIC) adapters on ESX / ESXi servers in lockdown mode
• Manage a UCNA’s DCB (Data Center Bridging) settings
• Manage a UCNA’s Fibre Channel over Ethernet (FCoE) Initialization Protocol (FIP)
• Manage adapter feature licenses
• Manage protocol personalities
• Firmware download for adapters on a specific cluster of hosts and for adapters on a specific host
• Change an adapter’s World Wide Port Name (WWPN) and World Wide Node Name (WWNN)
• Reset FC/FCoE adapter ports
• Update firmware and FC boot code (x86 BootBIOS) on a single adapter or multiple adapters (using Batch Update)
• Set global FC/FCoE driver parameters
• Run diagnostic tests on UCNA adapters
• View vital product data (VPD) for the selected adapter port
• View transceiver information for the selected adapter port
• View vNIC1 data for supported adapters
• Create and save reports about discovered storage area network (SAN) elements

The User Manual, which has a lot of fantastic information on both functionality and minimum driver and firmware requirements, is located HERE. Because of the HP specific issues I had getting this up and running, I strongly suggest that you RTFM before installing this software. To get this plugin to work with either the standard LightPulse (LP and LPe) cards or the next-gen OneConnect cards (CNAs), you must adhere to the minimum driver and firmware requirements.

For reference:

LightPulse adapters

• Driver version: lpfc820_400.8.2.0.76.20 (or later)
• Firmware: Latest firmware available for your LightPulse card on the Emulex website

OneConnect adapters

• FCoE driver: lpfc820_400.8.2.0.76.20 (or later)
• NIC driver: be2net-2.103.100.0 (or later)
• Firmware: 2.103.100.0 (or later)

IMPORTANT: IF YOU ARE RUNNING HP’S G7 SERIES BLADES WITH HP’S CUSTOMIZED ESXi IMAGE

Regardless of the stated requirements, do not install the Emulex Bundle! There is a conflict between the HP CIM provider (Offline Bundle 1.1) and the Emulex CIM provider. I will detail the issue and a workaround in my next post.

Getting back on track...

First, download the latest (2.0.0 A4 as of 6/19/2011) firmware for the Emulex LPe12000 HBA HERE. Be sure to grab the right download as there are vendor specific distributions listed.

Second, grab both the latest CIM Provider and vCenter Plug-in downloads HERE. You do not need the applications kit as that is only for ESX Classic. Note: Registration is required to download the vSphere plug-in.

Installing the vCenter Plug-In

The vCenter Plug-in install is pretty straightforward and coexistence is supported on the vCenter server providing that you change the default web service ports from 8080 / 8443 to something else. Otherwise, you will get a conflict with the VMware vCenter Web Services instance and the plug-in registration wizard will not load correctly. Below is a photo walkthrough of the install process.

Once the installation is complete, launch the Registration Utility to link the two web services together. This utility requires administrator rights to the vCenter instance and only serves to register the plug-in as available for all vSphere clients.

Once this is done, restart your vSphere client. You will now see the Emulex plug-in under the Plug-in Manager as well a "Emulex OneCommand" tab under "Hosts and Clusters" for both the cluster level and host level views, however, you will not be able to do any management until you install the CIM Provider on ESXi or the Applications Kit on ESX and reboot the targeted hosts.

Installing the CIM Provider on ESXi

There are a couple methods to do this, the easiest (in my opinion) being the VMware vSphere CLI. PowerCLI and vMA are options, but I won't cover them here. First, put the host in maintenance mode and then run the following command:

vihostupdate.pl --server esx1234.domain.com --username admin --password AdminP@ss -i -b drive:\directory\elx-esx-4.1.0-emulex-cim-provider-3.2.30.1-offline_bundle-364582.zip

Note: If you are running Dell's Customized ESXi 4.1 image, you already have a "compatible" Emulex CIM Provider, so you should be able to get some basic information out of the host right after you install the vCenter plugin. I would recommend updating to the latest version to get all the latest and greatest functionality.

After the installation, reboot the host. Once this is done, you should see the "Emulex CIM Provider" listed under the Hardware Status tab, as well as with a vihostupdate.pl -query command. You will also see information start to come out of the aforementioned "Emulex OneCommand" tab.

You can apply this firmware update in two ways, either at a host level or cluster level. I have not tested a cluster level update yet, but the idea is that you can blast out a firmware update to all of the hosts in your cluster at once. I am NOT SURE if this will automatically put hosts in maintenance mode (similar to the way VUM remediates a cluster), so TEST this before doing this in production (as you could cause availability issues if you don't stop the I/O before hand). Either way you want to do it, go to the Emulex OneCommand tab, select Maintenance (or batch update for the cluster view), browse to the extracted firmware file from the aforementioned download, and click OK. The plug-in will validate the file and only apply it to adapters that need it. You will get a status window with general progress and a Success / Failure return status. Once you get a success, that should be all the fiddling you will need to do. As always, if you are actually experiencing the issue detailed by Chad's blog, make sure to leverage your support contracts to make sure you are getting the full story (as well as vendor assistance if things do not as planned).

Note that Firmware Maintenance is only ONE small piece of this plug-in, and it would be a waste to entitle this blog post "Managing Emulex HBA's on ESXi" without some more detail about the rest of the plugin. I will not go over all of the ins and outs of this product, but here are some screens showing the type of information that you can attain from this plug-in.

The big areas of interest from my view are Driver Parameters, where you can do fine tuning, if needed, as well as fabric / lun view as you drill down farther into the tree of each adapter. As shown below, you can get REALLY detailed status on presented storage, which helps when translating "vSphere admin speak" to "SAN Guy speak" if those two people aren't one and the same.

As always, questions and comments are always welcome. Stay tuned for my next post on why NOT to use this with HP's Customized ESXi image (and specifically why not to use it with the HP G7 series blades).

Comments

[Bas]

Awesome post I really enjoyed reading this! I almost can’t wait for the next post about the G7 blades !
BTW isn't the easiest method to install the CIM provider importing the offline bundle in Update Manager ?

[Jon Kohler]

Bas,
RE VUM Deployment: I didn't see any support statement (could have missed it) from Emulex on whether their bundle actually works with VUM (not all offline bundles do). That said, I did try to import this into our VUM instance and it kicked back an error. Not 100% sure if it was the bundle or just VUM being VUM. If you try it and get different results, post back and let me know!

Thanks for the feedback, always appreciated.
Jon

[Darren]

Hi, Is there a workaround for the G7 blades please?

Cheers
Darren

[Jon Kohler]

Darren,
There is a workaround, I am still polishing that post and should have it up in the next day or so.

[Bas]

Hi Jon,

I created a Host Extension Baseline with only the emulex cim provider. Remediation of the patch works like a charm. I tested against ESXi 348481 build and vCenter build 345043.

Cheers
Bas

[Jon Kohler]

Bas,
Thanks for the update! Probably just something quirky with our VUM instance then, I will try to chase that down later.

Darren,
I am about to publish the workaround post, it should go live tonight (6/26/2011)