Emulex CIM Provider Issues with HP Hardware (and G7 / be2net woes)
Sunday, June 26, 2011 at 4:50PM I stumbled upon two nasty “bugs” with the Emulex CIM Provider and HP Hardware (the worst of which applies specifically to HP’s G7 Series blades). This post applies to ESXi 4.1 Update 1, Emulex CIM Provider 3.2.30, and HP’s Offline Bundle 1.1a.
Bug 1
The Emulex CIM Provider does not work when the HP Offline Bundle is loaded. When the two are loaded together, you will lose functionality on both the Emulex OneCommand tab as well as the Hardware Status tab. There may be additional loss of functionality that I did not see, so feel free to comment if you have any additional information. I verified this with both ESXi 4.1 Update 1 GA (348481) and the latest patched build as of 6/26/2011 (381591).
If you are running VMware’s version of ESXi 4.1 U1, the Emulex CIM Provider should work just fine on most of HP’s hardware (I will get to the G7’s in a minute). If you use HP’s media, you are going to be out of luck by default, as they bake their CIM Provider right into the image. If you are really hot to get this functionality from Emulex (IE, needing update firmware on a LPe12000 HBA), you can remove the HP bundle via Local or Remote TSM [esxupdate remove –b BundleID] or vSphere CLI [vihostupdate.pl --server 10.1.1.20 --username root -B BundleID --remove].
I opened up a case with Emulex on the matter and got the following responses from their tech support department (props to Tim M @ Emulex for responding to my case quickly and thoroughly!)
Emulex 1:
There is a known issue with the HP custom visor images for installing ESX and ESXi and the Emulex CIM providers that are required for proper operation of OCM for vCenter plug-in and OCM stand-alone.
Emulex is working with HP and VMware. Currently Emulex believes the issue to be with HP's custom image but final analysis is not complete. We have reproduced the issue in house and reported it to HP.
We currently have no work around for this issue. We recommend that you contact HP to see if they have a bootable image for doing firmware updates with the HP qualified firmware. We do not have a time frame for resolution of this issue at this time.
Emulex 2:
Update to the issue with HP provided customized visor image.
The issue I mentioned has been narrowed down to the "HP ESXi Offline Bundle for 4.1 U1". If that is not installed, the Emulex CIM provider can be installed and all functionality works. However, that bundle may include the drivers for HP LOMs (LAN on Motherboard) chips so the onboard 10 Gbe may not work without it.
We are continuing to work with VMware and HP on the issue.
Looks like we won't see a fix until HP revs their Offline Bundle. I opened a case with HP on the matter, but unlike Emulex, they wanted a SAID (HP purchased VMware Support ID) to provide any assistance. If someone can get something out of HP on this matter, please comment below!
Bug 2
The “bug” with the G7 Series blades is a bit nastier. When the Emulex CIM Provider is installed on an HP G7 series blade the onboard 1/10GB FlexNIC’s disappear after the host reboots. Notice the absence of vmnic0/1 in the DCUI below.
;)
;)
I activated Local TSM and removed the Emulex package with esxupdate and rebooted, but unfortunately, that did not fix the issue. Scanning through the logs in the local troubleshooter didn’t reveal much either, except that the host couldn’t bring up the management or vMotion networks. I ended up just reinstalling ESXi and my NIC’s were back in business.
;)
The Issue
The trouble here stems again from using HP’s Customized ESXi image, which is necessary on the G7 blades (as noted here and here). However, this “bug” is not related to the offline bundle but to the version of the be2net driver included with their April release of 4.1 U1 (took me another host rebuild to figure that one out).
Background
The BL460c G7’s onboard NIC (NC553i) is a rebranded BladeEngine 3 adapter, which uses the be2net driver. The BladeEngine platform was designed by ServerEngines, which Emulex acquired about a year ago. This adapter can provide FCoE functionality when used with HP’s VirtualConnect system. Emulex provides agnostic technology through their OneConnect product line.
This means that regardless of whether you are using FCoE or not, you still have to adhere to Emulex’s minimum requirements, since Emulex’s CIM Provider is designed to provide management over both integrated FCoE and FC HBA’s. Per Emulex, you must be using be2net version 2.103.100.0 (or later) and firmware revision 2.103.100.0 (or later).
The Rub
HP keeps a running list on what they have included with their releases for all of the 4.x ESXi media easily accessible. Per this list, HP’s 4.1 U1 media includes be2net 2.102.518.0, not high enough for Emulex’s plug-in to function. Also, our G7’s had firmware 3.102.517.703 loaded (technically 2.102.517.703 per HP), which is also not good enough.
The Workaround
BEFORE attempting to install the Emulex CIM Provider, Install the latest revision of be2net, 2.104.277.3, AND the latest release of the BladeEngine firmware, 2.104.281.0. Note: You will still have to remove HP’s Offline Bundle if you want OneCommand to work, which you should do AFTER installing be2net and the firmware but BEFORE installing the Emulex bundle.
The be2net update comes in VIB format, so the install is pretty straight forward. The firmware update is painful as it is NOT included on the 9.30 HP Firmware Maintenance DVD. The download, a bootable ISO, is time consuming.
;)
In the future, if you were to get Emulex OneCommand working, you will be able to extract the .UFI firmware file from the ISO and update this firmware on the fly.
Other Issues
In preparing the content for this post, I looked through VMware’s KB to see if there were any others advisories regarding the BladeEngine platform and found these four articles:
- ESX loses network connectivity with Emulex BladeEngine 10Gb Ethernet Controller be2net driver loaded
- ESX host stops when shutting down with the Emulex BladeEngine 10Gb Ethernet Controller be2net driver loaded
- Connecting or disconnecting the only virtual machine on a ESXi vDS using Emulex OneConnect NICs causes 100% host CPU and network connectivity loss
- The guest operating system that uses Emulex NIC using be2net driver on HP BL 465 G7 loses network connectivity
The Skinny
Going through all of these articles, there seems to be a pattern of issues with both connectivity and reliability with this platform. Regardless of if you are planning to implement Emulex OneCommand for vCenter, make sure all of your hosts have the latest be2net driver and NIC firmware loaded. This should help you avoid all of these nasty issues.
As always, comment below with your experiences, suggestions, or feedback!
Reader Comments (8)
Jon
Thanks for highlighting these issues. I want to let you and the end-user community know we take these problems very seriously, and that there is light at the end of this particular tunnel.
Emulex is aware of these issues when using the HP G7 blades and VMware image, and we are working hand in hand with both HP, VMware and our customers to establish root cause and deliver a timely resolution. We are presently testing what we believe to be a final solution that should require minimal change to existing customer environments.
Once the full qualification is complete we will immediately post the solution here, on our web site at http://www.emulex.com/support.html, and will update our documentation, so users are aware. In the meantime Emulex Tech Support can provide additional assistance if needed and will proactively notify all customers with registered cases when a solution becomes available. Any users that want to register a case can do so at http://www.emulex.com/support.html.
Nothing is more important to us than the value and out-of-box experience we bring to our customers. Occasionally we stub our toe …. when that happens, our priority is to get things back on track as quickly as we can.
Andrew Eastaugh
Snr Product Manager, Emulex OneCommand
Jon
As a follow up on the CIM Provider problem, this is fully resolved with the Emulex CIM Provider package v3.4 and higher, which can be downloaded from the following URL for vShere 4 platforms.
http://www.emulex.com/downloads/emulex/vmware/vsphere-41/management.html
The fix that we implemented does not require any uninstallation or modification to the VMware environment or HP Provider bundle .... just uninstall the existing Emulex CIM Providers and apply the new ones.
Although the root of the problem was essentially across multiple elements, we wanterd to provide our customers with the least disruptive single-point solution, and we found a way to do that by apply changes via the CIM Provider Developer kit (CIM PDK)
Andrew Eastaugh
Snr Product Manager, Emulex OneCommand
Hi Andrew,
I can let you know that 3.4.15.5 has exactly the same problem as described above.
Emulex has not resolved this issue.
Regards
Dean.
Correction.
Updating 3.4.15.5 on a system with an earlier CIM results in the same issue.
A fresh build with 3.4.15.5 works.
Regards
Dean
Dean
With reference to your posts, the update to CIM Providers v3.4 should work as either an upgrade to a prior release or as a fresh installation. For an upgrade from a prior release a system reboot may be required in some cases. ... not clear if that was part of your process.
Andrew Eastaugh
Snr Product Manager, Emulex OneCommand
As you already know, for some time there has been a problem installing both HP and Emulex CIM providers on the same system. The root cause was previously found to be the distribution of Emulex libraries in the HP CIM bundle ... essentially, both sets of providers were trying to own the same libariy files when only one owner is permitted.
I'm happy to report that HP has now corrected their CIM bundle so both sets of providers can now coexist. The hp-esxi5.0uX-bundle-1.1-37 was released on 12/16 and calls out the fix in the release notes. We have tested the new providers in our labs and found no problems
You can downoad the new HP Providers from the followinf URL
http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareDescription.jsp?lang=en&cc=us&prodTypeId=15351&prodSeriesId=4091412&prodNameId=4091432&swEnvOID=4115&swLang=8&mode=2&taskId=135&swItem=MTX-dff3173d40bd45b094300e80c1
Best regards
Andrew Eastaugh
Snr Product Manager - Emulex OneCommand Manager
Thank you Andrew for your consistent followup
Seeing that some traction is going on here. I have a Dell that is exhibiting a similar situation. I installed the Emulex CIM providers (and NO OTHER CIM providers) in ESX5i. I can register the devices but they cannot interrogate the card once I've rebooted, etc.
I installed the latest firmware and the latest iSCSI ESXi drivers. I did NOT install the Emulex Ethernet drivers as the cards aren't being used for Ethernet.
I'm trying to enable jumbo frames and I believe this is in the plugin/OMM package.