Saturday, October 14, 2017

HPE are the worst.

Everyone knows that HP is just terrible, right? The latest one that bit me is this:

ProLiant G7 Series Servers -VMWare ESXi 5.5 and 6.0 Host Loses Connection to the Network

What they say is that, basically, if you want to use VMware on a BL465 G7 and you've made the stupid decision to upgrade the firmware on that BL465 G7, bad luck. It's now a brick. And HPE don't care - "Note: HPE will not provide any additional fixes moving forward."  This is the same BL465G7 that I had to fight with to upgrade the Power Controller firmware

There are two problems. First, they say you need firmware 4.9.416.15, which was (Yay me for tracking it down!) almost impossible to find. Then they explicitly say not to use the HPE Images. Sigh.

But, Good News, Everyone! I did the hard work for you. The Emulex Firmware 4.9.416.15 is contained in Legacy_OneConnect-Flash-10.7.110.38. Just download that ISO, boot your blade from it, and it'll upgrade the firmware of the NC551i to 4.9.416.15

That's step one.  But it's still not going to work, as the drivers on the HPE ISOs are wrong, and come up with 'No Network Detected'.  So you need to roll your own ISO, and go down the chain of problems and issues that I had. But, if you don't want to actually do it, I've (again) done the hard work for you, so here is the complete, working ISO (build 5310538)

Here's what I had to do (and this is assuming you have the powercli tools already installed)

Get the bits and pieces from VMware.

1. The Latest VMW 6.5 HPE custom image
2. The elxnet 10.7.110.44 driver (Which works in 6.5) specified by HPE

Open up the driver, and extract the -offline-bundle.zip file and put that and the image into C:\HP (or wherever. I put them into C:\HP because I'm lazy. If you change the path, please engage brain before typing.)

Import them into Powershell

Remember, you can use tab expansion for the URL, you don't need to type them all in.

Add-EsxSoftwareDepot -DepotUrl C:\HP\VMware-ESXi-6.5.0-5310538-HPE-650.10.1.3.5-Oct2017-depot.zip
Add-EsxSoftwareDepot -DepotUrl C:\HP\VMW-ESX-6.0.0-elxnet-10.7.110.44-offline_bundle-4014430.zip

Make your new Image

You clone the HPE image to make your own. Let's call it 'BL465G7' to be imaginative

New-EsxImageProfile -CloneProfile HPE* -vendor xrobau -name "BL465G7"

Update (Downgrade, actually) the driver

You should now see both elxnet drivers

PowerCLI C:\hp> Get-EsxSoftwarePackage | findstr elx
elx-esx-libelxima.so     11.2.1238.0-03                 ELX        2/05/2017 4:2...
elxiscsi                 11.2.1238.0-1OEM.650.0.0.45... EMU        2/05/2017 4:2...
emulex-esx-elxnetcli     11.1.28.0-0.0.4564106          VMware     27/10/2016 4:...
elxnet                   10.7.110.44-1OEM.600.0.0.27... EMU        3/06/2016 7:4...
elxnet                   11.2.1149.0-1OEM.650.0.0.42... EMU        2/11/2016 8:3...

We want to REMOVE the one that's in the clone, and replace it with the older one

Remove-EsxSoftwarePackage BL465G7 elxnet
Add-EsxSoftwarePackage -imageprofile BL465G7 -SoftwarePackage "elxnet 10.7.110.44-1OEM*"

Fix HP's other mistakes

Well, we can't fix ALL of them. They've made so many. So, so many. But we can fix the one that's causing our machine to PSOD.  There's something broken with the Mellanox 4 drivers in the HP Image that causes the machine to PSOD on the HPE image. So you need to remove them.

Remove-EsxSoftwarePackage BL465G7 nmlx4-en
Remove-EsxSoftwarePackage BL465G7 nmlx4-core
Remove-EsxSoftwarePackage BL465G7 nmlx4-rdma

There's also an ongoing issue with the 'smx-provider' (Smart Array, aka your P410i) which also causes PSOD's, so rip that out, too.

Remove-EsxSoftwarePackage BL465G7 smx-provider

Finally, Build your ISO

Export-EsxImageProfile -ImageProfile BL465G7 -ExportToIso -filepath c:\HP\VMware-ESXi-BL465.G7-6.5.0

Just boot from that ISO, and you're off and running. No PSODs, and a NC551i that works!







2 comments:

Unknown said...

Very helpful post. This worked for ESXi 6.0 U3 as well.

Just want to note one thing, if you use the Emulex for iSCSi Storage as well, ESXi will not detect the storage adapters after the upgrade/update. You will have to downgrade the HP provided EMU drivers for ima-be2iscsi and scsi-be2isci to 10.7.110.10 or replace them in the custom image.

Stewart Saunders said...

Hopefully not too many people are having the problem. In case you did something silly like me and didnt make sure you were at firmware level 4.9.416.15 first, below is a link to the offline firmware.

https://support.hpe.com/hpsc/swd/public/detail?swItemId=MTX_3e9e9f26de844ae58f2ee11530#tab2

I only provide this because the dropbox in the post is actually for firmware 4.9.416.7