NetXen HP NC522SFP Network Flooding

I had a very fun weekend. It started at 4am Saturday with a migration of ~125 virtual machines from an old AMD based environment to a new Intel Nehalem based environment. Who could’ve known that within a few hours all hell would’ve broken loose.

Enter in problem of network flooding from the NetXen based HP branded NC522SFP.  Because all of the 10GbE ports from the (9) new ESXi servers were creating thousands of pause frames on the Cisco Nexus 5020 switches, I thought originally that it was an issue on the switch.  Talks with Cisco revealed nothing.  We attempted to disconnect one of the connected ports (each ESXi host is dual connected into a pair of N5Ks using vPC) to remove a potential spanning tree loop….no dice.

A reboot of the host resolved the problem, things appeared to be running normally and we decided to let it be and wait until Monday.

10 hours goes by, it is now Sunday morning and the problem returns.  First host loses storage (we’re doing NFS over 10GbE here), then two more…until all 9 in this cluster are pretty much toast.  I decide to open a ticket with VMware.  Wouldn’t you know, there is a potential known bug and resolution.

Bug 496013

Description: Some NetXen based 10GbE cards using the unm_nic and nx_nic drivers sometime flood the network with pause frames causing the port to become disabled.

Resolution: NetXen believes upgrading the firmware to version 4.0.516 will resolve the problem.

I’ve gone ahead and patched 4 of the hosts with this new firmware, so far it has been stable (knock on wood).   I’ll let you know if something happens.

Checking which version of the firmware you’re running is simple. From a command-line (ESX or ESXi hidden CLI), type ethtool -i <vmnic#> (replace vmnic# with the alias to the vmnic you’d like to check).  You should see output similar to:

driver: nx_nic

version: 4.0.301

firmware-version: 4.0.406

bus-info: 0000:07:00.0

Update – Utility CD with firmware patch now included…

As you can see above, the firmware is out of date. To update the firmware you will need to boot from a Linux utility CD that has the appropriate driver, you then run a firmware update utility provided by HP.  To make this process easy I have created a bootable SLAX utility CD with the drivers pre-loaded. You can download the ISO from here (file temporarily removed). Once booted run the installer located in the root filesystem (ie: ./CP011471.scexe).

Let me know if you have any questions.


Created on January 11, 2010 by Rick Scherer

Posted under ESX 3.5 Tips, ESXi 3.5 Tips, Networking, Storage, vSphere.

This blog has 22,016 views and 40 responses.

Tags: , , , , , , , , ,

1 Star2 Stars3 Stars4 Stars5 Stars (2 votes, average: 4.00 out of 5)
Loading...

VMware Releases VI3 Update 5

Sometime yesterday ESX(i) Update 5 finally hit VMware Update Manager, about 2 days after the official announcement and release on the VMware website. This announcement includes updates for ESX, ESXi and vCenter Server. In addition to Update 5 being released there were about 20 additional updates made available for ESX(i), including 16 which were marked as critical.

The following information provides highlights of some of the enhancements available in this release of VMware ESX Server, this information can be found in the VMware ESX(i) 3.5 U5 Release Notes:

Enablement of Intel Xeon Processor 3400 Series– Support for the Intel Xeon processor 3400 series has been added. Support includes Enhanced VMotion capabilities. For additional information on previous processor families supported by Enhanced VMotion, see Enhanced VMotion Compatibility (EVC) processor support (KB 1003212).

Driver Update for Broadcom bnx2 Network Controller– The driver for bnx2 controllers has been upgraded to version 1.6.9. This driver supports bootcode upgrade on bnx2 chipsets and requires bmapilnx and lnxfwnx2tools upgrade from Broadcom. This driver also adds support for Network Controller – Sideband Interface (NC-SI) for SOL (serial over LAN) applicable to Broadcom NetXtreme 5709 and 5716 chipsets.

Driver Update for LSI SCSI and SAS Controllers – The driver for LSI SCSI and SAS controllers is updated to version 2.06.74. This version of the driver is required to provide a better support for shared SAS environments.

Newly Supported Guest Operating Systems – Support for the following guest operating systems has been added specifically for this release:

For more complete information about supported guests included in this release, see the VMware Compatibility Guide: http://www.vmware.com/resources/compatibility/search.php?deviceCategory=software.

  • Windows 7 Enterprise (32-bit and 64-bit)
  • Windows 7 Ultimate (32-bit and 64-bit)
  • Windows 7 Professional (32-bit and 64-bit)
  • Windows 7 Home Premium (32-bit and 64-bit)
  • Windows 2008 R2 Standard Edition (64-bit)
  • Windows 2008 R2 Enterprise Edition (64-bit)
  • Windows 2008 R2 Datacenter Edition (64-bit)
  • Windows 2008 R2 Web Server (64-bit)
  • Ubuntu Desktop 9.04 (32-bit and 64-bit)
  • Ubuntu Server 9.04 (32-bit and 64-bit)

Newly Supported Management Agents – See VMware ESX Server Supported Hardware Lifecycle Management Agents for current information on supported management agents.

Newly Supported Network Cards –This release of ESX Server supports HP NC375T (NetXen) PCI Express Quad Port Gigabit Server Adapter.

Newly Supported SATA Controllers – This release of ESX Server supports the Intel Ibex Peak SATA AHCI controller.

In addition to the enhancements found in ESX(i) 3.5 U5, there is also one lonely enhancement made to vCenter Server 2.5 U5:

Support for High Consolidation in VMware HA Clusters– VirtualCenter 2.5 Update 5 includes significant performance and scalability improvements to VMware HA. Use VirtualCenter 2.5 Update 5 for environments with more than 35 virtual machines per host in an HA cluster.
For information on the ESX Server host settings required for this scalability improvement, see ESX Server host settings required for environments with up to 80 virtual machines per host in an HA Cluster (KB 1012002).

Updating your ESX servers can and should be done with VMware Update Manager. To upgrade your vCenter Server installation you’ll need to download the installation ISO or ZIP from the VMware website and perform an in-place upgrade. Be sure to create a backup of your vCenter Server database then follow the steps in the Installation Guide.


Created on December 6, 2009 by Rick Scherer

Posted under ESX 3.5 Tips, ESXi 3.5 Tips, vCenter.

This blog has 4,063 views and no responses.

Tags: , , , , ,

1 Star2 Stars3 Stars4 Stars5 Stars (No Ratings Yet)
Loading...

Before Host Profiles there was vicfg-cfgbackup.pl

This last weekend I was reminded of a RCLI command that creates a backup of your ESXi Host configuration.  A client had an ESXi host where the USB drive failed.  The system didn’t entirely crash, it pinged so HA didn’t kick-in however hostd and vpxa weren’t responding so management from vCenter was impossible. Performing a reboot of the host proved it was a USB failure when it failed to boot.

So, HA kicked in finally when the host was powered off – however the remaining hosts in their cluster did not have adequate resources to power everything back on.  We needed to get this failed ESX host back online and quick!

Brought in new USB stick with fresh ESXi 4.0 installed, plugged in and powered up.  In the TUI configured the Management Port IP address/gateway/DNS, etc.    But now what, all of the (NFS) Datastores, vSwitch configuration and advanced settings were toast…and they didn’t have Enterprise Plus.

Say Hello to vicfg-cfgbackup.pl — a RCLI command which could be found in the vMA.  This utility will allow you to create a backup and restore a full ESXi configuration.  Luckily we had this (backup) command running on a nightly basis so we knew the configuration was complete.  A simply command and the host was online.

Seeing that ESX is going to be phased out, this RCLI command is a valid alternative for those ESXi users that do not have Enterprise Plus licensing.

The commands vicfg-cfgbackup.pl (esxcfg-cfgbackup.pl) allow you to backup and restore the configuration of your ESXi host.

To backup the host you would run the command.

vicfg-cfgbackup.pl –server <server_name> -s <backup_file_name>

To restore your backup configuration to your host you would run. This will cause the host to reboot once the process is complete. NOTE: The host must be in Maintenance Mode for this to work. The backup configuration must also match the patch level of the ESXi install. You can add a -f to force if needed.

vicfg-cfgbackup.pl –server <server_name> -l <backup_file_name>


Created on October 12, 2009 by Rick Scherer

Posted under Backup & Recovery, ESXi 3.5 Tips, vSphere.

This blog has 20,991 views and 8 responses.

Tags: , , , , ,

1 Star2 Stars3 Stars4 Stars5 Stars (No Ratings Yet)
Loading...

Committing snapshots generates a content ID mismatch error

I had a big problem Monday AM on one of my core SAP VM instances, that also happens to have a SQL DB server on it. Our VCB process finishes up on late Sunday night, if you’re not aware of how VCB works, it basically creates a snapshot of the Virtual Machine, then mounts the now readable VMDK parent to a proxy server where your backup agent resides. Once the backup is complete the snapshot is committed.  This wasn’t the case Monday AM — the VM crashed and I was paged. Snapshot didn’t commit, parent VMDK could not be found, had to manually set Parent CID in the delta VMDK file then finally when I got it back online the SQL DB was corrupt :( — luckily I had a full SQL backup from the night before.

This is where VMware KB 1007969 comes into the story…

Read More…


Created on April 14, 2009 by Rick Scherer

Posted under ESX 3.5 Tips, ESXi 3.5 Tips, Storage.

This blog has 12,068 views and 6 responses.

Tags: , , , , , ,

1 Star2 Stars3 Stars4 Stars5 Stars (No Ratings Yet)
Loading...

VMware Enhanced VMotion Compatibility (EVC)

I recently ran into a situation with a customer where they needed to add another host to their cluster.  Problem is, their original hosts had single core AMD Opteron processors (Opteron 285) and the host they wanted to purchase came with either a Dual-Core or Quad-Core Opteron.  The customer was scared about the VMotion Compatibility between their older AMD hosts and the newer hosts they would be purchasing….so much that they wanted me to find them an old single-core system. I know EVC is nothing new, I’ve been enabling it when needed for almost 8 months now.  You may run into a situation where your customer (or boss) is a little hesitant in enabling this advanced feature. But rest assured, VMware has KB 1003212 to answer all your questions.

Read More…


Created on March 31, 2009 by Rick Scherer

Posted under ESX 3.5 Tips, ESXi 3.5 Tips.

This blog has 20,782 views and 2 responses.

Tags: , , , ,

1 Star2 Stars3 Stars4 Stars5 Stars (No Ratings Yet)
Loading...

VMware ESX 3.5 Update 4 Released

VMware has released the latest update to its ESX(i) 3.5 flagship product, Update 4.  It is strongly recommended that you upgrade to VMware vCenter 2.5 Update 4 prior to upgrading your ESX hosts.  Updates such as this one typically include a number of system improvements and also all of the patches available in-between it and the previous update available. Numerous driver additions and updates have been added to this update roll-up, including;

Read More…


Created on March 30, 2009 by Rick Scherer

Posted under ESX 3.5 Tips, ESXi 3.5 Tips.

This blog has 11,050 views and one response.

Tags: , , , ,

1 Star2 Stars3 Stars4 Stars5 Stars (1 votes, average: 5.00 out of 5)
Loading...

VMware vCenter Mobile Access

I guess I posted a little too soon about Rove’s Mobile Admin release that has full support for VMware vCenter.   VMware has just announced VMware vCenter Mobile Access (vCMA) which is due to be released BETA in April, this is a Virtual Appliance that links your mobile device/smart phone to your vCenter Server. More information can be found on the Technology Preview Site.

Read More…


Created on March 20, 2009 by Rick Scherer

Posted under ESX 3.5 Tips, ESXi 3.5 Tips, Monitoring, vSphere.

This blog has 4,971 views and no responses.

Tags: , , , ,

1 Star2 Stars3 Stars4 Stars5 Stars (No Ratings Yet)
Loading...

Installing the Performance Overview Plug-In in VirtualCenter 2.5 Update 4

For those of you upgrading to vCenter 2.5 U4 and and are planning to use the new Performance Overview Plugin, there are some additional steps required to making this work.  Pretty much the biggest thing is that you need the Java Development Kit installed on the vCenter server. All of this is covered in VMware KB 1008296, but for your convenience I’ve included the steps here.

Read More…


Created on February 27, 2009 by Rick Scherer

Posted under ESX 3.5 Tips, ESXi 3.5 Tips, Monitoring.

This blog has 8,760 views and 3 responses.

Tags: , , ,

1 Star2 Stars3 Stars4 Stars5 Stars (No Ratings Yet)
Loading...

VMware vCenter 2.5 U4 Released

A few days ago VMware released vCenter 2.5 U4, the latest installment of their vCenter Server.  I have yet to upgrade my environment(s), but I will be planning to do so very soon.  There are two new enhancements to vCenter with this release, and a handful of updates.

Read More…


Created on February 27, 2009 by Rick Scherer

Posted under ESX 3.5 Tips, ESXi 3.5 Tips.

This blog has 5,882 views and one response.

Tags:

1 Star2 Stars3 Stars4 Stars5 Stars (No Ratings Yet)
Loading...

5000 Virtual Desktops using 10GB Storage in Minutes

Most people have never heard about NetApp’s Rapid Cloning Utility (RCU) for VMware. This tool is the magic behind the famous YouTube video that show how NetApp can very quickly create and deploy a very large number of Virtual Desktops, while using very little disk space on the filer. With desktop virtualization being hot right now this is a great tool to get to know and use. The tool is available on the NOW tool chest (a free website for NetApp customers and partners).

In this video below, NetApp demonstrates the cost savings of their zero disk cost provisioning and data deduplication software. 5000 Virtual Desktops provisioned in minutes, while only taking a little over 10GB storage.

Read More…


Created on February 5, 2009 by Rick Scherer

Posted under ESX 3.5 Tips, ESXi 3.5 Tips, Good Reading, NetApp, Storage, VMware.

This blog has 4,425 views and 5 responses.

Tags: , , , , , ,

1 Star2 Stars3 Stars4 Stars5 Stars (2 votes, average: 5.00 out of 5)
Loading...