Two local customers have ran into the same situation in the past month….both are running on a HP c-Class blade infrastructure leveraging Virtual Connect/FLEX-10 as well as on-board Broadcom 10G cards leveraging the bnx2x driver – this mixed with ESX(i) 4.1 is an almost lethal combination.

Random network drops, port flapping, dropped packets….not good for a virtual infrastructure. Luckily HP is aware of the issue and has released Advisory c02476622 which states;

The Broadcom bnx2x VMware ESX Driver Version 1.54 does not function with HP Virtual Connect Device Control Channel (DCC) and SmartLink features on ProLiant and Integrity server blades configured with the NC532m or the NC532i adapter running firmware version 2.2.6. After installing or upgrading VMware ESX/ESXi 4.1 the following functionality is either not installed or is lost:

1. New installation – DCC and SmartLink functionality is unavailable in an HP Virtual Connect environment with the NC532m or NC532i Network Adapters after installing VMware ESX/ESXi 4.1.
2. Upgrade installation – If the bnx2x Asynchronous Driver Update CD version 1.52 was previously installed on a VMware ESX/ESXi 4.0 host, DCC/SmartLink capabilities will be lost after upgrading to VMware ESX/ESXi 4.1, which will overwrite the bnx2x driver version 1.52 with version 1.54 that is included with the base VMware ESX/ESXi 4.1operating system.
3. Network failover – ProLiant and Integrity server blades hosting VMware ESX/ESXi 4.1 may lose network failover capabilities that use the VMware ESX NIC teaming failover policy (vSwitch setting) “Link Status only.”

There is a work-around that leverages VMware Beacon Probing but honestly this has been a hit/miss type work-around.  In my opinion, stick with ESX 4.0 until HP works out the issue with the firmware/driver.

Created on November 11, 2010 by Rick Scherer

Posted under Networking, vSphere.

This blog has 4,161 views and 6 responses.

Tags: beacon probing, bnx2x, broadcom, dcc, esx 4.1, flex-10, hp, virtual connect

(No Ratings Yet)

 Loading ...

 Print This Post

Created on November 11, 2010 by Rick Scherer

Posted under Good Reading, vSphere.

This blog has 880 views and 8 responses.

Tags: oracle, rac, support, VMware, vsphere

(No Ratings Yet)

 Loading ...

 Print This Post

Just about an hour ago VMware lifted the NDA on VMware vSphere 4.1 and made available all of the information on this latest release as well as the bits for download available to the public.

I will be covering a more in-depth review on this latest release really soon but I did want my readers to know that the bits are available for download from the VMware website.

http://downloads.vmware.com/d/info/datacenter_downloads/vmware_vsphere_4/4

The upgrade to ESX(i) 4.1 should be relatively easy by using traditional update methods such as VMware Update Manager (VUM).  However, the upgrade to vCenter Server 4.1 is more of a migration since it will only support a full 64-bit environment. Still don’t fret, a vCenter server migration is pretty simple just make sure you have a FULL backup of your vCenter Server database.

Also a little FYI….rumor has it that this will be the final build containing a full ESX install (Service Console). Today might be a good day to start planning that migration to ESXi.

Created on July 13, 2010 by Rick Scherer

Posted under vCenter, vSphere.

This blog has 1,389 views and one response.

Tags: esx, esxi, upgrade, vcenter, VMware, vsphere

(No Ratings Yet)

 Loading ...

 Print This Post

EMC is running a webcast hosted by Chad Sakac next Thursday and the topic should be of an interest to any VMware or Storage administrator.  If you haven’t heard about VAAI and it’s great offloading capabilities I strongly urge you to register for this webcast.

Thursday, June 24, 2010, 11:00 am ET

Host: Chad Sakac, VP VMware Technology Alliance, EMC

Register Now by Clicking Here

During this discussion, you can learn about vStorage API for Array Integration (VAAI). There are some functions that your storage system can handle more efficiently than your virtual servers and their hosts can. VAAI are a set of APIs that allows VMware vSphere to offload specific operations to the storage system improving VMware performance and freeing up resources on your virtual servers by leveraging more efficient array-based operations as an alternative to VMware host-based operations.

Join EMC Virtual Geek, Chad Sakac on June 24 to get key updates and honest answers to your most demanding questions.

Register Now by Clicking Here

Created on June 18, 2010 by Rick Scherer

Posted under EMC, Storage, vSphere.

This blog has 669 views and one response.

Tags: EMC, vaai, VMware, vsphere, vstorage

(No Ratings Yet)

 Loading ...

 Print This Post

Everyone in the virtualization blogosphere has already covered this but I figured I should let my loyal readers know…

Earlier today VMware became aware of an issue affecting users of VMware View after applying Update 2 to their ESX 4.0 hosts. The problem only effects PCoIP, RDP works normally. There is a discussion of the problem in the VMware Communities here.

While our IT Teams work to resolve the issue, the Knowledge Base Team has responded by creating an up-to-the-minute live document at: http://kb.vmware.com/kb/1022830 and using @vmwarecares and @vmwarekb Twitter accounts to alert customers.

This Knowledge Base article will be updated as new information becomes available. If you have been affected by this, please read the KB.

We apologize for any inconvenience this may have caused you. If you know how to spread the word to your friends and colleagues, please do so.

Created on June 15, 2010 by Rick Scherer

Posted under View, vSphere.

This blog has 327 views and one response.

(No Ratings Yet)

 Loading ...

 Print This Post

VMware has just dropped the second major update bundle (Update 2) for their flagship virtualization management product, VMware vCenter server.  This update addresses a number of issues found since the release of Update 1 as well as a number of improvements, such as:

Guest Operating System Customization Improvements: vCenter Server now supports customization of the following guest operating systems:

• Windows XP Professional SP2 (x64) serviced by Windows Server 2003 SP2
• SLES 11 (x32 and x64)
• SLES 10 SP3 (x32 and x64)
• RHEL 5.5 Server Platform (x32 and x64)
• RHEL 5.4 Server Platform (x32 and x64)
• RHEL 4.8 Server Platform (x32 and 64)
• Debian 5.0 (x32 and x64)
• Debian 5.0 R1 (x32 and x64)
• Debian 5.0 R2 (x32 and x64)

Here are some of the key issues that are resolved in this update;

Read More…

Created on June 10, 2010 by Rick Scherer

Posted under vCenter, vSphere.

This blog has 9,637 views and one response.

Tags: vcenter, vcenter server, vcenter server update 2, vsphere, vsphere update 2

(No Ratings Yet)

 Loading ...

 Print This Post

You’ve guessed it, VMware has released Update 2 of their flagship bare-metal virtualization product ESX and ESXi. This update addresses a number of issues found since the release of Update 1 as well as a number of enhancements, such as:

Enablement of Fault Tolerance Functionality for Intel Xeon 56xx Series processors— vSphere 4.0 Update 1 supports the Intel Xeon 56xx Series processors without Fault Tolerance. vSphere 4.0 Update 2 enables Fault Tolerance functionality for the Intel Xeon 56xx Series processors.

Enablement of Fault Tolerance Functionality for Intel i3/i5 Clarkdale Series and Intel Xeon 34xx Clarkdale Series processors— vSphere 4.0 Update 1 supports the Intel i3/i5 Clarkdale Series and Intel Xeon 34xx Clarkdale Series processors without Fault Tolerance. vSphere 4.0 Update 2 enables Fault Tolerance functionality for the Intel i3/i5 Clarkdale Series and Intel Xeon 34xx Clarkdale Series processors.

Enablement of IOMMU Functionality for AMD Opteron 61xx and 41xx Series processors— vSphere 4.0 Update 1 supports the AMD Opteron 61xx and 41xx Series processors without input/output memory management unit (IOMMU). vSphere 4.0 Update 2 enables IOMMU functionality for the AMD Opteron 61xx and 41xx Series processors.

Enhancement of the esxtop/resxtop utility— vSphere 4.0 Update 2 includes an enhancement of the performance monitoring utilities, esxtop and resxtop. The esxtop/resxtop utilities now provide visibility into the performance of NFS datastores in that they display the following statistics for NFS datastores: Reads/s, writes/s, MBreads/s, MBwrtn/s, cmds/s, GAVG/s(guest latency).

Additional Guest Operating System Support— ESX/ESXi 4.0 Update 2 adds support for Ubuntu 10.04. For a complete list of supported guest operating systems with this release, see the VMware Compatibility Guide.

Read More…

Created on June 10, 2010 by Rick Scherer

Posted under vSphere.

This blog has 3,562 views and one response.

Tags: esx, esxi, u2, update 2, vsphere

(2 votes, average: 4.00 out of 5)

 Loading ...

 Print This Post

Last month at EMC World we released a storage virtualization product unlike any other. EMC VPLEX not only virtualizes back-end storage from other vendors it also allows shared storage to be presented active/active over long distances. This capability makes it possible for long distance vMotion and brings a new technical term to disaster planning, Disaster Avoidance.

Imagine you’re planning a datacenter shutdown due to a power outage or building maintenance, or perhaps that wildfire is making its way to your office location. Now you have the flexibility to be able to avoid major downtime by simply swinging your workloads to an off-site datacenter.

VMware has taken notice and has officially released a KB article detailing out full support for long distance vMotion with the EMC VPLEX product. More information on this KB article can be found at http://kb.vmware.com/kb/1021215. Perhaps you want to know more about VPLEX or even see a demo of it in action? Chad Sakac posted a great breakdown of how it works along with some videos on his website, check it out now.

Created on June 8, 2010 by Rick Scherer

Posted under Backup & Recovery, Storage, vCenter, vSphere.

This blog has 1,755 views and 10 responses.

Tags: disaster avoidance, disaster recovery, EMC, vmotion, vplex, vsphere

(4 votes, average: 3.25 out of 5)

 Loading ...

 Print This Post

As handful of patches have just been released by VMware for their flagship bare-metal virtualization products ESX and ESXi.

With no surprise to me the majority of the patches are for ESX and relate to security flaws and vulnerabilities found within the Service Console.  Keep in mind these vulnerabilities in no-way mean the virtual machines being hosted are at risk. These patches are typically for underlying services that the Service Console rely on, such as openssl, java, gzip and ntp. Sometimes these patches also resolve issues on how the Service Console communicates with the vmkernel layer as well as system devices.

Two of the patch bundles for ESXi share some common fixes with it’s ESX brother which cover a NTP vulnerability, a shared interrupt issue between the vmkernel and console as well as a patch that properly enables quiescing utilizing the Microsoft Windows VSS components found in Windows 2008 R2 and Windows 7.

More information on these patches can be found by reviewing the individual bundles;

ESX 4.0 – ESX400-201005001
Includes 9 updates, including fixes for NTP, gzip, bind, vmkernel, krb5, webCenter, Expat, sudo and gcc.

ESXi 4.0 – ESXi400-201005001
Includes two updates, ESXi400-201005401-SG for the ESXi firmware and ESXi400-201005402-BG for VMware Tools.

For updating your ESX(i) hosts, simply use Update Manager or download the patches from the VMware website and use the Host Update Utility to perform these updates.

Created on May 28, 2010 by Rick Scherer

Posted under vSphere.

This blog has 396 views and 2 responses.

Tags: esx, esxi, patch, updates, vsphere

(No Ratings Yet)

 Loading ...

 Print This Post

I had a very fun weekend. It started at 4am Saturday with a migration of ~125 virtual machines from an old AMD based environment to a new Intel Nehalem based environment. Who could’ve known that within a few hours all hell would’ve broken loose.

Enter in problem of network flooding from the NetXen based HP branded NC522SFP.  Because all of the 10GbE ports from the (9) new ESXi servers were creating thousands of pause frames on the Cisco Nexus 5020 switches, I thought originally that it was an issue on the switch.  Talks with Cisco revealed nothing.  We attempted to disconnect one of the connected ports (each ESXi host is dual connected into a pair of N5Ks using vPC) to remove a potential spanning tree loop….no dice.

A reboot of the host resolved the problem, things appeared to be running normally and we decided to let it be and wait until Monday.

10 hours goes by, it is now Sunday morning and the problem returns.  First host loses storage (we’re doing NFS over 10GbE here), then two more…until all 9 in this cluster are pretty much toast.  I decide to open a ticket with VMware.  Wouldn’t you know, there is a potential known bug and resolution.

Bug 496013

Description: Some NetXen based 10GbE cards using the unm_nic and nx_nic drivers sometime flood the network with pause frames causing the port to become disabled.

Resolution: NetXen believes upgrading the firmware to version 4.0.516 will resolve the problem.

I’ve gone ahead and patched 4 of the hosts with this new firmware, so far it has been stable (knock on wood).   I’ll let you know if something happens.

Checking which version of the firmware you’re running is simple. From a command-line (ESX or ESXi hidden CLI), type ethtool -i <vmnic#> (replace vmnic# with the alias to the vmnic you’d like to check).  You should see output similar to:

driver: nx_nic

version: 4.0.301

firmware-version: 4.0.406

bus-info: 0000:07:00.0

Update – Utility CD with firmware patch now included…

As you can see above, the firmware is out of date. To update the firmware you will need to boot from a Linux utility CD that has the appropriate driver, you then run a firmware update utility provided by HP.  To make this process easy I have created a bootable SLAX utility CD with the drivers pre-loaded. You can download the ISO from here (file temporarily removed). Once booted run the installer located in the root filesystem (ie: ./CP011471.scexe).

Let me know if you have any questions.

Created on January 11, 2010 by Rick Scherer

Posted under ESX 3.5 Tips, ESXi 3.5 Tips, Networking, Storage, vSphere.

This blog has 12,651 views and 39 responses.

Tags: 4.0.516, Bug, Bug 496013, esx, esxi, firmware, NC522SFP, network flood, NetXen, spanning tree

(2 votes, average: 4.00 out of 5)

 Loading ...

 Print This Post