Last week I came across a problem that really stumped me, it even stumped the Tier-1 and Tier-2 support at VMware.  I’m posting the symptoms on here in a hope that someone else has experienced this issue and can share some light.

How about a little background on the environment, vCenter Server 4.0 U1 and multiple ESX(i) hosts (3.5, 4.0, 4.0 U1).   The vCenter Server as well as a number of ESXi 4.0 hosts were upgraded to U1 a couple days after it was released,  this problem however happened ~8 days after the upgrade.

Symptom 1: All ESX(i) hosts disconnect from vCenter Server, however, they are still online and no VMs went down.  Within 15 minutes all hosts appear to be reconnected.

Symptom 2: After the hosts reconnect, the ESX hosts appear to be functioning normally. However, the ESXi hosts display an error on the Overview tab as well as in the Events tab; “Unable to Synchronize with host that is unavailable.”

Symptom 3: Random VMotions start, for no apparent reason (DRS engaged, yet no constraints causing DRS to be invoked).  However, these VMotions fail at 10% due to the fact that the source and destination host is not available.

Symptom 4: /var/log/messages file displays errors with keywords: [VpxdVmomi] Error getting vpxa info: SSL Exception: Unexpected EOF From hosts, blacklisting showing up.   — I apologize for paraphrasing.

So, all this starts happening and I start investigating….pulling logs, restarting vCenter, and just sit there stumped.  I did notice that the rui.crt on the vCenter server expired, but back in 2008.  I went ahead and renewed the certificate and even restarted the entire vCenter server.  No luck.  I engaged VMware Support and their Tier-1 and Tier-2 support were stumped,  nothing even showed up in their internal database on this issue.

Then it all disappeared.  Roughly 90 minutes after it started, the problem just went away and everything was good.

Have you seen this issue?  What were your troubleshooting steps?  Did you resolve it or figure out the resolution?

Created on December 9, 2009 by Rick Scherer

Posted under vSphere.

This blog has 2,370 views and 5 responses.

Tags: error, esx, esxi, ssl, troubleshooting, vcenter

(No Ratings Yet)

 Loading ...

 Print This Post

Sometime yesterday ESX(i) Update 5 finally hit VMware Update Manager, about 2 days after the official announcement and release on the VMware website. This announcement includes updates for ESX, ESXi and vCenter Server. In addition to Update 5 being released there were about 20 additional updates made available for ESX(i), including 16 which were marked as critical.

The following information provides highlights of some of the enhancements available in this release of VMware ESX Server, this information can be found in the VMware ESX(i) 3.5 U5 Release Notes:

Enablement of Intel Xeon Processor 3400 Series– Support for the Intel Xeon processor 3400 series has been added. Support includes Enhanced VMotion capabilities. For additional information on previous processor families supported by Enhanced VMotion, see Enhanced VMotion Compatibility (EVC) processor support (KB 1003212).

Driver Update for Broadcom bnx2 Network Controller– The driver for bnx2 controllers has been upgraded to version 1.6.9. This driver supports bootcode upgrade on bnx2 chipsets and requires bmapilnx and lnxfwnx2tools upgrade from Broadcom. This driver also adds support for Network Controller – Sideband Interface (NC-SI) for SOL (serial over LAN) applicable to Broadcom NetXtreme 5709 and 5716 chipsets.

Driver Update for LSI SCSI and SAS Controllers – The driver for LSI SCSI and SAS controllers is updated to version 2.06.74. This version of the driver is required to provide a better support for shared SAS environments.

Newly Supported Guest Operating Systems – Support for the following guest operating systems has been added specifically for this release:

For more complete information about supported guests included in this release, see the VMware Compatibility Guide: http://www.vmware.com/resources/compatibility/search.php?deviceCategory=software.

• Windows 7 Enterprise (32-bit and 64-bit)
• Windows 7 Ultimate (32-bit and 64-bit)
• Windows 7 Professional (32-bit and 64-bit)
• Windows 7 Home Premium (32-bit and 64-bit)
• Windows 2008 R2 Standard Edition (64-bit)
• Windows 2008 R2 Enterprise Edition (64-bit)
• Windows 2008 R2 Datacenter Edition (64-bit)
• Windows 2008 R2 Web Server (64-bit)
• Ubuntu Desktop 9.04 (32-bit and 64-bit)
• Ubuntu Server 9.04 (32-bit and 64-bit)

Newly Supported Management Agents – See VMware ESX Server Supported Hardware Lifecycle Management Agents for current information on supported management agents.

Newly Supported Network Cards –This release of ESX Server supports HP NC375T (NetXen) PCI Express Quad Port Gigabit Server Adapter.

Newly Supported SATA Controllers – This release of ESX Server supports the Intel Ibex Peak SATA AHCI controller.

In addition to the enhancements found in ESX(i) 3.5 U5, there is also one lonely enhancement made to vCenter Server 2.5 U5:

Support for High Consolidation in VMware HA Clusters- VirtualCenter 2.5 Update 5 includes significant performance and scalability improvements to VMware HA. Use VirtualCenter 2.5 Update 5 for environments with more than 35 virtual machines per host in an HA cluster.
For information on the ESX Server host settings required for this scalability improvement, see ESX Server host settings required for environments with up to 80 virtual machines per host in an HA Cluster (KB 1012002).

Updating your ESX servers can and should be done with VMware Update Manager. To upgrade your vCenter Server installation you’ll need to download the installation ISO or ZIP from the VMware website and perform an in-place upgrade. Be sure to create a backup of your vCenter Server database then follow the steps in the Installation Guide.

Created on December 6, 2009 by Rick Scherer

Posted under ESX 3.5 Tips, ESXi 3.5 Tips, vCenter.

This blog has 2,801 views and no responses.

Tags: esx, esxi, update, vcenter, vi3, virtual infrastructure

(No Ratings Yet)

 Loading ...

 Print This Post

VMware recently released a patch, ESX350-200904401-BG which resolves a number of issues, which can be found in KB1010126.

The biggest fix that has affected me lately is, “Fixes an issue where an unsuccessful online consolidation might cause a virtual machine to fail and become unusable because of a CID mismatch.”

I recently ran into some problems with some virtual machines that have high I/O failing during the commit of a snapshot, I discussed this in an earlier post which can be found here.

I’d highly recommend anyone running ESX 3.5 to apply this update as it resolves a lot of known issues that can affect VM performance and stability.  Always remember to follow proper patching procedures, thoroughly test the patch install and verify before placing production Virtual Machines back on the host.

Created on May 5, 2009 by Rick Scherer

Posted under ESX 3.5 Tips.

This blog has 2,747 views and one response.

Tags: esx, patch

(1 votes, average: 5.00 out of 5)

 Loading ...

 Print This Post

Created on April 14, 2009 by Rick Scherer

Posted under vSphere.

This blog has 2,147 views and one response.

Tags: esx, launch, training, trainsignal, VMware, vsphere

(2 votes, average: 5.00 out of 5)

 Loading ...

 Print This Post

I had a big problem Monday AM on one of my core SAP VM instances, that also happens to have a SQL DB server on it. Our VCB process finishes up on late Sunday night, if you’re not aware of how VCB works, it basically creates a snapshot of the Virtual Machine, then mounts the now readable VMDK parent to a proxy server where your backup agent resides. Once the backup is complete the snapshot is committed.  This wasn’t the case Monday AM — the VM crashed and I was paged. Snapshot didn’t commit, parent VMDK could not be found, had to manually set Parent CID in the delta VMDK file then finally when I got it back online the SQL DB was corrupt :( — luckily I had a full SQL backup from the night before.

This is where VMware KB 1007969 comes into the story…

Read More…

Created on April 14, 2009 by Rick Scherer

Posted under ESX 3.5 Tips, ESXi 3.5 Tips, Storage.

This blog has 5,259 views and 6 responses.

Tags: CID, commit, esx, esxi, snapshot, vcb, vmdk

(No Ratings Yet)

 Loading ...

 Print This Post

The past few weeks I’ve been working with a customer that only has ESXi, Veeam FastSCP has came in really handy lately! I can copy files between systems or between a system and my desktop, the flexibility it creates is amazing.  It also attempts to utilize the VMKernel API first, if it cannot make a connection that way it will fallback to SSH.  Did I forget to mention, its blazing fast!

Well, now is your chance to get Veeam Backup and Replication for 50% off! Don’t miss this great opportunity to get a solid backup utility for your ESX(i) infrastructure.  Oh yeah, it also works with ESXi Free Edition!

Through June 15, Veeam Software is extending a special upgrade offer that gives you budget relief, freeing you to license the new #1 for VMware backup and replication. Veeam has the functionality, reliability and performance you need now.

You can have both backup and replication in one product – now 50% off when you provide proof of purchase of a competing product. Learn more >>

Created on April 7, 2009 by Rick Scherer

Posted under Backup & Recovery.

This blog has 2,134 views and 2 responses.

Tags: backup, copy, esx, esxi, fastscp, replication, restore, veeam

(No Ratings Yet)

 Loading ...

 Print This Post

I recently ran into a situation with a customer where they needed to add another host to their cluster.  Problem is, their original hosts had single core AMD Opteron processors (Opteron 285) and the host they wanted to purchase came with either a Dual-Core or Quad-Core Opteron.  The customer was scared about the VMotion Compatibility between their older AMD hosts and the newer hosts they would be purchasing….so much that they wanted me to find them an old single-core system. I know EVC is nothing new, I’ve been enabling it when needed for almost 8 months now.  You may run into a situation where your customer (or boss) is a little hesitant in enabling this advanced feature. But rest assured, VMware has KB 1003212 to answer all your questions.

Read More…

Created on March 31, 2009 by Rick Scherer

Posted under ESX 3.5 Tips, ESXi 3.5 Tips.

This blog has 8,151 views and 2 responses.

Tags: esx, esxi, evc, vcenter, vmotion

(No Ratings Yet)

 Loading ...

 Print This Post

VMware has released the latest update to its ESX(i) 3.5 flagship product, Update 4.  It is strongly recommended that you upgrade to VMware vCenter 2.5 Update 4 prior to upgrading your ESX hosts.  Updates such as this one typically include a number of system improvements and also all of the patches available in-between it and the previous update available. Numerous driver additions and updates have been added to this update roll-up, including;

Read More…

Created on March 30, 2009 by Rick Scherer

Posted under ESX 3.5 Tips, ESXi 3.5 Tips.

This blog has 9,377 views and one response.

Tags: esx, esxi, patch, update, vcenter

(1 votes, average: 5.00 out of 5)

 Loading ...

 Print This Post

A lot of information about the upcoming 4.0 release of VMware’s Enterprise Virtualization product, vSphere, has been released at VMworld Europe 2009.  I’d like to share some of the exciting new capabilities in this upcoming release.

Read More…

Created on February 25, 2009 by Rick Scherer

Posted under VMworld, vSphere.

This blog has 19,926 views and 16 responses.

Tags: esx, maximums, vsphere

(2 votes, average: 3.50 out of 5)

 Loading ...

 Print This Post

Idealstor, a leading developer of disk-to-disk backup solutions, announced today the release of iBac VIP for VMware Virtual Center.  iBac VIP for Virtual Center was created to simplify VMware backups by offering a single license that backs up every virtual machine regardless of how many ESX hosts have been implemented.

Read More…

Created on February 2, 2009 by Rick Scherer

Posted under Backup & Recovery, Storage.

This blog has 2,360 views and no responses.

Tags: backup, esx, esxi, ibak, idealstor, vcb

(No Ratings Yet)

 Loading ...

 Print This Post