Just about an hour ago VMware lifted the NDA on VMware vSphere 4.1 and made available all of the information on this latest release as well as the bits for download available to the public.

I will be covering a more in-depth review on this latest release really soon but I did want my readers to know that the bits are available for download from the VMware website.

http://downloads.vmware.com/d/info/datacenter_downloads/vmware_vsphere_4/4

The upgrade to ESX(i) 4.1 should be relatively easy by using traditional update methods such as VMware Update Manager (VUM).  However, the upgrade to vCenter Server 4.1 is more of a migration since it will only support a full 64-bit environment. Still don’t fret, a vCenter server migration is pretty simple just make sure you have a FULL backup of your vCenter Server database.

Also a little FYI….rumor has it that this will be the final build containing a full ESX install (Service Console). Today might be a good day to start planning that migration to ESXi.

Posted under vCenter, vSphere

You’ve guessed it, VMware has released Update 2 of their flagship bare-metal virtualization product ESX and ESXi. This update addresses a number of issues found since the release of Update 1 as well as a number of enhancements, such as:

Enablement of Fault Tolerance Functionality for Intel Xeon 56xx Series processors— vSphere 4.0 Update 1 supports the Intel Xeon 56xx Series processors without Fault Tolerance. vSphere 4.0 Update 2 enables Fault Tolerance functionality for the Intel Xeon 56xx Series processors.

Enablement of Fault Tolerance Functionality for Intel i3/i5 Clarkdale Series and Intel Xeon 34xx Clarkdale Series processors— vSphere 4.0 Update 1 supports the Intel i3/i5 Clarkdale Series and Intel Xeon 34xx Clarkdale Series processors without Fault Tolerance. vSphere 4.0 Update 2 enables Fault Tolerance functionality for the Intel i3/i5 Clarkdale Series and Intel Xeon 34xx Clarkdale Series processors.

Enablement of IOMMU Functionality for AMD Opteron 61xx and 41xx Series processors— vSphere 4.0 Update 1 supports the AMD Opteron 61xx and 41xx Series processors without input/output memory management unit (IOMMU). vSphere 4.0 Update 2 enables IOMMU functionality for the AMD Opteron 61xx and 41xx Series processors.

Enhancement of the esxtop/resxtop utility— vSphere 4.0 Update 2 includes an enhancement of the performance monitoring utilities, esxtop and resxtop. The esxtop/resxtop utilities now provide visibility into the performance of NFS datastores in that they display the following statistics for NFS datastores: Reads/s, writes/s, MBreads/s, MBwrtn/s, cmds/s, GAVG/s(guest latency).

Additional Guest Operating System Support— ESX/ESXi 4.0 Update 2 adds support for Ubuntu 10.04. For a complete list of supported guest operating systems with this release, see the VMware Compatibility Guide.

Read More…

Posted under vSphere

As handful of patches have just been released by VMware for their flagship bare-metal virtualization products ESX and ESXi.

With no surprise to me the majority of the patches are for ESX and relate to security flaws and vulnerabilities found within the Service Console.  Keep in mind these vulnerabilities in no-way mean the virtual machines being hosted are at risk. These patches are typically for underlying services that the Service Console rely on, such as openssl, java, gzip and ntp. Sometimes these patches also resolve issues on how the Service Console communicates with the vmkernel layer as well as system devices.

Two of the patch bundles for ESXi share some common fixes with it’s ESX brother which cover a NTP vulnerability, a shared interrupt issue between the vmkernel and console as well as a patch that properly enables quiescing utilizing the Microsoft Windows VSS components found in Windows 2008 R2 and Windows 7.

More information on these patches can be found by reviewing the individual bundles;

ESX 4.0 - ESX400-201005001
Includes 9 updates, including fixes for NTP, gzip, bind, vmkernel, krb5, webCenter, Expat, sudo and gcc.

ESXi 4.0 - ESXi400-201005001
Includes two updates, ESXi400-201005401-SG for the ESXi firmware and ESXi400-201005402-BG for VMware Tools. 

For updating your ESX(i) hosts, simply use Update Manager or download the patches from the VMware website and use the Host Update Utility to perform these updates.

Posted under Uncategorized

I had a very fun weekend. It started at 4am Saturday with a migration of ~125 virtual machines from an old AMD based environment to a new Intel Nehalem based environment. Who could’ve known that within a few hours all hell would’ve broken loose.

Enter in problem of network flooding from the NetXen based HP branded NC522SFP.  Because all of the 10GbE ports from the (9) new ESXi servers were creating thousands of pause frames on the Cisco Nexus 5020 switches, I thought originally that it was an issue on the switch.  Talks with Cisco revealed nothing.  We attempted to disconnect one of the connected ports (each ESXi host is dual connected into a pair of N5Ks using vPC) to remove a potential spanning tree loop….no dice.

A reboot of the host resolved the problem, things appeared to be running normally and we decided to let it be and wait until Monday.

10 hours goes by, it is now Sunday morning and the problem returns.  First host loses storage (we’re doing NFS over 10GbE here), then two more…until all 9 in this cluster are pretty much toast.  I decide to open a ticket with VMware.  Wouldn’t you know, there is a potential known bug and resolution.

Bug 496013

Description: Some NetXen based 10GbE cards using the unm_nic and nx_nic drivers sometime flood the network with pause frames causing the port to become disabled.

Resolution: NetXen believes upgrading the firmware to version 4.0.516 will resolve the problem.

I’ve gone ahead and patched 4 of the hosts with this new firmware, so far it has been stable (knock on wood).   I’ll let you know if something happens.

Checking which version of the firmware you’re running is simple. From a command-line (ESX or ESXi hidden CLI), type ethtool -i <vmnic#> (replace vmnic# with the alias to the vmnic you’d like to check).  You should see output similar to:

driver: nx_nic

version: 4.0.301

firmware-version: 4.0.406

bus-info: 0000:07:00.0

Update - Utility CD with firmware patch now included…

As you can see above, the firmware is out of date. To update the firmware you will need to boot from a Linux utility CD that has the appropriate driver, you then run a firmware update utility provided by HP.  To make this process easy I have created a bootable SLAX utility CD with the drivers pre-loaded. You can download the ISO from here (file temporarily removed). Once booted run the installer located in the root filesystem (ie: ./CP011471.scexe).

Let me know if you have any questions.

Posted under ESX 3.5 Tips, ESXi 3.5 Tips, Networking, Storage, vSphere

Last week I came across a problem that really stumped me, it even stumped the Tier-1 and Tier-2 support at VMware.  I’m posting the symptoms on here in a hope that someone else has experienced this issue and can share some light.

How about a little background on the environment, vCenter Server 4.0 U1 and multiple ESX(i) hosts (3.5, 4.0, 4.0 U1).   The vCenter Server as well as a number of ESXi 4.0 hosts were upgraded to U1 a couple days after it was released,  this problem however happened ~8 days after the upgrade.

Symptom 1: All ESX(i) hosts disconnect from vCenter Server, however, they are still online and no VMs went down.  Within 15 minutes all hosts appear to be reconnected.

Symptom 2: After the hosts reconnect, the ESX hosts appear to be functioning normally. However, the ESXi hosts display an error on the Overview tab as well as in the Events tab; “Unable to Synchronize with host that is unavailable.”

Symptom 3: Random VMotions start, for no apparent reason (DRS engaged, yet no constraints causing DRS to be invoked).  However, these VMotions fail at 10% due to the fact that the source and destination host is not available.

Symptom 4: /var/log/messages file displays errors with keywords: [VpxdVmomi] Error getting vpxa info: SSL Exception: Unexpected EOF From hosts, blacklisting showing up.   — I apologize for paraphrasing.

So, all this starts happening and I start investigating….pulling logs, restarting vCenter, and just sit there stumped.  I did notice that the rui.crt on the vCenter server expired, but back in 2008.  I went ahead and renewed the certificate and even restarted the entire vCenter server.  No luck.  I engaged VMware Support and their Tier-1 and Tier-2 support were stumped,  nothing even showed up in their internal database on this issue.

Then it all disappeared.  Roughly 90 minutes after it started, the problem just went away and everything was good.

Have you seen this issue?  What were your troubleshooting steps?  Did you resolve it or figure out the resolution?

Posted under vSphere

Sometime yesterday ESX(i) Update 5 finally hit VMware Update Manager, about 2 days after the official announcement and release on the VMware website. This announcement includes updates for ESX, ESXi and vCenter Server. In addition to Update 5 being released there were about 20 additional updates made available for ESX(i), including 16 which were marked as critical.

The following information provides highlights of some of the enhancements available in this release of VMware ESX Server, this information can be found in the VMware ESX(i) 3.5 U5 Release Notes:

Enablement of Intel Xeon Processor 3400 Series– Support for the Intel Xeon processor 3400 series has been added. Support includes Enhanced VMotion capabilities. For additional information on previous processor families supported by Enhanced VMotion, see Enhanced VMotion Compatibility (EVC) processor support (KB 1003212).

Driver Update for Broadcom bnx2 Network Controller– The driver for bnx2 controllers has been upgraded to version 1.6.9. This driver supports bootcode upgrade on bnx2 chipsets and requires bmapilnx and lnxfwnx2tools upgrade from Broadcom. This driver also adds support for Network Controller - Sideband Interface (NC-SI) for SOL (serial over LAN) applicable to Broadcom NetXtreme 5709 and 5716 chipsets.

Driver Update for LSI SCSI and SAS Controllers – The driver for LSI SCSI and SAS controllers is updated to version 2.06.74. This version of the driver is required to provide a better support for shared SAS environments.

Newly Supported Guest Operating Systems – Support for the following guest operating systems has been added specifically for this release:

For more complete information about supported guests included in this release, see the VMware Compatibility Guide: http://www.vmware.com/resources/compatibility/search.php?deviceCategory=software.

• Windows 7 Enterprise (32-bit and 64-bit)
• Windows 7 Ultimate (32-bit and 64-bit)
• Windows 7 Professional (32-bit and 64-bit)
• Windows 7 Home Premium (32-bit and 64-bit)
• Windows 2008 R2 Standard Edition (64-bit)
• Windows 2008 R2 Enterprise Edition (64-bit)
• Windows 2008 R2 Datacenter Edition (64-bit)
• Windows 2008 R2 Web Server (64-bit)
• Ubuntu Desktop 9.04 (32-bit and 64-bit)
• Ubuntu Server 9.04 (32-bit and 64-bit)

Newly Supported Management Agents – See VMware ESX Server Supported Hardware Lifecycle Management Agents for current information on supported management agents.

Newly Supported Network Cards –This release of ESX Server supports HP NC375T (NetXen) PCI Express Quad Port Gigabit Server Adapter.

Newly Supported SATA Controllers – This release of ESX Server supports the Intel Ibex Peak SATA AHCI controller.

In addition to the enhancements found in ESX(i) 3.5 U5, there is also one lonely enhancement made to vCenter Server 2.5 U5:

Support for High Consolidation in VMware HA Clusters- VirtualCenter 2.5 Update 5 includes significant performance and scalability improvements to VMware HA. Use VirtualCenter 2.5 Update 5 for environments with more than 35 virtual machines per host in an HA cluster.
For information on the ESX Server host settings required for this scalability improvement, see ESX Server host settings required for environments with up to 80 virtual machines per host in an HA Cluster (KB 1012002).

Updating your ESX servers can and should be done with VMware Update Manager. To upgrade your vCenter Server installation you’ll need to download the installation ISO or ZIP from the VMware website and perform an in-place upgrade. Be sure to create a backup of your vCenter Server database then follow the steps in the Installation Guide.

Posted under ESX 3.5 Tips, ESXi 3.5 Tips, vCenter

I’m proud to say that HP has full support for ESXi 4 installed on SD or USB Flash memory.  They even offer the installable ISO pre-built with their CIM providers, available here. There is a catch though, they only support their approved SD or USB Flash memory….sorry, but you cannot BYOF (Bring Your Own Flash).  This is reflected on their download page:

HP VMware ESXi 4.0 solution requires the following:

VMware ESXi 4.0 free downloadable product. To upgrade your license to Enterprise Plus, purchase HP VMware ESXi 4 product 571979-B21.

HP ESXi 4 CIM Providers.

Registration on VMware ESXi hypervisor web page to obtain permanent license serial number.

Acquire your choice of HP supported and qualified media - any HP supported hard drive, USB or SD Flash devices listed below.

Please note that all devices will need to be imaged for ESXi 4.0.

Information about HP supported SD card and USB Flash drive:

• Supported SD card**:
  HP 4GB SD Flash Media
  HP Part Number 580387-B21
  [spare kit part number 583306-001]
• Supported USB Flash Drive**:
  HP 4GB USB Flash Media Drive Key
  HP Part Number 580385-B21
  [Spare kit part number 583307-001]

*Must be purchased separately.
**HP VMware ESXi 4.0 does not support any other USB or SD flash devices

Seems simple, doesn’t it?   Actually it works great and I have put a few dozen servers into production with this method, the best part is no more local HDD which saves even more power!

One question I often get is,  why do you need a 4GB drive, I though ESXi was only 32MB?  Well, even though it is true that ESXi is only 32MB you still need adequate space for the VI Client, VMware Tools (all Operating systems) and upgrade space (for future ESXi patches and releases).  Using a 4GB drive ensures that you’ll have enough space for everything.

There is one problem currently that I am facing, HP has both the SD Card and USB Flash Drive on back-order and there is no expected ETA for either of them!  This has delayed a major project I’m working on and I’ve had to resort to using temporary “junk” USB drives to get the customer by in the mean time.

If someone from HP is reading this, PLEASE get your OEM to produce some new ones ASAP!

Posted under Storage, vSphere

This last weekend I was reminded of a RCLI command that creates a backup of your ESXi Host configuration.  A client had an ESXi host where the USB drive failed.  The system didn’t entirely crash, it pinged so HA didn’t kick-in however hostd and vpxa weren’t responding so management from vCenter was impossible. Performing a reboot of the host proved it was a USB failure when it failed to boot.

So, HA kicked in finally when the host was powered off - however the remaining hosts in their cluster did not have adequate resources to power everything back on.  We needed to get this failed ESX host back online and quick!

Brought in new USB stick with fresh ESXi 4.0 installed, plugged in and powered up.  In the TUI configured the Management Port IP address/gateway/DNS, etc.    But now what, all of the (NFS) Datastores, vSwitch configuration and advanced settings were toast…and they didn’t have Enterprise Plus.

Say Hello to vicfg-cfgbackup.pl — a RCLI command which could be found in the vMA.  This utility will allow you to create a backup and restore a full ESXi configuration.  Luckily we had this (backup) command running on a nightly basis so we knew the configuration was complete.  A simply command and the host was online.

Seeing that ESX is going to be phased out, this RCLI command is a valid alternative for those ESXi users that do not have Enterprise Plus licensing.

The commands vicfg-cfgbackup.pl (esxcfg-cfgbackup.pl) allow you to backup and restore the configuration of your ESXi host.

To backup the host you would run the command.

vicfg-cfgbackup.pl –server <server_name> -s <backup_file_name>

To restore your backup configuration to your host you would run. This will cause the host to reboot once the process is complete. NOTE: The host must be in Maintenance Mode for this to work. The backup configuration must also match the patch level of the ESXi install. You can add a -f to force if needed.

vicfg-cfgbackup.pl –server <server_name> -l <backup_file_name>

Posted under Backup & Recovery, ESXi 3.5 Tips, vSphere

I had a big problem Monday AM on one of my core SAP VM instances, that also happens to have a SQL DB server on it. Our VCB process finishes up on late Sunday night, if you’re not aware of how VCB works, it basically creates a snapshot of the Virtual Machine, then mounts the now readable VMDK parent to a proxy server where your backup agent resides. Once the backup is complete the snapshot is committed.  This wasn’t the case Monday AM — the VM crashed and I was paged. Snapshot didn’t commit, parent VMDK could not be found, had to manually set Parent CID in the delta VMDK file then finally when I got it back online the SQL DB was corrupt :( — luckily I had a full SQL backup from the night before.

This is where VMware KB 1007969 comes into the story…

Read More…

Posted under ESX 3.5 Tips, ESXi 3.5 Tips, Storage

The past few weeks I’ve been working with a customer that only has ESXi, Veeam FastSCP has came in really handy lately! I can copy files between systems or between a system and my desktop, the flexibility it creates is amazing.  It also attempts to utilize the VMKernel API first, if it cannot make a connection that way it will fallback to SSH.  Did I forget to mention, its blazing fast!

Well, now is your chance to get Veeam Backup and Replication for 50% off! Don’t miss this great opportunity to get a solid backup utility for your ESX(i) infrastructure.  Oh yeah, it also works with ESXi Free Edition!

Through June 15, Veeam Software is extending a special upgrade offer that gives you budget relief, freeing you to license the new #1 for VMware backup and replication. Veeam has the functionality, reliability and performance you need now.

You can have both backup and replication in one product - now 50% off when you provide proof of purchase of a competing product. Learn more >>

Posted under Backup & Recovery