In this video we’re going to cover the installation, configuration and usage of the VMware vCenter 5.0 Server Appliance (vCSA). The vCSA is a brand new production ready Virtual Appliance that allows you to stand up vCenter Server in literally a few minutes. Once you watch the video you’re going to be like, “Hey Rick, that was more than 5 minutes!”.  For that I do apologize, but when you do watch it you will realize we’re doing a lot more than just installing vCenter 5.0.

First a little disclaimer. vCSA is not for everyone, but in my opinion it should definitely be looked at and should be leveraged wherever it can. vCSA is obviously the direction of where the vCenter Server product is going and hopefully relatively soon it should be at par with its Windows based big brother.

So, why isn’t it for everyone?

As of right now it does not have support for integration with VMware Update Manager (VUM), VMware vCenter Linked-Mode, VMware vSphere Storage Appliance (vSA), VMware vCenter Heartbeat and VMware View Composer. Another concern you may have is that it’s embedded database option (based on DB2) is limited to 5 Hosts and 50 Virtual Machines. Think of the embedded option to be similar to the SQL Express Option in vCenter Server for Windows, great for POC, Demo, Test and extremely small SMB situations, but not practical for production. The final nail in the coffin might be that it only supports Oracle to offer external DB functionality.

Some of those constraints are not going to be avoidable, for example if you require more than 1,000 hosts or 10,000 powered on virtual machines you’re obviously going to need Linked-Mode and the Windows based vCenter Server. If you’re looking to deploy a VDI solution based on VMware View, you’re going to need the Windows based vCenter Server as well. But, if you’re like the majority of VMware vSphere customers, have less than 1,000 hosts, are confident in VMware DRS and HA to protect your vCSA and are OK with the fact that you need Oracle for the external database (which you can virtualize as well)….the vCSA might be for you!

One last thing I wanted to comment on was VMware Update Manager, in my opinion the lack of VUM support for the vCSA might not be that big of an issue, and here’s why; With the introduction of vSphere 5.0, VMware also introduced a few new features, Auto Deploy and Image Builder. These features tied together with Host Profiles truly enable the concept of stateless ESXi. My thought is, if you need to update your ESXi host, simply update the Auto Deploy rule and reboot the machine. Upon the next boot it will automatically be updated and configured properly.  Obviously VUM does a lot more than just ESXi patching, but again, for the majority of vSphere customers they’d be just fine with Auto Deploy.

So have a view of this video to see just how easy it is. I have sped up some portions of the video, specifically the loading of the vSphere Client as well as the deployment of the vCSA OVF template. Also, I suggest watching the video in full-screen mode by clicking the icon on the bottom right of the video. If for whatever reason the video isn’t displaying, you can also use the following link to view; http://youtu.be/o2f1b1vYSis

Posted under vCenter, vSphere

This is going to be the first of many in my vSphere 5 Video Series where I’ll cover the basics to getting vSphere 5.0 installed, configured and operating. In this video see just how easy it is to do a bare-metal installation of ESXi 5.0. Out of all of the install and upgrade options available for vSphere 5.0 this by far is the easiest and cleanest method in my opinion.

Upgrading can be extremely easy as well, by leveraging VMware Update Manager (VUM) it allows existing configurations to be migrated and will even allow you to migrate from ESX to ESXi. One thing to keep in mind when doing a migration with VUM from vSphere 4 to vSphere 5 is that if you are using the ESX edition of vSphere 4 and have custom scripts, agents or modules loaded into ESX those will not be migrated into ESXi 5.0.

Whatever your situation might be, my recommendation has always been to do a fresh installation of ESXi then leverage Host Profiles to push the configuration to the host. Even if you’re not an Enterprise Plus customer you can still get all of the benefits, like Host Profiles, free for 60 days by simply not licensing the product. Remember, you must license before the 60 days are up to avoid any service disruption.

Another great feature of vSphere 5 is Image Builder and Auto Deploy, I’ll cover Auto Deploy into more detail later, but with Image Builder you can build custom ESXi builds that include third party drivers and other custom data. Don’t worry though, you can still do custom installations with kickstart if you’d like, but after you see Auto Deploy you’re not going to want to.

So have a view of this video to see just how easy it is. I have sped up some portions of the video, specifically the blade server booting, the hardware discovery process and actually installation portion. Also, I suggest watching the video in full-screen mode by clicking the icon on the bottom right of the video. If for whatever reason the video isn’t displaying, you can use the following link to view; http://youtu.be/aN9mc9YNiC0

Posted under vSphere

vmware has recently released a patch for their vSphere 4.0 product line, which affects both ESX and ESXi.

Details from vmware;

We are pleased to inform you that a new VMware ESX 4.0 Patch is available as of April 28, 2011.

Improvements included in this patch:

• An update for the Certificate Revocation List (CRL) to revoke an RSA key that HP uses for code signing certain software components
• Remediation of a denial of service possibility. By sending malicious network traffic an attacker could exhaust the available sockets which would prevent further connections to the host
• Refinements in handling of shared folders

Detailed information regarding resolved and known issues and enhancements can be found at ESX 4.0 Patch Release Notes

• http://kb.vmware.com/kb/1037260
• http://kb.vmware.com/kb/1037261

VMware ESX 4.0 Patch is available for download at:
Download VMware ESX 4.0 Patch http://www.vmware.com/patch/download/.

Thanks,

VMware vSphere Product Management Team

One of the patches included (ESX400-201104401-SG for ESX and ESXi400-201104401-SG for ESXi) resolves a couple different issues, one updates the Certification Revocation List (CRL) to revoke a key that HP uses for code-signing certain software components. HP server contains a new key pair and has re-signed the affected software components with the new key. What this means is that if you apply this patch on a HP server and you are using specific HP management agents (like the HP Management Agent for VMware ESX 4.x) you will need to download the software with the updated key and re-install it.

The other fix within the above mentioned patch resolves a potential denial of service attack against the vmkernel over it’s management interface. When an attacker exhausts all available sockets the ESX(i) host will become inaccessible via vCenter or the vSphere client. Virtual Machines will continue to run and have network connectivity, but the ESX(i) host may need to be rebooted in order to be able to connect to the machine again. The ESX(i) system might intermittently lose connectivity caused by applications that do not correctly close sockets. If this occurs, an error message similar to the following might be written to the vpxa log file:
socket() returns -1 (Cannot allocate memory)
An error message similar to the following might be written to the VMkernel log file:
socreate(type=2, proto=17) failed with error 55
The Common Vulnerabilities and Exposures Project (cve.mitre.org) has assigned the name CVE-2011-1785 to this issue. More information on this patch can be found in KB 1037258 (ESX) and KB 1037259 (ESXi).

Another patch, specific to ESXi (ESXi400-201104402-BG), has also been released. The only information on this patch can be found in KB 1037553 which states “This patch improves the way shared folders are handled.”.

Posted under vSphere

Yesterday, February 10th, VMware made available for general consumption U1 to their vSphere 4.1 product line. This update includes all prior patches as well as a number of new enhancements to the vSphere product suite.

This release provides the following improvements, I’ve included some notes along with the high-level updates:

VMware ESX/ESXi

• Support for up to 160 logical processors
  • Prepared for the release of the Westmere-EX processor
• Inclusion of additional drivers
  • 3ware and Neterion drives are now included
• Enablement of Intel Trusted Execution Technology (ESXi only)
  • More information on this can be found in KB 1033811 
• Additional guest operating system support
  • Provides support for RHEL 6, RHEL 5.6, SLES 11 SP1 for VMware, Ubuntu 10.10, and Solaris 10 Update 9 guest operating systems
• Bug and security fixes
  • A listing of resolved issues can be found in the release notes

VMware vCenter

• Additional guest operating system customization support
  • Windows 7 SP1, Windows Server 2008 R2 SP1, RHEL 6, RHEL5.5
• Additional vCenter Server database support
  • SQL 2008 R2, SQL 2005 SP3, Oracle 11g R2, DB2 9.7.2
• Bug and security fixes
  • A listing of resolved issues can be found in the release notes

VMware vCenter Update Manager

• The VMware vCenter Update Manager Utility to help users reconfigure the setup of Update Manager.
• Bug and security fixes.

VMware vCenter Orchestrator

• Bug Fixes

For additional details regarding the new fixes and improvements, please refer to the following release notes:
VMware ESX
VMware ESXi
VMware vCenter

VMware vSphere 4.1 Update 1 is available for download from the VMware website as well as through VMware Update Manager (for your ESX and ESXi hosts). Keep in mind, the only way you can switch from ESX to ESXi is to do a fresh install. I’d highly recommend using Host Profiles to make this migration quicker and easier.

Jason Boche has already noted one potential issue with using VMware Update Manager (VUM) to upgrade from 4.1 to 4.1 U1, check that out here.

Posted under vSphere

Just about an hour ago VMware lifted the NDA on VMware vSphere 4.1 and made available all of the information on this latest release as well as the bits for download available to the public.

I will be covering a more in-depth review on this latest release really soon but I did want my readers to know that the bits are available for download from the VMware website.

http://downloads.vmware.com/d/info/datacenter_downloads/vmware_vsphere_4/4

The upgrade to ESX(i) 4.1 should be relatively easy by using traditional update methods such as VMware Update Manager (VUM).  However, the upgrade to vCenter Server 4.1 is more of a migration since it will only support a full 64-bit environment. Still don’t fret, a vCenter server migration is pretty simple just make sure you have a FULL backup of your vCenter Server database.

Also a little FYI….rumor has it that this will be the final build containing a full ESX install (Service Console). Today might be a good day to start planning that migration to ESXi.

Posted under vCenter, vSphere

You’ve guessed it, VMware has released Update 2 of their flagship bare-metal virtualization product ESX and ESXi. This update addresses a number of issues found since the release of Update 1 as well as a number of enhancements, such as:

Enablement of Fault Tolerance Functionality for Intel Xeon 56xx Series processors— vSphere 4.0 Update 1 supports the Intel Xeon 56xx Series processors without Fault Tolerance. vSphere 4.0 Update 2 enables Fault Tolerance functionality for the Intel Xeon 56xx Series processors.

Enablement of Fault Tolerance Functionality for Intel i3/i5 Clarkdale Series and Intel Xeon 34xx Clarkdale Series processors— vSphere 4.0 Update 1 supports the Intel i3/i5 Clarkdale Series and Intel Xeon 34xx Clarkdale Series processors without Fault Tolerance. vSphere 4.0 Update 2 enables Fault Tolerance functionality for the Intel i3/i5 Clarkdale Series and Intel Xeon 34xx Clarkdale Series processors.

Enablement of IOMMU Functionality for AMD Opteron 61xx and 41xx Series processors— vSphere 4.0 Update 1 supports the AMD Opteron 61xx and 41xx Series processors without input/output memory management unit (IOMMU). vSphere 4.0 Update 2 enables IOMMU functionality for the AMD Opteron 61xx and 41xx Series processors.

Enhancement of the esxtop/resxtop utility— vSphere 4.0 Update 2 includes an enhancement of the performance monitoring utilities, esxtop and resxtop. The esxtop/resxtop utilities now provide visibility into the performance of NFS datastores in that they display the following statistics for NFS datastores: Reads/s, writes/s, MBreads/s, MBwrtn/s, cmds/s, GAVG/s(guest latency).

Additional Guest Operating System Support— ESX/ESXi 4.0 Update 2 adds support for Ubuntu 10.04. For a complete list of supported guest operating systems with this release, see the VMware Compatibility Guide.

Read More…

Posted under vSphere

As handful of patches have just been released by VMware for their flagship bare-metal virtualization products ESX and ESXi.

With no surprise to me the majority of the patches are for ESX and relate to security flaws and vulnerabilities found within the Service Console.  Keep in mind these vulnerabilities in no-way mean the virtual machines being hosted are at risk. These patches are typically for underlying services that the Service Console rely on, such as openssl, java, gzip and ntp. Sometimes these patches also resolve issues on how the Service Console communicates with the vmkernel layer as well as system devices.

Two of the patch bundles for ESXi share some common fixes with it’s ESX brother which cover a NTP vulnerability, a shared interrupt issue between the vmkernel and console as well as a patch that properly enables quiescing utilizing the Microsoft Windows VSS components found in Windows 2008 R2 and Windows 7.

More information on these patches can be found by reviewing the individual bundles;

ESX 4.0 - ESX400-201005001
Includes 9 updates, including fixes for NTP, gzip, bind, vmkernel, krb5, webCenter, Expat, sudo and gcc.

ESXi 4.0 - ESXi400-201005001
Includes two updates, ESXi400-201005401-SG for the ESXi firmware and ESXi400-201005402-BG for VMware Tools.

For updating your ESX(i) hosts, simply use Update Manager or download the patches from the VMware website and use the Host Update Utility to perform these updates.

Posted under vSphere

I had a very fun weekend. It started at 4am Saturday with a migration of ~125 virtual machines from an old AMD based environment to a new Intel Nehalem based environment. Who could’ve known that within a few hours all hell would’ve broken loose.

Enter in problem of network flooding from the NetXen based HP branded NC522SFP.  Because all of the 10GbE ports from the (9) new ESXi servers were creating thousands of pause frames on the Cisco Nexus 5020 switches, I thought originally that it was an issue on the switch.  Talks with Cisco revealed nothing.  We attempted to disconnect one of the connected ports (each ESXi host is dual connected into a pair of N5Ks using vPC) to remove a potential spanning tree loop….no dice.

A reboot of the host resolved the problem, things appeared to be running normally and we decided to let it be and wait until Monday.

10 hours goes by, it is now Sunday morning and the problem returns.  First host loses storage (we’re doing NFS over 10GbE here), then two more…until all 9 in this cluster are pretty much toast.  I decide to open a ticket with VMware.  Wouldn’t you know, there is a potential known bug and resolution.

Bug 496013

Description: Some NetXen based 10GbE cards using the unm_nic and nx_nic drivers sometime flood the network with pause frames causing the port to become disabled.

Resolution: NetXen believes upgrading the firmware to version 4.0.516 will resolve the problem.

I’ve gone ahead and patched 4 of the hosts with this new firmware, so far it has been stable (knock on wood).   I’ll let you know if something happens.

Checking which version of the firmware you’re running is simple. From a command-line (ESX or ESXi hidden CLI), type ethtool -i <vmnic#> (replace vmnic# with the alias to the vmnic you’d like to check).  You should see output similar to:

driver: nx_nic

version: 4.0.301

firmware-version: 4.0.406

bus-info: 0000:07:00.0

Update - Utility CD with firmware patch now included…

As you can see above, the firmware is out of date. To update the firmware you will need to boot from a Linux utility CD that has the appropriate driver, you then run a firmware update utility provided by HP.  To make this process easy I have created a bootable SLAX utility CD with the drivers pre-loaded. You can download the ISO from here (file temporarily removed). Once booted run the installer located in the root filesystem (ie: ./CP011471.scexe).

Let me know if you have any questions.

Posted under ESX 3.5 Tips, ESXi 3.5 Tips, Networking, Storage, vSphere

Last week I came across a problem that really stumped me, it even stumped the Tier-1 and Tier-2 support at VMware.  I’m posting the symptoms on here in a hope that someone else has experienced this issue and can share some light.

How about a little background on the environment, vCenter Server 4.0 U1 and multiple ESX(i) hosts (3.5, 4.0, 4.0 U1).   The vCenter Server as well as a number of ESXi 4.0 hosts were upgraded to U1 a couple days after it was released,  this problem however happened ~8 days after the upgrade.

Symptom 1: All ESX(i) hosts disconnect from vCenter Server, however, they are still online and no VMs went down.  Within 15 minutes all hosts appear to be reconnected.

Symptom 2: After the hosts reconnect, the ESX hosts appear to be functioning normally. However, the ESXi hosts display an error on the Overview tab as well as in the Events tab; “Unable to Synchronize with host that is unavailable.”

Symptom 3: Random VMotions start, for no apparent reason (DRS engaged, yet no constraints causing DRS to be invoked).  However, these VMotions fail at 10% due to the fact that the source and destination host is not available.

Symptom 4: /var/log/messages file displays errors with keywords: [VpxdVmomi] Error getting vpxa info: SSL Exception: Unexpected EOF From hosts, blacklisting showing up.   — I apologize for paraphrasing.

So, all this starts happening and I start investigating….pulling logs, restarting vCenter, and just sit there stumped.  I did notice that the rui.crt on the vCenter server expired, but back in 2008.  I went ahead and renewed the certificate and even restarted the entire vCenter server.  No luck.  I engaged VMware Support and their Tier-1 and Tier-2 support were stumped,  nothing even showed up in their internal database on this issue.

Then it all disappeared.  Roughly 90 minutes after it started, the problem just went away and everything was good.

Have you seen this issue?  What were your troubleshooting steps?  Did you resolve it or figure out the resolution?

Posted under vSphere

Sometime yesterday ESX(i) Update 5 finally hit VMware Update Manager, about 2 days after the official announcement and release on the VMware website. This announcement includes updates for ESX, ESXi and vCenter Server. In addition to Update 5 being released there were about 20 additional updates made available for ESX(i), including 16 which were marked as critical.

The following information provides highlights of some of the enhancements available in this release of VMware ESX Server, this information can be found in the VMware ESX(i) 3.5 U5 Release Notes:

Enablement of Intel Xeon Processor 3400 Series– Support for the Intel Xeon processor 3400 series has been added. Support includes Enhanced VMotion capabilities. For additional information on previous processor families supported by Enhanced VMotion, see Enhanced VMotion Compatibility (EVC) processor support (KB 1003212).

Driver Update for Broadcom bnx2 Network Controller– The driver for bnx2 controllers has been upgraded to version 1.6.9. This driver supports bootcode upgrade on bnx2 chipsets and requires bmapilnx and lnxfwnx2tools upgrade from Broadcom. This driver also adds support for Network Controller - Sideband Interface (NC-SI) for SOL (serial over LAN) applicable to Broadcom NetXtreme 5709 and 5716 chipsets.

Driver Update for LSI SCSI and SAS Controllers – The driver for LSI SCSI and SAS controllers is updated to version 2.06.74. This version of the driver is required to provide a better support for shared SAS environments.

Newly Supported Guest Operating Systems – Support for the following guest operating systems has been added specifically for this release:

For more complete information about supported guests included in this release, see the VMware Compatibility Guide: http://www.vmware.com/resources/compatibility/search.php?deviceCategory=software.

• Windows 7 Enterprise (32-bit and 64-bit)
• Windows 7 Ultimate (32-bit and 64-bit)
• Windows 7 Professional (32-bit and 64-bit)
• Windows 7 Home Premium (32-bit and 64-bit)
• Windows 2008 R2 Standard Edition (64-bit)
• Windows 2008 R2 Enterprise Edition (64-bit)
• Windows 2008 R2 Datacenter Edition (64-bit)
• Windows 2008 R2 Web Server (64-bit)
• Ubuntu Desktop 9.04 (32-bit and 64-bit)
• Ubuntu Server 9.04 (32-bit and 64-bit)

Newly Supported Management Agents – See VMware ESX Server Supported Hardware Lifecycle Management Agents for current information on supported management agents.

Newly Supported Network Cards –This release of ESX Server supports HP NC375T (NetXen) PCI Express Quad Port Gigabit Server Adapter.

Newly Supported SATA Controllers – This release of ESX Server supports the Intel Ibex Peak SATA AHCI controller.

In addition to the enhancements found in ESX(i) 3.5 U5, there is also one lonely enhancement made to vCenter Server 2.5 U5:

Support for High Consolidation in VMware HA Clusters- VirtualCenter 2.5 Update 5 includes significant performance and scalability improvements to VMware HA. Use VirtualCenter 2.5 Update 5 for environments with more than 35 virtual machines per host in an HA cluster.
For information on the ESX Server host settings required for this scalability improvement, see ESX Server host settings required for environments with up to 80 virtual machines per host in an HA Cluster (KB 1012002).

Updating your ESX servers can and should be done with VMware Update Manager. To upgrade your vCenter Server installation you’ll need to download the installation ISO or ZIP from the VMware website and perform an in-place upgrade. Be sure to create a backup of your vCenter Server database then follow the steps in the Installation Guide.

Posted under ESX 3.5 Tips, ESXi 3.5 Tips, vCenter