I had a very fun weekend. It started at 4am Saturday with a migration of ~125 virtual machines from an old AMD based environment to a new Intel Nehalem based environment. Who could’ve known that within a few hours all hell would’ve broken loose.

Enter in problem of network flooding from the NetXen based HP branded NC522SFP.  Because all of the 10GbE ports from the (9) new ESXi servers were creating thousands of pause frames on the Cisco Nexus 5020 switches, I thought originally that it was an issue on the switch.  Talks with Cisco revealed nothing.  We attempted to disconnect one of the connected ports (each ESXi host is dual connected into a pair of N5Ks using vPC) to remove a potential spanning tree loop….no dice.

A reboot of the host resolved the problem, things appeared to be running normally and we decided to let it be and wait until Monday.

10 hours goes by, it is now Sunday morning and the problem returns.  First host loses storage (we’re doing NFS over 10GbE here), then two more…until all 9 in this cluster are pretty much toast.  I decide to open a ticket with VMware.  Wouldn’t you know, there is a potential known bug and resolution.

Bug 496013

Description: Some NetXen based 10GbE cards using the unm_nic and nx_nic drivers sometime flood the network with pause frames causing the port to become disabled.

Resolution: NetXen believes upgrading the firmware to version 4.0.516 will resolve the problem.

I’ve gone ahead and patched 4 of the hosts with this new firmware, so far it has been stable (knock on wood).   I’ll let you know if something happens.

Checking which version of the firmware you’re running is simple. From a command-line (ESX or ESXi hidden CLI), type ethtool -i <vmnic#> (replace vmnic# with the alias to the vmnic you’d like to check).  You should see output similar to:

driver: nx_nic

version: 4.0.301

firmware-version: 4.0.406

bus-info: 0000:07:00.0

Update - Utility CD with firmware patch now included…

As you can see above, the firmware is out of date. To update the firmware you will need to boot from a Linux utility CD that has the appropriate driver, you then run a firmware update utility provided by HP.  To make this process easy I have created a bootable SLAX utility CD with the drivers pre-loaded. You can download the ISO from here (file temporarily removed). Once booted run the installer located in the root filesystem (ie: ./CP011471.scexe).

Let me know if you have any questions.

Posted under ESX 3.5 Tips, ESXi 3.5 Tips, Networking, Storage, vSphere

I was just invited by Cisco to be part of the BETA for the next release of the Nexus 1000V.  I’ll provide more details about the BETA after I install and test it out, but here are some teasers;

And guess what…?!   You can be part of the BETA too! Check out this link and sign up to be part of the distributed virtual switch revolution!

Posted under Networking, vSphere

Doing some speed tests of IP based storage gave me some good results. First I enabled Jumbo frames on my dedicated IP storage Ethernet card, then I set the MTU on the vSwitch and the VMKnic to 9000 (Scott Lowe has a great write-up on doing this on his website). I then mounted a NFS volume as Datastore and did a copy of a VMDK file which currently resides on a 7200rpm SATA 3 drive to the mounted NFS volume.

I copied the 8GB file in under 2 minutes and my average MbTX/s was around 525Mb (65MBps) - keep in mind, this was a copy from a local SATA disk (low IOPS) and I was able to get around 65MBps — not bad!

The biggest thing to remember, VMDK access needs low latency…not high throughput — this is why NFS has become so popular for VMDK storage.

Posted under Networking, Storage

Jason Boche posted this network diagram on VI3, it shows exactly what ports VC, ESX and other services needed for a functioning enterprise use.   This diagram is extremely helpful in troubleshooting network issues and also in helping put your companies security department at ease.  Thanks Jason!

Click to Enlarge

Posted under ESX 3.5 Tips, ESXi 3.5 Tips, Networking, Security, VMware

Today I got into a heated discussion with a “Virtualization Expert” at Gartner today about the risks associated with virtualizing your DMZ, primarily into the same environment as your non-DMZ servers.

Read More…

Posted under Networking, Security, VMware

In the event you need to change the VLAN of any of your Port Groups here is some good instruction on doing just that. Remember, VLANs are configured at the Port Group level and not the vSwitch level. This allows you to trunk as many VLANs you need to your vSwitch’s then create dedicated Port Groups for each of those Virtual LANs.

Read More…

Posted under ESX 3.5 Tips, ESXi 3.5 Tips, Networking, VMware

Actually this is not that difficult, but remember you will require console access to the server. Be sure to put the machine in Maintenance Mode then Disconnect it from Virtual Center. Then connect to the console of the ESX host;

Read More…

Posted under ESX 3.5 Tips, ESXi 3.5 Tips, Networking, VMware