NFS Datastores and what was their BIG issue…

This all started back about a year ago when I decided to move my datastores from Fibre Channel to NFS. The data was already on a NetApp FAS960c so I was enjoying thin provisioning and snapshots…but I wanted more!

We recently installed a new FAS6030 and was excited to try A-SIS (Data deduplication), so I decided that this was the perfect time to move all of our data over to the new filer and also change architectures and make the jump to NFS. I built out the vSwitch that has our VMKernel portgroup to include a few more NICs and proceeded to follow the NetApp & VMware VI3 Storage Best Practices Guide (TR3428).

Then I got to page number 9 (FYI, at the time this was version 3.1 of the document…I’ll get into why this is important later), where it proceeded to tell me to set NFS.LockDisable to 1 (0 is the default). Not really thinking anything about it I did it, which means I missed a unspoken Sys. Admin rule — always research suggested changes that could potentially affect other systems.

So, we were up and running…no cares or worries in the world! Everything was running great, I loved NFS and how easy it was to provision additional space on the fly, mount to other systems, etc…. (I still love it actually, it offers a ton of flexibility and there are a ton of cool things you can do to manage/backup/restore your data). It took almost a year but we had what appeared to be an outage, I couldn’t really put my finger on it at the time but VMware HA definitely kicked in (after doing some forensics after the fact I found that there wasn’t a hardware failure, actually one of our main core Cisco Switches did a spanning tree convergence which brought the entire network to a halt for roughly 25-30 seconds…JUST enough time for HA to think it was isolated).

So, what I found out from this little experience is that we crucially needed our NFS Locks to be in place and it was a VERY bad thing that we did by disabling them. What we saw happening is that all of the other primary nodes in the HA cluster (the first 4 hosts are primary by default) attempted to start the VMs when they thought the “node” failed. Ultimately it was very close to ALL of our VMs starting up on our first four hosts, and with locking disabled they were able to all start the VM and read/write the VMDK. Interestingly enough, NetApp removed the recommendation to set NFS.LockDisable=1 in a later revision of the TR3428 document.

So…why did NetApp initially recommend disabling NFS Locks? Well, in their initial testing the removal of VMware Snapshots took a long time over the NFS client (has to do with VMDK quiescing, lock removals, appending, etc… the one area NFS lacks because it is not a block level protocol). Their fix for the best practice guide was to remove NFS locks, but it sounds now like they didn’t do enough research before recommending this (and neither did I before implementing).

Until recently VMware has not had a fix for this “slow snapshot removal” issue, but thankfully they have released patch ESX350-200808401-BG which includes a fix just for that. But, it is a multi-step fix, so you will need to do the following;

1.) Download patch ESX350-200808401-BG.

2.) Identify an ESX server to apply the patch to.

3.) Using VMotion migrate the running VMs to other ESX nodes.

4.) Install this patch on the ESX server identified in step 2.

5.) In the adv. configuration settings ensure that NFS file locking is enabled;
NFS.LockDisable=0

advset.png
(Click on Image to Enlarge)
(this can also be done from the CLI with esxcfg-advcfg)

6.) Edit the following file: /etc/vmware/config file and add the line:
prefvmx.ConsolidateDeleteNFSLocks = “TRUE”

prefvmx.png
(Click on Image to Enlarge)
(For ESXi hosts, please read my blog on enabling service console access)

7.) Save the changes to the file.

8.) Reboot the ESX host.

9.) VMotion VMs back to patched/NFS lock enabled/prefvmx enabled host.

10.) Repeat steps 2 – 9 on each host in the cluster.

I’ve tested this and it does indeed return the .LCK files to my VMs and it also makes my snapshot removals very snappy.

For those of you that are interested in knowing what prefvmx.ConsolidateDeleteNFSLocks=TRUE actually does, here is a direct response from VMware;

“This changes the behavior of how locks are used while doing snapshot consolidations. [It] greatly improves performance of IO suspension- while maintaining file/data integrity of locking mechanism.”

So, good luck out there people. I know there are a lot of you that now have corrupt VMs because of the NFS.LockDisable issue. We just need to make sure people know of the above fix, and also refer to the latest Best Practice guide that NetApp provides, which is 4.2 (as of 10/13/2008) and can always be found here http://media.netapp.com/documents/tr-3428.pdf


Created on October 13, 2008 by Rick Scherer

Posted under ESX 3.5 Tips, ESXi 3.5 Tips, NetApp, Storage, VMware, VMware HA.

This blog has 35,946 views.

Tags: , , , , , , , , ,

1 Star2 Stars3 Stars4 Stars5 Stars (4 votes, average: 4.50 out of 5)
Loading...

15 Comments so far

  1. pancamo
    5:23 pm on October 14th, 2008

    I worked with netapp earier this year on the NFS locking issue and we were told that there was the issue with HA and locking. Since we don’t have HA enabled, turning off locking solved out vmotion slowness…

    I’m not sure why it took them so long to update the BP doc.

    dan (1000 VMs over NFS)
    http://viroptics.pancamo.com

  2. Rick Scherer
    4:09 pm on October 15th, 2008

    1000 VMs, whew… I’m not there yet, I have around 300 as of today. Congrats on the 2007 NetApp Innovation Award, I was nominated for this years Enterprise award so hopefully I’ll get it!

  3. Richpo
    9:28 pm on October 18th, 2008

    There’s a setting in VC 2.5 that allows you to change the default behavior of network isolated ESX server. I’ve had all my VMs power off on one cluster because the network had a hiccup. Long enough to power them off but not long enough to power the VMs backup. There’s also some time out issues that you can adjust so the ESX server doesn’t freak out and kill everything in less than 15 seconds. I never understood why you would want to kill the VM if you are using two or more network switches. If I had an outage on both switches then I’m hosed for all my ESX servers. Nice to know I haven’t been the only one Burned by this behavior. But in this case you got burned twice. But I have to deal with Hitachi storage which is almost as bad as haven’t no locks

  4. David Faul
    8:24 am on October 19th, 2008

    Did VMware say you need to edit the /etc/vmware/config file on ESXi hosts as well? Since it is unsupported to login to the console (although /etc/vmware/config exists), what do users do with these hosts?

  5. Rick Scherer
    3:52 pm on October 20th, 2008

    The following should be done under the request and supervision of a VMware Support Engineer.

    To access the console from ESXi;

    1.) Hit ALT-F1 from the Orange/Black summary screen.

    2.) Type the word “unsupported” (without the quotes).

    3.) Edit the /etc/vmware/config file following the instructions above.

  6. Rick Scherer
    3:56 pm on October 20th, 2008

    Richpo; There are ways to increase the threshold for VMware HA, by default its actually 14 seconds before isolation response kicks in. It is really all about preference, if you have a full network outage then obviously you probably do not want your VMs shut down, but what if just one host is isolated? There are pros and cons to each selection.

    But, like I said there are ways to increase the threshold, and also a way to include additional IPs to be checked (in addition to the HA member hosts and your default gateway). I’ll write up a blog shortly on how to do these.

  7. Nikolai
    9:30 am on October 22nd, 2008

    Minor correction: NFS.Lock.Disable should really be NFS.LockDisable.

    I can’t provide much more on the issue, aside from http://kb.vmware.com/kb/1006129 and the suggestion that anyone who needs more information regarding the VMware released patch, or issues with snapshots or data corruption in relation to NFS storage and the NFS.LockDisable setting needs to contact NetApp.

  8. Rick Scherer
    10:23 am on October 22nd, 2008

    Thanks Nikolai, once you start writing its easy to overlook simple things like that.

  9. gcleric
    5:46 am on November 20th, 2008

    As a Netapp customer I surely would like to see them reduce the price for NFS. I mean I can buy two Cisco 9124 FC switches, SFX modules, FC cables, and Netapp license for Fiber Channel for cheaper than Netapp’s NFS licenses.

    Netapp Lic for FC = $2500 per filer
    Netapp Lic for NFS = $13000 per filer

    Pitty….

  10. Rick Scherer
    7:49 pm on November 20th, 2008

    I agree, the NFS license is pretty hefty. But, depending on the computing environments you’ll be supporting, it may be beneficial to have it. For example, we are running a little over 100 Sun servers, 16 ESX servers and a slew of other misc. systems. The simplicity and other management benefits made it worth it for us to license NFS over 10 years ago.

  11. shane_allen
    1:59 pm on December 9th, 2008

    Rick,

    BTW, bravo for your Virtulization Forum 2008 Presentation. It was fun listening to the “doubters” about NetApp’s NFS performance.

    I just want to say thank you for the above posts. Within a couple of weeks of running our Exchange 2003 server and our ERP application server virtualized I was hit by NFS.LockDisable. I too had configured the ESX servers with NetApps TR3428 and one morning while Vmotion’ing our Exchange server and our ERP app server they both blew up. A panicked call to VMware and we found that the NFS.LockDisable was to to blame. Fortunately the NetApp snapshots saved me and after a simple cut and paste from the snapshot folder we were up and running inside of a 30 minutes. I then updated the NetApp /VMware engineers I had been working with..

  12. Joel
    7:44 am on January 14th, 2009

    Just got bit by this and still recovering. We run ESXi 3.5 and I’m having a hard time finding the ESXi equivalent of “ESX350-200808401-BG”. Anyone know what it is or if it exists?

  13. Rick Scherer
    10:03 am on January 14th, 2009

    Joel, you can apply that patch to ESX – you’ll then need to access the ESXi Support console to change the /etc/vmware/config file. Instructions to access this console can be found here; http://vmwaretips.com/wp/2008/10/20/access-the-esxi-service-console/

  14. Noe Hoyos
    9:28 am on December 3rd, 2009

    I’d like to undertand how database backups are integrated with VM quiescent. I guess that just quiescent the VM would have an impact on the database unless the DB is quiescent in the process. Is this a this a feature of NetApp for VI?
    I’ve seen your presentations but I am skeptical about the performance of NFS datastores (although I love the idea) for SAP systems. Where may I find more information?
    Thanks,
    Noe

  15. Rick Scherer
    1:24 pm on December 6th, 2009

    Noe,

    The Windows VSS driver will be envoked by any tool that utilizes VMware Snapshots just as long as you have VMware Tools installed on the Windows guest operating system.

    However, this is a best effort way to get the underlying database quiesced, it is not guaranteed that it’ll actually happen. Sometimes I have seen frozen I/O errors kicked back by SQL when doing this.

    As far as NFS performance, I personally have not seen any issues using NFS. The only exception is that NFS will not aggregate over multiple links, so if you have systems that require high throughput (SAP) it is recommended to use 10GbE.

    SAP is a perfect example of an application where the lack of proper planning could burn you. If you’re going to do SAP over NFS, I’d highly recommend 10GbE.

    There are some good websites that get into NFS performance, I’d check VMware VROOM! first.

Trackbacks

Leave a Comment

Name (required)

Email (required)

Website

Comments

More Blog Post

Previous Post: