Virtualizing your DMZ

Today I got into a heated discussion with a “Virtualization Expert” at Gartner today about the risks associated with virtualizing your DMZ, primarily into the same environment as your non-DMZ servers.

As you can see in my diagram below, this is my plan;  Create a completely isolated vSwitch with dedicated NICs for my DMZ portgroup, which is separated from the vSwitch that contains my Service Console, VMKernel and other Virtual Machine portgroup.

The discussion I got into this afternoon was that this was still a security risk, and presented a vulnerability to potentially all my other Virtual Machines on the host.   Frankly, I just don’t see it.  The only risk I could potentially ever see is if one of the guests in the DMZ portgroup was compromised, it could potentially affect other systems within that vSwitch — there should be no way it could cross talk to the other portgroups on other vSwitches in this system.  Right?  The Gartner engineer kept bringing up how someone could attack that vSwitch and gain access to the system to compromise it…then he kept giving examples about how this happened in Hyper-V (which has nothing to do with ESX).

So, I’m leaving this open…what is your opinion?  Regardless of Intrusion Protection Systems and Intrusion Detection Systems and Firewalls, etc. —  How strong is a vSwitch?  Can someone that hacks a guest within an isolated vSwitch (one with no Service Console on it) gain access to your host?

Are these so-called Experts at firms like Gartner and Forrester really Experts?

You tell me….


Created on December 5, 2008 by Rick Scherer

Posted under Networking, Security, VMware.

This blog has 26,218 views.

Tags: , , , , , ,

1 Star2 Stars3 Stars4 Stars5 Stars (3 votes, average: 4.33 out of 5)
Loading...

34 Comments so far

  1. John Troyer
    7:30 pm on December 5th, 2008

    One of the risks is simply human error if somebody connects the wrong VM to the wrong switch down the road.

    See this white paper for more detail and decision criteria for different DMZ topologies for virtualization
    http://www.vmware.com/resources/techresources/1052

  2. rodos
    9:08 pm on December 5th, 2008

    Are you able to ignore them? Have you checked out http://www.vmware.com/files/pdf/dmz_virtualization_vmware_infra_wp.pdf ?

    There is no risk of traffic between the vSwitches in your model. The risk is from human error and misconfiguration (which is really the same in the physical work IMHO).

    Rodos

  3. Louw
    11:42 pm on December 5th, 2008

    This “expert” obbiously don’t have a clue.

    Just saying “it could” doesn’t prove the reality and if a competitor has the problem also proves nothing about VMware infrastructure.

    This is an ongoing FUD attack without ANY basis in reality but unfortunately IT Managers don’t have the knowledge to debunk these type of “experts” from Gartner and will believe the lie and even promulgate it.

    Gartner should check they’re “experts” before they loose all validity from the technical community.

    PS. IDC has already gone that route.

    Louw

  4. Rick Scherer
    8:32 am on December 6th, 2008

    I really appreciate all the input on this subject so far. For those of you that haven’t read my bio, I’ve been considered an “expert” in virtualization for over 5 years now, and the reason I designed my proposal the way I did was because I knew it would provide the greatest protection because of its isolation. Like I said in the post, this “expert” really got under my skin….but because he was from Gartner my upper management will most likely follow his suggestion of completely separating it on a hardware level as well.

    Keep the comments coming, I will be sure to show all of this to my management for review (And proof that I am not alone in my beliefs!)

  5. Wayne Ingram
    9:33 pm on December 6th, 2008

    It always amuses we when people say that human error can cause an issue with the “collapsed DMZ”. How any people do you know that run the same IP subnet in their DMZ as in the normal network? I like most people run say 10.x.x.x internally but run 192.x.x.x in our dmz. If an admin accidentally selects the wrong vlan, the error will be picked up pretty quick when the guest vm fails to talk on the network due the wrong vlan been selected.

  6. Grant Sabesky
    12:47 pm on December 8th, 2008

    Following the Vmware “Best Practices” document is a good start and depending on your level of paranoia will dictate your infrastructure design. My general rule is use best practices when possible dictated by the environment, be conservative, somewhat paranoid and if you are concerned about anything being hacked, virtual or physical – disconnect from the network!!! Not an option today…but remember layer2 is not exempt from attacks…My 20 years of experience says go forward with a vSwitch(s)for a DMZ.
    Questions: Is your vSwtich0 connected to your internal network? What ports and protocols are being used to communicate with your DMZ hosts? How is VC communicating with your DMZ?

  7. Rick Scherer
    1:03 pm on December 8th, 2008

    vSwitch0 would still be within our private network, for internal Virtual Machines and service console access. vSwitch1 would be dedicated for DMZ based virtual machines.

  8. JLobster
    2:11 pm on December 8th, 2008

    Was this Gartner “expert’s” name Matt Cain? I have had issues with him spouting nothing but year-old MS FUD with one of my accounts, too. Very, very poorly informed.

  9. Rick Scherer
    2:55 pm on December 8th, 2008

    Wasn’t Matt, but I think this is a common practice among all uninformed “virtualization experts” — just spout out FUD because you don’t know anything about what your talking about.

    FUD + Unknowing Management = Pure Profit

  10. Grant Sabesky
    3:43 pm on December 8th, 2008

    Just for added security I would place a FW appliance between your DMZ and your internal network…just a thought

  11. Rick Scherer
    4:14 pm on December 8th, 2008

    Well since there is no link between the DMZ and our Internal network there is no way to put a virtual FW between them. There is already a physical firewall and IPS between our DMZ and Internal Network (two actually).

  12. Grant Sabesky
    4:17 pm on December 8th, 2008

    Does your internal VC talk to the DMZ VM’s? How do you communicate with the DMZ VM’s?

  13. Rick Scherer
    4:43 pm on December 8th, 2008

    Look at the diagram above, the Service Console/VKernel and internal network are all part of vSwitch0 — vSwitch1 is dedicated for DMZ guests.

    This ensures that your ESX host is still managed by an internal VC server, but you still can have DMZ guests on the same box for better utilization of resources.

  14. shane_allen
    1:42 pm on December 9th, 2008

    Rick,

    Before I fully collasped our DMZ and hosted it as you have outlined above within our VMware enviroment I spoke with the two authors of the VMware “DMZ Virtualization with VMware Infrastructure:” paper and they confirmed (and many of the posts already have) that the only risk to the VMware environment is the same risk with a physical DMZ. Admin’s not knowing what they are doing. If you can mitigate that it’s a great cost and management savings. I doubt the Gartner “expert” really knows what he is talking about.

  15. Cameron Moore
    1:19 pm on December 20th, 2008

    I’m starting to plan out an ESX rollout for this summer, so this is a great discussion.

    Generally speaking, would it be even more secure to isolate your VMKernel and Serv Cons onto a separate physical NIC and vSwitch?

  16. Rick Scherer
    2:16 pm on December 20th, 2008

    Hey Cameron, as a best practice you would want to separate the physical NICs of your Service Console and VMKernel, but I really do not see a security risk running them on the same vSwitch. The only risk is if you would run a DMZ network on the same vSwitch as your internal network – which is the same risk as doing this in a physical world.

    If your limited on NICs you can always use two NICs for your Serv Con. and your VMKernel, just set your active and standby settings in your failover order for each portgroup. (ie: Serv Con would be active on NIC 1 and standby on NIC 2, VMKernel would be active on NIC 2 and standby on NIC 1).

    Good luck with your deployment and feel free to submit any questions you may have to me.

  17. Cameron Moore
    2:38 pm on December 20th, 2008

    I think I was unclear in my original question…

    Is it more secure to have your internal VMs (Network0 in your diagram) on their own vSwitch? What I’m getting at is whether it would be a good idea to be just as paranoid about your internal VMs as you are with your DMZ VMs. If an internal VM gets compromised in your diagram, wouldn’t your ESX host be vulnerable?

    Is their some inherent vulnerability in the vSwitch technology or this simply a basic networking problem were there’s no firewall between the hosts on a vSwitch? The nature of the vulnerability is unclear to me at this point?

  18. Rick Scherer
    2:55 pm on December 20th, 2008

    It is not the vSwitch technology that is vulnerable, it is the fact of having different systems on the same lan segment (think of this the same as having multiple systems on the same physical switch with no firewall/ips type of protection).

    If you are worried about the possibility of having your internal network comprimised then I would see no problem separating the Serv Con and VMKernel into their own vSwitch. You can have a maximum of 248 vSwitches so don’t worry about having one extra :) Remember though, even though your separating your internal network from the Serv Con. and VMKernel, if they are on the same subnet it still wouldn’t matter because an attack can still be attempted.

    So lets sum up. The vulnerability isn’t in the vSwitch, its in putting different subnets on the same vSwitch (just like you wouldn’t do this on a physical switch). If you separate Serv Con./VMKernel and your VM network and they are on the same subnet, you are not achieving any stronger security goals.

  19. Cameron Moore
    8:55 pm on December 20th, 2008

    Rick,
    Thanks for the excellent clarification and the FUD debunking. I think what had me confused was when you said “the Gartner engineer kept bringing up how someone could attack that vSwitch and gain access to the system to compromise it.” Attacking the vSwitch sounded pretty ominous, but now I see what you are talking about. Thanks for your time!

  20. Tim Pierson
    6:55 am on April 7th, 2009

    I wrote a security class for VMWare and discussed this topic greatly in the course. The thing that no one has mentioned is that if the attacker has access to the vSwitch code then he also has access to the kernel. Why would he waste his time hacking into a vSwitch when he already owns the box. Keep in mind that vSwitches only process the frame data, nothing more. So the code they interpret would be very difficult to circumvent. This was purpously done by VMWare. Less complexity = more security.

    This could very well change with the addition of the vSafe API to debut this year. While its intent is to strengthen security, but it will also add complexity. Refer to statement above..

    I think this fear is there simply because of the unknown. Virtualization is a newer technology. Those technologies always get the blame for things that may or could go wrong. Remember when you implemented AD? Did it not also get the blame for a lot of things that went wrong during its implementation. I think it will just take time for the products to mature and develop a comfort level with the sys-admins.

  21. Rick Scherer
    7:32 am on April 7th, 2009

    I completely agree with you Tim, if the attacker is in the vSwitch that means they’ve already owned your box.

    Typical FUD that we will deal with most likely forever. The product is mature (5+ years for anything IT is mature), the problem is “most” firms like the ones mentioned above have security experts discussing risk on a product they really know nothing about.

  22. Jerry Cox
    4:48 pm on April 7th, 2009

    The real issue here isn’t the security of the vSwitch or whether we trust the administrator. As more and more applications are deployed in the virtual environment, it becomes necessary to segregate the virtual network, and to the extent possible, do this without messing up all those great benefits we get from virtualization in the first place – resource pooling, DRS, vMotion, etc. Moreover, most compliance related requirements (ex. PCI) require that the person separating the network be a different person than the system admin – in this case the VMWare admin, exactly to the points above.. What’s needed is a virtual firewall that sits in between virtual switches and controls traffic according to a corporate policy. If it has the ability to detect and/or prevent attacks that originate from VM’s within a virtual subnet (that do not even make it to the IDS/IPS gizmo sitting on the perimeter) so much the better.

    Provably resolving this issue will help us all gain the trust of application owners, auditors and analysts, allowing us to grow and expand our vEmpires.

    Check out the product that won best of show for security at VMWorld last year. VMSafe coming makes it even better.

  23. Rick Scherer
    1:41 pm on April 8th, 2009

    Hey Jerry, I think 3rd party vNetwork switches will resolve the PCI compliance issue. With vSphere you can now offload the network control back to the networking group.

    http://www.vmware.com/technology/virtual-datacenter-os/infrastructure.html

    The discussion on this post really discusses the FUD around placing DMZ and Non-DMZ virtual machines on the same host — technically there is no security risk just as long as the networks are on separate vSwitch.

  24. Pete Sneathen
    7:04 am on March 13th, 2012

    Rick,

    Awesome, this is just what I have been looking for.

    I noticed that you have 10 NICS in your host, and 4 are appear to be dedicated to DMZ VMs. Is this because you want 2 teamed for network traffic and 2 teamed for storage? How wuold shared storage be utilized between trusted/internal VMs on switch 0 and DMZ VMs on switch 1? Sorry, still fairly new to the storage game, and just shy of 1 year under my belt with virtaulization. Your feedback (and others) is much appreciated. Thanks.

  25. Rick Scherer
    8:58 am on March 13th, 2012

    The expectation of this design is that those (4) vNICS tied to vSwitch1 would strictly be for the traffic of those Virtual Machines within the DMZ. Back-end storage (VMFS datastores) would be mounted via either Fibre Channel or they can be routed through the VMkernel port (for iSCSI or NFS).

    Isolation could be taken to a whole new level by having a separate VMkernel (or separate fibre runs) strictly for storage of those DMZ virtual machines if your security policies specify that they must be separate.

  26. Adam
    6:28 pm on August 9th, 2012

    Why separate vSwitches? Wouldn’t separate port groups on different vlans on the same vSwitch that go through firewall/ips/ids to control/protect inbound/outboud traffic be sufficient?

    What’s the reason behind multiple vSwitches, from what I understand traffic across multiple port groups on the same vSwitch have to go out the pnic and potentially back in the same pnic is the destination is on the same host.

    I agree with statements above that if someone finds a way to compromise the underlying hypervisor they own the system and which vSwitch the VM is on is trivial.

    So, if we are protecting our DMZ assets with layer-4&7 firewalls/ips/ids… is that not enough in a virtualized environment?

    Does any of the above change with the introduction of NVP/NVGRE/VXLAN?

  27. Rick Scherer
    6:43 pm on August 9th, 2012

    @Adam – A lot of this changes with the introduction of different networking technologies. NVP and VXLAN introduce some very exciting possibilities. I’m fully welcoming the beginning of SDN.

    This was written back in 2008 before we even heard of a Distributed Virtual Switch (dvSwitch), so this design wouldn’t even apply in that model.

    The vSwitch isolation was to give my (previous) employers Chief Security Officer a higher level of comfort. It would’ve been impossible for me to explain how I can leverage a single vSwitch and dedicate pnics to the DMZ port group.

  28. Nicky
    12:04 pm on November 19th, 2014

    Rick,

    just come across this post, and two years on, we’re in much the same position. So called security experts who insist hosting a DMZ host and internal servers on the same ESXi host or farm, is insecure. We’re looking at much the same design as you have implemented. How did you get on with your design, and other than the two documents referenced here, do you have any other VMware best practice docs that I could use to beat the doubters around the head with?

  29. Rick Scherer
    12:12 pm on November 24th, 2014

    @Nicky,

    In that situation, keeping in mind that it was back in 2008, we ended up having to deploy a separate physical infrastructure for our DMZ environment.

    I’m seeing more and more now that virtualized options are become widely more accepted, this is especially prevalent when leveraging virtual network and security tools (vCNS, NSX, etc).

    You can find more information in the VMware vCNS DMZ Design and Deployment guide; http://www.vmware.com/files/pdf/techpaper/vCloud-Networking-Security-DMZ-Design-Deployment-Guide.pdf

    Thanks
    Rick

  30. Kewal Kishore
    10:40 am on November 25th, 2014

    I want to install Exchange 2010 EDGE server on esxi server. I want put this server in DMZ. I am new vmware esx and need some help to configure DMZ. Is there any article which I can refer to create DMZ. I have only single NIC on whitebox.

  31. Rick Scherer
    11:00 am on January 12th, 2015

    A single NIC will not work for this type of configuration. You’d want at least two NIC ports, one port to an internal private network for management and another port connected to your networks DMZ.

  32. G_Singh
    9:37 am on January 29th, 2015

    Hi Rick,

    I am looking to deploy a Web Server on a DMZ that will then talk to a SQL Server. I have in mind to put the Web Server on its only separate VLAN. I have 4 NIC’s available on the server. Is there anything to look out for or it it just a case of one NIC for internal, one NIC for the DMZ and then firewall rules to forward on the ports needed?

  33. Adam Gibson
    10:45 am on February 4th, 2015

    The vswitch is only part of the risk. It is things like client tools/drivers being exploited to compromise the host VM server that is the bigger reason for me to keep from deploying mixed DMZ/non-DMZ vm guests on the same ESX.

    http://www.blackhat.com/presentations/bh-usa-09/KORTCHINSKY/BHUSA09-Kortchinsky-Cloudburst-PAPER.pdf

    That is one example for vmware desktop. To think there is zero chance of some other exploit that can compromise the host from the vm on ESX would be a bet I wouldn’t want to make.

    We have separate DMZ and non-DMZ ESX throughout our infrastructure. It is much more expensive but worth the $$$ to me to know that there is physical separation when it comes to sensitive data to protect.

  34. Al
    7:38 am on May 15th, 2015

    I was nodding my head through most of this.. then I read: http://bradhedlund.com/2010/02/10/vswitch-illusion-dmz-virtualization/

    and I was nodding my head again… Damn this is confusing !

Leave a Comment

Name (required)

Email (required)

Website

Comments

More Blog Post