Cisco UCS Design Flaw? No Northboard FCoE Connectivity

Today Scott Lowe wrote a post on his blog explaining how Cisco UCS lacks Northbound FCoE connectivity, explained here;

I’m about halfway through the first day of Unified Computing System (UCS) training in San Jose, CA, and I’ve learned of what I think is a fairly significant limitation. The issue centers around what Cisco refers to as “northbound” traffic and how Fibre Channel over Ethernet (FCoE) is handled with northbound traffic.

Recall that a central part of UCS is the UCS 6100 series fabric interconnect. The 6100 series fabric interconnect has connectivity in two directions:

  • Southbound connectivity is connectivity aimed back at the fabric extenders in the blade chassis themselves.
  • Northbound connectivity is connectivity headed outside the UCS to other systems and networks.

All southbound traffic is 10Gbps Ethernet with FCoE. Northbound traffic can be 10Gbps Ethernet or Fibre Channel, but not FCoE. Based on the information I’ve been given (and if I’m incorrect please let me know in the comments), you cannot directly connect an FCoE-enabled storage array to a UCS. Even if your storage array has native FCoE interfaces, you can’t plug them into the UCS 6100 series fabric interconnects because that’s considered northbound traffic and you can’t use FCoE with northbound traffic.

I have a feeling customers who have purchased storage arrays with FCoE interfaces with the intention of hooking the arrays up directly to a UCS are going to be a bit upset when this information becomes more widely known.

If I’m working from incorrect or incomplete information, please feel free to speak up in the comments.

At first I was extremely shocked to hear this, this is pretty big news and I would be upset as a customer if I wasn’t able to directly attach my FCoE-enabled storage array directly to the UCS Fabric Interconnect.

After doing some research of my own I found the following;

Expansion Module Options for Cisco UCS 6100 Series Fabric Interconnects

The Cisco UCS 6100 Series is equipped to support a number of expansion module options (Figure 4).

• Ethernet module that provides 6 ports of 10 Gigabit Ethernet using the SFP+ interface

• Fibre Channel plus Ethernet module that provides 4 ports of 10 Gigabit Ethernet using the SFP+ interface; and 4 ports of 1/2/4-Gbps native Fibre Channel connectivity using the SFP interface

• Fibre Channel module that provides 8 ports of 1/2/4-Gbps native Fibre Channel using the SFP interface for transparent connectivity with existing Fibre Channel networks

So, no FCoE support on those expansion modules which is surprising.. however, it could be by design. I wouldn’t necessary want to attach my SAN directly to the fabric interconnect, but with the addition of a Nexus 5000 I could push FCoE direct to the Fabric Interconnect, thus avoiding unnecessary fibre connections.   I do find it funny though that the Nexus 5000 has full FCoE support (Northbound and Southbound), look at their expansion module offerings;

Expansion Module Options for the Cisco Nexus 5000 Series

The Cisco Nexus 5000 Series is equipped to support expansion modules that can be used to increase the number of 10 Gigabit Ethernet/FCoE ports or connect to Fibre Channel SANs with 4/2/1-Gbps Fibre Channel switch ports, or both. The Cisco Nexus 5010 supports one expansion module, and the Cisco Nexus 5020 supports any combination of two modules from the following offerings (Figure 4):

• Ethernet module that provides 6 10 Gigabit Ethernet/FCoE ports using the SFP+ interface

• Fibre Channel plus Ethernet module that provides 4 10 Gigabit Ethernet/FCoE ports using the SFP+ interface, and 4 ports of 4/2/1-Gbps native Fibre Channel connectivity using the SFP interface

• ·Fibre Channel module that provides 8 ports of 4/2/1-Gbps native Fibre Channel using the SFP interface for transparent connectivity with existing Fibre Channel networks

Hopefully someone from Cisco can explain why UCS was designed this way, feel free to comment on my blog or on Scott Lowe‘s.

UPDATE: A lot of chatter has been going on in Twitter and on Scott’s post. It sounds like FCoE will be supported in UCS with a future software upgrade.  Also interestingly enough, you cannot directly attach a FC device into the Fabric Interconnect. UCS operates all FC functions in NPIV mode, not FC switch mode. This means you need to uplink to a fibre channel switch, like an MDS (or N5K).  Check out Scott’s post for all the comments.

UPDATE 01/10/2011: With UCS Manager v1.4 now released supports direct attached storage, including direct FC connected storage, direct Ethernet connected storage (iSCSI/NFS) as well as FCoE connected storage. Dave Alexander has a great write-up on his blog. Steve Chambers also has a great write-up on his blog reviewing all of the great new updates in UCSM 1.4. One thing that I’m disappointed to not see yet is northbound FCoE support, don’t get me wrong having direct connect storage is amazing but say you have multiple UCS domains within a single datacenter, you’ll still need to connect dedicated IP and SAN fabrics to each Fabric Interconnect.


Created on July 27, 2009 by Rick Scherer

Posted under Good Reading, Storage.

This blog has 5,452 views and 4 responses.

Tags: , , ,

1 Star2 Stars3 Stars4 Stars5 Stars (No Ratings Yet)
Loading ... Loading ...

4 Comments so far

  1. Craig
    9:17 pm on September 21st, 2009

    I run in to the same disappointment as you did. To have Nexus 5000 be part of the UCS solution, it will end up become a show stopper to many potential customer who may not be big enough to require Nexus 5000 in the environment which they expected UCS 6100 will playing the same role. Cisco Should seriously look in to this.

  2. HP SFP
    12:54 am on February 26th, 2010

    Why We cannot directly connect an FCoE-enabled storage array to a UCS, what would be the problem if at all we connect them can you please explain.

  3. Mark
    11:12 am on March 8th, 2010

    I know this is an old post, but there is NO REQUIREMENT for an MDS switch to be in the mix with a UCS. The UCS can connect to ANY modern FC switch which supports NPIV devices attached.

    This is NO DIFFERENT from HP Virtual Connect FC, a Brocade FC blade server switch running in Access Gateway mode, or a Cisco MDS FC blade server switch running in NPV mode.

    Ethernet operates in a similar manner on UCS. Again, this is NO DIFFERENT form HP Virtual Connect Ethernet.

  4. pattho
    6:36 pm on January 5th, 2011

    Finally it is now supported as of 12/10
    FCoE direct into UCS much better ROI than better

Leave a Comment

Name (required)

Email (required)

Website

Comments

More Blog Post