Ethernet Storage Guy

FCoE Fibre Channel over Ethernet promises a great new future for block based storage protocols. I have found that in my conversations with many customers/clients that there is a significant amount of confusion around what FCoE actually is.

This post is an attempt to clear some of the basic confusion up. The FCoE technology in itself can spawn many conversations but this one will attempt to stay on track with the basics.

So how did FCoE come to be?

On April 5th, 2007, a group of companies published a press release entitled “Leading IT Vendors Propose Fibre Channel over Ethernet (FCoE) Standard”

The press release announces the concept of creating a new protocol specification to transport Fibre Channel storage traffic over native Ethernet Frames (FCoE) and was presented to ANSI on June 6th, 2007.

So What is FCoE?
If you are really interested take a look at the full specification.

FCoE is a protocol specification that maps Fibre Channel over Ethernet that is independent of the native Ethernet forwarding scheme. Thats alot of words but very important to understand what that means. This means that a FCoE frame traverses the ethernet fabric without utilizing the traditional ethernet process, utilizing broadcasts etc…

This architecture allows FCoE to seamlessly integrate with existing Fibre Channel. The forwarding mechanism is in essence the same as Fibre Channel plus all the zoning rules. This enables you to integrate a FCoE fabric with an existing SAN fabric. The two fabrics are interchangeable and all of the new switches which support FCoE (discussed below) also have card slots for FC connections to enable that convergence point between the two fabrics.

The new ethernet fabric, which was designed for FCoE, is called a couple of different things. The names generally refer to the same principal technology yet the names are usually tied to a manufactures marketing.

For Example:
DCE – Data Center Ethernet (Cisco)
CEE – Converged Enhanced Ethernet (Brocade, IBM)

We will use DCE for the purposes of this post.

DCE includes a few new features that are leveraged by FCoE and can be leveraged by other protocols as they are developed or re-engineered for DCE.

Features:
Priority-Based Flow Control IEEE 802.1Qbb - Provides class of service flow control. Traditional ethernet has flow control but it is one size fits all. Class of Service flow control allows us to give a preference to things being transmitted. FCoE, for example, uses a Class of Service flow control which will guarantee it’s delivery and preference in the event of congestion.

CoS Bandwidth Management IEEE 802.1Qaz - This new specification allows grouping of classes of traffic into lanes of bandwidth, once again ensuring that FCoE or other transmissions are given the preference they need.

Congestion Notification IEEE 802.1Qau - End to End Congestion Management for a Layer 2 Network. The ability to actually have the network notify as points in the network become congested, at the switch port or host.

L2 Multi-Path for Unicast & Multicast - This effectively can allow for the elimination of spanning-tree, in certain environments. Today when we create multiple links between two devices those links are not load-balanced completely or at all. In the case of switch to switch links spanning-tree will block one port and it will not be used for data traffic, unless the primary link fails. In the case of switch to host traffic you effectively need to bond two or more ports together to load-balance via a higher layer protocol. Which brings us back to switch links, you can bond switch links together via etherchannel yet these ports are load-balanced based on source and destination pairs. If you have 4x1Gbps connections bonded together, then one source and destination pair can only consume the bandwidth of one physical link. More on that some other time. L2 Multi-path is what Fibre Channel networks do today. Basically if you establish two connections on a switch to a host, FC will load-balance. L2 Multi-Path is a game changer and not alot of people understand it yet.

Lossless Service - This is key to FCoE being successful. Traditional ethernet discards frames when congestion occurs and the higher layer protocols will take care of ensuring the data gets retransmitted. Said simply, you can’t do that with FC traffic. The fabric (in this case Data Center Ethernet) needed to have an ability to enable a service that could guarantee the transport of data.

What do you need?

The above are the features of DCE now what do we need to run DCE and ultimately FCoE over DCE. Basically, you need new hardware. That hardware is detailed below.

Switch hardware - Cisco has released it’s Nexus product line. The Nexus 5000 is a Data Center Ethernet class switch which can deliver the lossless service and has ASICs on board that can forward frames with latency numbers that are far better than we have seen in any ethernet switch ever released. Brocade and others are planning releases of FCoE capable DCE/CEE switches.

NIC hardware - Making FCoE work requires supporting DCE features, end to end, and this obviously includes the host connection. A new class of NICs has emerged and those have many different marketing names as well. We are going to call them CNA’s (Converged Network Adapters). A CNA is 10Gbps Data Center Ethernet adapter (supporting features mentioned above) and designed specifically for transport of FCoE frames. Now lets crack open a CNA card and have a look under the covers, in a simplistic manner.

Here is basically what it is.
1.) 10Gbps DCE Ethernet NIC ASIC
2.) Legacy Ethernet NIC ASIC
3.) Legacy FC HBA ASIC

Basically, you pop this NIC in your server and install the drivers. One set of drivers will install a Ethernet NIC another will install a HBA. Traffic that comes from the “virtual HBA in the server will then be classified and transmitted properly out the 10Gbps connection adhering to the forwarding rules defined in the fabric. World Wide Names (WWNs) all that stuff remains the same from the FC world. IP will run on the legacy ethernet NIC and it will be transmitted out the 10Gbps DCE Ethernet NIC with the proper class of service for IP.

Which brings us to the thing that I find so many people confused about. That last paragraph was the first time I mentioned IP. IP is not apart of the FCoE stack. No IP addresses exist for FCoE components. The only time you need IP is if you want to actually run IP traffic on the CNA adapter. The CNA adapter will however ensure that IP traffic does not get in the way of FCoE. This brings me to a final point that people are often confused about. I have not once mentioned routing. FCoE is completely layer 2, meaning switched traffic, no routing is involved. Remember the FCoE specification allows for an independent means to forward traffic. Our forwarding mechanism for FCoE are the zones we define in the DCE/FC fabric, the same way we use to with FC. IP transmissions can routed on the same wire but that traffic is classfully separated from our FCoE traffic.

Cisco FCoE Video
IBM Redpaper – Introduction to FCoE
Convergence with FCoE and Lossless Ethernet
NetApp First to Announce Support for Native FCoE Storage