Beyond SAN and FCoE!

By Jerome Joseph
Next Generation Serviceability
San and FCOE 1 San and FCOE 2

As the need for shared storage became acute in the early 1990s, fundamental limitations in available technologies hampered efforts to meet this need. Network traffic with Ethernet was typically only 10 Mbps (or 100 if you were lucky), and direct-attached storage impeded the growth and clustering of servers. To overcome these limitations, new technology such as fibre channel was developed to network storage arrays with a reliable transport layer. Although the technology was made to work, it came at the cost of a proliferation of standards, semi-standards, and vendor-specific solutions which all conflicted with each other. Fibre channel protocol (FCP) provided elasticity with 10/8-bit encoding and additional frames for the IDLE state using K28.5 frames. SCSI command descriptor blocks (CDBs) were encoded in fibre channel exchange (FCE). Access control was provided with LOGIN mechanisms (like fabric login or FLOGI), FC zoning, and port security. Build fabric (BF), fabric shortest-path first (FSPF), and principal switch election were used to manage multiple SAN/FC switches.

Even after years of SAN administration, only a select group of engineers understand this complex set of technologies and their interactions and conflicts.

The Origin of Fibre Channel over Ethernet (FCoE )

San and FCOE 3

At the same time as SAN technology was advancing, Ethernet went through its own metamorphosis into 1Gbps, 10Gbps, and beyond. The availability of reliable GigE—with lossless switches providing low latency and better congestion avoidance—brought the relevance of SAN technology into question. Multiple vendors developed their own methods of encapsulating FC frames, notably converged enhanced ethernet (CEE) and datacenter bridging (DCB). The only adapter that has the complete set of single-root I/O virtualization (SR_IOV) and virtual network tagging (VN-Tag) functionality is the Cisco Palo adapter (M81KR), which works only with Cisco UCS.

In my view, FCoE is force fitting FC into GigE to extend the life of FC, merely so that vendors can sell expensive storage arrays and expensive switches that support FCoE, lossless ethernet, and priority flow control (PFC). But with advent of powerful CPUs, larger hard drives, faster solid-state devices, and reliable PCI storage devices, we don’t need storage arrays or SAN switches. Instead, we can bring storage back to the server.

Overcoming SAN and FCoE with Nutanix Complete Cluster

Modern storage arrays contain multiple disks formatted in vendor-specific RAID with a front-end cache and an NFS/FC/CNA adapter connected to an expensive SAN switch with FC cables. Hosts also require expensive adapters to interface with the storage array adapter.

Most array vendors take six months to a year to certify the latest HDDs, SSDs, adapters, and memory. Once the certification is complete, the process of upgrading the storage array is tedious.

San and FCOE 4

Nutanix, by contrast, makes it easier to adapt new technologies in this area.

Nutanix depends on standard GigE infrastructure, removing the need to buy separate storage processors, storage arrays, FC/FCoE infrastructure, and GigE infrastructure. Instead of the hodge-podge approach of migrating SAN from FC to FCoE with multiple (non)standards, Nutanix defines a future-safe approach. Whenever a new CPU, HDD, or network adapter comes into market, it can be incorporated into your Nutanix Complete Cluster and immediately improve performance.

Large storage vendors do not want disruption—rather they want slow a process of changing one component at a time, so that they have time to reinvent and have their customers buy overprovisioned switches, servers, storage arrays, and vendor-specific HDDs that will be end-of-life in a few years. Customers will be stuck re-provisioning, incrementally upgrading, migrating to new storage arrays and switches, learning new technologies, and spending training dollars, instead of being productive with a few basic and standard technologies.

Some aspects of FCoE—VN-Tag to create VIFs (SR_IOV) and lossless (or, wryly, lessloss) Ethernet—need to be incorporated into the next generation of Ethernet switches. Others—virtual fibre channel (VFC) interface and FCoE initialization protocol (FIP)—should be left behind.

It is time to move beyond the complexity of the past and build a flexible, agile, and standards-based datacenter with Nutanix Complete Cluster.