Analysis

posted on 19 June 2008 07:43

Various authorities and experts say we need a converged data center fabric with all other protocols carried on it. The benefit would be cheaper data center networking through simpler administration, and lower acquisition costs through expanded networking product volumes.

The fabric is needed to interconnect server resources and network ports to the outside world for Internet access, replication, DR links, e-mail etc. The server resource could itself be interconnected by the fabric to provide clustering facilities. The fabric is also needed to link server and storage resources and it could also be used to interconnect storage resources that are clustered together.

In short a converged data center fabric would provide server clustering, storage clustering, server-storage connectivity and server-external networking connectivity. A single fabric would be virtualized so that its bandwidth could be dynamically apportioned to the server, storage and external network links using it.

There seem to be just two candidates: Ethernet and InfiniBand. Ethernet because there is a roadmap from 1Gbit/s Ethernet (1GigE) to 10 GigE and 100GigE; bandwidth to spare, and InfiniBand because its current 20Gbit/s is plenty fast and a 40Gbit/s version is coming.

Each candidate has its problems in winning the data center convergence crown.

InfiniBand
InfiniBand dominates supercomputing and is popular in high-performance computing (HPC). But it is not popular for storage networking. Granted it does that role in supercomputing clusters and in HPC work but outside of that it is rarely seen.

Recently two straws in the wind have come along:-

1. LSI supporting InfiniBand interconnects to a storage array.
2. DataDirect supporting InfiniBand.

Two swallows don't make a summer. If HPC use in general enterprises spreads then that would give InfiniBand a greater presence¹. But for it to have a good shot at being the data center convergence fabric it needs to rapidly expand its use, say, for block-based storage networking, clustered scale-out file storage and general server clustering. Its cost is an inhibitor.

Ethernet
The big pair of problems here are, first, that Ethernet can drop packets and thus data. That's inexcusable in a data center backbone fabric. Secondly, its latency is unpredictable, packets can arrive late. This can cause some applications to temporarily break, such as ones relying on a Fibre Channel link to deliver data in a fixed time slot. Running Fibre Channel over Ethernet (FCOE) would make applications susceptible to this.

Other problems include Ethernet's relative lack of success as a server clustering link and the perception of it as a slow speed server-storage link (iSCSI) compared to Fibre Channel. The use of 10GigE should help counter that, but Fibre Channel could transition to 16Gbit/s in its next iteration which would make the playing field uneven again. The faster 100GigE is some way off.

There is a data center-class Ethernet standards effort. It could deliver product in, say 2011 but this is guesswork.

A big thing in favour of Erhernet is that its extension into data center fabric backbones is merely evolutionary. Everyone already has Ethernet so extending its adoption would be relatively easy.

So ...
Ethernet has its massive volume and installed base and road map in front of it. If data center-class Ethernet products arrive in 2011 and 2012 and if, by then, InfiniBand has not penetrated the worlds of storage networking and clustering and general server clustering then this writer's view is that it will stay in its supercomputing and HPC niches, which will be larger than today, but not as large as the market for a general data center fabric, which will become Ethernet.

^{[Chris Mellor.]}

^{1. NetApp Data ONTAP GX clusters can connect via a gateway to InfiniBand-linked servers.}