Why doesn't networking obey Moore's Law? - Eye on Blades Blog: Trends in Infrastructure -
Why doesn't networking obey Moore's Law?

Every 18 months, customers around the world expect to see server performance double for the same or less cost. As you know, it's been this way for some time; a consequence of a little idea called Moore's Law.  The good and bad of Moore's Law - great for customers, challenging for server businesses.  Our team would tell you that it just takes years of practice to keep up.

My question today is "why doesn't networking obey Moore's Law?"  The jump from 1GB to 10GB came years apart and with a huge jump in price. The most over-hyped next step is Fibre Channel over Ethernet (FCoE), well behind 8Gb Fibre Channel but more expensive. But why such a limited view on network consolidation and why is it still so far way?   Shouldn't the industry be focused on consolidating all Ethernet ports and cables to cut costs NOW, with 10GB performance delivered today?  We think so too. It just takes the will to rethink the sacred cows and challenge the conventional wisdom.  More on that in a minute.

I'm sure someone will throw out Metcalfe's Law or McGuire's Law of Mobility to argue my point here, but these benefits are in the consumer realm, not the data center.  I wish I could get the same switch upgrade bargins that I do on my cell phones every 2 years.

Continuing and extending the Moore's Law argument, I could ask the same question about price/performance for storage and facilities too.

Yesterday, I received an ad for a 1TB drives for $89!  We estimated that the same amount of capacity in a standard Fibre Channel SAN for the data center is around $20,000 for that first TB of storage.  How many drives are in the average infrastructure utilizing only 10% of their total capacity?  I suspect there are a lot of cheap terabytes to be had out there.

Facilities is a tougher issue because there are so many external forces driving the costs - energy and real estate are the obvious ones.  Still, the model for today's data centers are based on blueprints drawn up to support mainframe architectures in the 1960's.  Fresher ideas are being implemented today, but it's time to accelerate that innovation here as well.

That all brings us to, "how do we align network, storage and facilities more inline with Moore's Law?"

We think convergence is the key.  We talked last month about how that's a much more interesting idea than simply 'consolidation'.  Convergence is not only about less, it's about transformation to create a whole new world of possibilities.  That means combining things to create something new and usually makes the combined things obsolete.

Server and network convergence such as the combination of Virtual Connect and Flex-10 networking embedded in the BL495c virtualization blade is one example.  Moving storage closer to the compute, squeezing out the network is another compelling idea.  What about converging infrastructure systems and facilities management, power distribuion and cooling. 

What do you think it will take to align networks, storage and facilities closer to the improvements delivered to compute over the last couple of decades? 


Posted 03-12-2009 8:57 PM by newtonja

Comments

Omar Sultan wrote re: Why doesn't networking obey Moore's Law?
on 03-13-2009 6:39 AM

Jason:

You ask an interesting question about Moore's Law and networking.  Networking does in fact progress, but it changes by orders of magnitude 1Mb/s --> 10Mb/s (1985) --> 100Mb/s (1995) --> 1Gb/s (1998) --> 10Gb/s(2006).  Now, the rate of change in the network is driven by the demand.  The same way we would not see a need for quad-core processors if we still used DOS and WordPerfect, the rate of network upgrades is driven by the ability of the connected computers to fill the pipe.  Even today, the vast majority of computers can happily sit on on a GbE connection because it is more than adequate foe their needs.  The drive to 10GbE on the server side is driven by things like increased VM density, increased usage of video and the ability of more powerful CPUs to support more complex workloads with higher I/O requirements.

As for pricing, the "huge jump" in price is simply not there.  2008 ASP for a GbE port on a fixed managed switch is $111--for a 10GbE fixed managed switch, it is $876 (source: Dell'Oro), so for, for an equivalent amount of bandwidth, 10GbE is about 20% less expensive--not including the additional cabling, interface and switch costs of the the GbE approach.

Finally, all this makes the case for FCoE, which, by the way has shipping product today and should see a finalized standard by this summer. First of all, 10GbE offers more raw bandwidth than 8GbE FC.  In either case, an 8Gb FC pipe or 10GbE is still a larger pipe than most servers will fill on a regular basis unless used in one of the previously mentioned scenarios. A 10GbE unified fabric connection gives a server increased I/O bandwidth, while at the same time reducing overall TCO (less equipment up front, lowers operational and management costs in the long run).

Regards,

Omar Sultan

Cisco

Michael Kendall wrote re: Why doesn't networking obey Moore's Law?
on 03-13-2009 7:40 PM

Omar,

yes it is good that networking costs, on a per port basis, have come down appreciably. Even so, they have not dropped as quickly as server and storage on a per performance basis. Further, while overall IT server costs have gone down in aggregate, even as the raw compute capacity has greatly expanded, overall networking net costs have increased. Or put more simply, today most data centers spend less on servers per year and more on networking per year that previous periods.

Some of this is due to the economics of market share concentration. Theory states that as competition grows, industry player profitability becomes closer to the cost of capital. Servers have had extensive competition for years and as a result they have gross margins in the low twenty percent range. And this is why most governments encourage competition. It results in lower costs for buyers.

In the case of switches, there has been a concentration of market share with the result of gross margins closer to 60% plus. This is nowhere near the cost of capital  to develop and produce these products. The result is the higher costs we see for networking equipment. The good news for all of us is that competition is growing in this space.

One final thought on your post regarding FCoE. Your argument is that it is efficient to use excess capacity on 10 GbE networks for fibre channel traffic with the result of lower overall IT networking costs. This would be true if FCoE used a protocol as flexible and efficient as today's Ethernet and IP protocols, such as iSCSI, and therefore needed no new protocols and silicon to make it function. The problem is that the fibre channel protocol is very deterministic and inflexible, requiring different and new Ethernet protocols (CEE or what Cisco calls DCE) and new switch silicon to make it work over Ethernet.  And as you point out, these new protocols are not finished yet. And the new silicon is still new and more expensive.

So while there is some promise to FCoE it will probably take several spins of the technology and further testing to make this useful to customers. In the meantime there are perfectly good ways of combining data and storage traffic on today's Ethernet and lower cost infrastructure.

Regards,

Mike Kendall

HP

Omar Sultan wrote re: Why doesn't networking obey Moore's Law?
on 03-18-2009 5:30 AM

Mike:

Hmmm..the TAM and spend numbers I have seen would seem to say something different, but I guess it depends on how you cut the numbers.  I would agree that there are certainly different economics between server, storage and network for a number or reasons.

As far as the differences in margin between the product categories, I would offer that the difference comes from the ability create differentiated value for customers.  We are always welcome competition to the marketplace.  However, the lesson from the margin discrepancies is that success hinges on more  than just a low price.

As far as unified fabric goes, yes, we see it as delivering both long term and short term benefits to the customer--one of the reasons is because it builds upon the economics of Ethernet.  Certainly iSCSI is an option for customers with out an existing investment in FC, but for customers with a substantial FC investment, such as many of HP's customers, its not a cost effective approach, hence, FCoE.  Agreed, that FCoE can require new adaptors in the server and new switches, but I would disagree that this is more expensive.  For instance, Intel offers an FCoE stack for their existing 10GbE NICs and the Nexus 5000 is comparable to other 10GbE rack switches.  Looking at overall TCO, the possible incremental increase in capex is more than offset by opex savings.  We have a TCO calculator on Cisco.com to help folks figure out specifics (I am not going to post URLs, that would be bad form, but folks can find it if they are interested).

As far as "readiness" of FCoE, we have had a shipping solution, since last spring, in partnership with Emulex, Intel, QLogic, and VMware.  The FCoE standard if fairly close to be being finalized (by summer--there is an update on my blog) and, while the DCE/CEE standards are taking longer, they are making progress and we are committed to compliance to the final IEEE DCB standard.

We view FCoE as an access layer solution in the near term.  In light of that, as long as there is agreement on protocols on that access layer link, life is good.  Using the Nexus 5000 as an example, customers get the immediate benefits of FCoE to simplify their server infrastructure, the upstream 10Gb Ethernet and Fibre Channel links are fully standards compliant, so the solution can be easily integrated into the existing network.

Take care,

Omar Sultan

Cisco

Mark wrote re: Why doesn't networking obey Moore's Law?
on 03-21-2009 5:45 PM

Moore's Law states that the number of transistors on an integrated circuit doubles approximately every two years.  It has nothing to do with networking, which answers your question about why networking doesn't follow that trend. They are unrelated.

Add a Comment

(required)  
(optional)
(required)  
Remember Me?

Type the numbers and letters above:
Powered by Community Server (Non-Commercial Edition), by Telligent Systems