These days, server power efficiency is top of mind for everyone.  Five years ago, most customers had no idea how much power a server used.  Now, everyone knows - it's a lot.  In some cases, customers are making power consumption the primary criteria for vendor selection. 

So why is it so dang hard to get a straight story on exactly what options are the most power efficient?

In our experiences with power measurements, we found lots of ways to the results get skewed; whether intentionally or accidentally. We honestly try to avoid them, but here are some of the dirty little secrets a lot of vendors don't want you to know about their power testing results.  Consider these red flags next time someone spends lots of money in the Wall Street Journal to post a big claim with lots of fine print on power savings.

Lab Queens

One easy way for vendors to skew power results is to cherry-pick low power components.  Processors (even those in the same power grade) and memory DIMMs tend to consume significant power and can have wide variances in power consumed from one part to another. 

We call units built by cherry-picking components "Lab Queens".

Generally, these do not represent what a customer might actually be able to purchase.  Here's an example where we tried to avoid this scenario; the systems compared on this competitive power report on our website uses the exact same processors and DIMMs on all units tested, thereby eliminating differences due to component variances.  It would have been easy to create a Lab Queen and bump the results higher, artificially.

Configuration Errors

Peak power efficiency for a given system always requires the "best" configuration.  Memory power is mostly a function of DIMM count, and not capacity.  Therefore, to achieve minimum power, the minimum number of DIMMs should be used to realize the desired memory capacity. 

If you see a comparison where the DIMM count is different or not mentioned, you might notice the strong odor of dead fish. 

On the infrastructure side of the equation, a blade system can be configured with varying numbers of fans and power supplies.  Obviously some configurations will produce better power efficiency than others.  For instance, 4 power supplies run more efficiently than 6, assuming that 2+2 power redundancy delivers adequate power for a given configuration.  The most power efficient configurations though won't always be generated by a competitor when publishing a comparison.  Honestly, there are such big differences here that an apple to apple comparison is tough.  Only an HP BladeSystem dynamically throttles fan speeds based on demand across the different cooling zones in an enclosure.

Environmental Differences

The voltage level and room temperature can cause variances in the power supply efficiencies and the speed the system fans must run.  Consequently it is important that when comparing servers to verify that these parameters are held constant.  I once heard a story of an engineer setting up a box fan to blow air on his servers to try to reduce the fan power of the blades. 

Obviously he wasn't planning on counting the power from the box fan! 

Remarkably, there are published benchmarks that allow for external cooling to be deployed and not counted as a part of the system power.

Gaming the Benchmark

Any benchmark a vendor publishes is necessarily narrow in what it measures.  When looking at a published benchmark, you should consider how close the benchmark mimics your applications.  If the benchmark is similar to your applications, the results might be relevant to you.  If not, then you might do best just to ignore them.  Also, some benchmarks are very loose on configuration requirements, making it very easy to "game them" and making the results all but totally useless.  One benchmark that is broadly published in the industry falls into this category.

Where to go from here?

At this point, you are probably thinking "Wow! This is much more complicated than I really thought!"  And you are right.  Benchmarking fairly is very tricky business, even if you are trying to be fair.   The best results you can get are the ones you measure yourself.  In my next blog post, I'll comment on some of the techniques and challenges with measuring power yourself.