HPC Cluster Edge Blog - All Comments

re: SLURM: a Simple Resource Manager no more!

jeux — Mon, 21 Sep 2009 09:05:11 GMT

SLURM is an open-source resource manager designed for Linux clusters of all sizes. It provides three key functions. First it allocates exclusive and/or non-exclusive access to resources (computer nodes) to users for some duration of time so they can perform work. Second, it provides a framework for starting, executing, and monitoring work (typically a parallel job) on a set of allocated nodes. Finally, it arbitrates contention for resources by managing a queue of pending work.

re: Interconnect technologies: InfiniBand, 1G Ethernet, 10G Ethernet

disque dure externe — Sat, 19 Sep 2009 10:11:14 GMT

Off late, a vast number of interconnect technologies such as InfiniBand, Myrinet and Quadrics have been introduced into the System-Area Network (SAN) environment; the primary driving requirements of this environment being high-performance and a feature-rich interface. Ethernet, on the other hand, is already the ubiquitous technology for Wide-Area Network (WAN) environments. Traditionally, SAN technologies had been shut off from the WAN environment due to their incompatibility with the existing Ethernet compatible infrastructure. Similarly, Ethernet has traditionally not been considered a SAN interconnect due to its close to order-of-magnitude performance gap compared to other SAN interconnects such as InfiniBand, Myrinet and Quadrics (informally called Ethernot networks).

re: Do you suffer from cluster monitoring schizophrenia?

mark.seger@hp.com — Fri, 29 May 2009 10:50:17 GMT

Just to followup on my original post, I realized that by focusing on central vs local data collection I missed raising an even higher level decision that must be made and that is the purpose for the monitoring. Many people are simply looking for a high level view of the overall state of their cluster's health such as how many systems might be down, whether any systems are idle or perhaps not running close 100% CPU utilization - remember that in HPC, 100% is typically a good thing. There are certainly other things that can be identified in this manner as well such as memory utilization or networks or storage systems running at near capacity. This type of monitoring might also be useful for longer term capacity planning. All worthwhile causes and probably all met with a central monitoring approach.

But my next question then becomes what do you do when your central management system shows systems crashing or not running at their peak? Or any of a multitude of other problems? Since you've been collecting all this data in a central location, surely the answer must be in there and all you'd need to do is look for it. In fact, there is often enough data in here to help identify the problem.

However, I would claim that in many cases there isn't enough data since you had to trade off the level of detail being collected in favor of central collection. This is where I believe local data collection can come to your rescue.

-mark

re: HPC: The Innovator’s Great Divide?

guodong.zhang@hp.com — Wed, 27 May 2009 19:11:09 GMT

Speaking about how High Performance Computing (HPC) helps the competiveness of corporations, there is a recent article on Forbes.com titled "American Business's Secret Competitive Weapon: HPC" by Matthew Faraci that tells compelling stories on how companies could take advantage of HPC to cut cost, improve product designs, and reduce the time to market. Very interest article, and highly recommended to read :

www.forbes.com/.../high-performance-computing-leadership-managing-hpc.html