That's a shame, because I was hoping to be spectacularly long winded. Never mind :-) the rest can wait. Let's deal with these one at a time.

My calculator, unlike your attempt, was based on "HP LeftHand Best Practices Guide - What Not To Do", available at h20000.www2.hp.com/.../c01750870.pdf.

-----

Critical Data on Replication Level 0 Volumes

If your data is critical and it is stored on a volume configured with a replication level of 0, this data is at risk. If any storage node in the cluster fails, the data will not be available. It is best to have all volumes configured with a replication level 2 at a minimum.

-----

That's 35% usable, yes? The way I see it, my calculator still doesn't need batteries, and you need to correct this blog entry. Unless advising your customers to ignore best practices is standard advice from HP.

Now snapshot reserves. As I indicated to John Spiers (HP), Jim Haberkron (HP) and now you, NetApp's snapshot reserve is

1. thin provisioned (we allocate on write)

2. tunable from 0% to just short of 100%

3. not a hard limit, but a threshold at which actions are taken. Like a red area on a dial; to let you know that there might be an issue.

Actions can include deleteing old snapshots to make new space, ignoring it because the threshold is just that and there's enough real space on the system to continue, and so on.

Just like LeftHand, but with that extra functionality that our users find useful. And an AutoSupport function that flags this up for the user to generate reports, and for NetApp support to proactively use when we do our free six monthly storage audits (see here blogs.netapp.com/.../squeeze-until-t.html)

So this, quoting you;

----

"Because other vendors, NetApp included, have to set a reserve for each of those things. I don't know what their reserve - NetApp will have to answer that. And if you don't use the reserve, guess what - it's wasted space that can't easily be reclaimed."

----

is demonstrably a whole load of whooey.

Part 3.

You said;

----

With HP LeftHand, thin provisioning is leveraged throughout the architecture - with NetApp, its limited and space is committed when you create objects. Remember, HP LeftHand uses allocate on write - making the most of the storage capacity. With our thin provisioning, the capacity is always available to you when you need it without any pre-commit of space.

----

Wrong. Space is ONLY committed when the data is written. NetApp uses FlexVols, which are the virtualised containers; designed, from the ground up, as fully thin. No data -- no "reservation".

This is fundamental to the way NetApp WAFL works. Reserve 100GB, and nothing gets allocated. Write 20GB, and only 20GB gets written. That 80GB is still usable by other "objects".

Good practice has a sanity check in there. Reserve 10TB, and only have 1TB free of real storage? Are you sure? If the answer's yes, then a NetApp system will let you do it.  

I've taken the time to understand LeftHand's technology. Do me the favour of taking a little time to understand NetApp's.

The big graph at the bottom.

Jim Haberkorn used the same diagram, and I answered here; blogs.netapp.com/.../an-hp-lefthand-duplication-calculator.html

You're making the assumption that customers are losing performance. I paraphrase a reply on the blog (I think it's a NetApp reseller);

----

No one (NetApp or its partners) size systems based on the left hand side of that graph, and then *add* spindles to restore performance. The sizing is based on the right side of the graph, and any of that extra performance from low utilization is of benefit.

----

Jim Haberkorn (HP) felt the need to pen this frankly slanderous statement in reply.

----

If a NetApp sales team knows they can double their IOPS by configuring their system at 10% capacity instead of 75%, what is likely to happen in a competitive situation where performance is the deciding factor? Answer: Human nature being what it is, the sales team will perform their demo, POC, or benchmark at the configuration level that will get them the best performance and win the deal

----

John Martin from NetApp answered that for me.

----

Speaking as someone who faces these pressures on a regular basis, I can say that your "Human Nature" argument is fallacious. Reps who try that kind of thing dont last long at NetApp, actually, they dont even get hired. We can pick and choose our candidates, and trust me, we are very very picky. We've got some very good (not to mention conservative) sizing tools which we use all the time when putting configurations together. Maybe things are different at HP, or maybe your confusing us with some other vendor.

----

As to dissing George Wagner, HP LeftHand expert, this is what I said after the "Sorry George!".

----

Because if I spread out a volume across two or more nodes, and I don't nRAID2 it, then when the SPOF node goes down, I loose all my volumes, all my snaps, clones and all my data. Everything on my entire LeftHand system. nRAID isn't optional; it's not about high availability, it's about protecting against data loss, a different issue altogether.

----

That was never answered. What am I not understanding here, based on the "Best Practices: What Not To Do" document ("with a replication level of 0, this data is at risk"). Did I misread it, or did it not really mean "at risk"?

There's a whole load more, but I'm tired calling out your mistakes, you'll be tired running around getting some reasonable answers, and your readers will be sick and tired of both of us.

You're right, these kind of discussions don't serve our customers very well. The shame is, we wouldn't be having them if HP hadn't let its LeftHand story spin out of control.

I'm done. Have a good weekend.

Alex,

It appears that you still have a misunderstanding of LeftHand’s architecture and its benefits to customers. You continue to reference LeftHand’s configuration that delivers advanced data protection with HA and compare it to NetApp’s typical configuration without advanced data protection or HA.

Let’s start with Network RAID 0. You said this configuration has a SPOF. The single point of failure for a LeftHand system in Network RAID 0 is a disk shelf. If a disk shelf is down for any reason, all volumes spanning that disk shelf are down. With NetApp’s typical configuration, if a disk shelf is down, all volumes spanning that disk shelf are also down. NetApp requires that all volumes be Syncmirrored to protect against disk shelf failures and recommends Fabric MetroCluster for multi-site HA. Like I detailed in a previous post, NetApp’s storage utilization in a SyncMirror+Fabric MetroCluster configuration is 15.8%, and more than 2x the price/TB.

Here are excerpts from NetApp’s Best Practices and Resiliency Guide (wp_3437.pdf) that validate my points:

• “To protect business-critical data from any local storage-related issues, use local SyncMirror in conjunction with active-active configuration clusters.”
• “Local SyncMirror provides synchronous mirroring between two different volumes or aggregates on the same storage controller so that a duplicate copy of data exists, resulting in higher storage resiliency and data availability."
• “At the local level, SyncMirror provides storage resiliency capabilities that SnapMirror or even active-active configurations by themselves do not. The additional resiliency features that SyncMirror offers over SnapMirror in synchronous mode are protection against system downtime due to shelf failure, triple disk failure for RAID-DP groups, and Fibre Channel loop failure.”
• “NetApp recommends that MetroCluster be used with SyncMirror for the highest level of storage resiliency.”
• “Although both stretch MetroCluster and storage controllers in active-active configurations are supported up to 500 meters apart with 2Gb/sec connectivity, or 270 meters with 4Gb/sec speeds, this generally means that all controllers are in the same data center. Fabric MetroCluster allows the active-active configuration to be spread across data centers up to 100 kilometers apart. In the event of an outage at one data center, the second data center can assume all affected storage operations that were lost with the original data center. SyncMirror is required as part of MetroCluster to ensure that an identical copy of the data exists in the second data center in case the original data center is lost.”

Please do not compare LeftHand’s Network RAID 2 + RAID 5 to NetApp’s base configuration. The only valid comparison with LeftHand’s recommended configuration is NetApp with all volumes Syncmirrored locally, and replicated using Fabric MetroCluster if multi-site HA is a requirement. Otherwise Network RAID level 0 with RAID 5 or RAID 6 should be used for the comparison.

Thanks for reponding.

One nit though. You keep telling me how NetApp products work, whihc is cool, but without telling me or anyone else how LeftHand works. You keep saying "Please do not compare LeftHand’s Network RAID 2 + RAID 5 to NetApp’s base configuration." It's you that keeps doing the comparison, not me.

So let me try one more time.

Here's a sample system; a 2 node, 24 disk 1TB SATA LeftHand SAN. I'm going to try and make it HA.

1. I need to run nRAID2.  See footnote A for why.

2. I need to make a quorum, so I need another server running the Failover Manager (FOM). This is to avoid "split brain" and allow auotmated failover.

3. I run RAID5 at (5+1)*2 on each of the nodes. (No spares, we'll work out how that's handled at a later date.)

For my 24TB and 2 nodes and another server, I get 8TB usable. Right?

Footnotes

A.  "HP LeftHand Best Practices Guide - What Not To Do", available at h20000.www2.hp.com/.../c01750870.pdf.

-----

Critical Data on Replication Level 0 Volumes

If your data is critical and it is stored on a volume configured with a replication level of 0, this data is at risk. If any storage node in the cluster fails, the data will not be available. It is best to have all volumes configured with a replication level 2 at a minimum.

-----

>>1. I need to run nRAID2.  See footnote A for why.

Yes, to have storage node/disk shelf redundancy you need to run Network RAID. Just like NetApp, needs all volumes to be SyncMirrored to have disk shelf redundancy.  This is very similar to NetApp’s statement:  “To protect business-critical data from any local storage-related issues, use local SyncMirror in conjunction with active-active configuration clusters”

In reality customers create unreplicated volumes for temporary work space that’s used for database queries, temporary backups, and most fixed content if it is backed up somewhere. To say all volumes should be under Network RAID 2 is like saying all NetApp Aggregates should be Syncmirrored.

>>2. I need to make a quorum, so I need another server running the Failover Manager (FOM). This is to avoid "split brain" and allow automated failover.

Quorum management is inherent in the software and runs on every node. The external FOM is only required if you have 2 nodes, and possibly in some multi-site configurations, because there is no way to establish quorum with 2 nodes.

>>3. I run RAID5 at (5+1)*2 on each of the nodes. (No spares, we'll work out how that's handled at a later date.)

>>For my 24TB and 2 nodes and another server, I get 8.6 TB usable. Right?

You get 8.6TB 8TB if all volumes are Network RAID 2. With Parity Based Network RAID you can get 13TB or more depending how you set the hardware RAID.

Now we're getting somewhere.

A LeftHand node is not a NetApp shelf; it's a server. The reliability of a dumb shelf with next to no hardware and no software running on it is orders of magnitude greater than a server running Linux with SAN/iQ on top, acting as a node. Ask the boys over at the EVA part of the business, they'll tell you -- or take a look at the uptime stats from some research HP did on the uptime of various OSes; h20223.www2.hp.com/.../426962-0-0-225-121.html. That has Linux with a downtime of 57.12 hrs. I've never seen a shelf of disks with that kind of downtime, have you?

More questions

1. Is there any "lost write" support on the LeftHand RAID controller? Is data reliability suspect on SATA and SAS without checksum support?

2. No global spares. No sparing at all from what I can see; so when a disk fails, as it surely will, does the node run in degraded mode and possibly performance issues until a replacement is made? There's no "call home" facility, so does the customer have to note there's a problem and then call you?

3. What is Parity Based Network RAID, and does LeftHand acually do that? I thought it used mirroring, How would I create an HA LeftHand system with 13TB on two nodes?

4. So you're recommending running Network RAID 0 (i.e., no network raid) and taking backups for fixed content. As this is an iSCSI box, presumably he would serve this content up via NAS, either NFS or CIFS, requiring another server. Why not just put the fixed content on the cheap server? Or why doesn't he buy a FAS2020 HA box that does iSCSI, CIFS and NFS, is protected with dual parity RAID, can dedupe the fixed content and has an uptime in excess of 99.999%?

Alex, my friend: You can dish it out. You need to be a little better about taking it. If you think you are being treated unfairly by HP in these discussions then you need to go back and read some of your own blogs.  

In any case, I’d like to address your point about how I ‘slandered’ NetApp by suggesting that now and again, due to human nature, NetApp just might be tempted to size a configuration at the low-end of their capacity range whenever performance was a big factor in the purchasing decision. If you remember, in our previous blog discussion on the subject, it was stated by someone in NetApp’s defense that NetApp only configures systems based on the capacity listed on the right side of the chart – in other words, in the 50-100% capacity range – and further it was stated above that any sales team that did not follow those configuration rules would be fired.  I just took a look at NetApp’s last three SPEC SFS results and the used capacity was 36%, 18%, and 12%. So, tell me, why would a NetApp sales team be fired for following the same practices NetApp uses in their published benchmarks? Or maybe NetApp fires their test engineers for that as well. If so, you guys must have a lot of turnover.

But in any case, since you like to talk about NetApp performance, let’s continue the discussion on another blog that I will be posting shortly.

Best regards, Jim

>>Now we're getting somewhere.

We are? This discussion is about best practices and storage utilization and you’re conveniently ignoring NetApp’s recommendations in the Best Practices and Resiliency Guide, but I’m not surprised, we’re familiar with NetApp skirting their real issues like performance degradation as the SAN fills up, which LeftHand does not have.

I don’t see what this table has to do with anything.  I don’t see ProLiant Blade Systems or DL servers in the table. The numbers are based on industry data incorporating a variety of server vendors, including white box junk, plus it’s outdated. The good news is that SAN/iQ is hardware agnostic, unlike NetApp, and you might want to envision running it on HP Integrity NonStop Servers. Last I checked NetApp uses x86 chipsets for their controllers and runs a Linux/BSD kernel, so I wouldn’t be throwing rocks.

With NetApp’s legacy scale-up systems, GX being the only exception, the controller head services all disk shelves, whereas the LeftHand controller only services a single disk shelf, making redundant controllers per node less of an issue, especially with Network RAID.

Like I said before, LeftHand doesn’t have a lost write problem and has many features that insure data integrity and even corrects BER events on-the-fly, which substantially reduces disk failures. All this is storage 101. As far as “call home”, HP has a broad range of support options, one of which includes intelligent Health Check, which is able to predict potential problems and warn a customer to take certain actions proactively.

Parity Based Network RAID. Actually, I’m not supposed to talk about it, but stay tuned.

I know why he doesn’t buy a FAS2020. First of all, I didn’t think NetApp recommended running dedupe on their low-end Filers because of the overhead.  Based on TR3437, I’m calculating that the FAS2020 has 32.2% storage utilization.

If the FAS2020 box goes down your entire SAN is down, along with a long list of other shortfalls that I can elaborate on if you would like. As your FAS2020 grows to a maximum of 68 disks, and performance begins to degrade, it’s a controller rip-and-replace upgrade, software licensing uplift, support uplift , and downtime.

What customers like about LeftHand is you just add nodes and walk away, scaling non-disruptively from a FAS2000 equivalent sized and beyond, with no downtime, and no software licensing uplifts, and of course all features included.

And why not avoid the hassle of migrating all your file server shares to a NetApp Filer when you can use the LeftHand SAN as the storage for your existing file servers?

Overall it sounds like a much more economical way to go than the FAS2020.

I'm going to call it a day here. I still have lots of questions, but I suspect I'm not going to get them answered; so they'll just have to hang in the air.

How exactly does a LeftHand system avoid the lost write problem? How do you rebuild a multi TB node across Ethernet? How willl LeftHand's DSM support FCoE, or is it stranded on Ethernet iSCSI forever? Will dedupe remain a pipe dream for LeftHand users because of the multiMB unit of space allocation? Does HP have an uptime figure they quote for a node, and a methodolgy for supporting the claim since there's no call home facility?

Parity Based Network RAID. Now that sounds interesting. Thnks for the heads up, and good luck implementing your version of CleverSafe.

Thanks Alex,

Who's CleverSafe? I'll have to check them out. Yes, no loss writes. Rebuild times are what you want them to be because re-stripes can be throttled. A multiple TB node takes the better part of a day to re-build depending on the throttling setting, although if the node is down for any reason other than a multiple disk failure beyond the RAID coverage then it is incrementally rebuilt, which can take a matter of minutes.

Dedupe? I guess you haven’t been reading:  www.hp.com/go/deduplication

We support internal and external de-dupe solutions today. When customers understand the performance penalties and other shortfalls of NetApp’s in-band de-dupe, and the advantages of doing this with secondary storage, e.g.  HP, Data Domain and other leading solutions, most customers chose a “Data Domain like” secondary storage solution. I guess this is why NetApp was desperate to get their hands on Data Domain.

HP LeftHand availability with Network RAID 2 + RAID 5 is better than five 9s.

"Dedupe? I guess you haven’t been reading:  www.hp.com/go/deduplication". John, are you serious?

Com'on we all know that trick; what about end-to-end pervasive dedupe by LHN? I guess that's the real question here...(remember to throw in a solution for file and block data).

And then;

"...understand the performance penalties and other shortfalls of NetApp’s in-band de-dupe...". Any proof points?

The world has a different view already for two years:

news.moneycentral.msn.com/.../article.aspx

Geert,

LeftHand has many customers implementing dedupe successfully with HP technology or 3rd party solutions like Data Domain.

I’m not sure the world has a different view. Out of those 37,000 systems I assume most of those are secondary storage or VTL systems. If NetApp has a complete dedupe story across primary and secondary storage why would they  be willing to fork out billions for Data Domain? It just doesn’t make sense.

The NetApp performance hit for de-dupe on primary storage is well known and documented by NetApp.

Some new news from HP:

www.theregister.co.uk/.../hp_buying_ibrix

Hi Alex,

NetApp talks about having 99.999% uptime and providing true HA. I came across a Avanade PDF on NetApps website (media.netapp.com/.../Avanade_Testing_Center_NetApp_Whitepaper_Exchange.pdf)  that discusses testing the HA functionality of the FAS3050c array. I was quite shocked to find out on page 6 that both hard and graceful failovers caused 2 minutes and 27 seconds of user downtime. Maybe I'm wrong, but I don't see how that downtime is acceptable unless you are using it strictly for NAS. I would hate to see what would happen to a database or VMware if that were to take place while they were running.

I am currently a HP LeftHand Networks customer and have been very satisfied with their product. I have unintentionally as well as intentionally caused HA to kick in by failing LH node(s) and have never had a problem. I have never seen any user downtime because of it, and we are running Exchange, SQL, File, VMware, Oracle, etc. Unless NetApp has made some changes to improve the Filer failover time, there is really no comparison between HP LeftHand and NetApp. When it comes to High Availability, HP LeftHand provides a far superior product based on my own personal experience. Granted I do not have a NetApp system to gain personal experience with, so I have to take the reference PDF for what it is worth, especially since it is coming from NetApps website.

Thanks,

Chris

Chris-

If you look carefully at that Avanade paper, you will see those failover times have nothing to do with the NetApp storage array.  Those downtime numbers represent the time it took for MSCS to stop/start Exchange services gracefully or harshly.  In fact, Avanade didn't induce a storage array failure during their testing, so it would be erroneous for you to compare your personal LeftHand HA experience with NetApp based on that paper.

-Ken