To help your customers work out how much usable space they get for their virtualisation in this "new paradigm", I've put up a helpful space calculator here;

blogs.netapp.com/.../an-hp-lefthand-duplication-calculator.html

I make it 35% usable space "at no additional charge".

Calvin,

Given the copious storage-related misinformation in your response to Alex, it seems a good thing HP also sells servers and networks, etc...  To wit:

(1) NetApp's SPC results (which HP also posts to) reveal an industry-leading 67% storage utilization at max performance.  Your 25% estimate is pure conjecture and indicates either ignorance or malice on your part.

(2) You can restore data from free Snapshots at no extra cost.  SnapRestore is a unique feature which can instantly recover upto 16TB at a time.

(3) NetApp SnapMirror has dozens more capabilities than LH Remote Copy, not the least of which is dedupe support.

(4) SyncMirror is free, and is independent of MetroCluster. It supports 5 simultaneous disk failures before bringing a volume offline.  Network RAID can't match that.

(5) FAS Cluster Failover is not a product - it's a feature.  And all FAS systems include it by default.

(6) See SPC comment above.  Where are those recent HP performance number again?

I certainly hope the quality of HP's storage products is superior to the poor quality of competitive "analysis" displayed here!

/R

"What I’ve heard from our experts is that with NetApp [...]usable capacity is around 25%"

Your experts are guessing. Let's see the math. O, hang on, you did show us, and it was pointed out that you got it completely wrong.  blogs.netapp.com/.../deduplication-g.html

So you have capacity based pricing (watch out for that 35%!!!) and we have a-la-carte and a space guarantee. That makes LeftHand cheaper how?

NetApp's deduplication story appears to be somewhat fictitious. Not only does NetApp's dedupe software negatively impact performance and take up additional space using fingerprint files, the deduplication default configuration takes the free space generated from running the taxing deduplication process and places it in the snapshot reserve space. I assume this is because snapshots grow like crazy on deduplicated volumes. LeftHand has no snapshot reserve space, so I would say we save more space than a NetApp SAN running deduplication.

Here are my NetApp capcity utilization calculations:

Usable Disk Capacity Adjustment 10.5%

Data ONTAP and WAFL Reserve 10%

RAID Parity (RAID 4, 1 Parity+1 Spare per 14 drive shelf) 14.3%

Snapshot + LUN Overwrite Reserve 50%

Usable Capacity = 34.5%

With Double Drive Fault Protection

RAID Parity (RAID-DP, 2 Parity+1 Spare per 14 drive shelf 21.4%

Usable Capacity = 31.7%

Add Data Replication (redundancy across sites)

MetroCluster Data Replication 50%

Usable Capacity = 15.8%

You're looking silly on your assertion that NetApp deduplication is fictitious. Here's some info from one of the biggest hospitals in Europe;

dumps of VMWare: 49 % of savings

dumps of SQL and Sybase : 66 - 68 % of savings (7 full dumps of the same database)

User share with the sources of all our software : 28 %

VMWare production : still in test, but at least 50%

Patient invoices (more than one million PDF files / year): 52 %

Or what about our 50% virtualisation guarantee on a NetApp FAS system, or 35% guarantee using other arrays (including the EVA) behind a NetApp V Series?  

Reserves are exactly that; a reservation, not an occupation. It generates warnings when you exceed it, but the system doesn't stop running; it can delete older snaps, for instance. That's another silly assertion you make in claiming that reserves eat into usable space. Set it to 0% if you don't want the warnings.

RAID-4 is not recommended for new systems. Ever.

A 14 disk 1TB (base 10) RAID-DP group has 8.729TiB (that's base 2) or 9.372TB (base 10) usable, ignoring the spare, since they're global to the array, not to the shelf. That's 68% usable. A spare takes 828.416GiB (base 2). I'll let you do the math on that.

Replication is thin (remember that?). Only the used is replicated, not the reserves.

That's a lot more functionality and usable than the LeftHand 35%.

Follow up from my previous post; I've just discovered that the data on the site I quoted anonymously is now public. (Premature post-itis!)

UZ Leuven, a NetApp customer since 2000 and one of the largest healthcare providers in Europe, leveraged a wide range of NetApp storage efficiency technologies across several IT functions, including server virtualization, e-mail archiving, file sharing, and database management. As a result, UZ Leuven streamlined its business operations while saving valuable time and money. For example, UZ Leuven leveraged deduplication technology within its NetApp MetroCluster storage system to reduce data storage requirements by as much as 80% in its virtual server environment. Combined with NetApp SnapMirror®, deduplication helped UZ Leuven dramatically reduce the amount of disk space required for backup, from 4.2 terabytes to 800 gigabytes.

www.netapp.com/.../news-rel-20090623-storage-efficiency.html

John,

There is a few mistakes in your claims above which invalidates your calculation (and how many more will you make?)...

RAID4 is no longer (already for quite a while) the best practice recommendation, RAID-DP (high-performance RAID-6) is. So no point in mentioning RAID4 any longer.

Then your data-to-parity and spare-to-disk ratios are way off.... and it's not the first time you HP guys misrepresent this. Check media.netapp.com/.../tr-3437.pdf for the gist of things:

On the parity disks:

===============

"NetApp recommends using the default RAID group sizes when using RAID-DP." (page 11). This is 16 for FC disks, so 14 data and 2 parity - in contrast of your claimed 12+2 for each 14-disk shelf. Care to redo your calculation?

On the spares:

============

"Maintain two hot spares for each type of disk drive in the storage system to take advantage of Maintenance Center." (page 4).

Then the number of spares with RAID-DP increases (by 1 only!) when the total number of disks reaches 84, 112 and 336 respectively. See the complete table on page 13.

Again, care to redo your calculation?

Also, care to explain the 50% Snapshot + LUN Overwrite Reserve "requirement"?

Alex,

You said I'm looking silly on your assertion that NetApp deduplication is fictitious. Ok, let’s put some facts behind your deduplication functionality based on excerpts from your “NetApp Deduplication for FAS Frequently Asked Questions” and "Configuring NetApp Deduplication with LUN’s" documents:

IS NETAPP DEDUPLICATION SUITABLE FOR PRIMARY STORAGE?

Yes for “light duty” primary applications. What we mean by light-duty primary storage are volumes that contain primary (1st copy) data, but are not performance-driven. Some examples of this would be VMware VM’s, home directories, document directories, and application volumes that experience heavy I/O loads during the day but are quiescent at night and on weekends. These volumes might very well benefit from deduplication if the system has the performance headroom to support the additional overhead imposed by deduplication.

Note: A customer can use LeftHand’s SmartClone functionality to accomplish the same savings for VMware VM’s, home directories and document directories, without a performance hit.

IS THERE ANY WRITE PERFORMANCE OVERHEAD AFTER ENABLING DEDUPLICATION ON A VOLUME?

As data is stored on a deduplication-enabled volume, digital fingerprint files are also stored. Less than 10% write performance overhead is required for this process.

IS THERE ANY READ PERFORMANCE OVERHEAD AFTER A VOLUME IS DEDUPLICATED?

When data is read from a deduplication-enabled volume, the read performance penalty will vary depending upon the original vs. deduplicated block layout. Unless the data has been written sequentially, the read impact would be minimal. However, if an application depends upon fast read performance i.e. sequential block recording, deduplication’s impact on read performance should be carefully considered before implementation.

DOES DEDUPLICATION REQUIRE ANY SPACE OVERHEAD?

Yes. Deduplication creates fingerprint file metadata which requires 1% - 3% of the total volume size.

ARE THERE ANY SPECIAL CONSIDERATIONS IN USING DEDUPLICATION WITH LUNS?

NetApp deduplication operates regardless of protocol. However, LUNs are a somewhat special case because of reservations and space guarantees. The default mode of deduplication and LUNs is to apply any space savings to the snapshot reserve area.

"Deduplication in a block-based (FCP/iSCSI) LUN environment is slightly more complicated. This is because of the space guarantees and fractional reservations often utilized by LUNs. With space guarantees, for instance, a 500GB LUN that is created will consume exactly 500GB of physical disk space. If the data within the LUN is reduced through deduplication, the LUN will still reserve the same physical space capacity of 500GB, and the space savings will not be apparent to the user."

WHAT IMPACT WILL DEDUPLICATION HAVE ON FILE FRAGMENTATION, SPECIFICALLY IN A LUN FILE?

If data has been written to the volume (or LUN) sequentially, deduplication will certainly disrupt the sequential structure of the data. If a customer is concerned with this, it indicates the customer is performance-oriented and thus not a good candidate for deduplication. Remember - the main purpose of deduplication is to reduce space on infrequently to moderately accessed data. There will always be some performance penalty associated with deduplication.

ARE THERE ANY ENVIRONMENTS WHERE DEDUPLICATION IS NOT RECOMMENDED?

Deduplication is not recommended for customers in environments where performance is paramount. Also, if the system deploying deduplication sustains heavy read/write activity without any off-peak hours for post processing, we recommend that deduplication not be deployed on that system.

ARE THERE ANY VOLUME SIZE LIMITS ON DEDUPLICATION VOLUMES?

Yes. Current maximum volume sizes for NetApp deduplication are as follows:

R200 4 TB

FAS2020 500GB

FAS2050 1TB

FAS3020 1 TB

FAS3040 3 TB

FAS3140 3 TB

FAS3050 2 TB

FAS3070 6 TB

FAS3170 10 TB

FAS6030 10 TB

FAS6070 16 TB

CAN I ENABLE DEDUPLICATION ON SNAPLOCK VOLUMES?

SnapLock is currently not supported with NetApp deduplication

WHY WOULD I WANT TO STOP A DEDUPLICATION OPERATION?

Since deduplication consumes system resources during processing, you may wish to suspend deduplication during times when critical file replication, backup, archival, or restoration is occurring.

IF I USE NDMPCOPY TO MOVE A FILESYSTEM WHICH HAS BEEN DEDUPLICATED, WILL I HAVE TO DEDUPLICATE THE DESTINATION AGAIN?

Yes. NDMP reads data at the file level, not the block level. Deduplicated data will be “un-deduplicated” by this process. Therefore, data copied using NDMP will not be deduplicated at the destination.

With the performance hit, all the restrictions, and the fact that you "rob Peter to pay Paul" with snapshots for SAN volumes, people may understand why I said that your deduplication functionality is fictitious.

Stopped watching the news, John? SnapLock and dedupe are already supported for quite a while and now even assessed by a independent firm - www.netapp.com/.../news-rel-20090513.html:

“NetApp deduplication, in conjunction with SnapLock, meets SEC compliance standards because it retains all metadata associated with the data it deduplicates," said Bob Williams, president of Cohasset Associates, Inc. “Another important factor is that NetApp performs byte-level validation before any deduplication of data occurs.”

Also read here: media.netapp.com/.../ar-netapp-snaplock-compliance.pdf

The mentioned max volumes sizes are also already retired, and have been upped across all platforms. What is the data source your referring to above? One from your own email archive?

We have never been shy about being honest about the best use cases for dedupe. Evidence shows that many, many customers are happily implementing dedupe across a variety of applications, low key to high profile including demanding VMware deployments.

But what's your problem anyways? There is NO dedupe available on LHN, I guess that's a bigger disqualifier altogether. And that's not fictitious, but a FACT!

(PS - you may want to repost your response on my previous comment as I accidently managed to delete it. So you may want to remove access to those delete links for outsiders...)

Ok Geert, using the NetApp’s Storage Best Practices and Resiliency guide on a 4 disk shelf system:

Usable Disk Capacity Adjustment 300GB drives = 10.5%

Data ONTAP and WAFL Overhead = 10.5% (Note: LeftHand’s SW overhead is only 1.6%)

Core Dump Reserve (not mentioned) = 1%

Hot Spares 2/56 = 3.57%

RAID-DP 8/56 drives = 14.3%

Recommended Snapshot Reserve = 20%

On page 15: “NetApp recommends SyncMirror and active-active configurations for very high levels of storage resiliency.” This is required to match the resiliency of LeftHand’s Network RAID level 2. = 50%

Net Storage Utilization = 30.5%

Geert,

It looks like I’m behind on the fixes in your latest release, but the larger issues highlighted from your older documentation still remain – performance degradation, fragmentation, and the inability to use deduplicated space savings in SAN environments.  LeftHand customers proactively prevent duplication in the first place through the use of SmartClones, and we eliminate space wasting things like reserves.

I’m not so sure I would be bragging about the fact that you have deduplication and we don’t when considering its negatives. If it’s so great it makes me wonder why NetApp is so desperate to buy Data Domain.

John

Yes, Network RAID 2 can sustain any random 5 disk failure. In fact, a 4 array system configured in Network RAID level 2 can sustain up to 2 complete array failures, and up to 6 disk failures in each of the remaining 2 arrays, and all at the same time.

John

Please stop showing NetApp snapshot reserves as allocated space. While I appreciate you have a position to defend, misrepresenting a competitor's technology this way is reprehensible. Saying it over and over again does not make it true.  

You and your colleagues have been pointed at NetApp documentation that confirms this, have access to NetApp systems that demonstrate this, and have been repeatedly told this by me and others many times.

@John

Thanks for the extract of the best practices guide, but the point was that you used the word "fictitious", and you've done so again. Are you aware of this word's meaning? What is fictitious about NetApp's dedupe?

John,

    you use the following figures

Usable Disk Capacity Adjustment 300GB drives = 10.5%

Data ONTAP and WAFL Overhead = 10.5% (Note: LeftHand’s SW overhead is only 1.6%)

Core Dump Reserve (not mentioned) = 1%

Hot Spares 2/56 = 3.57%

RAID-DP 8/56 drives = 14.3%

Recommended Snapshot Reserve = 20%

The interesting thing here is that you do what most vendors do when comparing their capacity utilisation vs NetApp, and that is to include snapshot reserves for netapp, but fail to include anything comparable for their own platforms.

Lefthand, like most modern storage systems have emulated NetApp's snapshot capability, and just like Netapp have the capability to let administrators determine how much space is allocated to it, though unlike NetApp, they dont provide public best practice reccomendations on how much should be reserved, not even a rule of thumb (which is what the 20% figure is all about).

My understanding is that Lefthand would require some snaphot capacity to be reserved to allow a nRAID system to quickly resynchronise itself if there was ever a  break in communication between the two nodes. The quesiton I have is how much capacity should be reserved for this, or indeed, how much capacity should be reserved for any of the scenario's where a snapshot might be used on a Lefhand "array" ?.

If the answer is "it depends, its up to the admin" and then fail to reserve any snapshot storage in your usable capacity numbers then do the right thing and compare your system to a NetApp configuration without snapshot reservations.

BTW, when you show your capacity utilisatoin figures do you also factor in TB to TiB conversions ? And where the heck did you get 1% for coredump space ?

Personally I think the nRAID stuff is pretty cool, much kudos for some clever engineering, but from my perspective  mandating  an extra mirror just to provide a base level of HA seems both expensive and wasteful, I suspect that many others feel the same way.

So now what's the Net Storage Utilization on a 4 array Network RAID level 2 system?

Nice Healthy debate here but I think it is trying to subvert the basic tenant of John's original post which is that HP provides end to End virtualization solutions for customers including Servers, Storage and Networking.  We even enable customers to colocate Virtual Servers and Virtual Storage on their servers running VMWARE.    

NetApp is providing Storage and inherently is not able to provide customers with this level of value.  The answer is always going to be if you buy a server buy NetApp storage.  

At HP not only might you not have to buy a storage device to support VMWARE as e offer the only VMWARE certified Software SAN, but you can run that functionality on NON HP hardware.  

We also provide a fully scalable infrastructure solution with the HP BladeSsytem Matrix for the most demanding environments.

So back to John's

John,

We have no snapshot reserve space. In fact, there is no means of setting this in the GUI – it is non-existent. Who else can say that? Everything on a LeftHand SAN is allocate-on-write, including volumes, snapshots and remote replicas. The customer has a real-time view into what each of these is taking up on their SAN and only needs to be concerned about the total available storage in the SAN.

No aggregates, no snapshot reserves, no fractional overwrite reserves, no logs, no root volumes and no igroups. HP-LeftHand eliminates all these complex storage management tasks – storage virtualization at its best.  Maybe HP should support our VSA running on an HP blade server managing NetApp LUNS to reduce management complexity?

John

Fictitious implies fabrication and suggests artificiality or contrivance more than deliberate falsification or deception. The true benefit of NetApp dedupe to the customer is contrived. In other words, the negatives can often outweigh the positives for customers, but NetApp only talks about the benefits because it's a contrived plan to attract customers through feature allure.

John/CalvinZ

1. Sorry for coninuting the comment thread outside of the original posts topic, its a pet hate of mine. I'd be more than happy to continue this debate on the netapp storage efficiency blog on where I will soon be putting up a post on the subject of space reservations for snapshots.

2. My assumption about Lefthand's snapshot reservation capability, came from the followin website

www.nasi.com/leftHand-networks_snapshot.php

wich says

Administrators can pre-allocate capacity for snapshots, and grow them via pre-set or administrator-defined increments.

I'm not sure if they're misrepresenting your product, or if there is a substantial difference between "pre-allocating capacity" and creating a "reserve", or if as it would appear from your reply that an administrator has no way of managing the space consumed by snapshots on a left hand array.

Have a nice weekend

Regards

john

John,

The info at that link is a few years old and looks like it was written prior to releasing thin provisioning 2.0 with allocate-on-write technology, which eliminates all snapshot pre-reservation/reserves and associated management.

>>>John Spiers: Yes, Network RAID 2 can sustain any random 5 disk failure. In fact, a 4 array system configured in Network RAID level 2 can sustain up to 2 complete array failures, and up to 6 disk failures in each of the remaining 2 arrays, and all at the same time.

Care to explain how? What Raid-level is used within an array system? How many duplicates of data is required? What utilization will result in this config?

>>>John Spiers: Note: A customer can use LeftHand’s SmartClone functionality to accomplish the same savings for VMware VM’s, home directories and document directories, without a performance hit. &  LeftHand customers proactively prevent duplication in the first place through the use of SmartClones,….

This seems like comparing cows to apples …

Deduplication of source data has imho really nothing to do with thin provisioned cloning (SmartClones, FlexClones).

Regarding dedup:

• OK, might be that not all environments benefit from source data dedup, however the environments that do will contain most of the data.

• OK, there’ll be some fingerprinting overhead (“Deduplication creates fingerprint file metadata which requires 1% - 3% of the total volume size”) however if it saves you 30—70% of storage to store your dedup’d source data  ……

• OK, LUN sizes are fixed (“With space guarantees, for instance, a 500GB LUN that is created will consume exactly 500GB of physical disk space. If the data within the LUN is reduced through deduplication, the LUN will still reserve the same physical space capacity of 500GB, and the space savings will not be apparent to the user."), however as data grows (50-100%/year?) a 500 GB LUN lasts much longer than a non-dedup’d LUN, and that’s what the user will notice.

Just my $ 0.02.

Its been interesting and rather entertaining reading some of the comments from both HP and Netapp.. It would appear that a lot of what is being talked about is what VMware has started to call Storage Virtual Appliances. Folks like Stormagic and Falconstor offer something similar.. I've leveraged this technology (free from stormagic's website) to enable and give me VMotion and HA.. I'm in the process of downloading the Falconstor solution but must say I've been impressed so far with the fact that I don't need to buy seperate shared storage to give me these advanced server virtualization features...

Additionally, here's a solution to VDI like none other I've ever seen!