Around the Storage Block Blog : deduplication

Tales of cold dead fish

CalvinZ — Fri, 29 May 2009 13:46:00 GMT

By Lee Johns

Do you love Sushi? I certainly have come to. Great taste, healthy - what could be better? It seems everywhere I turn today there are new Sushi restaurants springing up. Sushi is it! It is one of the old adages of marketing that if Sushi had been marketed as "Cold Dead Fish", it would not have been as successful.

Now to the topic of this blog post. The new HP StorageWorks X1000 and X3000 Network Storage Systems - our unified storage products announced this week feature a capability called single instancing. These products provide an easy way to deploy unified file and block services to new and existing SAN environments and the single instancing feature eliminates copies of files on the system and can reduce the space consumed by up to 35%. Now I am not an expert but that seems to me to be deduplication. I understand that there are different types of deduplication. There are different types of sushi too. Apparently someone at Microsoft decided to use single instancing as the term for file deduplication. Whoever it was, they likely are not a fan of Sushi.

The new X1000 and X3000 HP StorageWorks are unified storage systems that combine file and application storage and feature file deduplication that can save up to 35% of space. There - I've said it. . .

Tweet this!

I've been personally impacted by lost tapes

jim hankins — Wed, 17 Sep 2008 17:55:00 GMT

Hi Folks,

Yesterday I received a letter in the mail at home that started off:

Dear Sir or Madam,

We are writing to let you know that computer tapes containing some of your personal information were lost while being transported to an off-site storage facility by our archive services vendor. While we have no reason to believe that this information has been accessed or used inappropriately, we deeply regret that this incident occurred....

So the first question I have is how does an archive vendor lose tapes? How hard can it be to take the tapes from your customer put them in a secure truck and drive them to the storage facility? Isn't that your whole business model - you will pick up, transport and store these tapes safely and securely 100% of the time?

Now I understand that any activity with humans involved cannot be guaranteed to work 100% of the time. So what really happened? A bit more of an explanation would have been helpful, such as the truck was in an inadvertent accident and the contents of the truck were spilled into a river or all over the highway and could not all be recovered. Without more details I'm left wondering did someone make off with the tapes by accident or on purpose? Or was this just sloppy work by the company?

Anyway, I hope this is a call to action for this company to do at least two things to prevent such an incident in the future.

1. Look into tape encryption such as the LTO-4 offers. I would have been more much pleased if that second sentence read "While the tapes were physically lost, the data they contained cannot be accessed or read by anyone because the data on the tapes is securely encrypted with sophisticated technology requiring encryption keys to make the data readable. Our security policy ensures that these keys are always stored in or transported to physically separate locations from the computer tapes."

2. Consider the use of replication and electronic vaulting for moving data off-site for archiving. With new technologies such as deduplication and low-bandwidth replication, this company would perhaps be able to reduce the amount of data that is stored on tapes and physically transported to archive storage. Again, I don't know the specifics here, but as an example let's say this company had four sites that they were backing up to data to tape and transporting those tapes to off-site archives. With replication and electronic vaulting, they could replicate data from three of their sites to just one site for backup to tapes and then only have to move tapes from the one site to archive storage thereby reducing their risk exposure by 75%.

If you're worried about how a similar incident could impact your company and what risks are involved HP is here to help. We can work with you to significantly reduce your data security exposure from the desktop to your data center. On the storage side, we offer a FREE storage security risk assessment. For more details on HP's other data security options beyond storage please check HP's Security web page.

Deduplication, online storage, and cannibals

CalvinZ — Mon, 11 Aug 2008 16:06:00 GMT

By Warren Smith, StorageWorks Competitive Intelligence Manager

"Every fight is a food fight when you're a cannibal."

That's an amusing quote by Demetri Martin, an American comedian. But it could be the motto of NetApp. Last week, NetApp announced that customers could use their V-series storage to dedupe primary storage of other storage manufacturers, including EMC, HDS, and yours truly, HP StorageWorks. Customers should ask themselves, "Is this a dinner invite from a cannibal?"

Without further NetApp explanation of how this deduplication facility would be realized in other vendor storage, what configuration constraints apply or any other substantive details, NetApp made the unilateral claim. The cynic in me wants to think that NetApp made this announcement because the advanced single instance storage (ASIS) deduplication in their storage works sub-par in their storage and they honestly wanted to offer customers the advice to try to use ASIS in someone else's storage, on the chance that it might perform better. But seriously, the cannibal instinct of some storage vendors typically seeks to propagate their technology problems across all possible systems and eat into sound working technology.

Can I share the reason that this offer from NetApp is not a good idea? And it is not because it comes from NetApp. This idea is not a good idea because applying deduplication in primary storage is injecting an invasive process and performance impacting process into the heart of your business operations. The ASIS deduplication process necessarily takes valuable compute cycles from the high priority business application processing that drives many businesses. NetApp appropriately warns customers about the performance impacting effect of ASIS.

The following quotes are taken directly from NetApp's Deduplication Best Practices section in the text referenced below:

"If there is very little new data, run deduplication infrequently, because it doesn't make sense to unnecessarily consume CPU resources."
"Use the auto mode so that deduplication only runs when significant additional data has been written to each particular flexible volume"
"Stagger deduplication schedules for the flexible volumes so it runs on alternative days."

Source: Technical Report: NetApp Deduplication for FAS Deployment and Implementation Guide, TR-3505. Network Appliance, Inc. 16 April 2008.

And for sure, NetApp wants customers to know that there are limitations to the ASIS deduplication functionality. In an article on the ChannelWeb website, Chris Cummings, senior director of data protection solutions for NetApp, is quoted:

"In order to activate the dedupe license, which is free, customers do need to spend about 10 minutes filling out a form that states that they know there is a chance of performance degradation when implementing the technology, depending on data type and other factors", Cummings said.

Source: http://www.crn.com/storage/209901632

Despite these vendor advisories to be careful with the use of ASIS, some customers may think, "That's OK, I can just run my dedupes at midnight." But we have seen testing data that the performance impacting effects of ASIS can also be experienced even later when the deduped volume is used in the normal applications operations. And for the many storage customers desperately seeking to reduce their data duplication, there is good news.

The intelligent place to perform data reduction is in secondary and lower tier storage and during backup operations, and away from primary storage. HP's Deduplication Strategy provides for just that, intelligent data reduction in the backup regime. Our Deduplication Strategy, which offers "Accelerated" Deduplication for enterprise customers and "Dynamic" Deduplication for smaller businesses, is also complimented by storage space efficiency designs in the HP StorageWorks EVA that include Dynamic Capacity Management, implemented via EVA Software.

And so, it is clear that the marketplace has a keen interest in data reduction methods and technologies that hold the possibility of reducing the volume of data under management in customer storage systems. It is also clear that some vendors have sought advantage for themselves with this marketplace phenomena and will continue to seek to capitalize on their untested hype and promises that are made with regard to their products. And remember, never accept an invitation from a cannibal for dinner.

Examples of Data Deduplication Ratios for File Serving, SQL and Exchange

jim hankins — Mon, 28 Jul 2008 20:46:00 GMT

By Jim Hankins

If you remember back in my HP Deduplication - Part 1 post when we announced our new deduplication products back in June, I said that the deduplication ratio you can expect from a product can vary based on a number of factors. We now can share with you deduplication test results from our D2D4000 Backup System conducted by a 3rd party, Binary Testing Ltd.

Binary Testing conducted testing that backed up and deduplicated data for file serving, SQL and Exchange environments with various data change rates over a simulated three month backup period. The results can be found here: http://h71028.www7.hp.com/ERC/downloads/4AA2-0799ENW.pdf

Again, your mileage may vary but this report should give you some idea of what's possible if your business runs these types of applications.

EMC: HP's Whac-A-Mole VTL (& HP Reponse Part 1)

jasontreu — Mon, 28 Jul 2008 16:55:00 GMT

I wanted to share a response I made on an EMC blog called, "The Backup Blog."

We will also be sharing a response for our VLS product as well. There was too much to respond to in one post! Stay tuned!

__________________________________

As a Product Manager for the HP StorageWorks D2D Backup System product line, I wanted to respond to your blog entry on July 16th, 2008, entitled "HP's Whac-A-Mole VTL".

First, I would like to agree with you that our D2D Backup Systems may be limited in scalability and capacity - that is, depending on one's perspective. For a large Enterprise or data center, this is certainly true. However, these products are positioned for small and mid-range businesses (SMB), as well as for remote/branch office sites (ROBO), that require improved backup and restore capabilities over traditional backup to tape (disk-to-tape) data protection schemes.

Rather than take the approach of many other storage vendors, including EMC, which try to "tweak" higher-end solutions to meet the needs of smaller customers (think square peg, round hole), HP has designed a portfolio of disk-based data protection solutions specifically for customers that have smaller budgets and storage requirements.

Our new D2D Backup Systems, which you reference, range from 3TB to 9TB, offer comparable (if not better) performance than other solutions within the same class of disk-based storage products, and are priced starting at $6500 for a complete system (including the deduplication software).

Additionally, by using inline, hash-based deduplication (which you fail to mention), we are compatible with a wide range of backup applications that customers already have installed - unlike the EMC Avamar solution which requires customers to "rip and replace" their current backup applications.

HP's D2D Backup Systems are easy to install (typically, in less than an hour) and are just as easy to manage, requiring little, if any, need for expensive installation/service/support contracts (ala EMC, IBM, Data Domain). Also, HP's D2D Backup Systems utilize target-based deduplication (again, which you fail to mention) which is far less dependent on client resources and much less likely to impact the availability and performance of client applications - unlike source-based deduplication solutions such as EMC Avamar.

For years, I have seen press releases and heard EMC executives tout their dedication to providing SMB solutions. EMC purchased Dantz Retrospect, partnered with Dell, and created the Insignia product line - all in the name of garnering market share in the fastest growing IT customer segment - the SMB segment. Yet, EMC continues to build Enterprise solutions, then remove a few hard drives, take away a bit software, and then claim it has solutions for the SMB segment. Sorry folks, that's not how it's done. You can't claim to cater to small businesses when your pricing starts at $20K to $50K for a low-end data protection solution. While EMC may sell some volume in the mid-range segment (everybody and their brother competes there!), I think you have missed the boat on smaller businesses which comprise 80% to 85% of worldwide businesses. Probably time to check your market research...

Lastly, on the topic of product names, as you so thoughtfully pointed out how bad "D2D" was, I think the uninformed IT customer would find EMC's product names quite amusing - Avamar, DL 3D, Centera, Clariion, and Celerra. My bet is that the uninformed IT customer would think that your either selling pharmaceuticals to middle-aged men (if you get my point) or selling props from an episode of Battlestar Galactica.

Thanks and look forward to our future conversations.

Mike Ewell
Product Manager for the HP StorageWorks D2D Backup System product line

Simply Business Protection for small and midsized IT environments

brad.parks — Wed, 02 Jul 2008 20:23:00 GMT

- by Brad Parks

We’re all well aware that data protection equates to business protection and some of the recent posts on this blog go into detail on how HP is fundamentally changing the rules of the game with our deduplication technologies embedded in our disk based backup solutions.I’d like to expand on some of these topics while specifically looking at the needs of small and midsized customers. A key challenge midsized businesses share with enterprises is the need to reduce the cost of protecting their data but there are unique challenges as well. Small and midsized businesses are some of the most exciting and dynamic environments to work in because there can be wild swings in customer demand and therefore data growth. IT Managers in these environments must wear a lot of hats and be as flexible as their IT environment in order to respond to changing business conditions. Consolidating backup and moving from pure tape based backup strategies to multi-tier disk to disk to tape (D2D2T) methodologies is one way that IT managers can be less reactive and more intelligent with regards to their data protection.As mentioned earlier, one of the most significant new advances in storage technologies is data deduplication. This backup technique eliminates redundant blocks of data from storage, by saving a single copy of identical data and replacing any further instances with pointers back to that one copy.

HP recently announced several new disk-based storage systems specifically designed to meet the needs of different business environments. For midmarket customers, HP is radically changing the economics of data protection in smaller business environments and remote offices 45% less cost than comparable products and can retrain up to 50x more backup data on the same amount of raw capacity.

Key benefits of these new HP StorageWorks D2D Backup Systems:

Maximizes disk space via synchronous backup operations that deduplicate data as it is stored to the disk.
Reduces the amount of disk and memory required to deduplicate data by leveraging proprietary technology created by HP Labs, the company’s central research and development arm.
Improves productivity and eliminates the need for training with a fully functional graphical user interface that does not require complex command lines for configuration.
Optimizes investment of primary backup system through compatibility with new versions of software and applications.

Read more about our announcement: www.hp.com/go/deduplication In addition to the deduplication products, we’ve also recently released an exciting new series of products for smaller customers who want the benefits of disk based backup for individual servers. Backup of standalone servers is still a significant challenge for those with only a few servers; you may not be ready to consolidate your backup to a centralized device but you still are looking for a technology that provides pain-free backup and rapid file restore. The HP StorageWorks RDX Removable Disk Backup System delivers an easy to use, affordable and rugged data protection solution for workstations and entry level servers. In fact, you can protect your entire system with the included continuous data protection software. Cost of ownership is reduced with long lasting removable disk cartridges and a forward and backward compatible docking station that does not require a costly upgrade to accommodate future higher capacity cartridges. The RDX Removable Disk Backup System offers fast disk based performance with the ability to store 160 GB or 320 GB of data on a single removable disk cartridge. Read more at www.hp.com/go/rdx

Five ideas to reduce cost of managing and protecting data

CalvinZ — Wed, 02 Jul 2008 15:46:00 GMT

Our own Patrick Eitenbichler recently discussed a list of five things customers should focus on to reduce the costs of managing and protecting data. The full article is in Processor Magazine and you can read it at this URL: http://tinyurl.com/593qrm

Here's a summary of the five points Patrick made:

1. Consolidation. Moving data onto centralized storage systems can help administrators avoid the fragmented capacity that leads to extra maintenance work, low disk utilization, and huge backup headaches.

2. Centralized backup. SMEs should look at disk-to-disk-to-tape backup solutions that initially store data on disk drives and eventually migrate it to tape for long-term data retention. Ensuring successful backups on a nightly basis is "mission-critical," says Eitenbichler.

3. Deduplication. Using de-duplication allows administrators to drastically reduce the amount of data stored on disk-based backup systems at data reduction ratios of 20:1 or even 40:1.

4. Thin provisioning. This technique eliminates wasted capacity by automatically sizing storage capacity needed by application requirements.

5. Data life cycle management. It sounds simple, but keeping an inventory of storage devices onsite, available capacity, and growth trends can allow administrators to delay additional purchases for several months.

HP Announces Deduplication - Part 2

jim hankins — Tue, 24 Jun 2008 15:01:00 GMT

- By Jim Hankins

As I stated in part 1, it's difficult to judge the merits of a disk-based backup or virtual tape products with deduplication capabilities based solely on the deduplication ratio that a particular vendor might be touting for its product. The deduplication ratio a customer will see is dependent on a number of variables unique to that customer's backup environment. What I'd like to examine in part 2 are some other features that a customer should consider when comparing disk-based backup or virtual products from different vendors.

As I also mentioned in part 1, HP is introducing deduplication capabilities into our existing HP StorageWorks Virtual Library Systems as well as introducing two brand new products, the HP StorageWorks D2D2500 Backup System and the HP StorageWorks D2D4000 Backup System. These products are intended to meet different customer needs depending on the size of the customer's backup environment. Therefore, it's important to examine the features of these two different products versus other competitive products based on these different customer requirements.

For medium to large scale enterprise backup environments, the key features of our VLS9000 product is its grid-like scalability and appliance-based management.

Grid-like scalability simply means customers can extend the performance and capacity of the VLS9000 by adding additional nodes (one node standard, expandable to eight) or by adding additional capacity per node. Each node of the VLS9000 provides 600 MB/sec and 30 to 80 TBs of capacity. Similar competitive products are offering only single node capability which means it's difficult to grow performance and capacity as the customer's needs grows.

All nodes and components of the VLS9000 can be managed as a single appliance. This allows the customer automatic management of all the hardware and software, including self-configuration, self-monitoring, self-diagnostics, performance load balancing, self-tuning and self-maintenance. Additionally, the entire capacity of the VLS9000 can be presented as a single virtual library. Similar competitive products come pre-configured from the factory, but any expansion or repair tasks can be tedious and complex, and if not done carefully can impact the system's performance and reliability.

For our brand new D2D2500 and D2D4000 Backup Systems, HP focused on delivering three key features for smaller backup environments. Those features are a very affordable price point, ease of deployment and management, and robust integration to physical tape.

One of HP's unique design advantages for our D2D2500/4000 products is our ability to leverage the economies of scale of our volume server and disk drive business. This allows us to build these new products and offer them at price points that are 30 to 40% below competitive product offerings in this customer segment.

The D2D2500/4000 also provides a fully functional graphical user interface (GUI) that allows the product to be easily setup and deployed. Ongoing management and configuration of the products is also performed via the easy to use GUI. Other competitive products provide only limited functionality from their user interface and some tasks require the customer to switch to a command line interface to complete important configuration steps. For some customers this can be a challenging and difficult way to setup and maintain the product.

Our D2D2500/4000 was also designed with tape integration fully in mind. Tape integration allows customers to copy data from the D2D2500/4000 directly to physical tape for offsite disaster recovery or for long-term archives. Tape integration is not available for many competing disk-based backup products, leaving the customer to piece together a disparate and perhaps non-interoperable solution for physical tape backup of their system.

And finally, one of the other exciting capabilities that deduplication enables is low bandwidth replication for disaster recovery. So for customers with multiple data centers or remote sites, deduplication and replication can be a powerful combination for a faster, more reliable way of moving backup data between sites. With deduplication, only the changed backup data needs to be sent between sites allowing customers to use more affordable low bandwidth WAN connections. HP will be offering replication on our VLS, D2D2500 and D2D4000 products before the end of the year.

HP Announces Deduplication - Part 1

jim hankins — Mon, 23 Jun 2008 18:11:00 GMT

-By Jim Hankins

Earlier today, HP announced new deduplication capabilities for customers who are considering deploying disk-based backup or virtual tape as part of their data protection processes. Deduplication is one of the most talked about new technologies in the storage industry today as customers continue to look for innovative ways to protect the ever growing amounts of data in their IT environments.

However, in talking with customers we found that there are very different needs for disk-based backup and deduplication depending on whether the customer wanted to use the technology in a larger scale "data center" type installation or in a smaller scale "office" type installation. Because of these very different needs, HP is offering its customers two different deduplication technologies.

First, HP is making available by license accelerated deduplication for our HP StorageWorks Virtual Library Systems. Our VLS products with accelerated deduplication technology are uniquely scalable for large data centers where both high performance and high capacity are required.

Second, HP is introducing two brand new products, the HP StorageWorks D2D2500 Backup System and the HP StorageWorks D2D4000 Backup Systems with dynamic deduplication technology. Dynamic deduplication was developed by HP specifically for smaller environments where low cost and ease-of-use are key customer needs. Dynamic deduplication is a built-in feature on the D2D2500 and D2D4000, rather than by licensed option.

For more information about the above products please see our announcement page at: www.hp.com/go/deduplication

One of the most frequent questions we heard from customers that we talked to about deduplication prior to our announcement was, "So what kind of deduplication ratios can I expect to get with HP's deduplication technologies?" We've done some internal testing that has shown it's possible to reach at least a 50:1 deduplication ratio, but the ratio that you will achieve in your environment depends on a number of variables. You may hear some other vendors quoting deduplication ratios that are much larger or smaller, but it all depends on a number of factors.

One of those factors is the type of data that the deduplication process is being applied against. Some data types lend themselves to being better candidates for deduplication than others. As an example, data from a PACS (Picture Archiving and Communication System) used in X-rays and other medical imaging will have very little duplicate data so the ratio would usually be quite low. In another example, a database, where there may be many records with empty fields or the same data in the same fields, would typically be a good candidate and could produce very high deduplication ratios.

Other factors to consider are what is your backup policy and the daily change rate of your data? Are you doing daily full and weekly full backups? Or are you doing daily incremental and weekly full backups? Is the daily change rate of your data 1%, 2% or even more? It is important to remember that the less your data changes the more benefit you'll see because over time the deduplication engine will see more and more of the same (duplicate) data during the backups.

Lastly, how you measure deduplication is important to the overall ratio. Are you measuring the deduplication ratio of just your last backup to the previous backup? Are you measuring the ratio over the aggregate of all backups stored? Or is the measurement somewhere in between?

Another word of caution here, some might think that deduplication means that you can buy a smaller disk- based backup system, but be aware that it may take many backups over a long span of time to yield substantial deduplication ratios. Initially, the amount of storage you buy with your disk-based backup or virtual tape product needs to be sized correctly to reflect your existing backup tape rotation strategy and expected data change rate within your environment.

HP believes that the various deduplication technologies in the industry are going to deliver relatively the same ratios, so it's much more important to consider other features such as the scalability, cost and ease-of-use of competing technologies.

In part 2, I will take a look at these other features more closely.