Designed for Failure: The Future of Storage Systems

As DCIG does ongoing product research in the Electronically Stored Information space, we have run across multiple examples of storage systems that are “designed for failure”–and that is a good thing. “Designed for failure” has no connection with the infamous “planned obsolescence” of the American auto industry in the 1960’s and 1970’s. Instead, “designed for failure” means that a system is architected in such a way that the system continues to function even if multiple nodes fail simultaneously.
The “designed for failure” philosophy is a key foundation for Internet-scale services. Indeed, the Internet Protocol itself is built on this philosophy. We see this philosophy in grid-style architectures, and more recently in object storage systems. The need for the “designed for failure” philosophy in large-scale systems is articulated by none other than James Hamilton, VP and Distinguished Engineer of Amazon Web Services, who observes, “Once the service has scaled beyond 10,000 servers and 50,000 disks, failures will occur multiple times a day.”
But a business doesn’t need to operate at Internet scale to suffer from the traditional “recover from failure” approach in its storage systems. According to Stephen Foskett of Gestalt IT, “With Seagate’s recent introduction of a 3 TB hard disk drive, we have reached the point where every typical 4- or 5-disk RAID 5 set is likely to encounter an unrecoverable read error in its lifetime.”
Unfortunately, most enterprise storage systems–even those that support RAID6–still reflect a “recover from failure” approach instead of a “designed for failure” philosophy. When it comes to storage systems, the key technologies for implementing “designed for failure” are erasure codes and information dispersal algorithms. Some storage vendors that have embraced the “designed for failure” approach include Amplidata, Cleversafe, Permabit, Gridstore and even Windows Azure Storage. 
Erasure Codes?
As explained by David Floyer, Wikibon Co-founder and CTO, “An erasure code provides redundancy by breaking objects up into smaller fragments and storing the fragments in different places. The key is that you can recover the data from any combination of a smaller number of those fragments.”
Although many people have never heard of erasure codes, they have been used for decades in data communication and data storage including CDs, DVDs, Digital Audio Tape (DAT), xDSL connections, and even communications protocols used by the Voyager space probes. For many enterprise storage arrays, erasure codes first appeared as part of RAID6. Adoption was slow because erasure coding is relatively CPU-intensive.
Traditional RAID Reaching Its Limits
Multi-terabyte hard drives and the ongoing increase in overall data storage requirements are stretching RAID to its limits. It can take many hours to rebuild a RAID array after a drive failure, with performance compromised throughout the rebuild process. 
Worse, parity-based RAID arrays with many terabytes of storage run an increased risk of encountering an unrecoverable read error that will cause the rebuild process to fail entirely–which means restoring the array from backup. As noted at the beginning of this article, the possibility of encountering unrecoverable read errors in a typical RAID array is real and growing. 
Achieving acceptable levels of data protection on top of parity-based RAID generally means adding replication to the mix. This generally means multiplying the amount of total storage space required by a factor of 2x or 3x.
Erasure Codes and Information Dispersal Algorithms Replace RAID
Erasure codes and information dispersal algorithms address critical shortcomings of RAID in today’s increasingly large storage subsystems. Among their many benefits, erasure codes:
  • Give the business more flexibility in determining how much protection is enough protection. One erasure code setup is EC 10/16 that breaks data into 16 chunks. Since only 10 chunks are needed to recover the data in this scenario, the storage system can suffer a failure of 6 of the 16 nodes and still function properly.
  • Use less space to provide a given level of data protection. In the EC 10/16 example above, requiring just 1.6x the original storage space.
  • Provide an opportunity for geographic dispersal of information more efficiently than current replication practices.
Erasure Code Trade-Offs
Erasure coding is more compute intensive than parity-based RAID data protection schemes. However, storage systems built with current multi-core processors can perform the necessary calculations without degrading storage system performance. Depending on how information dispersal is implemented it may subject information retrieval to WAN-based delays.
“Designed for Failure” Finding Its Way into DCIG Buyer’s Guides
The “designed for failure” philosophy is a good fit for a growing number of electronically stored information scenarios. Erasure codes and information dispersal algorithms will–and should–play a significant role in enterprise storage systems. Therefore, the forthcoming DCIG 2013 Enterprise Midrange Array Buyer’s Guide–and future DCIG buyer’s guides–will include erasure codes/information dispersal as a criterion in our evaluations. 
There is a good body of technical literature regarding erasure codes and information dispersal. Storage professionals will do well to acquaint themselves with the fundamentals of erasure coding and its potential applications across the data storage ecosystem, and especially how it might apply to their own business data storage and information archiving requirements. Those running large RAID 5 arrays should examine their options for implementing a more robust approach to data protection.
Ken Clipperton

About Ken Clipperton

Ken Clipperton is the Lead Analyst for Storage at DCIG, a group of analysts with IT industry expertise who provide informed, insightful, third party analysis and commentary on IT hardware, software and services. Within the data center, DCIG has a special focus on the enterprise data storage and electronically stored information (ESI) industries.

Leave a Reply