In the early 2000’s I was a big believer in appliance and/or array-based storage virtualization technology. To me, it seemed like the most logical choice to solve some of the most pressing problems such as data migrations, storage optimization and reducing storage networking’s overall management complexity that were confronting the deployment of storage networks in enterprise data centers. Yet here we find ourselves in 2015 and, while appliance and storage array-based storage virtualization still exists, it certainly never became the runaway success that many envisioned at the time. Here are my top 3 reasons as to what went wrong with this technology and why it has yet to fully realize its promise.
- It did not and still does not sufficiently scale to meet enterprise requirements. The big appeal to me of storage virtualization appliances and/or array controllers was that they could aggregate all of an infrastructure’s storage arrays and their capacity into one giant pool of storage which could then be centrally managed. As I came to learn, the problem with this philosophy was that none of the solutions could fully scale to manage all of the storage capacity in one’s data center and certainly not in my data center.
In the early 2000’s I was managing what seemed like an unimaginably large amount of storage capacity (four storage arrays with over 11TBs of storage capacity.) Even in that environment (considered small by any of today’s standards,) the storage virtualization solution I brought in-house only scaled to manage 1TB of capacity. So instead of simplifying my environment and presenting me with only one storage console to manage, it become just another one to manage which increased my complexity rather than reducing it.
- Storage virtualization vendors failed to represent the capabilities of high end arrays. One of the big claims that vendors of storage virtualization appliances made was that you could virtualize high end arrays such as the EMC Symmetrix, the IBM Shark or the HDS Tagmastore. This would eliminate or minimize the need to license their management software, increase application performance and simplify your environment even as you lowered storage costs.
While there was some merit in these claims, they failed to mention that by putting their virtualization appliance in front of these high end arrays you also lost some of the functionality of these high end arrays. For instance, if you had an Oracle Database that communicated directly with the front end controllers on an EMC Symmetrix for data management or performance reasons, that functionality largely went away once a storage virtualization appliance was put in front of it. While some of those issues have been addressed in modern storage virtualization appliances, they have not and will likely never be fully addressed.
- Heterogeneous environments were way more (and remain way more) complex than anyone likes to admit. Another claim made by storage virtualization appliances and/or controllers was the idea that organizations could connect any operating system to any storage array using any network interface and use these storage virtualization appliances to manage and move data between them. This would give organizations a great deal of flexibility to introduce any server and/or storage array onto their storage network and give them more negotiating power to boot.
What I came to learn and now fully understand is that this ideal is simply not an option in enterprise storage networking environments. These environments want tested and proven end-to-end configurations. In this way, if anything went wrong, there was a vendor that they could hold accountable to come in and fix the problem, end-to-end. When presented with this requirement, most storage virtualization vendors were typically unwilling to produce the certifications that illustrated such end-to-end interoperability or provide any guarantees that would resolve the issues that arose. In some cases, even if they did, no one believed they could deliver on them. As such, few enterprises were willing to bet on this type of technology.
Having provided these reasons as to why appliance and/or storage array-based storage virtualization failed to fully gain the wide adoption that many expected them to gain, one should not assume this technology has died off. If anything, storage vendors have learned these lessons with successful deployments of this technology now in place and I even see the adoption of this technology gaining some momentum. In an upcoming blog entry, I will share some thoughts and tips as to why it is seeing a rebirth and ways in which to successfully deploy these solutions in today’s data centers.