Broader Adoption of Inline Deduplication for Primary Storage May be Closer than you Think

While attending SNW last week and receiving a briefing from WhipTail Technologies CTO James Candelaria, Bob Farkaly, Director of Marketing for Exar’s Storage System Products, was never far away. Because of the brevity of my meeting with Candelaria, I never really had a chance to formally talk with Farkaly at the show and connect all of the dots between Exar and WhipTail, other than to assume some component of Exar was an integral component of WhipTail’s Racerunner solid state storage appliance.

It was not until after I got home this week and had a chance to do a little research that I could connect some of the dots. I first found a blog entry on Storage Switzerland‘s site that WhipTail had integrated Hifn’s BitWackr (Exar acquired Hifn in February 2009) so companies could compress and deduplicate data stored to WhipTail’s Racerunner appliance. Because of the speed of the solid states drives, it is now becoming feasible to introduce deduplication into the primary data path without negatively impacting application performance.

However not knowing much about Exar, I followed up with Farkaly and had a chance to talk with him this past Wednesday. Here are some of the insights he had to share:

  • The Hifn card that the WhipTail Racerunner uses to do compression and deduplication on has its origins in virtual tape libraries (VTLs). Many VTLs already in use in enterprise production environments use Hifn’s Express DR 1600 cards for compression.
  • The next generation of cards from Hifn will do compression, deduplication and data encryption.
  • It is this next generation of cards and its deduplication technology that were still in development when Exar acquired Hifn that contributed to Exar’s interest in purchasing Hifn.
  • These cards operate at the block level and are currently available for Windows and Linux servers (Racerunner has a Linux kernel).
  • There is nothing that precludes these cards from eventually being used by UNIX operating systems though no one has come Hifn/Exar asking them to develop drivers and test them for UNIX operating systems
  • Doing inline deduplication for primary storage is very different than doing it for VTLs and backup applications as data redundancy is much less common. Hifn is so far seeing the best results when used in conjunction with NAS interfaces where it is seeing data reductions in the 4:1 range

So what does this conversation with Farkaly tell me? Well, I don’t expect any of the major storage vendors to be announcing inline deduplication for primary storage in the next month. But that said the combination of WhipTail and Hifn prove that it can be successfully done and Farkaly indicated that more vendors were looking at this technology. While he could not give any indication of who was looking at it or if there were any time lines associated with it, the fact that Exar essentially delivers a turnkey way to implement compression and deduplication for primary storage makes it much more likely that you will more storage arrays coming to market sooner rather than later with this functionality.

Have a good weekend!

image_pdfimage_print
Jerome M. Wendt

About Jerome M. Wendt

President & Founder of DCIG, LLC Jerome Wendt is the President and Founder of DCIG, LLC., an independent storage analyst and consulting firm. Mr. Wendt founded the company in November 2007.

3 Comments

  • Anonymous says:

    I don’t understand the appeal of hardware assisted data deduplication solutions using something like Hifn’s Bitwackr. The deduplication performance bottleneck is mostly in disk IO. Having special hardware to compute MD5/SHA1 doesn’t help since the disk IO still dominates the performance load.
    The WhipTail solution illustrates the problem. It’s using SSD disk since traditional disks are not fast enough. That means a hardware-dedup + traditional disks are too slow, or the hardware-dedup doesn’t help.
    CPU is so fast and with multi-core these days that all the computation can be done by CPU. Also Hifn is not be able to compete with Intel in ramming up the CPU performance, i.e. commodity CPU works just fine.

  • Jerome M. Wendt says:

    I have since spoken to WhipTail and Exar (who purchased Hifn)about the question I raised. I have some more information to share about this topic which I hope to cover in a blog I am writing this Friday (2/12).

  • While you are rummaging through the WhipTail deliverables, would you have time to look at similar items from GreenBytes?
    https://www.getgreenbytes.com/products
    The new, little 1000 looks especially enticing for SME/SMSB environments. The entry point is sure affordable for something that has integrated SSD already.

Leave a Reply

Bitnami