'C' Fusion-io Run; Interview with Fusion-io Sr Director of Product Mgt Brent Compton Part IV
Fusion-io has taken some very impressive steps to lure developers to adopt their ioMemory platform with some impressive features in the SDK kit. As we have revealed in prior blog entries, its new APIs for block IO and key value stores grant developers access to flash in ways that its competitors simply do not. In this final blog entry in my interview series with Brent Compton, Senior Director of Product Management at Fusion-io, he describes new memory-access semantics exposed in the SDK.
Ben: You've described a pretty impressive feature set. What languages will the SDK be released in?
Brent: Initially in C.
Ben: Hence the comparisons with mmap of course! In the C language, I would create a pool of memory using the mmap function. Now I can do the same with one of your API calls and store things within that pool of memory. Do I have to pre-allocate that or can I dynamically allocate memory into that?
Brent: It varies by API family. For instance, with the directKey-Value Store API, a key-value store is created with the kv_init function call and ioMemory is dynamically allocated through kev_put and kv_batch_put calls.Inside of the memory-access semantics API family, memory allocation is much more similar to traditional malloc models.
With ExtendedMemory, for example, a process extends its virtual memory space through a malloc variation, giving the process a sense that DRAM has been transparently extended onto much larger and lower cost ioMemory. The application process is then able to access its entire dataset as if it were completely in memory, with transparent demand-paging from ioMemory occurring behind the scenes to ensure data is in DRAM when needed.
With the Auto Commit Memory API, a process expands its virtual memory space by allocating a persistent region of memory, known as Auto-Commit Memory. Like with standard memory, the process stores data using traditional memory interaction semantics, such as pointer dereferencing or memcpy. However, unlike standard memory, data stored in Auto-Commit Memory is persistent and retrievable after a system failure.
Ben: What applications do you see in the first wave of early adopters? You mentioned NoSQL as one example. What other high level types of applications can you see getting early adoption?
Brent: There are four groups of adopters that are very keen to use these APIs. Their keenness is reflected by their enthusiasm to work with these APIs during our early access program.
There are four categories of industry verticals with which we are working.
The first category is this explosion of unstructured databases or NoSQL key value stores. We are working with several companies in this space.
The second space is traditional relational databases. I mentioned the two illustrations earlier. One illustration is their interest in speeding up transaction logging, and the other is their interest in high-performance mechanisms for achieving atomicity of complex writes.
As an example of the second case, our work with the MySQL InnoDB engine eliminates double buffered writes. That is a big deal. They're expensive writes because of the added latency and media wear of writing everything twice. Especially for a medium like flash -- that is not something you want to do.
Third is the social networking community. They are clearly an innovative bunch and they have development staffs which create differentiation natively. They are very enthusiastic about native access to ioMemory.
The fourth vertical is the investment banking community. Those folks are just always looking just to shave latency in high-speed transactional systems.
Ben: You mentioned working with some partners?
Brent: We have already opened up to some early access partners. Like with a lot of new innovations, early access partners are hand-selected based on their ability to significantly improve their solution through these APIs, and for their ability to help us tune these innovations for the general market.
As you would imagine, we are growing the early access program thoughtfully. For instance we have a very compelling set of early access partners already working our direct key value store API.
One of them was on stage with us for the DEMO Spring announcement in April. Citrusleaf demonstrated 400,000 transactions/second with sub-millisecond latency on a single $25k server while running through our native directKey-Value Store APIs.
Another, Couchbase, issued a joint press release with us a week later describing our joint work to make their CouchDB Server run native on flash.
Ben: You would still consider the SDK beta at this point, correct?
Brent: Yes. The term we are using to describe it is "early access." Mostly to reflect our engagement with our partners' own development cycles. Because this is a software development kit, it inherently means our partners will be modifying their code. We are giving them early access so that they can get it into their code revision cycles.
Obviously young startup companies can move overnight, or over a weekend, when modifying their code to run native on ioMemory. In addition these aggressive young companies, we are also working with some big, commonly established brand name companies that have longer development cycles with much larger customer bases.
Our purpose right now is just to get the early-access SDK kit into their hands and get it into their development cycles. We are then looking at supporting the production-ready SDK simultaneously with them releasing production grade versions of their products running natively on ioMemory.
Ben: Thank you Brent. This has given me a chance to let my mind to run wild a little.
Brent: You're welcome! Thank you for opportunity.
In Part I of our interview series, we discussed how the Fusion-io SDK kit will help to unleash the next gen properties of flash.
In Part 2 of this interview series Brent will continue to discuss the primitives that developers will have access to including atomic multi-block writes. We also discuss how familiar the API will be to developers.
In Part 3 in this interview series we will discuss the "DirectFS" API, a native POSIX compliant direct file system layer and discuss the more technical aspects of how the SDK works.
Ben: You've described a pretty impressive feature set. What languages will the SDK be released in?
Brent: Initially in C.
Ben: Hence the comparisons with mmap of course! In the C language, I would create a pool of memory using the mmap function. Now I can do the same with one of your API calls and store things within that pool of memory. Do I have to pre-allocate that or can I dynamically allocate memory into that?
Brent: It varies by API family. For instance, with the directKey-Value Store API, a key-value store is created with the kv_init function call and ioMemory is dynamically allocated through kev_put and kv_batch_put calls.Inside of the memory-access semantics API family, memory allocation is much more similar to traditional malloc models.
With ExtendedMemory, for example, a process extends its virtual memory space through a malloc variation, giving the process a sense that DRAM has been transparently extended onto much larger and lower cost ioMemory. The application process is then able to access its entire dataset as if it were completely in memory, with transparent demand-paging from ioMemory occurring behind the scenes to ensure data is in DRAM when needed.
With the Auto Commit Memory API, a process expands its virtual memory space by allocating a persistent region of memory, known as Auto-Commit Memory. Like with standard memory, the process stores data using traditional memory interaction semantics, such as pointer dereferencing or memcpy. However, unlike standard memory, data stored in Auto-Commit Memory is persistent and retrievable after a system failure.
Ben: What applications do you see in the first wave of early adopters? You mentioned NoSQL as one example. What other high level types of applications can you see getting early adoption?
Brent: There are four groups of adopters that are very keen to use these APIs. Their keenness is reflected by their enthusiasm to work with these APIs during our early access program.
There are four categories of industry verticals with which we are working.
The first category is this explosion of unstructured databases or NoSQL key value stores. We are working with several companies in this space.
The second space is traditional relational databases. I mentioned the two illustrations earlier. One illustration is their interest in speeding up transaction logging, and the other is their interest in high-performance mechanisms for achieving atomicity of complex writes.
As an example of the second case, our work with the MySQL InnoDB engine eliminates double buffered writes. That is a big deal. They're expensive writes because of the added latency and media wear of writing everything twice. Especially for a medium like flash -- that is not something you want to do.
Third is the social networking community. They are clearly an innovative bunch and they have development staffs which create differentiation natively. They are very enthusiastic about native access to ioMemory.
The fourth vertical is the investment banking community. Those folks are just always looking just to shave latency in high-speed transactional systems.
Ben: You mentioned working with some partners?
Brent: We have already opened up to some early access partners. Like with a lot of new innovations, early access partners are hand-selected based on their ability to significantly improve their solution through these APIs, and for their ability to help us tune these innovations for the general market.
As you would imagine, we are growing the early access program thoughtfully. For instance we have a very compelling set of early access partners already working our direct key value store API.
One of them was on stage with us for the DEMO Spring announcement in April. Citrusleaf demonstrated 400,000 transactions/second with sub-millisecond latency on a single $25k server while running through our native directKey-Value Store APIs.
Another, Couchbase, issued a joint press release with us a week later describing our joint work to make their CouchDB Server run native on flash.
Ben: You would still consider the SDK beta at this point, correct?
Brent: Yes. The term we are using to describe it is "early access." Mostly to reflect our engagement with our partners' own development cycles. Because this is a software development kit, it inherently means our partners will be modifying their code. We are giving them early access so that they can get it into their code revision cycles.
Obviously young startup companies can move overnight, or over a weekend, when modifying their code to run native on ioMemory. In addition these aggressive young companies, we are also working with some big, commonly established brand name companies that have longer development cycles with much larger customer bases.
Our purpose right now is just to get the early-access SDK kit into their hands and get it into their development cycles. We are then looking at supporting the production-ready SDK simultaneously with them releasing production grade versions of their products running natively on ioMemory.
Ben: Thank you Brent. This has given me a chance to let my mind to run wild a little.
Brent: You're welcome! Thank you for opportunity.
In Part I of our interview series, we discussed how the Fusion-io SDK kit will help to unleash the next gen properties of flash.
In Part 2 of this interview series Brent will continue to discuss the primitives that developers will have access to including atomic multi-block writes. We also discuss how familiar the API will be to developers.
In Part 3 in this interview series we will discuss the "DirectFS" API, a native POSIX compliant direct file system layer and discuss the more technical aspects of how the SDK works.
Buyers Guides
Newsletter Signup
Leave a comment