NVMe has been one of the hottest topics in the storage industry for a while now. Rightfully so, it is making an impact in the storage industry and moving computing forward. For storage vendors and customers alike, it is important to be ready as certain aspects of NVMe become standard. There are two key aspects of NVMe, as it promises to solve backend bottlenecks to flash devices and improve the SAN performance to servers over fabrics. In this blog, I will share my perspective on NVMe flash modules and how, here at Violin, we solve the flash device performance aspect.
Many of the all-flash storage products available today rely solely on SATA or SAS Solid State Disks. These devices provide better performance than rotating media but, greatly reduce the performance of the raw flash. They are connected via SAS controllers with low powered CPUs and a small amount of SAS channels for IO traffic (lanes that are typically 8 X 12Gb/s). These cards are then attached to a SAS fabric switch which limits the total amount of IO from the drives. The IO, then, finally reaches the SSDs that have low power controllers, further increasing the latency of the IO. Through the entire stack you will see 2-3 Milliseconds of latency. That is roughly 20X higher than the latency of Violin Systems Extreme Performance Flash Storage Platform.
Storage vendors that are leveraging SSDs, are rapidly shifting to NVMe flash modules in an attempt to reduce latency and increase performance. The challenge with moving the IO to PCIe bus is you now have several interfaces competing for a small number of lanes. If you have a modern Intel dual socket platform, you have 80 total lanes (40 per CPU). If each controller has 2 IO cards you now have 64 for PCIe NVMe modules. Each of these modules connect via PCIe Gen 3x4, allowing a total of 16 modules to attach. With a small number of flash modules, it reduces how wide the IO can be dispersed. This is why we see some AFA platforms achieving only a 20% improvement on performance some vendors have rolled out new platforms and only improved performance by 20%.
At Violin Systems, we have a different approach with purpose built all-flash arrays. Our unique approach is built from the ground up to provide unparalleled performance. Within the Extreme Performance Flash Storage Platforms, we leverage a flash fabric architecture that is comprised of 64 Violin Intelligent Memory Modules (VIMM). These modules are on a 500MhZ bus that have 3 ports of IO and are assembled into a tall maple architecture. This tall maple architecture allows any 1 of 4 controllers to communicate with any VIMM. Each VIMM has a FPGA on it, giving the Violin Storage Platform 80 CPU/FPGAs per array, providing a significantly larger number of channels for IO. This allows a great deal of resiliency and ability to spread IO extremely wide. The Violin Flash Fabric Architecture also allows IO, garbage collection, and trimming to be scheduled to provide constant low latency read performance. This is how the Violin Storage Platforms can provide 2.2 Million IOPs or greater at sub millisecond latency, more than most NVMe products on the market.
NVMe will have a big impact in how storage protocols are reshaped, however, as a protocol it lacks maturity and provisions for enterprise arrays. Until a standard protocol evolves to allow for IO to be spread wide and have background tasks scheduled, Violin Systems will continue to be the performance leader with the innovation of the Flash Fabric Architecture. When Violin comes to market with products that support NVMe, the performance, reliability, and enterprise features in Concerto OS will be designed to take full advantage of the gains from NVMe. That’s why Violin does not need to rush to market with NVMe labeled products.
Tommy Scherer, Solutions Architect, Violin Systems