WavebreakMediaMicro - Fotolia
While we'll see some gains in traditional disk from current research, memory-based storage will only increase in popularity, density and reliability.
At a conference in the early 2010s, I was asked by a storage vendor's roaming video team what I thought about the future of hard disks. They seemed legitimately shocked when I said hard disks have no future: "My future data center storage will just be a giant blob of flash."
To be fair, it's been pretty obvious. Storage was the one thing holding computers back. A modern Intel E5-2699v3 has 18 cores running at 2.3 GHz. That's 41.4 billion clock cycles per second. But what happens when that CPU runs out of things to work on, and needs to go to disk for data, or to flush something to disk for safety? Accessing RAM is relatively fast, at 100 nanoseconds, but a traditional disk read takes 20 milliseconds. That's several orders of magnitude longer than RAM -- 828 million clock cycles wasted while the CPU sits and waits for disk.
Solid-state drives (SSDs) cut that to 1 millisecond, meaning you only waste 41 million clock cycles. The performance difference is measurable to applications and users, and there's movement to cut that even further. RAID controllers, queuing algorithms and caching mechanisms are all nearing end of life or being entirely rethought due to the data center SSD invasion.
Given the size of drives, RAID algorithms can no longer protect against unrecoverable read errors. SSD is also so fast that powerful CPUs are needed to do the math behind RAID and avoid a bottleneck. That drives prices and complexity up. That's why most storage manufacturers don't do RAID anymore, opting to store multiple copies of each block and use new techniques like erasure coding.
Sayonara, magnetic disks
If we're going to throw RAID away, do we need the SCSI protocol? Hardware manufacturers don't think so, since they've offered direct PCIe-based flash drives for several years. Dell, for instance, allows four 1.6 TB PCIe hot-pluggable flash modules in a 2U server. You could also install 5+ TB PCIe flash cards. And speaking of PCIe, RAID and SCSI aren't going to be protocols of choice for technology like Intel's optical PCIe, which extends the PCI Express bus across the data center, at 100 Gbps or more, to "blob of flash" storage arrays.
Storage density was one of the last ways that rotating magnetic disks still held on, with 6, 8, and 10 TB drives recently entering the market. Then SanDisk introduced an impressive 200 GB microSD card, decimating magnetic disk's density leadership. The microSD standard is 15 mm by 11 mm by 1 mm, about the size of a fingernail. A traditional 3.5 inch hard disk is 2,200 times larger than a microSD card, by volume, so we could get 450 TB of SSD in a disk bay. Imagine a 2U data center server with 5.4 petabytes of flash installed internally. Petabytes!
No, I didn't account for overhead in my calculations, and, yes, the SanDisk microSD card is consumer grade. But enterprise grade isn't far behind. Automated tiering will help speed adoption of data center SSD to allow the expensive write-capable flash to absorb most of the writes, and the cheap read-optimized flash to do the bulk storage. Deduplication and compression on all-flash arrays will drive up utilization. We're headed for a typical 4:1 deduplication ratio for 5 petabytes in 2U boxes.
About the author:
Bob Plankers is a virtualization and cloud architect at a major Midwestern university.