The Enterpise TLC Storage Era Begins: Rounding Up 13 SSDs With Samsung, Intel, and Memblaze

2018 was a busy year for consumer SSDs. With all the NAND flash manufacturers now shipping high-quality 3D NAND in volume, we've seen more competition than ever, and huge price drops. NVMe is starting to go mainstream and Samsung is no longer sitting atop that market segment unchallenged. But not all of the interesting SSD advancements have been in the consumer realm. We've reported on new datacenter SSD form factors, the introduction of QLC NAND and enterprise SSDs with staggering capacities, but we haven't been publishing much in the way of traditional product reviews for enterprise/datacenter SSDs.

Tackling the Enterprise SSD Market

Today we're looking at several recent models that cover the wide range of enterprise SSDs, from entry-level SATA drives based on the same hardware as mainstream consumer SSDs, up to giants that deliver 1M IOPS over a PCIe x8 connection while pulling more than 20W.

No two models in this review are direct competitors against each other, but together these drives represent almost every important product segment for enterprise/datacenter SSDs. The selection of drives reflects recent trends: the fact that MLC NAND is basically gone from the market, and now replaced by mature 3D TLC NAND. The state of 3D TLC is that it now offers plenty of endurance and performance when aggregated into the multi-TB drives that are most cost-effective.

For the few niches that still require the highest endurance and lowest latency that money can buy, alternative memories like Intel 3D XPoint and Samsung Z-NAND are filling the gap left by the disappearance of traditional MLC and SLC NAND flash. SAS SSDs still exist, but their relevance is waning now that NVMe SSDs have matured to the point of offering not just higher performance, but all the same reliability and management features that are standard in the SAS ecosystem.

In the consumer SSD market, there's not a big price between the cheapest NVMe SSDs and the fastest (NAND-based) NVMe SSDs, so it's difficult to divide the consumer NVMe market into multiple product segments that each contain reasonable products. The enterprise SSD market doesn't have this problem. There's a clear distinction between the lower-power NVMe drives that usually share a controller platform with a consumer product, and the high-end drives that use enterprise-only 16 to 32-channel controllers.

Probing Samsung, Intel, and Memblaze

The table below lists the drives that are the subject of this review. Three of the four new models Samsung announced during Q3 2018 are included: two SATA and one NVME, and a fourth (983 ZET Z-SSD) will be reviewed later. The Intel DC P4510 mid-range NVMe SSD and Optane DC P4800X have already been reviewed on AnandTech, but are being revisited now that we have more relevant drives to compare against. The Memblaze PBlaze5 is the most powerful flash-based SSD we have tested, and is a good representative of the top-tier enterprise SSDs that don't get sampled for review very often.

Not shown in the table but included in the review is the older Samsung PM863 SATA drive, which helps illustrate how things have progressed since the days when Samsung had a monopoly on 3D NAND. We do have several other enterprise SSDs on hand, but they're all either in service as boot drives for various testbeds, or are old planar NAND drives, so they're not included in this review.

To determine where an enterprise SSD fits into the competitive landscape, the first specifications to look at are the form factor, interface, and write endurance rating.

• Interface: As with consumer SSDs, in the enterprise storage market SATA drives are generally cheaper and much slower than NVMe drives. But in a server, the performance differences between SATA and NVMe can have a much larger impact on overall system performance than with an interactive desktop workload. SATA product lines usually top out at 4TB or 8TB; beyond that, the SATA interface bottleneck leads to unacceptably low performance-per-TB ratios. NVMe product lines usually start around 1TB and go up from there, because smaller capacities would not be able to make good use of PCIe bandwidth.
• Form Factor: Smaller form factors like M.2 and 7mm thick 2.5" drives cannot draw as much power as a PCIe add-in card, and this often leads to the physically smaller drives offering lower performance. The largest 16 and 32-channel SSD controllers cannot even fit on a 22mm wide M.2 card. This is why newer standards are coming into the market, like Samsung's NF1 and Intel's Ruler.
• Write Endurance: Endurance ratings indicate what kind of workload a drive is intended for. Drives with higher endurance ratings also have higher write performance, especially random write performance. Ratings of less than one drive write per day usually indicate a drive that is intended for a relatively read-heavy workload. As little as 0.1 or 0.2 DWPD can be plenty for a use case like a streaming video CDN node, which experiences little write volume and what writes it does have to contend with are mostly sequential. Drives with ratings of 5 DWPD or more are becoming rare, except for specialty models like Intel's Optane SSDs and Samsung's Z-SSDs.

Based on these guidelines, we can expect the Intel SSD DC P4510 to be a small step up from the Samsung 983 DCT, in price and power consumption if not performance. The Memblaze PBlaze5 D900 is another step up with its higher write endurance rating, and the PBlaze5 C900 is clearly the beefiest NAND-based drive in this bunch with its PCIe 3 x8 interface.

Test System

Intel provided our enterprise SSD test system, one of their 2U servers based on the Xeon Scalable platform (codenamed Purley). The system includes two Xeon Gold 6154 18-core Skylake-SP processors, and 16GB DDR4-2666 DIMMs on all twelve memory channels for a total of 192GB of DRAM. Each of the two processors provides 48 PCI Express lanes plus a four-lane DMI link. The allocation of these lanes is complicated. Most of the PCIe lanes from CPU1 are dedicated to specific purposes: the x4 DMI plus another x16 link go to the C624 chipset, and there's an x8 link to a connector for an optional SAS controller. This leaves CPU2 providing the PCIe lanes for most of the expansion slots, including most of the U.2 ports.

The enterprise SSD test system and most of our consumer SSD test equipment are housed in a StarTech RK2236BKF 22U fully-enclosed rack cabinet. During testing for this review, the front door on this rack was generally left open to allow better airflow, since the rack doesn't include exhaust fans of its own. The rack is currently installed in an unheated attic and it's the middle of winter, so this setup provided a reasonable approximation of a well-cooled datacenter.

The test system is running a Linux kernel from the most recent long-term support branch. This brings in the latest Meltdown/Spectre mitigations, though strategies for dealing with Spectre-style attacks are still evolving. The benchmarks in this review are all synthetic benchmarks, with most of the IO workloads generated using FIO. Server workloads are too widely varied for it to be practical to implement a comprehensive suite of application-level benchmarks, so we instead try to analyze performance on a broad variety of IO patterns.

Enterprise SSDs are specified for steady-state performance and don't include features like SLC caching, so the duration of benchmark runs doesn't have much effect on the score, so long as the drive was thoroughly preconditioned. Except where otherwise specified, for our tests that include random writes, the drives were prepared with at least two full drive writes of 4kB random writes. For all the other tests, the drives were prepared with at least two full sequential write passes.

Our drive power measurements are conducted with a Quarch XLC Programmable Power Module. This device supplies power to drives and logs both current and voltage simultaneously. With a 250kHz sample rate and precision down to a few mV and mA, it provides a very high resolution view into drive power consumption. For most of our automated benchmarks, we are only interested in averages over time spans on the order of at least a minute, so we configure the power module to average together its measurements and only provide about eight samples per second, but internally it is still measuring at 4µs intervals so it doesn't miss out on short-term power spikes.

Quarch provides a variety of power injection adapters for use with different drive form factors and connectors. For this review, we used two different adapters: one for U.2/SAS/SATA drives that delivers power over a thin ribbon cable that can fit between drives in a hot-swap bay, and a PCIe add-in card riser. For the Samsung 983 DCT M.2, we used a PCIe to M.2 adapter board that includes a 12V to 3.3V converter, and our measurements have been adjusted to account for the overhead of that converter. This is necessary because the 983 DCT M.2 is rated to draw a peak of 3.8A, and it is very difficult to deliver that much current at only 3.3V without serious voltage drop along the cable. Newer Quarch power modules include upgrades to improve high-current delivery and add automatic compensation for voltage drop along the cable, but we haven't had the chance to upgrade yet.