Memory

The memory setup on my system is a bit different than most. The memory on my sever is split between 64GB of standard error correcting registered memory, typical for servers, and 256GB of Optane flash memory in two DIMM slots. Optane is a pretty cool technology - unlike current consumer SSDs, Optane works really well at low queue depths, the number of outstanding operations. Normal SSDs that use even single level flash have to queue many operations in parallel to achieve good performance, Optane is able to get great performance (fast and low latency) with just one operation.

What this enabled beyond really awesome SSDs is the Datacenter persistent memory module, or DCPMM. Intel developed the DCPMM which puts Optane flash modules and a controller on the DDR4 bus as either bulk memory for the computer, or in several direct access modes. Unfortunately Optane seems to be a consumer failure though, as Intel has canceled all future consumer targeted Optane products. It was a very inexpensive way to transparently get a lot of performance from ZFS by having lots of RAM, without spending a lot of money.

FusionIO SSD

All the other hardware is pretty standard and boring, though the bulk storage SSD is somewhat interesting - it's an old FusionIO SSD that uses a FPGA and a bunch of flash chips to implement a very fast SSD, even considering available SSDs now. For example, it gets up to 345K/385K IOPS R/W. It doesn't slow down much as it fills past 90%, unlike modern SSDs, and uses MLC, not TLC, which gives it much more well-rounded performance in tail-end scenarios. Note these are note NVME and getting them working in Ubuntu 20.04 requires using this driver.