Building a 20TB ZFS file server (2011)

Friends and colleagues usually laugh when I tell them I have a 20TB file server at home. They are even more surprised when I tell them it's 3 years old. Anyway, for those of you that are curious, you will find below informations and pictures of the beast1. I'm using the amazing ZFS filesystem which I strongly recommends to everybody needing a reliable non-distributed filesystems.

ZFS

ZFS was a revolutionary file system when released in 2005 and it still kicks ass today. Until BTRFS matures, ZFS is the king of the hill when it comes to non-distributed filesystems. The hype around it was pretty big at the time and a lot was written about it. It was even argued that a physical system with the storage capacity that matches the 128-bit address range of ZFS would "literally evaporate all the oceans on earth" and would have the mass of a 1000 moons. No, I'm not making this up.

Hype aside, the reality is that ZFS has many great features. Using it made me realize how primitive Windows, Mac OS and (until recently) most Linuxes default filesystems are. Even though it is much more sophisticated, ZFS is very easy to manage thanks to its simple command line tool2 and is therefore a very pratical choice. Adding, removing, replacing disks, checking the health of the system or any other operation is always one simple command away.

ZFS for the masses

You don't want to use the command line on a FreeBSD or Solaris system just to use ZFS? Rest assured, FreeNAS is a FreeBSD-based appliance that covers your need. It provides an easy to use web interface to manage all aspects of your box.

List of components

ZFS is best supported on FreeBSD and Solaris and hardware support is a bit more limited on those operating systems. I basically made sure that the network card and disk controllers would be supported out of the box. This information is a Google search away: there are many many documented ZFS builds nowadays. Anyway, here's what I went with at the time:

  • Motherboard: Supermicro X8SIL-F
  • CPU: Intel Xeon 3440
  • Memory: 4 x 2GB DDR3 1333 ECC
  • Power supply: Corsair 750W
  • Hard disk drives: 10 x Samsung HD204UI (2TB)
  • Hard disk controller: Intel SASUC8I (LSI chipset)
  • Case: Norco RPC-450B
Arrays of hard drives
10 x Samsung HD204UI (2TB) bougth at 0.05$/GB
SATA controller
Intel RAID Controller SASUC8I (LSI chipset)
Bunch of SATA connectors
A bouquet of SATA connectors
There are fancy cases with 10+ hot-swappable bays for easy maintenance (simliar to the ones I have on the right). However, these are much more expensive and I have seen too many complaints about the cheap popular options like the Norco-4220 to justify the money (~400$). I don't mind opening the case to change a disk once in a while.
Mainboard with Intel CPU
CPU and memory

Price of hard drives

I bought my array of 2TB Samsung drives before the Thailand floods at roughly 100$ apiece. Unfortunately, the price per capacity (~0.05$/GB) is more or less the same right now but it should get much cheaper in the coming years provided no natural catastrophes strike hard drives factories.

Data provided by Backblaze. See their blog post for more details.

Conclusion

I'm pretty glad with the build. It already survived 2 hard disk drive failures without losing data and I don't think I will run out of space in the foreseeable future :)

  1. The beast's hostname is "neumann" in reference to John von Neumann, a badass scientist that possessed an eidetic memory. ↩

  2. It is by far the simplest RAID I have ever setup. For example, the following creates a new pool with two disks (da0 and da1) in mirror: "zpool create tank /dev/da0 /dev/da1". ↩

  • hardware
  • zfs
  • storage