TrueNAS Build and Installation Issues
Why Custom NAS Build?
One of my biggest regrets is the purchase of a 2-bay NAS (Synology DS220+). Since any redundancy requires a minimum number of 2 drives, with this NAS there's no way to extend capacity other than replacing the existing drives with larger-capacity ones. I started with 2 WD Red Pro 4TB HDDs but considered them to be too noisy at night and replaced them with 2 WD Red 2TB SSDs. Now I'm at a new place and I can put the NAS in a place where acoustics doesn't matter much but because both bays have been occupied I have to choose between the 2 HDDs or 2 SSDs.
The obvious option is to upgrade to a NAS with more drive bays. I'll skip 4-bay NAS's because all bays will be occupied once I put in all 4 drives and I'll have no room for expansion again. If I stick to Synology the choices are:
| Model | #Bays | Price |
|---|---|---|
| DS1621+ | 6 | $899.99 |
| DS1621xs+ | 6 | $1599.99 |
| DS1821+ | 8 | $999.99 |
Let's say they are... not cheap. I'm pretty sure many people will pay the price happily given how feature-rich DSM is (plus the warranty). However for me it's hard to justify paying such prices given that 90% of my current NAS use is simply storing and accessing files over the network.
Luckily there's a second and cheaper option: building your own NAS. An off-the-shelf NAS is just a computer in custom form factor running a custom BSD/Linux operating system. That means you can basically build one by building a PC with standard components and installing a NAS OS like TrueNAS and Unraid on it. The great thing about custom NAS is that you can typically get much more performance while spending much less at the same price point as off-the-shelf NAS's are often equipped with very low-power CPUs and just barely enough amount of RAM. This effect is further magnified if you can repurpose spare hardware or buy some used ones.
The Build
In my case, I have a few spare hardware, notably a Z170 motherboard, an SF600 PSU and the aforementioned 2 4TB hard drives. To make a PC, at minimum I'll need a CPU, a CPU cooler, some RAM, a boot drive and a case. And here's the part list:
| Component | Part | Price | Source |
|---|---|---|---|
| Motherboard | Gigabyte Z170X-UD5 | $0 | Spare |
| CPU | i3-6100T | $44 | eBay |
| Cooler | Arctic Alpine 12 CO | $16 | Amazon |
| RAM | 8GB x2 DDR4 3000MHz | $0 | Spare |
| Boot Drive | SK Hynix P31 500GB | $68 | Amazon |
| PSU | Corsair SF600 | $0 | Spare |
Since the motherboard only supports 6th and 7th gen Intel CPUs, I opted for a low-TDP 2C4T CPU which should be more than enough (and actually way powerful compared to the Celeron J4025 in my DS220+). The 2 sticks of RAM were pulled from my gaming PC which had 4 sticks. I didn't buy a case yet because I simply couldn't find one with multiple drive bays from eBay / Craigslist / FB Marketplace. Plus I wanted to make sure I'm comfortable with this option before I commit more $$$ to it.
For the OS, my choice is TrueNAS Core 13.0. It's popular among enthusiasts and it's open source but funded by a commercial company at the same time. There are two main flavors: Core and Scale. Core is mature, based on FreeBSD and has limited extra features. Scale is newer, based on Debian, more flexible but currently slightly less performant.
Installation Issues
I thought the installation would be the easiest part: writing a boot drive, disabling secure boot, install TrueNAS and use it. But I actually took quite some time troubleshooting 2 problems.
The first problem is that the installation got stuck with this error message:
CAM status: CCB request completed with an error
Initially I thought I messed up the disk image. I used Rufus to write the image to a 32GB USB drive and made sure DD write mode was used per official documentation. I noticed that the parameters I used (they were kept from last time when I wrote an Ubuntu image) were different from others used online. However the problem persisted after changing the parameters. I also tried different boot options (for some reason BIOS listed 2 for the same drive) but that didn't work either.
The motherboard comes with some USB 2.0 ports and some USB 3.1 ones. I specifically inserted the drive to a USB 3.1 port for better speed. It turns out that was the culprit. After moving the drive to a USB 2.0 port the installation completed without any hiccup.
The second problem is that the system refused to boot even though the installation was successful. The screen kept spewing the following message:
Reboot and Select proper Boot device, or Insert Boot Media in selected Boot device and press a key
The BIOS seems to get stuck in a bad state as I couldn't even enter BIOS or boot menu. After inserting the USB boot drive again I was able to enter the boot menu. I noticed that other than the USB drive, there are 2 options: "Disk" and "Disk (UEFI)". After setting "Disk (UEFI)" as the first boot option TrueNAS booted successfully.
My Experiences So Far
After playing with TrueNAS for a few days I would say the experience aligns with my expectations:
- System is relatively intuitive to use and seems reliable.
- I was able to set up a storage pool with 2-disk mirror and enable SMB share over it.
- ZFS's learning curve is pretty steep.
- TrueNAS Documentation is not great and a lot of solutions can only be found in their user forum or Reddit (Ubiquiti users: first time?).
One big surprise to me is that ZFS is actually less flexible than SHR. For one, some capacity will be wasted if you mix drives of different capacities in the same VDEV. For two, ZFS doesn't support dynamic expansion if the VDEV is not mirror. These are actually a big deal since it means you pretty much have to plan in advance how many drives you need and actually buy them at once; and you only get the capacity increase after you replace all drives in the VDEV with bigger drives. To be fair there are reasons behind these design decisions. But for average users SHR is better as you can simply add more drives as you go and maximize the available capacity.
Anyway I'm no average user so I'm happy to stick to the ZFS limitations. My plan is to keep the current mirror setup and get some more 4TB drives and set up a raidz1 VDEV once it hits 50% utilization. With this decision I ordered a Fractal Design Define R5 case with 8+2 bays for $150. I also considered cases of lower prices but either the build quality is subpar or cable management is terrible. I think it's a good investment since the case will stick forever even I upgrade components down the line.
So in total this NAS build cost me around $280, which is much much cheaper than the off-the-shelf offerings. I also get a much more transparent and hackable platform since TrueNAS Core is open source and based on FreeBSD. In the meantime I have to live with the lack of official support and the lack of add-on features like Synology Drive, Video Station, etc. So far I consider this to be a reasonable tradeoff.
Appendix: Intel I211 vs I219
The particular motherboard I used has 2 ethernet ports. The spec sheet simply describes them as "Dual Intel GbE LAN" so I thought they are the same. However, under TrueNAS one shows as "igb0" while the other as "em0". Why do they have different labels?
After some digging it turns out different chips are used!
$ pciconf -lv
igb0@pci0:10:0:0: class=0x020000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x1539 subvendor=0x1458 subdevice=0xe000
vendor = 'Intel Corporation'
device = 'I211 Gigabit Network Connection'
class = network
subclass = ethernet
em0@pci0:0:31:6: class=0x020000 rev=0x31 hdr=0x00 vendor=0x8086 device=0x15b8 subvendor=0x1458 subdevice=0xe000
vendor = 'Intel Corporation'
device = 'Ethernet Connection (2) I219-V'
class = network
subclass = ethernet
According to this post, I211 is marginally better than I219 while consuming slightly more power. Now it makes sense why the I211 chip (marked as LAN1) has a dedicated metal shroud on the motherboard.