Let’s build a home lab server from scratch with Linux

When you want to build a home lab server, there are hundreds or maybe thousands of different ways how to do it. I thought a lot about how I should approach this. Should I buy a server rack, or build a cheap server with older consumer hardware? Which type of Operating System should I use for my Hypervisor? Should I go with Microsoft Hyper-V (which is now free by the way), the good old VMware ESXi, or something community-powered like Proxmox?

But for me building a home lab server is more like a fun and testing project. So I mainly want to learn and understand the underlying technology. But it also needs to be rock solid and build upon common IT industry standards. So I decided to choose a Linux distribution as the main Operating System and install that on a Dell T410 Tower server. I will install the server with Ubuntu 20.04 LTS and install all tools like a Hypervisor, Storage System, and other services from scratch.


Chose the right hardware for your server

Note, this is no recommendation to buy the following server model. I just bought it because I got a pretty cheap offering on eBay. Any other server with similar components from HP should be also fine. But I still want to explain some of the reasons why I’ve chosen this hardware and what you should consider when buying hardware for your home lab server.

Consumer Hardware, Tower Server, or Rack Server?

If you want to set up a home lab server, there is nothing wrong with using the usual consumer hardware. If you don’t feel comfortable with server hardware and you just want an easy solution, just use a normal PC. However, I think it’s more interesting to get your hands on some real server hardware, you would only see in professional environments.

Tower servers have a big advantage over rack servers. They’re usually a bit quieter because they’re often used in offices. So when you don’t have a separate room, don’t buy a rack server, they produce a lot of noise. There are also some smaller tower server models which only support one CPU and are more like workstations than servers.

I decided to buy the Dell T410 tower server which is capable of 2 CPUs, 8 memory slots, 6 3,5” SATA or SAS drives with hot-swap, and 2 NICs. It was already outdated when I bought it, but having 2x intel CPUs still, gives me 8 cores and 16 threads for a very fair amount of money.

ZFS and Raid Controllers

ZFS is a file system that was developed by Sun Microsystems and part of their Solaris Operating Systems. Large parts were made open source before it was acquired by Oracle and the open-source version of ZFS was ported to Linux, Mac OS, and FreeBSD. It’s a rock-solid file system that comes with a lot of advanced features like redundancy, caching, and so forth.

But consider a few things when you plan to use ZFS as a file system on your home lab server. It’s best-practice to give ZFS direct access to the hard disk. This is a problem when using a Raid Controller, which is often used in server hardware. Those Raid Controllers act like a layer in between the device and the software, which can cause problems with ZFS. The best solution is to use a Raid Controller that offers a “passthrough” mechanism that disables the Raid-Layer and allows the system to directly access the hard disk.

You will a Raid Controller that offers a passthrough or can be flashed in the so-called “IT-mode”. Of course, the Raid Controller that was shipped with my server doesn’t have this mode and there was no firmware existing that would enable the “IT-mode”. Therefore, I needed to replace the existing Raid Controller with a pre-flashed IT firmware I could luckily buy on eBay.

Add an SSD and Hard Drives

Most Tower servers and Rack servers are capable of using SATA or SAS drives. I bought 4x 500GB 3,5” SATA hard drives on eBay, which allow me to practice with ZFS storage pools for a fair amount of money. But I still wanted the Operating System to be installed on an SSD hard drive. So I needed to buy an additional 250GB SSD drive and a cage to mount this drive in the server and an adapter to connect the internal fan control, the SSD, and DVD drive to the power supply. Note, most real server hardware doesn’t come with enough cables and connectors to connect many internal hard drives or devices, because you mainly will use the backplane to connect your 3,5” storage drives anyway.


Install the Operating System and ZFS Filesystem

Now, that I’ve replaced the Raid Controller, the noisy fan, and installed the SSD, I could install the Ubuntu Server 20.04. I’ve installed the Operating System on the internal 250GB SSD drive and formated this with the usual EXT4 filesystem. I didn’t touch the 4x 500GB SATA drives, yet. We will install the ZFS utilities on Ubuntu first and then build a redundant storage pool on all four hard drives manually.

Note, I’m not going through all the details of installing Ubuntu Server. If you need assistance you should check out the official documentation.

Install ZFS and manage storage pools

Now, we need to install the ZFS utilities to be able to use the filesystem within our Ubuntu Server.

sudo apt install zfsutils-linux

If you want to see if all your hard drives are connected, you can use the “lsblk” command. In my case I have 4x HDDs with 500GB and 1x SSD with 240GB on which the main operating system is installed.

xcad@xsrv1:~$ lsblk
NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
loop0    7:0    0    55M  1 loop /snap/core18/1880
loop1    7:1    0  55.3M  1 loop /snap/core18/1885
loop2    7:2    0  70.6M  1 loop /snap/lxd/16922
loop3    7:3    0    31M  1 loop /snap/snapd/9721
loop4    7:4    0    31M  1 loop /snap/snapd/9607
loop5    7:5    0  71.3M  1 loop /snap/lxd/16099
sda      8:0    0 232.9G  0 disk
├─sda1   8:1    0   512M  0 part /boot/efi
└─sda2   8:2    0 232.4G  0 part /
sdb      8:16   0 465.8G  0 disk
├─sdb1   8:17   0 465.8G  0 part
└─sdb9   8:25   0     8M  0 part
sdc      8:32   0 465.8G  0 disk
├─sdc1   8:33   0 465.8G  0 part
└─sdc9   8:41   0     8M  0 part
sdd      8:48   0 465.8G  0 disk
├─sdd1   8:49   0 465.8G  0 part
└─sdd9   8:57   0     8M  0 part
sde      8:64   0 465.8G  0 disk
├─sde1   8:65   0 465.8G  0 part
└─sde9   8:73   0     8M  0 part
sr0     11:0    1  1024M  0 rom

Three different ZFS options

Before we create a storage pool let me explain a bit more about ZFS, redundancy, and pools. In ZFS we have three main options to create a pool of multiple hard drives. We can simply “stripe” the drives, which will create a large storage pool, and just adds the disk space of all hard drives together. Note that this won’t have any redundancy, if we lose a drive, we lose data. The advantage of striping disks is that we can split the workload of reading and write operations because we can do these operations on more than one drive simultaneously. The performance increases the more drives we add to the pool.

If we want to add redundancy we can use a “mirror” pool, that will write the data to two drives. If one drive fails, we still have all the data. Another advantage is that we can also split the workload of reading operations because we can read the same data from two drives simultaneously. But it does not affect our write operations, because we need to write the same data to both drives and we will lose 50% of the disk space.

There is also the “raidz”, which exists in three different variants, but I will only cover the most common variant “raidz-1”. This needs a minimum of three drives and adds a parity section, similar to the hardware RAID-5 equivalent. The advantage is that we will only lose the disk space of one drive and we can also split the read workload. We also don’t lose any data if one drive fails. The performance increases and the disk space loss decreases as we add more drives to the raidz pool.

Set up a raidz storage pool

Let’s build a raidz storage pool with the following commands. I’m using all four disks to build the raidz which will provide us the disk space of 3 drives and redundancy.

sudo zpool create store raidz /dev/sda /dev/sdb /dev/sdc /dev/sdd
xcad@xsrv1:~$ sudo zpool list
NAME    SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
store  1.81T   888K  1.81T        -         -     0%     0%  1.00x    ONLINE  -
xcad@xsrv1:~$ zpool status
  pool: store
 state: ONLINE
  scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        store       ONLINE       0     0     0
          raidz1-0  ONLINE       0     0     0
            sda     ONLINE       0     0     0
            sdb     ONLINE       0     0     0
            sdc     ONLINE       0     0     0
            sdd     ONLINE       0     0     0

ZFS automatically mounts the storage pool with the name under the Linux root directory. In my case, this is /store. To measure the reading and write performance I’m using the command-line tool “dd” to write and read some test files to the partition.

xcad@xsrv1:~$ sudo dd if=/dev/zero of=/store/test1.img bs=1G count=1 oflag=dsync
1+0 records in
1+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 6.21784 s, 173 MB/s

xcad@xsrv1:~$ sudo dd if=/store/test1.img of=/dev/null bs=8k
131072+0 records in
131072+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 1.25948 s, 853 MB/s

If you want to destroy a ZFS pool, simply execute the following command. Note, this will delete all files in the storage pool.

xcad@xsrv1:~$ sudo zpool destroy store
xcad@xsrv1:~$ zpool status
no pools available

Set up a combination of mirror and stripe

In hardware RAIDs you may be familiar with something called RAID10, which is a combination of “stripe” and “mirror”. The advantage of combining different methods to get the best of both world. We can create a “stripe” of two “mirrors”, which requires four drives. But in this case, we get the additional write performance of a “stripe”, combined with the redundancy of “mirrors” with a disk space loss of only 50%. In ZFS you can also approach this scenario with the following command.

xcad@xsrv1:~$ sudo zpool create store mirror /dev/sda /dev/sdb mirror /dev/sdc /dev/sdd
xcad@xsrv1:~$ zpool status
  pool: store
 state: ONLINE
  scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        store       ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            sda     ONLINE       0     0     0
            sdb     ONLINE       0     0     0
          mirror-1  ONLINE       0     0     0
            sdc     ONLINE       0     0     0
            sdd     ONLINE       0     0     0

errors: No known data errors

Now, if we measure the performance and compare it to the raidz pool, you can see it increases the write performance, but the read performance is the same. That’s because the raidz pool needs to write to all disks, but read from two drives simultaneously. The “mirrored” “stripe” pool can read and write from/to all drives simultaneously.

xcad@xsrv1:~$ sudo dd if=/dev/zero of=/store/test1.img bs=1G count=1
1+0 records in
1+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 1.43098 s, 750 MB/s

xcad@xsrv1:~$ sudo dd if=/store/test1.img of=/dev/null bs=8k
131072+0 records in
131072+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 1.25672 s, 854 MB/s

Which setup you chose is up to you. A combination of “mirrors” and “stripes” will give you the best performance. If you want to lose as little disk space as possible by adding redundancy and read performance, you should choose a raidz.

ZFS has much more advanced features, like caching, snapshots, and so forth. If you’re interested and want to learn more about it, don’t forget to subscribe to this channel because I will always make great tutorials for IT professionals or home lab enthusiasts.


Install libvirt / QEMU and KVM

A good home lab server should also run a hypervisor which can be used to run virtual machines. This can be easily done with QEMU and KVM on a Linux server. While all-in-one solutions like Proxmox, VMware ESXi, and similar offer easy and simple configuration via a GUI. A hypervisor on Linux with QEMU and KVM consists of multiple components we will install in the command-line. But don’t worry, I will cover an easy solution to create and manage virtual machines with a GUI tool later. First, let’s install all the necessary components to run a hypervisor stack on Linux.

xcad@xsrv1:~$ sudo apt install -y qemu qemu-kvm libvirt-daemon virtinst bridge-utils

You should also check with the following command if your server is configured to use hardware acceleration.

xcad@xsrv1:~$ sudo kvm-ok

Check if the libvirtd service is running.

xcad@xsrv1:~$ sudo systemctl status libvirtd
● libvirtd.service - Virtualization daemon
     Loaded: loaded (/lib/systemd/system/libvirtd.service; enabled; vendor preset: enabled)
     Active: active (running) since Mon 2020-10-26 08:04:12 UTC; 17min ago
TriggeredBy: ● libvirtd-admin.socket
             ● libvirtd-ro.socket
             ● libvirtd.socket
       Docs: man:libvirtd(8)
             https://libvirt.org
   Main PID: 1534 (libvirtd)
      Tasks: 19 (limit: 32768)
     Memory: 91.0M
     CGroup: /system.slice/libvirtd.service
             ├─1534 /usr/sbin/libvirtd
             ├─1707 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/u>
             └─1708 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/u>

You can also add the permissions to manage virtual machines to your username.

xcad@xsrv1:~$ sudo usermod -aG kvm xcad

Now we can use the command-line tools virsh and virt-install to manage networking, storage, and virtual machines. But especially the management and installation of virtual machines from the command-line can be tricky and has some limitations. Easier is to use a GUI application with a VNC client that automatically connects to the virtual machines. On Linux, we can use the tool virt-manager which runs natively on Linux. Unfortunately, it doesn’t have a Windows Client, but there is a pretty easy trick to run it in WSL2.

First, you need to download and install the Vcxsrv X11 server on Windows, which you can download here. Then you need to add an environment variable in the .bashrc or .zshrc file of your WSL2.

export DISPLAY=$(cat /etc/resolv.conf | grep nameserver | awk '{print $2}'):0

Now, install virt-manager on your WSL2, or your Linux distro. To connect to your home lab server, you should use proper SSH authentication with private and public keys. If you want to learn more about it, I’ve written an article that teaches you everything about SSH and private and public key authentication. If you set up your SSH key authentication you can simply connect virt-manager via SSH to your home lab server.


Summary

This is how I installed and set up my home lab server from scratch with Linux. I hope this helped you with some tips and tricks and you could get an idea of how you could set up your own home lab server. Of course, there are so many different options and you should decide which set up fits best your needs.

Leave a Comment

I accept the Privacy Policy