Home cluster part 1 - Intro and Proxmox install

Figuring out how to make my own little homelab cluster
configuration
linux
proxmox
Published

November 21, 2022

Introduction

I’ve been running services on my home network for years. It started with running things bare metal on the same machine I used as my desktop for day to day work. That was a nice easy way to get started, but I was constantly running into conflicting updates, or accidentally breaking something when I tried to get some desktop service working. The next step was getting a dedicated machine solely for hosting services. This worked a lot better since my service requirements changed a lot less frequently than my desktop requirements, but I still ran into conflicting services, or breaking one service when I was testing something out on another. The next step was a dedicated machine, but running all my services in docker containers. That really helped with isolation and was also where I got serious about automating my environment with ansible, which generally meant that even a complete system wipeout only took me down for as long as it took to reinstall the base OS and re-run my ansible playbook.

Now it’s time for the next step in making my home server environment fancier - Proxmox.

Motivation

Why bother doing this? My current system works pretty well, but it still has a few issues I’d like to address with Proxmox. The first is that my base OS is a single point of failure. I can swap out individual container services with impunity, but swapping out or experimenting with different operating systems means taking my whole stack down. Of course this also means that if any of the physical components of my server fail all of my services are down until I can replace or repair those components / the whole system.

Switching to Proxmox addresses both of these issues. First, even on a single physical host I can run multiple virtual machines to test different operating system configurations without downtime. Second, I can cluster together Proxmox nodes such that even if a single host fails I can fail over to another one.

Realistically, this is way overkill for the importance of the services I’m running currently. My miniflux app going down for a while is just not that big a deal. One service that I’m planning to introduce in this environment that I was nervous to set up on my old stack for reliability reasons is home assistant but even in this new world I’m not going to have anything that won’t have an analog failback. It’s not like if this server goes down my furnace won’t work.

So honestly my motivation for this is mostly because I find it interesting to learn about clustering and want a freer hand to experiment with some new system administration and operation tools. Eventually I plan to figure out kubernetes on top of this stack, but that’s a whole other level of complexity so we’ll leave it alone for now.

Hardware selection

I waffled for a ridiculously long time deciding what hardware I wanted to run this on. For reference, my existing server is a Dell Optiplex 7020. It’s got an i5-4590, 8GB of RAM and has been frankly doing just fine with the services I’ve put on it. I know I’m adding a little overhead by adding virtualization, and eventually k8s to the stack, and I plan to add some more services (particularly on the operations side like logging and alerting tools) but honestly my compute requirements are quite modest. In terms of power budget I’m lucky enough to be in a region with fairly affordable electricity, so after running some scenarios on different hardware I concluded that it wasn’t going to be a major financial factor unless I went with lots of really old inefficient enterprise gear. However, from an environmental perspective I still want to limit my consumption.

I considered four general classes of hardware: single board computers (SBCs) like raspberry pi, off lease/refurbished enterprise desktops, used enterprise servers, and custom built PCs.

SBCs are often promoted in terms of being both affordable and low power consumption. At the time of this writing considering them is something of a non-starter thanks to supply chain issues, but even if that weren’t the case I decided against them. While a base board is definitely quite affordable, once you add on all the additional required components (storage, case, power supply) the price point is pretty comparable to a refurbished enterprise desktop for considerably lower performance, essentially no upgradability (particularly RAM, which is key) and the challenges of ARM (most software has builds for ARM, but there are still some gaps). They’re definitely the lowest in absolute power consumption, but I think the gap narrows quite a bit when considering the electricity to compute power ratio, and besides, power consumption isn’t the most important thing to me in this build.

I didn’t really consider used enterprise servers. The noise, power consumption and physical space requirements just didn’t seem to make sense. Maybe if I was based in the US where the used market seems thicker I could have picked something up for cheap enough to consider, but up in Canada the price performance just didn’t seem to be there. Some people recommend enterprise gear because it more closely approximates an enterprise environment and has some fancy management features that’s missing from the consumer market, not to mention extra reliability features like ECC RAM and redundant power supplies. None of those were compelling enough features for me to overcome the previously mentioned noise, power, and space considerations.

A custom built PC was definitely a serious consideration. I specced out a pretty beefy machine using PC partpicker that definitely would have handled whatever I threw at it for about $750 CAD. $2,250 would have gotten me a three node setup with plenty of power and the option to do things like throw in GPUs if I wanted to mess around with GPU accellerated machine learning or do lots of video transcoding. While that was technically within my budget it seemed a lot to commit to what was essentially an experiment.

Finally, after going through all those options I decided to follow the path of tiny mini micro and score some ultra small form factor (USFF) used PCs from eBay. They don’t have quite the low space or power consumption of an SBC but they’re pretty dang small and use mostly laptop components so they’re fairly power efficient. On the other hand, there’s a much better upgrade path for them compared to SBCs and they’re x86 so there’s much greater compatibility. They’re definitely not as high power as a custom build, but as you’ll see below they’re way cheaper, and I can always extend the cluster or swap them out to use as media boxes or something with beefier hardware later.

What I actually bought

  • HP ProDesk 400 G3
    • i5-7500T
    • 32GB RAM
    • 512 GB nvme SSD
    • 1TB WD Blue SATA SSD (purchased new and installed after)
    • 348.75 CAD total price - 252.75 CAD for the system 96 CAD for the SSD upgrade.
  • HP EliteDesk 800 G3
    • i5-7500T
    • 32GB RAM (purchased new and installed after, shipped with 4GB)
    • 512 GB nvme SSD (purchased new, came with 240 GB that I repurposed for a portable drive)
    • 1TB WD Blue SATA SSD (purchased new and installed after)
    • 456.63 CAD total price - 141.34 for the system, 125.15 for the RAM, 96 for the SSD, 94.14 for the nvme SSD
  • Dell 3060 Micro
    • i5-8500T
    • 32GB RAM (purchased new and installed after, shipped with 8GB)
    • 512 GB nvme SSD (purchased new, came with 240 GB that I repurposed for a portable drive)
    • 1TB WD Blue SATA SSD (purchased new and installed after)
    • 503.53 CAD total price - 180 for the system, 133.39 for the RAM, 96 for the SSD, 94.14 for the nvme SSD

As you can see the price varied between the nodes. I got lucky with the HP ProDesk because it was in Canada and came equipped with the RAM and nvme I wanted. Making those upgrades after on the other systems and ordering from the US increased the price. With further patience and luck maybe I could have saved a couple hundred bucks, but honestly I’d already been waiting a long time to get this project kicked off and I kind of think that ProDesk was a bit of a unicorn.

Installing Proxmox

The base proxmox install (should be) dead simple. Grab the ISO, rip it to a USB drive, boot it in your system and follow a couple prompts. My first issue with booting was that I couldn’t get into the boot menu or BIOS menu. It turned out that there was some weird incompatibility between my monitor and the host when using DisplayPort. Fortunately I had a DisplayPort to HDMI cable kicking around and using that let me get into the menu and select the right boot option. I had to go into the BIOS and turn off secure boot (Advanced -> Secure boot config -> legacy disable and secure boot disable in the HPs). That sounds sketchy, but it’s basically a requirement to boot anything other than Windows. While I was there I made sure that virtualization was enabled. In the HPs that was under Advanced -> System options -> and make sure “Virtualization Technology (VTx)” and “Virtualization Technologyy for directed I/O (VTd)” are checked. For the Dell the config is similar. F2 gets you into the BIOS. Under “Security” turn off “TPM 2.0” and under “Secure boot” turn off “secure boot”. There’s also a tab for “virtualization support”. Both configs were alread checked on mine, but that’s where it lives. The next weird thing I had happen was actually booting the Proxmox image. I tried multiple USB drives and image burners (balena and rufus) but just couldn’t get it working. Ubuntu server and debian both booted fine but Proxmox wouldn’t play ball. After a bunch of searching I saw a recommendation to try ventoy. I’d heard of ventoy before but dismissed it as just another thing that could go wrong when trying to get an installer booting. Ironically it worked right away and now I’m a total convert, since I can just dump ISO images into a folder on the boot drive and pick from any of them when I plug it into a system. As a PS I also switched over from using a regular USB 3.0 flash drive to a USB SSD drive and oh man, if you have one of those available I recommend it, so much faster. Once I got past those initial hiccups it was mostly straightforward. I just gave the node a hostname, set its IP, my password, time zone info and maybe a couple other things. The main sneaky thing to change was on the first menu, where you select the drive to install to. I had to click “Options”, select “ZFS RAID 0” and have it only use my nvme drive. I don’t need most of the ZFS features, but if you want to migrate VMs across your cluster easily they need to be on that sort of storage. For the Kube nodes I won’t care about that, but some VMs I will, and I want to save the SSD for persistent volumes for kubernetes. After that it rebooted, I confirmed I could access the admin portal throught the web interface and stuck it down in my utility room to continue the setup remotely from my workstation.

Base Proxmox config

I want most of the config for proxmox to be done using ansible, which I’ll cover in a follow up post. The only thing I did manually on the nodes was set them up as a cluster. That was actually ridiculously easy to do. The docs cover it well so I won’t go into it here.

Conclusion

That’s it for this post, I just wanted to introduce the topic and get to the point where I had the nodes set up. Subsequent posts will deal with further node configuration, as well as actually deploying VMs to the nodes.