Introduction

This is the third post ( part 1, part 2) documenting my adventures setting up a home cluster. In this one I will try a few different methods of getting VMs installed on proxmox. As with the previous posts, this is not intended to be a how to guide from an expert. I haven’t used proxmox before working on this project, so I’m mostly doing this to document what I do for future reference, or maybe provide others with the perspective of what it’s like to work on proxmox as a relative beginner.

All the code I reference in this post is in my recipes repository. Specifically, the ansible role to create templates is here and the terraform code is here

Basic menu driven install

Create the VM

The most obvious way to install a VM is through the UI. I know I won’t want to take this approach indefinitely as it involves manual work and isn’t reproducible (at least not easily), but it seems like the right place to start, both to ensure I don’t have any unforeseen issues with my setup, and also to provide a baseline for comparison when I try other methods later.

Selecting one of my nodes from the web interface I click “Create VM”. In the first tab I pick the node to install to and give it a name, we’ll do ubuntu-test for this. I could also assign it to a resource pool if I had any of those created but I don’t so I won’t. The other thing I can assign is a VM ID, which is the unique numeric identifier proxmox uses internally. At this point I’m fine to let proxmox manage that though so I’ll leave it on the default.

Checking the advanced options I can also configure the VM to start at boot so it will come back up if I reboot my cluster. I can also configure the order it starts/stops. The start at boot setting seems like it would be handy for production services, but I’m just testing so I’ll leave it for now.

On the next tab I can configure the OS. I’ve already configured my NAS (set up in part 2) to hold things like ISO images for installing and uploaded an Ubuntu 22.10 server image, so I’ll select that. The guest OS settings are already correctly set on Linux with a modern kernel so I’m all good there.

Next up is the System tab. The first option is Graphic Card. There’s a ton of options under this one, but at this point I don’t have any intention of installing anything that will care so I’ll leave it at default. Maybe at some point I’ll have a system with a GPU that I want to pass through, or will need a Windows server, but not right now. I also have to pick a machine option. Based on the docs as long as I don’t want to do PCIe passthrough I can stick with the default, so I will for now. Next I pick a SCSI controller. Again, referring to the docs the VirtIO SCSI Single option that it had selected by default seems perfect for me. There’s also a checkbox for Qemu Agent. Reading the docs this seems like a handy thing to have, so I’ll turn it on (looks like mostly it’s for cleaner shutdown and pausing during disk backups). The last thing on this tab is whether to enable TPM. Since I’m not making a Windows image I don’t need this, so I’ll leave it unchecked.

Following that we’re on to Disks. I can create multiple disks I’m sure, but for now let’s just set up one. First I make sure that the backing storage is my local-zfs storage, which is the NVME drive on the host, rather than my NAS. I haven’t configured the SSD in these hosts yet, I’m planning to set up ceph on them but that’s for a future post. The other basic thing to set on this page is disk size. I’m not planning to keep this image around, so I’ll stick with the default 32GB for now. The Bus/Device field defaults to the SCSI interface I set up on the last tab so that seems fine. There’s an option for cache mode as well. Right now I’m not really sure what that does, but from the docs the default of no cache seems like it will work for me, so I’ll leave it. Taking a look at the docs it seems like I want to have the Discard option checked so I’ll do that. From the docs IO Thread only seems like it really matters if I have multiple disks attached, but I don’t see the harm of turning it on so let’s do that. I’ll check SSD emulation since the underlying disk really is an SSD and the guest OS might as well think so too. I’ll uncheck the backup option on this one, since I’m planning to just destroy this VM shortly after I create it and I don’t need backups hanging around. I want to be able to try replicating this VM to different hosts, and I’d want this disk to be included, so I’ll leave skip replication unchecked. The last thing I have to pick is the Async IO option. From reading this it seems like the default io_uring will work for me, I’m not deep enough on how this sort of thing works to have strong opinions or requirements so I’ll go with the default.

Now we come to CPU. For sockets I’ll leave it at 1, since all my hosts have only 1 physical socket. For cores my hosts have either 4 or 6 cores, so there’s definitely no point going over 6. Since this is just a test machine let’s just give it 2. For CPU type I’m going to leave it on the Default (kvm64). From the docs on CPU type if I set the CPU to host it will exactly match the CPU flags and type of my host system, but I might have migration issues across hosts, since they’re not all the exact same CPU. The default will allow easier migration, but might disable some specific CPU flags that would be beneficial. For now I’ll stick with the easy option. There’s some other stuff for CPU limits and extra CPU flags here that I’m also going to leave alone for now.

Now we’re on to memory. Each of these hosts has 32GB of memory, so I don’t really have to be cheap here, at least while I’m testing. Under advanced I can set a memory floor and enable/disable ballooning. From the docs, I want to have ballooning enabled, even if I have the memory floor and ceiling set the same, so that my host can see how much memory the VM is actually using. If I was running a bunch of VMs with dynamic memory requirements I could see overallocating the max across them and setting the floor for each. In this case I’m just going to leave it at the default 2GB since I’m not going to actually run anything on this VM.

Almost done, next up is network. I’ve only got one bridge right now so I’ll leave that selected. I’m not currently doing any VLAN stuff in my network so I’ll leave the VLAN tag entry blank. For model I’ll stick with VirtIO as the docs say that’s the one to pick for best performance as long as your host supports it. The firewall option is checked by default. I haven’t looked into the proxmox firewall at all at this point, but let’s leave that on for now. I can also do some rate limiting and other fancy network config here but I’m going to leave those on the default for now.

The only thing to do now is confirm my settings and actually create the VM. I’ll check start after created so it fires up right away.

Configure the VM

After waiting a little bit I can see my newly created VM listed under the node I set it up on. Clicking into that VM and selecting the Console section I can see that I’m in the Ubuntu server installation wizard. Since this isn’t a post about installing Ubuntu server I’ll work through the menus without writing everything down. Going through the install worked fine until it came time to reboot and it failed to unmount the virtual CD-ROM that had the installation ISO. I went to the hardware tab on the VM in the proxmox interface, removed the CD-ROM and rebooted. After the reboot the VM came up fine, and I was able to ssh into it from my desktop.

Create a template image

After confirming the manual VM creation process worked, I started looking into automating the process. From what I could see, most automation tools like ansible or terraform require you to have a VM template created that they can use. There are also some tools to automate the creation of templates, but let’s not get ahead of ourselves. There’s tons of docs on getting a template created and I started getting a bit of analysis paralysis going through them all, so let’s try the Techno Tim guide and see where that leads.

I was able to mostly follow the guide, and other than having to change references from local-lvm to local-zfs for storage of the disk I didn’t encounter any obvious errors. I created a new machine and it started up, but I couldn’t access it over the web interface for the shell, or see info about its IP (no qemu guest agent installed). In the cloud-init config I set the IP to be DHCP, just to see what would happen. I wasn’t able to resolve the host by name to try and ssh in using that. After checking my router I found a host just named ubuntu and was able to ssh into its IP with the username and ssh key I specified in the cloud init template. For a further test I created a second clone from the template. One thing I noticed was that I had to create it on the same node as the template. Presumably I can migrate it later. The second VM came up, got a new ID in proxmox, and pulled a different IP showing a different MAC address in my router. I was able to ssh into it the same way. After confirming that I shut it down and migrated it to another node, just to see how that would go. The migration went fine, the VM came back up, got the same IP, and I was able to ssh back into it.

Thoughts on manual templates

This wasn’t too bad. There are a few tweaks I’d want to apply, like adding the guest agent into the machine, but overall template creation is pretty easy. I could see wanting to update my templates semi regularly when new versions of base OSs come out though, and I’d like to understand more of the theory behind how this actually works, since a lot of what I did was pretty much copy paste. To do that, I’ll clear these out and look into some other template creation options.

Cleaning up

I don’t want lingering stuff from this experiment hanging out on my nodes, so let’s go in and see what I have to get rid of. First is the created VMs - I can stop and remove them from the UI easily enough. Same deal for the template VM. I checked the local storage through the UI as well and it looks like any virtual disks I created were removed when I got rid of the VM. The only other thing to remove was that initially downloaded cloud image, so I went into the shell for the node and just ran rm to get rid of that.

Automate creating a template

Creating templates doesn’t immediately feel like the sort of thing I’ll have to do super often and will therefore want to automate. In researching how I would do it though I found a few use cases, namely to schedule the creation of updated templates to avoid a long running apt update && apt upgrade cycle on each newly created image. I found a nice looking post that had a fairly reasonable looking workflow using a few shell scripts. Further research suggested that the “production” way to do this would be with packer. That seems more complicated, but it would help me learn a more broadly applicable skill that I might be able to transfer to other projects. I honestly can’t tell if that’s a good use of my time or just yak shaving so I’ll try and get by without for now.

Following the post linked above, there’s only one pre-requisite on the proxmox nodes to run the scripts, and that’s libguestfs-tools in order to modify the cloud image bases we’ll be building templates from. That’s easy enough to add to the ansible playbook I’ve been using throughout this series to configure my proxmox nodes. After that there’s just four scripts I have to tweak slightly for my own requirements and then get onto at least one node. I can also set these up in ansible with some templates, which should make them easier to modify and otherwise maintain. Plus then I have them stored somewhere if I have to rebuild these nodes in the future. The repository associated with the above post is here.

As a starting point I copied in the scripts as templates into an ansible role. I swapped out all the variables that were hard coded in the scripts for ansible variables, and then set what I wanted to initially test on as ansible variables in the defaults for the role. The idea is that this way if I want to build multiple templates I can just call this role with a variables file that overrides the specific things I want to change (image id, cloud image). After a little bit of fiddling I got the files copied over and ran the script, which did create a template for me. So far one advantage over the manual template from before is that this image has qemu-guest-agent installed, so I can see the hosts IP in the summary tab. There’s still some more stuff I’d like to sort out though.

By changing the storage location of the template from local storage of the node to my NAS I was able to clone the template to another node in my cluster. That image then came up in a bootloop though. Even more fun, I couldn’t stop it from the web interface, which meant I couldn’t delete it. I had to ssh into the node it was running on, run ps aux | grep "/usr/bin/kvm -id <VM ID>" and then kill -9 that PID. Crazy. I tried creating it on the same node as the template with a target of local storage but got the same issue.

After looking at the docs it seems like if I want to automate building images from templates I’ll be using the template name, rather than a VM ID anyway. So I think I’ll try modifying the script to create a template with the same name but different ID on each of my nodes, which should let me provision VMs to any node. First let’s clean up the template some more though. As a future project maybe I’ll come back and figure out why building the template on my NAS causes it to boot loop, but that’s a problem for future me.

One thing I definitely want to be able to do is scale VMs I create off these templates up or down. 2 cores, 2GB of RAM and 2GB of disk will not always do it. To test this I create a VM from the template without modification and ssh into it. df -h confirms that I have 2GB of disk assigned to the VM by default. lscpu shows 2 cores. free -h confirms I have 2GB of RAM. Let’s turn the VM off and adjust those. From the Hardware tab of the VM from the proxmox UI I adjust the CPU up to 3 cores and RAM up to 4GB. Disk resizing cannot be done from the UI, so from the terminal based on the docs I run qm resize 100 scsi0 +5G to add 5GB to the disk. Let’s fire the machine back up and see what happened. lscpu indeed now shows 3 cores, that’s cool. free -h shows the updated amount of RAM. Even df -h shows the correct amount of disk. That last one is frankly surprising to me because the docs indicate that only the disks should have been resized, not the logical volumes or anything else, which would have meant some commands being run within the VM to make the space available. That has also been my experience with VMs at work. Maybe it’s something fancy in the Ubuntu image I’m using? Either way, pretty sweet for now.

Another thing to change is how the VM authenticates over SSH. The default way I’ve been doing it is to just put my laptop’s public key in ~/.ssh/authorized_keys using cloud-init. This works ok, but it’s not how I manage ssh in the rest of my network. See my earlier post for details, but the tldr is I want to us a certificate authority to allow any signed key to authenticate as a user, and all my host keys to authenticate themselves with a CA. The former is a bit of a convenience as I could just add a couple keys for my other devices to my keyfiles file and keep them up to date if I rotated keys. The Host key thing will be super handy though, since otherwise I’ll have to manually verify that I trust the host key of each new VM I start up, and if I ever tear down and replace a VM I’ll get errors on host validation. So let’s fix that. The first step is to copy the host CA key and the user CA public key into a folder on my NAS that’s accessible from proxmox so I can inject those keys into the templates. I could have ansible copy them over and encrypt them in the playbook, but I think having them on the NAS is slightly more convenient and secure, even if I’d have encrypted the keys with ansible-vault. Next I need to modify the template creation script to copy those files in, and modify sshd_config to use them. While I’m at it I can turn off password authentication over SSH for a little more security. This actually went surprisingly smoothly. I updated the build_image script to copy in the public key for the user CA as well as the private host CA key. Then I set it to run a very slightly modified version of the host setup script I created in my earlier post. After re-running the template build script and creating a VM from the new template I was able to ssh in from my laptop without having passed in a key file to the cloud-init template, or being prompted to validate the host key. Magic!

A note on name resolution. I think I remember seeing/reading this somewhere but it didn’t come back to me until I started troubleshooting. When I first create a machine from a template, if I tell it to use DHCP for IP address acquisition it gets a lease from my router with the hostname ubuntu. So if I want to ping/ssh the machine by name I have to use ubuntu.local.ipreston.net instead of <vm-name>.local.ipreston.net. I typically like to use DHCP on my servers and then just do static mappings in my router to pin them to an IP, rather than hard coding the IP itself, mostly so that I can get easy name resolution without having to put manual entries into my DNS. Even my proxmox nodes themselves which do use static IPs, I created a static mapping in my DHCP server to their MAC addresses so that all my IPs would be available in one place. Anyway, after you reboot the VM once it gets a lease with its actual hostname so just reboot it once, or manually alter the hostname when you do your static mapping.

Automate creating another template

Having a working Ubuntu template is pretty handy, but what if I want to branch out? Can I apply this approach to other distros? I’m pretty sure this approach will work fine with another debian based distro, and probably even another fairly standard Linux like CentOS will be fine (although I should test). But what about weird ones? Specifically I want to see if I can get this working on flatcar Linux since I want to try using it for my kubernetes nodes. Let’s walk before we try running though and extend to another version of Ubuntu.

The first thing I want to do is tweak how I’m numbering my templates. Right now each template gets a variable set for its whole VM ID. I’d like to break that out into chunks. The first digit should just always be 8 (at least for now) to indicate a template and keep it out of the range of actual VMs I’m deploying. The next one I’m thinking should be the node the template is created on, and then the last two digits can be an identifier for the specific template. This actually wasn’t bad at all. The one variable definition gets a little long, but basically I just go from one line in defaults of build_vm_id: "8000" to this.

This relies on all my proxmox nodes having hostnames of the format pve<Num> but I can work with that. The number of digits in my IDs will change if I get more than 9 nodes or 99 templates too, but I’m not really expecting that to happen, and I don’t even think that would necessarily break anything if it did, so I won’t worry about it for now.

With that slight modification to the role complete I set my playbook to call the role twice, modifying the variables from the defaults for just the template name, the template number, and the URL of the cloud image to build from for Ubuntu Jammy and Kinetic.

Around this time I realized it was going to be a little tedious running the template build script each time I added a template, so I added a handler to the role to execute the template build script whenever it made a change. It took a little bit of tweaking to figure out that I needed the full path of the script I wanted to run, as well as set it as the working directory so I could call the subscript that defines all the build variables. After those changes the handler triggered properly and built my templates whenever the script changed, or I added a new template to build.

At this point the general template creation process is working quite nicely for Ubuntu versions, but what about other distros? Let’s give debian a shot. I grabbed the cloudgeneric version of debian bullseye from their official cloud images page and plugged it into my playbook. No problem at all. The template built, I was able to build an image from it just the same as the Ubuntu ones.

Let’s get a bit braver and branch out to an even more different distro Rocky Linux. This one might come in handy if I want to try out anything enterprisey or just want to see what the Red Hat experience is like. I found their generic cloud images here and plugged the link into my playbook. The template built ok, but trying to run the VM I ran into problems where it got stuck on a line that said Probing EDD (edd=off to disable)... ok and just hung out there. Similar to the weird boot loop I got deploying from my NAS I wasn’t able to shut down the VM from the Web UI and had to go into the terminal on the node and ps aux | grep "/usr/bin/kvm -id <VM ID>" to find its PID and kill -9 it before I could remove the VM. I guess I have to do some troubleshooting. A little searching finds that this error is pretty common, although it doesn’t actually relate to the message, but something that’s happening after. There are a few potential kernel configs I might be able to change, but as I’m poking around in the machine I notice something interesting, it’s got way more disk to start than my Ubuntu templates did. I wonder if I’m somehow filling the disk, so I use that command from the previous section and resize the disk on a newly cloned template before starting it up. Disappointingly this did not solve the problem. Another weird thing I noticed during the start up is that CPU usage on the VM is pinned at right around 50%. Since I gave it 2 cores that suggests that one core is working flat out on something. Several of the posts indicated that after about 10 minutes the system would come up. That’s obviously a terrible startup time, but I’d like to give it a while to see if I at least have the same problem. So I go do some reading and let this VM run for a while… and discover that tragically the usually perfect strategy of ignoring a problem and hoping it goes away doesn’t work in this case.

Examine the template creation script and modify it

Something about how I have my VM configured is not playing nice with Rocky Linux. It could just be a very specific thing that I only want to modify for that distro, but I also just copy-pasted most of the other template creation parameters from some guy on the internet. So before I assume that my basic parameters are the best and it’s only Rocky that needs to be modified, let’s examine those options that I’m using and see if I want to modify any of them. Maybe while I’m at it I’ll fix my Rocky issue (or introduce new ones to working distros), but at a minimum I’ll have a better understanding of what’s going on.

The first little bit of the script downloads a cloud image, and then uses virt-customize to update packages, install a list of packages (just qemu-guest-agent and cloud-init by default), copy in a build-info file with some metadata about the template build, copy in some ssh related files and have a script to set them up on first boot (note to self, maybe that’s the part that’s breaking in Rocky, I’ve only tested that script in debian and Arch based distros so far). That stuff (except maybe the ssh part) is all straightforward and I understand what it’s trying to do, so let’s skip to the next line:

qm destroy ${build_vm_id}

Remove the old template before you build a new one, makes sense.

qm create ${build_vm_id} --memory ${vm_mem} --cores ${vm_cores} --net0 virtio,bridge=vmbr0 --name ${template_name}

Create a new VM (that we’ll turn into a template later) with an ID of build_vm_id, memory and cores set to our variables, and a virtio network adapter, which is what I did in the manual template creation. Finally we give it a name based on the template_name variable. So far so good, but I had a lot more options available when I built a VM manually earlier in this post, anything else I should set? Reading back through my manual config I set basically everything else to defaults so I think I’m good here. Let’s see what’s next.

qm importdisk ${build_vm_id} ${image_name} ${storage_location}

Ok, this is fine, I’m importing the disk image I downloaded and modified to the VM I created and putting it in the storage location I specify. All seems fine. Maybe I’ll need to revisit this if I take another crack at storing these templates on my NAS, but fine for now.

qm set ${build_vm_id} --scsihw ${scsihw} --scsi0 ${storage_location}:vm-${build_vm_id}-disk-0

Ok, here’s where I deviate from what I picked in the manual build. In my defaults (based on the script I copied in) I had scsihw set to virtio-scsi-pci, whereas in my manual build I went with virtio-scsi-single. I’m struggling to find the actual difference between these settings, but let’s change it for kicks for now.

qm set ${build_vm_id} --ide0 ${storage_location}:cloudinit

Add the cloud-init drive, seems fine. It’s emulating a CD drive so ide makes sense.

qm set ${build_vm_id} --nameserver ${nameserver} --ostype l26 --searchdomain ${searchdomain} --ciuser ${cloud_init_user}

Add a couple defaults to the cloud-init template and set the ostype to linux (l26). No worries there.

qm set ${build_vm_id} --boot c --bootdisk scsi0

--boot c tells it to boot from hard disk (as opposed to CD or network) and we set the bootdisk to the image that’s been mounted to the VM. Seems fine.

qm set ${build_vm_id} --agent enabled=1

This turns on qemu agent, which we want.

One thing I noticed from going through this is I had some lines that set multiple options, even though they weren’t necessarily related. So I cleaned that up to be one option per line. Easier to parse and modify that way.

I took a quick look back at the manual config section and didn’t see anything else that stood out, so I guess I have to get back to fixing Rocky Linux.

Get back to making Rocky Linux work (spoiler, I do not succeed)

Ok, that was a fun side quest, but let’s get back to figuring out Rocky. I re-run my template creation playbook, just in case that storage config changed anything. I also found a proxmox forum post where someone was having the same problem with a particular RHEL image, but no solution. That post also said it worked fine with RHEL 7 and the issue was with 6. I’m trying Rocky 9 (I believe they use the same version as RHEL for compatibility) so I don’t know if that’s helpful. This post suggests the output just means my console output is being redirected somewhere else, so I’m not seeing whatever the actual issue is. I guess I should fix that first regardless. One suggested solution there is to change the default tty from serial. An alternative approach there, is to check out the proxmox docs and enable serial out on the VM with qm set <VM ID> -serial0 socket. Let’s add that line to my template and see if I get anything. A little bit of progress in that it doesn’t just tell me I don’t have serial on that machine, but I also only see starting serial terminal on interface serial0 (press Ctrl+O to exit), which isn’t exactly informative. Let’s ditch my ssh script setup on first boot, just to make sure that’s not what’s hanging the template. Removing it from the template script gives me the same issue, so the problem is elsewhere. Just for kicks, let’s try a different Rocky cloud image. I found this blog that’s using the GenericCloud image rather than the GenericCloudBase image I was using. I’m not sure why I picked GenericCloudBase to begin with so let’s swap over and see what happens. Still nothing. Ok, the blog also has a bunch more cloud-init modules installed than I do. Maybe one of them will fix things. Let’s add them to the package list for that template. Still no luck. Ok, back to basics. We have a blog post where someone made a template, and apparently it worked. Let’s try manually working through those instructions and see what happens. Well, first problem their link goes to a pinned daily build of Rocky that’s no longer available on the site. Fine, we’ll do the latest one and hope that there’s not just some daily build issue that’s leading to all of this. So I get the same error following the guide. After a little more digging I find a link in the rocky vault to the exact image that the blog was using. Let’s apply my template to that image. Slightly better luck. It still bootloops, but I can actually see things in the console and by being very speedy I was even able to get a screencap of the kernel panic error it’s throwing.

Finding out that this was the issue led me to a proxmox forum post where it turns out lots of people are having this issue with Rocky 9 if they set their CPU to the default kvm64. I reset my VM to use host for the CPU. That fixed the boot loop error but led to another error. At this point I decided I didn’t feel like running Rocky Linux very much.

Try Arch Linux

Rocky was supposed to be an intermediate difficulty distro to test out my template process. I don’t actually have a use case for it, I just figured it would be more different than debian but less different than flatcar when it comes to testing. I’m really hoping that I just got unlucky and that other distros won’t be so hard. Let’s see if that’s correct. I don’t have an immediate use case for Rocky, but I do like running Arch, it’s what my current server has. Let’s try that. Building the template goes fine, and this one actually boots to a login prompt from the proxmox UI, so we’re on a happier path than with Rocky already. At first it seemed like Arch wasn’t updating my default user. I started up the image and tried to ssh in but got a connection refused error. Trying to ssh in as the default arch user that the image uses got a permission denied error instead. After some testing it turns out that cloud-init just takes longer to complete on first boot in Arch, I think because it has to do more package updating. If I just left the VM running for a bit I was able to ssh right in.

Try flatcar Linux

I’ve never used flatcar before, but it sounds interesting and this blog recommended it for self hosting kubernetes so I’d like to have it available in my environment. I found this repository which had its own scripts to create a flatcar template. Most of it looks broadly similar to the approach I’ve been taking, so let’s try it out. I notice that flatcar images come in .img.bz2 format instead of .img or .qcow2 like the other files I’ve downloaded. I may have to add in some logic to the script to extract images in that case. As a first step though I just tried running the whole workflow as is. That got me a template built, but the VM I created off it couldn’t find any bootable media, suggesting the disk creation didn’t work as intended. Probably because I have to extract it first. After adding a little bit of logic to my image building script:

if [ "${image_name##*.}" == "bz2" ]; then
    bzip2 -d ${image_name}
    image_name="${image_name%.*}"
fi

I got the VM to boot. I could auto login as user core over serial, but it looks like none of my other cloud-init config stuff worked. This post is already getting super long though so I’m going to save getting a fully working flatcar image for a separate post and declare victory on my general goal of “be able to make templates automatically”.

Automate deploying VMs from templates

I’m not entirely done with templates at this point. I still want to schedule their rebuild so that I can have fresh templates to build VMs, and I need to figure out some way to be able to deploy VMs based off these templates to any node, either by recreating them on each node, or figuring out why creating them off shared storage didn’t work before. How exactly I go about some of that might depend on how I actually want to deploy VMs from these templates though, so let’s figure that out and come back to templates later.

Once again I’m at a decision point. I can keep using ansible to deploy VMs, or I can switch to terraform. While for templates I avoided the hashicorp product and the associated learning curve, I’ve actually used terraform a bit before, and I know that it’s something I’ll want to learn for other applications like deploying cloud resources, so in this case it seems to be worth the extra effort to figure out.

In my recipes repository I made a terraform directory and a pve subdirectory under that to contain this project. That repository already has terraform configured from my devcontainer so I’m off to a good start. I created a main.tf file and put in just the required info for the proxmox provider so I could run terraform init. This seemed to work ok, so next up is to configure the connection to my cluster.

Following the docs for the provider I created a terraform user in my cluster. In theory I should automate this with ansible too as part of my proxmox setup, but I feel like I’m already getting pretty in the weeds here and would like to move towards actually accomplishing things. I guess I’ll end up regretting this if I need to entirely rebuild my cluster and can’t figure out why my terraform jobs aren’t running all of a sudden. After creating the user I added a pve_creds.env file to the directory (but also put it in .gitignore) with the username and password I’d just set up. This part is fragile too, since if I move to a different machine or delete this repository I’ll have to recreate that file. At some point in the future I’ll either figure out vault or maybe just set up the bitwarden CLI to properly retrieve secrets. But that’s a project for another day.

With my credentials (presumably) set up, let’s try and provision a machine. There’s all sorts of fancy stuff I can eventually do to parametrize the build and I will definitely set all of that up eventually, but for now let’s start with spinning up one machine using hard coded values.

resource "proxmox_vm_qemu" "test_server" {
  count       = 1
  name        = "test-vm-${count.index + 1}"
  target_node = "pve1"
  clone       = "ubuntujammytemplate"
  agent       = 1
  os_type     = "cloud-init"
  cores       = 3
  cpu         = "host"
  memory      = 4096
  bootdisk    = "scsi0"
  disk {
    slot = 0
    # set disk size here. leave it small for testing because expanding the disk takes time.
    size     = "10G"
    type     = "scsi"
    storage  = "local-zfs"
    iothread = 1
  }
  network {
    model  = "virtio"
    bridge = "vmbr0"
  }
  ipconfig0 = "ip=192.168.85.9${count.index + 1}/24,gw=192.168.85.1"
}

The first run of terraform plan was disappointing because I’d accidentally used the username for the terraform user in one post I’d been reading in the environment, but set a username based on the official docs on my datacenter, so I just got a permission denied. After that though it looked promising with the following:

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # proxmox_vm_qemu.test_server[0] will be created
  + resource "proxmox_vm_qemu" "test_server" {
      + additional_wait           = 0
      + agent                     = 1
      + automatic_reboot          = true
      + balloon                   = 0
      + bios                      = "seabios"
      + boot                      = "c"
      + bootdisk                  = "scsi0"
      + clone                     = "ubuntujammytemplate"
      + clone_wait                = 0
      + cores                     = 3
      + cpu                       = "host"
      + default_ipv4_address      = (known after apply)
      + define_connection_info    = true
      + force_create              = false
      + full_clone                = true
      + guest_agent_ready_timeout = 100
      + hotplug                   = "network,disk,usb"
      + id                        = (known after apply)
      + ipconfig0                 = "ip=192.168.85.91/24,gw=192.168.85.1"
      + kvm                       = true
      + memory                    = 4096
      + name                      = "test-vm-1"
      + nameserver                = (known after apply)
      + numa                      = false
      + onboot                    = false
      + oncreate                  = true
      + os_type                   = "cloud-init"
      + preprovision              = true
      + reboot_required           = (known after apply)
      + scsihw                    = (known after apply)
      + searchdomain              = (known after apply)
      + sockets                   = 1
      + ssh_host                  = (known after apply)
      + ssh_port                  = (known after apply)
      + tablet                    = true
      + target_node               = "pve1"
      + unused_disk               = (known after apply)
      + vcpus                     = 0
      + vlan                      = -1
      + vmid                      = (known after apply)

      + disk {
          + backup             = 0
          + cache              = "none"
          + file               = (known after apply)
          + format             = (known after apply)
          + iops               = 0
          + iops_max           = 0
          + iops_max_length    = 0
          + iops_rd            = 0
          + iops_rd_max        = 0
          + iops_rd_max_length = 0
          + iops_wr            = 0
          + iops_wr_max        = 0
          + iops_wr_max_length = 0
          + iothread           = 1
          + mbps               = 0
          + mbps_rd            = 0
          + mbps_rd_max        = 0
          + mbps_wr            = 0
          + mbps_wr_max        = 0
          + media              = (known after apply)
          + replicate          = 0
          + size               = "10G"
          + slot               = 0
          + ssd                = 0
          + storage            = "local-zfs"
          + storage_type       = (known after apply)
          + type               = "scsi"
          + volume             = (known after apply)
        }

      + network {
          + bridge    = "vmbr0"
          + firewall  = false
          + link_down = false
          + macaddr   = (known after apply)
          + model     = "virtio"
          + queues    = (known after apply)
          + rate      = (known after apply)
          + tag       = -1
        }
    }

Plan: 1 to add, 0 to change, 0 to destroy.

That looks like the sort of thing I want to do, let’s do it with a run of terraform apply… It worked! It took a fair bit longer than doing it manually runtime wise (about a minute and 30 seconds), but I had read that the proxmox API is kind of slow, so that wasn’t totally surprising. At the end of the process though I had a new VM with the name and IP I’d assigned with the CPU/RAM/Disk I’d specified that I was able to ssh into. Neat! There’s a lot more I’d like to do with terraform in terms of actually deploying machines I care about of course, but this post is already getting super long so let’s save that for later.

Automate creating templates across nodes

So it seems like terraform is going to be cloning templates based on their name, not their VM ID. This should mean as long as I have a template with the same name on each node, I should be able to provision VMs to any node I want from terraform. It’s definitely a bit of extra disk/CPU to do it this way, but it seems fairly clean and easy, so let’s give it at try. I think all I’ll have to do is change my playbook to target all of my nodes, since I already have logic to adjust the VM id of the template based on which node it’s on. Turns out that did indeed work, so not much to write about this one.

Automate scheduled rebuilds of the templates

Last thing to do before I wrap up this section of my homelab saga. That’s to schedule the creation of these templates. This way if there are updated images of a particular distro, or if the packages themselves need to be updated, any machines I create from the template will have those changes reflected regularly.

Using the cron module, I created a task to add an entry for each template:


- name: Create crontab entry for template creation
  ansible.builtin.cron:
    name: "Create template {{ template_name }}"
    minute: "{{ 5 * build_vm_template_num | int }}"
    hour: "{{ build_vm_host_digit }}"
    month: "*"
    day: "1"
    job: "cd {{ install_dir }} && ./build_image.sh"

I don’t want all these jobs to kick off at once, so I had the start hour vary based on the node, from 1am to 3am, and then the minute vary by the template number multiplied by 5 so each job should kick off in 5 minute intervals. If I end up with more than 12 (11?) templates I’ll have to revise this, but that’s a problem for future me.

I’m not 100% sure this will work, the crontab entries look fine, but I never totally trust a cron job until I’ve seen it execute at least once. Fortunately I’m writing this on the 30th of the month so I’ll be able to check back and see soon enough.

Conclusion and next steps

In this post I (over)designed a way to create Virtual Machines on my proxmox cluster for a variety of Linux distributions. To recap: so far in this series I have selected and acquired hardware, done a basic proxmox install on that hardware and clustered it, configured the underlying proxmox environment using ansible, created a number of template VMs for various Linux distributions, and set up the basic tooling necessary to create VMs from those templates using code with Terraform.

With all of this done, surely it’s time to actually start doing things on my cluster rather than setting it up, right? Not so fast! There’s still one more piece of low level infrastructure I need to set up before I’m fully ready to deploy VMs to actually do things in this environment, and that’s to configure distributed storage on the 1TB SSDs I have in each node using ceph. I’ll cover that in the next post.