Introduction

The next cluster component I need available before I can host actually useful workloads is persistent storage. The talos docs have sections for local and replicated with some recommendations and specific suggestions. I know I want both local storage to handle services that do their own replication (such as databases) as well as replicated storage for all the other services I run that don’t natively replicate storage. In this post I’ll document what storage providers I tested, the experience of configuring them, and their performance.

tldr

For replicated storage I’m going to go with longhorn, and for local I’m going to go with local-path-provisioner. Performance on replicated storage varied a bit between the engines I tested, but I’m pretty obviously bottlenecked by the 1gbps link I have for my nodes. At a future date I might look into putting 2.5gbps NICs in them and getting an updated switch, but that’s a project for another time.

How I benchmarked

I found this repo that has a spec for running storage benchmarks. It looked pretty good and I don’t know enough about storage benchmarking to do better so I went with it. Here’s an example benchmark job I created to test a storage class:

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: dbench-pv-claim
spec:
  storageClassName: local-path
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
---
apiVersion: batch/v1
kind: Job
metadata:
  name: dbench
spec:
  template:
    spec:
      containers:
        - name: dbench
          image: zayashv/dbench:latest
          imagePullPolicy: Always
          env:
            - name: DBENCH_MOUNTPOINT
              value: /data
              # - name: DBENCH_QUICK
              #   value: "yes"
              # - name: FIO_SIZE
              #   value: 1G
              # - name: FIO_OFFSET_INCREMENT
              #   value: 256M
              # - name: FIO_DIRECT
              #   value: "0"
          volumeMounts:
            - name: dbench-pv
              mountPath: /data
      restartPolicy: Never
      volumes:
        - name: dbench-pv
          persistentVolumeClaim:
            claimName: dbench-pv-claim
  backoffLimit: 4

Very straightforward, make a PersistentVolumeClaim using the storage class you care about, make a job that attaches the dbench container to that pvc.

From there I can apply the file and then check out the results of the job with kubectl logs jobs/dbench -f. After I’m done I delete the resource and deploy a similar manifest to test another type of storage. Not super sophisticated but it worked for what I needed.

Local storage

As both a baseline and because I need it I wanted to set up local path persistent storage to start. The Talos docs recommended local path provisioner so I tried it first. The installation from following the docs in talos (you need to do a couple talos specific installations) were easy to follow and when I was done I had a working storage class. Here’s the summary of the dbench results:

==================
= Dbench Summary =
==================
Random Read/Write IOPS: 174k/103k. BW: 1831MiB/s / 2467MiB/s
Average Latency (usec) Read/Write: 66.86/43.63
Sequential Read/Write: 3193MiB/s / 2835MiB/s
Mixed Random Read/Write IOPS: 77.8k/25.1k

This is nice and fast, how fast exactly we’ll see when we start comparing it to other storage classes.

NAS storage

Not everything the cluster accesses will be stored on it. I’ve got a Synology NAS full of big spinning disks so for anything large I’ll want to use that. Also, spinning disks outside the cluster seems like a good other end of the spectrum for performance compared to the local path on nvme drives I just tested.

I created a shared folder for storage testing on my NAS and added it as a pvc:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: nfs-pv
spec:
  capacity:
    storage: 10Gi # Size is arbitrary for NFS, adjust as needed
  accessModes:
    - ReadWriteMany # RWX for multiple pods
  persistentVolumeReclaimPolicy: Retain # Keeps data after PVC deletion
  storageClassName: nfs # Optional, define if using dynamic provisioning later
  mountOptions:
    - hard
    - nfsvers=4.1 # Adjust NFS version as needed
  nfs:
    server: 192.168.10.3 # Your NFS server IP or hostname
    path: /volume1/storagetesting # Specific folder in the NFS share

Then ran dbench:

==================
= Dbench Summary =
==================
Random Read/Write IOPS: 24.4k/15.6k. BW: 110MiB/s / 110MiB/s
Average Latency (usec) Read/Write: 588.77/528.61
Sequential Read/Write: 111MiB/s / 111MiB/s
Mixed Random Read/Write IOPS: 17.5k/5806

Dang, yup, that’s slow. Right around gigabit speed so to be expected, but still, slow.

Longhorn

The talos docs basically just say don’t use longhorn because it uses iscsi and that’s old. I imagine in a really production focused high performance environment that would be a valid critique. For my home lab where storage isn’t really my main focus and convenience and user friendliness matter maybe it’s not such a big deal.

The longhorn docs do have a talos specific install guide that will probably be helpful. It also has a guide for installation with argocd that I’ll refer to. I already did the talos specific configs a wile back before I got sidetracked by upgrading and a bunch of other challenges. So I can just focus on the argo stuff at this point.

I threw the helm chart in an app, added a manifest for the namespace that had a label for pod-security.kubernetes.io/enforce: privileged as well as an Ingress for Traefik so I could load the web UI.

Besides some stupid issues where I hadn’t applied the talos patches that I thought I had the install went fine.

Uninstalling

Quick note because this caught me out, if you want to remove longhorn, don’t just remove the app from argo. Follow the docs otherwise you will be in a horribly broken state where the namespace can’t be deleted because it’s wiped some of the pods that are responsible for cleaning up CRDs… it’s a whole thing.

Performance

==================
= Dbench Summary =
==================
Random Read/Write IOPS: 13.1k/8955. BW: 154MiB/s / 54.2MiB/s
Average Latency (usec) Read/Write: 539.46/664.56
Sequential Read/Write: 151MiB/s / 56.2MiB/s
Mixed Random Read/Write IOPS: 9449/3144

Slower than the NAS on writes, faster on reads. That makes sense, it’s got to write over gigabit multiple times whereas the NAS only has to do it once. I’m disappointed reads aren’t faster, I wouldn’t have expected that performance to be so much worse, I guess maybe it has to do some network checks to make sure nodes aren’t writing before it reads or something? Anyway, that’s good to be aware of.

Openebs Mayastor

Oh man, I had such a bad time with installing this. Openebs has very inconsistent choices about where it expects to be able to find files for its local volumes (for handling its own etcd cluster etc). There’s a github issue discussing it. Combined with talos needing to configure the kubelet to mount paths that containers need to access this is a recipe for misconfiguration. I messed around with so many different configs on the helm chart for openebs and the config for my kubelet between /var/local and /var/openebs/local and /var/local/openebs. I did learn a ton of troubleshooting techniques following the logs of various pods back to ultimately discovering that one pod all the others depended on couldn’t start because it couldn’t attach to a volume. The worst part was as near as I can tell even after I updated my config, removed openebs and reinstalled it, it was still trying to use the old paths. Eventually I wiped my cluster and started fresh and was able to get it working after that. I don’t want to hold this against the project too much, I’m sure a lot of the problem is my inexperience, but I sure spent a lot of time banging my head against tweaking those configs.

Once I got it working the performance was similar to longhorn:

==================
= Dbench Summary =
==================
Random Read/Write IOPS: 29.9k/13.8k. BW: 161MiB/s / 56.4MiB/s
Average Latency (usec) Read/Write: 306.22/339.67
Sequential Read/Write: 172MiB/s / 59.2MiB/s
Mixed Random Read/Write IOPS: 19.4k/6457

Besides making me sad when I tried to configure it, openebs also requires 2 cores per node running at 100% polling for changes. Since I only have 4 cores in each of my nodes that would be 50% of my CPU dedicated to handling storage. Given that, the lack of nice web UI (I won’t configure things with it but it’s nice for monitoring in longhorn), and the pain I suffered configuring it, I will be passing on using openebs.