Introduction

While messing around with setting up storage I learned a couple other neat things about setting up my cluster that don’t really fit elsewhere, but I don’t want to lose them.

Both of these tips come from checking out kevindurb’s repo so big shoutout to him.

Patches for talconfig

In all the docs and other repos I’d seen before this point patches were added as strings inside the talhelper config. So I had sections looking like this:

patches:
  - |-
    cluster:
      apiServer:
        admissionControl:
          - name: PodSecurity
            configuration:
              apiVersion: pod-security.admission.config.k8s.io/v1beta1
              kind: PodSecurityConfiguration
              exemptions:
                namespaces:
                  - openebs
      proxy:
        disabled: true
controlPlane:
  nodeLabels:
    openebs.io/engine: "mayastor"
  patches:
    - |-
      machine:
        sysctls:
          vm.nr_hugepages: "1024"
    - |-
      - op: add
        path: /machine/kubelet/extraMounts
        value:
          - destination: /var/openebs/local
            type: bind
            source: /var/openebs/local
            options:
              - bind
              - rshared
              - rw

It works, but it looks gross and you don’t get any nice yaml syntax highlighting in your editor.

What I learned you can do instead is make individual patch yaml files (similar to what I was doing before switching to talhelper) and then point to them. So instead of the above I had patch files like

cluster:
  apiServer:
    admissionControl:
      - name: PodSecurity
        configuration:
          apiVersion: pod-security.admission.config.k8s.io/v1beta1
          kind: PodSecurityConfiguration
          exemptions:
            namespaces:
              - openebs

and

machine:
  sysctls:
    vm.nr_hugepages: "1024"
  nodeLabels:
    openebs.io/engine: "mayastor"

and

machine:
  kubelet:
    extraMounts:
      - destination: /var/local
        type: bind
        source: /var/local
        options:
          - bind
          - rshared
          - rw

That I could then reference in talconfig.yaml:

controlPlane:
  patches:
    - "@./patches/mayastor-cp.yaml"
    - "@./patches/mayastor.yaml"
    - "@./patches/var-local.yaml"

This cleans things up considerably, and also makes it easier to keep track of what things are doing what. For instance I can combine the hugepages config and nodelabel configs into a mayastor patch that I know to apply on any node that will be running a mayastor disk pool. I didn’t end up going with mayastor for storage but this idea will carry over.

Combining helm apps with custom resources

For several of my apps I’ve wanted to install the base app with helm, but then have also had to install custom resources to go with them. Things like a ClusterSecretStore to go along with external secrets.

The way I’ve done this to date is to make a custom helm chart and list the parent app as a dependency. This works, but it feels a bit heavy for specifying a chart and then a couple extra resources. Instead I learned that I can just give my argo app multiple sources and specify my repo pointing to a resources folder as well as a helm source along with the custom values for that chart. Again using openebs as an example, it looks something like this:

  sources:
    - repoURL: https://openebs.github.io/openebs
      targetRevision: 4.2.0
      chart: openebs
      helm:
        valuesObject:
          localpv-provisioner:
            localpv:
              basePath: "/var/local/openebs"
            hostpathClass:
              enabled: true
              name: openebs-hostpath
              isDefaultClass: false
              basePath: "/var/local/openebs"
          mayastor:
            etcd:
              localpvScConfig:
                basePath: "/var/local/openebs"
            loki-stack:
              localpvScConfig:
                basePath: "/var/local/openebs"
            io_engine:
              envcontext: "iova-mode=pa"
            csi:
              node:
                initContainers:
                  enabled: false
          engines:
            local:
              lvm:
                enabled: false
              zfs:
                enabled: false
    - repoURL: https://github.com/ianepreston/homelab.git
      path: k8s/dev/storage/openebs/resources
      targetRevision: HEAD

Not too shabby. I’m not going to bother converting my existing apps to this pattern, but I’ll keep it in mind for future ones for sure.