Migrating My Homelab from MicroK8s to Talos Linux on Proxmox: What Actually Broke

On September 24, 2026 I shut down the three MicroK8s VMs that had run my homelab for years and switched every workload over to a new Talos Linux cluster on the same Proxmox hosts. The cluster runs my GitOps-managed apps: observability, n8n, analytics, a few side projects, and the tools I use to write this blog. It has been in production for a few days now, including a rolling reboot of all three nodes.

The move itself was not hard. Talos is well documented, and talosctl is one of the better CLIs in the Kubernetes ecosystem. What took time were the small things that MicroK8s had been doing for me quietly, which I only noticed when they stopped: which IP the kubelet advertises, how CoreDNS resolves my own domains, which Ceph messenger protocol the kernel client speaks, and why a snapshot backup of a VM can freeze etcd.

This post covers the architecture I ended up with, the config I generated, and a checklist of every gotcha, so you can skip the evenings I spent on them. If you have never looked at Talos before, start with my Talos Linux guide. If you are still choosing an immutable OS, my Talos vs Bottlerocket vs Flatcar comparison covers that decision.

Why I Left MicroK8s

MicroK8s served me well. It is a single snap: snap install microk8s, enable a few addons, join the nodes, and you have a cluster. For a homelab that is hard to beat, and I would still recommend it to anyone who wants Kubernetes on one Ubuntu machine by this afternoon.

Over time, three problems added up:

The node OS was one more thing to look after. Every MicroK8s node was a full Ubuntu VM with SSH, apt updates, unattended-upgrades, snap refreshes and the odd kernel that needed a reboot. My package update script handled it, but every VM was a pet with its own drift.

Snap refreshes were something I had to plan around. Pinning a channel works, but the addons (DNS, ingress, MetalLB, hostpath storage) are MicroK8s’ own versions of those components, with their own defaults. When something broke, I was debugging MicroK8s’ packaging on top of Kubernetes itself.

Recovery was not declarative. If a node died, rebuilding it meant a long runbook: install Ubuntu, install the snap, enable the same addons, rejoin. Talos turns that into “apply this YAML to a fresh VM”.

Talos takes the opposite approach: no SSH, no shell, no package manager, a read-only root filesystem, and one API (talosctl) through which you configure everything. The whole node is a machine config file. For a homelab that I want to forget about between upgrades, that was the deciding factor.

Target Architecture

The Proxmox cluster has three physical hosts with Ceph running across them. The Talos layout mirrors that:

Three control-plane nodes that also run workloads is a deliberate homelab trade-off. Etcd gets quorum and survives losing one host, and I don’t spend RAM on dedicated workers that would sit mostly idle. In a production cluster I would split them. Here, removing the NoSchedule taint (allowSchedulingOnControlPlanes: true in the cluster config) is the pragmatic choice.

The VIP is one of my favourite Talos features. You declare a shared IP on the control-plane network interface, and Talos runs the election between the nodes: whichever node holds etcd leadership answers on the VIP. No keepalived, no HAProxy VM, no external load balancer for the API. The kubeconfig points at the VIP, and my reverse proxy points ingress traffic at the same address.

Generating the Machine Config

Talos config generation is one command plus patches. I keep the cluster secrets (the CA, the etcd CA, and the bootstrap tokens) in my password manager, not in git, and generate everything else from versioned patch files:

talosctl gen config homelab https://<VIP>:6443 \
  --with-secrets secrets.yaml \
  --config-patch @machine/patch-common.yaml \
  -o base

talosctl machineconfig patch base/controlplane.yaml \
  -p @machine/patch-node1.yaml \
  -p @machine/patch-nodeip.yaml \
  -p @machine/patch-metrics.yaml \
  -p @machine/patch-apiserver-sans.yaml \
  -p @machine/patch-etcd-tuning.yaml \
  -o node1.yaml

Each patch has one job:

• patch-common: settings shared by every node (VIP, scheduling on control planes, and so on).
• patch-nodeN: hostname and static addresses for that specific node.
• patch-nodeip: which subnet the kubelet and etcd may use as the node’s address. This one matters more than you would think (next section).
• patch-metrics: exposes controller-manager, scheduler and etcd metrics to Prometheus.
• patch-apiserver-sans: extra names in the API server certificate so the DNS name and the VIP both validate.
• patch-etcd-tuning: etcd settings adjusted for VMs whose disks are shared with everything else on the host.

Keeping secrets.yaml out of the repo while versioning the patches means I can rebuild any node’s config from scratch in under a minute. Without the secrets file, a regenerated config would create a brand-new cluster identity that the existing nodes reject.

Why the nodeIP patch is mandatory with two networks

This was my first real surprise. Each VM has two NICs: the LAN and the Ceph network. With no guidance, the kubelet picked the Ceph network address as the node’s InternalIP on some nodes. Everything looked healthy (pods scheduled, kubectl get nodes was all green), but NodePorts and the host-network ingress were only listening on the Ceph network, which my reverse proxy cannot reach.

The fix is to tell the kubelet which subnet is valid. While you’re at it, pin etcd’s advertised subnet too:

machine:
  kubelet:
    nodeIP:
      validSubnets:
        - 192.168.1.0/24
cluster:
  etcd:
    advertisedSubnets:
      - 192.168.1.0/24

If your nodes have more than one interface, apply this patch before you bootstrap. Changing etcd’s advertised address after the fact is a lot more painful than setting it up front.

The CPU Type Gotcha: x86-64-v3 for Ceph CSI

My VMs started with Proxmox’s default CPU model, and Talos booted and ran fine. Then the ceph-csi pods went into CrashLoopBackOff with no useful log output. The current ceph-csi images are built for a newer x86-64 microarchitecture level, and the default virtual CPU does not advertise the instructions they need.

Switching the VM CPU type to x86-64-v3 (or host, if all your Proxmox hosts have the same CPU generation and you don’t care about live migration between different CPUs) fixed it immediately. Set it when you create the VMs. Changing it later means a VM restart, which with three control-plane nodes is fine, but it is one more rolling operation you could have avoided.

Ceph CSI: RBD and CephFS Against Proxmox Ceph

I did not want a second storage system, so the cluster uses Proxmox’s existing Ceph through two Helm releases: ceph-csi-rbd and ceph-csi-cephfs. I deliberately kept the same pools and the same StorageClass names as on MicroK8s, with CephFS as the default class. That way the manifests in my GitOps repo work unchanged on both clusters, which made moving apps one by one much easier.

Two notes on this setup.

The CSI secrets are not in git. Each driver needs a Ceph user key: one each for the RBD provisioner, RBD node plugin, CephFS provisioner and CephFS node plugin. I create them out of band with ceph auth get-key client.<id> and a kubectl create secret. Sealing them would also work. I simply prefer that Ceph admin material never passes through the repo, even encrypted.

CephFS needs the v1 messenger port. RBD happily talks to the monitors on port 3300 (messenger v2). CephFS did not. Mounts failed with this error:

mount error: no mds is up or the cluster is laggy

The MDS was perfectly healthy. The problem was that the kernel CephFS client does not negotiate msgr v2 unless you pass the ms_mode mount option. With v2-only monitor addresses, it simply could not talk to the cluster. Listing the monitors on port 6789 (msgr v1) in the CephFS driver config fixed it. RBD kept using 3300.

If you see “no mds is up” and ceph -s says the MDS is active, check the monitor ports before you touch the MDS.

Migrating Persistent Volumes Without Copying Data

This was the part I was most nervous about, and it turned out to be the most elegant. Both clusters talk to the same Ceph pools, so the RBD images holding the data are already where they need to be. The only question is how to get the new cluster to use them.

The answer is static PersistentVolumes that reuse the original volumeHandle. The procedure for each app:

1. On MicroK8s, set the PV’s persistentVolumeReclaimPolicy to Retain, so deleting the claim never deletes the image.
2. Take a Ceph snapshot of the image as a safety net.
3. Scale the app to zero on MicroK8s.
4. Wait until rbd status <pool>/<image> shows no watchers. That confirms the old node has released the image.
5. In the GitOps repo, add a static PV on Talos with the same volumeHandle, the same CSI driver and Retain, plus a PVC bound to it by name.
6. Let ArgoCD sync the app on Talos.

A trimmed static PV looks like this:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: n8n-data
spec:
  capacity:
    storage: 10Gi
  accessModes: ["ReadWriteOnce"]
  persistentVolumeReclaimPolicy: Retain
  storageClassName: ceph-rbd
  claimRef:
    namespace: n8n
    name: n8n-data
  csi:
    driver: rbd.csi.ceph.com
    fsType: ext4
    volumeHandle: <original-volume-handle-from-microk8s>
    volumeAttributes:
      clusterID: <ceph-fsid>
      pool: <pool>
      imageFeatures: layering
      staticVolume: "true"
    nodeStageSecretRef:
      name: csi-rbd-secret
      namespace: ceph-csi-rbd

Step 4 is not optional. My RBD images do not have the exclusive-lock feature, so nothing stops two clusters from mapping the same image at the same time. That is a textbook way to corrupt a filesystem. The same reasoning is why the MicroK8s VMs are now shut down with autostart disabled, and why they must stay that way: if they booted, they would try to remount volumes that Talos now owns.

One thing outside Kubernetes also bit me: my reverse proxy had a few custom locations with the old node IPs hardcoded. The main proxy target pointed at the new VIP, but a /api location on one host still went to a MicroK8s node. Search your proxy config for the old addresses before you shut anything down.

Ingress: Host Network, the public Class, and a 308 Loop

On MicroK8s, the ingress addon gives you an nginx controller with an IngressClass called public. All my Ingress manifests referenced it. Rather than rewriting every app, I installed ingress-nginx on Talos as a DaemonSet on the host network and named its IngressClass public as well. Ports 80 and 443 are open on all three nodes and therefore on the VIP, which is the only address my external reverse proxy needs to know.

Two gotchas came up.

Pod Security Admission blocks it by default. Talos enforces the baseline Pod Security Standard on every namespace. Host network and host ports violate it, so the controller pods were rejected until I labelled the namespace:

kubectl label namespace ingress \
  pod-security.kubernetes.io/enforce=privileged

Only do this on namespaces that really need it. This default is one of the things Talos gets right.

TLS terminated upstream causes a 308 redirect loop. My reverse proxy terminates TLS and forwards plain HTTP to the cluster. For any Ingress with a tls: section, ingress-nginx saw an HTTP request and answered with a 308 redirect to HTTPS. The proxy followed it, sent HTTP again, and so on. The fix is to make the controller trust the X-Forwarded-Proto header from the proxy:

controller:
  config:
    use-forwarded-headers: "true"

On a vanilla chart you have to say it explicitly.

I know ingress-nginx has been retired upstream, and it was a conscious choice to keep it for this migration. Changing the operating system, the storage path and the ingress controller in the same weekend is how you end up not knowing which change broke what. Gateway API is the next project. I wrote about how I approach that kind of move in lessons from the ingress-nginx migration, and the ingress-nginx alternatives post covers the options.

ArgoCD and Sealed Secrets: Reusing the Same Repo

I had one goal for GitOps: both clusters must be able to run from the same repository during the transition, so I could move apps one at a time and roll back by scaling down on one side and up on the other.

That turned out to be easy because every Application’s destination is https://kubernetes.default.svc, the in-cluster API, which means “whichever cluster this ArgoCD runs in”. I installed ArgoCD v3.3 from the official manifests on Talos, copied over argocd-cm, argocd-rbac-cm, argocd-secret and the repository credentials, and applied the same Application manifests. Apps that were still scaled to zero on MicroK8s synced normally on Talos. If you are new to ArgoCD, my ArgoCD guide covers the concepts I’m relying on here.

Sealed secrets need the old keys. Every SealedSecret in the repo was encrypted with the MicroK8s controller’s keys. A fresh controller on Talos would generate new keys and fail to decrypt all of them. I exported the existing key secrets (there were five, because the controller rotates keys), applied them in kube-system on Talos, and then installed the controller. It picked them up and decrypted everything on first sync.

The rotation deserves attention. sealed-secrets generates a new key every 30 days by default. Your offline backup of the keys therefore goes stale month by month. If you ever restore from it, anything sealed with a newer key will not decrypt. Re-exporting the keys periodically is now part of my runbook. For more on how secrets end up exposed (or not) inside a cluster, see discovering secrets in Kubernetes.

Manual Endpoints, because ArgoCD excludes them. My argocd-cm has a resource.exclusions rule for Endpoints and EndpointSlice, which is common and sensible: they’re usually generated by Kubernetes and would otherwise flood ArgoCD with noise. The side effect is that ArgoCD also ignores the hand-written Endpoints for Services without a selector. I use those to point Prometheus at targets outside the cluster (the NAS, an image proxy) and at control-plane components. After the migration, those ServiceMonitors had no targets. I now apply those Endpoints manually, and they’re documented in the cluster README so future me doesn’t lose another hour.

Control-Plane Metrics on Talos

kube-prometheus-stack expects to scrape the controller-manager, the scheduler and etcd. On Talos those components bind to localhost by default, so the targets show as down. The metrics patch fixes that:

cluster:
  controllerManager:
    extraArgs:
      bind-address: 0.0.0.0
  scheduler:
    extraArgs:
      bind-address: 0.0.0.0
  etcd:
    extraArgs:
      listen-metrics-urls: http://0.0.0.0:2381

There is a subtlety. Talos applied the controller-manager and scheduler changes live, but etcd only picked up the new metrics listener after a node reboot. For a while two of three etcd targets were up and one was down, which looked like a network problem until I realised only the rebooted node had the change. If you apply this patch to a running cluster, plan a rolling reboot.

CoreDNS: talosctl upgrade-k8s Overwrites Your Changes

MicroK8s’ DNS addon had a rewrite that sent all of my *.<my-domain> names to the reverse proxy’s LAN address. I had forgotten it existed. On Talos, one app could not reach another service through its public hostname, because my local DNS resolves that name to a VPN address that pods cannot route to.

The fix was a small hosts block in the CoreDNS Corefile that maps that hostname to the reverse proxy’s LAN IP. The catch: talosctl upgrade-k8s regenerates the CoreDNS ConfigMap, so the next Kubernetes upgrade would quietly remove the change. I now keep the live Corefile in the repo, and re-applying it is a step in my upgrade runbook. The cleaner long-term fix is to disable Talos’ built-in CoreDNS management and manage CoreDNS yourself with Helm, but for one hosts entry the runbook step is enough.

Backups: Don’t Snapshot Talos VMs, Snapshot etcd

This was the scariest incident of the migration. My Proxmox backup job ran in snapshot mode on the new Talos VMs like on every other VM. During the backup, the guest’s disk writes stalled long enough for etcd to start missing heartbeats. The API server became unresponsive, and the nodes did not recover cleanly until I rebooted them.

The lesson is that Talos nodes have no state worth backing up at the VM level. The OS is an immutable image, and the config is a YAML file you can regenerate. The only state that matters is etcd (plus the persistent volumes, which live in Ceph and have their own snapshots). So:

• The Talos VMs are excluded from the Proxmox backup job.
• A script on one of the Proxmox hosts runs talosctl etcd snapshot every night and keeps 14 days of snapshots on the NAS.
• A Prometheus alert fires if the snapshots stop appearing.

Restoring is one command against a freshly installed control-plane node:

talosctl -n <node> bootstrap --recover-from=./etcd-snapshot.db

Rehearse that restore on a throwaway VM before you need it. A backup you have never restored is only a hope.

MicroK8s vs Talos: How They Compare in Practice

The biggest change in daily work is debugging without SSH. My first reaction was to look for a way in, and there isn’t one. A few days in, I don’t miss it. talosctl dmesg, talosctl logs kubelet, talosctl dashboard and kubectl debug node/<name> (for the rare case where you need tools on the host) cover everything I have needed so far.

Gotchas Checklist

If you’re planning the same move, this is the list I wish I’d had:

• Set nodeIP.validSubnets (and etcd advertisedSubnets) before bootstrap if your VMs have more than one NIC.
• Use CPU type x86-64-v3 or newer on the Proxmox VMs, or ceph-csi will crash-loop.
• Point CephFS at the monitors on port 6789 (msgr v1) unless you set ms_mode. “no mds is up” is usually a protocol problem, not an MDS problem.
• Keep StorageClass and IngressClass names identical to the old cluster so manifests work in both during the transition.
• Migrate RBD volumes with static PVs reusing the volumeHandle, with Retain on both sides, and wait for zero watchers before mounting on the new cluster.
• Keep the old cluster off for good once its volumes have moved. Disable autostart.
• Grep your reverse proxy for the old node IPs, including custom locations.
• Label the ingress namespace privileged if the controller uses host networking.
• Enable use-forwarded-headers when TLS terminates in front of ingress-nginx.
• Copy the sealed-secrets keys before installing the controller, and re-export them after each 30-day rotation.
• Re-create hand-written Endpoints if ArgoCD excludes Endpoints from sync.
• Plan a rolling reboot after enabling etcd metrics.
• Keep your CoreDNS customisations in git and re-apply them after every talosctl upgrade-k8s.
• Exclude Talos VMs from VM-level snapshot backups and back up etcd with talosctl etcd snapshot instead. Rehearse the restore.
• Store secrets.yaml safely: it is the cluster’s identity. The rest of the config can be regenerated.

Frequently Asked Questions

Conclusion

Moving from MicroK8s to Talos Linux on Proxmox took a weekend of real work and a few evenings of gotchas. Almost none of those gotchas were Talos bugs. They were assumptions from the old cluster: which interface is the node’s address, what DNS rewrites exist, which Ceph protocol a client speaks, and whether a VM backup is harmless. Talos makes those assumptions visible because it does nothing you haven’t declared.

The result is a cluster where every node is a YAML file and a boot image, upgrades are a single talosctl command, and the only thing I really need to back up is etcd. Next up is replacing ingress-nginx with Gateway API, now that the platform underneath is no longer the variable.

If you’re planning a similar migration, start with the checklist above, keep both clusters able to run from the same GitOps repo, and move one app at a time.