Kubernetes

Every Kubernetes cluster runs on Linux. But the distribution you choose for your nodes determines how much time you spend patching, hardening, debugging SSH sessions, and dealing with configuration drift across your fleet. General-purpose distributions like Ubuntu and Debian were designed to run anything: web servers, desktops, databases, and yes, Kubernetes. That flexibility is also their biggest liability when your only job is running containers.

Related reading: Talos vs Bottlerocket vs Flatcar.

Talos Linux takes a radically different approach. It strips away everything a Kubernetes node does not need: there is no shell, no SSH daemon, no package manager, and no way to log in interactively. The entire operating system is managed through an API, and every change is declarative. If that sounds extreme, it is. But it solves real problems that traditional distributions cannot address without layers of additional tooling.

This guide is a comprehensive deep dive into Talos Linux: what it is, how its architecture works, how it compares to alternatives like Flatcar and Bottlerocket, how to install and operate it, and when you should (and should not) use it. Whether you are evaluating Talos for a production fleet or a homelab, this is everything you need to make an informed decision.

What Is Talos Linux

Talos Linux is a minimal, immutable operating system designed exclusively to run Kubernetes. It is developed by Sidero Labs and distributed as a single system image that boots into a Kubernetes-ready state. There is no general-purpose userland. No bash shell. No ability to SSH into a node and run commands. Every aspect of machine configuration — from network settings to Kubernetes component flags — is expressed in a YAML document called the machine config and applied through an authenticated gRPC API.

The core design principles are:

• Immutable — The root filesystem is read-only and mounted from a SquashFS image. You cannot install packages, modify system binaries, or alter the OS at runtime.
• API-driven — All management happens through talosctl, a CLI that communicates with the Talos API over mutual TLS. There is no SSH and no interactive console.
• Minimal — The OS ships only what Kubernetes needs: a Linux kernel, containerd, the kubelet, etcd (on control plane nodes), and the Talos machinery. The installed image is roughly 80 MB.
• Declarative — The desired machine state is defined in a YAML config. Applying a new config converges the node to the desired state, similar to how Kubernetes reconciles workloads.
• Secure by default — No shell access means no attack vector through compromised credentials. All API communication requires mutual TLS authentication. The attack surface is drastically smaller than any traditional distribution.

Talos supports bare metal, VMware vSphere, AWS, Azure, GCP, Hetzner, Equinix Metal, Oracle Cloud, and several other platforms. It also runs on single-board computers like Raspberry Pi and NVIDIA Jetson, making it viable for edge deployments. For a broader perspective on how immutable infrastructure fits into the Kubernetes ecosystem, see our Kubernetes security best practices guide.

Architecture Deep Dive

Understanding Talos at an architectural level is essential before deploying it. The design choices are unconventional compared to what most Linux administrators expect, and they explain both its strengths and its constraints.

The machined Daemon and API-Driven Management

At the heart of Talos is machined, a single PID-1 process that replaces systemd, init, and every other service manager. When a Talos node boots, machined starts, reads its machine configuration, and orchestrates the entire lifecycle: networking, disk setup, containerd, the kubelet, and etcd (on control plane nodes).

machined exposes a gRPC API over port 50000 (for the trustd/machine API) and port 50001 (for the maintenance API during initial provisioning). This is the only way to interact with the node. The talosctl CLI is the primary client, authenticating with mutual TLS certificates generated during cluster bootstrapping.

Key API operations include:

• talosctl apply-config — Push a new or updated machine configuration.
• talosctl upgrade — Trigger an in-place OS upgrade.
• talosctl dmesg — Stream kernel messages in real time.
• talosctl logs — Read logs from any Talos service (etcd, kubelet, containerd).
• talosctl get — Inspect resource state (network interfaces, disks, services).
• talosctl reset — Wipe a node and return it to maintenance mode.

This API-first model eliminates configuration drift by design. There is no way for an operator to SSH into a node, run an ad-hoc command, and leave the system in an undocumented state. Every change flows through the same declarative path.

System Partitions Layout

Talos partitions the disk into a well-defined layout that separates immutable system data from mutable state:

The STATE partition is critical: it persists the machine config and TLS certificates across reboots and upgrades. The EPHEMERAL partition holds everything that can be reconstructed — container images, pod volumes (emptyDir), and etcd data on control plane nodes. When you run talosctl reset, the EPHEMERAL partition is wiped, but STATE can optionally be preserved.

This layout means that an OS upgrade replaces the BOOT partition contents (kernel + initramfs) while leaving your machine configuration and Kubernetes state untouched. If an upgrade fails, Talos rolls back to the previous BOOT image automatically.

Boot Process and Kubernetes Bootstrapping

The Talos boot sequence is deterministic and fast, typically completing in under 60 seconds on modern hardware:

1. Firmware → Bootloader — UEFI or BIOS loads GRUB, which loads the Talos kernel and initramfs.
2. Kernel init → machined — The kernel starts machined as PID 1. There is no init system in between.
3. Machine config discovery — machined checks the STATE partition for an existing config. If none is found (first boot), it enters maintenance mode and listens on the maintenance API for a config to be applied.
4. Network configuration — Networking is brought up based on the machine config (DHCP or static).
5. Disk setup — Partitions are created or validated. The EPHEMERAL partition is formatted if missing.
6. containerd starts — The container runtime is launched.
7. etcd starts (control plane only) — etcd is started and joins the existing cluster, or waits for a bootstrap command.
8. kubelet starts — The kubelet registers the node with the Kubernetes API server.

The first control plane node requires a one-time bootstrap command (talosctl bootstrap) to initialize the etcd cluster and generate the Kubernetes control plane static pods. Subsequent control plane nodes join automatically.

Security Model: No SSH, Mutual TLS, API-Only

Talos Linux implements a zero-trust security model at the OS level. Every API request is authenticated using mutual TLS (mTLS). When you generate a cluster configuration with talosctl gen config, it produces a Certificate Authority (CA) that signs both the client (operator) and server (node) certificates.

The security implications are significant:

• No shell access — There is no /bin/sh, no /bin/bash, no login capability. Even if an attacker gains network access to the node, there is no shell to exploit.
• No SSH daemon — Port 22 is not open. There is no sshd binary on the system.
• No package manager — You cannot install tools, backdoors, or persistence mechanisms on the host.
• Read-only rootfs — Even with theoretical root access, the filesystem cannot be modified.
• Mutual TLS everywhere — The Talos API, etcd communication, and inter-node trust all use mTLS. Certificates can be rotated without downtime.

This does not make Talos invulnerable — kernel exploits and container escape vulnerabilities still apply. But it eliminates the most common attack vectors in Kubernetes node compromise: SSH credential theft, unauthorized package installation, and persistent rootkits.

Talos Linux vs Alternatives: Comparison Table

Choosing a node OS depends on your operational model, cloud provider, and team experience. Here is how Talos Linux compares to the most common alternatives for Kubernetes node operating systems.

Talos Linux vs Ubuntu / Debian

Ubuntu and Debian are the default choices for most Kubernetes deployments, especially when using kubeadm or managed installers. They work. But they carry everything a general-purpose OS includes: a package manager, a full shell, hundreds of system services, and thousands of binaries that your Kubernetes nodes never use.

The operational burden is real: you need to patch the OS independently from Kubernetes, harden SSH, configure unattended upgrades, manage user accounts, and run CIS benchmarks to verify compliance. With Talos, these concerns disappear because the attack surface simply does not exist. The trade-off is that you lose the ability to SSH in and debug problems the traditional way.

Talos Linux vs Flatcar Container Linux

Flatcar Container Linux (the successor to CoreOS Container Linux) is the closest philosophical match to Talos. Both use immutable root filesystems and image-based updates. However, Flatcar retains SSH access and a full shell, which means an operator can still log in and make ad-hoc changes. Flatcar uses Ignition for initial provisioning and systemd for service management.

The key difference is that Flatcar is a container-optimized general-purpose OS, while Talos is a Kubernetes-only OS. Flatcar can run arbitrary containers and system services. Talos runs only Kubernetes. If you need SSH as a safety net during your transition to immutable infrastructure, Flatcar is a pragmatic middle ground. If you want to enforce immutability with no escape hatches, Talos is the stronger choice.

Talos Linux vs Bottlerocket

Bottlerocket is AWS’s purpose-built container OS, designed for EKS and ECS. Like Talos, it has an immutable rootfs and an API-driven settings model. Unlike Talos, it provides an optional “admin container” that gives you a shell for debugging, and it is heavily optimized for the AWS ecosystem.

If you run exclusively on AWS with EKS, Bottlerocket is the path of least resistance. If you need a multi-cloud or bare-metal solution with integrated Kubernetes bootstrapping, Talos is significantly more flexible. Bottlerocket also does not bootstrap Kubernetes itself — it relies on EKS or an external installer.

Talos Linux vs RancherOS / k3OS

RancherOS and k3OS were early attempts at minimal container-focused Linux distributions. RancherOS ran the entire system as Docker containers. k3OS bundled k3s (lightweight Kubernetes) into the OS. Both projects have been deprecated or are in maintenance mode. Talos is the actively developed, production-grade successor to this category. If you are currently running k3OS, Talos is the natural migration path.

Installation and Cluster Bootstrap

Setting up a Talos cluster follows a consistent workflow regardless of the platform: generate configs, boot nodes, apply configs, bootstrap. Here is a step-by-step walkthrough.

Step 1: Install talosctl

Download the talosctl binary for your platform. On macOS with Homebrew:

brew install siderolabs/tap/talosctl

On Linux:

curl -sL https://talos.dev/install | sh

Step 2: Generate Machine Configurations

The talosctl gen config command generates a full set of machine configurations: one for control plane nodes, one for workers, and a talosconfig file containing the client credentials.

talosctl gen config my-cluster https://10.0.0.10:6443 
  --output-dir _out

This creates three files in the _out directory:

• controlplane.yaml — Machine config for control plane nodes.
• worker.yaml — Machine config for worker nodes.
• talosconfig — Client configuration with the CA certificate and client key for mTLS authentication.

The endpoint URL (https://10.0.0.10:6443) should point to the Kubernetes API server address — either a load balancer VIP or the IP of your first control plane node.

Step 3: Boot Nodes with Talos

How you boot depends on the platform:

• Bare metal — Write the Talos ISO or disk image to a USB drive or PXE boot. The node boots into maintenance mode, waiting for a config.
• VMware — Deploy the OVA template, or use the ISO in a VM. Talos provides official OVA images.
• AWS — Use the official Talos AMI. Launch EC2 instances with the AMI and pass the machine config as user-data.
• Azure / GCP — Use the official images from Sidero Labs’ image factory. Pass the machine config through the platform’s metadata service.

Step 4: Apply Configuration and Bootstrap

Once nodes are booted and in maintenance mode, apply the machine configs:

# Configure talosctl to use the generated credentials
export TALOSCONFIG="_out/talosconfig"

# Apply config to the first control plane node
talosctl apply-config --insecure 
  --nodes 10.0.0.10 
  --file _out/controlplane.yaml

# Apply config to worker nodes
talosctl apply-config --insecure 
  --nodes 10.0.0.20 
  --file _out/worker.yaml

The --insecure flag is required for the initial config application because the node does not yet have TLS certificates. After the config is applied, all subsequent communication uses mTLS.

Now bootstrap the Kubernetes cluster from the first control plane node:

# Set the endpoint and node
talosctl config endpoint 10.0.0.10
talosctl config node 10.0.0.10

# Bootstrap etcd and the control plane
talosctl bootstrap

This command initializes etcd, generates the Kubernetes PKI, and starts the control plane static pods. Within a minute or two, the Kubernetes API server is available.

Step 5: Retrieve kubeconfig and Verify

# Get the kubeconfig
talosctl kubeconfig -n 10.0.0.10

# Verify the cluster
kubectl get nodes
kubectl get pods -A

Essential talosctl Commands

Once the cluster is running, these are the commands you will use daily:

# Check node health
talosctl health --nodes 10.0.0.10

# Stream kernel messages (equivalent to dmesg -w)
talosctl dmesg --nodes 10.0.0.10 --follow

# View service logs
talosctl logs kubelet --nodes 10.0.0.10
talosctl logs etcd --nodes 10.0.0.10

# List running services
talosctl services --nodes 10.0.0.10

# Get machine config (current running config)
talosctl get machineconfig --nodes 10.0.0.10

# Inspect resource state
talosctl get members --nodes 10.0.0.10
talosctl get addresses --nodes 10.0.0.10

Day-2 Operations

Installation is only the beginning. The real value of Talos emerges in day-2 operations: upgrades, config changes, and cluster maintenance. This is where the declarative, API-driven model pays dividends.

Upgrading Talos Linux

Talos upgrades are performed node by node through the API. The process downloads the new OS image, writes it to the inactive boot partition, and reboots the node into the new version. If the upgrade fails, the node automatically rolls back to the previous image.

# Upgrade a single node
talosctl upgrade --nodes 10.0.0.10 
  --image ghcr.io/siderolabs/installer:v1.9.0

# Upgrade with --preserve to keep the EPHEMERAL partition
talosctl upgrade --nodes 10.0.0.10 
  --image ghcr.io/siderolabs/installer:v1.9.0 
  --preserve

For production clusters, follow this sequence: upgrade control plane nodes one at a time, verify etcd health after each, then upgrade workers in a rolling fashion. The --preserve flag is important if you want to keep downloaded container images and avoid re-pulling everything after the reboot.

Upgrading Kubernetes Version

Kubernetes version upgrades are separate from Talos OS upgrades. You can run a newer version of Kubernetes on an older Talos release (within compatibility bounds). The upgrade is triggered through talosctl:

talosctl upgrade-k8s --nodes 10.0.0.10 
  --to 1.31.0

This command orchestrates the upgrade of all control plane components (kube-apiserver, kube-controller-manager, kube-scheduler, kube-proxy) and then rolls the kubelet version across all nodes. It respects PodDisruptionBudgets and cordons/drains nodes before upgrading.

Customizing Machine Config with Patches

As your cluster evolves, you will need to modify machine configurations — adding a registry mirror, changing kubelet flags, or configuring network bonding. Talos supports config patches that overlay changes onto the base config without replacing the entire file.

# Create a patch file
cat > kubelet-patch.yaml << 'EOF'
machine:
  kubelet:
    extraArgs:
      max-pods: "150"
    extraMounts:
      - destination: /var/local-storage
        type: bind
        source: /var/local-storage
        options:
          - bind
          - rw
EOF

# Apply the patch
talosctl apply-config --nodes 10.0.0.20 
  --config-patch @kubelet-patch.yaml

Patches can also be applied at generation time with talosctl gen config --config-patch, which is ideal for encoding environment-specific overrides into your GitOps pipeline.

etcd Management

Talos manages etcd as a first-class service, not as a manually deployed component. Common etcd operations are available through talosctl:

# Check etcd member list
talosctl etcd members --nodes 10.0.0.10

# Take an etcd snapshot (backup)
talosctl etcd snapshot db.snapshot --nodes 10.0.0.10

# Remove a failed etcd member
talosctl etcd remove-member --nodes 10.0.0.10 

# Force a new etcd cluster from a single node (disaster recovery)
talosctl etcd forfeit-leadership --nodes 10.0.0.10

Regular etcd snapshots are non-negotiable for any production cluster. Automate this with a CronJob that calls the Talos API or runs talosctl etcd snapshot from an external host.

Limitations and When NOT to Use Talos Linux

Talos is not the right choice for every environment. Understanding its limitations is just as important as understanding its strengths.

No SSH Debugging

The most immediate pain point: when something goes wrong, you cannot SSH into the node and poke around. You are limited to what the Talos API exposes — logs, dmesg, service status, and resource state. For most Kubernetes issues, this is sufficient. But for low-level kernel or hardware debugging, you may need to boot the node from a different OS temporarily.

Talos does offer a talosctl dashboard command that provides a real-time TUI (text UI) showing CPU, memory, network, and service status. Combined with talosctl logs and talosctl dmesg, you can troubleshoot most problems. But the learning curve is real, especially for teams accustomed to reaching for htop and journalctl.

Learning Curve for Traditional Sysadmins

If your team manages infrastructure through SSH, Ansible playbooks, and shell scripts, Talos requires a fundamental shift in operational practices. There is no way to "just install" a debugging tool on a node. Everything must be done through the API or through Kubernetes workloads (DaemonSets with host-level access). This shift is valuable in the long run, but it requires investment in training and new workflows.

Custom Kernel Modules

Talos ships a specific kernel build with a curated set of modules. If your workload requires a custom kernel module (GPU drivers, specific storage drivers, or out-of-tree network drivers), you need to build a custom Talos image using the Talos image factory or the imager tool. This is supported but adds operational complexity compared to distributions where you can simply apt install a kernel module package.

Sidero Labs provides an Image Factory service that lets you build custom Talos images with additional system extensions (like NVIDIA drivers, iSCSI tools, or ZFS support) through a web interface or API.

Workloads Requiring Host-Level Access

Some workloads expect to interact with the host OS directly: log collectors that read /var/log, monitoring agents that read /proc, or security tools that install kernel modules. Most of these work in Talos (containerd's runtime allows host path mounts), but some assume a traditional Linux userland that simply does not exist. Evaluate your specific stack before committing.

Real-World Use Cases

Homelab and Learning

Talos is an excellent choice for homelab Kubernetes clusters. It runs on Raspberry Pi 4/5, Intel NUCs, and old laptops. The entire OS fits in minimal storage, and the declarative config model means you can rebuild your cluster from scratch in minutes by reapplying your machine configs. Many homelab operators use Talos with ArgoCD or Flux for a fully GitOps-managed stack.

Edge and Retail

Edge deployments benefit from Talos's small footprint, immutable design, and remote management. A retail chain with 500 store locations running local Kubernetes clusters can manage every node through the Talos API without ever needing physical or SSH access. The A/B upgrade mechanism ensures that a bad update does not brick a remote device.

Production Multi-Cloud Clusters

Talos provides a consistent node OS across AWS, Azure, GCP, and bare metal. This is valuable for organizations that run Kubernetes on multiple providers and want a single operational model for node management. Instead of maintaining separate AMIs, Azure images, and GCP images with different toolchains, you maintain one set of Talos machine configs with platform-specific patches.

Security-Sensitive Environments

For regulated industries (finance, healthcare, government), Talos's security posture simplifies compliance. The absence of SSH, shell, and package management eliminates entire categories of CIS benchmark requirements. Audit teams appreciate that there is no way for a rogue operator to install unauthorized software on the node OS. The immutable image model also simplifies forensics: if the OS hash does not match the known-good image, the node has been tampered with.

Frequently Asked Questions