Design decisions
Why it is built this way.
Every part of the lab had an easier or more popular alternative. These are the choices that shaped it, what each one was chosen over, and what it costs. Each one is written down in the lab's repository before anything is built.
Backups, not replicas
- Chosen
- ZFS on the host, with a mirror for irreplaceable data and backups that leave the house.
- Instead of
- Ceph or Longhorn replicating across the nodes.
- Why
- On one server every replica lands on the same disks. That triples the writes and adds no durability, because the failure it is meant to survive takes all copies at once.
- The price
- A dead disk pair means a restore, not a failover. The restore is tested so that this stays an inconvenience.
No redundancy for media, on purpose
- Chosen
- One large single disk for films and series, kept apart from everything else.
- Instead of
- Mirroring the media library too.
- Why
- Media can be fetched again. Photos and documents cannot. Paying for a second large disk protects the part that matters least.
- The price
- If that disk dies, the library has to be rebuilt by hand. Nothing irreplaceable is on it.
The manager never runs on what it manages
- Chosen
- Sidero Omni in its own virtual machine beside the cluster, with a login that does not depend on the cluster.
- Instead of
- Running the manager inside the cluster, or signing in to it through the cluster's own identity provider.
- Why
- Recovering the cluster must never require the cluster. When every node is down, the tool that rebuilds them still has to start and still has to let me in.
- The price
- One more machine to keep patched, and one more login to look after.
Disks for pods through a narrow key
- Chosen
- Proxmox CSI, which only holds a scoped token that can create and attach disks.
- Instead of
- A storage driver that logs in to the host as root over SSH.
- Why
- The cluster faces the internet. A break-in there should not come with root access to the server underneath everything.
- The price
- No volume snapshots inside Kubernetes. Volumes are copied out as encrypted backups instead.
Kubernetes, though Compose would do
- Chosen
- Kubernetes for every app in the lab.
- Instead of
- Docker Compose on a single machine.
- Why
- Compose would run every service here with a fraction of the effort. Running Kubernetes properly is the point of the lab, so it gets the harder tool on purpose.
- The price
- More moving parts than a family needs. Each one has to earn its place on the stack map.
An operating system with no shell
- Chosen
- Talos Linux: immutable, configured through an API, with no SSH and no package manager.
- Instead of
- k3s or kubeadm on a general-purpose Linux.
- Why
- Nodes that serve the internet should have as little on them as possible. Upgrades become image swaps, and the configuration lives in a file, not in shell history.
- The price
- No quick fix by logging in. Every change goes through the configuration, even at two in the morning.
One door, with a login in front
- Chosen
- Envoy Gateway on the Gateway API, which asks Authentik before a request reaches the apps behind it.
- Instead of
- ingress-nginx, and Cilium's own Gateway API implementation.
- Why
- ingress-nginx is retired and gets no more security fixes. Cilium's gateway has no hook for an external login, which would have meant a separate login per app.
- The price
- A heavier gateway than the minimum. A lighter one is noted as the fallback.
Secrets never enter Git, not even encrypted
- Chosen
- A hosted vault outside the lab. The cluster fetches secrets from it, and Git only holds references.
- Instead of
- Encrypted secrets committed next to the manifests.
- Why
- An encrypted file is still a target, and its key has to live somewhere. A vault outside the lab survives a full outage. The few roots of trust stay in a password manager, because storing them inside the system would mean needing the system to recover it.
- The price
- A dependency on an outside service. Running secrets keep working when it is unreachable; only changes wait.
A backup counts once it has been restored
- Chosen
- A restore drill that brings a backup back on its own and checks the result. The live status shows when it last passed.
- Instead of
- Trusting that the backup jobs report success.
- Why
- A backup job can succeed for months while writing something that cannot be restored. The only proof is a restore.
- The price
- Extra load and one more thing that can raise an alarm.
A VPN instead of a mesh overlay
- Chosen
- WireGuard on the router as the only way to the admin side from outside.
- Instead of
- Tailscale, which most homelab guides recommend.
- Why
- Tailscale solves a missing public address. This connection has one, so the overlay would only have been one more way in to secure, run by someone else.
- The price
- The apps answer web traffic from one country only. Abroad, the family connects the tunnel first. That is the main downside, and it is accepted.
This website lives apart from the lab
- Chosen
- A static site at the edge. The lab pushes a small signed status to it; nothing reaches into the lab.
- Instead of
- Hosting the site on the cluster it describes.
- Why
- A status page that goes down with the thing it reports on tells you nothing. Pushing out also means no inbound path for the site.
- The price
- The status can be a few minutes old. The page says how old it is.