Copyright © KC Green

Fuchsbau

From Mastodon to snac: A Lighter Fediverse Stack

 infrastructure 

Why the lab dropped PostgreSQL, Redis, and object storage for a filesystem-backed snac instance — with VolSync restic backups.

The lab used to run Mastodon on Kubernetes: PostgreSQL, a Redis-compatible cache, and S3-shaped media storage on top of the usual edge stack. That worked, but it was heavy for a small personal instance — more moving parts than the workload justified.

The fediverse home at this-is-fine.social now runs snac instead: a minimal ActivityPub server in portable C, no database, all state on a single filesystem tree, backed up off-cluster with the same VolSync + restic pattern as other stateful apps.

Pocket ID as the Lab OIDC Provider (Zot Example)

 infrastructure 

A small IdP at auth.this-is-fine.io — native OpenID login for the Zot registry.

Pocket ID is a small OpenID Connect provider — enough for a homelab without Keycloak. The lab runs it at https://auth.this-is-fine.io with Flux and an HTTPRoute on Envoy Gateway. Below is how Zot uses native OpenID against that issuer.

If you already run a homelab IdP, the interesting part is how little application code must change when the app speaks OIDC natively. Zot is a clear example: configure issuer URL, client ID, and redirect URIs, mount a secret, and the registry UI handles login without an OAuth sidecar in front of it.

From Harbor to Zot on ARM64

 infrastructure 

Why the lab dropped Harbor for a lighter OCI registry after moving Talos nodes to aarch64.

Harbor is a full registry product: scanning, replication, projects, and several backing services. On aarch64 it is also heavy, and it was a poor fit for four RK1 boards (ARM support is still catching up). After moving Talos from x86-64 to ARM64, Harbor stayed disabled in Git; Zot serves oci.this-is-fine.io instead.

Harbor shines when you want built-in scanning UI, replication policies, and a large operations surface. The lab mainly needed push, pull, sign, and admit — a smaller registry that speaks OCI and runs comfortably on ARM boards was the better fit.

Cilium BGP and a UniFi UDM-SE as the Lab iBGP Router

 infrastructure 

Advertise Kubernetes LoadBalancer IPs into the LAN with Cilium’s BGP control plane and the UDM gateway as peer.

You can get MetalLB-style service IPs without MetalLB. Cilium includes a BGP control plane that advertises LoadBalancer addresses on the LAN. The lab peers worker nodes with a UniFi UDM-SE as the default-gateway iBGP peer so phones, laptops, and the tailnet can reach Envoy Gateway VIPs without NodePorts.

Without BGP (or something like MetalLB), a LoadBalancer Service on bare metal often sits Pending forever. Advertising routes into the LAN gateway fixes that: the UDM learns /32 (or the pool) and forwards traffic to the node Cilium chose for the Service.

Envoy Gateway and the Move from Traefik to Gateway API

 infrastructure 

Three shared gateways replace Traefik-style CRDs — with Headscale as a worked example (public API, private UI).

The lab used to run Traefik with its own CRDs (IngressRoute, Middleware, and friends). Gateway API standardises routes; Envoy Gateway is the controller here — one Helm install, three shared Gateways, and per-app HTTPRoute resources instead of Traefik-only objects.

Traefik is excellent at the edge, but its CRDs are controller-specific. Moving to Gateway API was less about feature envy and more about portability: the same HTTPRoute can be read by another implementation if you ever switch controllers. Envoy Gateway is the implementation here; the routes stay standard Kubernetes objects.

Cosign and Kyverno for ZeroClaw Container Images

 infrastructure   security 

Sign container images in CI, store signatures in OCI — Kyverno refuses unsigned pods at admission.

For images you build yourself, a practical supply-chain loop is: build in CI, sign the digest, verify at admission. The lab uses Cosign (Sigstore), a private Zot registry at oci.this-is-fine.io, and Kyverno verifyImages so unsigned ZeroClaw pods do not start.

CI holds the private signing key; the cluster policy carries the matching public key.

Signing answers a simple question: did this image come from your build pipeline? Scanning for CVEs is still worth doing, but signature verification stops casual image substitution even when a tag name looks familiar.

Running Mastodon in the Lab (Fediverse Stack)

 infrastructure 

What Mastodon needs under the hood — and how a small Kubernetes stack covers it with GitOps.

Mastodon is a federated microblog server: local timelines plus ActivityPub links to other instances. Running it means managing state (PostgreSQL, media, cache) and edge (HTTPS and /.well-known for discovery). The lab instance is at https://mastodon.this-is-fine.social.

It ships as a Helm release through Flux, same GitOps style as the rest of the lab.

Federation is the interesting part socially and technically. Other servers discover you through HTTPS hostnames and /.well-known endpoints; if DNS or TLS drifts, federation breaks even when the pod is healthy. That is why the lab treats edge (Gateway API, cert-manager, external-dns) as first-class dependencies, not an afterthought.

IRC in the Lab: InspIRCd and a ZNC Bouncer

 infrastructure 

Classic IRC on Kubernetes — InspIRCd for the network, ZNC so clients can disconnect without losing context.

IRC is still a simple model: one server, many channels, text over a long-lived TCP session. InspIRCd runs the network; ZNC is a bouncer that stays logged in when your client disconnects and replays backlog when you return.

In the lab both are plain Deployments under Flux, with TLS from cert-manager and a Gateway API TCPRoute for TLS on port 6697.

IRC is a good contrast to HTTP-only homelab apps: clients expect a long-lived TCP session and often TLS on a non-443 port. Gateway API TCPRoute covers that without bolting IRC into an HTTP ingress controller as a special case.

Cluster-Wide Tailscale: Headscale, Tailnet DNS, and Cross-Cluster Routes

 infrastructure 

Self-hosted Headscale plus a few Kubernetes pieces — private mesh without the official TS operator.

Tailscale builds a WireGuard mesh with little configuration. Headscale is an open control server for the same clients — you run policy and issue keys yourself. The lab does not use the Tailscale Kubernetes operator; a handful of Deployments and DaemonSets do the job instead.

The goal is one tailnet for laptops and nodes, with Kubernetes APIs and internal HTTP on *.tif.internal without putting those names on the public internet.

Self-hosting the control plane means you own ACL files, preauth keys, and MagicDNS base domains. The trade-off is operational work: upgrades, backups, and policy edits are yours. For a multi-cluster lab that already runs GitOps everywhere else, that trade-off is acceptable.

VolSync, Volume Snapshots, and Restic Off-Site Backups

 infrastructure 

CSI snapshots plus VolSync and restic — encrypted off-site backups via Vault and External Secrets.

Stateful apps need point-in-time copies and a copy off the cluster. The lab uses VolSync with the restic mover: Kubernetes creates a VolumeSnapshot, VolSync runs restic against it, and encrypted data lands in a remote repository. The restic URL and password live in Vault and reach the cluster through External Secrets.

VolSync sits in the middle: you already run the CSI snapshot controller and a storage class that supports snapshots (Rook-Ceph block volumes in the lab). VolSync watches a ReplicationSource, triggers on a schedule, and spins up a short-lived mover job. You get off-site copies without shelling into pods to run restic by hand.