Copyright © KC Green

Flux

Running Mastodon in the Lab (Fediverse Stack)

 infrastructure 

What Mastodon needs under the hood — and how a small Kubernetes stack covers it with GitOps.

Mastodon is a federated microblog server: local timelines plus ActivityPub links to other instances. Running it means managing state (PostgreSQL, media, cache) and edge (HTTPS and /.well-known for discovery). The lab instance is at https://mastodon.this-is-fine.social.

It ships as a Helm release through Flux, same GitOps style as the rest of the lab.

Federation is the interesting part socially and technically. Other servers discover you through HTTPS hostnames and /.well-known endpoints; if DNS or TLS drifts, federation breaks even when the pod is healthy. That is why the lab treats edge (Gateway API, cert-manager, external-dns) as first-class dependencies, not an afterthought.

VolSync, Volume Snapshots, and Restic Off-Site Backups

 infrastructure 

CSI snapshots plus VolSync and restic — encrypted off-site backups via Vault and External Secrets.

Stateful apps need point-in-time copies and a copy off the cluster. The lab uses VolSync with the restic mover: Kubernetes creates a VolumeSnapshot, VolSync runs restic against it, and encrypted data lands in a remote repository. The restic URL and password live in Vault and reach the cluster through External Secrets.

VolSync sits in the middle: you already run the CSI snapshot controller and a storage class that supports snapshots (Rook-Ceph block volumes in the lab). VolSync watches a ReplicationSource, triggers on a schedule, and spins up a short-lived mover job. You get off-site copies without shelling into pods to run restic by hand.

ZeroClaw in the Lab: a GitOps Agent on Matrix

 infrastructure 

An ops-oriented agent on Kubernetes — Matrix chat in, Flux and kubectl out, MCP for the web.

ZeroClaw (nickname Claw) is a small AI agent for cluster work: check Flux status, explain failing pods, dry-run Renovate, draft Git patches. It shares the same GitOps monorepo as the cluster — it is not a second control plane.

You talk to it on Matrix at matrix.this-is-fine.social. Web fetches go through MCP to a Scrapling sidecar instead of built-in browser tools, so there is one audited path for HTTP.

The agent is deliberately ops-focused. It is not a general chatbot for the public internet; it reads cluster state, follows skills checked into Git, and proposes changes that still pass human review. That keeps expectations aligned with what automation can safely do inside a production-shaped lab.

Lab GitOps: Talos, Flux, and a Monorepo

 infrastructure 

Why the lab pairs immutable Talos nodes with Flux CD — and which projects make that workable on ARM.

The lab is a Kubernetes testbed: try cluster layouts here before they go to production. After a short manual bootstrap, Flux keeps the cluster in sync with Git — the usual GitOps loop of declared state in version control and controllers that apply diffs.

Nodes run Talos Linux on Turing Pi RK1 boards. Images come from the Talos Image Factory (metal-arm64, sbc-rockchip / turingrk1). Talos keeps the node OS small and API-driven: you do not SSH in to patch packages. That pushes complexity into Kubernetes manifests, which fits a GitOps workflow.

Everything after the first bootstrap lives in one monorepo: cluster overlays, shared operators, Helm charts, and application kustomizations. Talos machine config (talconfig) is versioned there too, but applied with talosctl, not Flux. Drawing that line clearly avoids pretending the entire world is reconciled from Git when the hypervisor and disk layout are still operator steps.