Two-Server Fleet: Cyclops Minion Plan

Unknown — Sat, 29 Aug 2026 00:00:00 +0000

PLAN: Two-Server Fleet — Claw (primary) + Cyclops (worker)

Status: PROPOSED v1.1 (not yet executed). Cyclops is already cleaned up and minion-ready. v1.1 amendments from design review: fingerprint-verified key exchange, base-ubuntu-first isolation, whole-file Caddy ownership, UNMANAGED contract, backup key chain (3 homes, 2 physical), single write path. Author: Hermes · Date: 2026-08-29 · Repo: github.com/fakeid30/homelab (mirrored to Forgejo)

1. Current state (done)

	Claw (claw.eazyplugins.com)	Cyclops (160.191.163.106)
OS	AlmaLinux 10.2	Ubuntu 24.04 (keep — no reinstall)
Specs	4 vCPU / 7.5GB / 44GB free	2 vCPU / 3.8GB / 20GB free (post-cleanup)
Role	Primary — ingress + user-facing services	Worker — will host Forgejo (+ future heavy apps)
Stack	Caddy → 20× Anubis → 56 rootless quadlets	Caddy (2 vhosts) → forgejo + anubis-cyclops
IaC	Masterless salt, 196 states, validated	none yet (cleaned by hand, backup taken)
Running now	59 containers	openclaw-gateway (bare node, DO NOT TOUCH) + forgejo + anubis-cyclops + caddy

Access: ssh nirjhor@cyclops.painlesswp.com works from Claw (key-based). Sudo pass: [REDACTED — user knows it]. Forgejo: HTTP via Caddy (forgejo.cyclops.painlesswp.com), SSH on port 2222, repo chorompotro/homelab (mirrors GitHub).

2. Target architecture

Users ──► *.eazyplugins.com ──► CLAW (sole ingress, TLS, Anubis chain)
                                    │
                                    ├─ local quadlets (unchanged)
                                    │
                                    └─ WireGuard tunnel 10.0.0.1 ↔ 10.0.0.2
                                              │
                                         CYCLOPS: worker services
                                         (forgejo stays on its own domain,
                                          future: immich, heavy/batch jobs)

DNS: zero changes ever. Wildcard *.eazyplugins.com → Claw covers future services.
URLs never change for users; only Caddy's upstream target shows where bytes go.
Claw keeps all TLS certs; Cyclops never needs Let's Encrypt.

3. Phase plan

Phase A — Trust fabric (~1h total)

WireGuard (~30 min): both nodes; Claw=10.0.0.1, Cyclops=10.0.0.2. Key exchange via each box's existing SSH session — never via the tunnel being established or any third-party paste/bootstrap channel. Tunnel carries: salt, cross-node upstreams, backups.
Salt master on Claw (dnf install salt-master, bind 4505/4506 to tunnel interface only).
Salt minion on Cyclops (apt from salt repo), master=10.0.0.1.
Key acceptance — fingerprint rule: auto_accept: False permanently. Accept only after salt-key -f <minion> on Claw and salt-call key.finger (or local salt-key -f mymaster) on Cyclops produce matching fingerprints, compared out-of-band (each box's own SSH session — not over the tunnel, not via chat relay of one side only).

Phase B — State tree refactor (3–4h; risk center is the Debian branch)

Grains: Claw roles:[primary, ingress, anubis]; Cyclops roles:[worker, forgejo].
base-ubuntu.sls FIRST, in isolation (the risk item — Debian package/service names, AppArmor paths vs SELinux, Ubuntu caddy repo):
- Write it, then dry-run standalone on Cyclops as root: sudo salt-call --local state.apply base-ubuntu test=True — testing what will actually run, not what master-pushed top.sls will do later.
- Gate: 0 failures or it doesn't exist. Only then does top.sls include it.
Remaining Cyclops states (after base-ubuntu is known-good):
- forgejo.sls — codifies the container definition only (quadlet file, image, port 2222→22 mapping, network). Data dir gets file.directory (existence + ownership) — no recurse, no prune, no clean.
- caddy-cyclops.sls — owns the whole Caddyfile (file.managed, full contents from repo copy). No sed-merges, no partial management. Manual on-disk edits get overwritten on next apply — drift surfaces as a diff, which is the feature. Emergency edits go through the repo (edit state → apply), never on-disk.
Pillar per-node: Cyclops never receives Claw's secrets. OS branches via map.jinja (os_family: Debian) — ~20 lines.

Phase C — First workloads (optional, later)

WireGuard-first app to prove the pattern (e.g. Immich on Cyclops: Caddy vhost on Claw → 10.0.0.2:2283, new Anubis instance on Claw, no DNS change).
Backups — restic over the tunnel. Repo per direction: claw→cyclops (Claw critical data), cyclops→claw (forgejo). Key chain (decided now, three homes, all independent):
1. Repo passwords in salt pillar on the master (delivered per-node)
2. Pen-drive copy (offline)
3. Printed copy (offline, different physical location from the pen-drive) DR chain: machine dies → rebuild from salt → pillar restores restic passwords → restore. Both machines die → offline copies + GitHub repo rebuild. The key has exactly three homes; this is closed now, not in Phase C.

Phase D — Day-2 (optional, when >2 nodes or automation wanted)

state.orchestrate for cross-node ordering; reactors/beacons for event-driven automation.
Scale rule: 3rd+ server ⇒ this design pays off; at 2 nodes master-minion is architecturally right but operationally optional (salt-ssh is the 70% fallback with no master risk).

4. Why this design

Claw as sole ingress keeps: one Anubis pattern, one Caddyfile, all URLs stable, one TLS story. Cost: Claw is a SPOF for Cyclops-hosted user services (documented trade).
Ubuntu stays: salt is cross-OS by design; only cost is the Debian-family branch in states. Reinstalling to Alma is not needed (revisit only if Cyclops gets rebuilt anyway).
Memory is federated, not pooled: Claw RAM serves Claw services; Cyclops' 2.6GB free serves workers. You place services; salt enforces placement.

5. Risks & mitigations

Risk	Mitigation
Claw down → Cyclops user services dark	Accepted (homelab); document which services die with Claw
Master down → no convergence (nodes keep running last config)	salt-ssh break-glass; nodes are NOT dependent at runtime
Tunnel down → cross-node services + salt break	Uptime Kuma check on 10.0.0.2
Drift (manual changes on Cyclops)	Whole-file state ownership + re-apply; drift surfaces as diff
Mixed-OS state bugs	base-ubuntu-first isolation; root dry-run standalone before top.sls inclusion
MITM during key exchange	Fingerprint match out-of-band; auto_accept False permanently

6. CONVENTIONS (binding, also mirrored in repo)

Single write path: all applies originate from Claw (salt 'cyclops' state.apply ...). salt-call --local on Cyclops exists for exactly one purpose: read-only validation (test=True dry-runs, as root). No local applies, no minion-side cron applies, no on-disk quick fixes — the fix path is edit-state-on-Claw → apply from Claw. One writer = one definition of "applied."
Whole-file ownership: salt-managed config files (Caddyfile, quadlets) are generated wholesale from the repo. No merges.
Data is never state: container definitions are managed; data dirs get existence+ownership only.
Validation gate: root dry-run 0 failures before any apply; endpoints 200 after.
Unmanaged list lives in UNMANAGED.md (repo root); additions require a commit explaining why.

7. UNMANAGED (excluded from all state, see UNMANAGED.md)

openclaw-gateway.service — no state may reference it, stop it, or depend on it. (Root's salt wouldn't see user units anyway — the exclusion is deliberate, not incidental.)
~/forgejo/data — definition managed, data never.
~/saltstack, ~/package.json — openclaw-adjacent artifacts, outside management.

8. Execution checklist (when given the GO)

WireGuard up both sides (SSH-swapped keys), ping 10.0.0.2 ↔ 10.0.0.1
salt-master on Claw (bind tunnel interface; file_roots → homelab repo path; auto_accept: False)
salt-minion on Cyclops; fingerprints compared out-of-band; key accepted
salt 'cyclops' test.ping → True
Grains (roles) set both nodes
base-ubuntu.sls written → standalone root dry-run on Cyclops → 0 failures → then top.sls inclusion
forgejo.sls (definition-only) + caddy-cyclops.sls (whole-file) written; dry-run → 0 failures
Real apply from Claw → verify forgejo + openclaw + endpoints still 200 after
Pillar top.sls per-node split; secrets audited (Cyclops gets only its own)
UNMANAGED.md + this conventions section committed
Commit tree changes → push GitHub + Forgejo
RUNBOOK.md updated: two-node rebuild procedure
(Phase C, later) restic repos + key chain deployed; first tunnel service

9. Reference facts

Cyclops SSH from Claw is passwordless (key). Forgejo git push = SSH port 2222, key nirjhor@claw.hackershostel.com (verified).
OpenClaw on Cyclops = systemd user service openclaw-gateway.service, node process, port 18789, fronted by anubis-cyclops — never modify.
Pre-cleanup backup of Cyclops (forgejo data + authelia) sits at ~/backups/cyclops-preclean-backup.tar.gz on Claw (192KB, deletable).
Salt validation convention: full salt-call --local state.apply test=True as root must be 0 failures before any push (Claw baseline: 196/0).

10. OBSERVED REALITY (2026-08-29 post-cleanup inspection) — salt must codify THIS

The user deployed significant infrastructure by hand after cleanup. Salt states for Cyclops must codify the following as-is (not the older assumptions):

Forgejo (significantly reworked)

Image: codeberg.org/forgejo/forgejo:16 (v16.0.3) — upgraded from gitea 9.0.3
Quadlet: ~/.config/containers/systemd/forgejo.container (rewritten 2026-08-29 17:27)
- Ports: 3026:3000 (HTTP), 2222:22 (git SSH)
- Volumes: ~/forgejo/data:/data (unmanaged), /etc/timezone+/etc/localtime (ro)
- Env: GITEA_CUSTOM=/data/gitea, USER_UID/GID=1000, AutoUpdate=registry, Restart=on-failure
app.ini: DOMAIN/SSH_DOMAIN = forgejo.cyclops.painlesswp.com, ROOT_URL https, SSH_PORT=2222
Data: ~/forgejo/data — gitea+git+ssh dirs, host keys generated, 10MB — UNMANAGED (per UNMANAGED.md)
Legacy ~/forgejo/docker-compose.yml still on disk (superseded by quadlet; candidate for removal)

NEW: Forgejo Actions CI + docs site

forgejo-runner (~/.local/bin/forgejo-runner 13.0.0) — systemd user unit forgejo-runner.service, WorkingDirectory ~/runner, DOCKER_HOST→podman.sock
Workflow .forgejo/workflows/docs.yml in repo: on push→main, zola builds site → /mnt_pages/homelab → runner publishes to ~/forgejo-pages/homelab
docs.cyclops.painlesswp.com — Caddy file_server serving the built site (LIVE, 200)

NEW: SearXNG (back, by user decision)

Container searxng (image 2026.8.29-d226b78bc), started 14:36 via plain podman run — NO quadlet, no systemd unit (restart=unless-stopped only). ⚠ Not boot-persistent, not auto-updated. Salt state needed if this stays.

Caddy on Cyclops (3 vhosts now, still NO anubis on forgejo/docs vhosts)

cyclops.painlesswp.com → anubis-cyclops → openclaw (unchanged)
forgejo.cyclops.painlesswp.com → localhost:3026 (direct, no anubis)
docs.cyclops.painlesswp.com → file_server ~/forgejo-pages/homelab (direct, no anubis)

URL rename (done)

forgeho. → forgejo.cyclops.painlesswp.com everywhere: app.ini, Caddyfile, claw-side ssh config, git remotes, docs. Old hostname dead.

Salt implications (delta from v1.1 plan)

forgejo.sls must pin image codeberg.org/forgejo/forgejo:16 + the two PublishPorts + GITEA_CUSTOM env
NEW forgejo-runner.sls: user unit + binary + config.yml (runner is CI-critical — definition managed, secrets/registration NOT)
NEW docs-publish handling: either manage ~/forgejo-pages as unmanaged output dir (Actions-owned) or exclude — it's CI-written, salt must never clean it
Decide: searxng quadlet-ized (recommended) or removed
Decide: anubis in front of forgejo/docs vhosts (v1.1 plan assumed Anubis universal; current manual Caddyfile has none on these two)

ADDENDUM (same day): single SearXNG decision

Cyclops' local searxng container REMOVED (was unit-less, not boot-persistent)
OpenClaw on Cyclops now uses claw's searxng via https://search.eazyplugins.com
Claw Caddyfile: /search?format=json from 160.191.163.106 (cyclops) bypasses Anubis, IP-gated handle block; all other traffic still goes through Anubis challenge
salt delta: caddy.sls (claw) must include this @searx_internal block; openclaw config on cyclops stays UNMANAGED (part of .openclaw, openclaw rule adjacency)

Homelab