PLAN: Two-Server Fleet — Claw (primary) + Cyclops (worker)

Status: PROPOSED v1.1 (not yet executed). Cyclops is already cleaned up and minion-ready. v1.1 amendments from design review: fingerprint-verified key exchange, base-ubuntu-first isolation, whole-file Caddy ownership, UNMANAGED contract, backup key chain (3 homes, 2 physical), single write path. Author: Hermes · Date: 2026-08-29 · Repo: github.com/fakeid30/homelab (mirrored to Forgejo)

1. Current state (done)

Access: ssh nirjhor@cyclops.painlesswp.com works from Claw (key-based). Sudo pass: [REDACTED — user knows it]. Forgejo: HTTP via Caddy (forgejo.cyclops.painlesswp.com), SSH on port 2222, repo chorompotro/homelab (mirrors GitHub).

2. Target architecture

Users ──► *.eazyplugins.com ──► CLAW (sole ingress, TLS, Anubis chain)
                                    │
                                    ├─ local quadlets (unchanged)
                                    │
                                    └─ WireGuard tunnel 10.0.0.1 ↔ 10.0.0.2
                                              │
                                         CYCLOPS: worker services
                                         (forgejo stays on its own domain,
                                          future: immich, heavy/batch jobs)

• DNS: zero changes ever. Wildcard *.eazyplugins.com → Claw covers future services.
• URLs never change for users; only Caddy's upstream target shows where bytes go.
• Claw keeps all TLS certs; Cyclops never needs Let's Encrypt.

3. Phase plan

Phase A — Trust fabric (~1h total)

1. WireGuard (~30 min): both nodes; Claw=10.0.0.1, Cyclops=10.0.0.2. Key exchange via each box's existing SSH session — never via the tunnel being established or any third-party paste/bootstrap channel. Tunnel carries: salt, cross-node upstreams, backups.
2. Salt master on Claw (dnf install salt-master, bind 4505/4506 to tunnel interface only).
3. Salt minion on Cyclops (apt from salt repo), master=10.0.0.1.
4. Key acceptance — fingerprint rule: auto_accept: False permanently. Accept only after salt-key -f <minion> on Claw and salt-call key.finger (or local salt-key -f mymaster) on Cyclops produce matching fingerprints, compared out-of-band (each box's own SSH session — not over the tunnel, not via chat relay of one side only).

Phase B — State tree refactor (3–4h; risk center is the Debian branch)

5. Grains: Claw roles:[primary, ingress, anubis]; Cyclops roles:[worker, forgejo].
6. base-ubuntu.sls FIRST, in isolation (the risk item — Debian package/service names, AppArmor paths vs SELinux, Ubuntu caddy repo):
   • Write it, then dry-run standalone on Cyclops as root: sudo salt-call --local state.apply base-ubuntu test=True — testing what will actually run, not what master-pushed top.sls will do later.
   • Gate: 0 failures or it doesn't exist. Only then does top.sls include it.
7. Remaining Cyclops states (after base-ubuntu is known-good):
   • forgejo.sls — codifies the container definition only (quadlet file, image, port 2222→22 mapping, network). Data dir gets file.directory (existence + ownership) — no recurse, no prune, no clean.
   • caddy-cyclops.sls — owns the whole Caddyfile (file.managed, full contents from repo copy). No sed-merges, no partial management. Manual on-disk edits get overwritten on next apply — drift surfaces as a diff, which is the feature. Emergency edits go through the repo (edit state → apply), never on-disk.
8. Pillar per-node: Cyclops never receives Claw's secrets. OS branches via map.jinja (os_family: Debian) — ~20 lines.

Phase C — First workloads (optional, later)

9. WireGuard-first app to prove the pattern (e.g. Immich on Cyclops: Caddy vhost on Claw → 10.0.0.2:2283, new Anubis instance on Claw, no DNS change).
10. Backups — restic over the tunnel. Repo per direction: claw→cyclops (Claw critical data), cyclops→claw (forgejo). Key chain (decided now, three homes, all independent):
    1. Repo passwords in salt pillar on the master (delivered per-node)
    2. Pen-drive copy (offline)
    3. Printed copy (offline, different physical location from the pen-drive) DR chain: machine dies → rebuild from salt → pillar restores restic passwords → restore. Both machines die → offline copies + GitHub repo rebuild. The key has exactly three homes; this is closed now, not in Phase C.

Phase D — Day-2 (optional, when >2 nodes or automation wanted)

11. state.orchestrate for cross-node ordering; reactors/beacons for event-driven automation.
12. Scale rule: 3rd+ server ⇒ this design pays off; at 2 nodes master-minion is architecturally right but operationally optional (salt-ssh is the 70% fallback with no master risk).

4. Why this design

• Claw as sole ingress keeps: one Anubis pattern, one Caddyfile, all URLs stable, one TLS story. Cost: Claw is a SPOF for Cyclops-hosted user services (documented trade).
• Ubuntu stays: salt is cross-OS by design; only cost is the Debian-family branch in states. Reinstalling to Alma is not needed (revisit only if Cyclops gets rebuilt anyway).
• Memory is federated, not pooled: Claw RAM serves Claw services; Cyclops' 2.6GB free serves workers. You place services; salt enforces placement.

5. Risks & mitigations

6. CONVENTIONS (binding, also mirrored in repo)

1. Single write path: all applies originate from Claw (salt 'cyclops' state.apply ...). salt-call --local on Cyclops exists for exactly one purpose: read-only validation (test=True dry-runs, as root). No local applies, no minion-side cron applies, no on-disk quick fixes — the fix path is edit-state-on-Claw → apply from Claw. One writer = one definition of "applied."
2. Whole-file ownership: salt-managed config files (Caddyfile, quadlets) are generated wholesale from the repo. No merges.
3. Data is never state: container definitions are managed; data dirs get existence+ownership only.
4. Validation gate: root dry-run 0 failures before any apply; endpoints 200 after.
5. Unmanaged list lives in UNMANAGED.md (repo root); additions require a commit explaining why.

7. UNMANAGED (excluded from all state, see UNMANAGED.md)

• openclaw-gateway.service — no state may reference it, stop it, or depend on it. (Root's salt wouldn't see user units anyway — the exclusion is deliberate, not incidental.)
• ~/forgejo/data — definition managed, data never.
• ~/saltstack, ~/package.json — openclaw-adjacent artifacts, outside management.

8. Execution checklist (when given the GO)

• WireGuard up both sides (SSH-swapped keys), ping 10.0.0.2 ↔ 10.0.0.1
• salt-master on Claw (bind tunnel interface; file_roots → homelab repo path; auto_accept: False)
• salt-minion on Cyclops; fingerprints compared out-of-band; key accepted
• salt 'cyclops' test.ping → True
• Grains (roles) set both nodes
• base-ubuntu.sls written → standalone root dry-run on Cyclops → 0 failures → then top.sls inclusion
• forgejo.sls (definition-only) + caddy-cyclops.sls (whole-file) written; dry-run → 0 failures
• Real apply from Claw → verify forgejo + openclaw + endpoints still 200 after
• Pillar top.sls per-node split; secrets audited (Cyclops gets only its own)
• UNMANAGED.md + this conventions section committed
• Commit tree changes → push GitHub + Forgejo
• RUNBOOK.md updated: two-node rebuild procedure
• (Phase C, later) restic repos + key chain deployed; first tunnel service

9. Reference facts

• Cyclops SSH from Claw is passwordless (key). Forgejo git push = SSH port 2222, key nirjhor@claw.hackershostel.com (verified).
• OpenClaw on Cyclops = systemd user service openclaw-gateway.service, node process, port 18789, fronted by anubis-cyclops — never modify.
• Pre-cleanup backup of Cyclops (forgejo data + authelia) sits at ~/backups/cyclops-preclean-backup.tar.gz on Claw (192KB, deletable).
• Salt validation convention: full salt-call --local state.apply test=True as root must be 0 failures before any push (Claw baseline: 196/0).

10. OBSERVED REALITY (2026-08-29 post-cleanup inspection) — salt must codify THIS

The user deployed significant infrastructure by hand after cleanup. Salt states for Cyclops must codify the following as-is (not the older assumptions):

Forgejo (significantly reworked)

• Image: codeberg.org/forgejo/forgejo:16 (v16.0.3) — upgraded from gitea 9.0.3
• Quadlet: ~/.config/containers/systemd/forgejo.container (rewritten 2026-08-29 17:27)
  • Ports: 3026:3000 (HTTP), 2222:22 (git SSH)
  • Volumes: ~/forgejo/data:/data (unmanaged), /etc/timezone+/etc/localtime (ro)
  • Env: GITEA_CUSTOM=/data/gitea, USER_UID/GID=1000, AutoUpdate=registry, Restart=on-failure
• app.ini: DOMAIN/SSH_DOMAIN = forgejo.cyclops.painlesswp.com, ROOT_URL https, SSH_PORT=2222
• Data: ~/forgejo/data — gitea+git+ssh dirs, host keys generated, 10MB — UNMANAGED (per UNMANAGED.md)
• Legacy ~/forgejo/docker-compose.yml still on disk (superseded by quadlet; candidate for removal)

NEW: Forgejo Actions CI + docs site

• forgejo-runner (~/.local/bin/forgejo-runner 13.0.0) — systemd user unit forgejo-runner.service, WorkingDirectory ~/runner, DOCKER_HOST→podman.sock
• Workflow .forgejo/workflows/docs.yml in repo: on push→main, zola builds site → /mnt_pages/homelab → runner publishes to ~/forgejo-pages/homelab
• docs.cyclops.painlesswp.com — Caddy file_server serving the built site (LIVE, 200)

NEW: SearXNG (back, by user decision)

• Container searxng (image 2026.8.29-d226b78bc), started 14:36 via plain podman run — NO quadlet, no systemd unit (restart=unless-stopped only). ⚠ Not boot-persistent, not auto-updated. Salt state needed if this stays.

Caddy on Cyclops (3 vhosts now, still NO anubis on forgejo/docs vhosts)

• cyclops.painlesswp.com → anubis-cyclops → openclaw (unchanged)
• forgejo.cyclops.painlesswp.com → localhost:3026 (direct, no anubis)
• docs.cyclops.painlesswp.com → file_server ~/forgejo-pages/homelab (direct, no anubis)

URL rename (done)

• forgeho. → forgejo.cyclops.painlesswp.com everywhere: app.ini, Caddyfile, claw-side ssh config, git remotes, docs. Old hostname dead.

Salt implications (delta from v1.1 plan)

1. forgejo.sls must pin image codeberg.org/forgejo/forgejo:16 + the two PublishPorts + GITEA_CUSTOM env
2. NEW forgejo-runner.sls: user unit + binary + config.yml (runner is CI-critical — definition managed, secrets/registration NOT)
3. NEW docs-publish handling: either manage ~/forgejo-pages as unmanaged output dir (Actions-owned) or exclude — it's CI-written, salt must never clean it
4. Decide: searxng quadlet-ized (recommended) or removed
5. Decide: anubis in front of forgejo/docs vhosts (v1.1 plan assumed Anubis universal; current manual Caddyfile has none on these two)

ADDENDUM (same day): single SearXNG decision

• Cyclops' local searxng container REMOVED (was unit-less, not boot-persistent)
• OpenClaw on Cyclops now uses claw's searxng via https://search.eazyplugins.com
• Claw Caddyfile: /search?format=json from 160.191.163.106 (cyclops) bypasses Anubis, IP-gated handle block; all other traffic still goes through Anubis challenge
• salt delta: caddy.sls (claw) must include this @searx_internal block; openclaw config on cyclops stays UNMANAGED (part of .openclaw, openclaw rule adjacency)