Cyclops — Worker Node of the Dual-Node Fleet

Node: cyclops.painlesswp.com · Ubuntu 24.04 (AppArmor) · public 160.191.163.106 Role: fleet worker — app services behind claw's ingress, reachable only over the fleet WireGuard tunnel (10.0.0.2). Managed by: SaltStack as a minion of claw's master (10.0.0.1). Source of truth is the salt-fleet repo (Forgejo chorompotro/salt-fleet, GitHub mirror fakeid30/salt-fleet) under cyclops/salt/minion/*.sls — staged to claw /srv/salt/minion/ by tools/deploy.sh, applied with sudo /opt/saltvenv/bin/salt cyclops state.apply <minion.state>. Live as of 2026-09-14.

1. The model — two servers, ONE ingress

Internet → *.eazyplugins.com DNS → claw (160.250.95.39) Caddy TLS
         → Anubis (per-service, on claw) → 10.0.0.2:<port> over WireGuard → cyclops container

• Claw (AlmaLinux 10.2, SELinux enforcing) — sole public ingress. Caddy terminates TLS, Anubis bot-challenges, WireGuard hub (10.0.0.1). Hosts the claw-local services and one Anubis instance per cyclops-backed service.
• Cyclops (Ubuntu 24.04, AppArmor) — worker. No public ports beyond 22/80/443/2222 (Forgejo git-ssh). Services bind tunnel-only 10.0.0.2:<port> and are reached by claw's Caddy over WireGuard. ufw trusts the WG subnet wholesale.
• WireGuard: claw 10.0.0.1 ↔ cyclops 10.0.0.2; keys live in salt-fleet pillars (server:wireguard, minion:wireguard).

Why cyclops? Claw was RAM-bound; cyclops has headroom for stateful apps and their databases, and it moves heavier/untrusted workloads off the ingress box. Accepted trade-off: claw is a single point of failure for ingress — if claw is down, cyclops-hosted services are unreachable. Cyclops has no public path of its own, by design.

2. Service placement (current)

Every cyclops user-facing service is fronted by an Anubis instance on claw (host network, 127.0.0.1:89XX → 10.0.0.2:<port>):

3. Salt management (source of truth)

Cyclops is a salt minion (id cyclops) of claw's master (10.0.0.1, over WireGuard). Authoritative states live in the salt-fleet repo (~/salt-fleet on claw), directory cyclops/salt/minion/, applied via run0 /opt/saltvenv/bin/salt 'cyclops' state.apply <state>. Pushes hit Forgejo, pass the CI render gate, then deploy by webhook to claw /srv/salt + /srv/pillar.

Top-level state list (claw/salt/top.sls, 'cyclops' section) — this is the authoritative inventory:

Apply pattern (from claw): run0 /opt/saltvenv/bin/salt 'cyclops' state.apply minion.<state> --timeout=180 Secrets live in claw pillar claw/pillar/minion.sls under minion: (per-service blocks) + restic:. Never committed to the homelab docs repo.

⚠️ ~/saltstack on cyclops is a STALE leftover from the pre-fleet standalone era (beaverhabits/privatebin/searxng/your_spotify states — those services do NOT run on cyclops anymore). Not used since minion mode was adopted. Do not apply anything from it.

4. Running containers (live inventory)

Quadlet files: ~/.config/containers/systemd/*.container (rootless podman; managed via systemctl --user as nirjhor — salt runs as root and cannot see user units directly). Podman networks: forgejo_default, outline-net, kitchenowl-net (+ transient CI job networks).

5. WireGuard (fleet tunnel)

• Interface wg0, address 10.0.0.2/24; peer = claw (10.0.0.1:51820), keepalive 25s.
• Config /etc/wireguard/wg0.conf is salt-managed (minion.wireguard, mode 600, private key from pillar minion:wireguard).
• The interface is brought up once at boot by systemd; config changes need a manual sudo wg syncconf wg0 (or systemctl restart wg-quick@wg0 only when the tunnel is down). Do not watch→restart from salt.
• Carries: salt (master 10.0.0.1), restic SFTP leg, claw Caddy → 10.0.0.2 app ports, forgejo-runner callbacks.

6. Firewall (ufw)

minion.ufw — default deny incoming, allow outgoing. Explicit allows:

• 22/tcp ssh
• 80/tcp + 443/tcp cyclops-local Caddy (docs/forgejo/openclaw vhosts)
• 2222/tcp forgejo git-ssh
• from 10.0.0.0/24 — full fleet trust (salt, restic sftp, cross-node)
• App ports (3000/3001/9091/9092/39325/45189) stay bound but are public-inaccessible by default-deny.

7. Backups (restic)

Nightly 04:00 root timer (restic-backup.timer), script /usr/local/bin/restic-backup-cyclops.sh (salt-managed via minion.backup).

• Two repos (3-home principle):
  • sftp:nirjhor@10.0.0.1:/home/nirjhor/restic-repos/cyclops (claw disk)
  • b2:claw-cyclops:/cyclops (Backblaze B2)
• PATHS: ~/forgejo/data, ~/vaultwarden/data, ~/outline, ~/kitchenowl-app, ~/.config/containers
• Retention: keep-daily 7, keep-weekly 5, keep-monthly 6, prune.
• Verify: weekly restore drill Sunday 09:30 driven from claw (restic-verify), reports → fleet-status repo.

8. Cyclops-local Caddy (/etc/caddy/Caddyfile, salt-managed minion.caddy-cyclops)

This docs site = Zola build of the homelab repo, published by Forgejo Actions (publish-docs workflow, .forgejo/workflows/docs.yml) on every push to main → output /mnt_pages/homelab (bind-mounted ~/forgejo-pages/homelab), served by Caddy. To update these pages: edit ~/homelab on claw and push to Forgejo (git push forgejo main), not by editing ~/forgejo-pages directly. Site rendering (Zola 0.23.6 + Linkita theme, 2026-09-13 revision, both vendored/pinned — see below): code blocks are syntax-highlighted via [markdown.highlighting] in config.toml; mermaid diagrams render from GitHub-style fences on pages generated by tools/sync-site-content.sh (it rewrites fences to Zola <mermaid> components — canonical repo files keep raw fences for GitHub/gist). On manually-authored pages, write diagrams directly as Zola mermaid component invocations.

Toolchain pin — update these as a PAIR. Zola is pinned in .forgejo/workflows/docs.yml (the curl line that installs the binary) and the Linkita theme is vendored at themes/linkita/ (a plain copy, not a submodule, with no local patches — replacing it wholesale is safe). The two release in lockstep: Linkita's 2026-09-12 revision is "updates for Zola 0.23.5" and its 2026-09-13 revision is "updates for Zola 0.23.6", and Zola 0.23.6's own changelog entry exists to ship the Tera change that enables the theme's switch to component implicit params. Upgrading one without the other can leave the theme on component call signatures the Zola binary no longer accepts. Procedure: bump the CI pin → refresh themes/linkita/ from upstream master → bash tools/sync-site-content.sh → zola build → diff the output against the previous build (page count, then text-extract each page and confirm the prose is byte-identical). The 0.23.4→0.23.6 paired update on 2026-09-23 changed exactly one visible string site-wide (footer "Powered by Zola" → "Made with Zola") and nothing else.

This page is generated: cyclops/README.md → content/cyclops/index.md, alongside the claw pages (claw/README.md, claw/config.md, claw/guide.md, claw/CHANGELOG.md). Edit the repo-root file, never the generated copy.

9. Maintenance playbook (handover)

Add/update a service on cyclops (cross-node pattern):

1. Write/extend cyclops/salt/minion/<name>.sls in ~/salt-fleet (quadlet + data dir + network, PublishPort=10.0.0.2:<port>:<cport> for tunnel-only apps).
2. Add the state to 'cyclops' list in claw/salt/top.sls.
3. Add any secrets to claw/pillar/minion.sls (minion:<name>:...) and the claw-side anubis entry (host_port, network: null, target: http://10.0.0.2:<port>) to claw/pillar/server.sls.
4. Commit + push salt-fleet → Forgejo webhook auto-deploys (stages to /srv/salt, /srv/pillar on claw).
5. run0 /opt/saltvenv/bin/salt 'cyclops' state.apply minion.<name> --timeout=180
6. Claw side: run0 /opt/saltvenv/bin/salt-call --local state.apply server.anubis then restart the anubis instance (run0 systemctl --user -M nirjhor@ restart anubis-<name>.service); server.caddy applies the vhost.
7. Verify: public vhost 200/302/401; app over WG answers; add policy file in claw/salt/anubis/ for /api/+/assets/ surfaces (bookorbit/kitchenowl lesson).

Moving a service claw → cyclops (the kitchenowl recipe, 2026-09-07):

1. Write the cyclops state cyclops/salt/minion/<svc>.sls mirroring kitchenowl.sls / outline.sls / vaultwarden.sls: data dir (owned by the right uid), a dedicated podman network (<svc>-net), quadlet(s) publishing 10.0.0.2:<port> only, secrets from minion:<svc> pillar, restic PATHS += data dir.
2. Add to the cyclops branch of claw/salt/top.sls; add the JWT/secret block to claw/pillar/minion.sls (cyclops-only pillar).
3. Claw side: delete the old stack from stacks.sls, flip the anubis pillar entry to {network: null, target: http://10.0.0.2:<port>, policy: <existing>} and the Caddy vhost to import anubis 10.0.0.2:<port> 127.0.0.1:<89XX>.
4. Migrate data before starting cyclops containers: stop the claw services (systemd stop), PRAGMA wal_checkpoint(TRUNCATE) for SQLite, tar+scp the data dir to cyclops. (Careful: extract into the exact target dir — tar xzf from ~ puts data/ at ~/data.)
5. Apply minion.<svc> on cyclops first; verify data reads (sqlite3 counts / API); THEN flip claw anubis + caddy. Rollback = keep old claw quadlets stopped, not deleted, until the new node is verified.
6. Doc-sync: config.md tables + README mermaid + guide vhost table + RUNBOOK (counts, DR dirs, setup row) in the same commit wave.

Drift check: claw runs drift-check daily 05:37 → /var/log/drift-report.txt; cyclops states must read 0 failures. Auto-update: podman-auto-update.timer daily 00:15 (user), AutoUpdate=registry on quadlets. openclaw gateway: native user service on cyclops (:18789) — excluded from all salt states (UNMANAGED.md). Update with ~/openclaw-safe-update.sh [version|latest], never openclaw update — the built-in updater's staged swap is broken on this host (see §10). Never via salt.

10. OpenClaw (native process — UNMANAGED by salt)

Systemd user service openclaw-gateway.service (node, loopback :18789), fronted by anubis-cyclops via cyclops.painlesswp.com. State under ~/.openclaw on cyclops. Current version 2026.9.4 (3a9d69d).

Updating. Use ~/openclaw-safe-update.sh [version|latest]. The built-in openclaw update failed six consecutive attempts (2026-09-13) with Failed: global install swap — Exit code: 1 … Installation recovery is unverified, leaving the gateway failed and the bot offline — even though the candidate itself passed all seven validation checks (global update, migration rehearsal, doctor lint, config validation, plugin resolution, migration continuation, gateway canary). Two compounding causes:

1. Stop timeout too short. The unit allowed 30s to stop while the gateway legitimately drains active work for up to 315s, so systemd SIGKILLed it mid-swap. Fixed by the drop-in ~/.config/systemd/user/openclaw-gateway.service.d/stop-timeout.conf → TimeoutStopSec=360. Verify with systemctl --user show openclaw-gateway -p TimeoutStopUSec (expect 6min); a healthy stop shows no State 'stop-sigterm' timed out in the journal.
2. npm project manifest. npm is invoked with cwd=$HOME and --allow-scripts requires a valid project package.json; ~/package.json had no name field, so npm aborted with Cannot destructure property 'name' of '.for'. Fixed (backup ~/package.json.bak-20260913); installs now run from the scratch project ~/.openclaw/npm-project.

The wrapper: stops the gateway (clean drain) → tars the tree to ~/backups/ → npm-installs from the scratch project with the full --allow-scripts list → openclaw plugins update --all → restarts → verifies HTTP 200 and version → auto-rolls back from the tarball on failure. A ~3-minute drain on stop is normal, not a hang.

Provider policy: zai-only. Only zai is enabled, plus ollama for embeddings (memory search and dreaming depend on it); 19 other provider plugins are disabled. Default model pinned to agents.defaults.model.primary = zai/glm-5.3-flash, fallbacks glm-5.2 → glm-5.1 → glm-5-turbo. agents.defaults.modelPolicy.allow lists zai models only, so anything outside it cannot be selected. Dreaming is pinned to a zai model with allowModelOverride. Note: a provider absent from plugins.entries is still enabled by default — disabling needs an explicit entry.

Source of truth: Forgejo chorompotro/openclaw-cyclops (private). Contains sanitized config (secrets replaced by ${...} placeholders), workspace and memory notes, non-bundled skills, this systemd wiring (drop-in nesting preserved), the safe-update scripts, and CLI-derived exports (cron/plugins/skills/models/agents). Working copy ~/openclaw-cyclops on cyclops; openclaw-cyclops-sync.timer pushes every 15 minutes (idempotent). Restore with scripts/bootstrap.sh --dry-run then bootstrap.sh. Bulk runtime state — agents/*.sqlite, state/openclaw.sqlite, memory embeddings, caches, media — is deliberately not in the repo; restore it from restic if needed.

Monitoring gap (not implemented). No monitoring agent runs on cyclops; dozzle/glances/uptime-kuma/diun are claw-only, so cyclops is covered by external checks at best. Options scoped: Uptime Kuma push/HTTP monitors (agent-less), a Dozzle agent container, Glances server mode, or a second Diun instance on cyclops.

11. Known gotchas (read before touching)

• Salt runs as root → can't see user systemd units. Drive rootless units via run0 systemctl --user -M nirjhor@ … (or sudo -u nirjhor XDG_RUNTIME_DIR=/run/user/1000 systemctl --user …). Cyclops salt states use cmd.run → sudo -u nirjhor env XDG_RUNTIME_DIR=/run/user/1000 DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus systemctl --user …; on claw interactive shells the equivalent is run0 systemctl --user -M nirjhor@ ….
• No SELinux — AppArmor. Omit :z/:Z from mounts; file.directory uid strings work the same, but the package module is apt.
• Rootless podman uid map differs from claw. Cyclops /etc/subuid is nirjhor:100000:65536 (claw is 524288). Container postgres uid 70 → host 100069 on cyclops; container root → host 1000. Compute from the live subuid, never copy claw's numbers.
• Outline postgres data dir ownership: must be host-uid 100069 (container postgres uid 70 mapped through rootless 100000:65536). Never chown to nirjhor — the container then fails with Permission denied on pg_filenode.map.
• No passwordless sudo for the agent SSH key on cyclops — root-file checks via run0 /opt/saltvenv/bin/salt 'cyclops' cmd.run '…'.
• Vaultwarden intentionally has NO anubis (browser extension / mobile / CLI clients can't solve challenges) — app-level master-password auth only.
• Forgejo app.ini lives inside ~/forgejo/data (GITEA_CUSTOM) — user-managed, not salt-owned; salt only ensures the quadlet + data-dir existence (never prune).
• KitchenOwl data is a live SQLite (WAL mode) under ~/kitchenowl-app/data — restic-covered; don't copy while hot without PRAGMA wal_checkpoint(TRUNCATE) first.
• uWSGI/app IPv6 binds: an app binding [::]:port may not accept IPv4-mapped connections on netavark the way it does on claw — test through the actual front (nginx uwsgi_pass, Caddy) rather than raw HTTP curls to the app port (kitchenowl-back speaks the uwsgi protocol on :5000, not HTTP).
• Slow-VM timers: podman-auto-update.timer and the 04:00 restic system timer both run here — verify after any change (slow-VM 30s health-check misses are a known failure mode).
• openclaw update is broken on this host. The staged swap aborts (Installation recovery is unverified, exit 1) and leaves the gateway failed — the bot goes offline. Update with ~/openclaw-safe-update.sh instead; see §10 for the two root causes and the recovery procedure.

12. Repos

• salt-fleet — all salt states + pillar (source of truth). Push → Forgejo → CI render gate → webhook deploy to claw /srv/salt.
• homelab — docs (this file, claw/*.md, content/) + Forgejo CI builds the docs site.
• fleet-status — machine-generated restic verify reports.
• openclaw-cyclops — OpenClaw config/workspace/skills/wiring for this node, auto-synced every 15 min (see §10).
• Gists (167988534e…, 13c6d2d3a0…) are migrated/frozen — never update, never delete.