HackersHostel — Homelab Config Reference

Ingress node: claw.eazyplugins.com · AlmaLinux 10.2 · Bangladesh (UTC+6) Worker node: cyclops (cyclops.painlesswp.com, Ubuntu 24.04) — WG 10.0.0.2; services behind claw's Caddy over WireGuard Stack: Caddy (reverse proxy, on claw) · Podman (rootless, UID 1000) · Quadlet (systemd) · Anubis (bot protection) SELinux: Enforcing (claw) / AppArmor (cyclops) · DNS: Wildcard *.eazyplugins.com Last updated: 2026-09-07

Infrastructure

Caddy @skip Matcher

Private IPs bypass Anubis — internal access works without bot challenges:

@skip {
    remote_ip 127.0.0.1 ::1 10.0.0.0/8 172.16.0.0/12 192.168.0.0/16
}
reverse_proxy @skip localhost:<backend_port>
reverse_proxy localhost:<anubis_port> {
    header_up X-Real-Ip {remote_host}
}

Deployed Services (Sep 2026)

Web Applications (with Anubis)

Web Applications — Anubis on claw host-net → cyclops over WireGuard

Web Applications (Anubis on host network)

Direct Access — NO Anubis

Non-Web Services

Removed / Retired

• CrowdSec + firewall-bouncer — removed Aug 2026 (RAM + log spam; logrotate now handles syslog)
• TeamSpeak 3 (mbentley/teamspeak) — replaced by TS6 Beta
• OpenClaw — was Cyclops-era; not present on Claw

Backups (restic)

Claw data is backed up nightly at 04:00 (root timer restic-backup.timer, script /usr/local/bin/restic-backup-claw.sh, salt state server.backup) to two repos: sftp to cyclops over WireGuard (sftp:nirjhor@10.0.0.2:~/restic-repos/claw) and Backblaze B2 (b2:$BUCKET:/claw, client-side encrypted). Covers all app data dirs, named volumes, pg_dumps, Caddyfile, salt tree. Retention: hourly 24 / daily 7 / weekly 5 / monthly 6, pruned by the nightly job only. Every backup script in the fleet runs plain restic unlock (dead-PID locks only) before touching a repo, so a crashed or manual restic invocation can never fail a scheduled leg.

Radicale gets an extra hourly increment (restic-radicale-hourly.timer at :45 → /usr/local/bin/restic-radicale-hourly.sh): calendar data changes all day, so the nightly-only 24h RPO was too coarse. Same two repos, tagged hourly, dedupe keeps the cost trivial (~1MB/h worst case). The Sunday 09:30 restic-verify.timer drill (restic check + real restic dump restore proof) covers these repos; reports land in the fleet-status repo.

Secrets (repo passwords, B2 keys) live in claw salt pillar under server:restic — never in the repo.

Oddities & Quirks

1. Rootless Podman UID Mapping

Container UID 65534 (nobody) does not map to host UID 65534. The mapping is:

Container UID 65534 → Host UID 524288 + 65534 - 1 = 589821

Verify with podman exec <container> cat /proc/self/uid_map:

0       1000          1
         1     524288      65536

Affected: PrivateBin (runs as nobody inside container). Data dir must be owned by host UID 589821, not 65534.

Fix in Quadlet:

[Service]
ExecStartPre=sudo chown -R 589821:589821 /home/nirjhor/privatebin-app/data/

Consequence for backups: data files are 0600 owned by a container-mapped subuid (e.g. 525286), so a tar run as the normal user silently skips them — 46 of 51 files in the YourSpotify mongo dir. The archive still looks plausible, so compare file lists rather than trusting the exit code. run0 is unreliable for this (transient Remote peer disconnected). Use podman unshare instead: it enters the user namespace, where those subuid-owned files are readable with no host root.

podman unshare tar czf backup.tar.gz -C /home/nirjhor/your_spotify your_spotify_db

Verify by comparing the archive against podman unshare find <dir> -type f — the user-level find cannot traverse 0700 subdirs such as diagnostic.data/, so it under-reports as well.

2. SELinux Enforcing + Podman

Bind mounts need :z suffix for SELinux relabeling:

Volume=/home/nirjhor/privatebin-app/data:/srv/data:z

Without :z, SELinux blocks container writes even with correct Unix ownership.

3. Anubis Cannot Use 127.0.0.1 on Default Network

This is the biggest gotcha. Anubis containers on the default pasta network targeting http://127.0.0.1:<port> will fail with connection refused. The 127.0.0.1 refers to the container's own loopback, not the host.

Three correct patterns:

Pattern A — Quadlet services on shared network

Both Anubis and backend on homelab-internal, Anubis targets by container name:

# Anubis
[Container]
Network=homelab-internal
PublishPort=8907:8923
Environment=BIND=:8923
Environment=TARGET=http://privatebin:8080

Pattern B — Compose services

Anubis joins the compose network (both must share the same custom bridge network):

[Container]
Network=your_spotify_default
PublishPort=3003:8923
Environment=BIND=:8923
Environment=TARGET=http://your_spotify_client:3000

⚠️ Network rule: The compose stack must define a named custom bridge network, and the Anubis Quadlet must join that same network. The default pasta network does not work.

Pattern C — Host-network backends

Anubis uses Network=host and binds to localhost:

[Container]
Network=host
Environment=BIND=127.0.0.1:8904
Environment=TARGET=http://127.0.0.1:8000

Do NOT do this (broken):

# ❌ WRONG — Anubis on pasta network, can't reach host 127.0.0.1
[Container]
PublishPort=8907:8923
Environment=TARGET=http://127.0.0.1:8086

4. Anubis X-Real-Ip Header Required

Anubis returns HTTP 500 if X-Real-Ip header is missing. Caddy must forward it:

reverse_proxy localhost:<anubis_port> {
    header_up X-Real-Ip {remote_host}
}

5. Flame Dashboard

No sqlite3 in container — must copy DB out, edit on host, copy back:

podman cp flame:/app/data/db.sqlite /tmp/flame.db
sqlite3 /tmp/flame.db "UPDATE apps SET isPinned = 1 WHERE isPinned = 0;"
podman cp /tmp/flame.db flame:/app/data/db.sqlite
podman restart flame

PUT API returns 401 even with valid Authorization: Flame <key> header. Direct SQLite edits are the only reliable way.

Unpinned apps hidden — New apps default to isPinned=0 and only show in the collapsed "Other Apps" section. Set isPinned=1 to show on main dashboard.

Broken MDI icons — Some Material Design Icons don't render (e.g., photo_library). Use favicon URLs as fallback:

UPDATE apps SET icon = 'https://example.com/favicon.ico' WHERE name = 'ServiceName';

Password with ! — Use single quotes in shell to avoid history expansion.

6. Dozzle Healthcheck

Built-in healthcheck checks Docker socket connectivity. It reports unhealthy on Podman even though the service works fine. Ignore the status.

7. Glances Container Monitoring

Glances needs the Podman socket mounted to list containers:

Volume=/run/user/1000/podman/podman.sock:/var/run/docker.sock:ro
PodmanArgs=--pid=host

The --pid=host flag is required for process monitoring.

8. podman-auto-update

podman-auto-update.timer runs daily (~20:45). 62 of the 64 Quadlet units carry AutoUpdate=registry + Label=io.containers.autoupdate=registry. Works. Compose stacks are NOT covered — they have their own weekly *-update.timer services (ActualBudget, Uptime Kuma, YourSpotify) doing podman compose pull && up -d --force-recreate. Jitsi also has a weekly jitsi-meet-update.timer (added 2026-09-21), but it does NOT use that bare pattern — see Quirk 10.

Cyclops side (verified 2026-09-21): cyclops runs its own podman-auto-update.timer (system unit, daily ~00:00, ExecStartPost=podman image prune -f) and all 10 of its quadlets carry AutoUpdate=registry — confirmed by podman auto-update --dry-run, which lists every cyclops container under the registry policy. This is not theoretical: Vaultwarden was bumped to 1.37.3 (Web-Vault 2026.7.0) by the 2026-09-14 run. Note the two nodes therefore have separate auto-update timers, both daily — claw ~20:45, cyclops ~00:00.

9. Logrotate

/etc/logrotate.d/syslog — weekly, rotate 2, maxsize 200M, covers messages/secure/btmp/wtmp with rsyslog HUP postrotate. Installed Aug 2026 after 1.8GB of unrotated CrowdSec+Hermes spam accumulated.

10. Jitsi Meet

Managed by podman-compose@docker-jitsi-meet.service. Config at ~/.jitsi-meet-cfg/ (owned 1000:1000). Images pinned stable-11248 (2026-09-21) from ghcr.io/jitsi/. JVB UDP 10000. No firewalld.

Auto-update (added 2026-09-21): jitsi-meet-update.timer (weekly, Persistent=true) runs ~/jitsi-safe-update.sh, which resolves the newest released stable-* tag from GitHub, compares it to the JITSI_IMAGE_VERSION pin in .env, and only then backs up .env, pulls, recreates and health-checks — rolling back to the previous tag if the stack does not come up. The bare fleet pattern (podman compose pull && up -d --force-recreate) is deliberately NOT used here: the .env pins a specific stable-<n> tag, which never moves, so a plain pull is a permanent no-op — and on the rolling stable tag Jitsi occasionally ships config-affecting changes. The script keeps .env as a truthful record of the deployed release and reuses the old images for a real rollback.

The pre-existing jitsi-meet-update.service was dead on arrival: it pointed WorkingDirectory at %h/jitsi-meet, which does not exist (the project lives at %h/jitsi/docker-jitsi-meet), so even if enabled it would have failed — and it was never enabled. Fixed and enabled with the above.

Health check used by the script (a "running container" is not proof of a working stack): all 4 containers up, jicofo logging Added new videobridge (proves JVB registered — the usual silent failure), and https://meet.eazyplugins.com/ returning 200.

In-place ICE restart enabled (2026-09-21): ENABLE_ICE_RESTART=1 and ENABLE_ICE_RESTART_ON_NETWORK_CHANGE=1 in .env, so an ongoing call survives a network change (WiFi↔cellular, brief drop) instead of dying. Both are off by default upstream — 2.0.11248 ships the client option but leaves it false, so the capability was dormant until set. Only the web service reads these vars (it generates config.js); JVB and jicofo gained unconditional in-place ICE restart support in the same release, so the flag is the whole switch. Verified served: config.enableIceRestart = true / enableIceRestartOnNetworkChange = true. Note the auto-update script only ever rewrites the JITSI_IMAGE_VERSION line, so these flags survive upgrades.

⚠️ Where Jitsi's live config actually lives (2026-09-21): the images render templates from /defaults/ into /run/<component>/config/ at every container start, and that is what the processes read (jicofo runs with -Dconfig.file=/run/jicofo/config/jicofo.conf). The host bind-mounted ~/.jitsi-meet-cfg/<c>/*.conf files are a stale snapshot — on this box every one of them is frozen at 2026-08-14 — and are not the running config. Reading them will make a correctly-applied change look like it silently failed: setting ENABLE_CODEC_OPUS_RED left /config/jicofo.conf with an empty audio {} block while /run/jicofo/config/jicofo.conf had opus.red.enabled = true all along. Diagnose from /run/<c>/config/, from the served config.js, or from podman exec <c> env — never from the host config dir.

Opus RED enabled (2026-09-21): ENABLE_OPUS_RED=1 (web → config.enableOpusRed) and ENABLE_CODEC_OPUS_RED=1 (jicofo → jicofo.conference.audio.opus.red.enabled). In-band redundancy (RFC 2198) so packet loss degrades into something recoverable rather than robotic audio — costs 30-50% of audio bitrate (+10-20 kbps/stream), negligible against the ~116 Mbps uplink measured on this box. Both halves are required; setting only one silently does nothing. Verified: served config.enableOpusRed = true, runtime opus.red.enabled = true, bridge registered, 0 errors.

Permissions-Policy killed microphone (2026-06-03): Caddy's (security) snippet must keep Permissions-Policy "geolocation=()" — adding camera=(), microphone=() blocks getUserMedia in Chrome. Site-level header directives cannot override snippet block-style header directives; edit the snippet source.

Java heaps capped (Aug 2026): JVB_JAVA_OPTIONS=-Xmx512m -Xms128m and JICOFO_MAX_MEMORY=256m in .env (was 3072m each). Fine for 2-5 person calls.

CRITICAL: anubis-jitsi is the Anubis PoW proxy — NOT a Jitsi instance. docker-jitsi-meet_* containers are the real backend. Never compose down the real stack thinking it's obsolete.

11. RustDesk (Quadlet)

Not a web service — no Caddy or Anubis. Network=host:

• hbbs (rustdesk-hbbs.container): TCP 21115, 21116, 21118 + UDP 21116
• hbbr (rustdesk-hbbr.container): TCP 21117, 21119

Public key: HdN2MHZj3faACaA5RICY6yvsl0GyUKS0z4trHMqefi4= Data: ~/.local/share/containers/storage/volumes/rustdesk_rustdesk_data/

12. Uptime Kuma — :latest is v1, not v2

The :latest Docker tag is pinned to the v1 branch. v2 requires the explicit :2 tag (current: louislam/uptime-kuma:2).

13. Tandoor Gunicorn Workers

GUNICORN_WORKERS env var (not WEB_CONCURRENCY) controls workers — confirmed via boot.sh inside the container. Set to 1 in the Quadlet (Aug 2026), saved ~218MB RAM.

14. Hermes Services

• hermes-dashboard.service — dashboard on 0.0.0.0:9119 (host net), fronted by anubis-hermes (BIND 127.0.0.1:8918)
• hermes-gateway.service — messaging gateway (Telegram)

15. DeepSeek Harness — tried and removed (Aug 28, 2026)

Built dsh (deepseek-harness) from source into a Podman image, quadlet + Caddy vhost deepseek.eazyplugins.com. Failed end-to-end: app hard-blocks 0.0.0.0 binds, --trusted-host fixed the host-fence 403, but the settings/provider UI stayed broken behind reverse proxy ("settings are unavailable in this browser"). Removed entirely — container, image, quadlet, volume, Caddy block, cron.

16. Diun — alerts must name the host that runs the image

Diun runs on claw and watches a flat, hand-maintained image list (~/.local/share/diun/diun-images.yml). The default notification template ends with the host that ran the check — always claw — which reads as if claw hosts the image. It doesn't; diun has no concept of where an image is deployed, only which registries to query.

Since 2026-09-14 every entry carries metadata.host (claw, cyclops, or claw+cyclops for images deployed on both) and notif.telegram.templateBody renders it as a Runs on: line. Missing annotation prints an explicit "(not recorded in diun-images.yml)" fallback, so the field fails loudly instead of silently attributing the image to the wrong node. The mapping was verified against live podman ps on both nodes; annotate or re-verify whenever a service moves between hosts.

Known gap: the list covers 30 images only — cyclops's forgejo, outline, postgres:17-alpine and redis:7-alpine are not watched at all, nor are claw's bookorbit, pgvector, mongo or the postgres/redis variants. A second Diun instance on cyclops would fix both the attribution and the gap (it would also make the hostname truthful by construction).

Backups from the change: ~/.local/share/diun/*.bak-*.

17. Salt states for rootless units — three silent-convergence traps (fixed 2026-09-21)

All of claw's services are rootless user units, but salt-call --local runs as root. That mismatch produced three separate ways for server.quadlets to claim success while doing nothing — all found by applying an unrelated image bump, all now fixed:

1. service.running cannot see user units. The quadlet macro used service.running for 21 services. It resolves against the system systemd, so every apply reported The named service X.service is not available — 21 failures per run, while the containers were running fine. state.apply test=True hides this (it reports "not present" instead), which is why it survived until someone ran a real apply. Now driven explicitly: sudo -u USER env XDG_RUNTIME_DIR=/run/user/1000 DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus systemctl --user … (USER = the owning user)
2. enable --now never restarts a running unit. A quadlet change was written, reported as success, and the old container kept running — silently skipping the image or config change. Fixed with a <name>-restart state guarded by an onchanges dependency on the quadlet file.
3. cmd.run's unless is evaluated as root too. The homelab-internal-network guard checked podman network exists as root, which cannot see rootless networks → the guard always "failed" → every apply tried to re-create an existing network (podman exits 125 = failed state). The guard now queries the user's podman explicitly.

Consequence: before these fixes server.quadlets could not complete a clean run at all. Treat test=True as necessary but not sufficient — it under-reports this entire class of failure. After the fix: 67 states, 0 failures on a real run.

18. Phantom units — check for dangling .wants symlinks (fixed 2026-09-21)

systemctl --user list-timers --all listed vaultwarden-update.timer with every column -, and is-enabled returned not-found. The cause was not an empty unit file but a dangling symlink: ~/.config/systemd/user/timers.target.wants/vaultwarden-update.timer pointed at a unit that no longer exists (Vaultwarden moved to cyclops, and its units were deleted on claw while the enable-symlink was left behind). A companion leftover, ~/.config/systemd/user/podman-compose@vaultwarden-app.service (a real file, disabled, not in default.target.wants), still pointed at ~/vaultwarden-app — which would have started a second Vaultwarden on claw against a stale database had anyone enabled it. Both removed; ~/vaultwarden-app/data/ (a pre-migration snapshot holding db.sqlite3 + rsa_key.pem, last written 2026-06-12) was deliberately left untouched.

Lesson: when a service is decommissioned, systemctl --user disable it before deleting the unit files — deleting files alone leaves a dangling enable-symlink that makes the unit look present-but-dead in every listing. To audit for this class: find ~/.config/systemd -xtype l lists every broken symlink.

19. MongoDB major upgrades — fcV, and the moving mongo:8 tag (2026-09-23)

MongoDB upgrades one major at a time. The data dir carries a featureCompatibilityVersion (fcV) and a newer major refuses to start if that document is too old — the error names the versions it accepts:

UPGRADE PROBLEM: ... Invalid feature compatibility version value '6.0'. Expected one of the following versions: '8.3', '8.2', '8.0'

So 6.0 → 8.0 is two hops (6.0 → 7.0 → 8.0). Each hop: swap the image → confirm the server starts and the data reads → then db.adminCommand({setFeatureCompatibilityVersion: "<x.y>", confirm: true}).

The trap: mongo:8 is no longer the 8.0 line — it currently resolves to 8.3.11, whose accepted-fcV list is only 8.3/8.2/8.0, so it cannot start on a 7.0 data dir at all. The 8.0 line accepts 7.0/7.3/8.0. Pin mongo:8.0 explicitly; a bare mongo:8 fails the upgrade with a confusing "Wrong mongod version". Check what a tag really points at first: podman run --rm <image> --version.

8.0 is a Major, 8.3 is a minor — deliberately staying on 8.0. (Corrected: earlier notes in this repo called 8.3.11 a "rapid release"; it is not. 8.1/8.2 were rapid releases and are both EOL; since 8.2 MongoDB ships minor releases, and 8.3 is the current one — "as stable as major releases and suitable for production workloads".) Two reasons not to move:

• MongoDB scopes on-premises minor releases to specific use cases — Search, Vector Search, enhanced Queryable Encryption. YourSpotify uses none of them, so 8.3 buys nothing here.
• MongoDB's own guidance: "If you want a predictable support window and manual control over upgrades, choose a Major release." 8.0 is the Major (5-year lifecycle, full on-prem support).

It is technically reachable now — 8.3 starts happily on a fcV-8.0 data dir (tested 2026-09-23 on a synthetic dir; mongod startup complete). It failed before the upgrade only because the dir's fcV was too old. So this is a deliberate choice, not a limitation. Note minor→minor upgrades must be sequential (none may be skipped), so adopting a minor line is a path you keep walking.

Three places pinned mongo:6 and all had to change — the live quadlet, the salt state (server/stacks.sls), and ~/your_spotify/docker-compose.yml. That last one matters because yourspotify-update.timer runs podman compose pull && up -d --force-recreate weekly, which would have silently reverted the pin.

Traps met while diagnosing a failing start:

• Quadlets run podman run --rm, so a container that exits is deleted — podman logs and podman ps -a then report "no container ... found", and mongod's output never reaches the journal (podman runs it detached). To capture a failing start, run the image manually against a copy of the data.
• Always test images against data with the :z label. Without it SELinux blocks the entrypoint's chown/find and a healthy image looks broken (find: '/data/db': Permission denied).

YourSpotify: 6.0.28 → 7.0.43 → 8.0.32 (fcV 6.0 → 7.0 → 8.0), data dir 511 MB → 399 MB. Pre-upgrade backup + scripts in ~/backups/mongo-upgrade-20260923/.

20. Image hygiene — the prune traps (2026-09-23)

podman image prune is not the whole story, and a blanket podman image prune -a on this fleet is destructive. Two traps, both hit for real:

1. buildah working containers hide in plain sight and pin images. podman image prune -f silently removed nothing while a 3.56 GB dangling layer sat there, because six abandoned buildah working containers were holding it and node:22-slim. They are invisible to podman ps -a and podman ps -a shows only a "no container" for them — list them with podman ps -a --external (they show status Storage). They came from an abandoned attempt to containerise Hermes (~/.hermes/hermes-agent/Dockerfile, 2026-08-25); Hermes actually runs natively from ~/.hermes/hermes-agent/venv/bin/hermes via systemd, so the whole build was cruft. Removal order matters: podman rm -f the working containers first, then the dangling layer and its base image become prunable.

2. Never blanket-prune on cyclops — two images are load-bearing and one cannot be re-pulled.

• localhost/salt-ci:latest (1.9 GB) — a locally built image, referenced by the salt-fleet CI workflow (runs-on: salt-ci in .forgejo/workflows/salt-check.yml). The Forgejo runner consumes it on demand, so no container is ever long-lived and it always looks unused. It cannot be re-pulled — only rebuilt.
• ghcr.io/catthehacker/ubuntu:act-latest (1.6 GB) — the runner's CI image, same on-demand pattern.

Kept deliberately on claw, both unused by containers but earn their space:

• mongo:7 — the only local image that can open the fcV-6.0 pre-upgrade backup; mongo 8.0 cannot.
• jitsi stable-11146-2 was kept here for jitsi-safe-update.sh rollback, but it is no longer local (verified 2026-10-11: only stable-11248, the current pin, is present). rollback() re-pulls, so a failed update still rolls back — via re-pull rather than a local image. The design premise ("old images already there") no longer holds; treat rollback as re-pull-based.

Result: claw 47 GB → 39 GB (61 → 42 images, 0 dangling); cyclops 27 GB → 26 GB (14 → 11 images, 0 dangling). ~9 GB fleet-wide. All 64 claw + 10 cyclops containers verified still running after, and every container's image confirmed still present. Note podman images "SIZE" over-counts shared layers — the disk delta is the honest number, and it is always smaller than the sum of the removed images.

21. Jev urgency gate — a calibrated alert brain (2026-09-27)

Problem: warning signals nobody watches (OpenClaw's own memory-pressure warnings fired twice unnoticed; disk creep; restic hiccups). Options were ignore → find out at outage, or ping-everything → alert fatigue. Hermes-as-judge works but is a full agent turn per check.

Shipped: ~/.hermes/scripts/jev-urgency-gate.sh + jev-gate.{service,timer} (user unit, hourly, Persistent=true). Hourly sweep of claw (disk, swap, failed units, OOM kills, OpenClaw memory-pressure warnings, restic nightly + radicale hourly log tails) and cyclops over SSH (same signals) → hour-over-hour deltas → one Jev call (typesafe/jev via CommandCode Provider API) with page_nirjhor (noul probability) + severity (score: ignore / digest / page) → route: p > 0.8 → Telegram page, 0.4–0.8 → ~/.hermes/jev-gate/digest.txt, < 0.4 → silent. Every judgment in ~/.hermes/jev-gate/gate.log. Runs 24×/day for fractions of a cent ($0.04/M input tokens).

Tuning lesson (measured live): raw absolute signals page falsely — "swap_used=100%" scored p(page)=0.95 on an all-green fleet. Reframing the same state with deltas + baseline context ("100% of 512 MB swap is the weeks-stable baseline, only a RISING number matters; dnf-makecache is a known EPEL-mirror transient") dropped it to p=0.5 → digest. Jev weights context, so feed context, not levels. Baselines text lives in the script's STATE heredoc — update it there when baselines move.

Key/auth note: the CommandCode key is read from ~/.hermes/.env (HERMES_CUSTOM_API_COMMANDCODE_AI_API_KEY) at runtime and the Authorization header is built in the script — no token in the script itself. ⚠️ Agent tool output redacts Bearer <var-pattern> to Bearer *** when grepping/reading the script — the file bytes are fine; verify auth by behavior (curl -w '%{http_code}'), never by grepping for the token.

Digest delivery: the morning cron/agent should read ~/.hermes/jev-gate/digest.txt (it currently requires asking). Thresholds live in the script's route block if tuning is needed.

Service Directory Structure

~/
├── .config/containers/systemd/     # 64 Quadlet .container files + anubis@.container template
├── .config/systemd/user/           # podman-compose@.service, *-update.{service,timer} overrides
├── privatebin-app/data/            # PrivateBin paste storage
├── beaverhabits-app/beaver/        # BeaverHabits SQLite DB
├── radicale-app/{config,data}      # Radicale CalDAV data
├── teamspeak-app/                  # (legacy TS3 data) — TS6 uses named volume ts6-data
├── tandoor-app/docker-compose.yml  # + named volumes tandoor_tandoor_*
├── vikunja-app/                    # vikunja_default network
├── memos-app/memos-data/           # Memos files (DB in postgres)
├── linkwarden-app/docker-compose.yml
├── your_spotify/docker-compose.yml # + your_spotify_db/
├── uptimekuma-app/{data,docker-compose.yml}
├── actualbudget-app/               # named volumes actualbudget-app_*
├── jitsi/docker-jitsi-meet/        # compose + .env (heaps capped)
├── .jitsi-meet-cfg/                # Jitsi config (owned 1000:1000)
├── srv/audiobookshelf/             # /srv/audiobookshelf/{config,metadata,audiobooks,podcasts}
├── .local/share/
│   ├── searxng/                    # SearXNG settings.yml
│   ├── flame/data/                 # Flame DB + uploads
│   ├── paperless/{data,media,export,consume}
│   ├── lubelogger/{data,keys}
│   ├── diun/                       # diun.yml + diun-images.yml
│   └── containers/storage/volumes/ # postgres/redis/media named volumes

Cap — Screen Recording Stack (added 2026-09-26)

Self-hosted Loom alternative (github.com/CapSoftware/Cap). 5 containers on claw, rootless quadlets:

• URL: https://cap.eazyplugins.com (Caddy -> anubis 3005 -> cap-web 3030)
• Secrets: ~/cap/.env (0600) + podman secrets (cap-mysql-password, cap-mysql-root-password, cap-minio-user, cap-minio-password). Garage keys: capkey (id [REDACTED]) in ~/cap/garage.toml (0600, replace:False in salt)
• Salt: server.cap state (quadlets + policy; env/garage guarded by replace:False). Salt does NOT manage podman secrets or garage layout/bucket/key — one-time deploy steps.
• Garage S3: public endpoint https://cap-minio.eazyplugins.com (Caddy direct -> 127.0.0.1:3027, NO anubis — S3 clients can't solve PoW). Presigned multipart uploads verified end-to-end (SigV4).
  • GOTCHA: Garage v1.x rejects UNSIGNED-PAYLOAD/STREAMING hashes; fixed by v1.0.1 -> v2.4.1 (data/layout survive; bind mount ~/cap/minio).
  • GOTCHA 2: AWS SDK v3 presigner embeds x-amz-checksum-mode/crc32 of an EMPTY body into presigned URLs (flexible checksums, default = WHEN_SUPPORTED) — any raw PUT against such a URL gets 400 InvalidDigest. Fixed with AWS_REQUEST_CHECKSUM_CALCULATION=WHEN_REQUIRED + AWS_RESPONSE_CHECKSUM_VALIDATION=WHEN_REQUIRED in ~/cap/.env.
  • GOTCHA 3: Cap's env contract is S3_PUBLIC_ENDPOINT/S3_INTERNAL_ENDPOINT (+ CAP_AWS_* for creds). S3_PUBLIC_URL is playback-only — without S3_PUBLIC_ENDPOINT, presigned upload URLs default to amazonaws.com.
• Email: Resend API. RESEND_API_KEY (reused from Outline's Resend account on cyclops, ~/outline/outline.env SMTP_PASSWORD — same key) + RESEND_FROM_DOMAIN=support.painlesswp.com. Login = 6-digit code to your email.
  • GOTCHA: eazyplugins.com is NOT verified in Resend — sends from it fail with 403 (silent: browser gets 302, error only server-side). Only support.painlesswp.com is verified. To send from eazyplugins.com: add Resend's DKIM/SPF/MX records at registrar, verify at resend.com/domains, then flip RESEND_FROM_DOMAIN.
  • Test sign-in end-to-end: GET /api/auth/csrf for csrfToken, POST urlencoded email+csrfToken to /api/auth/signin/email.
• STATUS (2026-09-27): uploads + share-link playback working end-to-end. Gotchas 1–3 above were the full root-cause chain; Caddy answers CORS preflights on the S3 vhost (Garage has no native CORS).
  • Flame icon: Flame bundles @mdi/js v5 — videocam (v6 name) doesn't exist there; use video-outline.

Anubis Port Allocation (24 instances, v1.28.1)

All instances: container listens on :8923 (BIND), published to the unique host port. Metrics disabled or bound to 127.0.0.1. Shared ED25519 key across all 23 (anubis-cap has its own key — generated at deploy). OG_PASSTHROUGH=true, COOKIE_EXPIRATION_TIME=168h, USE_REMOTE_ADDRESS=false. FreeLLMAPI app is live (freellm.eazyplugins.com, port 3004) but its anubis front was removed 2026-09-03 — direct reverse_proxy, no anubis-freellmapi instance.