Claw — Changelog

2026-10-19

  • Forgejo runner CI repaired + codified — two silent-failure classes fixed on cyclops: (1) salt-ci CI image had been pruned (every salt-fleet render check failed in ~2s for 12 days while webhook deploys masked it) — rebuilt from tools/salt-ci.Dockerfile; (2) runner offered no ubuntu-latest label, so any GitHub-style workflow (commandcode-runs) queued forever — label added mapping to the already-local catthehacker act image. runner/config.yml is now salt-managed (minion/runner-config.yml) so labels survive rebuilds; runner restart wired to config changes. Verified live: salt-fleet run 110 + commandcode-runs run 9 green (125 cucumber scenarios / 535 steps — the stateful-fake-host apply-path suite). Forgejo v17 gotcha logged: Actions API id/index collision makes rerun-by-API re-run the wrong run; retrigger with an empty commit instead.
  • Forgejo v16.0.6 → v17.0.0 on cyclops (v16 EOL 2026-10-29; v17 supported to 2027-01-28) — salt bump forgejo:16→:17 (salt-fleet ebf098a), webhook-deployed, state applied clean (5/5). Pre-upgrade data backup forgejo-data-pre-v17-20261019.tgz (57 MB, sha256 4ed958f3…) on claw ~/backups/ + cyclops /tmp, taken via podman unshare tar with the service stopped (UID-mapped secrets unreachable as plain user; root not needed). Docs: cyclops README + docs-site tables → :17. Breaking-change audit: action-run webhook events deprecated in favor of workflow-run events — the fleet's salt-fleet push webhook is push-event based, unaffected. Ops note: forgejo's git SSH (port 2222) is the container itself — pushes to the salt-fleet/homelab repos fail while forgejo is stopped (cost one failed push during the backup window).

2026-10-11

  • Fleet standardized on mise + Ruby 4.x — mise (v2026.10.7) installed on both nodes, ruby@4.0.7 via precompiled ruby-build (YJIT on; ZJIT not in precompiled builds — deliberate, see bench), weekly mise-update.timer (user unit, Sun 09:00 +06) on each node. Distro rubies retired: claw ruby 3.3 + ruby4.0 (dnf, zero reverse-deps), cyclops ruby3.2/ruby-full + 8 orphan leaves (apt). ruby resolves fleet-wide via mise activate (bashrc) + shims PATH (profile — Ubuntu's bashrc early-returns non-interactive shells, which is why both are needed). Salt: new fleet-wide server/mise.sls in top.sls for both nodes; gotcha rediscovered: deploy.sh maps flat claw/salt/*.sls → /srv/salt/server/<name>.sls, so top must say server.mise (a - mise ref = "No matching sls"); webhook staging lag hit twice — byte-diff before applying. Before removal on claw: rpm --whatrequires empty + zero ruby-shebang scripts fleet-wide. Benchmark first (~/bench-jit-20261011/): 4.0's rewritten json lib is 2.4× faster than 3.3, ZJIT never beat base/YJIT at 4.0.6 — upgrade justified by the stdlib, not the JITs.
  • Diun vaultwarden watch removed + claw KitchenOwl leftovers decommissioned (2026-10-11) — Diun: dropped vaultwarden/server:latest from the watch list (30 entries now) — Vaultwarden lives on cyclops and its claw watch entry was the likely re-puller of the stale claw image removed earlier today; cyclops keeps updating it via its own AutoUpdate=registry quadlet. KitchenOwl: removed the claw-hosted kitchenowl-back/kitchenowl-web containers, quadlets and the empty kitchenowl_default network — the public vhost has routed to cyclops (10.0.0.2:3013 via anubis-kitchenowl) since the 2026-09-07 migration, and the safety evidence was checked before removal: 0 HTTP requests to claw's backend in 72h of logs, and cyclops' DB is strictly newer (max item id 663 / latest 2026-10-09 vs claw's 656 / 2026-09-11 — no unique claw data). ~/kitchenowl-app/data/ preserved untouched per the data-preservation rule. Verified post-removal: kitchenowl.eazyplugins.com 200, anubis-kitchenowl up, cyclops containers untouched. Claw now: 64 quadlets (62 AutoUpdate-labeled), 67 containers. Counts synced across config.md/guide.md/content pages.
  • Doc truth-sweep + claw disk cleanup — removed the regenerable pip cache (2.9 GB) and a re-pulled vaultwarden/server:latest image (303 MB; claw copy stale since the cyclops migration — likely re-pulled by Diun, which still watches vaultwarden/server:latest; removing that watch entry is an open decision): claw 44 GB → 41 GB (56% → 52%), all 69 containers + every image verified intact. Re-derived every current-state count from live and fixed drift: config.md anubis 23→24 instances (header was stale vs its own port table), quadlets 61→66 (64 AutoUpdate-labeled) in config.md + guide.md, Diun 27→31 images in config.md + guide.md, content pages 61/62→66/69, salt-fleet pillar comment pre2→v1.28.1, RUNBOOK claw total ~62→~69 (24 anubis). Jitsi stable-11146-2 rollback image verified no longer local — Quirk updated (rollback() re-pulls; the "old images already local" premise is gone). Anubis port table + service tables verified correct against live (24/24 match).

2026-10-09

  • Anubis fleet v1.28.1 "Wuk Lamat Echo 1" (stable, first 1.28 stable line): bumped claw 24/24 + cyclops 2/2 from v1.28.0-pre3. Adds stable WASM PoW framework (challenges still disabled by default), 1.28.0 skipped due to systemd-unit-breaking change reverted in .1. Pre3 security content retained (GHSA-5gwm-367w-7fgj, Thoth→GeoIP). All images/quadlets/salt/diun/docs synced; pre3 images pruned both nodes.

  • Jev urgency gate deployed (2026-09-27) — hourly jev-gate.timer (user unit) sweeps claw + cyclops (disk, swap, failed units, OOM kills, OpenClaw memory-pressure warnings, restic legs), computes hour-over-hour deltas, and asks the Jev decision model (typesafe/jev via CommandCode Provider API, key from ~/.hermes/.env) for a calibrated page probability: p > 0.8 → Telegram page, 0.4–0.8 → ~/.hermes/jev-gate/digest.txt, < 0.4 → silent; every judgment logged to gate.log. Cost ~fractions of a cent/day. Live-tuning lesson: raw absolute signals page falsely (swap=100% baseline scored p=0.95); feeding deltas + baseline context ("weeks-stable, only rising matters") brought it to p=0.5 → digest. Jev weights context — feed context, not levels. Trigger context: OpenClaw's own memory-pressure warnings had fired twice on cyclops unnoticed (RSS 1.6 GiB vs its 1.5 GiB threshold after ~3 days uptime; heap only ~500 MB — Node RSS bloat, fixed by gateway restart per its own prescribed action, RSS → 550 MB). Docs: config.md Quirk 21, guide.md §15.4.

  • Docs toolchain: Zola 0.23.4 → 0.23.6 + Linkita theme 2026-08-19 → 2026-09-13 (2026-09-23) — done as a paired update because the two release in lockstep: Linkita's 2026-09-12 revision is "updates for Zola 0.23.5" and its 2026-09-13 revision is "updates for Zola 0.23.6", while Zola 0.23.6's changelog entry is literally "Update tera version to support component implicit params" — the enabler for the theme's switch to component implicit params. Bumping one alone risks the theme calling components with signatures the binary no longer accepts. CI pin (.forgejo/workflows/docs.yml) and the vendored themes/linkita/ (167 → 192 files; a plain copy, not a submodule, with no local patches) both updated. Verified by diffing the builds, not by "it built": 15 HTML pages before and after, 0 files removed, 18 added (theme search-index language files), no build errors or warnings, and a text extraction of every page shows the prose byte-identical — the only site-wide change is one footer string, "Powered by Zola" → "Made with Zola". This also de-risked the update: there is no templates/ directory in the repo, so the theme's implicit-params change cannot break local overrides we don't have. Zola 0.23.5 also brings real fixes now in use (taxonomy lookups via slugified names, locale on the date filter, dark borders on resized PNGs, get_hash caching). Documented the pin + procedure in cyclops/README.md so the next person pairs them.

  • Anubis upgraded fleetwide to v1.28.0-pre3 "Wuk Lamat" (2026-10-07) — 24 instances on claw (23 bridge + cap/jitsi/outline/kitchenowl/speedtest/vaultwarden/hermes host-net), all live quadlets bumped) plus anubis-forgejo and the unmanaged anubis-cyclops on cyclops. Headline fix: GHSA-5gwm-367w-7fgj — an attacker controlling PTR records for their IP could set multiple PTRs pointing at search-engine hostnames and bypass Anubis entirely. Also: Thoth removed (THOTH_* envs now deprecated log-only; ASN descriptions come from MaxMind GeoIP — no fleet impact, Thoth was never wired here), retry-once for transient cookie loss (#1916), DYNAMIC_COOKIE_SUFFIX, and a batch of small hardening fixes (WASM challenge fail-open, challenge solutions bound to issuing rule, path-traversal edges, JA4H header Set-vs-Add). Sources bumped in all four canonical spots + diun watch list: /srv/pillar/server.sls, /srv/salt/server/anubis.sls (fallback), /srv/salt/server/quadlets.sls (diun), /srv/salt/minion/anubis-cyclops.sls (was on :latest — now pinned), ~/.local/share/diun/diun-images.yml; local mirror ~/fleet-setup/roots kept in step; the unmanaged anubis-cyclops quadlet edited in place per UNMANAGED.md contract. COOKIE_PARTITIONED=false retained. Verified: every claw instance reports version v1.28.0-pre3 from its startup log, cyclops both instances pre3, 24/24 public vhosts serve the challenge (dozzle 401 = basicauth front door, correct), machine-surface policies intact post-upgrade (jitsi /config.js = 200 JS, bookorbit /api = JSON 404 not challenge, speedtest /backend = raw 200), old pre2 image pruned on both nodes, openclaw front 200.

  • Salt drift audit + jitsi.sls landmine defused (2026-09-23) — a full test=True pass over the whole tree (305 states) reported 0 failures, but reading its pending-content list rather than its failure count exposed a state that would have undone the last two days' work: server.jitsi's jitsi-env hardcodes the entire ~/jitsi/docker-jitsi-meet/.env, and it still pinned stable-11146-1 while live was stable-11248 — plus it lacked all four ENABLE_* flags added 2026-09-21. Applying it would have downgraded Jitsi by 139 commits and silently disabled in-place ICE restart and Opus RED, with salt reporting no errors, and (via watch: file: jitsi-env) restarted the stack to do it. Fixed: pin bumped, the four flags added, and the block reshaped to reproduce the live .env byte-for-byte (the only remaining difference was a one-line # Memory caps comment salt wrote but the live file never had, which shifted every subsequent line; the note now lives as a YAML comment outside contents: where it cannot leak into the file). Verified: jitsi-env now reports "is in the correct state" — a true no-op — so the file cannot be clobbered again. Also documented in jitsi.sls as a CO-OWNERSHIP WARNING. Rule recorded: any state that hardcodes contents: for a file the fleet also edits out-of-band needs this check, because test=True reports such drift as a pending change, never as a failure. Also found and documented (not changed — owner's call): the live stack units are 8 concrete podman-compose@<stack>.service files that no salt state manages; jitsi.sls instead writes the template podman-compose@.service, which does not exist on the box and is used by nothing. And server.stacks still fails 17 states on a real apply while test=True reports 0 — the same root-vs-rootless bug class fixed in server.quadlets on 2026-09-21 but missed in stacks.sls (5 network unless guards running as root + 12 service.running states naming rootless user units). RUNBOOK gained §6.1–6.4 covering all of it. Stale version references corrected across jitsi.sls, anubis.sls, pillar/server.sls, bootstrap-claw.sh, anubis/default-policy.yaml and RUNBOOK.md (anubis 19 instances (v1.27.0) → 23 (v1.28.0-pre2)). Also corrected in the guide: Jitsi was still documented as "manually pinned stable-11146-2", two releases stale and missing the auto-update path entirely.

  • Image cleanup on both nodes (2026-09-23) — ~9 GB reclaimed with zero service impact; all 64 claw + 10 cyclops containers verified still running afterwards and every container's image confirmed still present. claw 47 GB → 39 GB (61 → 42 images, 0 dangling): removed an orphaned 3.56 GB dangling build layer, six abandoned buildah working containers, node:22-slim, mongo:6, mongo:8/8.3 (8.3.11), jitsi stable-11146-1 (×4), vaultwarden/server:latest (Vaultwarden lives on cyclops), authelia:latest (Authelia removed Sep 3), anubis v1.28.0-pre1/v1.27.0/v1.25.0, mbentley/teamspeak (TS3, superseded by TS6), http-https-echo, node:22-slim, and the curl/alpine utility images. cyclops 27 GB → 26 GB (14 → 11 images, 0 dangling): a 601 MB untagged LibreSpeed leftover + anubis v1.28.0-pre1. Trap 1 — a 3.56 GB dangling image that podman image prune refused to remove: it was pinned by six buildah working containers that are invisible to podman ps -a (list with podman ps -a --external; status Storage). They were leftovers of an abandoned attempt to containerise Hermes (~/.hermes/hermes-agent/Dockerfile, 2026-08-25) — Hermes actually runs natively from ~/.hermes/hermes-agent/venv/bin/hermes via systemd, so the build was pure cruft. Order matters: remove the working containers first, then the layer and its base become prunable. Trap 2 — podman image prune -a would have broken CI: on cyclops, localhost/salt-ci:latest (1.9 GB) looks unused because no container ever outlives a CI job (the runner consumes it on demand) — but it is a locally built image that cannot be re-pulled, referenced by runs-on: salt-ci in the salt-fleet workflow; ghcr.io/catthehacker/ubuntu:act-latest (1.6 GB) is the same pattern. Both kept. Also kept deliberately on claw, though unused by containers: mongo:7 — the only local image that can open the fcV-6.0 pre-upgrade backup (mongo 8.0 cannot) — and jitsi stable-11146-2, the rollback target jitsi-safe-update.sh expects to find local. Docs: Quirk 20. Note podman images SIZE over-counts shared layers, so the disk delta is the honest figure and is always smaller than the sum of the removals.

  • YourSpotify MongoDB upgraded 6.0.28 → 7.0.43 → 8.0.32 (2026-09-23) — two hops, because MongoDB only upgrades one major at a time: the data dir's featureCompatibilityVersion gates startup and a newer major refuses a too-old fcV (Invalid feature compatibility version value '6.0'). Each hop was: stop → swap image → confirm the server starts and the data reads → setFeatureCompatibilityVersion. Verified: 8.0.32, fcV 8.0, and all 9 collections and 21 indexes identical to the pre-upgrade baseline (3,281 docs). Data dir shrank 511 MB → 399 MB. Pre-upgrade backup (208 MB, gzip-verified, sha256 recorded, 77/77 files) plus baseline JSON and scripts in ~/backups/mongo-upgrade-20260923/; /data/db is already inside restic's set, so there are two independent copies. Finding — mongo:8 is a trap: it now resolves to 8.3.11, whose accepted-fcV list is only 8.3/8.2/8.0, so it cannot start on a 7.0 data dir at all; the upgrade failed with a confusing "Wrong mongod version" until the pin became mongo:8.0 (accepts 7.0/7.3/8.0). Correction (same day): an earlier version of this entry called 8.3.11 "a rapid release" — wrong. 8.1/8.2 were rapid releases (both now EOL); since 8.2 MongoDB ships minor releases and 8.3 is the current one. Landing on mongo:8.0 is therefore a deliberate choice, not a limitation: 8.0 is the Major (5-year lifecycle, full on-prem support, and MongoDB's own advice for a predictable support window), while 8.3 is a minor whose on-prem availability MongoDB scopes to specific use cases (Search, Vector Search, enhanced Queryable Encryption) that YourSpotify does not use. Verified that 8.3 would now work — it starts on a fcV-8.0 dir (tested on a synthetic dir, mongod startup complete) — it just buys nothing here. Minor→minor upgrades must also be sequential, so adopting a minor line is a path you keep walking. Pinned to mongo:8.0 in all three places that previously held mongo:6 — live quadlet, salt server/stacks.sls, and ~/your_spotify/docker-compose.yml. The compose pin mattered: yourspotify-update.timer runs podman compose pull && up -d --force-recreate weekly and would have reverted the pin. AutoUpdate=registry now tracks 8.0.x patches. App verified end to end — server logs Connected to database !, all three services active, api-spotify and spotify both HTTP 200. Docs: Quirk 19. Also found (not fixed — needs your call): server.stacks has the same latent bug class already fixed in server.quadlets — 17 states fail on a real apply (5 podman network create guards that run the unless check as root and can't see rootless networks, plus 12 service.running states on rootless user units reporting "service not available"). Nothing is broken — the services run from their quadlets — but Salt is not enforcing them. test=True reported 0 failures on the same run, re-confirming the RUNBOOK caveat that test=True under-reports this class.

  • Jitsi: Opus RED enabled (2026-09-21) — in-band audio redundancy (RFC 2198) so packet loss on a lossy last mile degrades into recoverable audio instead of robotic dropouts, at a cost of 30-50% of audio bitrate (+10-20 kbps/stream, negligible against the measured ~116 Mbps uplink). Requires both halves and setting only one does nothing: ENABLE_OPUS_RED=1 (web, → config.enableOpusRed) and ENABLE_CODEC_OPUS_RED=1 (jicofo, → jicofo.conference.audio.opus.red.enabled). Verified: served config.enableOpusRed = true, runtime opus { red { enabled = true } }, bridge registered, 0 jicofo/web errors. .env backed up to .env.bak-20260921-171321. Checked no conference was active before bouncing jicofo (jicofo holds the focus, so recreating it ends live meetings). Finding — where the live config really lives: the Jitsi images render /defaults/ templates into /run/<component>/config/ on every start and the processes read those (jicofo: -Dconfig.file=/run/jicofo/config/jicofo.conf). The host bind-mounted ~/.jitsi-meet-cfg/<c>/*.conf files are a stale snapshot (all frozen at 2026-08-14 here) and are not what runs — reading them made the jicofo half of this change look like it had silently failed (empty audio {} block) when it had in fact applied. Corrected the guidance that said to delete config subdirs before --force-recreate to force regeneration; it is not needed and the host copies are not authoritative. Always diagnose from /run/<c>/config/, the served config.js, or podman exec <c> env.

  • Jitsi in-place ICE restart enabled (2026-09-21) — calls now survive a network change (WiFi↔cellular, brief drop) instead of dropping. ENABLE_ICE_RESTART=1 + ENABLE_ICE_RESTART_ON_NETWORK_CHANGE=1 in .env; both default to false upstream, so the capability shipped in 2.0.11248 but was dormant. Only the web service reads the vars (it generates config.js) — JVB and jicofo have the in-place ICE restart support built in unconditionally. Verified on the served config: config.enableIceRestart = true, enableIceRestartOnNetworkChange = true. .env backed up to .env.bak-20260921-162058 first. Required --force-recreate web: podman compose up -d does not recreate a container when only its env changes, so the first attempt silently did nothing (web still showed the old uptime and no ICE env vars) — worth remembering for any future .env change on this stack. The recreate also restarted prosody and jvb; JVB re-registered cleanly (jicofo logged "Lost a bridge" then "Added new videobridge", version 2.3.318).

  • Claw phantom-unit leftovers removed (2026-09-21) — vaultwarden-update.timer was reported by me as "empty files"; that was wrong twice over. It was a dangling enable-symlink in timers.target.wants/ pointing at a unit that had been deleted (Vaultwarden moved to cyclops; its units were removed on claw but the enable-symlink was left behind), which is why the unit showed as present-but-dead in every listing (is-enabled → not-found). Removed, along with podman-compose@vaultwarden-app.service — a real, disabled, decommissioned instance file still pointing at ~/vaultwarden-app, which would have started a second Vaultwarden on claw against a stale database if ever enabled. ~/vaultwarden-app/data/ (pre-migration snapshot: db.sqlite3 + rsa_key.pem, last written 2026-06-12) was left untouched per the data-preservation rule. Separately confirmed Vaultwarden does not run on claw at all; on cyclops it is already correctly auto-updating — quadlet carries AutoUpdate=registry, podman auto-update --dry-run lists it under the registry policy, and the 2026-09-14 run actually bumped it to 1.37.3 / Web-Vault 2026.7.0. Audit recipe for this class of leftover: find ~/.config/systemd -xtype l.

  • Jitsi Meet updated to stable-11248 and brought under automatic maintenance (2026-09-21) — was one stable release behind on stable-11146-2 (2026-09-03). Components moved with it: JVB 2.3-307 → 2.3-318, jicofo 1.0-1189 → 1.0-1205, meet-web 1.0.9365 → 1.0.9442. Recreated via podman compose pull && up -d, .env backed up to .env.bak-20260921-154735 first (existing convention). Verified healthy: all 4 containers up, JVB registered with jicofo (the usual silent failure point), meet.eazyplugins.com 200, config.js and app.bundle.min.js 200, zero real errors — the only log matches are the harmless absent sip-communicator.properties and the idempotent "User exists" re-registration on recreate. Auto-update: the pre-existing jitsi-meet-update.service/.timer were dead on arrival — WorkingDirectory=%h/jitsi-meet does not exist (the project is at %h/jitsi/docker-jitsi-meet) and the timer had never been enabled, so Jitsi had no update path at all. Replaced with ~/jitsi-safe-update.sh: resolve the newest released stable-* from GitHub → compare against the .env pin → back up → pin → pull → recreate → health-check → real rollback to the previous tag on failure (the old images are still local, and .env is the record of what is deployed). Timer enabled; next run Mon 28 Sep 00:00, Persistent=true so a missed window catches up. The bare compose pull && up -d --force-recreate fleet pattern is deliberately NOT used here: the .env pins a specific stable-<n> tag, which never moves, so a plain pull is a permanent no-op. Drill-verified end to end — the timer fired the service, which resolved upstream_latest=stable-11248 against deployed=stable-11248 and correctly no-op'd (exit 0).

  • Anubis upgraded fleetwide to v1.28.0-pre2 (2026-09-21) — 23 instances on claw (via pillar server:anubis:image) plus anubis-forgejo and the unmanaged anubis-cyclops on cyclops. The headline is a security fix: X-Anubis-Rule, X-Anubis-Action and X-Anubis-Status were client-supplied headers that Anubis forwarded downstream, so anything trusting X-Anubis-Status: PASS could be fooled into believing a request had been cleared; pre2 clears client values, sets its own, and strips Connection tokens naming them. Also included: consistent subrequest-auth redirect validation, configured deny status for non-gzip challenge clients, and challenge extensions. Verified: all 23 claw instances report version: v1.28.0-pre2, cyclops both instances on pre2, public endpoints serving challenges (200), the COOKIE_PARTITIONED=false workaround retained unchanged.

  • Three silent-convergence bugs in server.quadlets fixed (2026-09-21) — found only by running a real apply, which failed 21 states while test=True reported zero. (1) The quadlet macro used service.running for 21 rootless services, which resolves against the system systemd as root and therefore always reported "service not available"; now driven via sudo -u USER … systemctl --user. (2) enable --now never restarts a running unit, so quadlet changes were written, reported successful, and the old container kept running — fixed with per-quadlet restart states guarded by onchanges. (3) The homelab-internal-network guard evaluated podman network exists as root, which cannot see rootless networks, so every apply tried to re-create an existing network (podman exit 125 = failure); the guard now queries the user's podman. server.quadlets now completes a clean real run: 67 states, 0 failures. Lesson recorded: test=True is necessary but not sufficient for this class of failure.

  • Diun config drift repaired in salt (2026-09-21) — diun-config and diun-images are salt-managed with hardcoded contents, but the live files had gained a custom templateBody (the "Runs on: HOST" line) and 27 per-entry metadata.host annotations that salt did not know about. The next server.quadlets apply would therefore have silently reverted both. Salt now carries the annotated list and the template body (the Go template is wrapped in a Jinja raw block so the template braces survive rendering), plus a diun-config-reload state so a config change actually restarts Diun — it reads notif.* only at startup. Anubis's own tag in the watch list is also bumped to pre2.

  • Diun alerts now name the host that runs the image (2026-09-14) — the default template ended with the host that ran the check (always claw), which reads as if claw hosts the image. Diun has no notion of deployment, so the host is now recorded per entry: all 27 entries in diun-images.yml carry metadata.host (claw, cyclops, or claw+cyclops for images on both — mapped against live podman ps on both nodes) and notif.telegram.templateBody renders a Runs on: line, with an explicit "(not recorded)" fallback so a missing annotation fails loudly rather than attributing the image to the wrong node. Backups ~/.local/share/diun/*.bak-20260914-095556. Known gap: 27 images watched only — cyclops's forgejo/outline/postgres/redis and claw's bookorbit/pgvector/mongo are not in the list.

  • OpenClaw on cyclops: update repair + safe update path (2026-09-13) — openclaw update to 2026.9.4 failed six consecutive attempts with global install swap — Exit code: 1 … Installation recovery is unverified, leaving the gateway failed (Telegram bot offline) even though the candidate passed all seven validation checks. Two causes: (1) the unit allowed 30s to stop while the gateway legitimately drains active work for up to 315s, so systemd SIGKILLed it mid-swap — fixed with drop-in openclaw-gateway.service.d/stop-timeout.conf (TimeoutStopSec=360; verified 6min, zero SIGKILLs since); (2) npm runs with cwd=$HOME and --allow-scripts needs a valid project manifest, but ~/package.json had no name (Cannot destructure property 'name' of '.for') — fixed, installs now run from the scratch project ~/.openclaw/npm-project. Landed 2026.9.4 (3a9d69d) via npm; new ~/openclaw-safe-update.sh (stop → backup → install → plugins update --all → verify → auto-rollback) replaces the built-in updater; duckduckgo/perplexity/searxng/zai plugins re-pinned to 9.4.

  • OpenClaw provider policy → zai-only, default model pinned (2026-09-13) — 19 provider plugins disabled; only zai remains, plus ollama for embeddings (memory search and dreaming depend on it). Primary zai/glm-5.3-flash, fallbacks glm-5.2 → glm-5.1 → glm-5-turbo, modelPolicy.allow restricted to zai so nothing else is selectable, dreaming pinned to a zai model with allowModelOverride. Verified with a live turn (provider=zai, fallbackUsed=false); stale openrouter free-model cache removed.

  • Forgejo chorompotro/openclaw-cyclops created (2026-09-13) — source of truth for cyclops's OpenClaw instance, mirroring the hermes-claw pattern: sanitized config (secrets replaced by ${...} placeholders), workspace + memory notes, non-bundled skills, systemd wiring with drop-in nesting preserved, safe-update scripts, and CLI exports (cron/plugins/skills/models/agents). The exporter redacts credential shapes while copying — it caught a live LiveKit API secret and JWTs sitting in the agent's own memory notes — then re-scans every written file and aborts the sync on any hit; verified by fresh clone: zero occurrences of the live secret values. Sync via user timer every 15 min, idempotent; restore via bootstrap.sh --dry-run then bootstrap.sh.

  • Restic stale-lock hardening (2026-09-12) — a manual restic forget --dry-run left a lock with a dead PID; the 04:00 claw B2 leg failed once (sftp leg + all radicale hourlies unaffected). All 5 fleet backup scripts (nightly ×2 nodes, hourly radicale, verify) now run plain restic unlock (dead-PID locks only) before each repo touch — a crashed/manual run can no longer fail a scheduled leg. Incident + rerun-verified green; salt-fleet ec85af2+6121e37+fa40f58.

  • Radicale hourly backup increment (2026-09-11) — new restic-radicale-hourly.timer (:45, claw, root): restic pass over ~/radicale-app only, to both claw repos (sftp cyclops + B2). Nightly-only backup = 24h RPO, too coarse for all-day CalDAV edits → now ~1h. Nightly forget gained --keep-hourly 24; hourly job never prunes. Verified: live run both legs OK, B2 restore drill sha256-identical, forget dry-run keeps all hourly snapshots. Salt-fleet commit 356b782.

  • Docs site rendering fixes (2026-09-09) — docs.cyclops.painlesswp.com (Zola 0.23 + linkita theme): syntax highlighting was dead (empty [markdown] config → Zola default none) → added [markdown.highlighting] style=inline + light/dark themes; mermaid never rendered (config.extra.mermaid unset, and Zola 0.23/Tera2 removed the old mermaid() shortcode syntax entirely → theme migrated to <mermaid> components) → tools/sync-site-content.sh now wraps GitHub-style mermaid fences in <mermaid> component invocations in the generated copy (canonical repo files keep raw fences for GitHub/gist), config.extra.mermaid=true loads the JS.

  • BookOrbit anubis policy (2026-09-09) — new bookorbit-policy.yaml path-scoped ALLOW ^/(api|assets)/: SPA fetch() calls were receiving challenge HTML inside API responses → "JSON.parse: unexpected character" in UI; then /assets/foliate/view.js (ebook reader) was served as challenge HTML → "Failed to load foliate/view.js". API returns JSON, reader assets serve as JS, UI keeps argon2id challenge.

  • KitchenOwl migrated to cyclops (2026-09-07) — claw → cyclops over WG (10.0.0.2:3013), data preserved (SQLite checkpoint + copy; 1 household, 655 items intact). anubis-kitchenowl flipped to claw host-net :8926 → WG. Docs restructured for dual-node (claw ingress + cyclops worker: vaultwarden/outline/kitchenowl/speedtest).

  • Speedtest deployed (2026-09-07) — LibreSpeed on cyclops (speedtest.eazyplugins.com): quadlet WG-only 10.0.0.2:3012→8080, MODE=standalone (stateless), anubis-speedtest :8929 (host net) with path-scoped policy (ALLOW /backend/ measurement surface, challenge UI), no gzip on vhost (would corrupt measurements), diun watching ghcr.io/librespeed/speedtest:latest, Flame id 41.

  • KitchenOwl deployed (2026-09-07) — kitchenowl.eazyplugins.com: web :8096 + flask back on kitchenowl_default/homelab-internal, anubis-kitchenowl :8926 with path-scoped policy (ALLOW /api/ for mobile apps), JWT secret in pillar, diun watching both images.

  • AUTHELIA REMOVED fleet-wide (2026-09-03) — SSO → per-app native auth + Anubis. Caddy gates stripped, portal deleted, OIDC purged from all apps (outline/linkwarden/vikunja/memos/lubelogger/tandoor/forgejo).

  • Anubis cookie-domain fix (2026-09-03): alias map completed (beaverhabits→habits, searxng→search, it-tools→tools) — mismatched cookie domain caused infinite challenge loops.

  • Dozzle: dropped authelia forward-proxy auth (dozzle has no native auth) → Caddy basic_auth (nirjhor, shared with files).

  • Files (files.eazyplugins.com): SELinux 403 fix — /srv/files relabeled httpd_sys_content_t (was container_file_t); enforced by server.sls.

  • Anubis removed from in front of freellm vhost (2026-09-03) — freellm app stays live, direct reverse_proxy now.

  • Jitsi images pinned stable-11146-2 (2026-09-03, was -1).

  • WP box (160.250.95.56): basic auth removed from all 4 sites.

  • claw-service + Forgejo SSH remote wired (2026-08-29)

2026-09-26 — Cap screen-recording stack deployed (claw)

  • New service: Cap (open-source Loom alternative) at https://cap.eazyplugins.com
  • 5 rootless quadlets: cap-web (:3030), cap-media-server (internal), cap-mysql, cap-minio (Garage S3 :3027), anubis-cap (:3005, 24th instance)
  • MinIO images yanked from docker hub/quay (Oct 2025 license change) → replaced with Garage (AGPL, S3-compatible)
  • Anubis path-scoped policy: /api/ + /r/ allowed for desktop app/share links; web UI behind PoW challenge
  • Salt: server.cap state added to top.sls; secrets NOT in salt (~/cap/.env 0600 + podman secrets + garage.toml 0600, all replace:False)
  • Caddy: cap.eazyplugins.com block via fleet import anubis convention
  • Flame dashboard: Cap entry added (apps table, icon video-outline — videocam not in Flame's bundled MDI v5)
  • salt-fleet commit adfe06f
  • CONFIRMED WORKING (2026-09-27): Cap uploads + share-link playback end-to-end after fixes below (root causes were S3 vhost missing, Garage v1 hash rejection, S3_PUBLIC_ENDPOINT unset, SDK v3 checksum poison). User-verified.
  • Flame icon fix (2026-09-27): Cap tile showed no icon — Flame bundles @mdi/js v5, where videocam (v6 name) doesn't exist. Row updated to video-outline (run0 sqlite3 UPDATE).
  • Email wired: Resend API, key reused from Outline's account (cyclops), FROM domain support.painlesswp.com. Login = 6-digit code.
    • GOTCHA: eazyplugins.com NOT verified in Resend — sends 403 silently (browser sees 302). Fix = DKIM/SPF/MX at registrar + resend.com/domains verify, then flip RESEND_FROM_DOMAIN.
  • CORS fix (2026-09-26): Garage has NO native CORS — browser preflights (OPTIONS) got 403, blocking web-dashboard uploads and any cross-origin video fetch. Caddy now answers OPTIONS 204 + ACAO/ACAM/ACAH headers on cap-minio vhost (salt 73e19e0).
    • Note: 'Could not load a playable video source' on old share links = upload never completed (0 parts landed pre-fix); no video data to recover. Re-record required.
  • Uploads fixed AGAIN (2026-09-26, 2nd round): (3) S3_PUBLIC_ENDPOINT was unset — presigned URLs defaulted to amazonaws.com (S3_PUBLIC_URL is playback-only); (4) AWS SDK v3 flexible checksums embedded crc32-of-empty-body into presigned part URLs -> garage 400 InvalidDigest on every PUT. Fix: S3_PUBLIC_ENDPOINT set + AWS_REQUEST_CHECKSUM_CALCULATION=WHEN_REQUIRED in .env; garage bumped v2.0.0 -> v2.4.1. Full multipart flow verified via SDK v3 + raw fetch PUT.
  • Uploads fixed (2026-09-26): two causes — (1) cap-minio.eazyplugins.com S3 vhost didn't exist (presigned URLs went nowhere; TLS internal error), added to Caddy direct->3027 without anubis; (2) Garage v1.0.1 rejects UNSIGNED-PAYLOAD/STREAMING content hashes, upgraded to v2.0.0 (data intact). Multipart upload verified end-to-end via boto3 SigV4. salt-fleet d6dfe55