eazyplugins.com — Complete Homelab Server Setup Guide

AlmaLinux 10.2 · Rootless Podman · Caddy · Anubis Bot Protection · Quadlet

Last updated: 2026-08-28 · Server: claw.eazyplugins.com (formerly claw.hackershostel.com)

Table of Contents

1. Server Specs
2. OS Installation & Base Config
3. Firewall & Security
4. Caddy Reverse Proxy
5. Anubis Bot Protection
6. Podman & Quadlet Setup
7. Shared Network & Storage
8. Services — Quadlet Containers
9. Services — Podman Compose Projects
10. Cockpit Management
11. File Browser (Caddy Native)
12. Hermes
13. Kdump (Kernel Crash Dumps)
14. Service Table
15. Maintenance & Backups

1. Server Specs

2. OS Installation & Base Config

2.1 Install AlmaLinux 10

Install AlmaLinux 10.2 from ISO. Choose Minimal install. During setup:

• Set hostname: claw.eazyplugins.com
• Create user nirjhor with sudo access
• Enable network interface, configure static IP if needed
• Enable SELinux (Enforcing mode)
• Set timezone: Asia/Dhaka

2.2 Post-Install Updates

sudo dnf update -y && sudo dnf upgrade -y
sudo reboot

2.3 Essential Packages

sudo dnf install -y \
  vim curl wget git htop tmux jq \
  cockpit cockpit-podman \
  caddy \
  kexec-tools kdump-utils makedumpfile \
  podman podman-compose \
  gh

Note: firewalld is intentionally NOT in this list (see §3).

2.4 Enable User Lingering (Critical for Rootless Systemd)

Without lingering, user systemd services stop when you log out.

sudo loginctl enable-linger nirjhor

Verify:

loginctl show-user nirjhor | grep Linger
# Linger=yes

3. Firewall & Security

3.1 No firewalld — Current Model

firewalld is not installed on this server. The security model is:

• All web services bind to 127.0.0.1 / localhost or sit behind Caddy TLS
• Caddy handles TLS termination on 80/443
• Anubis PoW challenges block bots at the app layer
• Non-web services (RustDesk, TeamSpeak) bind specific ports directly

Historical note: an earlier iteration ran firewalld with empty rules and CrowdSec with an iptables bouncer. Both were removed (CrowdSec in Aug 2026 for RAM/log-spam reasons). SSH brute-force attempts (btmp grows fast) are mitigated by key-only auth below.

3.2 SSH Hardening

# /etc/ssh/sshd_config (or drop-in)
PermitRootLogin without-password   # key-only root, or 'no'
PasswordAuthentication no

Verify effective config:

sudo sshd -T | grep -E '^passwordauthentication|^permitrootlogin'

3.3 SELinux

Keep SELinux Enforcing. All containers should run with SELinux labels. If a specific container has issues, use --security-opt label=disable only as a last resort (required for Dozzle to access the podman socket).

Verify:

getenforce
# Enforcing
sestatus

3.4 Logrotate

Ensure /etc/logrotate.d/syslog exists (weekly, rotate 2, maxsize 200M, messages/secure/btmp/wtmp + rsyslog HUP postrotate). Without it, unrotated syslog grows to gigabytes.

4. Caddy Reverse Proxy

4.1 Install & Configure

Caddy is installed via RPM (caddy-2.11.4-2.el10).

sudo systemctl enable --now caddy

4.2 Caddyfile

Create /etc/caddy/Caddyfile (see the Caddyfile file in this gist for the full production version):

{
	email admin@eazyplugins.com
	log {
		format json
		output file /var/log/caddy/access.log {
			roll_size 50mb
			roll_keep 5
		}
	}
	servers {
		trusted_proxies static private_ranges
		protocols h1 h2 h3
	}
}

# --- Snippets ---

(anubis) {
	@skip {
		remote_ip 127.0.0.1 ::1 10.0.0.0/8 172.16.0.0/12 192.168.0.0/16
	}
	reverse_proxy @skip {args[0]} {
		header_up X-Real-Ip {remote_host}
	}
	reverse_proxy {args[1]} {
		header_up X-Real-Ip {remote_host}
	}
}

(security) {
	header {
		X-Content-Type-Options "nosniff"
		X-Frame-Options "DENY"
		Referrer-Policy "strict-origin-when-cross-origin"
		Permissions-Policy "geolocation=()"
		-Server
	}
	encode gzip
	handle_errors {
		respond "{err.status} {err.message}" {err.status}
	}
}

# --- Catch-all ---
:80 {
	respond "eazyplugins.com" 444
}

# --- Services with Anubis ---
search.eazyplugins.com     { import security; import anubis localhost:8080  localhost:8903 }
dashboard.eazyplugins.com  { import security; import anubis localhost:5005  localhost:8901 }
health.eazyplugins.com     { import security; import anubis localhost:61208 localhost:8902 }
api-spotify.eazyplugins.com{ import security; reverse_proxy localhost:8082 }
spotify.eazyplugins.com    { import security; import anubis localhost:3002  localhost:3003 }
meet.eazyplugins.com       { import security; import anubis localhost:8000  localhost:8904 }

dozzle.eazyplugins.com {
    import security
    basic_auth * {
        nirjhor $2a$14$<GENERATE_BCRYPT_HASH>   # shared nirjhor credential (files/dozzle); dozzle has no native auth
    }
    import anubis localhost:8085 localhost:8906
}
privatebin.eazyplugins.com { import security; import anubis localhost:8086  localhost:8907 }
tools.eazyplugins.com      { import security; import anubis localhost:8087  localhost:8908 }
habits.eazyplugins.com     { import security; import anubis localhost:8089  localhost:8909 }
recipes.eazyplugins.com    { import security; import anubis localhost:8088  localhost:8910 }
tasks.eazyplugins.com      { import security; import anubis localhost:8090  localhost:8911 }
memos.eazyplugins.com      { import security; import anubis localhost:5230  localhost:8912 }
bookmarks.eazyplugins.com  { import security; import anubis localhost:3010  localhost:8913 }
budget.eazyplugins.com     { import security; import anubis localhost:8091  localhost:8915 }
calendar.eazyplugins.com   { import security; reverse_proxy 127.0.0.1:5232 }
status.eazyplugins.com     { import security; import anubis localhost:3001  localhost:8916 }
hermes.eazyplugins.com     { import security; import anubis localhost:9119  localhost:8918 }
paperless.eazyplugins.com  { import security; import anubis localhost:8092  localhost:8921 }
lubelogger.eazyplugins.com { import security; import anubis localhost:8094  localhost:8922 }
freellm.eazyplugins.com    { import security; reverse_proxy localhost:3004 }  # anubis removed 2026-09-03

# --- Audiobookshelf — NO Anubis (socket.io WebSocket needs h1) ---
books.eazyplugins.com {
	header {
		X-Content-Type-Options "nosniff"
		X-Frame-Options "DENY"
		Referrer-Policy "strict-origin-when-cross-origin"
		Permissions-Policy "geolocation=()"
		-Server
	}
	reverse_proxy localhost:13378
}

# --- Cockpit (custom headers for iframe/WebSocket support) ---
cockpit.eazyplugins.com {
	header {
		X-Content-Type-Options "nosniff"
		X-Frame-Options "sameorigin"
		Referrer-Policy "strict-origin-when-cross-origin"
		Permissions-Policy "geolocation=()"
		-Server
	}
	encode gzip
	reverse_proxy localhost:9090 {
		transport http {
			versions 1.1
		}
		header_up X-Forwarded-Proto https
	}
}

# --- File Browser (Caddy native, no container) ---
files.eazyplugins.com {
	header {
		X-Content-Type-Options "nosniff"
		X-Frame-Options "DENY"
		Referrer-Policy "strict-origin-when-cross-origin"
		Permissions-Policy "geolocation=()"
		-Server
	}
	basic_auth * {
		nirjhor $2a$14$<GENERATE_BCRYPT_HASH>
	}
	root * /srv/files
	file_server browse
}

4.3 Generate Bcrypt Hash for Basic Auth

caddy hash-password --plaintext '<YOUR_PASSWORD>'

4.4 Reload Caddy

sudo caddy validate --config /etc/caddy/Caddyfile --adapter caddyfile
sudo systemctl reload caddy

4.5 DNS Records

Wildcard A record to 160.250.95.39:

*.eazyplugins.com    A    160.250.95.39
eazyplugins.com      A    160.250.95.39

Caddy will automatically obtain Let's Encrypt certificates for each subdomain on first request.

4.6 File Browser Directory

sudo mkdir -p /srv/files
sudo chown caddy:caddy /srv/files

5. Anubis Bot Protection

Anubis (TecharoHQ) sits in front of each service and challenges suspicious traffic with a JavaScript proof-of-work test. Real browsers pass instantly; bots get blocked.

5.1 Deployment Model (current)

Anubis runs from the container image (ghcr.io/techarohq/anubis:v1.28.1) — no host binary, no /opt checkout. One Quadlet per service (24 total). Each instance:

• Listens on container port 8923 (BIND=:8923), published to its unique host port (89XX range)
• Targets its backend by container name on a shared podman network (or 127.0.0.1 when Network=host)
• Uses a shared ED25519 key (64-hex ED25519_PRIVATE_KEY_HEX env in each Quadlet)

Generate a key:

openssl genpkey -algorithm Ed25519 -out /dev/stdout 2>/dev/null \
  | openssl pkey -outform DER 2>/dev/null | tail -c 32 | xxd -p -c 64

5.2 Anubis Quadlet (real production example — anubis-paperless.container)

[Unit]
Description=Anubis Bot Protection - Paperless-ngx
After=network-online.target network.target
Wants=network-online.target
StartLimitIntervalSec=60
StartLimitBurst=5

[Container]
Image=ghcr.io/techarohq/anubis:v1.28.1
ContainerName=anubis-paperless
PublishPort=8921:8923
Network=homelab-internal
Environment=SLOG_LEVEL=INFO
Environment=SERVE_ROBOTS_TXT=true
Environment=COOKIE_DOMAIN=paperless.eazyplugins.com
Environment=REDIRECT_DOMAINS=paperless.eazyplugins.com
Environment=COOKIE_EXPIRATION_TIME=168h
Environment=ED25519_PRIVATE_KEY_HEX=<64_HEX_CHARS>
Environment=BIND=:8923
Environment=TARGET=http://paperless-ngx:8000
Environment=USE_REMOTE_ADDRESS=false
Environment=OG_PASSTHROUGH=true
Environment=OG_EXPIRY_TIME=24h
AutoUpdate=registry
Pull=missing

[Service]
Restart=always
RestartSec=5
Type=notify
NotifyAccess=all

[Install]
WantedBy=default.target

There is also an anubis@.container template unit kept for ad-hoc instances (PublishPort=89%i:8923), but all 23 production instances use standalone files like the one above.

5.3 Host-Network Variant (Jitsi, Hermes)

[Container]
Network=host
Environment=BIND=127.0.0.1:8904
Environment=TARGET=http://127.0.0.1:8000
Environment=METRICS_BIND=127.0.0.1:8914

5.4 Anubis Configuration Notes

• COOKIE_EXPIRATION_TIME=168h — Cookies last 7 days
• USE_REMOTE_ADDRESS=false — Uses X-Real-Ip header (set by Caddy) instead of REMOTE_ADDR
• OG_PASSTHROUGH=true + OG_EXPIRY_TIME=24h — OpenGraph passthrough for link previews
• SERVE_ROBOTS_TXT=true — Anubis serves its robots.txt
• Metrics either disabled or bound to 127.0.0.1
• Private IP ranges bypass Anubis entirely via Caddy's @skip matcher

6. Podman & Quadlet Setup

6.1 Rootless Podman

Podman runs rootless under user nirjhor (UID 1000). No containers run as root.

podman info --format '{{.Host.RemoteSocket.Path}}'
podman info --format '{{.Security.Rootless}}'
# true

6.2 Quadlet

Quadlet converts .container files in ~/.config/containers/systemd/ into systemd user services. 64 Quadlets as of Oct 2026.

Reload after adding/editing unit files:

systemctl --user daemon-reload

Gotcha: generated units are transient — systemctl --user enable <name>.service fails with "transient or generated". Start with systemctl --user start <name>; auto-start comes from [Install] WantedBy=default.target in the .container file.

6.3 Podman Compose via Systemd

Helper service at ~/.config/systemd/user/podman-compose@.service:

[Unit]
Description=Podman Compose for %i
After=network-online.target
Wants=network-online.target

[Service]
Type=oneshot
RemainAfterExit=yes
WorkingDirectory=/home/nirjhor/%i
ExecStart=/usr/bin/podman compose up -d
ExecStop=/usr/bin/podman compose down
TimeoutStartSec=300
TimeoutStopSec=60

[Install]
WantedBy=default.target

Usage (Jitsi):

systemctl --user start podman-compose@docker-jitsi-meet

Updates are automated weekly (jitsi-meet-update.timer → ~/jitsi-safe-update.sh). To update by hand, run the same script — do not hand-edit JITSI_IMAGE_VERSION and compose up without a health check, since the pin is what the script compares against:

~/jitsi-safe-update.sh
systemctl --user list-timers jitsi-meet-update.timer
journalctl --user -u jitsi-meet-update.service -n 20

7. Shared Network & Storage

7.1 Shared Podman Network

podman network create homelab-internal

Most standalone Quadlets join homelab-internal (subnet 10.89.8.0/24) so Anubis can reach backends by container name. Compose stacks keep their own networks (tandoor_default, vikunja_default, memos_memos, your_spotify_default, linkwarden_linkwarden); backends needing Anubis from a compose network publish a host port instead.

7.2 Data Directories

mkdir -p ~/.local/share/{searxng,flame/data,paperless/{data,media,export,consume},lubelogger/{data,keys}}
mkdir -p ~/beaverhabits-app/beaver ~/privatebin-app/data ~/radicale-app/{data,config}
mkdir -p ~/actualbudget-app ~/linkwarden-app ~/memos-app ~/tandoor-app ~/uptimekuma-app ~/vikunja-app ~/your_spotify
sudo mkdir -p /srv/audiobookshelf/{config,metadata,audiobooks,podcasts}

8. Services — Quadlet Containers

General Quadlet File Pattern

Create files at ~/.config/containers/systemd/<name>.container:

[Unit]
Description=<Service Name>
After=network-online.target network.target
Wants=network-online.target
StartLimitIntervalSec=60
StartLimitBurst=5

[Container]
Image=<registry/image:tag>
Label=io.containers.autoupdate=registry
ContainerName=<name>
PublishPort=<host>:<container>
Network=homelab-internal
Volume=/host/path:/container/path:z
Environment=KEY=value
AutoUpdate=registry
Pull=missing

[Service]
Restart=always
RestartSec=5

[Install]
WantedBy=default.target

:z on bind mounts (SELinux), :Z only for exclusive single-container mounts. Multi-backend stacks (postgres/redis) get After= dependencies on their db Quadlet services.

9. Services — Podman Compose Projects

Compose stacks live in ~/<app>/docker-compose.yml:

Weekly update pattern (compose stacks not covered by podman-auto-update):

# ~/.config/systemd/user/uptimekuma-update.{service,timer}
[Service]
Type=oneshot
WorkingDirectory=%h/uptimekuma-app
ExecStart=/bin/bash -c '/usr/bin/podman compose pull && /usr/bin/podman compose up -d --force-recreate'

[Timer]
OnCalendar=weekly

10. Cockpit Management

Cockpit on 9090, proxied at cockpit.eazyplugins.com (h1 transport, WebSocket-safe headers). Requires cockpit.socket:

sudo systemctl enable --now cockpit.socket

11. File Browser (Caddy Native)

files.eazyplugins.com — no container. Caddy file_server browse + basic_auth on /srv/files. See Caddyfile §4.2.

12. Hermes

Hermes Agent runs as systemd user services:

• hermes-dashboard.service — hermes dashboard --insecure --no-open --host 0.0.0.0 --port 9119 (host network)
• hermes-gateway.service — messaging platform gateway

Dashboard is fronted by anubis-hermes (host network, 127.0.0.1:8918 → 127.0.0.1:9119) and exposed at hermes.eazyplugins.com.

13. Kdump (Kernel Crash Dumps)

Installed and active (kexec-tools kdump-utils makedumpfile), crashkernel reserves 256MB.

sudo systemctl enable --now kdump
kdumpctl status

Known quirk: kdumpctl status as non-root prints a /run/lock/kdump: Permission denied line before the real status — run with sudo.

14. Service Table

See config.md in this gist for the complete live table (domains, ports, networks, Anubis allocation, quirks).

Quick reference — public endpoints:

15. Maintenance & Backups

15.1 Automatic Updates

• Quadlet containers: podman-auto-update.timer (daily ~20:45) — 62 of 64 units with AutoUpdate=registry
• Compose stacks: weekly *-update.timer services (ActualBudget, Uptime Kuma, YourSpotify) running podman compose pull && up -d --force-recreate
• Jitsi: jitsi-meet-update.timer (weekly, Persistent=true) runs ~/jitsi-safe-update.sh — deliberately NOT the bare pattern above, because .env pins a specific stable-<n> tag, so a plain pull is a permanent no-op. The script resolves the newest released stable-*, compares it to the pin, then backs up / pulls / recreates / health-checks and rolls back on failure. Currently stable-11248. See config.md Quirk 10. ⚠️ The pin is co-owned: salt-fleet claw/salt/jitsi.sls bakes the same JITSI_IMAGE_VERSION into the .env, so after any Jitsi update bump it there too, or the next state.apply server.jitsi rolls Jitsi back (see salt-fleet RUNBOOK §6.2)
• Diun: watches 30 images 4x/day, notifies Telegram; detects drift even where auto-update is off
• dnf-automatic: security updates for host packages

15.2 Backups

• Radicale: weekly radicale-backup.timer → ~/.hermes/scripts/radicale-backup.sh
• Postgres volumes: named volumes under ~/.local/share/containers/storage/volumes/ — include in any backup regime
• Everything else: bind-mount directories under $HOME + /srv/audiobookshelf

15.3 Health Checks

# All containers up?
podman ps --format '{{.Names}} {{.Status}}' | grep -v Up || echo ALL UP

# Every public site answering?
for d in search dashboard health dozzle status hermes; do
  curl -s -o /dev/null -w "%{http_code} $d\n" https://$d.eazyplugins.com
done

# RAM hogs
podman stats --no-stream --format '{{.Name}}: {{.MemUsage}}' | sort -t: -k2 -hr | head -10

15.4 Jev Urgency Gate (hourly, calibrated alerting)

jev-gate.timer (user unit) runs ~/.hermes/scripts/jev-urgency-gate.sh hourly: sweeps claw + cyclops (disk, swap, failed units, OOM kills, OpenClaw memory-pressure warnings, restic log tails), computes hour-over-hour deltas, and asks the Jev decision model (CommandCode API, key in ~/.hermes/.env) for a page probability. Routing: p > 0.8 → Telegram page · 0.4–0.8 → ~/.hermes/jev-gate/digest.txt · < 0.4 → silent. Log: ~/.hermes/jev-gate/gate.log. Full design + tuning notes: config.md Quirk 21.