monlet/docs/ops/backup-restore.md

Backup and restore

All persistent state lives in PostgreSQL (agents, checks, events, incidents, notification_outbox). The server is stateless; the agent local spool is best-effort and not part of the canonical state (ADR-0006).

What to back up

• The Monlet PostgreSQL database (logical or physical).
• The server MONLET_AUTH_TOKEN and .env (kept in your secret store, not in backups of the DB).
• The agent config.toml per host (kept in your config management).

Logical backup (recommended for small deployments)

PGPASSWORD=monlet pg_dump \
  -h $PGHOST -U monlet -d monlet \
  --format=custom --no-owner --no-privileges \
  --file=monlet-$(date -u +%Y%m%dT%H%M%SZ).dump

Restore into an empty database:

createdb -h $PGHOST -U postgres monlet
PGPASSWORD=monlet pg_restore \
  -h $PGHOST -U monlet -d monlet \
  --no-owner --no-privileges \
  monlet-YYYYMMDDTHHMMSSZ.dump

The schema is owned by Alembic. Restoring a logical dump from the same Monlet version is safe. After restore, run alembic upgrade head only if you restored from an older version.

Retention

• events grows unbounded in v1. Decide a retention policy that matches your storage budget (typical: 7–30 days). Stage 6+ may add server-side trimming; until then, run a manual cron:

  DELETE FROM events WHERE observed_at < now() - interval '14 days';
  

  checks.last_event_id is not a FK, so trimming events does not break current state.

• incidents and notification_outbox are small. Trim outbox state IN ('sent','failed','discarded') rows older than 7 days if storage is tight.

Disaster recovery

• Restore the database.
• Restart the server. Agents will resume sending; spooled events on each agent are replayed and de-duplicated by event_id (ADR-0006).
• The agent spool is bounded (10 000 events / 50 MiB FIFO, ADR-0005). During a long outage the oldest events get dropped to make room for new ones, so observations from the start of the outage are lost first. Plan recovery time accordingly — for an agent producing ~1 event/s per check, ~10 000 events is roughly 2–3 hours of buffering for a handful of checks.
• If agents were also lost, only events that were not yet persisted to the spool file are lost in addition.