50 lines
2.2 KiB
Markdown
50 lines
2.2 KiB
Markdown
# Backup and restore
|
||
|
||
All persistent state lives in PostgreSQL (`agents`, `checks`, `events`, `incidents`, `notification_outbox`). The server is stateless; the agent local spool is best-effort and not part of the canonical state (ADR-0006).
|
||
|
||
## What to back up
|
||
|
||
- The Monlet PostgreSQL database (logical or physical).
|
||
- The server `MONLET_AUTH_TOKEN` and `.env` (kept in your secret store, not in backups of the DB).
|
||
- The agent `config.toml` per host (kept in your config management).
|
||
|
||
## Logical backup (recommended for small deployments)
|
||
|
||
```sh
|
||
PGPASSWORD=monlet pg_dump \
|
||
-h $PGHOST -U monlet -d monlet \
|
||
--format=custom --no-owner --no-privileges \
|
||
--file=monlet-$(date -u +%Y%m%dT%H%M%SZ).dump
|
||
```
|
||
|
||
Restore into an empty database:
|
||
|
||
```sh
|
||
createdb -h $PGHOST -U postgres monlet
|
||
PGPASSWORD=monlet pg_restore \
|
||
-h $PGHOST -U monlet -d monlet \
|
||
--no-owner --no-privileges \
|
||
monlet-YYYYMMDDTHHMMSSZ.dump
|
||
```
|
||
|
||
The schema is owned by Alembic. Restoring a logical dump from the same Monlet version is safe. After restore, run `alembic upgrade head` only if you restored from an older version.
|
||
|
||
## Retention
|
||
|
||
- `events` grows unbounded in v1. Decide a retention policy that matches your storage budget (typical: 7–30 days). Stage 6+ may add server-side trimming; until then, run a manual cron:
|
||
|
||
```sql
|
||
DELETE FROM events WHERE observed_at < now() - interval '14 days';
|
||
```
|
||
|
||
`checks.last_event_id` is not a FK, so trimming `events` does not break current state.
|
||
|
||
- `incidents` and `notification_outbox` are small. Trim outbox `state IN ('sent','failed','discarded')` rows older than 7 days if storage is tight.
|
||
|
||
## Disaster recovery
|
||
|
||
- Restore the database.
|
||
- Restart the server. Agents will resume sending; spooled events on each agent are replayed and de-duplicated by `event_id` (ADR-0006).
|
||
- The agent spool is bounded (10 000 events / 50 MiB FIFO, ADR-0005). During a long outage the oldest events get dropped to make room for new ones, so observations from the start of the outage are lost first. Plan recovery time accordingly — for an agent producing ~1 event/s per check, ~10 000 events is roughly 2–3 hours of buffering for a handful of checks.
|
||
- If agents were also lost, only events that were not yet persisted to the spool file are lost in addition.
|