Implement Monlet MVP stack and UI updates

docs/ops/backup-restore.md

@@ -0,0 +1,49 @@
+# Backup and restore
+
+All persistent state lives in PostgreSQL (`agents`, `checks`, `events`, `incidents`, `notification_outbox`). The server is stateless; the agent local spool is best-effort and not part of the canonical state (ADR-0006).
+
+## What to back up
+
+- The Monlet PostgreSQL database (logical or physical).
+- The server `MONLET_AUTH_TOKEN` and `.env` (kept in your secret store, not in backups of the DB).
+- The agent `config.toml` per host (kept in your config management).
+
+## Logical backup (recommended for small deployments)
+
+```sh
+PGPASSWORD=monlet pg_dump \
+  -h $PGHOST -U monlet -d monlet \
+  --format=custom --no-owner --no-privileges \
+  --file=monlet-$(date -u +%Y%m%dT%H%M%SZ).dump
+```
+
+Restore into an empty database:
+
+```sh
+createdb -h $PGHOST -U postgres monlet
+PGPASSWORD=monlet pg_restore \
+  -h $PGHOST -U monlet -d monlet \
+  --no-owner --no-privileges \
+  monlet-YYYYMMDDTHHMMSSZ.dump
+```
+
+The schema is owned by Alembic. Restoring a logical dump from the same Monlet version is safe. After restore, run `alembic upgrade head` only if you restored from an older version.
+
+## Retention
+
+- `events` grows unbounded in v1. Decide a retention policy that matches your storage budget (typical: 7–30 days). Stage 6+ may add server-side trimming; until then, run a manual cron:
+
+  ```sql
+  DELETE FROM events WHERE observed_at < now() - interval '14 days';
+  ```
+
+  `checks.last_event_id` is not a FK, so trimming `events` does not break current state.
+
+- `incidents` and `notification_outbox` are small. Trim outbox `state IN ('sent','failed','discarded')` rows older than 7 days if storage is tight.
+
+## Disaster recovery
+
+- Restore the database.
+- Restart the server. Agents will resume sending; spooled events on each agent are replayed and de-duplicated by `event_id` (ADR-0006).
+- The agent spool is bounded (10 000 events / 50 MiB FIFO, ADR-0005). During a long outage the oldest events get dropped to make room for new ones, so observations from the start of the outage are lost first. Plan recovery time accordingly — for an agent producing ~1 event/s per check, ~10 000 events is roughly 2–3 hours of buffering for a handful of checks.
+- If agents were also lost, only events that were not yet persisted to the spool file are lost in addition.

docs/ops/deployment-checklist.md

@@ -0,0 +1,57 @@
+# First deployment checklist
+
+A pragmatic list for the first production-ish Monlet stand-up. Adjust per environment.
+
+## 1. Secrets
+
+- [ ] Generate `MONLET_AUTH_TOKEN` (`openssl rand -hex 32`) and store it in your secret manager.
+- [ ] Generate a PostgreSQL password and store it in your secret manager.
+- [ ] Decide which notifier channels you want and gather their credentials (Telegram bot token + chat id, webhook URL, Alertmanager URL).
+
+## 2. Database
+
+- [ ] Provision PostgreSQL 16+.
+- [ ] Create `monlet` database and role with `CREATE`/`USAGE` on the schema.
+- [ ] Set `MONLET_DATABASE_URL=postgresql+asyncpg://...`.
+- [ ] Run `alembic upgrade head` from a one-shot container or the server image.
+- [ ] Verify `\dt` shows `agents, checks, events, incidents, notification_outbox`.
+
+## 3. Server
+
+- [ ] Set env vars per `server/.env.example` (see also `server/README.md`).
+- [ ] Start container; verify `GET /api/v1/health` and `/api/v1/ready` return 200.
+- [ ] Verify `/metrics` exposes `monlet_server_*`.
+- [ ] Confirm an authenticated `GET /api/v1/agents` returns `{ items: [], next_cursor: null }`.
+- [ ] Confirm logs show no unredacted token/cookie material.
+
+## 4. Agent (per host)
+
+- [ ] Install binary (see `agent/systemd/`).
+- [ ] Render `config.toml` from `agent/config.example.toml` with hostname or explicit `agent_id`, and `[server].token`.
+- [ ] Start under systemd; verify a heartbeat reaches the server (`GET /api/v1/agents`).
+- [ ] Run one configured check and verify a row appears in `/api/v1/checks` and an event in `/api/v1/events/query`.
+
+## 5. Notifications
+
+- [ ] Enable at least one notifier in server env (`MONLET_NOTIFIER_*`).
+- [ ] Force a `critical` event from one agent and confirm an outbox row reaches `sent` (`GET /api/v1/notifiers/outbox`).
+- [ ] Resolve the check and confirm a `resolved` outbox row is emitted.
+
+## 6. Web UI
+
+- [ ] Set `MONLET_API_BASE_URL` and `MONLET_API_TOKEN` for the web container.
+- [ ] Browse `/`, `/agents`, `/checks`, `/incidents`, `/events`, `/outbox`.
+- [ ] Verify the web image does not log the token (`docker logs` greps clean).
+
+## 7. Observability (optional)
+
+- [ ] Add Monlet server to your Prometheus scrape config (`deploy/prometheus/prometheus.yml`).
+- [ ] Import `deploy/grafana/dashboards/monlet-overview.json` for the starter dashboard.
+- [ ] If using Alertmanager as a notifier, point `MONLET_NOTIFIER_ALERTMANAGER_URL` at it.
+
+## 8. Operational
+
+- [ ] Schedule database backups (`docs/ops/backup-restore.md`).
+- [ ] Document token rotation owner and cadence (`docs/ops/token-rotation.md`).
+- [ ] Add `monlet_server_open_incidents` and `monlet_server_agents{status="dead"}` to your dashboards.
+- [ ] Decide an `events` retention policy.

docs/ops/token-rotation.md

@@ -0,0 +1,45 @@
+# Token rotation
+
+Monlet v1 uses a single shared Bearer token (`MONLET_AUTH_TOKEN`) for all agent and UI endpoints (ADR-0007). Per-agent tokens and dual-token acceptance are out of scope for v1.
+
+## Why rotation is disruptive in v1
+
+The server validates exactly one token at a time. Agents that present the old token after the server has switched receive `401`, which the agent treats as a **non-retryable** failure and drops the affected batch from its local spool (`agent/internal/client/client.go`, `agent/internal/app/app.go`). Events flushed during the mismatch window are lost — they are not re-spooled and not re-sent.
+
+Plan rotations during a low-traffic window and either stop the agents first (so spooled events persist on disk until they restart with the new token) or accept the event-loss window.
+
+## When to rotate
+
+- A suspected leak (token committed, log exposure, lost laptop with `config.toml`).
+- Operator turnover.
+- Routine schedule (e.g., quarterly).
+
+## Procedure
+
+1. **Generate a new token**:
+
+   ```sh
+   openssl rand -hex 32
+   ```
+
+2. **Stop or pause agents** (recommended). Either:
+   - `systemctl stop monlet-agent` on each host; spooled events remain on disk and replay after restart, or
+   - skip this step and accept that any events sent during steps 3–4 are lost on the `401`.
+
+3. **Restart server** with the new `MONLET_AUTH_TOKEN`. Verify:
+
+   ```sh
+   curl -sf -H "Authorization: Bearer $NEW" http://server:8000/api/v1/agents | jq .
+   ```
+
+4. **Update agent `config.toml`** (`[server].token`) on each host and start agents.
+
+5. **Update Web UI** (`MONLET_API_TOKEN`) and restart.
+
+6. **Revoke the old token**: remove the old value from secret stores, CI variables, and operator notes.
+
+## Notes
+
+- The token is the only auth in v1; protect it like a database password.
+- The token never appears in metrics labels, incident keys, or notification labels (ADR-0007, redaction policy).
+- The server redacts `Authorization`/`Cookie`-style headers before logging (`monlet_server/redaction.py`).