Harden production workflows and agent admission

docs/ops/backup-restore.md

@@ -6,7 +6,7 @@ All persistent state lives in PostgreSQL (`agents`, `checks`, `events`, `inciden
 
 - The Monlet PostgreSQL database (logical or physical).
 - The server `MONLET_AUTH_TOKEN` and `.env` (kept in your secret store, not in backups of the DB).
-- The agent `config.toml` per host (kept in your config management).
+- The agent `config.toml` per host and local `state_dir/agent.key` if you need to preserve accepted identity without re-acceptance.
 
 ## Logical backup (recommended for small deployments)
 
@@ -31,19 +31,15 @@ The schema is owned by Alembic. Restoring a logical dump from the same Monlet ve
 
 ## Retention
 
-- `events` grows unbounded in v1. Decide a retention policy that matches your storage budget (typical: 7–30 days). Stage 6+ may add server-side trimming; until then, run a manual cron:
-
-  ```sql
-  DELETE FROM events WHERE observed_at < now() - interval '14 days';
-  ```
-
-  `checks.last_event_id` is not a FK, so trimming `events` does not break current state.
-
-- `incidents` and `notification_outbox` are small. Trim outbox `state IN ('sent','failed','discarded')` rows older than 7 days if storage is tight.
+- `events` is partitioned by `received_at` (one monthly partition per range). Retention is controlled by `MONLET_EVENTS_RETENTION_MONTHS` (default 36). The partition maintenance loop drops old partitions on `MONLET_PARTITION_MAINTENANCE_INTERVAL_SEC` — disk is reclaimed by `DROP PARTITION`, not by row-level `DELETE`, so reclaim is bulk-fast and lock-light. Do NOT run row-level `DELETE FROM events` — it bloats the table and conflicts with partition rotation.
+- `event_ingest_dedup` is the idempotency table; TTL `MONLET_EVENT_DEDUP_RETENTION_DAYS` (default 30) prunes rows in bounded batches each detector tick. After dedup TTL elapses an agent replaying a very old spooled event will be accepted again (no double-side effects because the corresponding `events` partition is already dropped on the same horizon).
+- `notification_outbox` retention is bounded by `MONLET_OUTBOX_RETENTION_MAX_ROWS` (default 50000) and pruned in batches of `MONLET_OUTBOX_PRUNE_BATCH_SIZE` per tick (PH-012). Only terminal states (`sent`, `failed`, `discarded`) are eligible — active rows (`pending`, `sending`, `retry`) are never deleted.
+- `incidents` is small and not pruned automatically; size scales with operator activity rather than ingestion rate.
 
 ## Disaster recovery
 
 - Restore the database.
 - Restart the server. Agents will resume sending; spooled events on each agent are replayed and de-duplicated by `event_id` (ADR-0006).
 - The agent spool is bounded (10 000 events / 50 MiB FIFO, ADR-0005). During a long outage the oldest events get dropped to make room for new ones, so observations from the start of the outage are lost first. Plan recovery time accordingly — for an agent producing ~1 event/s per check, ~10 000 events is roughly 2–3 hours of buffering for a handful of checks.
+- Spool overflow and non-retryable (4xx) server rejections are observable via `monlet_agent_events_dropped_total{reason}` with `reason ∈ spool_overflow_events | spool_overflow_bytes | send_non_retryable`. Alert on any non-zero rate. Operator action: investigate the agent log (one warning per ~30s drop burst), the server (for 4xx), and whether spool limits need raising in `state_dir`.
 - If agents were also lost, only events that were not yet persisted to the spool file are lost in addition.

docs/ops/deployment-checklist.md

@@ -28,8 +28,8 @@ A pragmatic list for the first production-ish Monlet stand-up. Adjust per enviro
 ## 4. Agent (per host)
 
 - [ ] Install binary (see `agent/systemd/`).
-- [ ] Render `config.toml` from `agent/config.example.toml` with hostname or explicit `agent_id`, and `[server].token`.
-- [ ] Start under systemd; verify a heartbeat reaches the server (`GET /api/v1/agents`).
+- [ ] Render `config.toml` from `agent/config.example.toml`; omit `agent_id` to use the OS hostname, or set explicit `agent_id` when the host name is not a stable Monlet identifier.
+- [ ] Start under systemd; verify the host appears as `pending` in `GET /api/v1/agents`, then accept it in the UI/API.
 - [ ] Run one configured check and verify a row appears in `/api/v1/checks` and an event in `/api/v1/events/query`.
 
 ## 5. Notifications
@@ -41,6 +41,7 @@ A pragmatic list for the first production-ish Monlet stand-up. Adjust per enviro
 ## 6. Web UI
 
 - [ ] Set `MONLET_API_BASE_URL`, `MONLET_API_TOKEN`, and optional `MONLET_WEB_TIME_ZONE` for the web container.
+- [ ] Protect the web UI with either `MONLET_WEB_AUTH_USERNAME`/`MONLET_WEB_AUTH_PASSWORD` or a trusted reverse proxy that sets `X-Forwarded-User` plus `MONLET_WEB_TRUST_PROXY_AUTH=true`.
 - [ ] Set optional `MONLET_NOTIFICATION_TIME_ZONE` for human-readable notifier timestamps.
 - [ ] Browse `/`, `/agents`, `/checks`, `/incidents`, `/events`, `/outbox`.
 - [ ] Verify the web image does not log the token (`docker logs` greps clean).

docs/ops/token-rotation.md

@@ -1,20 +1,33 @@
 # Token rotation
 
-Monlet v1 uses a single shared Bearer token (`MONLET_AUTH_TOKEN`) for all agent and UI endpoints (ADR-0007). Per-agent tokens and dual-token acceptance are out of scope for v1.
+Monlet uses split Bearer-token auth (ADR-0007):
 
-## Why rotation is disruptive in v1
+- **UI token** — `MONLET_AUTH_TOKEN` env. Accepted by every UI/operator endpoint. The variable holds one or more space/comma-separated tokens; the server accepts any of them, so rotation is zero-loss.
+- **Agent key** — generated locally by each agent and stored in `state_dir/agent.key`. It is not configured on the server; operators accept, block, remove, or blacklist keys through the Agents UI/API.
 
-The server validates exactly one token at a time. Agents that present the old token after the server has switched receive `401`, which the agent treats as a **non-retryable** failure and drops the affected batch from its local spool (`agent/internal/client/client.go`, `agent/internal/app/app.go`). Events flushed during the mismatch window are lost — they are not re-spooled and not re-sent.
+Rotate UI tokens independently from agent admission. Changing `MONLET_AUTH_TOKEN` does not affect accepted agents.
 
-Plan rotations during a low-traffic window and either stop the agents first (so spooled events persist on disk until they restart with the new token) or accept the event-loss window.
+## How multi-token overlap works
+
+`MONLET_AUTH_TOKEN` is a list: tokens separated by spaces and/or commas. Examples:
+
+```sh
+MONLET_AUTH_TOKEN="single-token"
+MONLET_AUTH_TOKEN="old-token new-token"
+MONLET_AUTH_TOKEN="old-token,new-token"
+```
+
+All listed tokens are accepted. The middleware compares in constant time across the full list, so timing-side-channels do not leak which token matched.
+
+Logs and metrics never include token values.
 
 ## When to rotate
 
-- A suspected leak (token committed, log exposure, lost laptop with `config.toml`).
+- Suspected UI token leak.
 - Operator turnover.
 - Routine schedule (e.g., quarterly).
 
-## Procedure
+## Zero-loss procedure
 
 1. **Generate a new token**:
 
@@ -22,24 +35,39 @@ Plan rotations during a low-traffic window and either stop the agents first (so
    openssl rand -hex 32
    ```
 
-2. **Stop or pause agents** (recommended). Either:
-   - `systemctl stop monlet-agent` on each host; spooled events remain on disk and replay after restart, or
-   - skip this step and accept that any events sent during steps 3–4 are lost on the `401`.
+2. **Add the new token alongside the old one** in the server environment:
 
-3. **Restart server** with the new `MONLET_AUTH_TOKEN`. Verify:
+   ```sh
+   MONLET_AUTH_TOKEN="$OLD $NEW"
+   ```
+
+   Restart (or reload) the server. The old token still works, the new one is now accepted as well.
+
+3. **Move clients to the new token**:
+   - Update Web UI `MONLET_API_TOKEN` and restart.
+   - Update any operator scripts or curl examples to use the new value.
+
+4. **Verify everything uses the new token**:
 
    ```sh
    curl -sf -H "Authorization: Bearer $NEW" http://server:8000/api/v1/agents | jq .
+   # Optional: tail server logs for unauthorized attempts during the window.
    ```
 
-4. **Update agent `config.toml`** (`[server].token`) on each host and start agents.
+5. **Remove the old token** from the server environment:
 
-5. **Update Web UI** (`MONLET_API_TOKEN`) and restart.
+   ```sh
+   MONLET_AUTH_TOKEN="$NEW"
+   ```
 
-6. **Revoke the old token**: remove the old value from secret stores, CI variables, and operator notes.
+   Restart the server. From this point the old token is revoked.
+
+6. **Scrub the old value** from secret stores, CI variables, and operator notes.
 
 ## Notes
 
-- The token is the only auth in v1; protect it like a database password.
-- The token never appears in metrics labels, incident keys, or notification labels (ADR-0007, redaction policy).
+- Tokens are the only auth in v1; protect them like a database password.
+- Tokens never appear in metrics labels, incident keys, or notification labels (ADR-0007, redaction policy).
 - The server redacts `Authorization`/`Cookie`-style headers before logging (`monlet_server/redaction.py`).
+- The middleware uses `hmac.compare_digest` against every accepted token; rotation does not introduce a timing oracle.
+- To rotate an individual agent key, delete `state_dir/agent.key` on that host and restart the agent. The server will show a pending replacement for the same hostname; accept it after verifying the host.