Harden production workflows and agent admission

docs/ops/backup-restore.md

@@ -6,7 +6,7 @@ All persistent state lives in PostgreSQL (`agents`, `checks`, `events`, `inciden
 
 - The Monlet PostgreSQL database (logical or physical).
 - The server `MONLET_AUTH_TOKEN` and `.env` (kept in your secret store, not in backups of the DB).
-- The agent `config.toml` per host (kept in your config management).
+- The agent `config.toml` per host and local `state_dir/agent.key` if you need to preserve accepted identity without re-acceptance.
 
 ## Logical backup (recommended for small deployments)
 
@@ -31,19 +31,15 @@ The schema is owned by Alembic. Restoring a logical dump from the same Monlet ve
 
 ## Retention
 
-- `events` grows unbounded in v1. Decide a retention policy that matches your storage budget (typical: 7–30 days). Stage 6+ may add server-side trimming; until then, run a manual cron:
-
-  ```sql
-  DELETE FROM events WHERE observed_at < now() - interval '14 days';
-  ```
-
-  `checks.last_event_id` is not a FK, so trimming `events` does not break current state.
-
-- `incidents` and `notification_outbox` are small. Trim outbox `state IN ('sent','failed','discarded')` rows older than 7 days if storage is tight.
+- `events` is partitioned by `received_at` (one monthly partition per range). Retention is controlled by `MONLET_EVENTS_RETENTION_MONTHS` (default 36). The partition maintenance loop drops old partitions on `MONLET_PARTITION_MAINTENANCE_INTERVAL_SEC` — disk is reclaimed by `DROP PARTITION`, not by row-level `DELETE`, so reclaim is bulk-fast and lock-light. Do NOT run row-level `DELETE FROM events` — it bloats the table and conflicts with partition rotation.
+- `event_ingest_dedup` is the idempotency table; TTL `MONLET_EVENT_DEDUP_RETENTION_DAYS` (default 30) prunes rows in bounded batches each detector tick. After dedup TTL elapses an agent replaying a very old spooled event will be accepted again (no double-side effects because the corresponding `events` partition is already dropped on the same horizon).
+- `notification_outbox` retention is bounded by `MONLET_OUTBOX_RETENTION_MAX_ROWS` (default 50000) and pruned in batches of `MONLET_OUTBOX_PRUNE_BATCH_SIZE` per tick (PH-012). Only terminal states (`sent`, `failed`, `discarded`) are eligible — active rows (`pending`, `sending`, `retry`) are never deleted.
+- `incidents` is small and not pruned automatically; size scales with operator activity rather than ingestion rate.
 
 ## Disaster recovery
 
 - Restore the database.
 - Restart the server. Agents will resume sending; spooled events on each agent are replayed and de-duplicated by `event_id` (ADR-0006).
 - The agent spool is bounded (10 000 events / 50 MiB FIFO, ADR-0005). During a long outage the oldest events get dropped to make room for new ones, so observations from the start of the outage are lost first. Plan recovery time accordingly — for an agent producing ~1 event/s per check, ~10 000 events is roughly 2–3 hours of buffering for a handful of checks.
+- Spool overflow and non-retryable (4xx) server rejections are observable via `monlet_agent_events_dropped_total{reason}` with `reason ∈ spool_overflow_events | spool_overflow_bytes | send_non_retryable`. Alert on any non-zero rate. Operator action: investigate the agent log (one warning per ~30s drop burst), the server (for 4xx), and whether spool limits need raising in `state_dir`.
 - If agents were also lost, only events that were not yet persisted to the spool file are lost in addition.