monlet/docs/ops/deployment-checklist.md

First deployment checklist

A pragmatic list for the first production-ish Monlet stand-up. Adjust per environment.

1. Secrets

• Generate MONLET_AUTH_TOKEN (openssl rand -hex 32) and store it in your secret manager.
• Generate a PostgreSQL password and store it in your secret manager.
• Decide which notifier channels you want and gather their credentials (Telegram bot token + chat id, webhook URL, Alertmanager URL).

2. Database

• Provision PostgreSQL 16+.
• Create monlet database and role with CREATE/USAGE on the schema.
• Set MONLET_DATABASE_URL=postgresql+asyncpg://....
• Keep database/server runtime timezone at UTC; only UI/notifier presentation should use local timezones.
• Run alembic upgrade head from a one-shot container or the server image.
• Verify \dt shows agents, checks, events, incidents, notification_outbox.

3. Server

• Set env vars per server/.env.example (see also server/README.md).
• Start container; verify GET /api/v1/health and /api/v1/ready return 200.
• Verify /metrics exposes monlet_server_*.
• Confirm an authenticated GET /api/v1/agents returns { items: [], next_cursor: null }.
• Confirm logs show no unredacted token/cookie material.

4. Agent (per host)

• Install binary (see agent/systemd/).
• Render config.toml from agent/config.example.toml; omit agent_id to use the OS hostname, or set explicit agent_id when the host name is not a stable Monlet identifier.
• Start under systemd; verify the host appears as pending in GET /api/v1/agents, then accept it in the UI/API.
• Run one configured check and verify a row appears in /api/v1/checks and an event in /api/v1/events/query.

5. Notifications

• Enable at least one notifier in server env (MONLET_NOTIFIER_*).
• Force a critical event from one agent and confirm an outbox row reaches sent (GET /api/v1/notifiers/outbox).
• Resolve the check and confirm a resolved outbox row is emitted.

6. Web UI

• Set MONLET_API_BASE_URL, MONLET_API_TOKEN, and optional MONLET_WEB_TIME_ZONE for the web container.
• Protect the web UI with either MONLET_WEB_AUTH_USERNAME/MONLET_WEB_AUTH_PASSWORD or a trusted reverse proxy that sets X-Forwarded-User plus MONLET_WEB_TRUST_PROXY_AUTH=true.
• Set optional MONLET_NOTIFICATION_TIME_ZONE for human-readable notifier timestamps.
• Browse /, /agents, /checks, /incidents, /events, /outbox.
• Verify the web image does not log the token (docker logs greps clean).

7. Observability (optional)

• Add Monlet server to your Prometheus scrape config (deploy/prometheus/prometheus.yml).
• Import deploy/grafana/dashboards/monlet-overview.json for the starter dashboard.
• If using Alertmanager as a notifier, point MONLET_NOTIFIER_ALERTMANAGER_URL at it.

8. Operational

• Schedule database backups (docs/ops/backup-restore.md).
• Document token rotation owner and cadence (docs/ops/token-rotation.md).
• Add monlet_server_open_incidents and monlet_server_agents{status="dead"} to your dashboards.
• Decide an events retention policy.