# Monlet Server

FastAPI server: heartbeat/event ingestion, current check state, incident lifecycle, stale/dead detector, notification outbox, Prometheus metrics.

## Stack

- Python 3.14+
- `uv`
- FastAPI, Pydantic v2
- SQLAlchemy 2.x async + `asyncpg`
- Alembic (sync `psycopg`)
- `structlog`, `prometheus-client`
- PostgreSQL 16+

## Local development

```sh
cd server
UV_PROJECT_ENVIRONMENT=.venv UV_CACHE_DIR=../.cache/uv uv sync --extra dev
```

Run a Postgres instance (Docker or local), set `MONLET_DATABASE_URL`, then:

```sh
UV_PROJECT_ENVIRONMENT=.venv UV_CACHE_DIR=../.cache/uv uv run alembic upgrade head
UV_PROJECT_ENVIRONMENT=.venv UV_CACHE_DIR=../.cache/uv uv run fastapi dev monlet_server/main.py
```

## Environment

| Var | Default | Purpose |
|---|---|---|
| `MONLET_AUTH_TOKEN` | required | UI/operator Bearer token. Space/comma-separated list; any accepted. `changeme` is rejected. |
| `MONLET_DATABASE_URL` | `postgresql+asyncpg://monlet:monlet@127.0.0.1:5432/monlet` | Async DSN |
| `MONLET_LOG_LEVEL` | `INFO` | Log level |
| `MONLET_HOST` | `0.0.0.0` | HTTP bind host for direct server runs |
| `MONLET_PORT` | `8000` | HTTP bind port for direct server runs |
| `MONLET_BODY_LIMIT_BYTES` | `1048576` | Request body limit (ADR-0005) |
| `MONLET_OUTPUT_MAX_BYTES` | `8192` | Per-event output limit |
| `MONLET_MAX_BATCH_EVENTS` | `200` | Maximum events per ingestion batch |
| `MONLET_STALE_AFTER_SEC` | `20` | Stale threshold |
| `MONLET_DEAD_AFTER_SEC` | `30` | Dead threshold |
| `MONLET_DETECTOR_TICK_SEC` | `5` | Detector tick |
| `MONLET_ENABLE_DETECTOR` | `true` | Toggle detector task |
| `MONLET_DB_POOL_SIZE` | `10` | Base async DB pool size |
| `MONLET_DB_MAX_OVERFLOW` | `10` | Extra DB connections above pool size |
| `MONLET_DB_POOL_TIMEOUT_SEC` | `30` | Seconds to wait for a DB connection |
| `MONLET_DB_POOL_RECYCLE_SEC` | `1800` | Recycle DB connections by age |
| `MONLET_EVENTS_RETENTION_MONTHS` | `36` | Months of `events` partitions to keep; `0` disables drop |
| `MONLET_EVENTS_FUTURE_PARTITIONS` | `3` | Future month partitions to pre-create |
| `MONLET_EVENT_DEDUP_RETENTION_DAYS` | `30` | Idempotency window for `event_id` |
| `MONLET_EVENT_DEDUP_PRUNE_BATCH_SIZE` | `5000` | Dedup rows pruned per maintenance tick |
| `MONLET_MAX_PENDING_AGENTS` | `10000` | Maximum pending agent keys before heartbeat rejects with 429 |
| `MONLET_PENDING_AGENT_TTL_DAYS` | `7` | Pending key cleanup TTL; `0` disables pruning |
| `MONLET_PENDING_AGENT_PRUNE_BATCH_SIZE` | `1000` | Pending cleanup rows per detector tick |
| `MONLET_PARTITION_MAINTENANCE_INTERVAL_SEC` | `3600` | Partition create/drop worker interval |
| `MONLET_OUTBOX_RETENTION_MAX_ROWS` | `50000` | Maximum retained terminal outbox rows; `0` disables pruning |
| `MONLET_OUTBOX_PRUNE_BATCH_SIZE` | `5000` | Outbox rows pruned per maintenance tick |
| `MONLET_ENABLE_NOTIFIER_WORKER` | `true` | Toggle outbox worker |
| `MONLET_NOTIFIER_TICK_SEC` | `5` | Worker poll interval |
| `MONLET_NOTIFIER_BATCH_SIZE` | `20` | Rows claimed per tick |
| `MONLET_NOTIFIER_CONCURRENCY` | `4` | Parallel notification deliveries per tick |
| `MONLET_NOTIFIER_MAX_ATTEMPTS` | `8` | Terminal-failure threshold (ADR-0005) |
| `MONLET_NOTIFIER_LEASE_SEC` | `300` | Recovery lease for stuck `sending` rows |
| `MONLET_NOTIFIER_HTTP_TIMEOUT_SEC` | `10` | Outbound HTTP timeout |
| `MONLET_NOTIFIER_INCLUDE_OUTPUT` | `true` | Include redacted check output in notifications |
| `MONLET_NOTIFIER_MAX_OUTPUT_BYTES` | `1024` | Maximum notification output bytes after redaction |
| `MONLET_NOTIFICATION_TIME_ZONE` | `UTC` | IANA timezone for human-readable notification timestamps |
| `MONLET_NOTIFIER_DEBUG_ENABLED` | `true` | Log-only notifier |
| `MONLET_NOTIFIER_TELEGRAM_ENABLED` | `false` | Telegram notifier on/off |
| `MONLET_NOTIFIER_TELEGRAM_TOKEN` | `` | Telegram bot token |
| `MONLET_NOTIFIER_TELEGRAM_CHAT_ID` | `` | Telegram chat id |
| `MONLET_NOTIFIER_WEBHOOK_ENABLED` | `false` | Generic webhook notifier on/off |
| `MONLET_NOTIFIER_WEBHOOK_URL` | `` | Webhook URL |
| `MONLET_NOTIFIER_WEBHOOK_TOKEN` | `` | Optional Bearer token for webhook |
| `MONLET_NOTIFIER_ALERTMANAGER_ENABLED` | `false` | Alertmanager notifier on/off |
| `MONLET_NOTIFIER_ALERTMANAGER_URL` | `` | Alertmanager base URL (e.g. `http://am:9093`) |

## Tests

Tests use `testcontainers-python` to spin up a real PostgreSQL container; Docker must be available.

```sh
UV_PROJECT_ENVIRONMENT=.venv UV_CACHE_DIR=../.cache/uv uv run pytest -q
```

## Docker

Image installs dependencies inside the container only; host venv is never mounted.

```sh
docker build -t monlet-server .
```

## Endpoints

All routes under `/api/v1` (per `api/openapi.yaml`). `/metrics` and `/api/v1/health`, `/api/v1/ready` are open.

- UI/operator endpoints require `Authorization: Bearer <MONLET_AUTH_TOKEN>`.
- Agent ingestion endpoints accept the per-agent key generated and stored by the agent. A new key creates a pending agent; events are ignored until an operator accepts it.