101 lines
4.9 KiB
Markdown
101 lines
4.9 KiB
Markdown
# Monlet Server
|
|
|
|
FastAPI server: heartbeat/event ingestion, current check state, incident lifecycle, stale/dead detector, notification outbox, Prometheus metrics.
|
|
|
|
## Stack
|
|
|
|
- Python 3.14+
|
|
- `uv`
|
|
- FastAPI, Pydantic v2
|
|
- SQLAlchemy 2.x async + `asyncpg`
|
|
- Alembic (sync `psycopg`)
|
|
- `structlog`, `prometheus-client`
|
|
- PostgreSQL 16+
|
|
|
|
## Local development
|
|
|
|
```sh
|
|
cd server
|
|
UV_PROJECT_ENVIRONMENT=.venv UV_CACHE_DIR=../.cache/uv uv sync --extra dev
|
|
```
|
|
|
|
Run a Postgres instance (Docker or local), set `MONLET_DATABASE_URL`, then:
|
|
|
|
```sh
|
|
UV_PROJECT_ENVIRONMENT=.venv UV_CACHE_DIR=../.cache/uv uv run alembic upgrade head
|
|
UV_PROJECT_ENVIRONMENT=.venv UV_CACHE_DIR=../.cache/uv uv run fastapi dev monlet_server/main.py
|
|
```
|
|
|
|
## Environment
|
|
|
|
| Var | Default | Purpose |
|
|
|---|---|---|
|
|
| `MONLET_AUTH_TOKEN` | required | UI/operator Bearer token. Space/comma-separated list; any accepted. `changeme` is rejected. |
|
|
| `MONLET_DATABASE_URL` | `postgresql+asyncpg://monlet:monlet@127.0.0.1:5432/monlet` | Async DSN |
|
|
| `MONLET_LOG_LEVEL` | `INFO` | Log level |
|
|
| `MONLET_HOST` | `0.0.0.0` | HTTP bind host for direct server runs |
|
|
| `MONLET_PORT` | `8000` | HTTP bind port for direct server runs |
|
|
| `MONLET_BODY_LIMIT_BYTES` | `1048576` | Request body limit (ADR-0005) |
|
|
| `MONLET_OUTPUT_MAX_BYTES` | `8192` | Per-event output limit |
|
|
| `MONLET_MAX_BATCH_EVENTS` | `200` | Maximum events per ingestion batch |
|
|
| `MONLET_STALE_AFTER_SEC` | `20` | Stale threshold |
|
|
| `MONLET_DEAD_AFTER_SEC` | `30` | Dead threshold |
|
|
| `MONLET_DETECTOR_TICK_SEC` | `5` | Detector tick |
|
|
| `MONLET_ENABLE_DETECTOR` | `true` | Toggle detector task |
|
|
| `MONLET_DB_POOL_SIZE` | `10` | Base async DB pool size |
|
|
| `MONLET_DB_MAX_OVERFLOW` | `10` | Extra DB connections above pool size |
|
|
| `MONLET_DB_POOL_TIMEOUT_SEC` | `30` | Seconds to wait for a DB connection |
|
|
| `MONLET_DB_POOL_RECYCLE_SEC` | `1800` | Recycle DB connections by age |
|
|
| `MONLET_EVENTS_RETENTION_MONTHS` | `36` | Months of `events` partitions to keep; `0` disables drop |
|
|
| `MONLET_EVENTS_FUTURE_PARTITIONS` | `3` | Future month partitions to pre-create |
|
|
| `MONLET_EVENT_DEDUP_RETENTION_DAYS` | `30` | Idempotency window for `event_id` |
|
|
| `MONLET_EVENT_DEDUP_PRUNE_BATCH_SIZE` | `5000` | Dedup rows pruned per maintenance tick |
|
|
| `MONLET_MAX_PENDING_AGENTS` | `10000` | Maximum pending agent keys before heartbeat rejects with 429 |
|
|
| `MONLET_PENDING_AGENT_TTL_DAYS` | `7` | Pending key cleanup TTL; `0` disables pruning |
|
|
| `MONLET_PENDING_AGENT_PRUNE_BATCH_SIZE` | `1000` | Pending cleanup rows per detector tick |
|
|
| `MONLET_PARTITION_MAINTENANCE_INTERVAL_SEC` | `3600` | Partition create/drop worker interval |
|
|
| `MONLET_OUTBOX_RETENTION_MAX_ROWS` | `50000` | Maximum retained terminal outbox rows; `0` disables pruning |
|
|
| `MONLET_OUTBOX_PRUNE_BATCH_SIZE` | `5000` | Outbox rows pruned per maintenance tick |
|
|
| `MONLET_ENABLE_NOTIFIER_WORKER` | `true` | Toggle outbox worker |
|
|
| `MONLET_NOTIFIER_TICK_SEC` | `5` | Worker poll interval |
|
|
| `MONLET_NOTIFIER_BATCH_SIZE` | `20` | Rows claimed per tick |
|
|
| `MONLET_NOTIFIER_CONCURRENCY` | `4` | Parallel notification deliveries per tick |
|
|
| `MONLET_NOTIFIER_MAX_ATTEMPTS` | `8` | Terminal-failure threshold (ADR-0005) |
|
|
| `MONLET_NOTIFIER_LEASE_SEC` | `300` | Recovery lease for stuck `sending` rows |
|
|
| `MONLET_NOTIFIER_HTTP_TIMEOUT_SEC` | `10` | Outbound HTTP timeout |
|
|
| `MONLET_NOTIFIER_INCLUDE_OUTPUT` | `true` | Include redacted check output in notifications |
|
|
| `MONLET_NOTIFIER_MAX_OUTPUT_BYTES` | `1024` | Maximum notification output bytes after redaction |
|
|
| `MONLET_NOTIFICATION_TIME_ZONE` | `UTC` | IANA timezone for human-readable notification timestamps |
|
|
| `MONLET_NOTIFIER_DEBUG_ENABLED` | `true` | Log-only notifier |
|
|
| `MONLET_NOTIFIER_TELEGRAM_ENABLED` | `false` | Telegram notifier on/off |
|
|
| `MONLET_NOTIFIER_TELEGRAM_TOKEN` | `` | Telegram bot token |
|
|
| `MONLET_NOTIFIER_TELEGRAM_CHAT_ID` | `` | Telegram chat id |
|
|
| `MONLET_NOTIFIER_WEBHOOK_ENABLED` | `false` | Generic webhook notifier on/off |
|
|
| `MONLET_NOTIFIER_WEBHOOK_URL` | `` | Webhook URL |
|
|
| `MONLET_NOTIFIER_WEBHOOK_TOKEN` | `` | Optional Bearer token for webhook |
|
|
| `MONLET_NOTIFIER_ALERTMANAGER_ENABLED` | `false` | Alertmanager notifier on/off |
|
|
| `MONLET_NOTIFIER_ALERTMANAGER_URL` | `` | Alertmanager base URL (e.g. `http://am:9093`) |
|
|
|
|
## Tests
|
|
|
|
Tests use `testcontainers-python` to spin up a real PostgreSQL container; Docker must be available.
|
|
|
|
```sh
|
|
UV_PROJECT_ENVIRONMENT=.venv UV_CACHE_DIR=../.cache/uv uv run pytest -q
|
|
```
|
|
|
|
## Docker
|
|
|
|
Image installs dependencies inside the container only; host venv is never mounted.
|
|
|
|
```sh
|
|
docker build -t monlet-server .
|
|
```
|
|
|
|
## Endpoints
|
|
|
|
All routes under `/api/v1` (per `api/openapi.yaml`). `/metrics` and `/api/v1/health`, `/api/v1/ready` are open.
|
|
|
|
- UI/operator endpoints require `Authorization: Bearer <MONLET_AUTH_TOKEN>`.
|
|
- Agent ingestion endpoints accept the per-agent key generated and stored by the agent. A new key creates a pending agent; events are ignored until an operator accepts it.
|