Monlet Server
FastAPI server: heartbeat/event ingestion, current check state, incident lifecycle, stale/dead detector, notification outbox, Prometheus metrics.
Stack
- Python 3.14+
uv- FastAPI, Pydantic v2
- SQLAlchemy 2.x async +
asyncpg - Alembic (sync
psycopg) structlog,prometheus-client- PostgreSQL 16+
Local development
cd server
UV_PROJECT_ENVIRONMENT=.venv UV_CACHE_DIR=../.cache/uv uv sync --extra dev
Run a Postgres instance (Docker or local), set MONLET_DATABASE_URL, then:
UV_PROJECT_ENVIRONMENT=.venv UV_CACHE_DIR=../.cache/uv uv run alembic upgrade head
UV_PROJECT_ENVIRONMENT=.venv UV_CACHE_DIR=../.cache/uv uv run fastapi dev monlet_server/main.py
Environment
| Var | Default | Purpose |
|---|---|---|
MONLET_AUTH_TOKEN |
required | UI/operator Bearer token. Space/comma-separated list; any accepted. changeme is rejected. |
MONLET_DATABASE_URL |
postgresql+asyncpg://monlet:monlet@127.0.0.1:5432/monlet |
Async DSN |
MONLET_LOG_LEVEL |
INFO |
Log level |
MONLET_HOST |
0.0.0.0 |
HTTP bind host for direct server runs |
MONLET_PORT |
8000 |
HTTP bind port for direct server runs |
MONLET_BODY_LIMIT_BYTES |
1048576 |
Request body limit (ADR-0005) |
MONLET_OUTPUT_MAX_BYTES |
8192 |
Per-event output limit |
MONLET_MAX_BATCH_EVENTS |
200 |
Maximum events per ingestion batch |
MONLET_STALE_AFTER_SEC |
20 |
Stale threshold |
MONLET_DEAD_AFTER_SEC |
30 |
Dead threshold |
MONLET_DETECTOR_TICK_SEC |
5 |
Detector tick |
MONLET_ENABLE_DETECTOR |
true |
Toggle detector task |
MONLET_DB_POOL_SIZE |
10 |
Base async DB pool size |
MONLET_DB_MAX_OVERFLOW |
10 |
Extra DB connections above pool size |
MONLET_DB_POOL_TIMEOUT_SEC |
30 |
Seconds to wait for a DB connection |
MONLET_DB_POOL_RECYCLE_SEC |
1800 |
Recycle DB connections by age |
MONLET_EVENTS_RETENTION_MONTHS |
36 |
Months of events partitions to keep; 0 disables drop |
MONLET_EVENTS_FUTURE_PARTITIONS |
3 |
Future month partitions to pre-create |
MONLET_EVENT_DEDUP_RETENTION_DAYS |
30 |
Idempotency window for event_id |
MONLET_EVENT_DEDUP_PRUNE_BATCH_SIZE |
5000 |
Dedup rows pruned per maintenance tick |
MONLET_MAX_PENDING_AGENTS |
10000 |
Maximum pending agent keys before heartbeat rejects with 429 |
MONLET_PENDING_AGENT_TTL_DAYS |
7 |
Pending key cleanup TTL; 0 disables pruning |
MONLET_PENDING_AGENT_PRUNE_BATCH_SIZE |
1000 |
Pending cleanup rows per detector tick |
MONLET_PARTITION_MAINTENANCE_INTERVAL_SEC |
3600 |
Partition create/drop worker interval |
MONLET_OUTBOX_RETENTION_MAX_ROWS |
50000 |
Maximum retained terminal outbox rows; 0 disables pruning |
MONLET_OUTBOX_PRUNE_BATCH_SIZE |
5000 |
Outbox rows pruned per maintenance tick |
MONLET_ENABLE_NOTIFIER_WORKER |
true |
Toggle outbox worker |
MONLET_NOTIFIER_TICK_SEC |
5 |
Worker poll interval |
MONLET_NOTIFIER_BATCH_SIZE |
20 |
Rows claimed per tick |
MONLET_NOTIFIER_CONCURRENCY |
4 |
Parallel notification deliveries per tick |
MONLET_NOTIFIER_MAX_ATTEMPTS |
8 |
Terminal-failure threshold (ADR-0005) |
MONLET_NOTIFIER_LEASE_SEC |
300 |
Recovery lease for stuck sending rows |
MONLET_NOTIFIER_HTTP_TIMEOUT_SEC |
10 |
Outbound HTTP timeout |
MONLET_NOTIFIER_INCLUDE_OUTPUT |
true |
Include redacted check output in notifications |
MONLET_NOTIFIER_MAX_OUTPUT_BYTES |
1024 |
Maximum notification output bytes after redaction |
MONLET_NOTIFICATION_TIME_ZONE |
UTC |
IANA timezone for human-readable notification timestamps |
MONLET_NOTIFIER_DEBUG_ENABLED |
true |
Log-only notifier |
MONLET_NOTIFIER_TELEGRAM_ENABLED |
false |
Telegram notifier on/off |
MONLET_NOTIFIER_TELEGRAM_TOKEN |
`` | Telegram bot token |
MONLET_NOTIFIER_TELEGRAM_CHAT_ID |
`` | Telegram chat id |
MONLET_NOTIFIER_WEBHOOK_ENABLED |
false |
Generic webhook notifier on/off |
MONLET_NOTIFIER_WEBHOOK_URL |
`` | Webhook URL |
MONLET_NOTIFIER_WEBHOOK_TOKEN |
`` | Optional Bearer token for webhook |
MONLET_NOTIFIER_ALERTMANAGER_ENABLED |
false |
Alertmanager notifier on/off |
MONLET_NOTIFIER_ALERTMANAGER_URL |
`` | Alertmanager base URL (e.g. http://am:9093) |
Tests
Tests use testcontainers-python to spin up a real PostgreSQL container; Docker must be available.
UV_PROJECT_ENVIRONMENT=.venv UV_CACHE_DIR=../.cache/uv uv run pytest -q
Docker
Image installs dependencies inside the container only; host venv is never mounted.
docker build -t monlet-server .
Endpoints
All routes under /api/v1 (per api/openapi.yaml). /metrics and /api/v1/health, /api/v1/ready are open.
- UI/operator endpoints require
Authorization: Bearer <MONLET_AUTH_TOKEN>. - Agent ingestion endpoints accept the per-agent key generated and stored by the agent. A new key creates a pending agent; events are ignored until an operator accepts it.