|
|
|
@@ -0,0 +1,231 @@
|
|
|
|
|
|
|
|
English | [Русский](CONFIG_REFERENCE.ru.md)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
# Monlet: Configuration Reference
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Monlet consists of agents running on hosts and a central server. The agent generates its local key in `state_dir/agent.key`; the server sees a new key as `pending` until an operator accepts the agent in the UI.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
## Agent TOML
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
File: usually `/etc/monlet/agent.toml`.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
```toml
|
|
|
|
|
|
|
|
# Optional. If omitted, the OS hostname is used.
|
|
|
|
|
|
|
|
# agent_id = "host-01"
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
state_dir = "/var/lib/monlet-agent"
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
[labels]
|
|
|
|
|
|
|
|
env = "prod"
|
|
|
|
|
|
|
|
role = "api"
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
[server]
|
|
|
|
|
|
|
|
enabled = true
|
|
|
|
|
|
|
|
url = "https://monlet.example.com"
|
|
|
|
|
|
|
|
heartbeat_interval = "10s"
|
|
|
|
|
|
|
|
batch_interval = "10s"
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
[metrics]
|
|
|
|
|
|
|
|
enabled = true
|
|
|
|
|
|
|
|
listen = "127.0.0.1:9465"
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
[[checks]]
|
|
|
|
|
|
|
|
id = "disk_root"
|
|
|
|
|
|
|
|
name = "Root disk usage"
|
|
|
|
|
|
|
|
command = '''
|
|
|
|
|
|
|
|
/usr/local/lib/monlet/check_disk_root.sh --warn 80 --crit 90
|
|
|
|
|
|
|
|
'''
|
|
|
|
|
|
|
|
interval = "60s"
|
|
|
|
|
|
|
|
timeout = "10s"
|
|
|
|
|
|
|
|
notifications_enabled = true
|
|
|
|
|
|
|
|
resource_limits = { cpu_time = "5s", memory = "256MiB", open_files = 128 }
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
[[checks]]
|
|
|
|
|
|
|
|
id = "daily_backup"
|
|
|
|
|
|
|
|
name = "Daily backup freshness"
|
|
|
|
|
|
|
|
command = "/usr/local/lib/monlet/check_backup_freshness.sh"
|
|
|
|
|
|
|
|
cron = "CRON_TZ=UTC 15 6 * * *"
|
|
|
|
|
|
|
|
timeout = "2m"
|
|
|
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Keys:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
- `agent_id` - stable agent ID; optional, defaults to hostname.
|
|
|
|
|
|
|
|
- `state_dir` - local state, agent key, and spool.
|
|
|
|
|
|
|
|
- `[labels]` - inventory labels; do not put secrets here.
|
|
|
|
|
|
|
|
- `server.enabled` - send heartbeats/events to the server.
|
|
|
|
|
|
|
|
- `server.url` - server base URL.
|
|
|
|
|
|
|
|
- `metrics.enabled` - expose Prometheus `/metrics`.
|
|
|
|
|
|
|
|
- `metrics.listen` - `/metrics` listen address.
|
|
|
|
|
|
|
|
- `checks[].id` - stable check ID.
|
|
|
|
|
|
|
|
- `checks[].command` - shell command string; one-line `"..."` and multi-line `'''...'''` forms are supported.
|
|
|
|
|
|
|
|
- `checks[].interval` - run period (`60s`, `5m`, `1h`); required when `checks[].cron` is not set.
|
|
|
|
|
|
|
|
- `checks[].cron` - cron-style schedule; required when `checks[].interval` is not set. Standard 5-field specs, `@hourly`/`@daily`, and `CRON_TZ=...` are supported; seconds-format specs, `@every`, and `@reboot` are rejected.
|
|
|
|
|
|
|
|
- `checks[].timeout` - timeout; timed-out checks return `critical`.
|
|
|
|
|
|
|
|
- `checks[].notifications_enabled` - allow server-side notifications for this check.
|
|
|
|
|
|
|
|
- `checks[].resource_limits` - optional process limits: CPU time, virtual memory, open files.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Each check must define exactly one schedule mode: `interval` or `cron`. Interval checks run immediately after start/reload and then on the configured period. Cron checks wait for the next cron slot and do not run immediately on start.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Exit code: `0=ok`, `1=warning`, `2=critical`, `3+=unknown`.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Agent env:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
- `MONLET_AGENT_MAX_CHECKS` - maximum checks in config (default 256). This bounds `{check_id}` metric cardinality. Must be a positive integer; exceeding the limit is a config load error.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Reload:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
- `systemctl reload monlet-agent` reloads the local TOML via `SIGHUP`.
|
|
|
|
|
|
|
|
- Invalid config is not applied; the old config keeps running.
|
|
|
|
|
|
|
|
- A running check is not killed on reload; it runs until completion or its `timeout`.
|
|
|
|
|
|
|
|
- When a schedule changes, an interval check runs after the current in-flight run; a cron check waits for the next cron slot.
|
|
|
|
|
|
|
|
- Changing `agent_id`, `state_dir`, `server.*`, or `metrics.*` requires restart.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
## Server env
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
File: `.env` or container environment.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
```env
|
|
|
|
|
|
|
|
MONLET_AUTH_TOKEN=replace-with-random-ui-token
|
|
|
|
|
|
|
|
MONLET_DATABASE_URL=postgresql+asyncpg://monlet:monlet@postgres:5432/monlet
|
|
|
|
|
|
|
|
MONLET_LOG_LEVEL=INFO
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
MONLET_HOST=0.0.0.0
|
|
|
|
|
|
|
|
MONLET_PORT=8000
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
MONLET_BODY_LIMIT_BYTES=1048576
|
|
|
|
|
|
|
|
MONLET_OUTPUT_MAX_BYTES=8192
|
|
|
|
|
|
|
|
MONLET_MAX_BATCH_EVENTS=200
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
MONLET_ENABLE_DETECTOR=true
|
|
|
|
|
|
|
|
MONLET_STALE_AFTER_SEC=20
|
|
|
|
|
|
|
|
MONLET_DEAD_AFTER_SEC=30
|
|
|
|
|
|
|
|
MONLET_DETECTOR_TICK_SEC=5
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
MONLET_DB_POOL_SIZE=10
|
|
|
|
|
|
|
|
MONLET_DB_MAX_OVERFLOW=10
|
|
|
|
|
|
|
|
MONLET_DB_POOL_TIMEOUT_SEC=30
|
|
|
|
|
|
|
|
MONLET_DB_POOL_RECYCLE_SEC=1800
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
MONLET_EVENTS_RETENTION_MONTHS=36
|
|
|
|
|
|
|
|
MONLET_EVENTS_FUTURE_PARTITIONS=3
|
|
|
|
|
|
|
|
MONLET_PARTITION_MAINTENANCE_INTERVAL_SEC=3600
|
|
|
|
|
|
|
|
MONLET_OUTBOX_RETENTION_MAX_ROWS=50000
|
|
|
|
|
|
|
|
MONLET_OUTBOX_PRUNE_BATCH_SIZE=5000
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
MONLET_EVENT_DEDUP_RETENTION_DAYS=30
|
|
|
|
|
|
|
|
MONLET_EVENT_DEDUP_PRUNE_BATCH_SIZE=5000
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
MONLET_MAX_PENDING_AGENTS=10000
|
|
|
|
|
|
|
|
MONLET_PENDING_AGENT_TTL_DAYS=7
|
|
|
|
|
|
|
|
MONLET_PENDING_AGENT_PRUNE_BATCH_SIZE=1000
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
MONLET_ENABLE_NOTIFIER_WORKER=true
|
|
|
|
|
|
|
|
MONLET_NOTIFIER_TICK_SEC=5
|
|
|
|
|
|
|
|
MONLET_NOTIFIER_BATCH_SIZE=20
|
|
|
|
|
|
|
|
MONLET_NOTIFIER_CONCURRENCY=4
|
|
|
|
|
|
|
|
MONLET_NOTIFIER_MAX_ATTEMPTS=8
|
|
|
|
|
|
|
|
MONLET_NOTIFIER_LEASE_SEC=300
|
|
|
|
|
|
|
|
MONLET_NOTIFIER_HTTP_TIMEOUT_SEC=10
|
|
|
|
|
|
|
|
MONLET_NOTIFIER_INCLUDE_OUTPUT=true
|
|
|
|
|
|
|
|
MONLET_NOTIFIER_MAX_OUTPUT_BYTES=1024
|
|
|
|
|
|
|
|
MONLET_NOTIFICATION_TIME_ZONE=UTC
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
MONLET_NOTIFIER_DEBUG_ENABLED=true
|
|
|
|
|
|
|
|
MONLET_NOTIFIER_TELEGRAM_ENABLED=false
|
|
|
|
|
|
|
|
MONLET_NOTIFIER_WEBHOOK_ENABLED=false
|
|
|
|
|
|
|
|
MONLET_NOTIFIER_ALERTMANAGER_ENABLED=false
|
|
|
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Basics:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
- `MONLET_AUTH_TOKEN` - Bearer token(s) for UI/operator API, not for agents. Space/comma-separated lists support overlap rotation.
|
|
|
|
|
|
|
|
- `MONLET_DATABASE_URL` - PostgreSQL DSN (`postgresql+asyncpg://...`).
|
|
|
|
|
|
|
|
- `MONLET_LOG_LEVEL` - log level.
|
|
|
|
|
|
|
|
- `MONLET_HOST` / `MONLET_PORT` - HTTP server host and port (default `0.0.0.0:8000`).
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Ingestion limits:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
- `MONLET_BODY_LIMIT_BYTES` - maximum request body size (default 1 MiB = 1048576).
|
|
|
|
|
|
|
|
- `MONLET_OUTPUT_MAX_BYTES` - stored event `output` limit (default 8192).
|
|
|
|
|
|
|
|
- `MONLET_MAX_BATCH_EVENTS` - maximum events in one batch (default 200).
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Liveness detector:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
- `MONLET_ENABLE_DETECTOR` - enable stale/dead background recalculation (default `true`).
|
|
|
|
|
|
|
|
- `MONLET_STALE_AFTER_SEC` - seconds without heartbeat before an agent becomes `stale`.
|
|
|
|
|
|
|
|
- `MONLET_DEAD_AFTER_SEC` - seconds without heartbeat before an agent becomes `dead`.
|
|
|
|
|
|
|
|
- `MONLET_DETECTOR_TICK_SEC` - liveness recalculation period.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
DB pool:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
- `MONLET_DB_POOL_SIZE` - persistent pool connections (default 10).
|
|
|
|
|
|
|
|
- `MONLET_DB_MAX_OVERFLOW` - extra connections above the pool (default 10).
|
|
|
|
|
|
|
|
- `MONLET_DB_POOL_TIMEOUT_SEC` - wait time for a free connection (default 30).
|
|
|
|
|
|
|
|
- `MONLET_DB_POOL_RECYCLE_SEC` - recycle connections by age (default 1800).
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Storage and maintenance:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
- `MONLET_EVENTS_RETENTION_MONTHS` - event partition retention (default 36).
|
|
|
|
|
|
|
|
- `MONLET_EVENTS_FUTURE_PARTITIONS` - future monthly partitions to pre-create (default 3).
|
|
|
|
|
|
|
|
- `MONLET_PARTITION_MAINTENANCE_INTERVAL_SEC` - partition maintenance and prune period (default 3600).
|
|
|
|
|
|
|
|
- `MONLET_OUTBOX_RETENTION_MAX_ROWS` - upper bound for outbox rows; extra terminal rows are pruned (default 50000).
|
|
|
|
|
|
|
|
- `MONLET_OUTBOX_PRUNE_BATCH_SIZE` - terminal outbox prune batch size (default 5000).
|
|
|
|
|
|
|
|
- `MONLET_EVENT_DEDUP_RETENTION_DAYS` - `event_id` idempotency record retention (default 30).
|
|
|
|
|
|
|
|
- `MONLET_EVENT_DEDUP_PRUNE_BATCH_SIZE` - dedup prune batch size (default 5000).
|
|
|
|
|
|
|
|
- `MONLET_MAX_PENDING_AGENTS` - maximum unaccepted new agents (default 10000).
|
|
|
|
|
|
|
|
- `MONLET_PENDING_AGENT_TTL_DAYS` - old pending-key cleanup TTL (default 7).
|
|
|
|
|
|
|
|
- `MONLET_PENDING_AGENT_PRUNE_BATCH_SIZE` - old pending-key prune batch size (default 1000).
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Notifier worker:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
- `MONLET_ENABLE_NOTIFIER_WORKER` - enable notification delivery worker (default `true`).
|
|
|
|
|
|
|
|
- `MONLET_NOTIFIER_TICK_SEC` - outbox poll period (default 5).
|
|
|
|
|
|
|
|
- `MONLET_NOTIFIER_BATCH_SIZE` - outbox rows claimed per tick (default 20).
|
|
|
|
|
|
|
|
- `MONLET_NOTIFIER_CONCURRENCY` - parallel deliveries inside one tick (default 4).
|
|
|
|
|
|
|
|
- `MONLET_NOTIFIER_MAX_ATTEMPTS` - maximum delivery attempts before permanent failure (default 8).
|
|
|
|
|
|
|
|
- `MONLET_NOTIFIER_LEASE_SEC` - row lease duration during delivery, to prevent another worker from claiming it (default 300).
|
|
|
|
|
|
|
|
- `MONLET_NOTIFIER_HTTP_TIMEOUT_SEC` - HTTP delivery timeout (default 10).
|
|
|
|
|
|
|
|
- `MONLET_NOTIFIER_INCLUDE_OUTPUT` - include `output` text in notifications (default `true`).
|
|
|
|
|
|
|
|
- `MONLET_NOTIFIER_MAX_OUTPUT_BYTES` - notification output truncation after redaction (default 1024).
|
|
|
|
|
|
|
|
- `MONLET_NOTIFICATION_TIME_ZONE` - timezone for human-readable notification timestamps.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Notifier channels:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
- Telegram: `MONLET_NOTIFIER_TELEGRAM_ENABLED=true`, `MONLET_NOTIFIER_TELEGRAM_TOKEN`, `MONLET_NOTIFIER_TELEGRAM_CHAT_ID`.
|
|
|
|
|
|
|
|
- Webhook: `MONLET_NOTIFIER_WEBHOOK_ENABLED=true`, `MONLET_NOTIFIER_WEBHOOK_URL`, optional `MONLET_NOTIFIER_WEBHOOK_TOKEN`.
|
|
|
|
|
|
|
|
- Alertmanager: `MONLET_NOTIFIER_ALERTMANAGER_ENABLED=true`, `MONLET_NOTIFIER_ALERTMANAGER_URL`.
|
|
|
|
|
|
|
|
- Debug/log: `MONLET_NOTIFIER_DEBUG_ENABLED=true`.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
## Web env
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
The web UI reads API/auth/timezone settings from env. Without auth settings, read-only pages are available locally, but admission mutations return `403`.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
```env
|
|
|
|
|
|
|
|
MONLET_API_BASE_URL=http://127.0.0.1:8000
|
|
|
|
|
|
|
|
MONLET_API_TOKEN=replace-with-server-token
|
|
|
|
|
|
|
|
MONLET_WEB_TIME_ZONE=UTC
|
|
|
|
|
|
|
|
MONLET_WEB_AUTH_USERNAME=admin
|
|
|
|
|
|
|
|
MONLET_WEB_AUTH_PASSWORD=replace-with-strong-password
|
|
|
|
|
|
|
|
MONLET_WEB_SESSION_SECRET=replace-with-random-32-bytes-min
|
|
|
|
|
|
|
|
MONLET_WEB_SESSION_TTL_SEC=604800
|
|
|
|
|
|
|
|
NEXT_PUBLIC_MONLET_POLL_MS=10000
|
|
|
|
|
|
|
|
# Or, instead of local login, trust reverse-proxy auth:
|
|
|
|
|
|
|
|
# MONLET_WEB_TRUST_PROXY_AUTH=true
|
|
|
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
- `MONLET_API_BASE_URL` - server base URL for server-side fetch.
|
|
|
|
|
|
|
|
- `MONLET_API_TOKEN` - Bearer token for UI requests to the server; it is not sent to the browser.
|
|
|
|
|
|
|
|
- `MONLET_WEB_TIME_ZONE` - IANA timezone used to render timestamps (default `UTC`).
|
|
|
|
|
|
|
|
- `MONLET_WEB_AUTH_USERNAME` / `MONLET_WEB_AUTH_PASSWORD` - local cookie-login credentials.
|
|
|
|
|
|
|
|
- `MONLET_WEB_SESSION_SECRET` - session signing secret, at least 32 bytes; shorter values make UI auth misconfigured.
|
|
|
|
|
|
|
|
- `MONLET_WEB_SESSION_TTL_SEC` - session lifetime (default 604800 = 7 days).
|
|
|
|
|
|
|
|
- `MONLET_WEB_TRUST_PROXY_AUTH` - `true` trusts the `X-Forwarded-User` header from a reverse proxy. Mutually exclusive with local login; configuring both modes is an error (UI returns `500`).
|
|
|
|
|
|
|
|
- `NEXT_PUBLIC_MONLET_POLL_MS` - browser polling interval in milliseconds (default 10000); for production Next.js builds, this is a build-time variable.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
## Minimal Flow
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1. Server starts with PostgreSQL and `MONLET_AUTH_TOKEN`.
|
|
|
|
|
|
|
|
2. Agent starts from TOML, creates `state_dir/agent.key`, and sends heartbeat.
|
|
|
|
|
|
|
|
3. The new agent appears in UI as `pending`.
|
|
|
|
|
|
|
|
4. Operator accepts the agent.
|
|
|
|
|
|
|
|
5. After accept, the server starts accounting for events, checks, incidents, and notifications.
|
Stanislav Rossovskii