monlet/CONFIG_REFERENCE.md

English | Русский

Monlet: Configuration Reference

Monlet consists of agents running on hosts and a central server. The agent generates its local key in state_dir/agent.key; the server sees a new key as pending until an operator accepts the agent in the UI.

Agent TOML

File: usually /etc/monlet/agent.toml.

# Optional. If omitted, the OS hostname is used.
# agent_id = "host-01"

state_dir = "/var/lib/monlet-agent"

[labels]
env = "prod"
role = "api"

[server]
enabled = true
url = "https://monlet.example.com"
heartbeat_interval = "10s"
batch_interval = "10s"

[metrics]
enabled = true
listen = "127.0.0.1:9465"

[[checks]]
id = "disk_root"
name = "Root disk usage"
command = '''
/usr/local/lib/monlet/check_disk_root.sh --warn 80 --crit 90
'''
interval = "60s"
timeout = "10s"
notifications_enabled = true
resource_limits = { cpu_time = "5s", memory = "256MiB", open_files = 128 }

[[checks]]
id = "daily_backup"
name = "Daily backup freshness"
command = "/usr/local/lib/monlet/check_backup_freshness.sh"
cron = "CRON_TZ=UTC 15 6 * * *"
timeout = "2m"

Keys:

• agent_id - stable agent ID; optional, defaults to hostname.
• state_dir - local state, agent key, and spool.
• [labels] - inventory labels; do not put secrets here.
• server.enabled - send heartbeats/events to the server.
• server.url - server base URL.
• metrics.enabled - expose Prometheus /metrics.
• metrics.listen - /metrics listen address.
• checks[].id - stable check ID.
• checks[].command - shell command string; one-line "..." and multi-line '''...''' forms are supported.
• checks[].interval - run period (60s, 5m, 1h); required when checks[].cron is not set.
• checks[].cron - cron-style schedule; required when checks[].interval is not set. Standard 5-field specs, @hourly/@daily, and CRON_TZ=... are supported; seconds-format specs, @every, and @reboot are rejected.
• checks[].timeout - timeout; timed-out checks return critical.
• checks[].notifications_enabled - allow server-side notifications for this check.
• checks[].resource_limits - optional process limits: CPU time, virtual memory, open files.

Each check must define exactly one schedule mode: interval or cron. Interval checks run immediately after start/reload and then on the configured period. Cron checks wait for the next cron slot and do not run immediately on start.

Exit code: 0=ok, 1=warning, 2=critical, 3+=unknown.

Agent env:

• MONLET_AGENT_MAX_CHECKS - maximum checks in config (default 256). This bounds {check_id} metric cardinality. Must be a positive integer; exceeding the limit is a config load error.

Reload:

• systemctl reload monlet-agent reloads the local TOML via SIGHUP.
• Invalid config is not applied; the old config keeps running.
• A running check is not killed on reload; it runs until completion or its timeout.
• When a schedule changes, an interval check runs after the current in-flight run; a cron check waits for the next cron slot.
• Changing agent_id, state_dir, server.*, or metrics.* requires restart.

Server env

File: .env or container environment.

MONLET_AUTH_TOKEN=replace-with-random-ui-token
MONLET_DATABASE_URL=postgresql+asyncpg://monlet:monlet@postgres:5432/monlet
MONLET_LOG_LEVEL=INFO

MONLET_HOST=0.0.0.0
MONLET_PORT=8000

MONLET_BODY_LIMIT_BYTES=1048576
MONLET_OUTPUT_MAX_BYTES=8192
MONLET_MAX_BATCH_EVENTS=200

MONLET_ENABLE_DETECTOR=true
MONLET_STALE_AFTER_SEC=20
MONLET_DEAD_AFTER_SEC=30
MONLET_DETECTOR_TICK_SEC=5

MONLET_DB_POOL_SIZE=10
MONLET_DB_MAX_OVERFLOW=10
MONLET_DB_POOL_TIMEOUT_SEC=30
MONLET_DB_POOL_RECYCLE_SEC=1800

MONLET_EVENTS_RETENTION_MONTHS=36
MONLET_EVENTS_FUTURE_PARTITIONS=3
MONLET_PARTITION_MAINTENANCE_INTERVAL_SEC=3600
MONLET_OUTBOX_RETENTION_MAX_ROWS=50000
MONLET_OUTBOX_PRUNE_BATCH_SIZE=5000

MONLET_EVENT_DEDUP_RETENTION_DAYS=30
MONLET_EVENT_DEDUP_PRUNE_BATCH_SIZE=5000

MONLET_MAX_PENDING_AGENTS=10000
MONLET_PENDING_AGENT_TTL_DAYS=7
MONLET_PENDING_AGENT_PRUNE_BATCH_SIZE=1000

MONLET_ENABLE_NOTIFIER_WORKER=true
MONLET_NOTIFIER_TICK_SEC=5
MONLET_NOTIFIER_BATCH_SIZE=20
MONLET_NOTIFIER_CONCURRENCY=4
MONLET_NOTIFIER_MAX_ATTEMPTS=8
MONLET_NOTIFIER_LEASE_SEC=300
MONLET_NOTIFIER_HTTP_TIMEOUT_SEC=10
MONLET_NOTIFIER_INCLUDE_OUTPUT=true
MONLET_NOTIFIER_MAX_OUTPUT_BYTES=1024
MONLET_NOTIFICATION_TIME_ZONE=UTC

MONLET_NOTIFIER_DEBUG_ENABLED=true
MONLET_NOTIFIER_TELEGRAM_ENABLED=false
MONLET_NOTIFIER_WEBHOOK_ENABLED=false
MONLET_NOTIFIER_ALERTMANAGER_ENABLED=false

Basics:

• MONLET_AUTH_TOKEN - Bearer token(s) for UI/operator API, not for agents. Space/comma-separated lists support overlap rotation.
• MONLET_DATABASE_URL - PostgreSQL DSN (postgresql+asyncpg://...).
• MONLET_LOG_LEVEL - log level.
• MONLET_HOST / MONLET_PORT - HTTP server host and port (default 0.0.0.0:8000).

Ingestion limits:

• MONLET_BODY_LIMIT_BYTES - maximum request body size (default 1 MiB = 1048576).
• MONLET_OUTPUT_MAX_BYTES - stored event output limit (default 8192).
• MONLET_MAX_BATCH_EVENTS - maximum events in one batch (default 200).

Liveness detector:

• MONLET_ENABLE_DETECTOR - enable stale/dead background recalculation (default true).
• MONLET_STALE_AFTER_SEC - seconds without heartbeat before an agent becomes stale.
• MONLET_DEAD_AFTER_SEC - seconds without heartbeat before an agent becomes dead.
• MONLET_DETECTOR_TICK_SEC - liveness recalculation period.

DB pool:

• MONLET_DB_POOL_SIZE - persistent pool connections (default 10).
• MONLET_DB_MAX_OVERFLOW - extra connections above the pool (default 10).
• MONLET_DB_POOL_TIMEOUT_SEC - wait time for a free connection (default 30).
• MONLET_DB_POOL_RECYCLE_SEC - recycle connections by age (default 1800).

Storage and maintenance:

• MONLET_EVENTS_RETENTION_MONTHS - event partition retention (default 36).
• MONLET_EVENTS_FUTURE_PARTITIONS - future monthly partitions to pre-create (default 3).
• MONLET_PARTITION_MAINTENANCE_INTERVAL_SEC - partition maintenance and prune period (default 3600).
• MONLET_OUTBOX_RETENTION_MAX_ROWS - upper bound for outbox rows; extra terminal rows are pruned (default 50000).
• MONLET_OUTBOX_PRUNE_BATCH_SIZE - terminal outbox prune batch size (default 5000).
• MONLET_EVENT_DEDUP_RETENTION_DAYS - event_id idempotency record retention (default 30).
• MONLET_EVENT_DEDUP_PRUNE_BATCH_SIZE - dedup prune batch size (default 5000).
• MONLET_MAX_PENDING_AGENTS - maximum unaccepted new agents (default 10000).
• MONLET_PENDING_AGENT_TTL_DAYS - old pending-key cleanup TTL (default 7).
• MONLET_PENDING_AGENT_PRUNE_BATCH_SIZE - old pending-key prune batch size (default 1000).

Notifier worker:

• MONLET_ENABLE_NOTIFIER_WORKER - enable notification delivery worker (default true).
• MONLET_NOTIFIER_TICK_SEC - outbox poll period (default 5).
• MONLET_NOTIFIER_BATCH_SIZE - outbox rows claimed per tick (default 20).
• MONLET_NOTIFIER_CONCURRENCY - parallel deliveries inside one tick (default 4).
• MONLET_NOTIFIER_MAX_ATTEMPTS - maximum delivery attempts before permanent failure (default 8).
• MONLET_NOTIFIER_LEASE_SEC - row lease duration during delivery, to prevent another worker from claiming it (default 300).
• MONLET_NOTIFIER_HTTP_TIMEOUT_SEC - HTTP delivery timeout (default 10).
• MONLET_NOTIFIER_INCLUDE_OUTPUT - include output text in notifications (default true).
• MONLET_NOTIFIER_MAX_OUTPUT_BYTES - notification output truncation after redaction (default 1024).
• MONLET_NOTIFICATION_TIME_ZONE - timezone for human-readable notification timestamps.

Notifier channels:

• Telegram: MONLET_NOTIFIER_TELEGRAM_ENABLED=true, MONLET_NOTIFIER_TELEGRAM_TOKEN, MONLET_NOTIFIER_TELEGRAM_CHAT_ID.
• Webhook: MONLET_NOTIFIER_WEBHOOK_ENABLED=true, MONLET_NOTIFIER_WEBHOOK_URL, optional MONLET_NOTIFIER_WEBHOOK_TOKEN.
• Alertmanager: MONLET_NOTIFIER_ALERTMANAGER_ENABLED=true, MONLET_NOTIFIER_ALERTMANAGER_URL.
• Debug/log: MONLET_NOTIFIER_DEBUG_ENABLED=true.

Web env

The web UI reads API/auth/timezone settings from env. Without auth settings, read-only pages are available locally, but admission mutations return 403.

MONLET_API_BASE_URL=http://127.0.0.1:8000
MONLET_API_TOKEN=replace-with-server-token
MONLET_WEB_TIME_ZONE=UTC
MONLET_WEB_AUTH_USERNAME=admin
MONLET_WEB_AUTH_PASSWORD=replace-with-strong-password
MONLET_WEB_SESSION_SECRET=replace-with-random-32-bytes-min
MONLET_WEB_SESSION_TTL_SEC=604800
NEXT_PUBLIC_MONLET_POLL_MS=10000
# Or, instead of local login, trust reverse-proxy auth:
# MONLET_WEB_TRUST_PROXY_AUTH=true

• MONLET_API_BASE_URL - server base URL for server-side fetch.
• MONLET_API_TOKEN - Bearer token for UI requests to the server; it is not sent to the browser.
• MONLET_WEB_TIME_ZONE - IANA timezone used to render timestamps (default UTC).
• MONLET_WEB_AUTH_USERNAME / MONLET_WEB_AUTH_PASSWORD - local cookie-login credentials.
• MONLET_WEB_SESSION_SECRET - session signing secret, at least 32 bytes; shorter values make UI auth misconfigured.
• MONLET_WEB_SESSION_TTL_SEC - session lifetime (default 604800 = 7 days).
• MONLET_WEB_TRUST_PROXY_AUTH - true trusts the X-Forwarded-User header from a reverse proxy. Mutually exclusive with local login; configuring both modes is an error (UI returns 500).
• NEXT_PUBLIC_MONLET_POLL_MS - browser polling interval in milliseconds (default 10000); for production Next.js builds, this is a build-time variable.

Minimal Flow

1. Server starts with PostgreSQL and MONLET_AUTH_TOKEN.
2. Agent starts from TOML, creates state_dir/agent.key, and sends heartbeat.
3. The new agent appears in UI as pending.
4. Operator accepts the agent.
5. After accept, the server starts accounting for events, checks, incidents, and notifications.