monlet/agent/README.md

Monlet Agent

Go agent. Reads agent.toml, runs configured commands on schedule, captures status/output, pushes contract-valid heartbeat and events batches to the Monlet server with Bearer auth, and survives outages via a local on-disk spool.

Build and run

cd agent
GOTOOLCHAIN=local GOMODCACHE=$(pwd)/../.cache/go/pkg/mod GOCACHE=$(pwd)/../.cache/go/build GOBIN=$(pwd)/../.cache/go/bin go test ./...
GOTOOLCHAIN=local GOMODCACHE=$(pwd)/../.cache/go/pkg/mod GOCACHE=$(pwd)/../.cache/go/build GOBIN=$(pwd)/../.cache/go/bin go run ./cmd/monlet-agent -config config.example.toml

Do not use go install or go env -w. Module cache, build cache, and tool binaries stay under repo .cache/.

Configuration

See config.example.toml. Required fields: top-level state_dir, at least one [[checks]] (id, command, interval, timeout), and at least one output enabled: server.enabled = true or metrics.enabled = true. agent_id is optional; if omitted, it defaults to the OS hostname. hostname is not a config key.

Key	Default	Notes
server.enabled	false	Enables heartbeat/event push. Requires server.url when true. The agent generates and stores its own Bearer key in state_dir/agent.key.
metrics.enabled	false	Enables /metrics.
labels	{}	Optional heartbeat labels. Max 31 custom labels because monlet_agent_version is generated by the agent. Secret-like keys are rejected.
server.heartbeat_interval	10s	ADR-0005
server.batch_interval	10s	events flush cadence
metrics.listen	127.0.0.1:9465	low-cardinality only
checks[].command	required	Shell string executed as /bin/sh -c. Both forms accepted: command = "check_disk --warn 80 --crit 90" and triple-quoted command = '''...'''. Argv arrays are rejected. See Command security contract.
checks[].notifications_enabled	true	Set false for checks that must not create server notifications.
checks[].resource_limits	{}	Optional per-check best-effort ulimit values: cpu_time, memory, open_files.

Examples — one-line and multi-line forms:

[[checks]]
id = "uptime"
command = "test $(cut -d. -f1 /proc/uptime) -gt 60"
interval = "60s"
timeout = "5s"

[[checks]]
id = "disk_root"
command = '''
set -eu
/usr/local/lib/monlet/check_disk_root.sh
'''
interval = "60s"
timeout = "10s"
resource_limits = { cpu_time = "5s", memory = "256MiB", open_files = 128 }

Command security contract

checks[].command is the only public field for what a check runs. The agent executes it as /bin/sh -c <command> on the host where the agent runs.

• Local-config only. Commands are loaded from the agent's TOML file on disk. The server does not push config and does not execute remote commands. Anyone able to edit the agent config or replace the agent binary already has local root-equivalent power, so shell execution adds no new attack surface beyond that file.
• Argv arrays (command = ["..."]) are intentionally rejected by config validation. Use a shell string for both short and long commands.
• Quoting and escaping are the operator's responsibility. Triple-quoted '''...''' is recommended for multi-line scripts and avoids most escaping.
• Do not put secrets in command. Pass them via the systemd unit Environment= / EnvironmentFile= and reference them from the script. Label keys containing token, secret, password, credential, authorization, cookie, api_key/apikey are rejected by config validation to keep secrets out of heartbeats.
• Each run uses exec.CommandContext with Setpgid. On timeout the whole process group is killed and the check result becomes critical.
• Optional resource_limits are applied inside the child shell before the configured command runs. cpu_time uses ulimit -t and maps SIGXCPU to critical; memory uses virtual memory/address-space ulimit -v; open_files uses ulimit -n. These limits are best-effort and are not a cgroup/RSS limit.
• Combined stdout+stderr is truncated by UTF-8 byte length to 8 KiB with marker ...[truncated N bytes] before being persisted, exposed, or notified.

Config reload

Send SIGHUP or run systemctl reload monlet-agent to reload the local TOML file.

• Reload first validates the full new config. On error, the old config keeps running.
• Hot-reloadable: labels and checks, including command, interval, timeout, notifications_enabled, dedupe_key, and resource_limits.
• Restart required: agent_id, state_dir, server.*, and metrics.*.
• A check already running during reload is allowed to finish under its old config. Removed or changed checks do not start new old-config runs; changed checks start with the new config after the in-flight run finishes.

Behaviour

• Per-check scheduler with no-overlap: a tick is skipped if the previous run is still in flight (monlet_agent_check_skipped_total).
• Check commands run through /bin/sh -c; the old argv-array form is intentionally unsupported.
• Timeout via exec.CommandContext with Setpgid so the process group is killed on expiry; timeout → critical.
• Exit code mapping: 0=ok, 1=warning, 2=critical, 3+=unknown (ADR-0005).
• output (stdout+stderr) is truncated by UTF-8 byte length to 8 KiB with marker ...[truncated N bytes].
• event_id is UUIDv7 generated on check completion and preserved across spool replay (ADR-0006 idempotency).
• Spool: per-event JSON files in state_dir/spool/, FIFO, limits 10 000 events / 50 MiB, drop oldest on overflow (ADR-0005). Drops are counted in monlet_agent_events_dropped_total{reason} with bounded reasons; one warning is logged per ~30s drop burst, not per event.
• Event flushes are split before POST so each JSON request body stays within the 1 MiB server limit.
• Retry backoff for heartbeat and events: 1s → 30s, exp ×2, jitter ±20%. 4xx are dropped (not retried) to avoid hot loop; dropped batches are counted as reason=send_non_retryable and logged once per batch.
• On shutdown the agent asks workers to stop cleanly; if they do not drain within 30s, the process exits non-zero so the service manager can restart or mark failure.
• Bearer auth header on both endpoints uses the local state_dir/agent.key value (ADR-0007).
• Heartbeats include configured labels plus reserved monlet_agent_version=<binary version>.
• Heartbeats include explicit feature flags: push from server.enabled, metrics from metrics.enabled.

Prometheus metrics

Exposed on metrics.listen at /metrics. Labels are static (check_id, status, kind). The agent caps checks at 256 (PH-019) and each id must match [A-Za-z0-9._:-]+; do not embed dynamic values (timestamps, sequence numbers, hostnames) in id because metric cardinality scales with it on Prometheus.

• monlet_agent_check_runs_total{check_id,status}
• monlet_agent_check_skipped_total{check_id}
• monlet_agent_check_duration_seconds{check_id} (histogram)
• monlet_agent_check_status{check_id} (0=ok, 1=warning, 2=critical, 3=unknown)
• monlet_agent_check_exit_code{check_id}
• monlet_agent_check_last_run_timestamp_seconds{check_id}
• monlet_agent_check_last_success_timestamp_seconds{check_id}
• monlet_agent_check_interval_seconds{check_id}
• monlet_agent_checks_configured
• monlet_agent_config_load_success
• monlet_agent_config_reload_total{result}
• monlet_agent_config_last_reload_success_timestamp_seconds
• monlet_agent_resource_limit_apply_failures_total{check_id,resource}
• monlet_agent_events_channel_full_total
• monlet_agent_spool_events, monlet_agent_spool_bytes
• monlet_agent_send_attempts_total{kind}, monlet_agent_send_failures_total{kind} (kind ∈ heartbeat, events)
• monlet_agent_events_accepted_total, monlet_agent_events_deduplicated_total
• monlet_agent_events_dropped_total{reason} — reason ∈ spool_overflow_events | spool_overflow_bytes | send_non_retryable
• monlet_agent_last_heartbeat_timestamp_seconds
• monlet_agent_build_info{version,push,metrics} (constant 1)

systemd

See systemd/monlet-agent.service.example.