monlet/agent/README.md

Monlet Agent

Go agent. Reads agent.toml, runs configured commands on schedule, captures status/output, pushes contract-valid heartbeat and events batches to the Monlet server with Bearer auth, and survives outages via a local on-disk spool.

Build and run

cd agent
GOTOOLCHAIN=local GOMODCACHE=$(pwd)/../.cache/go/pkg/mod GOCACHE=$(pwd)/../.cache/go/build GOBIN=$(pwd)/../.cache/go/bin go test ./...
GOTOOLCHAIN=local GOMODCACHE=$(pwd)/../.cache/go/pkg/mod GOCACHE=$(pwd)/../.cache/go/build GOBIN=$(pwd)/../.cache/go/bin go run ./cmd/monlet-agent -config config.example.toml

Do not use go install or go env -w. Module cache, build cache, and tool binaries stay under repo .cache/.

Configuration

See config.example.toml. Required fields: top-level state_dir, at least one [[checks]] (id, command, interval, timeout), and at least one output enabled: server.enabled = true or metrics.enabled = true. agent_id is optional; if omitted, it defaults to the OS hostname. hostname is not a config key.

Key	Default	Notes
server.enabled	false	Enables heartbeat/event push. Requires server.url when true. The agent generates and stores its own Bearer key in state_dir/agent.key.
metrics.enabled	false	Enables /metrics.
labels	{}	Optional heartbeat labels. Max 31 custom labels because monlet_agent_version is generated by the agent. Secret-like keys are rejected.
server.heartbeat_interval	10s	ADR-0005
server.batch_interval	10s	events flush cadence
metrics.listen	127.0.0.1:9465	low-cardinality only
checks[].command	required	Shell string executed as /bin/sh -c. Both forms accepted: command = "check_disk --warn 80 --crit 90" and triple-quoted command = '''...'''. Argv arrays are rejected. See Command security contract.
checks[].notifications_enabled	true	Set false for checks that must not create server notifications.

Examples — one-line and multi-line forms:

[[checks]]
id = "uptime"
command = "test $(cut -d. -f1 /proc/uptime) -gt 60"
interval = "60s"
timeout = "5s"

[[checks]]
id = "disk_root"
command = '''
set -eu
/usr/local/lib/monlet/check_disk_root.sh
'''
interval = "60s"
timeout = "10s"

Command security contract

checks[].command is the only public field for what a check runs. The agent executes it as /bin/sh -c <command> on the host where the agent runs.

• Local-config only. Commands are loaded from the agent's TOML file on disk. The server does not push config and does not execute remote commands; see docs/architecture/security.md#no-remote-execution. Anyone able to edit the agent config or replace the agent binary already has local root-equivalent power, so shell execution adds no new attack surface beyond that file.
• Argv arrays (command = ["..."]) are intentionally rejected by config validation. Use a shell string for both short and long commands.
• Quoting and escaping are the operator's responsibility. Triple-quoted '''...''' is recommended for multi-line scripts and avoids most escaping.
• Do not put secrets in command. Pass them via the systemd unit Environment= / EnvironmentFile= and reference them from the script. Label keys containing token, secret, password, credential, authorization, cookie, api_key/apikey are rejected by config validation to keep secrets out of heartbeats.
• Each run uses exec.CommandContext with Setpgid. On timeout the whole process group is killed and the check result becomes critical.
• Combined stdout+stderr is truncated by UTF-8 byte length to 8 KiB with marker ...[truncated N bytes] before being persisted, exposed, or notified.

Behaviour

• Per-check scheduler with no-overlap: a tick is skipped if the previous run is still in flight (monlet_agent_check_skipped_total).
• Check commands run through /bin/sh -c; the old argv-array form is intentionally unsupported.
• Timeout via exec.CommandContext with Setpgid so the process group is killed on expiry; timeout → critical.
• Exit code mapping: 0=ok, 1=warning, 2=critical, 3+=unknown (ADR-0005).
• output (stdout+stderr) is truncated by UTF-8 byte length to 8 KiB with marker ...[truncated N bytes].
• event_id is UUIDv7 generated on check completion and preserved across spool replay (ADR-0006 idempotency).
• Spool: per-event JSON files in state_dir/spool/, FIFO, limits 10 000 events / 50 MiB, drop oldest on overflow (ADR-0005). Drops are counted in monlet_agent_events_dropped_total{reason} with bounded reasons; one warning is logged per ~30s drop burst, not per event.
• Retry backoff for heartbeat and events: 1s → 30s, exp ×2, jitter ±20%. 4xx are dropped (not retried) to avoid hot loop; dropped batches are counted as reason=send_non_retryable and logged once per batch.
• Bearer auth header on both endpoints uses the local state_dir/agent.key value (ADR-0007).
• Heartbeats include configured labels plus reserved monlet_agent_version=<binary version>.
• Heartbeats include explicit feature flags: push from server.enabled, metrics from metrics.enabled.

Prometheus metrics

Exposed on metrics.listen at /metrics. Labels are static (check_id, status, kind). The agent caps checks at 256 (PH-019) and each id must match [A-Za-z0-9._:-]+; do not embed dynamic values (timestamps, sequence numbers, hostnames) in id because metric cardinality scales with it on Prometheus.

• monlet_agent_check_runs_total{check_id,status}
• monlet_agent_check_skipped_total{check_id}
• monlet_agent_check_duration_seconds{check_id} (histogram)
• monlet_agent_check_status{check_id} (0=ok, 1=warning, 2=critical, 3=unknown)
• monlet_agent_check_exit_code{check_id}
• monlet_agent_check_last_run_timestamp_seconds{check_id}
• monlet_agent_check_last_success_timestamp_seconds{check_id}
• monlet_agent_check_interval_seconds{check_id}
• monlet_agent_checks_configured
• monlet_agent_spool_events, monlet_agent_spool_bytes
• monlet_agent_send_attempts_total{kind}, monlet_agent_send_failures_total{kind} (kind ∈ heartbeat, events)
• monlet_agent_events_accepted_total, monlet_agent_events_deduplicated_total
• monlet_agent_events_dropped_total{reason} — reason ∈ spool_overflow_events | spool_overflow_bytes | send_non_retryable
• monlet_agent_last_heartbeat_timestamp_seconds
• monlet_agent_build_info{version,push,metrics} (constant 1)

systemd

See systemd/monlet-agent.service.example.