monlet/agent/README.md

Monlet Agent

Go agent (Stage 2 MVP). Reads agent.toml, runs configured commands on schedule, captures status/output, pushes contract-valid heartbeat and events batches to the Monlet server with Bearer auth, and survives outages via a local on-disk spool.

Build and run

cd agent
GOTOOLCHAIN=local GOMODCACHE=$(pwd)/../.cache/go/pkg/mod GOCACHE=$(pwd)/../.cache/go/build GOBIN=$(pwd)/../.cache/go/bin go test ./...
GOTOOLCHAIN=local GOMODCACHE=$(pwd)/../.cache/go/pkg/mod GOCACHE=$(pwd)/../.cache/go/build GOBIN=$(pwd)/../.cache/go/bin go run ./cmd/monlet-agent -config config.example.toml

Do not use go install or go env -w. Module cache, build cache, and tool binaries stay under repo .cache/.

Configuration

See config.example.toml. Required fields: top-level state_dir, at least one [[checks]] (id, command, interval, timeout), and at least one output enabled: server.enabled = true or metrics.enabled = true. agent_id is optional; if omitted, it defaults to hostname.

Key	Default	Notes
server.enabled	false	Enables heartbeat/event push. Requires server.url and server.token when true.
metrics.enabled	false	Enables /metrics.
labels	{}	Optional heartbeat labels. Max 31 custom labels because monlet_agent_version is generated by the agent. Secret-like keys are rejected.
server.heartbeat_interval	30s	ADR-0005
server.batch_interval	10s	events flush cadence
metrics.listen	127.0.0.1:9465	low-cardinality only
checks[].command	required	Shell string executed as /bin/sh -c; TOML multi-line strings are supported.
checks[].notifications_enabled	true	Set false for checks that must not create server notifications.

Example:

[[checks]]
id = "disk_root"
command = '''
set -eu
/usr/local/lib/monlet/check_disk_root.sh
'''
interval = "60s"
timeout = "10s"

Behaviour

• Per-check scheduler with no-overlap: a tick is skipped if the previous run is still in flight (monlet_agent_check_skipped_total).
• Check commands run through /bin/sh -c; the old argv-array form is intentionally unsupported.
• Timeout via exec.CommandContext with Setpgid so the process group is killed on expiry; timeout → unknown.
• Exit code mapping: 0=ok, 1=warning, 2=critical, 3+=unknown (ADR-0005).
• output (stdout+stderr) is truncated by UTF-8 byte length to 8 KiB with marker ...[truncated N bytes].
• event_id is UUIDv7 generated on check completion and preserved across spool replay (ADR-0006 idempotency).
• Spool: per-event JSON files in state_dir/spool/, FIFO, limits 10 000 events / 50 MiB, drop oldest on overflow (ADR-0005).
• Retry backoff for heartbeat and events: 1s → 30s, exp ×2, jitter ±20%. 4xx are dropped (not retried) to avoid hot loop.
• Bearer auth header on both endpoints (ADR-0007).
• Heartbeats include configured labels plus reserved monlet_agent_version=<binary version>.
• Heartbeats include explicit feature flags: push from server.enabled, metrics from metrics.enabled.

Prometheus metrics

Exposed on metrics.listen at /metrics. Labels are static (check_id, status, kind):

• monlet_agent_check_runs_total{check_id,status}
• monlet_agent_check_skipped_total{check_id}
• monlet_agent_check_duration_seconds{check_id} (histogram)
• monlet_agent_check_status{check_id} (0=ok, 1=warning, 2=critical, 3=unknown)
• monlet_agent_check_exit_code{check_id}
• monlet_agent_check_last_run_timestamp_seconds{check_id}
• monlet_agent_check_last_success_timestamp_seconds{check_id}
• monlet_agent_check_interval_seconds{check_id}
• monlet_agent_checks_configured
• monlet_agent_spool_events, monlet_agent_spool_bytes
• monlet_agent_send_attempts_total{kind}, monlet_agent_send_failures_total{kind} (kind ∈ heartbeat, events)
• monlet_agent_events_accepted_total, monlet_agent_events_deduplicated_total
• monlet_agent_last_heartbeat_timestamp_seconds
• monlet_agent_build_info{version,push,metrics} (constant 1)

systemd

See systemd/monlet-agent.service.example.