# Monlet Agent

Go agent. Reads `agent.toml`, runs configured commands on schedule, captures status/output, pushes contract-valid `heartbeat` and `events` batches to the Monlet server with Bearer auth, and survives outages via a local on-disk spool.

## Build and run

```sh
cd agent
GOTOOLCHAIN=local GOMODCACHE=$(pwd)/../.cache/go/pkg/mod GOCACHE=$(pwd)/../.cache/go/build GOBIN=$(pwd)/../.cache/go/bin go test ./...
GOTOOLCHAIN=local GOMODCACHE=$(pwd)/../.cache/go/pkg/mod GOCACHE=$(pwd)/../.cache/go/build GOBIN=$(pwd)/../.cache/go/bin go run ./cmd/monlet-agent -config config.example.toml
```

Do not use `go install` or `go env -w`. Module cache, build cache, and tool binaries stay under repo `.cache/`.

## Configuration

See `config.example.toml`. Required fields: top-level `state_dir`, at least one `[[checks]]` (`id`, `command`, `timeout`, and exactly one of `interval` or `cron`), and at least one output enabled: `server.enabled = true` or `metrics.enabled = true`. `agent_id` is optional; if omitted, it defaults to the OS hostname. `hostname` is not a config key.

| Key | Default | Notes |
|---|---|---|
| `server.enabled` | `false` | Enables heartbeat/event push. Requires `server.url` when true. The agent generates and stores its own Bearer key in `state_dir/agent.key`. |
| `metrics.enabled` | `false` | Enables `/metrics`. |
| `labels` | `{}` | Optional heartbeat labels. Max 31 custom labels because `monlet_agent_version` is generated by the agent. Secret-like keys are rejected. |
| `server.heartbeat_interval` | `10s` | ADR-0005 |
| `server.batch_interval` | `10s` | events flush cadence |
| `metrics.listen` | `127.0.0.1:9465` | low-cardinality only |
| `checks[].command` | required | Shell string executed as `/bin/sh -c`. Both forms accepted: `command = "check_disk --warn 80 --crit 90"` and triple-quoted `command = '''...'''`. Argv arrays are rejected. See [Command security contract](#command-security-contract). |
| `checks[].interval` | required unless `cron` is set | Go duration such as `60s`, `5m`, or `1h`. Interval checks run once immediately on start/reload, then on the interval. |
| `checks[].cron` | required unless `interval` is set | Standard 5-field cron expression, macros such as `@hourly`/`@daily`, and optional `CRON_TZ=...`. Seconds fields, `@every`, and `@reboot` are rejected. Cron checks wait for the next scheduled slot; they do not run immediately on start/reload. |
| `checks[].notifications_enabled` | `true` | Set `false` for checks that must not create server notifications. |
| `checks[].resource_limits` | `{}` | Optional per-check best-effort `ulimit` values: `cpu_time`, `memory`, `open_files`. |

Examples — interval, cron, and multi-line forms:

```toml
[[checks]]
id = "uptime"
command = "test $(cut -d. -f1 /proc/uptime) -gt 60"
interval = "60s"
timeout = "5s"

[[checks]]
id = "daily_backup"
command = "/usr/local/lib/monlet/check_backup_freshness.sh"
cron = "CRON_TZ=UTC 15 6 * * *"
timeout = "2m"

[[checks]]
id = "disk_root"
command = '''
set -eu
/usr/local/lib/monlet/check_disk_root.sh
'''
interval = "60s"
timeout = "10s"
resource_limits = { cpu_time = "5s", memory = "256MiB", open_files = 128 }
```

## Command security contract

`checks[].command` is the only public field for what a check runs. The agent
executes it as `/bin/sh -c <command>` on the host where the agent runs.

- Local-config only. Commands are loaded from the agent's TOML file on disk.
  The server does not push config and does not execute remote commands. Anyone
  able to edit the agent config or replace the agent binary already has local
  root-equivalent power, so shell execution adds no new attack surface beyond
  that file.
- Argv arrays (`command = ["..."]`) are intentionally rejected by config
  validation. Use a shell string for both short and long commands.
- Quoting and escaping are the operator's responsibility. Triple-quoted
  `'''...'''` is recommended for multi-line scripts and avoids most escaping.
- Do not put secrets in `command`. Pass them via the systemd unit
  `Environment=` / `EnvironmentFile=` and reference them from the script.
  Label keys containing `token`, `secret`, `password`, `credential`,
  `authorization`, `cookie`, `api_key`/`apikey` are rejected by config
  validation to keep secrets out of heartbeats.
- Each run uses `exec.CommandContext` with `Setpgid`. On timeout the whole
  process group is killed and the check result becomes `critical`.
- Optional `resource_limits` are applied inside the child shell before the
  configured command runs. `cpu_time` uses `ulimit -t` and maps SIGXCPU to
  `critical`; `memory` uses virtual memory/address-space `ulimit -v`; `open_files`
  uses `ulimit -n`. These limits are best-effort and are not a cgroup/RSS limit.
- Combined stdout+stderr is truncated by UTF-8 byte length to 8 KiB with marker
  `...[truncated N bytes]` before being persisted, exposed, or notified.

## Config reload

Send `SIGHUP` or run `systemctl reload monlet-agent` to reload the local TOML file.

- Reload first validates the full new config. On error, the old config keeps running.
- Hot-reloadable: labels and checks, including `command`, `interval`, `cron`,
  `timeout`, `notifications_enabled`, `dedupe_key`, and `resource_limits`.
- Restart required: `agent_id`, `state_dir`, `server.*`, and `metrics.*`.
- A check already running during reload is allowed to finish under its old config.
  Removed or changed checks do not start new old-config runs. Changed interval
  checks start with the new config after the in-flight run finishes; changed cron
  checks wait for the next cron slot.

## Behaviour

- Per-check scheduler with no-overlap: a tick/cron slot is skipped if the previous run is still in flight (`monlet_agent_check_skipped_total`).
- Check commands run through `/bin/sh -c`; the old argv-array form is intentionally unsupported.
- Timeout via `exec.CommandContext` with `Setpgid` so the process group is killed on expiry; timeout → `critical`.
- Exit code mapping: 0=ok, 1=warning, 2=critical, 3+=unknown (ADR-0005).
- `output` (stdout+stderr) is truncated by **UTF-8 byte length** to 8 KiB with marker `...[truncated N bytes]`.
- `event_id` is UUIDv7 generated on check completion and preserved across spool replay (ADR-0006 idempotency).
- Spool: per-event JSON files in `state_dir/spool/`, FIFO, limits 10 000 events / 50 MiB, drop oldest on overflow (ADR-0005). Drops are counted in `monlet_agent_events_dropped_total{reason}` with bounded reasons; one warning is logged per ~30s drop burst, not per event.
- Event flushes are split before POST so each JSON request body stays within the 1 MiB server limit.
- Retry backoff for heartbeat and events: 1s → 30s, exp ×2, jitter ±20%. 4xx are dropped (not retried) to avoid hot loop; dropped batches are counted as `reason=send_non_retryable` and logged once per batch.
- On shutdown the agent asks workers to stop cleanly; if they do not drain within 30s, the process exits non-zero so the service manager can restart or mark failure.
- Bearer auth header on both endpoints uses the local `state_dir/agent.key` value (ADR-0007).
- Heartbeats include configured labels plus reserved `monlet_agent_version=<binary version>`.
- Heartbeats include explicit feature flags: `push` from `server.enabled`, `metrics` from `metrics.enabled`.

## Prometheus metrics

Exposed on `metrics.listen` at `/metrics`. Labels are static (`check_id`, `status`, `kind`). The agent caps `checks` at 256 (PH-019) and each `id` must match `[A-Za-z0-9._:-]+`; do not embed dynamic values (timestamps, sequence numbers, hostnames) in `id` because metric cardinality scales with it on Prometheus.

- `monlet_agent_check_runs_total{check_id,status}`
- `monlet_agent_check_skipped_total{check_id}`
- `monlet_agent_check_duration_seconds{check_id}` (histogram)
- `monlet_agent_check_status{check_id}` (0=ok, 1=warning, 2=critical, 3=unknown)
- `monlet_agent_check_exit_code{check_id}`
- `monlet_agent_check_last_run_timestamp_seconds{check_id}`
- `monlet_agent_check_last_success_timestamp_seconds{check_id}`
- `monlet_agent_check_interval_seconds{check_id}` (interval checks only)
- `monlet_agent_checks_configured`
- `monlet_agent_config_load_success`
- `monlet_agent_config_reload_total{result}`
- `monlet_agent_config_last_reload_success_timestamp_seconds`
- `monlet_agent_resource_limit_apply_failures_total{check_id,resource}`
- `monlet_agent_events_channel_full_total`
- `monlet_agent_spool_events`, `monlet_agent_spool_bytes`
- `monlet_agent_send_attempts_total{kind}`, `monlet_agent_send_failures_total{kind}` (`kind` ∈ `heartbeat`, `events`)
- `monlet_agent_events_accepted_total`, `monlet_agent_events_deduplicated_total`
- `monlet_agent_events_dropped_total{reason}` — `reason ∈ spool_overflow_events | spool_overflow_bytes | send_non_retryable`
- `monlet_agent_last_heartbeat_timestamp_seconds`
- `monlet_agent_build_info{version,push,metrics}` (constant 1)

## systemd

See `systemd/monlet-agent.service.example`.