60 lines
3.3 KiB
Markdown
60 lines
3.3 KiB
Markdown
# Monlet Agent
|
||
|
||
Go agent (Stage 2 MVP). Reads `agent.toml`, runs configured commands on schedule, captures status/output, pushes contract-valid `heartbeat` and `events` batches to the Monlet server with Bearer auth, and survives outages via a local on-disk spool.
|
||
|
||
## Build and run
|
||
|
||
```sh
|
||
cd agent
|
||
GOTOOLCHAIN=local GOMODCACHE=$(pwd)/../.cache/go/pkg/mod GOCACHE=$(pwd)/../.cache/go/build GOBIN=$(pwd)/../.cache/go/bin go test ./...
|
||
GOTOOLCHAIN=local GOMODCACHE=$(pwd)/../.cache/go/pkg/mod GOCACHE=$(pwd)/../.cache/go/build GOBIN=$(pwd)/../.cache/go/bin go run ./cmd/monlet-agent -config config.example.toml
|
||
```
|
||
|
||
Do not use `go install` or `go env -w`. Module cache, build cache, and tool binaries stay under repo `.cache/`.
|
||
|
||
## Configuration
|
||
|
||
See `config.example.toml`. Required sections: top-level `state_dir`, `[server]` (`url`, `token`), at least one `[[checks]]` (`id`, `command`, `interval`, `timeout`). `agent_id` is optional — if empty, the agent generates and persists a UUID under `state_dir/agent_id`.
|
||
|
||
| Key | Default | Notes |
|
||
|---|---|---|
|
||
| `mode` | `hybrid` | `prometheus_only` (only `/metrics`), `push_only` (only heartbeat+events), `hybrid` (both). `prometheus_only` does not require `server.url`/`server.token`. |
|
||
| `server.heartbeat_interval` | `30s` | ADR-0005 |
|
||
| `server.batch_interval` | `10s` | events flush cadence |
|
||
| `metrics.listen` | `127.0.0.1:9465` | low-cardinality only |
|
||
| `checks[].notification_owner` | `server` | `server`, `prometheus`, `none` |
|
||
|
||
## Behaviour
|
||
|
||
- Per-check scheduler with no-overlap: a tick is skipped if the previous run is still in flight (`monlet_agent_check_skipped_total`).
|
||
- Timeout via `exec.CommandContext` with `Setpgid` so the process group is killed on expiry; timeout → `unknown`.
|
||
- Exit code mapping: 0=ok, 1=warning, 2=critical, 3+=unknown (ADR-0005).
|
||
- `output` (stdout+stderr) is truncated by **UTF-8 byte length** to 8 KiB with marker `...[truncated N bytes]`.
|
||
- `event_id` is UUIDv7 generated on check completion and preserved across spool replay (ADR-0006 idempotency).
|
||
- Spool: per-event JSON files in `state_dir/spool/`, FIFO, limits 10 000 events / 50 MiB, drop oldest on overflow (ADR-0005).
|
||
- Retry backoff for heartbeat and events: 1s → 30s, exp ×2, jitter ±20%. 4xx are dropped (not retried) to avoid hot loop.
|
||
- Bearer auth header on both endpoints (ADR-0007).
|
||
|
||
## Prometheus metrics
|
||
|
||
Exposed on `metrics.listen` at `/metrics`. Labels are static (`check_id`, `status`, `kind`):
|
||
|
||
- `monlet_agent_check_runs_total{check_id,status}`
|
||
- `monlet_agent_check_skipped_total{check_id}`
|
||
- `monlet_agent_check_duration_seconds{check_id}` (histogram)
|
||
- `monlet_agent_check_status{check_id}` (0=ok, 1=warning, 2=critical, 3=unknown)
|
||
- `monlet_agent_check_exit_code{check_id}`
|
||
- `monlet_agent_check_last_run_timestamp_seconds{check_id}`
|
||
- `monlet_agent_check_last_success_timestamp_seconds{check_id}`
|
||
- `monlet_agent_check_interval_seconds{check_id}`
|
||
- `monlet_agent_checks_configured`
|
||
- `monlet_agent_spool_events`, `monlet_agent_spool_bytes`
|
||
- `monlet_agent_send_attempts_total{kind}`, `monlet_agent_send_failures_total{kind}` (`kind` ∈ `heartbeat`, `events`)
|
||
- `monlet_agent_events_accepted_total`, `monlet_agent_events_deduplicated_total`
|
||
- `monlet_agent_last_heartbeat_timestamp_seconds`
|
||
- `monlet_agent_build_info{version,mode}` (constant 1)
|
||
|
||
## systemd
|
||
|
||
See `systemd/monlet-agent.service.example`.
|