Harden production workflows and agent admission
This commit is contained in:
Stanislav Rossovskii
@@ -1,6 +1,6 @@
|
||||
# Monlet Agent
|
||||
|
||||
Go agent (Stage 2 MVP). Reads `agent.toml`, runs configured commands on schedule, captures status/output, pushes contract-valid `heartbeat` and `events` batches to the Monlet server with Bearer auth, and survives outages via a local on-disk spool.
|
||||
Go agent. Reads `agent.toml`, runs configured commands on schedule, captures status/output, pushes contract-valid `heartbeat` and `events` batches to the Monlet server with Bearer auth, and survives outages via a local on-disk spool.
|
||||
|
||||
## Build and run
|
||||
|
||||
@@ -14,22 +14,28 @@ Do not use `go install` or `go env -w`. Module cache, build cache, and tool bina
|
||||
|
||||
## Configuration
|
||||
|
||||
See `config.example.toml`. Required fields: top-level `state_dir`, at least one `[[checks]]` (`id`, `command`, `interval`, `timeout`), and at least one output enabled: `server.enabled = true` or `metrics.enabled = true`. `agent_id` is optional; if omitted, it defaults to `hostname`.
|
||||
See `config.example.toml`. Required fields: top-level `state_dir`, at least one `[[checks]]` (`id`, `command`, `interval`, `timeout`), and at least one output enabled: `server.enabled = true` or `metrics.enabled = true`. `agent_id` is optional; if omitted, it defaults to the OS hostname. `hostname` is not a config key.
|
||||
|
||||
| Key | Default | Notes |
|
||||
|---|---|---|
|
||||
| `server.enabled` | `false` | Enables heartbeat/event push. Requires `server.url` and `server.token` when true. |
|
||||
| `server.enabled` | `false` | Enables heartbeat/event push. Requires `server.url` when true. The agent generates and stores its own Bearer key in `state_dir/agent.key`. |
|
||||
| `metrics.enabled` | `false` | Enables `/metrics`. |
|
||||
| `labels` | `{}` | Optional heartbeat labels. Max 31 custom labels because `monlet_agent_version` is generated by the agent. Secret-like keys are rejected. |
|
||||
| `server.heartbeat_interval` | `30s` | ADR-0005 |
|
||||
| `server.heartbeat_interval` | `10s` | ADR-0005 |
|
||||
| `server.batch_interval` | `10s` | events flush cadence |
|
||||
| `metrics.listen` | `127.0.0.1:9465` | low-cardinality only |
|
||||
| `checks[].command` | required | Shell string executed as `/bin/sh -c`; TOML multi-line strings are supported. |
|
||||
| `checks[].command` | required | Shell string executed as `/bin/sh -c`. Both forms accepted: `command = "check_disk --warn 80 --crit 90"` and triple-quoted `command = '''...'''`. Argv arrays are rejected. See [Command security contract](#command-security-contract). |
|
||||
| `checks[].notifications_enabled` | `true` | Set `false` for checks that must not create server notifications. |
|
||||
|
||||
Example:
|
||||
Examples — one-line and multi-line forms:
|
||||
|
||||
```toml
|
||||
[[checks]]
|
||||
id = "uptime"
|
||||
command = "test $(cut -d. -f1 /proc/uptime) -gt 60"
|
||||
interval = "60s"
|
||||
timeout = "5s"
|
||||
|
||||
[[checks]]
|
||||
id = "disk_root"
|
||||
command = '''
|
||||
@@ -40,23 +46,47 @@ interval = "60s"
|
||||
timeout = "10s"
|
||||
```
|
||||
|
||||
## Command security contract
|
||||
|
||||
`checks[].command` is the only public field for what a check runs. The agent
|
||||
executes it as `/bin/sh -c <command>` on the host where the agent runs.
|
||||
|
||||
- Local-config only. Commands are loaded from the agent's TOML file on disk.
|
||||
The server does not push config and does not execute remote commands; see
|
||||
`docs/architecture/security.md#no-remote-execution`. Anyone able to edit the
|
||||
agent config or replace the agent binary already has local root-equivalent
|
||||
power, so shell execution adds no new attack surface beyond that file.
|
||||
- Argv arrays (`command = ["..."]`) are intentionally rejected by config
|
||||
validation. Use a shell string for both short and long commands.
|
||||
- Quoting and escaping are the operator's responsibility. Triple-quoted
|
||||
`'''...'''` is recommended for multi-line scripts and avoids most escaping.
|
||||
- Do not put secrets in `command`. Pass them via the systemd unit
|
||||
`Environment=` / `EnvironmentFile=` and reference them from the script.
|
||||
Label keys containing `token`, `secret`, `password`, `credential`,
|
||||
`authorization`, `cookie`, `api_key`/`apikey` are rejected by config
|
||||
validation to keep secrets out of heartbeats.
|
||||
- Each run uses `exec.CommandContext` with `Setpgid`. On timeout the whole
|
||||
process group is killed and the check result becomes `critical`.
|
||||
- Combined stdout+stderr is truncated by UTF-8 byte length to 8 KiB with marker
|
||||
`...[truncated N bytes]` before being persisted, exposed, or notified.
|
||||
|
||||
## Behaviour
|
||||
|
||||
- Per-check scheduler with no-overlap: a tick is skipped if the previous run is still in flight (`monlet_agent_check_skipped_total`).
|
||||
- Check commands run through `/bin/sh -c`; the old argv-array form is intentionally unsupported.
|
||||
- Timeout via `exec.CommandContext` with `Setpgid` so the process group is killed on expiry; timeout → `unknown`.
|
||||
- Timeout via `exec.CommandContext` with `Setpgid` so the process group is killed on expiry; timeout → `critical`.
|
||||
- Exit code mapping: 0=ok, 1=warning, 2=critical, 3+=unknown (ADR-0005).
|
||||
- `output` (stdout+stderr) is truncated by **UTF-8 byte length** to 8 KiB with marker `...[truncated N bytes]`.
|
||||
- `event_id` is UUIDv7 generated on check completion and preserved across spool replay (ADR-0006 idempotency).
|
||||
- Spool: per-event JSON files in `state_dir/spool/`, FIFO, limits 10 000 events / 50 MiB, drop oldest on overflow (ADR-0005).
|
||||
- Retry backoff for heartbeat and events: 1s → 30s, exp ×2, jitter ±20%. 4xx are dropped (not retried) to avoid hot loop.
|
||||
- Bearer auth header on both endpoints (ADR-0007).
|
||||
- Spool: per-event JSON files in `state_dir/spool/`, FIFO, limits 10 000 events / 50 MiB, drop oldest on overflow (ADR-0005). Drops are counted in `monlet_agent_events_dropped_total{reason}` with bounded reasons; one warning is logged per ~30s drop burst, not per event.
|
||||
- Retry backoff for heartbeat and events: 1s → 30s, exp ×2, jitter ±20%. 4xx are dropped (not retried) to avoid hot loop; dropped batches are counted as `reason=send_non_retryable` and logged once per batch.
|
||||
- Bearer auth header on both endpoints uses the local `state_dir/agent.key` value (ADR-0007).
|
||||
- Heartbeats include configured labels plus reserved `monlet_agent_version=<binary version>`.
|
||||
- Heartbeats include explicit feature flags: `push` from `server.enabled`, `metrics` from `metrics.enabled`.
|
||||
|
||||
## Prometheus metrics
|
||||
|
||||
Exposed on `metrics.listen` at `/metrics`. Labels are static (`check_id`, `status`, `kind`):
|
||||
Exposed on `metrics.listen` at `/metrics`. Labels are static (`check_id`, `status`, `kind`). The agent caps `checks` at 256 (PH-019) and each `id` must match `[A-Za-z0-9._:-]+`; do not embed dynamic values (timestamps, sequence numbers, hostnames) in `id` because metric cardinality scales with it on Prometheus.
|
||||
|
||||
- `monlet_agent_check_runs_total{check_id,status}`
|
||||
- `monlet_agent_check_skipped_total{check_id}`
|
||||
@@ -70,6 +100,7 @@ Exposed on `metrics.listen` at `/metrics`. Labels are static (`check_id`, `statu
|
||||
- `monlet_agent_spool_events`, `monlet_agent_spool_bytes`
|
||||
- `monlet_agent_send_attempts_total{kind}`, `monlet_agent_send_failures_total{kind}` (`kind` ∈ `heartbeat`, `events`)
|
||||
- `monlet_agent_events_accepted_total`, `monlet_agent_events_deduplicated_total`
|
||||
- `monlet_agent_events_dropped_total{reason}` — `reason ∈ spool_overflow_events | spool_overflow_bytes | send_non_retryable`
|
||||
- `monlet_agent_last_heartbeat_timestamp_seconds`
|
||||
- `monlet_agent_build_info{version,push,metrics}` (constant 1)
|
||||
|
||||
|
||||
Reference in New Issue
Block a user