Add web auth, infinite-scroll, agent admission and review fixes across agent/server/web
This commit is contained in:
@@ -26,6 +26,7 @@ See `config.example.toml`. Required fields: top-level `state_dir`, at least one
|
||||
| `metrics.listen` | `127.0.0.1:9465` | low-cardinality only |
|
||||
| `checks[].command` | required | Shell string executed as `/bin/sh -c`. Both forms accepted: `command = "check_disk --warn 80 --crit 90"` and triple-quoted `command = '''...'''`. Argv arrays are rejected. See [Command security contract](#command-security-contract). |
|
||||
| `checks[].notifications_enabled` | `true` | Set `false` for checks that must not create server notifications. |
|
||||
| `checks[].resource_limits` | `{}` | Optional per-check best-effort `ulimit` values: `cpu_time`, `memory`, `open_files`. |
|
||||
|
||||
Examples — one-line and multi-line forms:
|
||||
|
||||
@@ -44,6 +45,7 @@ set -eu
|
||||
'''
|
||||
interval = "60s"
|
||||
timeout = "10s"
|
||||
resource_limits = { cpu_time = "5s", memory = "256MiB", open_files = 128 }
|
||||
```
|
||||
|
||||
## Command security contract
|
||||
@@ -67,9 +69,25 @@ executes it as `/bin/sh -c <command>` on the host where the agent runs.
|
||||
validation to keep secrets out of heartbeats.
|
||||
- Each run uses `exec.CommandContext` with `Setpgid`. On timeout the whole
|
||||
process group is killed and the check result becomes `critical`.
|
||||
- Optional `resource_limits` are applied inside the child shell before the
|
||||
configured command runs. `cpu_time` uses `ulimit -t` and maps SIGXCPU to
|
||||
`critical`; `memory` uses virtual memory/address-space `ulimit -v`; `open_files`
|
||||
uses `ulimit -n`. These limits are best-effort and are not a cgroup/RSS limit.
|
||||
- Combined stdout+stderr is truncated by UTF-8 byte length to 8 KiB with marker
|
||||
`...[truncated N bytes]` before being persisted, exposed, or notified.
|
||||
|
||||
## Config reload
|
||||
|
||||
Send `SIGHUP` or run `systemctl reload monlet-agent` to reload the local TOML file.
|
||||
|
||||
- Reload first validates the full new config. On error, the old config keeps running.
|
||||
- Hot-reloadable: labels and checks, including `command`, `interval`, `timeout`,
|
||||
`notifications_enabled`, `dedupe_key`, and `resource_limits`.
|
||||
- Restart required: `agent_id`, `state_dir`, `server.*`, and `metrics.*`.
|
||||
- A check already running during reload is allowed to finish under its old config.
|
||||
Removed or changed checks do not start new old-config runs; changed checks start
|
||||
with the new config after the in-flight run finishes.
|
||||
|
||||
## Behaviour
|
||||
|
||||
- Per-check scheduler with no-overlap: a tick is skipped if the previous run is still in flight (`monlet_agent_check_skipped_total`).
|
||||
@@ -79,7 +97,9 @@ executes it as `/bin/sh -c <command>` on the host where the agent runs.
|
||||
- `output` (stdout+stderr) is truncated by **UTF-8 byte length** to 8 KiB with marker `...[truncated N bytes]`.
|
||||
- `event_id` is UUIDv7 generated on check completion and preserved across spool replay (ADR-0006 idempotency).
|
||||
- Spool: per-event JSON files in `state_dir/spool/`, FIFO, limits 10 000 events / 50 MiB, drop oldest on overflow (ADR-0005). Drops are counted in `monlet_agent_events_dropped_total{reason}` with bounded reasons; one warning is logged per ~30s drop burst, not per event.
|
||||
- Event flushes are split before POST so each JSON request body stays within the 1 MiB server limit.
|
||||
- Retry backoff for heartbeat and events: 1s → 30s, exp ×2, jitter ±20%. 4xx are dropped (not retried) to avoid hot loop; dropped batches are counted as `reason=send_non_retryable` and logged once per batch.
|
||||
- On shutdown the agent asks workers to stop cleanly; if they do not drain within 30s, the process exits non-zero so the service manager can restart or mark failure.
|
||||
- Bearer auth header on both endpoints uses the local `state_dir/agent.key` value (ADR-0007).
|
||||
- Heartbeats include configured labels plus reserved `monlet_agent_version=<binary version>`.
|
||||
- Heartbeats include explicit feature flags: `push` from `server.enabled`, `metrics` from `metrics.enabled`.
|
||||
@@ -97,6 +117,11 @@ Exposed on `metrics.listen` at `/metrics`. Labels are static (`check_id`, `statu
|
||||
- `monlet_agent_check_last_success_timestamp_seconds{check_id}`
|
||||
- `monlet_agent_check_interval_seconds{check_id}`
|
||||
- `monlet_agent_checks_configured`
|
||||
- `monlet_agent_config_load_success`
|
||||
- `monlet_agent_config_reload_total{result}`
|
||||
- `monlet_agent_config_last_reload_success_timestamp_seconds`
|
||||
- `monlet_agent_resource_limit_apply_failures_total{check_id,resource}`
|
||||
- `monlet_agent_events_channel_full_total`
|
||||
- `monlet_agent_spool_events`, `monlet_agent_spool_bytes`
|
||||
- `monlet_agent_send_attempts_total{kind}`, `monlet_agent_send_failures_total{kind}` (`kind` ∈ `heartbeat`, `events`)
|
||||
- `monlet_agent_events_accepted_total`, `monlet_agent_events_deduplicated_total`
|
||||
|
||||
Reference in New Issue
Block a user