creator/monlet
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Harden production workflows and agent admission

Browse Source
This commit is contained in:
Stanislav Rossovskii
2026-05-28 14:19:27 +04:00
parent a2e88b4e76
commit 37b1a1d6d6
109 changed files with 4927 additions and 894 deletions
Download Patch File Download Diff File Expand all files Collapse all files

53
agent/README.md
View File

@@ -1,6 +1,6 @@
# Monlet Agent
Go agent (Stage 2 MVP). Reads `agent.toml`, runs configured commands on schedule, captures status/output, pushes contract-valid `heartbeat` and `events` batches to the Monlet server with Bearer auth, and survives outages via a local on-disk spool.
Go agent. Reads `agent.toml`, runs configured commands on schedule, captures status/output, pushes contract-valid `heartbeat` and `events` batches to the Monlet server with Bearer auth, and survives outages via a local on-disk spool.
## Build and run
@@ -14,22 +14,28 @@ Do not use `go install` or `go env -w`. Module cache, build cache, and tool bina
## Configuration
See `config.example.toml`. Required fields: top-level `state_dir`, at least one `[[checks]]` (`id`, `command`, `interval`, `timeout`), and at least one output enabled: `server.enabled = true` or `metrics.enabled = true`. `agent_id` is optional; if omitted, it defaults to `hostname`.
See `config.example.toml`. Required fields: top-level `state_dir`, at least one `[[checks]]` (`id`, `command`, `interval`, `timeout`), and at least one output enabled: `server.enabled = true` or `metrics.enabled = true`. `agent_id` is optional; if omitted, it defaults to the OS hostname. `hostname` is not a config key.
| Key | Default | Notes |
|---|---|---|
| `server.enabled` | `false` | Enables heartbeat/event push. Requires `server.url` and `server.token` when true. |
| `server.enabled` | `false` | Enables heartbeat/event push. Requires `server.url` when true. The agent generates and stores its own Bearer key in `state_dir/agent.key`. |
| `metrics.enabled` | `false` | Enables `/metrics`. |
| `labels` | `{}` | Optional heartbeat labels. Max 31 custom labels because `monlet_agent_version` is generated by the agent. Secret-like keys are rejected. |
| `server.heartbeat_interval` | `30s` | ADR-0005 |
| `server.heartbeat_interval` | `10s` | ADR-0005 |
| `server.batch_interval` | `10s` | events flush cadence |
| `metrics.listen` | `127.0.0.1:9465` | low-cardinality only |
| `checks[].command` | required | Shell string executed as `/bin/sh -c`; TOML multi-line strings are supported. |
| `checks[].command` | required | Shell string executed as `/bin/sh -c`. Both forms accepted: `command = "check_disk --warn 80 --crit 90"` and triple-quoted `command = '''...'''`. Argv arrays are rejected. See [Command security contract](#command-security-contract). |
| `checks[].notifications_enabled` | `true` | Set `false` for checks that must not create server notifications. |
Example:
Examples — one-line and multi-line forms:
```toml
[[checks]]
id = "uptime"
command = "test $(cut -d. -f1 /proc/uptime) -gt 60"
interval = "60s"
timeout = "5s"
[[checks]]
id = "disk_root"
command = '''
@@ -40,23 +46,47 @@ interval = "60s"
timeout = "10s"
```
## Command security contract
`checks[].command` is the only public field for what a check runs. The agent
executes it as `/bin/sh -c <command>` on the host where the agent runs.
- Local-config only. Commands are loaded from the agent's TOML file on disk.
The server does not push config and does not execute remote commands; see
`docs/architecture/security.md#no-remote-execution`. Anyone able to edit the
agent config or replace the agent binary already has local root-equivalent
power, so shell execution adds no new attack surface beyond that file.
- Argv arrays (`command = ["..."]`) are intentionally rejected by config
validation. Use a shell string for both short and long commands.
- Quoting and escaping are the operator's responsibility. Triple-quoted
`'''...'''` is recommended for multi-line scripts and avoids most escaping.
- Do not put secrets in `command`. Pass them via the systemd unit
`Environment=` / `EnvironmentFile=` and reference them from the script.
Label keys containing `token`, `secret`, `password`, `credential`,
`authorization`, `cookie`, `api_key`/`apikey` are rejected by config
validation to keep secrets out of heartbeats.
- Each run uses `exec.CommandContext` with `Setpgid`. On timeout the whole
process group is killed and the check result becomes `critical`.
- Combined stdout+stderr is truncated by UTF-8 byte length to 8 KiB with marker
`...[truncated N bytes]` before being persisted, exposed, or notified.
## Behaviour
- Per-check scheduler with no-overlap: a tick is skipped if the previous run is still in flight (`monlet_agent_check_skipped_total`).
- Check commands run through `/bin/sh -c`; the old argv-array form is intentionally unsupported.
- Timeout via `exec.CommandContext` with `Setpgid` so the process group is killed on expiry; timeout → `unknown`.
- Timeout via `exec.CommandContext` with `Setpgid` so the process group is killed on expiry; timeout → `critical`.
- Exit code mapping: 0=ok, 1=warning, 2=critical, 3+=unknown (ADR-0005).
- `output` (stdout+stderr) is truncated by **UTF-8 byte length** to 8 KiB with marker `...[truncated N bytes]`.
- `event_id` is UUIDv7 generated on check completion and preserved across spool replay (ADR-0006 idempotency).
- Spool: per-event JSON files in `state_dir/spool/`, FIFO, limits 10 000 events / 50 MiB, drop oldest on overflow (ADR-0005).
- Retry backoff for heartbeat and events: 1s → 30s, exp ×2, jitter ±20%. 4xx are dropped (not retried) to avoid hot loop.
- Bearer auth header on both endpoints (ADR-0007).
- Spool: per-event JSON files in `state_dir/spool/`, FIFO, limits 10 000 events / 50 MiB, drop oldest on overflow (ADR-0005). Drops are counted in `monlet_agent_events_dropped_total{reason}` with bounded reasons; one warning is logged per ~30s drop burst, not per event.
- Retry backoff for heartbeat and events: 1s → 30s, exp ×2, jitter ±20%. 4xx are dropped (not retried) to avoid hot loop; dropped batches are counted as `reason=send_non_retryable` and logged once per batch.
- Bearer auth header on both endpoints uses the local `state_dir/agent.key` value (ADR-0007).
- Heartbeats include configured labels plus reserved `monlet_agent_version=<binary version>`.
- Heartbeats include explicit feature flags: `push` from `server.enabled`, `metrics` from `metrics.enabled`.
## Prometheus metrics
Exposed on `metrics.listen` at `/metrics`. Labels are static (`check_id`, `status`, `kind`):
Exposed on `metrics.listen` at `/metrics`. Labels are static (`check_id`, `status`, `kind`). The agent caps `checks` at 256 (PH-019) and each `id` must match `[A-Za-z0-9._:-]+`; do not embed dynamic values (timestamps, sequence numbers, hostnames) in `id` because metric cardinality scales with it on Prometheus.
- `monlet_agent_check_runs_total{check_id,status}`
- `monlet_agent_check_skipped_total{check_id}`
@@ -70,6 +100,7 @@ Exposed on `metrics.listen` at `/metrics`. Labels are static (`check_id`, `statu
- `monlet_agent_spool_events`, `monlet_agent_spool_bytes`
- `monlet_agent_send_attempts_total{kind}`, `monlet_agent_send_failures_total{kind}` (`kind` ∈ `heartbeat`, `events`)
- `monlet_agent_events_accepted_total`, `monlet_agent_events_deduplicated_total`
- `monlet_agent_events_dropped_total{reason}` — `reason ∈ spool_overflow_events | spool_overflow_bytes | send_non_retryable`
- `monlet_agent_last_heartbeat_timestamp_seconds`
- `monlet_agent_build_info{version,push,metrics}` (constant 1)

12
agent/cmd/monlet-agent/main.go
View File

@@ -9,6 +9,7 @@ import (
"path/filepath"
"syscall"
"github.com/monlet/agent/internal/agentkey"
"github.com/monlet/agent/internal/app"
"github.com/monlet/agent/internal/client"
"github.com/monlet/agent/internal/config"
@@ -43,11 +44,20 @@ func main() {
os.Exit(1)
}
key := ""
if cfg.PushesToServer() {
key, err = agentkey.LoadOrCreate(cfg.StateDir)
if err != nil {
log.Error("agent key", "err", err)
os.Exit(1)
}
}
a := &app.App{
Cfg: cfg,
AgentID: agentID,
Version: version,
Client: client.New(cfg.Server.URL, cfg.Server.Token, agentID),
Client: client.New(cfg.Server.URL, key, agentID),
Spool: sp,
Metrics: metrics.New(),
Log: log,

13
agent/config.example.toml
View File

@@ -1,4 +1,5 @@
hostname = "example-host-01"
# Optional. If omitted, the OS hostname is used as agent_id.
# agent_id = "example-host-01"
state_dir = "/var/lib/monlet-agent"
[labels]
@@ -8,8 +9,7 @@ role = "api"
[server]
enabled = true
url = "http://127.0.0.1:8000"
token = "replace-me"
heartbeat_interval = "30s"
heartbeat_interval = "10s"
batch_interval = "10s"
[metrics]
@@ -24,3 +24,10 @@ command = '''
'''
interval = "60s"
timeout = "10s"
[[checks]]
id = "uptime"
name = "Uptime probe"
command = "test $(cut -d. -f1 /proc/uptime) -gt 60"
interval = "60s"
timeout = "5s"

75
agent/internal/agentkey/agentkey.go Normal file
View File

@@ -0,0 +1,75 @@
package agentkey
import (
"crypto/rand"
"encoding/hex"
"fmt"
"os"
"path/filepath"
"strings"
)
const (
fileName = "agent.key"
keyBytes = 32
keyLen = keyBytes * 2
)
func LoadOrCreate(stateDir string) (string, error) {
if err := os.MkdirAll(stateDir, 0o700); err != nil {
return "", err
}
if err := os.Chmod(stateDir, 0o700); err != nil {
return "", err
}
path := filepath.Join(stateDir, fileName)
if info, err := os.Stat(path); err == nil {
if !info.Mode().IsRegular() {
return "", fmt.Errorf("agent key is not a regular file: %s", path)
}
if info.Mode().Perm() != 0o600 {
if err := os.Chmod(path, 0o600); err != nil {
return "", err
}
}
b, err := os.ReadFile(path)
if err != nil {
return "", err
}
return validate(strings.TrimSpace(string(b)), path)
} else if !os.IsNotExist(err) {
return "", err
}
key, err := generate()
if err != nil {
return "", err
}
tmp := fmt.Sprintf("%s.%d.tmp", path, os.Getpid())
defer os.Remove(tmp)
if err := os.WriteFile(tmp, []byte(key+"\n"), 0o600); err != nil {
return "", err
}
if err := os.Rename(tmp, path); err != nil {
return "", err
}
return key, nil
}
func validate(key string, path string) (string, error) {
if len(key) != keyLen {
return "", fmt.Errorf("invalid agent key in %s", path)
}
if _, err := hex.DecodeString(key); err != nil {
return "", fmt.Errorf("invalid agent key in %s", path)
}
return strings.ToLower(key), nil
}
func generate() (string, error) {
var b [keyBytes]byte
if _, err := rand.Read(b[:]); err != nil {
return "", err
}
return hex.EncodeToString(b[:]), nil
}

75
agent/internal/agentkey/agentkey_test.go Normal file
View File

@@ -0,0 +1,75 @@
package agentkey
import (
"os"
"path/filepath"
"testing"
)
func TestLoadOrCreatePersistsKey(t *testing.T) {
dir := t.TempDir()
k1, err := LoadOrCreate(dir)
if err != nil {
t.Fatal(err)
}
k2, err := LoadOrCreate(dir)
if err != nil {
t.Fatal(err)
}
if k1 != k2 {
t.Fatal("key was not persisted")
}
if len(k1) != keyLen {
t.Fatalf("unexpected key length %d", len(k1))
}
info, err := os.Stat(filepath.Join(dir, fileName))
if err != nil {
t.Fatal(err)
}
if got := info.Mode().Perm(); got != 0o600 {
t.Fatalf("unexpected key mode %o", got)
}
}
func TestLoadOrCreateRejectsInvalidExistingKey(t *testing.T) {
dir := t.TempDir()
if err := os.WriteFile(filepath.Join(dir, fileName), []byte("bad\n"), 0o600); err != nil {
t.Fatal(err)
}
if _, err := LoadOrCreate(dir); err == nil {
t.Fatal("expected invalid key error")
}
}
func TestLoadOrCreateRejectsInvalidHexExistingKey(t *testing.T) {
dir := t.TempDir()
if err := os.WriteFile(filepath.Join(dir, fileName), []byte("zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz\n"), 0o600); err != nil {
t.Fatal(err)
}
if _, err := LoadOrCreate(dir); err == nil {
t.Fatal("expected invalid key error")
}
}
func TestLoadOrCreateRepairsKeyPermissions(t *testing.T) {
dir := t.TempDir()
key := "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
path := filepath.Join(dir, fileName)
if err := os.WriteFile(path, []byte(key+"\n"), 0o644); err != nil {
t.Fatal(err)
}
got, err := LoadOrCreate(dir)
if err != nil {
t.Fatal(err)
}
if got != key {
t.Fatalf("unexpected key %q", got)
}
info, err := os.Stat(path)
if err != nil {
t.Fatal(err)
}
if got := info.Mode().Perm(); got != 0o600 {
t.Fatalf("unexpected key mode %o", got)
}
}

37
agent/internal/app/app.go
View File

@@ -22,11 +22,23 @@ type App struct {
Spool *spool.Spool
Metrics *metrics.Metrics
Log *slog.Logger
lastDropLog time.Time
}
const (
dropReasonOverflowEvents = "spool_overflow_events"
dropReasonOverflowBytes = "spool_overflow_bytes"
dropReasonSendRejected = "send_non_retryable"
dropLogThrottle = 30 * time.Second
)
func (a *App) Run(ctx context.Context) error {
events := make(chan scheduler.Event, 256)
// Account drops accumulated during spool Open (e.g. shrunk limits on restart).
a.recordSpoolDrops(a.Spool.Drops())
a.Metrics.ChecksConfigured.Set(float64(len(a.Cfg.Checks)))
a.Metrics.BuildInfo.WithLabelValues(a.Version, boolLabel(a.Cfg.PushesToServer()), boolLabel(a.Cfg.ExposesMetrics())).Set(1)
for _, c := range a.Cfg.Checks {
@@ -108,6 +120,7 @@ func (a *App) spoolWriter(ctx context.Context, events <-chan scheduler.Event) {
if err := a.Spool.Enqueue(ev); err != nil {
a.Log.Error("spool enqueue", "err", err)
}
a.recordSpoolDrops(a.Spool.Drops())
a.Metrics.SpoolDepth.Set(float64(a.Spool.Len()))
a.Metrics.SpoolBytes.Set(float64(a.Spool.Bytes()))
}
@@ -188,7 +201,9 @@ func (a *App) sendLoop(ctx context.Context) {
}
continue
}
// non-retryable: drop to avoid hot loop
// non-retryable: drop to avoid hot loop.
a.Metrics.EventsDropped.WithLabelValues(dropReasonSendRejected).Add(float64(len(seqs)))
a.Log.Warn("events dropped: server rejected batch", "count", len(seqs), "reason", dropReasonSendRejected)
a.Spool.Commit(seqs)
attempt = 0
continue
@@ -214,6 +229,26 @@ func sleep(ctx context.Context, d time.Duration) bool {
}
}
func (a *App) recordSpoolDrops(d spool.DropStats) {
if d.OverflowEvents == 0 && d.OverflowBytes == 0 {
return
}
if d.OverflowEvents > 0 {
a.Metrics.EventsDropped.WithLabelValues(dropReasonOverflowEvents).Add(float64(d.OverflowEvents))
}
if d.OverflowBytes > 0 {
a.Metrics.EventsDropped.WithLabelValues(dropReasonOverflowBytes).Add(float64(d.OverflowBytes))
}
now := time.Now()
if now.Sub(a.lastDropLog) >= dropLogThrottle {
a.Log.Warn("spool drops",
"events_overflow", d.OverflowEvents,
"bytes_overflow", d.OverflowBytes,
)
a.lastDropLog = now
}
}
func boolLabel(v bool) string {
if v {
return "true"

3
agent/internal/app/app_test.go
View File

@@ -31,7 +31,7 @@ func TestPrometheusOnlyDoesNotPush(t *testing.T) {
cfg := &config.Config{
AgentID: "a1",
StateDir: dir,
Server: config.ServerConfig{Enabled: boolPtr(false), URL: srv.URL, Token: "t", HeartbeatInterval: config.Duration{Duration: 50 * time.Millisecond}, BatchInterval: config.Duration{Duration: 50 * time.Millisecond}},
Server: config.ServerConfig{Enabled: boolPtr(false), URL: srv.URL, HeartbeatInterval: config.Duration{Duration: 50 * time.Millisecond}, BatchInterval: config.Duration{Duration: 50 * time.Millisecond}},
Metrics: config.MetricsConfig{Enabled: true, Listen: "127.0.0.1:0"},
Checks: []config.CheckConfig{{
ID: "c1",
@@ -106,7 +106,6 @@ func TestEndToEndPushAndReplay(t *testing.T) {
Server: config.ServerConfig{
Enabled: boolPtr(true),
URL: srv.URL,
Token: "t",
HeartbeatInterval: config.Duration{Duration: 100 * time.Millisecond},
BatchInterval: config.Duration{Duration: 100 * time.Millisecond},
},

5
agent/internal/client/client.go
View File

@@ -29,7 +29,10 @@ func New(baseURL, token, agentID string) *Client {
baseURL: strings.TrimRight(baseURL, "/"),
token: token,
agentID: agentID,
http: &http.Client{Timeout: 15 * time.Second},
http: &http.Client{
Timeout: 15 * time.Second,
Transport: &http.Transport{DisableKeepAlives: true},
},
}
}

37
agent/internal/config/config.go
View File

@@ -4,6 +4,7 @@ import (
"fmt"
"os"
"regexp"
"strconv"
"strings"
"time"
"unicode/utf8"
@@ -14,19 +15,24 @@ import (
var idPattern = regexp.MustCompile(`^[A-Za-z0-9._:-]+$`)
const (
maxIDLen = 128
maxLabelKeyLen = 64
maxLabelValueLen = 256
maxLabels = 32
maxIDLen = 128
maxLabelKeyLen = 64
maxLabelValueLen = 256
maxLabels = 32
// PH-019: bounded check inventory keeps {check_id} metric cardinality
// predictable. ValidateID already restricts each check_id to a stable
// identifier alphabet; this cap bounds the total. Override via
// MONLET_AGENT_MAX_CHECKS for fleets that genuinely need more.
defaultMaxChecks = 256
AgentVersionLabel = "monlet_agent_version"
defaultHeartbeat = 30 * time.Second
defaultHeartbeat = 10 * time.Second
defaultBatch = 10 * time.Second
defaultMetricsAddr = "127.0.0.1:9465"
)
type Config struct {
AgentID string `toml:"agent_id"`
Hostname string `toml:"hostname"`
Hostname string `toml:"-"`
StateDir string `toml:"state_dir"`
Labels map[string]string `toml:"labels"`
Server ServerConfig `toml:"server"`
@@ -37,7 +43,6 @@ type Config struct {
type ServerConfig struct {
Enabled *bool `toml:"enabled"`
URL string `toml:"url"`
Token string `toml:"token"`
HeartbeatInterval Duration `toml:"heartbeat_interval"`
BatchInterval Duration `toml:"batch_interval"`
}
@@ -124,11 +129,8 @@ func (c *Config) Validate() error {
if c.Server.URL == "" {
return fmt.Errorf("server.url is required when server.enabled = true")
}
if c.Server.Token == "" {
return fmt.Errorf("server.token is required when server.enabled = true")
}
} else if c.Server.URL != "" || c.Server.Token != "" {
return fmt.Errorf("server.url/token require server.enabled = true")
} else if c.Server.URL != "" {
return fmt.Errorf("server.url requires server.enabled = true")
}
if !c.PushesToServer() && !c.ExposesMetrics() {
return fmt.Errorf("server.enabled or metrics.enabled must be true")
@@ -136,6 +138,17 @@ func (c *Config) Validate() error {
if len(c.Checks) == 0 {
return fmt.Errorf("at least one check is required")
}
maxChecks := defaultMaxChecks
if v, ok := os.LookupEnv("MONLET_AGENT_MAX_CHECKS"); ok {
n, err := strconv.Atoi(v)
if err != nil || n <= 0 {
return fmt.Errorf("MONLET_AGENT_MAX_CHECKS must be a positive integer, got %q", v)
}
maxChecks = n
}
if len(c.Checks) > maxChecks {
return fmt.Errorf("too many checks: %d (max %d) — bounded for metric cardinality (PH-019)", len(c.Checks), maxChecks)
}
seen := make(map[string]struct{}, len(c.Checks))
for i := range c.Checks {
ch := &c.Checks[i]

83
agent/internal/config/config_test.go
View File

@@ -24,7 +24,6 @@ state_dir = "/tmp/x"
[server]
enabled = true
url = "http://localhost"
token = "t"
[[checks]]
id = "c1"
@@ -57,7 +56,6 @@ state_dir = "/tmp/x"
[server]
enabled = true
url = "http://localhost"
token = "t"
[[checks]]
id = "c1"
@@ -87,7 +85,6 @@ state_dir = "/tmp/x"
[server]
enabled = true
url = "http://localhost"
token = "t"
[[checks]]
id = "c1"
@@ -100,6 +97,36 @@ timeout = "5s"
}
}
func TestEmptyCommandRejected(t *testing.T) {
for name, cmd := range map[string]string{
"empty": `""`,
"whitespace": `" \t "`,
} {
t.Run(name, func(t *testing.T) {
_, err := Load(writeTmp(t, `
agent_id = "host-1"
state_dir = "/tmp/x"
[server]
enabled = true
url = "http://localhost"
[[checks]]
id = "c1"
command = `+cmd+`
interval = "10s"
timeout = "5s"
`))
if err == nil {
t.Fatal("expected empty command to be rejected")
}
if !strings.Contains(err.Error(), "command is required") {
t.Fatalf("unexpected error: %v", err)
}
})
}
}
func TestLoadLabels(t *testing.T) {
c, err := Load(writeTmp(t, `
agent_id = "host-1"
@@ -112,7 +139,6 @@ role = "api"
[server]
enabled = true
url = "http://localhost"
token = "t"
[[checks]]
id = "c1"
@@ -148,7 +174,6 @@ state_dir = "/tmp/x"
[server]
enabled = true
url = "http://localhost"
token = "t"
[[checks]]
id = "c1"
command = "true"
@@ -167,7 +192,6 @@ state_dir = "/tmp/x"
[server]
enabled = true
url = "http://localhost"
token = "t"
[[checks]]
id = "c1"
command = "true"
@@ -188,7 +212,6 @@ state_dir = "/tmp/x"
[server]
enabled = true
url = "http://localhost"
token = "t"
[[checks]]
id = "c1"
command = "true"
@@ -268,7 +291,28 @@ interval = "10s"
timeout = "5s"
`))
if err == nil {
t.Fatal("push_only must require server.url/token")
t.Fatal("push_only must require server.url")
}
}
func TestServerTokenRejected(t *testing.T) {
_, err := Load(writeTmp(t, `
state_dir = "/tmp/x"
[server]
enabled = true
url = "http://localhost"
token = "obsolete"
[[checks]]
id = "c1"
command = "true"
interval = "10s"
timeout = "5s"
`))
if err == nil {
t.Fatal("expected obsolete server.token to be rejected")
}
if !strings.Contains(err.Error(), "server.token") {
t.Fatalf("unexpected error: %v", err)
}
}
@@ -279,7 +323,6 @@ state_dir = "/tmp/x"
[server]
enabled = true
url = "http://localhost"
token = "t"
[[checks]]
id = "c1"
command = "true"
@@ -291,6 +334,27 @@ timeout = "5s"
}
}
func TestHostnameConfigKeyRejected(t *testing.T) {
_, err := Load(writeTmp(t, `
hostname = "host-1"
state_dir = "/tmp/x"
[server]
enabled = true
url = "http://localhost"
[[checks]]
id = "c1"
command = "true"
interval = "10s"
timeout = "5s"
`))
if err == nil {
t.Fatal("expected hostname key to be rejected")
}
if !strings.Contains(err.Error(), "hostname") {
t.Fatalf("unexpected error: %v", err)
}
}
func TestNotificationsEnabledCanDisable(t *testing.T) {
c, err := Load(writeTmp(t, minimal+`
notifications_enabled = false
@@ -329,7 +393,6 @@ state_dir = "/tmp/x"
[server]
enabled = true
url = "http://localhost"
token = "t"
[[checks]]
id = "c1"
command = "true"

2
agent/internal/ids/ids.go
View File

@@ -19,7 +19,7 @@ func ResolveAgentID(configured, hostname string) (string, error) {
if id == "" {
return "", fmt.Errorf("hostname is empty")
}
if err := config.ValidateID("hostname as agent_id", id); err != nil {
if err := config.ValidateID("OS hostname as agent_id", id); err != nil {
return "", err
}
return id, nil

7
agent/internal/metrics/metrics.go
View File

@@ -28,6 +28,7 @@ type Metrics struct {
SendFailures *prometheus.CounterVec
EventsAccepted prometheus.Counter
EventsDedup prometheus.Counter
EventsDropped *prometheus.CounterVec
LastHeartbeat prometheus.Gauge
BuildInfo *prometheus.GaugeVec
}
@@ -86,6 +87,10 @@ func New() *Metrics {
Name: "monlet_agent_events_deduplicated_total",
Help: "Events the server reported as duplicates.",
})
m.EventsDropped = prometheus.NewCounterVec(prometheus.CounterOpts{
Name: "monlet_agent_events_dropped_total",
Help: "Events dropped by the agent before server acceptance, by reason (low-cardinality enum).",
}, []string{"reason"})
m.CheckStatus = prometheus.NewGaugeVec(prometheus.GaugeOpts{
Name: "monlet_agent_check_status",
Help: "Last observed status per check: 0=ok,1=warning,2=critical,3=unknown.",
@@ -123,7 +128,7 @@ func New() *Metrics {
m.CheckInterval, m.ChecksConfigured,
m.SpoolDepth, m.SpoolBytes,
m.SendAttempts, m.SendFailures,
m.EventsAccepted, m.EventsDedup,
m.EventsAccepted, m.EventsDedup, m.EventsDropped,
m.LastHeartbeat, m.BuildInfo)
return m
}

5
agent/internal/runner/runner.go
View File

@@ -62,8 +62,9 @@ func Run(ctx context.Context, argv []string, timeout time.Duration) Result {
}
if cctx.Err() == context.DeadlineExceeded {
r.Status = StatusUnknown
r.ExitCode = -1
r.Status = StatusCritical
r.ExitCode = 2
r.Output = fmt.Sprintf("critical: check timed out after %s", timeout)
return r
}
if err != nil {

5
agent/internal/runner/runner_test.go
View File

@@ -97,9 +97,12 @@ func TestRunUnknownExit(t *testing.T) {
func TestRunTimeout(t *testing.T) {
r := Run(context.Background(), []string{"sh", "-c", "sleep 5"}, 200*time.Millisecond)
if r.Status != StatusUnknown {
if r.Status != StatusCritical || r.ExitCode != 2 {
t.Fatalf("got %+v", r)
}
if !strings.Contains(r.Output, "critical: check timed out after 200ms") {
t.Fatalf("output: %q", r.Output)
}
if r.DurationMs > 2000 {
t.Fatalf("duration too long: %d", r.DurationMs)
}

35
agent/internal/spool/spool.go
View File

@@ -24,10 +24,19 @@ type Spool struct {
maxEvents int
maxBytes int64
mu sync.Mutex
nextSeq uint64
files []entry // sorted ascending by seq
bytes int64
mu sync.Mutex
nextSeq uint64
files []entry // sorted ascending by seq
bytes int64
dropEvents int
dropBytes int
}
// DropStats reports events dropped due to bounded-spool overflow since the
// previous call to Drops. Counters are reset on read.
type DropStats struct {
OverflowEvents int
OverflowBytes int
}
type entry struct {
@@ -135,13 +144,31 @@ func (s *Spool) dropOldestLocked() {
if len(s.files) == 0 {
return
}
byCount := len(s.files) > s.maxEvents
old := s.files[0]
_ = os.Remove(filepath.Join(s.dir, old.name))
s.bytes -= old.size
s.files = s.files[1:]
if byCount {
s.dropEvents++
} else {
s.dropBytes++
}
}
}
// Drops returns and resets the count of events dropped due to overflow since
// the previous call. Callers use this to feed bounded-cardinality metrics and
// throttled logs without spamming on every Enqueue.
func (s *Spool) Drops() DropStats {
s.mu.Lock()
defer s.mu.Unlock()
d := DropStats{OverflowEvents: s.dropEvents, OverflowBytes: s.dropBytes}
s.dropEvents = 0
s.dropBytes = 0
return d
}
// Peek returns up to limit oldest items. Does not remove them.
func (s *Spool) Peek(limit int) ([]Item, error) {
s.mu.Lock()

11
agent/internal/spool/spool_test.go
View File

@@ -55,6 +55,13 @@ func TestDropOldestByCount(t *testing.T) {
if items[0].Event.EventID != "e2" {
t.Fatalf("expected oldest dropped, got %q", items[0].Event.EventID)
}
d := s.Drops()
if d.OverflowEvents != 2 || d.OverflowBytes != 0 {
t.Fatalf("drop stats: %+v", d)
}
if again := s.Drops(); again.OverflowEvents != 0 || again.OverflowBytes != 0 {
t.Fatalf("Drops must reset, got %+v", again)
}
}
func TestDropOldestByBytes(t *testing.T) {
@@ -71,6 +78,10 @@ func TestDropOldestByBytes(t *testing.T) {
if s.Bytes() > 3000 {
t.Fatalf("bytes=%d > 3000", s.Bytes())
}
d := s.Drops()
if d.OverflowBytes == 0 {
t.Fatalf("expected bytes-overflow drops to be counted, got %+v", d)
}
}
func TestReopenAppliesLimits(t *testing.T) {