commit dcea096327eeebfbda42ff8b62355c6aa55cafd7 Author: Stanislav Rossovskii Date: Tue May 26 16:41:27 2026 +0400 init monlet repo with stage 0-2 (docs, contract, agent MVP) diff --git a/.github/workflows/README.md b/.github/workflows/README.md new file mode 100644 index 0000000..de63ba8 --- /dev/null +++ b/.github/workflows/README.md @@ -0,0 +1,16 @@ +# CI Workflows + +Placeholder for future GitHub Actions workflows. + +Planned jobs: + +- Go test for `agent/`; +- `gofmt -l` and `go vet` for `agent/`; +- `go test -race` for Go stage checkpoints or concurrency-heavy changes; +- Python tests and migrations for `server/`; +- PostgreSQL constraint tests for `server/`; +- Agent-to-Server integration smoke; +- TypeScript check, lint, and build for `web/`; +- OpenAPI validation for `api/openapi.yaml`. +- OpenAPI example payload validation for `examples/heartbeat.json` and `examples/event-batch.json`. +- Docker Compose smoke for `/health`, `/ready`, `/metrics`, and example ingest. diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..a278423 --- /dev/null +++ b/.gitignore @@ -0,0 +1,26 @@ +.cache/ + +# Python +**/.venv/ +**/__pycache__/ +*.py[cod] +.pytest_cache/ +.mypy_cache/ +.ruff_cache/ + +# Node +**/node_modules/ +**/.next/ +**/out/ +**/dist/ + +# Go +**/bin/ +**/coverage.out + +# Local env +*.local +.env +!.env.example +server/.env +!server/.env.example diff --git a/README.md b/README.md new file mode 100644 index 0000000..6549190 --- /dev/null +++ b/README.md @@ -0,0 +1,62 @@ +# Monlet + +Monlet is a lightweight event and check monitoring system. + +It is split into three independent applications: + +- `agent/` - a small Go binary installed on monitored hosts. +- `server/` - a Python/FastAPI backend that owns inventory, state, incidents, and notifications. +- `web/` - a read-only Next.js dashboard. + +The first project phase is documentation and planning bootstrap only. Core business logic is intentionally not implemented yet. + +## MVP + +Monlet v1 focuses on local checks, central state, and simple read-only visibility: + +- agents run local checks from TOML config; +- agents expose Prometheus metrics and/or push facts to the server; +- server ingests heartbeats and check events idempotently; +- server owns current state, incident lifecycle, and notifier retries; +- web UI reads server APIs and does not mutate state. + +## Non-goals for v1 + +- remote command execution; +- remote config push; +- full RBAC; +- multi-tenant enterprise model; +- complex silences or inhibition engine; +- Grafana replacement; +- long-term log storage; +- full incident management platform; +- Kubernetes operator; +- plugin marketplace; +- distributed server cluster. + +## Repository Map + +- `docs/` - architecture, ADRs, development, and operations docs. +- `plan/` - stage workflow and exit checkpoints. +- `api/` - versioned OpenAPI contract. +- `agent/` - Go agent placeholder and deployment notes. +- `server/` - Python backend placeholder. +- `web/` - Next.js frontend placeholder. +- `deploy/` - local and observability deployment examples. +- `examples/` - example checks and configs. + +## Start Here + +Read in this order: + +1. `docs/README.md` +2. `docs/architecture/overview.md` +3. `ROADMAP.md` +4. `TODO.md` +5. `plan/README.md` +6. `AGENTS.md` +7. `CLAUDE.md` + +## Current Status + +Stage 0 is the only active stage: documentation bootstrap. Do not start agent, server, or web core logic until Stage 0 exits and the API contract is reviewed. diff --git a/agent/README.md b/agent/README.md new file mode 100644 index 0000000..f5b8fa2 --- /dev/null +++ b/agent/README.md @@ -0,0 +1,59 @@ +# Monlet Agent + +Go agent (Stage 2 MVP). Reads `agent.toml`, runs configured commands on schedule, captures status/output, pushes contract-valid `heartbeat` and `events` batches to the Monlet server with Bearer auth, and survives outages via a local on-disk spool. + +## Build and run + +```sh +cd agent +GOTOOLCHAIN=local GOMODCACHE=$(pwd)/../.cache/go/pkg/mod GOCACHE=$(pwd)/../.cache/go/build GOBIN=$(pwd)/../.cache/go/bin go test ./... +GOTOOLCHAIN=local GOMODCACHE=$(pwd)/../.cache/go/pkg/mod GOCACHE=$(pwd)/../.cache/go/build GOBIN=$(pwd)/../.cache/go/bin go run ./cmd/monlet-agent -config config.example.toml +``` + +Do not use `go install` or `go env -w`. Module cache, build cache, and tool binaries stay under repo `.cache/`. + +## Configuration + +See `config.example.toml`. Required sections: top-level `state_dir`, `[server]` (`url`, `token`), at least one `[[checks]]` (`id`, `command`, `interval`, `timeout`). `agent_id` is optional — if empty, the agent generates and persists a UUID under `state_dir/agent_id`. + +| Key | Default | Notes | +|---|---|---| +| `mode` | `hybrid` | `prometheus_only` (only `/metrics`), `push_only` (only heartbeat+events), `hybrid` (both). `prometheus_only` does not require `server.url`/`server.token`. | +| `server.heartbeat_interval` | `30s` | ADR-0005 | +| `server.batch_interval` | `10s` | events flush cadence | +| `metrics.listen` | `127.0.0.1:9465` | low-cardinality only | +| `checks[].notification_owner` | `server` | `server`, `prometheus`, `none` | + +## Behaviour + +- Per-check scheduler with no-overlap: a tick is skipped if the previous run is still in flight (`monlet_agent_check_skipped_total`). +- Timeout via `exec.CommandContext` with `Setpgid` so the process group is killed on expiry; timeout → `unknown`. +- Exit code mapping: 0=ok, 1=warning, 2=critical, 3+=unknown (ADR-0005). +- `output` (stdout+stderr) is truncated by **UTF-8 byte length** to 8 KiB with marker `...[truncated N bytes]`. +- `event_id` is UUIDv7 generated on check completion and preserved across spool replay (ADR-0006 idempotency). +- Spool: per-event JSON files in `state_dir/spool/`, FIFO, limits 10 000 events / 50 MiB, drop oldest on overflow (ADR-0005). +- Retry backoff for heartbeat and events: 1s → 30s, exp ×2, jitter ±20%. 4xx are dropped (not retried) to avoid hot loop. +- Bearer auth header on both endpoints (ADR-0007). + +## Prometheus metrics + +Exposed on `metrics.listen` at `/metrics`. Labels are static (`check_id`, `status`, `kind`): + +- `monlet_agent_check_runs_total{check_id,status}` +- `monlet_agent_check_skipped_total{check_id}` +- `monlet_agent_check_duration_seconds{check_id}` (histogram) +- `monlet_agent_check_status{check_id}` (0=ok, 1=warning, 2=critical, 3=unknown) +- `monlet_agent_check_exit_code{check_id}` +- `monlet_agent_check_last_run_timestamp_seconds{check_id}` +- `monlet_agent_check_last_success_timestamp_seconds{check_id}` +- `monlet_agent_check_interval_seconds{check_id}` +- `monlet_agent_checks_configured` +- `monlet_agent_spool_events`, `monlet_agent_spool_bytes` +- `monlet_agent_send_attempts_total{kind}`, `monlet_agent_send_failures_total{kind}` (`kind` ∈ `heartbeat`, `events`) +- `monlet_agent_events_accepted_total`, `monlet_agent_events_deduplicated_total` +- `monlet_agent_last_heartbeat_timestamp_seconds` +- `monlet_agent_build_info{version,mode}` (constant 1) + +## systemd + +See `systemd/monlet-agent.service.example`. diff --git a/agent/cmd/monlet-agent/main.go b/agent/cmd/monlet-agent/main.go new file mode 100644 index 0000000..905ff20 --- /dev/null +++ b/agent/cmd/monlet-agent/main.go @@ -0,0 +1,65 @@ +package main + +import ( + "context" + "flag" + "log/slog" + "os" + "os/signal" + "path/filepath" + "syscall" + + "github.com/monlet/agent/internal/app" + "github.com/monlet/agent/internal/client" + "github.com/monlet/agent/internal/config" + "github.com/monlet/agent/internal/ids" + "github.com/monlet/agent/internal/metrics" + "github.com/monlet/agent/internal/spool" +) + +var version = "dev" + +func main() { + cfgPath := flag.String("config", "config.toml", "path to TOML config") + flag.Parse() + + log := slog.New(slog.NewJSONHandler(os.Stderr, &slog.HandlerOptions{Level: slog.LevelInfo})) + + cfg, err := config.Load(*cfgPath) + if err != nil { + log.Error("load config", "err", err) + os.Exit(1) + } + + agentID, err := ids.LoadOrCreateAgentID(cfg.StateDir, cfg.AgentID) + if err != nil { + log.Error("agent_id", "err", err) + os.Exit(1) + } + + sp, err := spool.Open(filepath.Join(cfg.StateDir, "spool"), spool.DefaultMaxEvents, spool.DefaultMaxBytes) + if err != nil { + log.Error("open spool", "err", err) + os.Exit(1) + } + + a := &app.App{ + Cfg: cfg, + AgentID: agentID, + Version: version, + Client: client.New(cfg.Server.URL, cfg.Server.Token, agentID, version), + Spool: sp, + Metrics: metrics.New(), + Log: log, + } + + ctx, cancel := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM) + defer cancel() + + log.Info("monlet-agent starting", "agent_id", agentID, "version", version, "checks", len(cfg.Checks)) + if err := a.Run(ctx); err != nil { + log.Error("run", "err", err) + os.Exit(1) + } + log.Info("monlet-agent stopped") +} diff --git a/agent/config.example.toml b/agent/config.example.toml new file mode 100644 index 0000000..a9ab9ff --- /dev/null +++ b/agent/config.example.toml @@ -0,0 +1,22 @@ +agent_id = "example-host-01" +hostname = "example-host-01" +mode = "hybrid" +state_dir = "/var/lib/monlet-agent" + +[server] +url = "http://127.0.0.1:8000" +token = "replace-me" +heartbeat_interval = "30s" +batch_interval = "10s" + +[metrics] +enabled = true +listen = "127.0.0.1:9465" + +[[checks]] +id = "disk_root" +name = "Root disk usage" +command = ["/usr/local/lib/monlet/check_disk_root.sh"] +interval = "60s" +timeout = "10s" +notification_owner = "server" diff --git a/agent/go.mod b/agent/go.mod new file mode 100644 index 0000000..442e49a --- /dev/null +++ b/agent/go.mod @@ -0,0 +1,22 @@ +module github.com/monlet/agent + +go 1.25.7 + +require ( + github.com/BurntSushi/toml v1.6.0 + github.com/google/uuid v1.6.0 + github.com/prometheus/client_golang v1.23.2 +) + +require ( + github.com/beorn7/perks v1.0.1 // indirect + github.com/cespare/xxhash/v2 v2.3.0 // indirect + github.com/kr/text v0.2.0 // indirect + github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect + github.com/prometheus/client_model v0.6.2 // indirect + github.com/prometheus/common v0.66.1 // indirect + github.com/prometheus/procfs v0.16.1 // indirect + go.yaml.in/yaml/v2 v2.4.2 // indirect + golang.org/x/sys v0.35.0 // indirect + google.golang.org/protobuf v1.36.8 // indirect +) diff --git a/agent/go.sum b/agent/go.sum new file mode 100644 index 0000000..6f54a97 --- /dev/null +++ b/agent/go.sum @@ -0,0 +1,50 @@ +github.com/BurntSushi/toml v1.6.0 h1:dRaEfpa2VI55EwlIW72hMRHdWouJeRF7TPYhI+AUQjk= +github.com/BurntSushi/toml v1.6.0/go.mod h1:ukJfTF/6rtPPRCnwkur4qwRxa8vTRFBF0uk2lLoLwho= +github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM= +github.com/beorn7/perks v1.0.1/go.mod h1:G2ZrVWU2WbWT9wwq4/hrbKbnv/1ERSJQ0ibhJ6rlkpw= +github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs= +github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs= +github.com/creack/pty v1.1.9/go.mod h1:oKZEueFk5CKHvIhNR5MUki03XCEU+Q6VDXinZuGJ33E= +github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c= +github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= +github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8= +github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU= +github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0= +github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= +github.com/klauspost/compress v1.18.0 h1:c/Cqfb0r+Yi+JtIEq73FWXVkRonBlf0CRNYc8Zttxdo= +github.com/klauspost/compress v1.18.0/go.mod h1:2Pp+KzxcywXVXMr50+X0Q/Lsb43OQHYWRCY2AiWywWQ= +github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE= +github.com/kr/pretty v0.3.1/go.mod h1:hoEshYVHaxMs3cyo3Yncou5ZscifuDolrwPKZanG3xk= +github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY= +github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE= +github.com/kylelemons/godebug v1.1.0 h1:RPNrshWIDI6G2gRW9EHilWtl7Z6Sb1BR0xunSBf0SNc= +github.com/kylelemons/godebug v1.1.0/go.mod h1:9/0rRGxNHcop5bhtWyNeEfOS8JIWk580+fNqagV/RAw= +github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 h1:C3w9PqII01/Oq1c1nUAm88MOHcQC9l5mIlSMApZMrHA= +github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822/go.mod h1:+n7T8mK8HuQTcFwEeznm/DIxMOiR9yIdICNftLE1DvQ= +github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM= +github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= +github.com/prometheus/client_golang v1.23.2 h1:Je96obch5RDVy3FDMndoUsjAhG5Edi49h0RJWRi/o0o= +github.com/prometheus/client_golang v1.23.2/go.mod h1:Tb1a6LWHB3/SPIzCoaDXI4I8UHKeFTEQ1YCr+0Gyqmg= +github.com/prometheus/client_model v0.6.2 h1:oBsgwpGs7iVziMvrGhE53c/GrLUsZdHnqNwqPLxwZyk= +github.com/prometheus/client_model v0.6.2/go.mod h1:y3m2F6Gdpfy6Ut/GBsUqTWZqCUvMVzSfMLjcu6wAwpE= +github.com/prometheus/common v0.66.1 h1:h5E0h5/Y8niHc5DlaLlWLArTQI7tMrsfQjHV+d9ZoGs= +github.com/prometheus/common v0.66.1/go.mod h1:gcaUsgf3KfRSwHY4dIMXLPV0K/Wg1oZ8+SbZk/HH/dA= +github.com/prometheus/procfs v0.16.1 h1:hZ15bTNuirocR6u0JZ6BAHHmwS1p8B4P6MRqxtzMyRg= +github.com/prometheus/procfs v0.16.1/go.mod h1:teAbpZRB1iIAJYREa1LsoWUXykVXA1KlTmWl8x/U+Is= +github.com/rogpeppe/go-internal v1.10.0 h1:TMyTOH3F/DB16zRVcYyreMH6GnZZrwQVAoYjRBZyWFQ= +github.com/rogpeppe/go-internal v1.10.0/go.mod h1:UQnix2H7Ngw/k4C5ijL5+65zddjncjaFoBhdsK/akog= +github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U= +github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U= +go.uber.org/goleak v1.3.0 h1:2K3zAYmnTNqV73imy9J1T3WC+gmCePx2hEGkimedGto= +go.uber.org/goleak v1.3.0/go.mod h1:CoHD4mav9JJNrW/WLlf7HGZPjdw8EucARQHekz1X6bE= +go.yaml.in/yaml/v2 v2.4.2 h1:DzmwEr2rDGHl7lsFgAHxmNz/1NlQ7xLIrlN2h5d1eGI= +go.yaml.in/yaml/v2 v2.4.2/go.mod h1:081UH+NErpNdqlCXm3TtEran0rJZGxAYx9hb/ELlsPU= +golang.org/x/sys v0.35.0 h1:vz1N37gP5bs89s7He8XuIYXpyY0+QlsKmzipCbUtyxI= +golang.org/x/sys v0.35.0/go.mod h1:BJP2sWEmIv4KK5OTEluFJCKSidICx8ciO85XgH3Ak8k= +google.golang.org/protobuf v1.36.8 h1:xHScyCOEuuwZEc6UtSOvPbAT4zRh0xcNRYekJwfqyMc= +google.golang.org/protobuf v1.36.8/go.mod h1:fuxRtAxBytpl4zzqUh6/eyUujkJdNiuEkXntxiD/uRU= +gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= +gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk= +gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c/go.mod h1:JHkPIbrfpd72SG/EVd6muEfDQjcINNoR0C8j2r3qZ4Q= +gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA= +gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= diff --git a/agent/internal/app/app.go b/agent/internal/app/app.go new file mode 100644 index 0000000..41cd180 --- /dev/null +++ b/agent/internal/app/app.go @@ -0,0 +1,212 @@ +package app + +import ( + "context" + "errors" + "log/slog" + "sync" + "time" + + "github.com/monlet/agent/internal/client" + "github.com/monlet/agent/internal/config" + "github.com/monlet/agent/internal/metrics" + "github.com/monlet/agent/internal/scheduler" + "github.com/monlet/agent/internal/spool" +) + +type App struct { + Cfg *config.Config + AgentID string + Version string + Client *client.Client + Spool *spool.Spool + Metrics *metrics.Metrics + Log *slog.Logger +} + +func (a *App) Run(ctx context.Context) error { + events := make(chan scheduler.Event, 256) + + a.Metrics.ChecksConfigured.Set(float64(len(a.Cfg.Checks))) + a.Metrics.BuildInfo.WithLabelValues(a.Version, a.Cfg.Mode).Set(1) + for _, c := range a.Cfg.Checks { + a.Metrics.CheckInterval.WithLabelValues(c.ID).Set(c.Interval.Duration.Seconds()) + } + + var wg sync.WaitGroup + + if a.Cfg.ExposesMetrics() { + wg.Add(1) + go func() { + defer wg.Done() + if err := a.Metrics.Serve(ctx, a.Cfg.Metrics.Listen); err != nil { + a.Log.Error("metrics server", "err", err) + } + }() + } + + wg.Add(1) + go func() { + defer wg.Done() + scheduler.Run(ctx, a.AgentID, a.Cfg.Checks, events, scheduler.Hooks{ + OnSkipped: func(id string) { a.Metrics.CheckSkipped.WithLabelValues(id).Inc() }, + OnResult: func(ev scheduler.Event) { + a.Metrics.CheckRuns.WithLabelValues(ev.CheckID, ev.Status).Inc() + a.Metrics.CheckDuration.WithLabelValues(ev.CheckID).Observe(float64(ev.DurationMs) / 1000.0) + a.Metrics.CheckStatus.WithLabelValues(ev.CheckID).Set(metrics.StatusCode(ev.Status)) + a.Metrics.CheckExitCode.WithLabelValues(ev.CheckID).Set(float64(ev.ExitCode)) + ts := float64(ev.ObservedAt.Unix()) + a.Metrics.CheckLastRun.WithLabelValues(ev.CheckID).Set(ts) + if ev.Status == "ok" { + a.Metrics.CheckLastSuccess.WithLabelValues(ev.CheckID).Set(ts) + } + }, + }) + }() + + if a.Cfg.PushesToServer() { + wg.Add(1) + go func() { + defer wg.Done() + a.spoolWriter(ctx, events) + }() + wg.Add(1) + go func() { + defer wg.Done() + a.heartbeatLoop(ctx) + }() + wg.Add(1) + go func() { + defer wg.Done() + a.sendLoop(ctx) + }() + } else { + // drain scheduler events to avoid blocking it + wg.Add(1) + go func() { + defer wg.Done() + for { + select { + case <-ctx.Done(): + return + case <-events: + } + } + }() + } + + wg.Wait() + return nil +} + +func (a *App) spoolWriter(ctx context.Context, events <-chan scheduler.Event) { + for { + select { + case <-ctx.Done(): + return + case ev := <-events: + if err := a.Spool.Enqueue(ev); err != nil { + a.Log.Error("spool enqueue", "err", err) + } + a.Metrics.SpoolDepth.Set(float64(a.Spool.Len())) + a.Metrics.SpoolBytes.Set(float64(a.Spool.Bytes())) + } + } +} + +func (a *App) heartbeatLoop(ctx context.Context) { + interval := a.Cfg.Server.HeartbeatInterval.Duration + attempt := 0 + for { + hb := client.HeartbeatRequest{ + AgentID: a.AgentID, + ObservedAt: time.Now().UTC(), + Hostname: a.Cfg.Hostname, + Version: a.Version, + Mode: a.Cfg.Mode, + } + a.Metrics.SendAttempts.WithLabelValues("heartbeat").Inc() + err := a.Client.SendHeartbeat(ctx, hb) + if err != nil { + a.Metrics.SendFailures.WithLabelValues("heartbeat").Inc() + if errors.Is(err, context.Canceled) { + return + } + a.Log.Warn("heartbeat send failed", "err", err) + if client.Retryable(err) { + attempt++ + if sleep(ctx, client.Delay(attempt)) { + return + } + continue + } + } else { + a.Metrics.LastHeartbeat.Set(float64(time.Now().Unix())) + } + attempt = 0 + if sleep(ctx, interval) { + return + } + } +} + +func (a *App) sendLoop(ctx context.Context) { + interval := a.Cfg.Server.BatchInterval.Duration + attempt := 0 + for { + items, err := a.Spool.Peek(client.MaxBatchEvents) + if err != nil { + a.Log.Error("spool peek", "err", err) + } + if len(items) == 0 { + if sleep(ctx, interval) { + return + } + continue + } + evs := make([]scheduler.Event, len(items)) + seqs := make([]uint64, len(items)) + for i, it := range items { + evs[i] = it.Event + seqs[i] = it.Seq + } + a.Metrics.SendAttempts.WithLabelValues("events").Inc() + resp, err := a.Client.SendEvents(ctx, evs) + if err != nil { + a.Metrics.SendFailures.WithLabelValues("events").Inc() + if errors.Is(err, context.Canceled) { + return + } + a.Log.Warn("events send failed", "err", err) + if client.Retryable(err) { + attempt++ + if sleep(ctx, client.Delay(attempt)) { + return + } + continue + } + // non-retryable: drop to avoid hot loop + a.Spool.Commit(seqs) + attempt = 0 + continue + } + attempt = 0 + a.Spool.Commit(seqs) + a.Metrics.EventsAccepted.Add(float64(resp.Accepted)) + a.Metrics.EventsDedup.Add(float64(resp.Deduplicated)) + a.Metrics.SpoolDepth.Set(float64(a.Spool.Len())) + a.Metrics.SpoolBytes.Set(float64(a.Spool.Bytes())) + } +} + +// sleep returns true if context was cancelled while waiting. +func sleep(ctx context.Context, d time.Duration) bool { + t := time.NewTimer(d) + defer t.Stop() + select { + case <-ctx.Done(): + return true + case <-t.C: + return false + } +} diff --git a/agent/internal/app/app_test.go b/agent/internal/app/app_test.go new file mode 100644 index 0000000..bd2ddaf --- /dev/null +++ b/agent/internal/app/app_test.go @@ -0,0 +1,171 @@ +package app + +import ( + "context" + "encoding/json" + "io" + "log/slog" + "net/http" + "net/http/httptest" + "path/filepath" + "sync" + "sync/atomic" + "testing" + "time" + + "github.com/monlet/agent/internal/client" + "github.com/monlet/agent/internal/config" + "github.com/monlet/agent/internal/metrics" + "github.com/monlet/agent/internal/spool" +) + +func TestPrometheusOnlyDoesNotPush(t *testing.T) { + var hits int32 + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + atomic.AddInt32(&hits, 1) + w.WriteHeader(http.StatusAccepted) + })) + defer srv.Close() + + dir := t.TempDir() + cfg := &config.Config{ + AgentID: "a1", + Mode: "prometheus_only", + StateDir: dir, + Server: config.ServerConfig{URL: srv.URL, Token: "t", HeartbeatInterval: config.Duration{Duration: 50 * time.Millisecond}, BatchInterval: config.Duration{Duration: 50 * time.Millisecond}}, + Metrics: config.MetricsConfig{Enabled: false}, + Checks: []config.CheckConfig{{ + ID: "c1", + Command: []string{"sh", "-c", "echo hi"}, + Interval: config.Duration{Duration: 100 * time.Millisecond}, + Timeout: config.Duration{Duration: 50 * time.Millisecond}, + NotificationOwner: "server", + }}, + } + sp, _ := spool.Open(filepath.Join(dir, "spool"), 100, 1<<20) + a := &App{ + Cfg: cfg, AgentID: "a1", Version: "t", + Client: client.New(srv.URL, "t", "a1", "t"), + Spool: sp, Metrics: metrics.New(), + Log: slog.New(slog.NewTextHandler(io.Discard, nil)), + } + ctx, cancel := context.WithTimeout(context.Background(), 400*time.Millisecond) + defer cancel() + _ = a.Run(ctx) + if atomic.LoadInt32(&hits) != 0 { + t.Fatalf("prometheus_only must not hit server, got %d requests", atomic.LoadInt32(&hits)) + } + if sp.Len() != 0 { + t.Fatalf("prometheus_only must not spool events, got %d", sp.Len()) + } +} + +func TestEndToEndPushAndReplay(t *testing.T) { + var ( + mu sync.Mutex + seenEventIDs = make(map[string]int) + hbCount int32 + fail atomic.Bool + ) + fail.Store(true) + + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + switch r.URL.Path { + case "/api/v1/heartbeat": + atomic.AddInt32(&hbCount, 1) + w.WriteHeader(http.StatusAccepted) + _, _ = io.WriteString(w, `{"accepted":true}`) + case "/api/v1/events": + if fail.Load() { + http.Error(w, "down", http.StatusServiceUnavailable) + return + } + var req client.EventBatchRequest + _ = json.NewDecoder(r.Body).Decode(&req) + mu.Lock() + for _, e := range req.Events { + seenEventIDs[e.EventID]++ + } + mu.Unlock() + w.WriteHeader(http.StatusAccepted) + _ = json.NewEncoder(w).Encode(client.EventBatchResponse{Accepted: len(req.Events)}) + } + })) + defer srv.Close() + + dir := t.TempDir() + cfg := &config.Config{ + AgentID: "a1", + Hostname: "h", + Mode: "hybrid", + StateDir: dir, + Server: config.ServerConfig{ + URL: srv.URL, + Token: "t", + HeartbeatInterval: config.Duration{Duration: 100 * time.Millisecond}, + BatchInterval: config.Duration{Duration: 100 * time.Millisecond}, + }, + Metrics: config.MetricsConfig{Enabled: false}, + Checks: []config.CheckConfig{{ + ID: "c1", + Command: []string{"sh", "-c", "echo hi"}, + Interval: config.Duration{Duration: 150 * time.Millisecond}, + Timeout: config.Duration{Duration: 100 * time.Millisecond}, + NotificationOwner: "server", + }}, + } + sp, err := spool.Open(filepath.Join(dir, "spool"), 100, 1<<20) + if err != nil { + t.Fatal(err) + } + a := &App{ + Cfg: cfg, + AgentID: "a1", + Version: "test", + Client: client.New(cfg.Server.URL, "t", "a1", "test"), + Spool: sp, + Metrics: metrics.New(), + Log: slog.New(slog.NewTextHandler(io.Discard, nil)), + } + + ctx, cancel := context.WithCancel(context.Background()) + done := make(chan struct{}) + go func() { _ = a.Run(ctx); close(done) }() + + // Phase 1: server failing, events accumulate in spool. + time.Sleep(400 * time.Millisecond) + if sp.Len() == 0 { + cancel() + <-done + t.Fatal("expected spooled events while server failing") + } + + // Phase 2: server recovers. + fail.Store(false) + deadline := time.Now().Add(3 * time.Second) + for time.Now().Before(deadline) { + mu.Lock() + got := len(seenEventIDs) + mu.Unlock() + if got > 0 && sp.Len() == 0 { + break + } + time.Sleep(50 * time.Millisecond) + } + cancel() + <-done + + mu.Lock() + defer mu.Unlock() + if len(seenEventIDs) == 0 { + t.Fatal("no events delivered after recovery") + } + for id, n := range seenEventIDs { + if n > 1 { + t.Fatalf("event_id %s sent %d times (should be deduped by server, but agent should commit on first 2xx)", id, n) + } + } + if atomic.LoadInt32(&hbCount) == 0 { + t.Fatal("no heartbeats observed") + } +} diff --git a/agent/internal/client/backoff.go b/agent/internal/client/backoff.go new file mode 100644 index 0000000..b8a9b81 --- /dev/null +++ b/agent/internal/client/backoff.go @@ -0,0 +1,29 @@ +package client + +import ( + "math/rand/v2" + "time" +) + +const ( + BackoffMin = 1 * time.Second + BackoffMax = 30 * time.Second +) + +// Delay computes exponential backoff with ±20% jitter. +// attempt starts at 1 for the first failed try. +func Delay(attempt int) time.Duration { + if attempt < 1 { + attempt = 1 + } + d := BackoffMin + for i := 1; i < attempt; i++ { + d *= 2 + if d >= BackoffMax { + d = BackoffMax + break + } + } + jitter := 1.0 + (rand.Float64()*0.4 - 0.2) + return time.Duration(float64(d) * jitter) +} diff --git a/agent/internal/client/client.go b/agent/internal/client/client.go new file mode 100644 index 0000000..7990967 --- /dev/null +++ b/agent/internal/client/client.go @@ -0,0 +1,135 @@ +package client + +import ( + "bytes" + "context" + "encoding/json" + "fmt" + "io" + "net/http" + "strings" + "time" + + "github.com/monlet/agent/internal/scheduler" +) + +const ( + MaxBatchEvents = 200 +) + +type Client struct { + baseURL string + token string + agentID string + version string + http *http.Client +} + +func New(baseURL, token, agentID, version string) *Client { + return &Client{ + baseURL: strings.TrimRight(baseURL, "/"), + token: token, + agentID: agentID, + version: version, + http: &http.Client{Timeout: 15 * time.Second}, + } +} + +type HeartbeatRequest struct { + AgentID string `json:"agent_id"` + ObservedAt time.Time `json:"observed_at"` + Hostname string `json:"hostname"` + Version string `json:"version"` + Mode string `json:"mode,omitempty"` + Labels map[string]string `json:"labels,omitempty"` +} + +type EventBatchRequest struct { + AgentID string `json:"agent_id"` + Events []scheduler.Event `json:"events"` +} + +type EventBatchResponse struct { + Accepted int `json:"accepted"` + Deduplicated int `json:"deduplicated"` +} + +func (c *Client) SendHeartbeat(ctx context.Context, hb HeartbeatRequest) error { + return c.post(ctx, "/api/v1/heartbeat", hb, nil) +} + +func (c *Client) SendEvents(ctx context.Context, events []scheduler.Event) (*EventBatchResponse, error) { + if len(events) == 0 { + return &EventBatchResponse{}, nil + } + if len(events) > MaxBatchEvents { + return nil, fmt.Errorf("batch exceeds %d events", MaxBatchEvents) + } + req := EventBatchRequest{AgentID: c.agentID, Events: events} + var resp EventBatchResponse + if err := c.post(ctx, "/api/v1/events", req, &resp); err != nil { + return nil, err + } + return &resp, nil +} + +func (c *Client) post(ctx context.Context, path string, body any, out any) error { + b, err := json.Marshal(body) + if err != nil { + return err + } + req, err := http.NewRequestWithContext(ctx, http.MethodPost, c.baseURL+path, bytes.NewReader(b)) + if err != nil { + return err + } + req.Header.Set("Authorization", "Bearer "+c.token) + req.Header.Set("Content-Type", "application/json") + resp, err := c.http.Do(req) + if err != nil { + return err + } + defer resp.Body.Close() + if resp.StatusCode/100 != 2 { + msg, _ := io.ReadAll(io.LimitReader(resp.Body, 1024)) + return &HTTPError{Status: resp.StatusCode, Body: string(msg)} + } + if out != nil { + return json.NewDecoder(resp.Body).Decode(out) + } + return nil +} + +type HTTPError struct { + Status int + Body string +} + +func (e *HTTPError) Error() string { return fmt.Sprintf("http %d: %s", e.Status, e.Body) } + +// Retryable reports whether the error should trigger a retry (network errors +// and 5xx are retryable; 4xx are dropped to avoid hot-loop on bad payload). +func Retryable(err error) bool { + if err == nil { + return false + } + var he *HTTPError + if asErr(err, &he) { + return he.Status >= 500 + } + return true +} + +func asErr(err error, target **HTTPError) bool { + for e := err; e != nil; { + if h, ok := e.(*HTTPError); ok { + *target = h + return true + } + u, ok := e.(interface{ Unwrap() error }) + if !ok { + return false + } + e = u.Unwrap() + } + return false +} diff --git a/agent/internal/client/client_test.go b/agent/internal/client/client_test.go new file mode 100644 index 0000000..d3d680b --- /dev/null +++ b/agent/internal/client/client_test.go @@ -0,0 +1,86 @@ +package client + +import ( + "context" + "encoding/json" + "io" + "net/http" + "net/http/httptest" + "testing" + "time" + + "github.com/monlet/agent/internal/scheduler" +) + +func TestSendHeartbeatSuccess(t *testing.T) { + var got HeartbeatRequest + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.URL.Path != "/api/v1/heartbeat" || r.Header.Get("Authorization") != "Bearer tok" { + http.Error(w, "bad", http.StatusUnauthorized) + return + } + _ = json.NewDecoder(r.Body).Decode(&got) + w.WriteHeader(http.StatusAccepted) + _, _ = io.WriteString(w, `{"accepted":true}`) + })) + defer srv.Close() + c := New(srv.URL, "tok", "a1", "v0") + err := c.SendHeartbeat(context.Background(), HeartbeatRequest{AgentID: "a1", ObservedAt: time.Now().UTC(), Hostname: "h", Version: "v0"}) + if err != nil { + t.Fatal(err) + } + if got.AgentID != "a1" { + t.Fatalf("got %+v", got) + } +} + +func TestSendEventsBatch(t *testing.T) { + var got EventBatchRequest + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + _ = json.NewDecoder(r.Body).Decode(&got) + w.WriteHeader(http.StatusAccepted) + _, _ = io.WriteString(w, `{"accepted":2,"deduplicated":0}`) + })) + defer srv.Close() + c := New(srv.URL, "tok", "a1", "v0") + resp, err := c.SendEvents(context.Background(), []scheduler.Event{{EventID: "x"}, {EventID: "y"}}) + if err != nil { + t.Fatal(err) + } + if resp.Accepted != 2 || got.AgentID != "a1" || len(got.Events) != 2 { + t.Fatalf("resp=%+v got=%+v", resp, got) + } +} + +func TestSendEventsTooBig(t *testing.T) { + c := New("http://x", "t", "a1", "v0") + big := make([]scheduler.Event, MaxBatchEvents+1) + _, err := c.SendEvents(context.Background(), big) + if err == nil { + t.Fatal("expected error") + } +} + +func TestRetryableClassification(t *testing.T) { + if Retryable(&HTTPError{Status: 400, Body: ""}) { + t.Fatal("4xx should not retry") + } + if !Retryable(&HTTPError{Status: 503}) { + t.Fatal("5xx should retry") + } + if !Retryable(context.DeadlineExceeded) { + t.Fatal("network error should retry") + } +} + +func TestBackoffBounds(t *testing.T) { + for a := 1; a <= 10; a++ { + d := Delay(a) + if d < BackoffMin*4/5 { + t.Fatalf("attempt %d: delay %s below min", a, d) + } + if d > BackoffMax*6/5 { + t.Fatalf("attempt %d: delay %s above max", a, d) + } + } +} diff --git a/agent/internal/config/config.go b/agent/internal/config/config.go new file mode 100644 index 0000000..c8fce74 --- /dev/null +++ b/agent/internal/config/config.go @@ -0,0 +1,180 @@ +package config + +import ( + "fmt" + "os" + "regexp" + "time" + + "github.com/BurntSushi/toml" +) + +var idPattern = regexp.MustCompile(`^[A-Za-z0-9._:-]+$`) + +const ( + maxIDLen = 128 + defaultHeartbeat = 30 * time.Second + defaultBatch = 10 * time.Second + defaultMetricsAddr = "127.0.0.1:9465" +) + +type Config struct { + AgentID string `toml:"agent_id"` + Hostname string `toml:"hostname"` + Mode string `toml:"mode"` + StateDir string `toml:"state_dir"` + Server ServerConfig `toml:"server"` + Metrics MetricsConfig `toml:"metrics"` + Checks []CheckConfig `toml:"checks"` +} + +type ServerConfig struct { + URL string `toml:"url"` + Token string `toml:"token"` + HeartbeatInterval Duration `toml:"heartbeat_interval"` + BatchInterval Duration `toml:"batch_interval"` +} + +type MetricsConfig struct { + Enabled bool `toml:"enabled"` + Listen string `toml:"listen"` +} + +type CheckConfig struct { + ID string `toml:"id"` + Name string `toml:"name"` + Command []string `toml:"command"` + Interval Duration `toml:"interval"` + Timeout Duration `toml:"timeout"` + NotificationOwner string `toml:"notification_owner"` + DedupeKey string `toml:"dedupe_key"` +} + +type Duration struct{ time.Duration } + +func (d *Duration) UnmarshalText(b []byte) error { + v, err := time.ParseDuration(string(b)) + if err != nil { + return err + } + d.Duration = v + return nil +} + +func Load(path string) (*Config, error) { + var c Config + if _, err := toml.DecodeFile(path, &c); err != nil { + return nil, fmt.Errorf("read config: %w", err) + } + c.applyDefaults() + if err := c.Validate(); err != nil { + return nil, err + } + return &c, nil +} + +func (c *Config) applyDefaults() { + if c.Mode == "" { + c.Mode = "hybrid" + } + if c.Hostname == "" { + if h, err := os.Hostname(); err == nil { + c.Hostname = h + } + } + if c.Server.HeartbeatInterval.Duration == 0 { + c.Server.HeartbeatInterval.Duration = defaultHeartbeat + } + if c.Server.BatchInterval.Duration == 0 { + c.Server.BatchInterval.Duration = defaultBatch + } + if c.Metrics.Listen == "" { + c.Metrics.Listen = defaultMetricsAddr + } +} + +func (c *Config) Validate() error { + if c.AgentID != "" { + if err := ValidateID("agent_id", c.AgentID); err != nil { + return err + } + } + switch c.Mode { + case "prometheus_only", "push_only", "hybrid": + default: + return fmt.Errorf("invalid mode %q", c.Mode) + } + if c.StateDir == "" { + return fmt.Errorf("state_dir is required") + } + if c.PushesToServer() { + if c.Server.URL == "" { + return fmt.Errorf("server.url is required for mode %q", c.Mode) + } + if c.Server.Token == "" { + return fmt.Errorf("server.token is required for mode %q", c.Mode) + } + } + if len(c.Checks) == 0 { + return fmt.Errorf("at least one check is required") + } + seen := make(map[string]struct{}, len(c.Checks)) + for i := range c.Checks { + ch := &c.Checks[i] + if err := ValidateID("checks[].id", ch.ID); err != nil { + return err + } + if _, dup := seen[ch.ID]; dup { + return fmt.Errorf("duplicate check id %q", ch.ID) + } + seen[ch.ID] = struct{}{} + if len(ch.Command) == 0 { + return fmt.Errorf("check %q: command is required", ch.ID) + } + if ch.Interval.Duration <= 0 { + return fmt.Errorf("check %q: interval must be > 0", ch.ID) + } + if ch.Timeout.Duration <= 0 { + return fmt.Errorf("check %q: timeout must be > 0", ch.ID) + } + if ch.Timeout.Duration > ch.Interval.Duration { + return fmt.Errorf("check %q: timeout must be <= interval", ch.ID) + } + switch ch.NotificationOwner { + case "", "server", "prometheus", "none": + default: + return fmt.Errorf("check %q: invalid notification_owner %q", ch.ID, ch.NotificationOwner) + } + if ch.NotificationOwner == "" { + ch.NotificationOwner = "server" + } + if len(ch.DedupeKey) > 256 { + return fmt.Errorf("check %q: dedupe_key too long", ch.ID) + } + } + return nil +} + +// PushesToServer is true when the mode requires heartbeat/events push. +func (c *Config) PushesToServer() bool { + return c.Mode == "push_only" || c.Mode == "hybrid" +} + +// ExposesMetrics is true when the agent should serve /metrics. +// `metrics.enabled` is an independent gate; both must agree. +func (c *Config) ExposesMetrics() bool { + return c.Metrics.Enabled && (c.Mode == "prometheus_only" || c.Mode == "hybrid") +} + +func ValidateID(field, v string) error { + if v == "" { + return fmt.Errorf("%s is empty", field) + } + if len(v) > maxIDLen { + return fmt.Errorf("%s exceeds %d chars", field, maxIDLen) + } + if !idPattern.MatchString(v) { + return fmt.Errorf("%s has invalid characters", field) + } + return nil +} diff --git a/agent/internal/config/config_test.go b/agent/internal/config/config_test.go new file mode 100644 index 0000000..99e7c33 --- /dev/null +++ b/agent/internal/config/config_test.go @@ -0,0 +1,159 @@ +package config + +import ( + "os" + "path/filepath" + "testing" +) + +func writeTmp(t *testing.T, body string) string { + t.Helper() + p := filepath.Join(t.TempDir(), "c.toml") + if err := os.WriteFile(p, []byte(body), 0o600); err != nil { + t.Fatal(err) + } + return p +} + +const minimal = ` +agent_id = "host-1" +state_dir = "/tmp/x" + +[server] +url = "http://localhost" +token = "t" + +[[checks]] +id = "c1" +command = ["true"] +interval = "10s" +timeout = "5s" +` + +func TestLoadMinimal(t *testing.T) { + c, err := Load(writeTmp(t, minimal)) + if err != nil { + t.Fatal(err) + } + if c.Mode != "hybrid" { + t.Errorf("default mode: %q", c.Mode) + } + if c.Server.HeartbeatInterval.Duration == 0 { + t.Error("default heartbeat not applied") + } + if c.Checks[0].NotificationOwner != "server" { + t.Errorf("default notification_owner: %q", c.Checks[0].NotificationOwner) + } +} + +func TestValidateBadID(t *testing.T) { + body := minimal + "\n" + _, err := Load(writeTmp(t, body[:0]+` +agent_id = "bad id!" +state_dir = "/tmp/x" +[server] +url = "http://localhost" +token = "t" +[[checks]] +id = "c1" +command = ["true"] +interval = "10s" +timeout = "5s" +`)) + if err == nil { + t.Fatal("expected error for bad agent_id") + } +} + +func TestValidateTimeoutExceedsInterval(t *testing.T) { + _, err := Load(writeTmp(t, ` +agent_id = "x" +state_dir = "/tmp/x" +[server] +url = "http://localhost" +token = "t" +[[checks]] +id = "c1" +command = ["true"] +interval = "5s" +timeout = "10s" +`)) + if err == nil { + t.Fatal("expected error for timeout > interval") + } +} + +func TestPrometheusOnlyAllowsMissingServer(t *testing.T) { + c, err := Load(writeTmp(t, ` +mode = "prometheus_only" +state_dir = "/tmp/x" +[[checks]] +id = "c1" +command = ["true"] +interval = "10s" +timeout = "5s" +`)) + if err != nil { + t.Fatal(err) + } + if c.PushesToServer() { + t.Fatal("prometheus_only must not push") + } +} + +func TestPushOnlyRequiresServer(t *testing.T) { + _, err := Load(writeTmp(t, ` +mode = "push_only" +state_dir = "/tmp/x" +[[checks]] +id = "c1" +command = ["true"] +interval = "10s" +timeout = "5s" +`)) + if err == nil { + t.Fatal("push_only must require server.url/token") + } +} + +func TestExposesMetricsGate(t *testing.T) { + cases := []struct { + mode string + enabled bool + want bool + }{ + {"hybrid", true, true}, + {"hybrid", false, false}, + {"prometheus_only", true, true}, + {"push_only", true, false}, + } + for _, tc := range cases { + c := &Config{Mode: tc.mode, Metrics: MetricsConfig{Enabled: tc.enabled}} + if got := c.ExposesMetrics(); got != tc.want { + t.Errorf("mode=%s enabled=%v want=%v got=%v", tc.mode, tc.enabled, tc.want, got) + } + } +} + +func TestValidateDuplicateCheckID(t *testing.T) { + _, err := Load(writeTmp(t, ` +agent_id = "x" +state_dir = "/tmp/x" +[server] +url = "http://localhost" +token = "t" +[[checks]] +id = "c1" +command = ["true"] +interval = "10s" +timeout = "5s" +[[checks]] +id = "c1" +command = ["true"] +interval = "10s" +timeout = "5s" +`)) + if err == nil { + t.Fatal("expected duplicate id error") + } +} diff --git a/agent/internal/eventid/eventid.go b/agent/internal/eventid/eventid.go new file mode 100644 index 0000000..c0d631f --- /dev/null +++ b/agent/internal/eventid/eventid.go @@ -0,0 +1,12 @@ +package eventid + +import "github.com/google/uuid" + +// New returns a UUIDv7 in canonical lower-case form. +func New() (string, error) { + id, err := uuid.NewV7() + if err != nil { + return "", err + } + return id.String(), nil +} diff --git a/agent/internal/eventid/eventid_test.go b/agent/internal/eventid/eventid_test.go new file mode 100644 index 0000000..e1f4a5a --- /dev/null +++ b/agent/internal/eventid/eventid_test.go @@ -0,0 +1,20 @@ +package eventid + +import ( + "regexp" + "testing" +) + +var pattern = regexp.MustCompile(`^[0-9a-f]{8}-[0-9a-f]{4}-7[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$`) + +func TestNewMatchesContract(t *testing.T) { + for i := 0; i < 100; i++ { + id, err := New() + if err != nil { + t.Fatal(err) + } + if !pattern.MatchString(id) { + t.Fatalf("invalid UUIDv7: %q", id) + } + } +} diff --git a/agent/internal/ids/ids.go b/agent/internal/ids/ids.go new file mode 100644 index 0000000..144deb1 --- /dev/null +++ b/agent/internal/ids/ids.go @@ -0,0 +1,40 @@ +package ids + +import ( + "fmt" + "os" + "path/filepath" + "strings" + + "github.com/google/uuid" + + "github.com/monlet/agent/internal/config" +) + +// LoadOrCreateAgentID returns configured ID if non-empty, otherwise loads +// state_dir/agent_id, generating and persisting it on first run. +func LoadOrCreateAgentID(stateDir, configured string) (string, error) { + if configured != "" { + return configured, nil + } + if err := os.MkdirAll(stateDir, 0o755); err != nil { + return "", fmt.Errorf("create state dir: %w", err) + } + path := filepath.Join(stateDir, "agent_id") + b, err := os.ReadFile(path) + if err == nil { + id := strings.TrimSpace(string(b)) + if err := config.ValidateID("agent_id", id); err != nil { + return "", fmt.Errorf("persisted agent_id invalid: %w", err) + } + return id, nil + } + if !os.IsNotExist(err) { + return "", err + } + id := uuid.NewString() + if err := os.WriteFile(path, []byte(id+"\n"), 0o600); err != nil { + return "", err + } + return id, nil +} diff --git a/agent/internal/ids/ids_test.go b/agent/internal/ids/ids_test.go new file mode 100644 index 0000000..7290d7a --- /dev/null +++ b/agent/internal/ids/ids_test.go @@ -0,0 +1,33 @@ +package ids + +import ( + "os" + "path/filepath" + "testing" +) + +func TestConfiguredWins(t *testing.T) { + id, err := LoadOrCreateAgentID(t.TempDir(), "fixed") + if err != nil || id != "fixed" { + t.Fatalf("got %q %v", id, err) + } +} + +func TestGenerateAndReuse(t *testing.T) { + dir := t.TempDir() + id1, err := LoadOrCreateAgentID(dir, "") + if err != nil { + t.Fatal(err) + } + id2, err := LoadOrCreateAgentID(dir, "") + if err != nil { + t.Fatal(err) + } + if id1 != id2 { + t.Fatalf("not stable: %q vs %q", id1, id2) + } + b, _ := os.ReadFile(filepath.Join(dir, "agent_id")) + if len(b) == 0 { + t.Fatal("agent_id not persisted") + } +} diff --git a/agent/internal/metrics/metrics.go b/agent/internal/metrics/metrics.go new file mode 100644 index 0000000..78efd1d --- /dev/null +++ b/agent/internal/metrics/metrics.go @@ -0,0 +1,152 @@ +package metrics + +import ( + "context" + "errors" + "net/http" + "time" + + "github.com/prometheus/client_golang/prometheus" + "github.com/prometheus/client_golang/prometheus/promhttp" +) + +type Metrics struct { + Registry *prometheus.Registry + + CheckRuns *prometheus.CounterVec + CheckSkipped *prometheus.CounterVec + CheckDuration *prometheus.HistogramVec + CheckStatus *prometheus.GaugeVec + CheckExitCode *prometheus.GaugeVec + CheckLastRun *prometheus.GaugeVec + CheckLastSuccess *prometheus.GaugeVec + CheckInterval *prometheus.GaugeVec + ChecksConfigured prometheus.Gauge + SpoolDepth prometheus.Gauge + SpoolBytes prometheus.Gauge + SendAttempts *prometheus.CounterVec + SendFailures *prometheus.CounterVec + EventsAccepted prometheus.Counter + EventsDedup prometheus.Counter + LastHeartbeat prometheus.Gauge + BuildInfo *prometheus.GaugeVec +} + +// StatusCode maps a CheckResultEvent status string to a numeric gauge value. +func StatusCode(status string) float64 { + switch status { + case "ok": + return 0 + case "warning": + return 1 + case "critical": + return 2 + default: + return 3 + } +} + +func New() *Metrics { + reg := prometheus.NewRegistry() + m := &Metrics{Registry: reg} + m.CheckRuns = prometheus.NewCounterVec(prometheus.CounterOpts{ + Name: "monlet_agent_check_runs_total", + Help: "Number of check runs by id and status.", + }, []string{"check_id", "status"}) + m.CheckSkipped = prometheus.NewCounterVec(prometheus.CounterOpts{ + Name: "monlet_agent_check_skipped_total", + Help: "Number of ticks skipped due to overlap.", + }, []string{"check_id"}) + m.CheckDuration = prometheus.NewHistogramVec(prometheus.HistogramOpts{ + Name: "monlet_agent_check_duration_seconds", + Help: "Check execution duration.", + Buckets: prometheus.DefBuckets, + }, []string{"check_id"}) + m.SpoolDepth = prometheus.NewGauge(prometheus.GaugeOpts{ + Name: "monlet_agent_spool_events", + Help: "Current spool depth in events.", + }) + m.SpoolBytes = prometheus.NewGauge(prometheus.GaugeOpts{ + Name: "monlet_agent_spool_bytes", + Help: "Current spool size in bytes.", + }) + m.SendAttempts = prometheus.NewCounterVec(prometheus.CounterOpts{ + Name: "monlet_agent_send_attempts_total", + Help: "Outbound send attempts by kind.", + }, []string{"kind"}) + m.SendFailures = prometheus.NewCounterVec(prometheus.CounterOpts{ + Name: "monlet_agent_send_failures_total", + Help: "Outbound send failures by kind.", + }, []string{"kind"}) + m.EventsAccepted = prometheus.NewCounter(prometheus.CounterOpts{ + Name: "monlet_agent_events_accepted_total", + Help: "Events accepted by the server.", + }) + m.EventsDedup = prometheus.NewCounter(prometheus.CounterOpts{ + Name: "monlet_agent_events_deduplicated_total", + Help: "Events the server reported as duplicates.", + }) + m.CheckStatus = prometheus.NewGaugeVec(prometheus.GaugeOpts{ + Name: "monlet_agent_check_status", + Help: "Last observed status per check: 0=ok,1=warning,2=critical,3=unknown.", + }, []string{"check_id"}) + m.CheckExitCode = prometheus.NewGaugeVec(prometheus.GaugeOpts{ + Name: "monlet_agent_check_exit_code", + Help: "Last exit code per check.", + }, []string{"check_id"}) + m.CheckLastRun = prometheus.NewGaugeVec(prometheus.GaugeOpts{ + Name: "monlet_agent_check_last_run_timestamp_seconds", + Help: "Unix timestamp of the last completed run per check.", + }, []string{"check_id"}) + m.CheckLastSuccess = prometheus.NewGaugeVec(prometheus.GaugeOpts{ + Name: "monlet_agent_check_last_success_timestamp_seconds", + Help: "Unix timestamp of the last ok run per check.", + }, []string{"check_id"}) + m.CheckInterval = prometheus.NewGaugeVec(prometheus.GaugeOpts{ + Name: "monlet_agent_check_interval_seconds", + Help: "Configured interval per check.", + }, []string{"check_id"}) + m.ChecksConfigured = prometheus.NewGauge(prometheus.GaugeOpts{ + Name: "monlet_agent_checks_configured", + Help: "Number of checks configured.", + }) + m.LastHeartbeat = prometheus.NewGauge(prometheus.GaugeOpts{ + Name: "monlet_agent_last_heartbeat_timestamp_seconds", + Help: "Unix timestamp of the last successful heartbeat ack.", + }) + m.BuildInfo = prometheus.NewGaugeVec(prometheus.GaugeOpts{ + Name: "monlet_agent_build_info", + Help: "Constant 1 with build labels.", + }, []string{"version", "mode"}) + reg.MustRegister(m.CheckRuns, m.CheckSkipped, m.CheckDuration, + m.CheckStatus, m.CheckExitCode, m.CheckLastRun, m.CheckLastSuccess, + m.CheckInterval, m.ChecksConfigured, + m.SpoolDepth, m.SpoolBytes, + m.SendAttempts, m.SendFailures, + m.EventsAccepted, m.EventsDedup, + m.LastHeartbeat, m.BuildInfo) + return m +} + +// Serve blocks until ctx is done; the listener is closed on shutdown. +func (m *Metrics) Serve(ctx context.Context, addr string) error { + mux := http.NewServeMux() + mux.Handle("/metrics", promhttp.HandlerFor(m.Registry, promhttp.HandlerOpts{})) + srv := &http.Server{Addr: addr, Handler: mux, ReadHeaderTimeout: 5 * time.Second} + + errCh := make(chan error, 1) + go func() { errCh <- srv.ListenAndServe() }() + + select { + case <-ctx.Done(): + shutCtx, cancel := context.WithTimeout(context.Background(), 3*time.Second) + defer cancel() + _ = srv.Shutdown(shutCtx) + return nil + case err := <-errCh: + if errors.Is(err, http.ErrServerClosed) { + return nil + } + return err + } +} diff --git a/agent/internal/metrics/metrics_test.go b/agent/internal/metrics/metrics_test.go new file mode 100644 index 0000000..0bed268 --- /dev/null +++ b/agent/internal/metrics/metrics_test.go @@ -0,0 +1,34 @@ +package metrics + +import ( + "io" + "net/http" + "net/http/httptest" + "strings" + "testing" + + "github.com/prometheus/client_golang/prometheus/promhttp" +) + +func TestMetricsExposeRegistered(t *testing.T) { + m := New() + m.CheckRuns.WithLabelValues("c1", "ok").Inc() + m.SpoolDepth.Set(7) + srv := httptest.NewServer(promhttp.HandlerFor(m.Registry, promhttp.HandlerOpts{})) + defer srv.Close() + resp, err := http.Get(srv.URL) + if err != nil { + t.Fatal(err) + } + defer resp.Body.Close() + body, _ := io.ReadAll(resp.Body) + s := string(body) + for _, want := range []string{ + `monlet_agent_check_runs_total{check_id="c1",status="ok"} 1`, + `monlet_agent_spool_events 7`, + } { + if !strings.Contains(s, want) { + t.Errorf("missing %q in output", want) + } + } +} diff --git a/agent/internal/runner/runner.go b/agent/internal/runner/runner.go new file mode 100644 index 0000000..dafaf26 --- /dev/null +++ b/agent/internal/runner/runner.go @@ -0,0 +1,154 @@ +package runner + +import ( + "bytes" + "context" + "errors" + "fmt" + "os/exec" + "syscall" + "time" + "unicode/utf8" +) + +var replacementRune = []byte("�") + +const ( + MaxOutputBytes = 8192 + truncMarkerFmt = "...[truncated %d bytes]" +) + +type Status string + +const ( + StatusOK Status = "ok" + StatusWarning Status = "warning" + StatusCritical Status = "critical" + StatusUnknown Status = "unknown" +) + +type Result struct { + Status Status + ExitCode int + DurationMs int64 + Output string + OutputTruncated bool +} + +// Run executes argv with timeout. Stdout+stderr are captured into a single +// buffer; both are truncated to 8 KiB UTF-8 bytes if exceeded. +func Run(ctx context.Context, argv []string, timeout time.Duration) Result { + start := time.Now() + cctx, cancel := context.WithTimeout(ctx, timeout) + defer cancel() + + cmd := exec.CommandContext(cctx, argv[0], argv[1:]...) + cmd.SysProcAttr = sysProcAttr() + cmd.Cancel = func() error { return killGroup(cmd) } + cmd.WaitDelay = 2 * time.Second + + var buf bytes.Buffer + cmd.Stdout = &buf + cmd.Stderr = &buf + + err := cmd.Run() + dur := time.Since(start).Milliseconds() + + out, truncated := TruncateUTF8(buf.Bytes(), MaxOutputBytes) + r := Result{ + DurationMs: dur, + Output: out, + OutputTruncated: truncated, + } + + if cctx.Err() == context.DeadlineExceeded { + r.Status = StatusUnknown + r.ExitCode = -1 + return r + } + if err != nil { + var ee *exec.ExitError + if errors.As(err, &ee) { + r.ExitCode = ee.ExitCode() + } else { + r.Status = StatusUnknown + r.ExitCode = -1 + return r + } + } + r.Status = MapExit(r.ExitCode) + return r +} + +func MapExit(code int) Status { + switch code { + case 0: + return StatusOK + case 1: + return StatusWarning + case 2: + return StatusCritical + default: + return StatusUnknown + } +} + +// TruncateUTF8 returns a string whose UTF-8 byte length is <= max and a +// truncation flag. If truncated, appends "...[truncated N bytes]" marker. +func TruncateUTF8(b []byte, max int) (string, bool) { + // Sanitize invalid bytes first; otherwise json.Marshal replaces them with + // U+FFFD (3 bytes each) after our size check and the wire payload can + // exceed the contract limit. + b = bytes.ToValidUTF8(b, replacementRune) + if len(b) <= max { + return string(b), false + } + budget := max - 32 + if budget < 0 { + budget = 0 + } + for i := 0; i < 5; i++ { + cut := safeBoundary(b, budget) + marker := fmt.Sprintf(truncMarkerFmt, len(b)-cut) + if cut+len(marker) <= max { + return string(b[:cut]) + marker, true + } + budget = max - len(marker) - 1 + if budget < 0 { + budget = 0 + } + } + cut := safeBoundary(b, max/2) + marker := fmt.Sprintf(truncMarkerFmt, len(b)-cut) + return string(b[:cut]) + marker, true +} + +func safeBoundary(b []byte, limit int) int { + if limit >= len(b) { + return len(b) + } + for i := limit; i > 0; i-- { + if utf8.RuneStart(b[i]) { + if utf8.Valid(b[:i]) { + return i + } + } + } + return 0 +} + +func sysProcAttr() *syscall.SysProcAttr { + return &syscall.SysProcAttr{Setpgid: true} +} + +func killGroup(cmd *exec.Cmd) error { + if cmd.Process == nil { + return nil + } + pgid, err := syscall.Getpgid(cmd.Process.Pid) + if err == nil { + _ = syscall.Kill(-pgid, syscall.SIGKILL) + return nil + } + return cmd.Process.Kill() +} diff --git a/agent/internal/runner/runner_test.go b/agent/internal/runner/runner_test.go new file mode 100644 index 0000000..582f6b8 --- /dev/null +++ b/agent/internal/runner/runner_test.go @@ -0,0 +1,113 @@ +package runner + +import ( + "context" + "strings" + "testing" + "time" + "unicode/utf8" +) + +func TestMapExit(t *testing.T) { + cases := map[int]Status{0: StatusOK, 1: StatusWarning, 2: StatusCritical, 3: StatusUnknown, 255: StatusUnknown} + for code, want := range cases { + if got := MapExit(code); got != want { + t.Errorf("MapExit(%d)=%s want %s", code, got, want) + } + } +} + +func TestTruncateUTF8Short(t *testing.T) { + out, tr := TruncateUTF8([]byte("hi"), 100) + if tr || out != "hi" { + t.Fatalf("got %q tr=%v", out, tr) + } +} + +func TestTruncateUTF8Long(t *testing.T) { + in := strings.Repeat("a", 9000) + out, tr := TruncateUTF8([]byte(in), MaxOutputBytes) + if !tr { + t.Fatal("expected truncated") + } + if len([]byte(out)) > MaxOutputBytes { + t.Fatalf("byte len %d > %d", len([]byte(out)), MaxOutputBytes) + } + if !strings.Contains(out, "truncated") { + t.Fatalf("no marker: %q", out[len(out)-50:]) + } +} + +func TestTruncateUTF8SanitizesInvalidBytes(t *testing.T) { + // invalid UTF-8 bytes that json.Marshal would replace with U+FFFD (3 bytes each) + in := append([]byte("hello"), 0xff, 0xfe, 0xfd) + out, tr := TruncateUTF8(in, 100) + if tr { + t.Fatal("should fit") + } + if !utf8.ValidString(out) { + t.Fatalf("output must be valid UTF-8: %q", out) + } +} + +func TestTruncateUTF8Multibyte(t *testing.T) { + in := strings.Repeat("ё", 5000) // 2 bytes per rune = 10000 bytes + out, tr := TruncateUTF8([]byte(in), MaxOutputBytes) + if !tr { + t.Fatal("expected truncated") + } + if !utf8.ValidString(out) { + t.Fatal("invalid UTF-8 after truncate") + } + if len([]byte(out)) > MaxOutputBytes { + t.Fatalf("byte len %d > %d", len([]byte(out)), MaxOutputBytes) + } +} + +func TestRunOK(t *testing.T) { + r := Run(context.Background(), []string{"sh", "-c", "echo ok"}, 2*time.Second) + if r.Status != StatusOK || r.ExitCode != 0 { + t.Fatalf("got %+v", r) + } + if !strings.Contains(r.Output, "ok") { + t.Fatalf("output: %q", r.Output) + } +} + +func TestRunWarning(t *testing.T) { + r := Run(context.Background(), []string{"sh", "-c", "exit 1"}, 2*time.Second) + if r.Status != StatusWarning || r.ExitCode != 1 { + t.Fatalf("got %+v", r) + } +} + +func TestRunCritical(t *testing.T) { + r := Run(context.Background(), []string{"sh", "-c", "exit 2"}, 2*time.Second) + if r.Status != StatusCritical { + t.Fatalf("got %+v", r) + } +} + +func TestRunUnknownExit(t *testing.T) { + r := Run(context.Background(), []string{"sh", "-c", "exit 7"}, 2*time.Second) + if r.Status != StatusUnknown || r.ExitCode != 7 { + t.Fatalf("got %+v", r) + } +} + +func TestRunTimeout(t *testing.T) { + r := Run(context.Background(), []string{"sh", "-c", "sleep 5"}, 200*time.Millisecond) + if r.Status != StatusUnknown { + t.Fatalf("got %+v", r) + } + if r.DurationMs > 2000 { + t.Fatalf("duration too long: %d", r.DurationMs) + } +} + +func TestRunNoExec(t *testing.T) { + r := Run(context.Background(), []string{"/does/not/exist/zzz"}, 1*time.Second) + if r.Status != StatusUnknown { + t.Fatalf("got %+v", r) + } +} diff --git a/agent/internal/scheduler/scheduler.go b/agent/internal/scheduler/scheduler.go new file mode 100644 index 0000000..62de024 --- /dev/null +++ b/agent/internal/scheduler/scheduler.go @@ -0,0 +1,113 @@ +package scheduler + +import ( + "context" + "sync" + "time" + + "github.com/monlet/agent/internal/config" + "github.com/monlet/agent/internal/eventid" + "github.com/monlet/agent/internal/runner" +) + +// Event is a contract-shaped check result emitted by the scheduler. +type Event struct { + EventID string `json:"event_id"` + CheckID string `json:"check_id"` + ObservedAt time.Time `json:"observed_at"` + Status string `json:"status"` + ExitCode int `json:"exit_code"` + DurationMs int64 `json:"duration_ms"` + Output string `json:"output,omitempty"` + OutputTruncated bool `json:"output_truncated,omitempty"` + NotificationOwner string `json:"notification_owner,omitempty"` + IncidentKey string `json:"incident_key,omitempty"` +} + +type Hooks struct { + OnSkipped func(checkID string) + OnResult func(ev Event) +} + +// Run starts one goroutine per check. Emits events to out. Blocks until ctx is done. +func Run(ctx context.Context, agentID string, checks []config.CheckConfig, out chan<- Event, h Hooks) { + var wg sync.WaitGroup + for i := range checks { + c := checks[i] + wg.Add(1) + go func() { + defer wg.Done() + runCheck(ctx, agentID, c, out, h) + }() + } + wg.Wait() +} + +func runCheck(ctx context.Context, agentID string, c config.CheckConfig, out chan<- Event, h Hooks) { + t := time.NewTicker(c.Interval.Duration) + defer t.Stop() + var running bool + var mu sync.Mutex + + exec := func() { + mu.Lock() + if running { + mu.Unlock() + if h.OnSkipped != nil { + h.OnSkipped(c.ID) + } + return + } + running = true + mu.Unlock() + defer func() { + mu.Lock() + running = false + mu.Unlock() + }() + + res := runner.Run(ctx, c.Command, c.Timeout.Duration) + id, err := eventid.New() + if err != nil { + return + } + ev := Event{ + EventID: id, + CheckID: c.ID, + ObservedAt: time.Now().UTC(), + Status: string(res.Status), + ExitCode: res.ExitCode, + DurationMs: res.DurationMs, + Output: res.Output, + OutputTruncated: res.OutputTruncated, + NotificationOwner: c.NotificationOwner, + IncidentKey: incidentKey(agentID, c), + } + if h.OnResult != nil { + h.OnResult(ev) + } + select { + case out <- ev: + case <-ctx.Done(): + } + } + + // Fire first run immediately so smoke tests don't wait a full interval. + go exec() + + for { + select { + case <-ctx.Done(): + return + case <-t.C: + go exec() + } + } +} + +func incidentKey(agentID string, c config.CheckConfig) string { + if c.DedupeKey != "" { + return c.DedupeKey + } + return agentID + ":" + c.ID +} diff --git a/agent/internal/scheduler/scheduler_test.go b/agent/internal/scheduler/scheduler_test.go new file mode 100644 index 0000000..9f917b2 --- /dev/null +++ b/agent/internal/scheduler/scheduler_test.go @@ -0,0 +1,52 @@ +package scheduler + +import ( + "context" + "sync/atomic" + "testing" + "time" + + "github.com/monlet/agent/internal/config" +) + +func mkCheck(id, cmd string, interval, timeout time.Duration) config.CheckConfig { + return config.CheckConfig{ + ID: id, + Command: []string{"sh", "-c", cmd}, + Interval: config.Duration{Duration: interval}, + Timeout: config.Duration{Duration: timeout}, + NotificationOwner: "server", + } +} + +func TestEmitsEvents(t *testing.T) { + ctx, cancel := context.WithTimeout(context.Background(), 500*time.Millisecond) + defer cancel() + out := make(chan Event, 10) + go Run(ctx, "a1", []config.CheckConfig{mkCheck("c1", "echo hi", 200*time.Millisecond, 100*time.Millisecond)}, out, Hooks{}) + select { + case ev := <-out: + if ev.CheckID != "c1" || ev.Status != "ok" { + t.Fatalf("unexpected event: %+v", ev) + } + if ev.IncidentKey != "a1:c1" { + t.Fatalf("incident_key: %q", ev.IncidentKey) + } + case <-ctx.Done(): + t.Fatal("no event") + } +} + +func TestNoOverlapSkips(t *testing.T) { + ctx, cancel := context.WithTimeout(context.Background(), 600*time.Millisecond) + defer cancel() + out := make(chan Event, 10) + var skipped int32 + hooks := Hooks{OnSkipped: func(string) { atomic.AddInt32(&skipped, 1) }} + c := mkCheck("slow", "sleep 1", 100*time.Millisecond, 900*time.Millisecond) + go Run(ctx, "a1", []config.CheckConfig{c}, out, hooks) + <-ctx.Done() + if atomic.LoadInt32(&skipped) == 0 { + t.Fatal("expected at least one skipped tick") + } +} diff --git a/agent/internal/spool/spool.go b/agent/internal/spool/spool.go new file mode 100644 index 0000000..3751495 --- /dev/null +++ b/agent/internal/spool/spool.go @@ -0,0 +1,213 @@ +package spool + +import ( + "encoding/json" + "fmt" + "os" + "path/filepath" + "sort" + "strconv" + "strings" + "sync" + + "github.com/monlet/agent/internal/scheduler" +) + +const ( + DefaultMaxEvents = 10_000 + DefaultMaxBytes = 50 * 1024 * 1024 + fileExt = ".json" +) + +type Spool struct { + dir string + maxEvents int + maxBytes int64 + + mu sync.Mutex + nextSeq uint64 + files []entry // sorted ascending by seq + bytes int64 +} + +type entry struct { + seq uint64 + size int64 + name string +} + +type Item struct { + Seq uint64 + Event scheduler.Event +} + +func Open(dir string, maxEvents int, maxBytes int64) (*Spool, error) { + if err := os.MkdirAll(dir, 0o755); err != nil { + return nil, err + } + if maxEvents <= 0 { + maxEvents = DefaultMaxEvents + } + if maxBytes <= 0 { + maxBytes = DefaultMaxBytes + } + s := &Spool{dir: dir, maxEvents: maxEvents, maxBytes: maxBytes} + if err := s.rescan(); err != nil { + return nil, err + } + s.mu.Lock() + s.dropOldestLocked() + s.mu.Unlock() + return s, nil +} + +func (s *Spool) rescan() error { + ents, err := os.ReadDir(s.dir) + if err != nil { + return err + } + s.files = s.files[:0] + s.bytes = 0 + for _, e := range ents { + if e.IsDir() || !strings.HasSuffix(e.Name(), fileExt) { + continue + } + base := strings.TrimSuffix(e.Name(), fileExt) + seq, err := strconv.ParseUint(base, 10, 64) + if err != nil { + continue + } + info, err := e.Info() + if err != nil { + return err + } + s.files = append(s.files, entry{seq: seq, size: info.Size(), name: e.Name()}) + s.bytes += info.Size() + if seq >= s.nextSeq { + s.nextSeq = seq + 1 + } + } + sort.Slice(s.files, func(i, j int) bool { return s.files[i].seq < s.files[j].seq }) + return nil +} + +func (s *Spool) Len() int { + s.mu.Lock() + defer s.mu.Unlock() + return len(s.files) +} + +func (s *Spool) Bytes() int64 { + s.mu.Lock() + defer s.mu.Unlock() + return s.bytes +} + +func (s *Spool) Enqueue(ev scheduler.Event) error { + data, err := json.Marshal(ev) + if err != nil { + return err + } + s.mu.Lock() + defer s.mu.Unlock() + + seq := s.nextSeq + s.nextSeq++ + name := fmt.Sprintf("%014d%s", seq, fileExt) + tmp := filepath.Join(s.dir, name+".tmp") + final := filepath.Join(s.dir, name) + if err := os.WriteFile(tmp, data, 0o600); err != nil { + return err + } + if err := os.Rename(tmp, final); err != nil { + _ = os.Remove(tmp) + return err + } + size := int64(len(data)) + s.files = append(s.files, entry{seq: seq, size: size, name: name}) + s.bytes += size + s.dropOldestLocked() + return nil +} + +func (s *Spool) dropOldestLocked() { + for len(s.files) > s.maxEvents || s.bytes > s.maxBytes { + if len(s.files) == 0 { + return + } + old := s.files[0] + _ = os.Remove(filepath.Join(s.dir, old.name)) + s.bytes -= old.size + s.files = s.files[1:] + } +} + +// Peek returns up to limit oldest items. Does not remove them. +func (s *Spool) Peek(limit int) ([]Item, error) { + s.mu.Lock() + defer s.mu.Unlock() + if limit > len(s.files) { + limit = len(s.files) + } + out := make([]Item, 0, limit) + corrupt := make(map[uint64]struct{}) + for i := 0; i < limit; i++ { + e := s.files[i] + b, err := os.ReadFile(filepath.Join(s.dir, e.name)) + if err != nil { + if os.IsNotExist(err) { + corrupt[e.seq] = struct{}{} + continue + } + s.dropCorruptLocked(corrupt) + return out, err + } + var ev scheduler.Event + if err := json.Unmarshal(b, &ev); err != nil { + _ = os.Remove(filepath.Join(s.dir, e.name)) + corrupt[e.seq] = struct{}{} + continue + } + out = append(out, Item{Seq: e.seq, Event: ev}) + } + s.dropCorruptLocked(corrupt) + return out, nil +} + +func (s *Spool) dropCorruptLocked(seqs map[uint64]struct{}) { + if len(seqs) == 0 { + return + } + kept := s.files[:0] + for _, e := range s.files { + if _, bad := seqs[e.seq]; bad { + s.bytes -= e.size + continue + } + kept = append(kept, e) + } + s.files = kept +} + +// Commit removes items with the given sequence numbers. +func (s *Spool) Commit(seqs []uint64) { + if len(seqs) == 0 { + return + } + want := make(map[uint64]struct{}, len(seqs)) + for _, x := range seqs { + want[x] = struct{}{} + } + s.mu.Lock() + defer s.mu.Unlock() + kept := s.files[:0] + for _, e := range s.files { + if _, ok := want[e.seq]; ok { + _ = os.Remove(filepath.Join(s.dir, e.name)) + s.bytes -= e.size + continue + } + kept = append(kept, e) + } + s.files = kept +} diff --git a/agent/internal/spool/spool_test.go b/agent/internal/spool/spool_test.go new file mode 100644 index 0000000..2b1817b --- /dev/null +++ b/agent/internal/spool/spool_test.go @@ -0,0 +1,129 @@ +package spool + +import ( + "fmt" + "os" + "path/filepath" + "strings" + "testing" + + "github.com/monlet/agent/internal/scheduler" +) + +func mkEvent(id string) scheduler.Event { + return scheduler.Event{EventID: id, CheckID: "c1", Status: "ok"} +} + +func TestEnqueuePeekCommit(t *testing.T) { + s, err := Open(t.TempDir(), 10, 1<<20) + if err != nil { + t.Fatal(err) + } + for i := 0; i < 3; i++ { + if err := s.Enqueue(mkEvent(fmt.Sprintf("e%d", i))); err != nil { + t.Fatal(err) + } + } + items, err := s.Peek(10) + if err != nil { + t.Fatal(err) + } + if len(items) != 3 { + t.Fatalf("got %d items", len(items)) + } + if items[0].Event.EventID != "e0" { + t.Fatalf("FIFO violated: %q", items[0].Event.EventID) + } + s.Commit([]uint64{items[0].Seq, items[1].Seq}) + if s.Len() != 1 { + t.Fatalf("len=%d", s.Len()) + } +} + +func TestDropOldestByCount(t *testing.T) { + s, err := Open(t.TempDir(), 3, 1<<20) + if err != nil { + t.Fatal(err) + } + for i := 0; i < 5; i++ { + _ = s.Enqueue(mkEvent(fmt.Sprintf("e%d", i))) + } + if s.Len() != 3 { + t.Fatalf("len=%d", s.Len()) + } + items, _ := s.Peek(10) + if items[0].Event.EventID != "e2" { + t.Fatalf("expected oldest dropped, got %q", items[0].Event.EventID) + } +} + +func TestDropOldestByBytes(t *testing.T) { + big := strings.Repeat("x", 1000) + s, err := Open(t.TempDir(), 1_000_000, 3000) + if err != nil { + t.Fatal(err) + } + for i := 0; i < 10; i++ { + ev := mkEvent(fmt.Sprintf("e%d", i)) + ev.Output = big + _ = s.Enqueue(ev) + } + if s.Bytes() > 3000 { + t.Fatalf("bytes=%d > 3000", s.Bytes()) + } +} + +func TestReopenAppliesLimits(t *testing.T) { + dir := t.TempDir() + s, _ := Open(dir, 10, 1<<20) + for i := 0; i < 5; i++ { + _ = s.Enqueue(mkEvent(fmt.Sprintf("e%d", i))) + } + // Reopen with tighter limit: should drop oldest down to maxEvents. + s2, err := Open(dir, 2, 1<<20) + if err != nil { + t.Fatal(err) + } + if s2.Len() != 2 { + t.Fatalf("expected limits applied on Open, len=%d", s2.Len()) + } + items, _ := s2.Peek(10) + if items[0].Event.EventID != "e3" { + t.Fatalf("expected oldest dropped, got %q", items[0].Event.EventID) + } +} + +func TestPeekSkipsCorruptAndCleansList(t *testing.T) { + dir := t.TempDir() + s, _ := Open(dir, 10, 1<<20) + _ = s.Enqueue(mkEvent("e0")) + _ = s.Enqueue(mkEvent("e1")) + // Corrupt the first file on disk. + items, _ := s.Peek(10) + if err := os.WriteFile(filepath.Join(dir, fmt.Sprintf("%014d.json", items[0].Seq)), []byte("not-json"), 0o600); err != nil { + t.Fatal(err) + } + out, _ := s.Peek(10) + if len(out) != 1 || out[0].Event.EventID != "e1" { + t.Fatalf("got %+v", out) + } + if s.Len() != 1 { + t.Fatalf("corrupt entry not pruned from list, len=%d", s.Len()) + } +} + +func TestReopenPreservesOrder(t *testing.T) { + dir := t.TempDir() + s, _ := Open(dir, 100, 1<<20) + for i := 0; i < 3; i++ { + _ = s.Enqueue(mkEvent(fmt.Sprintf("e%d", i))) + } + s2, err := Open(dir, 100, 1<<20) + if err != nil { + t.Fatal(err) + } + items, _ := s2.Peek(10) + if len(items) != 3 || items[0].Event.EventID != "e0" || items[2].Event.EventID != "e2" { + t.Fatalf("bad reopen: %+v", items) + } +} diff --git a/agent/systemd/monlet-agent.service.example b/agent/systemd/monlet-agent.service.example new file mode 100644 index 0000000..892e9c0 --- /dev/null +++ b/agent/systemd/monlet-agent.service.example @@ -0,0 +1,21 @@ +[Unit] +Description=Monlet Agent +After=network-online.target +Wants=network-online.target + +[Service] +Type=simple +ExecStart=/usr/local/bin/monlet-agent -config /etc/monlet/agent.toml +Restart=always +RestartSec=5s +User=monlet +Group=monlet +StateDirectory=monlet-agent +NoNewPrivileges=true +PrivateTmp=true +ProtectSystem=strict +ProtectHome=true +ReadWritePaths=/var/lib/monlet-agent + +[Install] +WantedBy=multi-user.target diff --git a/api/README.md b/api/README.md new file mode 100644 index 0000000..583e611 --- /dev/null +++ b/api/README.md @@ -0,0 +1,11 @@ +# API Contract + +`openapi.yaml` is the first versioned Monlet API contract. + +Rules: + +- all public endpoints are under `/api/v1`; +- agent-facing endpoints use bearer token auth; +- check result ingestion is idempotent by `event_id`; +- incident keys are stable and must not include dynamic values; +- implementation must update this contract when public API changes. diff --git a/api/openapi.yaml b/api/openapi.yaml new file mode 100644 index 0000000..086a5cf --- /dev/null +++ b/api/openapi.yaml @@ -0,0 +1,565 @@ +openapi: 3.1.0 +info: + title: Monlet API + version: 1.0.0 + description: | + Monlet v1 API contract. Stage 1 frozen. + + All endpoints under `/api/v1`. Authentication uses a single shared Bearer token (see ADR-0007). Agent-facing ingestion endpoints (`/heartbeat`, `/events`) and read-only UI-facing endpoints share the same security scheme in v1. +servers: + - url: http://127.0.0.1:8000 +security: + - bearerAuth: [] +paths: + /api/v1/health: + get: + summary: Health check + security: [] + responses: + "200": + description: Service is alive. + content: + application/json: + schema: + $ref: "#/components/schemas/HealthResponse" + /api/v1/ready: + get: + summary: Readiness check + security: [] + responses: + "200": + description: Service is ready. + content: + application/json: + schema: + $ref: "#/components/schemas/HealthResponse" + "503": + description: Service is not ready. + content: + application/json: + schema: + $ref: "#/components/schemas/ErrorResponse" + /api/v1/heartbeat: + post: + summary: Ingest agent heartbeat + requestBody: + required: true + content: + application/json: + schema: + $ref: "#/components/schemas/HeartbeatRequest" + responses: + "202": + description: Heartbeat accepted. + content: + application/json: + schema: + $ref: "#/components/schemas/AcceptedResponse" + "400": + $ref: "#/components/responses/ValidationError" + "401": + $ref: "#/components/responses/Unauthorized" + "413": + $ref: "#/components/responses/PayloadTooLarge" + "500": + $ref: "#/components/responses/InternalError" + /api/v1/events: + post: + summary: Ingest agent check result events + requestBody: + required: true + content: + application/json: + schema: + $ref: "#/components/schemas/EventBatchRequest" + responses: + "202": + description: Events accepted. + content: + application/json: + schema: + $ref: "#/components/schemas/EventBatchAcceptedResponse" + "400": + $ref: "#/components/responses/ValidationError" + "401": + $ref: "#/components/responses/Unauthorized" + "413": + $ref: "#/components/responses/PayloadTooLarge" + "500": + $ref: "#/components/responses/InternalError" + /api/v1/agents: + get: + summary: List agents + parameters: + - $ref: "#/components/parameters/Cursor" + - $ref: "#/components/parameters/Limit" + responses: + "200": + description: Agent list. + content: + application/json: + schema: + allOf: + - $ref: "#/components/schemas/PageEnvelope" + - type: object + required: [items] + properties: + items: + type: array + items: + $ref: "#/components/schemas/Agent" + "401": + $ref: "#/components/responses/Unauthorized" + /api/v1/agents/{agent_id}: + get: + summary: Get agent detail + parameters: + - $ref: "#/components/parameters/AgentId" + responses: + "200": + description: Agent detail. + content: + application/json: + schema: + $ref: "#/components/schemas/Agent" + "401": + $ref: "#/components/responses/Unauthorized" + "404": + $ref: "#/components/responses/NotFound" + /api/v1/checks: + get: + summary: List current check states + parameters: + - $ref: "#/components/parameters/Cursor" + - $ref: "#/components/parameters/Limit" + responses: + "200": + description: Check state list. + content: + application/json: + schema: + allOf: + - $ref: "#/components/schemas/PageEnvelope" + - type: object + required: [items] + properties: + items: + type: array + items: + $ref: "#/components/schemas/CheckState" + "401": + $ref: "#/components/responses/Unauthorized" + /api/v1/incidents: + get: + summary: List incidents + parameters: + - name: state + in: query + required: false + schema: + type: string + enum: [open, resolved] + - $ref: "#/components/parameters/Cursor" + - $ref: "#/components/parameters/Limit" + responses: + "200": + description: Incident list. + content: + application/json: + schema: + allOf: + - $ref: "#/components/schemas/PageEnvelope" + - type: object + required: [items] + properties: + items: + type: array + items: + $ref: "#/components/schemas/Incident" + "401": + $ref: "#/components/responses/Unauthorized" + /api/v1/events/query: + get: + summary: Query stored events + parameters: + - name: agent_id + in: query + required: false + schema: + type: string + maxLength: 128 + pattern: "^[A-Za-z0-9._:-]+$" + - name: check_id + in: query + required: false + schema: + type: string + maxLength: 128 + pattern: "^[A-Za-z0-9._:-]+$" + - $ref: "#/components/parameters/Cursor" + - $ref: "#/components/parameters/Limit" + responses: + "200": + description: Event list. + content: + application/json: + schema: + allOf: + - $ref: "#/components/schemas/PageEnvelope" + - type: object + required: [items] + properties: + items: + type: array + items: + $ref: "#/components/schemas/StoredCheckResultEvent" + "401": + $ref: "#/components/responses/Unauthorized" + /api/v1/notifiers/outbox: + get: + summary: List notification outbox items + parameters: + - $ref: "#/components/parameters/Cursor" + - $ref: "#/components/parameters/Limit" + responses: + "200": + description: Outbox list. + content: + application/json: + schema: + allOf: + - $ref: "#/components/schemas/PageEnvelope" + - type: object + required: [items] + properties: + items: + type: array + items: + $ref: "#/components/schemas/NotificationOutboxItem" + "401": + $ref: "#/components/responses/Unauthorized" +components: + securitySchemes: + bearerAuth: + type: http + scheme: bearer + description: Shared static Bearer token in v1. See ADR-0007. + parameters: + AgentId: + name: agent_id + in: path + required: true + schema: + type: string + maxLength: 128 + pattern: "^[A-Za-z0-9._:-]+$" + Cursor: + name: cursor + in: query + required: false + description: Opaque pagination cursor returned in `next_cursor`. + schema: + type: string + maxLength: 512 + Limit: + name: limit + in: query + required: false + schema: + type: integer + minimum: 1 + maximum: 500 + default: 100 + responses: + Unauthorized: + description: Authentication failed. + content: + application/json: + schema: + $ref: "#/components/schemas/ErrorResponse" + NotFound: + description: Resource not found. + content: + application/json: + schema: + $ref: "#/components/schemas/ErrorResponse" + PayloadTooLarge: + description: Payload exceeds configured limit (1 MiB). + content: + application/json: + schema: + $ref: "#/components/schemas/ErrorResponse" + ValidationError: + description: Request body or parameters failed validation. + content: + application/json: + schema: + $ref: "#/components/schemas/ErrorResponse" + InternalError: + description: Unexpected server error. + content: + application/json: + schema: + $ref: "#/components/schemas/ErrorResponse" + schemas: + HealthResponse: + type: object + required: [status] + properties: + status: + type: string + enum: [ok] + ErrorResponse: + type: object + required: [error] + properties: + error: + type: object + required: [code, message, request_id] + properties: + code: + type: string + enum: + - unauthorized + - not_found + - validation + - payload_too_large + - rate_limited + - internal + - not_ready + message: + type: string + request_id: + type: string + description: Server-generated request id, mirrored from X-Request-Id. + PageEnvelope: + type: object + properties: + next_cursor: + type: [string, "null"] + description: Pass back as `cursor` to fetch the next page. Null/absent if no more results. + AcceptedResponse: + type: object + required: [accepted] + properties: + accepted: + type: boolean + EventBatchAcceptedResponse: + type: object + required: [accepted, deduplicated] + properties: + accepted: + type: integer + minimum: 0 + description: Number of events stored on this request. + deduplicated: + type: integer + minimum: 0 + description: Number of events whose `event_id` was already known. + HeartbeatRequest: + type: object + required: [agent_id, observed_at, hostname, version] + properties: + agent_id: + type: string + maxLength: 128 + pattern: "^[A-Za-z0-9._:-]+$" + observed_at: + type: string + format: date-time + hostname: + type: string + maxLength: 253 + version: + type: string + maxLength: 64 + mode: + type: string + enum: [prometheus_only, push_only, hybrid] + labels: + type: object + maxProperties: 32 + propertyNames: + maxLength: 64 + pattern: "^[A-Za-z0-9._:-]+$" + additionalProperties: + type: string + maxLength: 256 + description: Up to 32 labels. Key max 64, value max 256. + EventBatchRequest: + type: object + required: [agent_id, events] + properties: + agent_id: + type: string + maxLength: 128 + pattern: "^[A-Za-z0-9._:-]+$" + events: + type: array + minItems: 1 + maxItems: 200 + items: + $ref: "#/components/schemas/CheckResultEvent" + CheckResultEvent: + type: object + description: | + Event body used inside `EventBatchRequest.events`. The owning `agent_id` + is supplied once at the batch envelope; per-event `agent_id` is intentionally + absent so requests cannot mix or spoof other agents under a shared token. + Server response endpoints (`GET /events/query`) include `agent_id` separately + in their item schema. + required: + - event_id + - check_id + - observed_at + - status + - exit_code + - duration_ms + properties: + event_id: + type: string + description: UUIDv7 string (lower-case, with dashes). See ADR-0006. + minLength: 36 + maxLength: 36 + pattern: "^[0-9a-f]{8}-[0-9a-f]{4}-7[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$" + check_id: + type: string + maxLength: 128 + pattern: "^[A-Za-z0-9._:-]+$" + observed_at: + type: string + format: date-time + status: + type: string + enum: [ok, warning, critical, unknown] + exit_code: + type: integer + duration_ms: + type: integer + minimum: 0 + output: + type: string + maxLength: 8192 + description: | + Limit is 8 KiB measured in UTF-8 **bytes** (not characters). Agent truncates + on a UTF-8 boundary before send and appends marker `...[truncated N bytes]`. + `maxLength: 8192` here is a coarse schema-level upper bound; servers MUST + additionally reject events whose UTF-8 byte length exceeds 8192 with + `400 validation`. + output_truncated: + type: boolean + default: false + notification_owner: + type: string + enum: [server, prometheus, none] + incident_key: + type: string + maxLength: 256 + StoredCheckResultEvent: + description: Stored event returned by `GET /events/query`. Includes the resolved `agent_id` and server-side `received_at`. + allOf: + - $ref: "#/components/schemas/CheckResultEvent" + - type: object + required: [agent_id, received_at] + properties: + agent_id: + type: string + maxLength: 128 + pattern: "^[A-Za-z0-9._:-]+$" + received_at: + type: string + format: date-time + Agent: + type: object + required: [agent_id, hostname, status, last_seen_at] + properties: + agent_id: + type: string + hostname: + type: string + status: + type: string + enum: [alive, stale, dead] + last_seen_at: + type: string + format: date-time + version: + type: string + mode: + type: string + enum: [prometheus_only, push_only, hybrid] + labels: + type: object + additionalProperties: + type: string + CheckState: + type: object + required: [agent_id, check_id, status, last_observed_at] + properties: + agent_id: + type: string + check_id: + type: string + status: + type: string + enum: [ok, warning, critical, unknown] + last_observed_at: + type: string + format: date-time + exit_code: + type: integer + incident_key: + type: string + Incident: + type: object + required: [id, incident_key, state, severity, opened_at] + properties: + id: + type: string + incident_key: + type: string + state: + type: string + enum: [open, resolved] + severity: + type: string + enum: [warning, critical, unknown] + agent_id: + type: string + check_id: + type: string + opened_at: + type: string + format: date-time + resolved_at: + type: [string, "null"] + format: date-time + summary: + type: string + NotificationOutboxItem: + type: object + required: [id, notifier, state, incident_id, event_type, attempts] + properties: + id: + type: string + notifier: + type: string + state: + type: string + enum: [pending, sending, sent, retry, failed, discarded] + incident_id: + type: string + event_type: + type: string + enum: [firing, resolved] + attempts: + type: integer + minimum: 0 + next_attempt_at: + type: [string, "null"] + format: date-time + last_error: + type: [string, "null"] + updated_at: + type: string + format: date-time diff --git a/deploy/README.md b/deploy/README.md new file mode 100644 index 0000000..fcfeb55 --- /dev/null +++ b/deploy/README.md @@ -0,0 +1,15 @@ +# Deploy + +Deployment examples live here. + +Planned files: + +- Docker Compose for local development; +- PostgreSQL service; +- optional Prometheus scrape config; +- optional Alertmanager config; +- optional Grafana dashboard stub. + +No production-ready deployment files are implemented in Stage 0. + +Docker examples must not install Python, Node, or Go project dependencies on the host. Dependency installation belongs inside images/containers or repo-local development directories. diff --git a/deploy/alertmanager/README.md b/deploy/alertmanager/README.md new file mode 100644 index 0000000..cec8e69 --- /dev/null +++ b/deploy/alertmanager/README.md @@ -0,0 +1,9 @@ +# Alertmanager + +Placeholder for optional Alertmanager integration examples. + +Planned examples: + +- receiver config for Monlet-originated alerts; +- stable label conventions; +- routing notes for avoiding duplicate notifications. diff --git a/deploy/grafana/README.md b/deploy/grafana/README.md new file mode 100644 index 0000000..c7fb723 --- /dev/null +++ b/deploy/grafana/README.md @@ -0,0 +1,11 @@ +# Grafana + +Placeholder for Grafana dashboard examples. + +Planned dashboards: + +- agents alive/stale/dead; +- check state counts; +- open incidents; +- notification outbox state; +- server ingestion and notifier failures. diff --git a/deploy/prometheus/README.md b/deploy/prometheus/README.md new file mode 100644 index 0000000..75d1612 --- /dev/null +++ b/deploy/prometheus/README.md @@ -0,0 +1,9 @@ +# Prometheus + +Placeholder for Prometheus examples. + +Planned examples: + +- scrape Monlet agent `/metrics`; +- scrape Monlet server `/metrics`; +- sample rules that avoid duplicate alerting when `notification_owner = server`. diff --git a/docs/operations/deployment.md b/docs/operations/deployment.md new file mode 100644 index 0000000..ed3fa1c --- /dev/null +++ b/docs/operations/deployment.md @@ -0,0 +1,68 @@ +# Deployment + +This is an initial deployment plan, not a production runbook. + +## Agent + +Agent runs as a systemd service. + +Expected paths: + +- binary: `/usr/local/bin/monlet-agent`; +- config: `/etc/monlet/agent.toml`; +- state: `/var/lib/monlet-agent`; +- logs: journald. + +The agent should support graceful shutdown and avoid overlapping runs for the same check. + +## Server + +Server runs in Docker. + +Production storage: + +- PostgreSQL. + +Local/demo storage: + +- SQLite may be supported later, but must be documented as non-production. + +Required deployment features: + +- TLS-ready reverse proxy; +- agent API tokens; +- request size limits; +- structured logs; +- `/health`; +- `/ready`; +- `/metrics`. + +Docker image rule: + +- build server dependencies inside the image; +- do not require Python packages installed on the host; +- do not mount host virtualenvs into the container; +- local non-Docker development uses `server/.venv` only. + +## Web + +Web is read-only in v1. + +Auth plan: + +- reverse proxy basic auth or equivalent gateway auth; +- no full RBAC in v1. + +## Observability + +Optional examples: + +- Prometheus scrape config; +- Alertmanager route config; +- Grafana dashboard stub. + +Metrics must avoid high-cardinality labels. + +## Backups + +PostgreSQL backup/restore docs are required before first production use. diff --git a/examples/README.md b/examples/README.md new file mode 100644 index 0000000..af2a020 --- /dev/null +++ b/examples/README.md @@ -0,0 +1,5 @@ +# Examples + +Examples for checks, configs, and local demos. + +Stage 0 includes only placeholders. diff --git a/examples/checks/example_check.sh b/examples/checks/example_check.sh new file mode 100755 index 0000000..c0ec280 --- /dev/null +++ b/examples/checks/example_check.sh @@ -0,0 +1,3 @@ +#!/bin/sh +echo "ok" +exit 0 diff --git a/examples/event-batch.json b/examples/event-batch.json new file mode 100644 index 0000000..16d4352 --- /dev/null +++ b/examples/event-batch.json @@ -0,0 +1,29 @@ +{ + "agent_id": "host-01.prod", + "events": [ + { + "event_id": "018f5b3a-0a3d-7c5b-9a1e-2c4f6a8b0c11", + "check_id": "disk_root", + "observed_at": "2026-05-26T12:00:05Z", + "status": "ok", + "exit_code": 0, + "duration_ms": 42, + "output": "/ 18% used (180G/1.0T)", + "output_truncated": false, + "notification_owner": "server", + "incident_key": "host-01.prod:disk_root" + }, + { + "event_id": "018f5b3a-1f7e-7d2a-8b3c-9e1d2f3a4b5c", + "check_id": "postgres_up", + "observed_at": "2026-05-26T12:00:05Z", + "status": "critical", + "exit_code": 2, + "duration_ms": 1503, + "output": "could not connect to server: Connection refused", + "output_truncated": false, + "notification_owner": "server", + "incident_key": "host-01.prod:postgres_up" + } + ] +} diff --git a/examples/heartbeat.json b/examples/heartbeat.json new file mode 100644 index 0000000..3f3d0bd --- /dev/null +++ b/examples/heartbeat.json @@ -0,0 +1,12 @@ +{ + "agent_id": "host-01.prod", + "observed_at": "2026-05-26T12:00:00Z", + "hostname": "host-01.prod.example.com", + "version": "0.1.0", + "mode": "push_only", + "labels": { + "env": "prod", + "region": "eu-central-1", + "role": "api" + } +} diff --git a/server/.env.example b/server/.env.example new file mode 100644 index 0000000..d847af7 --- /dev/null +++ b/server/.env.example @@ -0,0 +1,6 @@ +MONLET_ENV=local +MONLET_LOG_LEVEL=info +MONLET_DATABASE_URL=postgresql+psycopg://monlet:monlet@localhost:5432/monlet +MONLET_AGENT_TOKEN_BOOTSTRAP=replace-me +MONLET_MAX_REQUEST_BYTES=1048576 +MONLET_MAX_CHECK_OUTPUT_BYTES=8192 diff --git a/server/README.md b/server/README.md new file mode 100644 index 0000000..f90978c --- /dev/null +++ b/server/README.md @@ -0,0 +1,28 @@ +# Monlet Server + +Placeholder for the Python/FastAPI server. + +Planned stack: + +- Python 3.14; +- uv; +- FastAPI; +- Pydantic; +- SQLAlchemy 2.x style; +- Alembic; +- PostgreSQL; +- httpx for outbound notifier calls; +- structured logging; +- Prometheus metrics; +- Docker image. + +No server core logic is implemented in Stage 0. + +## Planned Commands + +```sh +UV_PROJECT_ENVIRONMENT=.venv UV_CACHE_DIR=../.cache/uv uv sync +UV_PROJECT_ENVIRONMENT=.venv UV_CACHE_DIR=../.cache/uv uv run fastapi dev monlet_server/main.py +``` + +Do not install Python packages globally on the host. Docker builds install server dependencies inside the image; local development uses `server/.venv` and repo-local `.cache/uv`. diff --git a/web/README.md b/web/README.md new file mode 100644 index 0000000..c79b0e1 --- /dev/null +++ b/web/README.md @@ -0,0 +1,32 @@ +# Monlet Web + +Placeholder for the read-only dashboard. + +Planned stack: + +- Next.js App Router; +- TypeScript; +- Tailwind CSS; +- read-only API client; +- reverse-proxy/basic-auth friendly deployment. + +No web UI is implemented in Stage 0. + +## Planned Commands + +```sh +npm_config_cache=../.cache/npm npm install +npm_config_cache=../.cache/npm npm run dev +``` + +Do not install Node packages globally. Project dependencies live in `web/node_modules`; npm cache lives under repository `.cache/npm`. + +## Planned Pages + +- overview; +- agents; +- agent detail; +- checks; +- incidents; +- events; +- notifiers/outbox.