diff --git a/.github/workflows/README.md b/.github/workflows/README.md index de63ba8..5eef5a4 100644 --- a/.github/workflows/README.md +++ b/.github/workflows/README.md @@ -1,16 +1,16 @@ # CI Workflows -Placeholder for future GitHub Actions workflows. +`ci.yml` runs on pushes to `main` and on pull requests. It cancels older runs for +the same ref. -Planned jobs: +Jobs: -- Go test for `agent/`; -- `gofmt -l` and `go vet` for `agent/`; -- `go test -race` for Go stage checkpoints or concurrency-heavy changes; -- Python tests and migrations for `server/`; -- PostgreSQL constraint tests for `server/`; -- Agent-to-Server integration smoke; -- TypeScript check, lint, and build for `web/`; -- OpenAPI validation for `api/openapi.yaml`. -- OpenAPI example payload validation for `examples/heartbeat.json` and `examples/event-batch.json`. -- Docker Compose smoke for `/health`, `/ready`, `/metrics`, and example ingest. +- `openapi` - lints `api/openapi.yaml` with pinned `@redocly/cli`. +- `agent` - runs `gofmt`, `go vet`, `go test ./...`, and `go test -race ./...`. +- `server` - installs with `uv`, then runs Ruff and pytest. +- `web` - installs with npm, regenerates API types, then runs lint, typecheck, + build, and Playwright smoke. +- `stack-smoke` - runs `deploy/smoke.sh` after server and agent jobs pass. + +Keep tool versions pinned in the workflow or in the component lockfiles. Do not +replace pinned CI tools with `latest`. diff --git a/CONFIG_REFERENCE.ru.md b/CONFIG_REFERENCE.ru.md index f03a31d..d0e1977 100644 --- a/CONFIG_REFERENCE.ru.md +++ b/CONFIG_REFERENCE.ru.md @@ -36,6 +36,13 @@ interval = "60s" timeout = "10s" notifications_enabled = true resource_limits = { cpu_time = "5s", memory = "256MiB", open_files = 128 } + +[[checks]] +id = "daily_backup" +name = "Daily backup freshness" +command = "/usr/local/lib/monlet/check_backup_freshness.sh" +cron = "CRON_TZ=UTC 15 6 * * *" +timeout = "2m" ``` Ключи: @@ -49,11 +56,14 @@ resource_limits = { cpu_time = "5s", memory = "256MiB", open_files = 128 } - `metrics.listen` — адрес `/metrics`. - `checks[].id` — стабильный ID проверки. - `checks[].command` — shell-команда строкой; однострочно в `"..."`, многострочно в `'''...'''`. -- `checks[].interval` — период запуска. +- `checks[].interval` — период запуска (`60s`, `5m`, `1h`); обязателен, если не задан `checks[].cron`. +- `checks[].cron` — cron-style расписание; обязателен, если не задан `checks[].interval`. Поддерживаются стандартные 5 полей, `@hourly`/`@daily` и `CRON_TZ=...`; seconds-format, `@every` и `@reboot` отклоняются. - `checks[].timeout` — таймаут; при таймауте результат `critical`. - `checks[].notifications_enabled` — разрешить серверные уведомления по этой проверке. - `checks[].resource_limits` — опциональные лимиты процесса: CPU-время, virtual memory, open files. +Для каждой проверки нужно задать ровно один способ расписания: `interval` или `cron`. Interval-проверки запускаются сразу после старта/reload, затем по периоду. Cron-проверки ждут ближайший cron-slot и не запускаются немедленно при старте. + Exit code: `0=ok`, `1=warning`, `2=critical`, `3+=unknown`. Env агента: @@ -65,6 +75,7 @@ Reload: - `systemctl reload monlet-agent` перечитывает локальный TOML через `SIGHUP`. - Ошибочный config не применяется, старый продолжает работать. - Уже запущенный check при reload не убивается, а доживает до своего `timeout`. +- При смене расписания interval-check запускается после текущего in-flight run; cron-check ждёт следующий cron-slot. - Для изменения `agent_id`, `state_dir`, `server.*`, `metrics.*` нужен restart. ## Server env @@ -97,12 +108,14 @@ MONLET_EVENTS_RETENTION_MONTHS=36 MONLET_EVENTS_FUTURE_PARTITIONS=3 MONLET_PARTITION_MAINTENANCE_INTERVAL_SEC=3600 MONLET_OUTBOX_RETENTION_MAX_ROWS=50000 +MONLET_OUTBOX_PRUNE_BATCH_SIZE=5000 MONLET_EVENT_DEDUP_RETENTION_DAYS=30 MONLET_EVENT_DEDUP_PRUNE_BATCH_SIZE=5000 MONLET_MAX_PENDING_AGENTS=10000 MONLET_PENDING_AGENT_TTL_DAYS=7 +MONLET_PENDING_AGENT_PRUNE_BATCH_SIZE=1000 MONLET_ENABLE_NOTIFIER_WORKER=true MONLET_NOTIFIER_TICK_SEC=5 @@ -154,10 +167,12 @@ Liveness-детектор: - `MONLET_EVENTS_FUTURE_PARTITIONS` — сколько будущих месячных партиций создавать заранее (по умолчанию 3). - `MONLET_PARTITION_MAINTENANCE_INTERVAL_SEC` — период обслуживания партиций и prune (по умолчанию 3600). - `MONLET_OUTBOX_RETENTION_MAX_ROWS` — верхняя граница строк в outbox; лишние обрезаются (по умолчанию 50000). +- `MONLET_OUTBOX_PRUNE_BATCH_SIZE` — размер пакета при очистке terminal outbox-записей (по умолчанию 5000). - `MONLET_EVENT_DEDUP_RETENTION_DAYS` — хранение записей идемпотентности `event_id` (по умолчанию 30). - `MONLET_EVENT_DEDUP_PRUNE_BATCH_SIZE` — размер пакета при очистке dedup-записей (по умолчанию 5000). - `MONLET_MAX_PENDING_AGENTS` — лимит новых непринятых агентов (по умолчанию 10000). - `MONLET_PENDING_AGENT_TTL_DAYS` — очистка старых pending-ключей (по умолчанию 7). +- `MONLET_PENDING_AGENT_PRUNE_BATCH_SIZE` — размер пакета при очистке старых pending-ключей (по умолчанию 1000). Notifier worker: @@ -181,21 +196,29 @@ Notifier каналы: ## Web env -Веб-UI читает auth-настройки из env. Без них read-only страницы доступны локально, но admission-мутации возвращают `403`. +Веб-UI читает API/auth/timezone-настройки из env. Без auth-настроек read-only страницы доступны локально, но admission-мутации возвращают `403`. ```env +MONLET_API_BASE_URL=http://127.0.0.1:8000 +MONLET_API_TOKEN=replace-with-server-token +MONLET_WEB_TIME_ZONE=UTC MONLET_WEB_AUTH_USERNAME=admin MONLET_WEB_AUTH_PASSWORD=replace-with-strong-password MONLET_WEB_SESSION_SECRET=replace-with-random-32-bytes-min MONLET_WEB_SESSION_TTL_SEC=604800 +NEXT_PUBLIC_MONLET_POLL_MS=10000 # или вместо локального логина — доверие reverse-proxy: # MONLET_WEB_TRUST_PROXY_AUTH=true ``` +- `MONLET_API_BASE_URL` — базовый URL сервера для server-side fetch. +- `MONLET_API_TOKEN` — Bearer-токен для запросов UI к серверу; в браузер не передаётся. +- `MONLET_WEB_TIME_ZONE` — IANA timezone для отображения дат (по умолчанию `UTC`). - `MONLET_WEB_AUTH_USERNAME` / `MONLET_WEB_AUTH_PASSWORD` — учётные данные локального cookie-логина. - `MONLET_WEB_SESSION_SECRET` — секрет подписи сессии, минимум 32 байта; короче — UI считает auth неправильно настроенным. - `MONLET_WEB_SESSION_TTL_SEC` — срок жизни сессии (по умолчанию 604800 = 7 дней). - `MONLET_WEB_TRUST_PROXY_AUTH` — `true` включает доверие заголовку `X-Forwarded-User` от reverse-proxy. Несовместимо с локальным логином: одновременная настройка обоих режимов — ошибка (UI отвечает `500`). +- `NEXT_PUBLIC_MONLET_POLL_MS` — интервал браузерного polling в миллисекундах (по умолчанию 10000); для production-сборок Next.js это build-time переменная. ## Минимальная схема diff --git a/README.md b/README.md index f165997..9f9042b 100644 --- a/README.md +++ b/README.md @@ -110,8 +110,8 @@ The server owns incident and notification lifecycle. The agent sends facts only. The web UI is read-only for monitoring state; agent admission and blacklist controls are the v1 mutation surface. -Public API contracts live in `api/openapi.yaml`. Design and operations docs live -under `docs/`. +Public API contracts live in `api/openapi.yaml`. Design and operations docs start +at `docs/index.md`. ## Non-Goals @@ -134,6 +134,7 @@ Start with: 3. [server/README.md](server/README.md) 4. [web/README.md](web/README.md) 5. [api/README.md](api/README.md) +6. [docs/index.md](docs/index.md) Dependency caches and virtual environments are kept inside the repository. See the component READMEs for current local development commands. diff --git a/README.ru.md b/README.ru.md index 370a7c0..7038ffb 100644 --- a/README.ru.md +++ b/README.ru.md @@ -112,7 +112,7 @@ Server владеет lifecycle инцидентов и уведомлений. controls - mutation surface для v1. Публичный API contract лежит в `api/openapi.yaml`. Design и operations docs -лежат в `docs/`. +начинаются с `docs/index.md`. ## Не цели @@ -135,6 +135,7 @@ Monlet 0.1.0 намеренно не является: 3. [server/README.md](server/README.md) 4. [web/README.md](web/README.md) 5. [api/README.md](api/README.md) +6. [docs/index.md](docs/index.md) Dependency caches и virtual environments хранятся внутри репозитория. Актуальные команды local development описаны в component README. diff --git a/agent/README.md b/agent/README.md index 796af9b..103780a 100644 --- a/agent/README.md +++ b/agent/README.md @@ -14,7 +14,7 @@ Do not use `go install` or `go env -w`. Module cache, build cache, and tool bina ## Configuration -See `config.example.toml`. Required fields: top-level `state_dir`, at least one `[[checks]]` (`id`, `command`, `interval`, `timeout`), and at least one output enabled: `server.enabled = true` or `metrics.enabled = true`. `agent_id` is optional; if omitted, it defaults to the OS hostname. `hostname` is not a config key. +See `config.example.toml`. Required fields: top-level `state_dir`, at least one `[[checks]]` (`id`, `command`, `timeout`, and exactly one of `interval` or `cron`), and at least one output enabled: `server.enabled = true` or `metrics.enabled = true`. `agent_id` is optional; if omitted, it defaults to the OS hostname. `hostname` is not a config key. | Key | Default | Notes | |---|---|---| @@ -25,10 +25,12 @@ See `config.example.toml`. Required fields: top-level `state_dir`, at least one | `server.batch_interval` | `10s` | events flush cadence | | `metrics.listen` | `127.0.0.1:9465` | low-cardinality only | | `checks[].command` | required | Shell string executed as `/bin/sh -c`. Both forms accepted: `command = "check_disk --warn 80 --crit 90"` and triple-quoted `command = '''...'''`. Argv arrays are rejected. See [Command security contract](#command-security-contract). | +| `checks[].interval` | required unless `cron` is set | Go duration such as `60s`, `5m`, or `1h`. Interval checks run once immediately on start/reload, then on the interval. | +| `checks[].cron` | required unless `interval` is set | Standard 5-field cron expression, macros such as `@hourly`/`@daily`, and optional `CRON_TZ=...`. Seconds fields, `@every`, and `@reboot` are rejected. Cron checks wait for the next scheduled slot; they do not run immediately on start/reload. | | `checks[].notifications_enabled` | `true` | Set `false` for checks that must not create server notifications. | | `checks[].resource_limits` | `{}` | Optional per-check best-effort `ulimit` values: `cpu_time`, `memory`, `open_files`. | -Examples — one-line and multi-line forms: +Examples — interval, cron, and multi-line forms: ```toml [[checks]] @@ -37,6 +39,12 @@ command = "test $(cut -d. -f1 /proc/uptime) -gt 60" interval = "60s" timeout = "5s" +[[checks]] +id = "daily_backup" +command = "/usr/local/lib/monlet/check_backup_freshness.sh" +cron = "CRON_TZ=UTC 15 6 * * *" +timeout = "2m" + [[checks]] id = "disk_root" command = ''' @@ -81,16 +89,17 @@ executes it as `/bin/sh -c ` on the host where the agent runs. Send `SIGHUP` or run `systemctl reload monlet-agent` to reload the local TOML file. - Reload first validates the full new config. On error, the old config keeps running. -- Hot-reloadable: labels and checks, including `command`, `interval`, `timeout`, - `notifications_enabled`, `dedupe_key`, and `resource_limits`. +- Hot-reloadable: labels and checks, including `command`, `interval`, `cron`, + `timeout`, `notifications_enabled`, `dedupe_key`, and `resource_limits`. - Restart required: `agent_id`, `state_dir`, `server.*`, and `metrics.*`. - A check already running during reload is allowed to finish under its old config. - Removed or changed checks do not start new old-config runs; changed checks start - with the new config after the in-flight run finishes. + Removed or changed checks do not start new old-config runs. Changed interval + checks start with the new config after the in-flight run finishes; changed cron + checks wait for the next cron slot. ## Behaviour -- Per-check scheduler with no-overlap: a tick is skipped if the previous run is still in flight (`monlet_agent_check_skipped_total`). +- Per-check scheduler with no-overlap: a tick/cron slot is skipped if the previous run is still in flight (`monlet_agent_check_skipped_total`). - Check commands run through `/bin/sh -c`; the old argv-array form is intentionally unsupported. - Timeout via `exec.CommandContext` with `Setpgid` so the process group is killed on expiry; timeout → `critical`. - Exit code mapping: 0=ok, 1=warning, 2=critical, 3+=unknown (ADR-0005). @@ -115,7 +124,7 @@ Exposed on `metrics.listen` at `/metrics`. Labels are static (`check_id`, `statu - `monlet_agent_check_exit_code{check_id}` - `monlet_agent_check_last_run_timestamp_seconds{check_id}` - `monlet_agent_check_last_success_timestamp_seconds{check_id}` -- `monlet_agent_check_interval_seconds{check_id}` +- `monlet_agent_check_interval_seconds{check_id}` (interval checks only) - `monlet_agent_checks_configured` - `monlet_agent_config_load_success` - `monlet_agent_config_reload_total{result}` diff --git a/agent/config.example.toml b/agent/config.example.toml index 17342b8..91fbc55 100644 --- a/agent/config.example.toml +++ b/agent/config.example.toml @@ -32,3 +32,11 @@ name = "Uptime probe" command = "test $(cut -d. -f1 /proc/uptime) -gt 60" interval = "60s" timeout = "5s" + +# Cron-style schedules are also supported. Use exactly one of interval or cron. +# [[checks]] +# id = "daily_backup" +# name = "Daily backup freshness" +# command = "/usr/local/lib/monlet/check_backup_freshness.sh" +# cron = "CRON_TZ=UTC 15 6 * * *" +# timeout = "2m" diff --git a/agent/go.mod b/agent/go.mod index 442e49a..13ab7e1 100644 --- a/agent/go.mod +++ b/agent/go.mod @@ -6,6 +6,8 @@ require ( github.com/BurntSushi/toml v1.6.0 github.com/google/uuid v1.6.0 github.com/prometheus/client_golang v1.23.2 + github.com/prometheus/client_model v0.6.2 + github.com/robfig/cron/v3 v3.0.1 ) require ( @@ -13,7 +15,6 @@ require ( github.com/cespare/xxhash/v2 v2.3.0 // indirect github.com/kr/text v0.2.0 // indirect github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect - github.com/prometheus/client_model v0.6.2 // indirect github.com/prometheus/common v0.66.1 // indirect github.com/prometheus/procfs v0.16.1 // indirect go.yaml.in/yaml/v2 v2.4.2 // indirect diff --git a/agent/go.sum b/agent/go.sum index 6f54a97..1ad6eb8 100644 --- a/agent/go.sum +++ b/agent/go.sum @@ -31,6 +31,8 @@ github.com/prometheus/common v0.66.1 h1:h5E0h5/Y8niHc5DlaLlWLArTQI7tMrsfQjHV+d9Z github.com/prometheus/common v0.66.1/go.mod h1:gcaUsgf3KfRSwHY4dIMXLPV0K/Wg1oZ8+SbZk/HH/dA= github.com/prometheus/procfs v0.16.1 h1:hZ15bTNuirocR6u0JZ6BAHHmwS1p8B4P6MRqxtzMyRg= github.com/prometheus/procfs v0.16.1/go.mod h1:teAbpZRB1iIAJYREa1LsoWUXykVXA1KlTmWl8x/U+Is= +github.com/robfig/cron/v3 v3.0.1 h1:WdRxkvbJztn8LMz/QEvLN5sBU+xKpSqwwUO1Pjr4qDs= +github.com/robfig/cron/v3 v3.0.1/go.mod h1:eQICP3HwyT7UooqI/z+Ov+PtYAWygg1TEWWzGIFLtro= github.com/rogpeppe/go-internal v1.10.0 h1:TMyTOH3F/DB16zRVcYyreMH6GnZZrwQVAoYjRBZyWFQ= github.com/rogpeppe/go-internal v1.10.0/go.mod h1:UQnix2H7Ngw/k4C5ijL5+65zddjncjaFoBhdsK/akog= github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U= diff --git a/agent/internal/app/app.go b/agent/internal/app/app.go index 1cb5893..00debc0 100644 --- a/agent/internal/app/app.go +++ b/agent/internal/app/app.go @@ -47,9 +47,7 @@ func (a *App) Run(ctx context.Context) error { a.Metrics.ConfigLoadSuccess.Set(1) a.Metrics.ChecksConfigured.Set(float64(len(cfg.Checks))) a.Metrics.BuildInfo.WithLabelValues(a.Version, boolLabel(cfg.PushesToServer()), boolLabel(cfg.ExposesMetrics())).Set(1) - for _, c := range cfg.Checks { - a.Metrics.CheckInterval.WithLabelValues(c.ID).Set(c.Interval.Duration.Seconds()) - } + a.recordCheckScheduleMetrics(cfg.Checks) var wg sync.WaitGroup @@ -256,9 +254,7 @@ func (a *App) Reload(newCfg *config.Config, newAgentID string) error { a.Metrics.ConfigReloadTotal.WithLabelValues("success").Inc() a.Metrics.ConfigLastReloadSuccess.Set(float64(time.Now().Unix())) a.Metrics.ChecksConfigured.Set(float64(len(newCfg.Checks))) - for _, c := range newCfg.Checks { - a.Metrics.CheckInterval.WithLabelValues(c.ID).Set(c.Interval.Duration.Seconds()) - } + a.recordCheckScheduleMetrics(newCfg.Checks) a.schedulerMu.RLock() mgr := a.scheduler @@ -275,6 +271,16 @@ func (a *App) RecordReloadFailure() { a.Metrics.ConfigReloadTotal.WithLabelValues("failure").Inc() } +func (a *App) recordCheckScheduleMetrics(checks []config.CheckConfig) { + for _, c := range checks { + if c.UsesInterval() { + a.Metrics.CheckInterval.WithLabelValues(c.ID).Set(c.Interval.Duration.Seconds()) + } else { + a.Metrics.CheckInterval.DeleteLabelValues(c.ID) + } + } +} + func (a *App) config() *config.Config { a.cfgMu.RLock() defer a.cfgMu.RUnlock() diff --git a/agent/internal/app/app_test.go b/agent/internal/app/app_test.go index 31b3297..e3bba62 100644 --- a/agent/internal/app/app_test.go +++ b/agent/internal/app/app_test.go @@ -17,6 +17,7 @@ import ( "github.com/monlet/agent/internal/config" "github.com/monlet/agent/internal/metrics" "github.com/monlet/agent/internal/spool" + dto "github.com/prometheus/client_model/go" ) func TestPrometheusOnlyDoesNotPush(t *testing.T) { @@ -257,6 +258,92 @@ func TestReloadAcceptsLabelsAndChecks(t *testing.T) { } } +func TestReloadUpdatesIntervalMetricForCron(t *testing.T) { + oldCfg := &config.Config{ + AgentID: "a1", + Hostname: "h", + StateDir: "/tmp/old", + Server: config.ServerConfig{ + Enabled: boolPtr(true), + URL: "http://server", + HeartbeatInterval: config.Duration{Duration: 10 * time.Second}, + BatchInterval: config.Duration{Duration: 10 * time.Second}, + }, + Metrics: config.MetricsConfig{Enabled: false}, + Checks: []config.CheckConfig{{ + ID: "c1", + Command: "true", + Interval: config.Duration{Duration: time.Second}, + Timeout: config.Duration{Duration: time.Second}, + }}, + } + a := &App{ + Cfg: oldCfg, + AgentID: "a1", + Metrics: metrics.New(), + } + a.recordCheckScheduleMetrics(oldCfg.Checks) + if got, ok := checkIntervalMetric(t, a, "c1"); !ok || got != 1 { + t.Fatalf("interval metric missing before reload: value=%v ok=%v", got, ok) + } + + cronCfg := *oldCfg + cronCfg.Checks = []config.CheckConfig{{ + ID: "c1", + Command: "true", + Cron: "0 * * * *", + Timeout: config.Duration{Duration: time.Second}, + }} + if err := a.Reload(&cronCfg, "a1"); err != nil { + t.Fatal(err) + } + if got, ok := checkIntervalMetric(t, a, "c1"); ok { + t.Fatalf("interval metric must be deleted for cron check, got %v", got) + } + + intervalCfg := cronCfg + intervalCfg.Checks = []config.CheckConfig{{ + ID: "c1", + Command: "true", + Interval: config.Duration{Duration: 2 * time.Second}, + Timeout: config.Duration{Duration: time.Second}, + }} + if err := a.Reload(&intervalCfg, "a1"); err != nil { + t.Fatal(err) + } + if got, ok := checkIntervalMetric(t, a, "c1"); !ok || got != 2 { + t.Fatalf("interval metric missing after interval reload: value=%v ok=%v", got, ok) + } +} + +func checkIntervalMetric(t *testing.T, a *App, checkID string) (float64, bool) { + t.Helper() + families, err := a.Metrics.Registry.Gather() + if err != nil { + t.Fatal(err) + } + for _, family := range families { + if family.GetName() != "monlet_agent_check_interval_seconds" { + continue + } + for _, metric := range family.GetMetric() { + if labelValue(metric.GetLabel(), "check_id") == checkID && metric.GetGauge() != nil { + return metric.GetGauge().GetValue(), true + } + } + } + return 0, false +} + +func labelValue(labels []*dto.LabelPair, name string) string { + for _, label := range labels { + if label.GetName() == name { + return label.GetValue() + } + } + return "" +} + func boolPtr(v bool) *bool { return &v } diff --git a/agent/internal/config/config.go b/agent/internal/config/config.go index 8f47e74..722b57c 100644 --- a/agent/internal/config/config.go +++ b/agent/internal/config/config.go @@ -11,6 +11,7 @@ import ( "unicode/utf8" "github.com/BurntSushi/toml" + "github.com/robfig/cron/v3" ) var idPattern = regexp.MustCompile(`^[A-Za-z0-9._:-]+$`) @@ -58,6 +59,7 @@ type CheckConfig struct { Name string `toml:"name"` Command string `toml:"command"` Interval Duration `toml:"interval"` + Cron string `toml:"cron"` Timeout Duration `toml:"timeout"` NotificationsEnabled *bool `toml:"notifications_enabled"` DedupeKey string `toml:"dedupe_key"` @@ -211,14 +213,24 @@ func (c *Config) Validate() error { if strings.TrimSpace(ch.Command) == "" { return fmt.Errorf("check %q: command is required", ch.ID) } - if ch.Interval.Duration <= 0 { - return fmt.Errorf("check %q: interval must be > 0", ch.ID) - } if ch.Timeout.Duration <= 0 { return fmt.Errorf("check %q: timeout must be > 0", ch.ID) } - if ch.Timeout.Duration > ch.Interval.Duration { + if ch.Interval.Duration < 0 { + return fmt.Errorf("check %q: interval must be > 0", ch.ID) + } + ch.Cron = strings.TrimSpace(ch.Cron) + hasInterval := ch.Interval.Duration > 0 + hasCron := ch.Cron != "" + switch { + case hasInterval == hasCron: + return fmt.Errorf("check %q: exactly one of interval or cron is required", ch.ID) + case hasInterval && ch.Timeout.Duration > ch.Interval.Duration: return fmt.Errorf("check %q: timeout must be <= interval", ch.ID) + case hasCron: + if _, err := ParseCronSchedule(ch.Cron); err != nil { + return fmt.Errorf("check %q: invalid cron: %w", ch.ID, err) + } } if len(ch.DedupeKey) > 256 { return fmt.Errorf("check %q: dedupe_key too long", ch.ID) @@ -253,6 +265,42 @@ func (c CheckConfig) NotificationsOn() bool { return c.NotificationsEnabled == nil || *c.NotificationsEnabled } +func (c CheckConfig) UsesInterval() bool { + return c.Interval.Duration > 0 +} + +func (c CheckConfig) UsesCron() bool { + return strings.TrimSpace(c.Cron) != "" +} + +func ParseCronSchedule(spec string) (cron.Schedule, error) { + spec = strings.TrimSpace(spec) + if spec == "" { + return nil, fmt.Errorf("cron is empty") + } + withoutTZ := specWithoutTZ(spec) + lower := strings.ToLower(withoutTZ) + if strings.HasPrefix(lower, "@every") { + return nil, fmt.Errorf("@every is not supported; use interval") + } + if lower == "@reboot" { + return nil, fmt.Errorf("@reboot is not supported") + } + return cron.ParseStandard(spec) +} + +func specWithoutTZ(spec string) string { + fields := strings.Fields(spec) + if len(fields) == 0 { + return "" + } + first := fields[0] + if strings.HasPrefix(first, "CRON_TZ=") || strings.HasPrefix(first, "TZ=") { + return strings.ToLower(strings.Join(fields[1:], " ")) + } + return strings.ToLower(strings.Join(fields, " ")) +} + func validateResourceLimits(checkID string, limits ResourceLimits) error { if limits.CPUTime.Duration < 0 { return fmt.Errorf("check %q: resource_limits.cpu_time must be > 0", checkID) diff --git a/agent/internal/config/config_test.go b/agent/internal/config/config_test.go index 07dcb94..3535a73 100644 --- a/agent/internal/config/config_test.go +++ b/agent/internal/config/config_test.go @@ -48,9 +48,84 @@ func TestLoadMinimal(t *testing.T) { } } +func TestLoadCronCheck(t *testing.T) { + c, err := Load(writeTmp(t, ` + agent_id = "host-1" + state_dir = "/tmp/x" + + [server] + enabled = true + url = "http://localhost" + + [[checks]] + id = "c1" + command = "true" + cron = "CRON_TZ=UTC 0 6 * * *" + timeout = "5m" + `)) + if err != nil { + t.Fatal(err) + } + if !c.Checks[0].UsesCron() || c.Checks[0].UsesInterval() { + t.Fatalf("unexpected schedule mode: %+v", c.Checks[0]) + } +} + +func TestValidateCheckScheduleChoice(t *testing.T) { + for name, body := range map[string]string{ + "both_interval_and_cron": ` + interval = "10s" + cron = "0 * * * *" + timeout = "5s" + `, + "neither_interval_nor_cron": ` + timeout = "5s" + `, + "invalid_cron": ` + cron = "not a cron" + timeout = "5s" + `, + "seconds_cron": ` + cron = "0 0 * * * *" + timeout = "5s" + `, + "every_cron": ` + cron = "@every 10s" + timeout = "5s" + `, + "tz_every_cron": ` + cron = "CRON_TZ=UTC @every 10s" + timeout = "5s" + `, + "reboot_cron": ` + cron = "@reboot" + timeout = "5s" + `, + } { + t.Run(name, func(t *testing.T) { + _, err := Load(writeTmp(t, ` + agent_id = "host-1" + state_dir = "/tmp/x" + + [server] + enabled = true + url = "http://localhost" + + [[checks]] + id = "c1" + command = "true" + `+body+` + `)) + if err == nil { + t.Fatal("expected schedule validation error") + } + }) + } +} + func TestLoadResourceLimits(t *testing.T) { c, err := Load(writeTmp(t, ` -agent_id = "host-1" + agent_id = "host-1" state_dir = "/tmp/x" [server] diff --git a/agent/internal/scheduler/scheduler.go b/agent/internal/scheduler/scheduler.go index c4360b0..76fbb7e 100644 --- a/agent/internal/scheduler/scheduler.go +++ b/agent/internal/scheduler/scheduler.go @@ -11,6 +11,8 @@ import ( "github.com/monlet/agent/internal/runner" ) +var parseCronSchedule = config.ParseCronSchedule + // Event is a contract-shaped check result emitted by the scheduler. type Event struct { EventID string `json:"event_id"` @@ -207,8 +209,10 @@ func (w *checkWorker) loop(initial config.CheckConfig) { w.wg.Done() }() cfg := initial - t := time.NewTicker(cfg.Interval.Duration) + t := time.NewTimer(time.Hour) + stopTimer(t) defer t.Stop() + var timerCh <-chan time.Time running := false stopping := false @@ -230,7 +234,26 @@ func (w *checkWorker) loop(initial config.CheckConfig) { }() } - start(cfg) + arm := func(c config.CheckConfig, immediateInterval bool) bool { + delay, err := nextDelay(c, time.Now(), immediateInterval) + if err != nil { + return false + } + resetTimer(t, delay) + timerCh = t.C + return true + } + disarm := func() { + stopTimer(t) + timerCh = nil + } + + if !arm(cfg, true) { + if w.hooks.OnStopped != nil { + w.hooks.OnStopped(cfg.ID) + } + return + } for { select { @@ -257,11 +280,21 @@ func (w *checkWorker) loop(initial config.CheckConfig) { continue } cfg = next - t.Reset(cfg.Interval.Duration) + if cfg.UsesCron() { + runAfterCurrent = false + if !arm(cfg, false) { + return + } + continue + } if running { runAfterCurrent = true + disarm() } else { start(cfg) + if !arm(cfg, false) { + return + } } case <-w.doneCh: running = false @@ -281,26 +314,35 @@ func (w *checkWorker) loop(initial config.CheckConfig) { } if runAfterCurrent { runAfterCurrent = false + startAfterDrain := true for { select { case next := <-w.updateCh: if !reflect.DeepEqual(cfg, next) { cfg = next - t.Reset(cfg.Interval.Duration) + startAfterDrain = !cfg.UsesCron() } default: - start(cfg) + if startAfterDrain { + start(cfg) + } + if !arm(cfg, false) { + return + } goto nextLoop } } nextLoop: continue } - case <-t.C: + case <-timerCh: if stopping { continue } start(cfg) + if !arm(cfg, false) { + return + } } } } @@ -358,3 +400,41 @@ func incidentKey(agentID string, c config.CheckConfig) string { } return agentID + ":" + c.ID } + +func nextDelay(c config.CheckConfig, now time.Time, immediateInterval bool) (time.Duration, error) { + if c.UsesCron() { + schedule, err := parseCronSchedule(c.Cron) + if err != nil { + return 0, err + } + return delayUntil(now, schedule.Next(now)), nil + } + if immediateInterval { + return 0, nil + } + return c.Interval.Duration, nil +} + +func delayUntil(now, next time.Time) time.Duration { + if next.Before(now) { + return 0 + } + return next.Sub(now) +} + +func resetTimer(t *time.Timer, d time.Duration) { + if d < 0 { + d = 0 + } + stopTimer(t) + t.Reset(d) +} + +func stopTimer(t *time.Timer) { + if !t.Stop() { + select { + case <-t.C: + default: + } + } +} diff --git a/agent/internal/scheduler/scheduler_test.go b/agent/internal/scheduler/scheduler_test.go index d47fcf9..f9658ad 100644 --- a/agent/internal/scheduler/scheduler_test.go +++ b/agent/internal/scheduler/scheduler_test.go @@ -8,6 +8,7 @@ import ( "time" "github.com/monlet/agent/internal/config" + "github.com/robfig/cron/v3" ) func runScheduler(ctx context.Context, agentID string, checks []config.CheckConfig, out chan<- Event, h Hooks) { @@ -27,6 +28,32 @@ func mkCheck(id, cmd string, interval, timeout time.Duration) config.CheckConfig } } +type fakeCronSchedule struct { + delay time.Duration +} + +func (s fakeCronSchedule) Next(t time.Time) time.Time { + return t.Add(s.delay) +} + +func withFakeCron(t *testing.T, delay time.Duration) { + t.Helper() + old := parseCronSchedule + parseCronSchedule = func(string) (cron.Schedule, error) { + return fakeCronSchedule{delay: delay}, nil + } + t.Cleanup(func() { parseCronSchedule = old }) +} + +func mkCronCheck(id, cmd string, timeout time.Duration) config.CheckConfig { + return config.CheckConfig{ + ID: id, + Command: cmd, + Cron: "0 * * * *", + Timeout: config.Duration{Duration: timeout}, + } +} + func TestEmitsEvents(t *testing.T) { ctx, cancel := context.WithTimeout(context.Background(), 500*time.Millisecond) defer cancel() @@ -45,6 +72,41 @@ func TestEmitsEvents(t *testing.T) { } } +func TestCronDoesNotRunImmediately(t *testing.T) { + withFakeCron(t, 200*time.Millisecond) + ctx, cancel := context.WithTimeout(context.Background(), 100*time.Millisecond) + defer cancel() + out := make(chan Event, 10) + m := NewManager(ctx, "a1", out, Hooks{}) + m.Update([]config.CheckConfig{mkCronCheck("cron", "echo cron", 100*time.Millisecond)}) + select { + case ev := <-out: + t.Fatalf("cron check ran before next slot: %+v", ev) + case <-ctx.Done(): + } + m.Stop() + m.Wait() +} + +func TestCronEmitsOnDueSlot(t *testing.T) { + withFakeCron(t, 30*time.Millisecond) + ctx, cancel := context.WithTimeout(context.Background(), time.Second) + defer cancel() + out := make(chan Event, 10) + m := NewManager(ctx, "a1", out, Hooks{}) + m.Update([]config.CheckConfig{mkCronCheck("cron", "echo cron", 100*time.Millisecond)}) + select { + case ev := <-out: + if ev.CheckID != "cron" || ev.Status != "ok" { + t.Fatalf("unexpected event: %+v", ev) + } + case <-ctx.Done(): + t.Fatal("no cron event") + } + m.Stop() + m.Wait() +} + func TestNoOverlapSkips(t *testing.T) { ctx, cancel := context.WithTimeout(context.Background(), 600*time.Millisecond) defer cancel() @@ -59,6 +121,23 @@ func TestNoOverlapSkips(t *testing.T) { } } +func TestCronNoOverlapSkips(t *testing.T) { + withFakeCron(t, 50*time.Millisecond) + ctx, cancel := context.WithTimeout(context.Background(), 250*time.Millisecond) + defer cancel() + out := make(chan Event, 10) + var skipped int32 + hooks := Hooks{OnSkipped: func(string) { atomic.AddInt32(&skipped, 1) }} + m := NewManager(ctx, "a1", out, hooks) + m.Update([]config.CheckConfig{mkCronCheck("cron", "sleep 0.2", 500*time.Millisecond)}) + <-ctx.Done() + m.Stop() + m.Wait() + if atomic.LoadInt32(&skipped) == 0 { + t.Fatal("expected at least one skipped cron slot") + } +} + func TestReportsFullEventChannel(t *testing.T) { ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second) defer cancel() diff --git a/deploy/alertmanager/README.md b/deploy/alertmanager/README.md index cec8e69..d2184e1 100644 --- a/deploy/alertmanager/README.md +++ b/deploy/alertmanager/README.md @@ -1,9 +1,18 @@ # Alertmanager -Placeholder for optional Alertmanager integration examples. +Example Alertmanager configuration for the optional observability profile. -Planned examples: +`alertmanager.yml` is intentionally safe by default: it routes to a `null` +receiver, so alerts are grouped but dropped. Before using it outside local +development, add a real receiver and set `route.receiver` to that receiver. -- receiver config for Monlet-originated alerts; -- stable label conventions; -- routing notes for avoiding duplicate notifications. +Monlet can also send notifications to Alertmanager through the server notifier: + +```env +MONLET_NOTIFIER_ALERTMANAGER_ENABLED=true +MONLET_NOTIFIER_ALERTMANAGER_URL=http://alertmanager:9093 +``` + +Keep labels stable and low-cardinality. The example groups by `alertname`, +`agent_id`, and `check_id`; do not add timestamps, event IDs, or output text to +Alertmanager labels. diff --git a/deploy/grafana/README.md b/deploy/grafana/README.md index c7fb723..72c525f 100644 --- a/deploy/grafana/README.md +++ b/deploy/grafana/README.md @@ -1,11 +1,20 @@ # Grafana -Placeholder for Grafana dashboard examples. +Starter Grafana provisioning for the optional observability profile. -Planned dashboards: +Files: -- agents alive/stale/dead; -- check state counts; -- open incidents; -- notification outbox state; -- server ingestion and notifier failures. +- `provisioning/datasources/prometheus.yml` - adds Prometheus at `http://prometheus:9090`. +- `provisioning/dashboards/monlet.yml` - loads dashboards from `/var/lib/grafana/dashboards`. +- `dashboards/monlet-overview.json` - starter dashboard for agent status, open incidents, event ingest rate, and notification rate. + +Run it with: + +```sh +cd deploy +MONLET_AUTH_TOKEN=$(openssl rand -hex 16) docker compose --profile observability up +``` + +Grafana is exposed at . The local default is +`admin` / `admin` unless `GF_ADMIN_PASSWORD` is set. Change it before exposing +Grafana outside local development. diff --git a/deploy/prometheus/README.md b/deploy/prometheus/README.md index b29e014..1c1028f 100644 --- a/deploy/prometheus/README.md +++ b/deploy/prometheus/README.md @@ -1,9 +1,20 @@ # Prometheus -Placeholder for Prometheus examples. +Example scrape configuration for the optional observability profile in +`deploy/docker-compose.yml`. -Planned examples: +Run it with: -- scrape Monlet agent `/metrics`; -- scrape Monlet server `/metrics`; -- sample rules for checks that use `notifications_enabled = false`. +```sh +cd deploy +MONLET_AUTH_TOKEN=$(openssl rand -hex 16) docker compose --profile observability up +``` + +The example config scrapes: + +- `server:8000/metrics` as `monlet-server` +- `host.docker.internal:9100/metrics` as `monlet-agent` + +Adjust the agent target for your environment. In the showcase stack one demo +agent exposes metrics on host port `9465`; production agents usually expose +`metrics.listen` only on the host or a private monitoring network. diff --git a/docs/index.md b/docs/index.md new file mode 100644 index 0000000..036756c --- /dev/null +++ b/docs/index.md @@ -0,0 +1,16 @@ +# Monlet Docs + +This directory contains operator-facing docs that do not fit in the root or component READMEs. + +## Start Here + +- [First deployment checklist](ops/deployment-checklist.md) - step-by-step production-ish stand-up checklist. +- [Backup and restore](ops/backup-restore.md) - PostgreSQL backup, retention, and disaster recovery notes. +- [Token rotation](ops/token-rotation.md) - UI token overlap and agent-key rotation. +- [Deployment notes](ops/deployment.md) - deployment shape and production boundaries. + +## Conventions + +- Component setup lives in `agent/README.md`, `server/README.md`, `web/README.md`, and `deploy/README.md`. +- API contracts live in `api/openapi.yaml`; `api/README.md` only describes contract rules. +- Keep docs factual and current. Avoid placeholder docs when a real config or script already exists. diff --git a/docs/operations/deployment.md b/docs/operations/deployment.md deleted file mode 100644 index b60443e..0000000 --- a/docs/operations/deployment.md +++ /dev/null @@ -1,68 +0,0 @@ -# Deployment - -This is an initial deployment plan, not a production runbook. - -## Agent - -Agent runs as a systemd service. - -Expected paths: - -- binary: `/usr/local/bin/monlet-agent`; -- config: `/etc/monlet/agent.toml`; -- state: `/var/lib/monlet-agent`; -- logs: journald. - -The agent should support graceful shutdown and avoid overlapping runs for the same check. - -## Server - -Server runs in Docker. - -Production storage: - -- PostgreSQL. - -Local/demo storage: - -- SQLite may be supported later, but must be documented as non-production. - -Required deployment features: - -- TLS-ready reverse proxy; -- agent API tokens; -- request size limits; -- structured logs; -- `/health`; -- `/ready`; -- `/metrics`. - -Docker image rule: - -- build server dependencies inside the image; -- do not require Python packages installed on the host; -- do not mount host virtualenvs into the container; -- local non-Docker development uses `server/.venv` only. - -## Web - -Web is read-only for monitoring state in v1. Agent admission and blacklist controls are the explicit write exception. - -Auth plan: - -- local web login or equivalent gateway auth through a trusted reverse proxy; -- no full RBAC in v1. - -## Observability - -Optional examples: - -- Prometheus scrape config; -- Alertmanager route config; -- Grafana dashboard stub. - -Metrics must avoid high-cardinality labels. - -## Backups - -PostgreSQL backup/restore docs are required before first production use. diff --git a/docs/ops/deployment.md b/docs/ops/deployment.md new file mode 100644 index 0000000..361be18 --- /dev/null +++ b/docs/ops/deployment.md @@ -0,0 +1,66 @@ +# Deployment Notes + +These notes describe the intended production shape. Use +[`docs/ops/deployment-checklist.md`](deployment-checklist.md) for the +step-by-step first stand-up checklist. + +## Agent + +The agent runs on monitored hosts as a systemd service. + +Expected paths: + +- binary: `/usr/local/bin/monlet-agent` +- config: `/etc/monlet/agent.toml` +- state: `/var/lib/monlet-agent` +- logs: journald + +The agent reads only local config. It does not accept remote command execution or +remote config push from the server. Per-check scheduling is local to the agent, +and overlapping runs for the same check are skipped. + +## Server + +The server runs as a containerized FastAPI application backed by PostgreSQL 16+. +It owns inventory, current check state, incidents, notification outbox state, +retention, and public `/api/v1` contracts. + +Required production boundaries: + +- TLS and rate limiting at a reverse proxy +- PostgreSQL as canonical storage +- `MONLET_AUTH_TOKEN` stored in a secret manager +- request body limits enabled +- structured logs with auth material redacted +- `/api/v1/health`, `/api/v1/ready`, and `/metrics` exposed for operations + +Docker images install dependencies inside the image. Do not mount host virtual +environments into server containers. + +## Web + +The web app is an operator dashboard. Monitoring pages are read-only; agent +admission and blacklist controls are the explicit v1 write surface. + +Protect the UI with either local web login or a trusted auth proxy. If local +login is enabled, rate-limit `/login` at the reverse proxy. Monlet v1 does not +provide full RBAC. + +## Observability + +The local deploy stack includes optional Prometheus, Alertmanager, and Grafana +examples under `deploy/`. Production deployments should at minimum scrape: + +- Monlet server `/metrics` +- each agent `/metrics` +- reverse-proxy and PostgreSQL metrics from the surrounding environment + +Metric labels must stay low-cardinality. Do not put timestamps, request IDs, or +unbounded host-local values into check IDs or labels. + +## Backups + +PostgreSQL is the canonical state. Follow +[`docs/ops/backup-restore.md`](backup-restore.md) before first production +use. Agent spools are best-effort buffers and are not a substitute for database +backups. diff --git a/examples/README.md b/examples/README.md index bc3723a..5a6a868 100644 --- a/examples/README.md +++ b/examples/README.md @@ -1,3 +1,12 @@ # Examples -Examples for checks, configs, and local demos. +Small payloads and scripts used by local smoke tests and manual API checks. + +Files: + +- `heartbeat.json` - example agent heartbeat request body. +- `event-batch.json` - example event batch request body. +- `checks/example_check.sh` - minimal shell check that follows the Monlet exit-code contract. + +Use these examples as templates only. Real agents normally send heartbeat and +event payloads through the Go agent, not by hand-written curl requests. diff --git a/server/.env.example b/server/.env.example index 9a3367a..1cc7da7 100644 --- a/server/.env.example +++ b/server/.env.example @@ -1,25 +1,47 @@ MONLET_AUTH_TOKEN=replace-me-with-random-token MONLET_DATABASE_URL=postgresql+asyncpg://monlet:monlet@localhost:5432/monlet MONLET_LOG_LEVEL=INFO +MONLET_HOST=0.0.0.0 +MONLET_PORT=8000 + MONLET_BODY_LIMIT_BYTES=1048576 MONLET_OUTPUT_MAX_BYTES=8192 +MONLET_MAX_BATCH_EVENTS=200 + +MONLET_ENABLE_DETECTOR=true MONLET_STALE_AFTER_SEC=20 MONLET_DEAD_AFTER_SEC=30 MONLET_DETECTOR_TICK_SEC=5 -MONLET_ENABLE_DETECTOR=true + +MONLET_DB_POOL_SIZE=10 +MONLET_DB_MAX_OVERFLOW=10 +MONLET_DB_POOL_TIMEOUT_SEC=30 +MONLET_DB_POOL_RECYCLE_SEC=1800 + MONLET_EVENTS_RETENTION_MONTHS=36 +MONLET_EVENTS_FUTURE_PARTITIONS=3 +MONLET_PARTITION_MAINTENANCE_INTERVAL_SEC=3600 +MONLET_OUTBOX_RETENTION_MAX_ROWS=50000 +MONLET_OUTBOX_PRUNE_BATCH_SIZE=5000 + +MONLET_EVENT_DEDUP_RETENTION_DAYS=30 +MONLET_EVENT_DEDUP_PRUNE_BATCH_SIZE=5000 + MONLET_MAX_PENDING_AGENTS=10000 MONLET_PENDING_AGENT_TTL_DAYS=7 MONLET_PENDING_AGENT_PRUNE_BATCH_SIZE=1000 -MONLET_OUTBOX_RETENTION_MAX_ROWS=50000 MONLET_ENABLE_NOTIFIER_WORKER=true MONLET_NOTIFIER_TICK_SEC=5 MONLET_NOTIFIER_BATCH_SIZE=20 +MONLET_NOTIFIER_CONCURRENCY=4 MONLET_NOTIFIER_MAX_ATTEMPTS=8 MONLET_NOTIFIER_LEASE_SEC=300 MONLET_NOTIFIER_HTTP_TIMEOUT_SEC=10 +MONLET_NOTIFIER_INCLUDE_OUTPUT=true +MONLET_NOTIFIER_MAX_OUTPUT_BYTES=1024 MONLET_NOTIFICATION_TIME_ZONE=UTC + MONLET_NOTIFIER_DEBUG_ENABLED=true MONLET_NOTIFIER_TELEGRAM_ENABLED=false MONLET_NOTIFIER_TELEGRAM_TOKEN= diff --git a/server/README.md b/server/README.md index 1c91e9f..b873cb1 100644 --- a/server/README.md +++ b/server/README.md @@ -33,26 +33,38 @@ UV_PROJECT_ENVIRONMENT=.venv UV_CACHE_DIR=../.cache/uv uv run fastapi dev monlet | `MONLET_AUTH_TOKEN` | required | UI/operator Bearer token. Space/comma-separated list; any accepted. `changeme` is rejected. | | `MONLET_DATABASE_URL` | `postgresql+asyncpg://monlet:monlet@127.0.0.1:5432/monlet` | Async DSN | | `MONLET_LOG_LEVEL` | `INFO` | Log level | +| `MONLET_HOST` | `0.0.0.0` | HTTP bind host for direct server runs | +| `MONLET_PORT` | `8000` | HTTP bind port for direct server runs | | `MONLET_BODY_LIMIT_BYTES` | `1048576` | Request body limit (ADR-0005) | | `MONLET_OUTPUT_MAX_BYTES` | `8192` | Per-event output limit | +| `MONLET_MAX_BATCH_EVENTS` | `200` | Maximum events per ingestion batch | | `MONLET_STALE_AFTER_SEC` | `20` | Stale threshold | | `MONLET_DEAD_AFTER_SEC` | `30` | Dead threshold | | `MONLET_DETECTOR_TICK_SEC` | `5` | Detector tick | | `MONLET_ENABLE_DETECTOR` | `true` | Toggle detector task | +| `MONLET_DB_POOL_SIZE` | `10` | Base async DB pool size | +| `MONLET_DB_MAX_OVERFLOW` | `10` | Extra DB connections above pool size | +| `MONLET_DB_POOL_TIMEOUT_SEC` | `30` | Seconds to wait for a DB connection | +| `MONLET_DB_POOL_RECYCLE_SEC` | `1800` | Recycle DB connections by age | | `MONLET_EVENTS_RETENTION_MONTHS` | `36` | Months of `events` partitions to keep; `0` disables drop | | `MONLET_EVENTS_FUTURE_PARTITIONS` | `3` | Future month partitions to pre-create | | `MONLET_EVENT_DEDUP_RETENTION_DAYS` | `30` | Idempotency window for `event_id` | +| `MONLET_EVENT_DEDUP_PRUNE_BATCH_SIZE` | `5000` | Dedup rows pruned per maintenance tick | | `MONLET_MAX_PENDING_AGENTS` | `10000` | Maximum pending agent keys before heartbeat rejects with 429 | | `MONLET_PENDING_AGENT_TTL_DAYS` | `7` | Pending key cleanup TTL; `0` disables pruning | | `MONLET_PENDING_AGENT_PRUNE_BATCH_SIZE` | `1000` | Pending cleanup rows per detector tick | | `MONLET_PARTITION_MAINTENANCE_INTERVAL_SEC` | `3600` | Partition create/drop worker interval | | `MONLET_OUTBOX_RETENTION_MAX_ROWS` | `50000` | Maximum retained terminal outbox rows; `0` disables pruning | +| `MONLET_OUTBOX_PRUNE_BATCH_SIZE` | `5000` | Outbox rows pruned per maintenance tick | | `MONLET_ENABLE_NOTIFIER_WORKER` | `true` | Toggle outbox worker | | `MONLET_NOTIFIER_TICK_SEC` | `5` | Worker poll interval | | `MONLET_NOTIFIER_BATCH_SIZE` | `20` | Rows claimed per tick | +| `MONLET_NOTIFIER_CONCURRENCY` | `4` | Parallel notification deliveries per tick | | `MONLET_NOTIFIER_MAX_ATTEMPTS` | `8` | Terminal-failure threshold (ADR-0005) | | `MONLET_NOTIFIER_LEASE_SEC` | `300` | Recovery lease for stuck `sending` rows | | `MONLET_NOTIFIER_HTTP_TIMEOUT_SEC` | `10` | Outbound HTTP timeout | +| `MONLET_NOTIFIER_INCLUDE_OUTPUT` | `true` | Include redacted check output in notifications | +| `MONLET_NOTIFIER_MAX_OUTPUT_BYTES` | `1024` | Maximum notification output bytes after redaction | | `MONLET_NOTIFICATION_TIME_ZONE` | `UTC` | IANA timezone for human-readable notification timestamps | | `MONLET_NOTIFIER_DEBUG_ENABLED` | `true` | Log-only notifier | | `MONLET_NOTIFIER_TELEGRAM_ENABLED` | `false` | Telegram notifier on/off | diff --git a/web/README.md b/web/README.md index 570a6b2..c161ec2 100644 --- a/web/README.md +++ b/web/README.md @@ -41,6 +41,7 @@ Do not install Node packages globally. Project dependencies live in `web/node_mo | `MONLET_WEB_SESSION_SECRET` | _(none)_ | Secret used to sign the web session cookie; required with local web login; at least 32 bytes | | `MONLET_WEB_SESSION_TTL_SEC` | `604800` | Local web session lifetime | | `MONLET_WEB_TRUST_PROXY_AUTH` | `false` | Trust `X-Forwarded-User` from an upstream auth proxy | +| `NEXT_PUBLIC_MONLET_POLL_MS` | `10000` | Browser polling interval in ms; build-time for production Next.js bundles | The API token never reaches the browser — all server calls happen in Server Components / route handlers. Agent admission actions are blocked unless local web login is configured or trusted proxy auth is enabled.