Add cron schedules and sync docs
Some checks failed
ci / openapi (push) Failing after 7s
ci / agent (push) Failing after 5s
ci / server (push) Failing after 6s
ci / stack-smoke (push) Has been skipped
ci / web (push) Failing after 5s

This commit is contained in:
Stanislav Rossovskii
2026-06-23 19:18:01 +04:00
parent f5ec97fcb9
commit d6f9335398
24 changed files with 638 additions and 131 deletions

View File

@@ -1,16 +1,16 @@
# CI Workflows
Placeholder for future GitHub Actions workflows.
`ci.yml` runs on pushes to `main` and on pull requests. It cancels older runs for
the same ref.
Planned jobs:
Jobs:
- Go test for `agent/`;
- `gofmt -l` and `go vet` for `agent/`;
- `go test -race` for Go stage checkpoints or concurrency-heavy changes;
- Python tests and migrations for `server/`;
- PostgreSQL constraint tests for `server/`;
- Agent-to-Server integration smoke;
- TypeScript check, lint, and build for `web/`;
- OpenAPI validation for `api/openapi.yaml`.
- OpenAPI example payload validation for `examples/heartbeat.json` and `examples/event-batch.json`.
- Docker Compose smoke for `/health`, `/ready`, `/metrics`, and example ingest.
- `openapi` - lints `api/openapi.yaml` with pinned `@redocly/cli`.
- `agent` - runs `gofmt`, `go vet`, `go test ./...`, and `go test -race ./...`.
- `server` - installs with `uv`, then runs Ruff and pytest.
- `web` - installs with npm, regenerates API types, then runs lint, typecheck,
build, and Playwright smoke.
- `stack-smoke` - runs `deploy/smoke.sh` after server and agent jobs pass.
Keep tool versions pinned in the workflow or in the component lockfiles. Do not
replace pinned CI tools with `latest`.

View File

@@ -36,6 +36,13 @@ interval = "60s"
timeout = "10s"
notifications_enabled = true
resource_limits = { cpu_time = "5s", memory = "256MiB", open_files = 128 }
[[checks]]
id = "daily_backup"
name = "Daily backup freshness"
command = "/usr/local/lib/monlet/check_backup_freshness.sh"
cron = "CRON_TZ=UTC 15 6 * * *"
timeout = "2m"
```
Ключи:
@@ -49,11 +56,14 @@ resource_limits = { cpu_time = "5s", memory = "256MiB", open_files = 128 }
- `metrics.listen` — адрес `/metrics`.
- `checks[].id` — стабильный ID проверки.
- `checks[].command` — shell-команда строкой; однострочно в `"..."`, многострочно в `'''...'''`.
- `checks[].interval` — период запуска.
- `checks[].interval` — период запуска (`60s`, `5m`, `1h`); обязателен, если не задан `checks[].cron`.
- `checks[].cron` — cron-style расписание; обязателен, если не задан `checks[].interval`. Поддерживаются стандартные 5 полей, `@hourly`/`@daily` и `CRON_TZ=...`; seconds-format, `@every` и `@reboot` отклоняются.
- `checks[].timeout` — таймаут; при таймауте результат `critical`.
- `checks[].notifications_enabled` — разрешить серверные уведомления по этой проверке.
- `checks[].resource_limits` — опциональные лимиты процесса: CPU-время, virtual memory, open files.
Для каждой проверки нужно задать ровно один способ расписания: `interval` или `cron`. Interval-проверки запускаются сразу после старта/reload, затем по периоду. Cron-проверки ждут ближайший cron-slot и не запускаются немедленно при старте.
Exit code: `0=ok`, `1=warning`, `2=critical`, `3+=unknown`.
Env агента:
@@ -65,6 +75,7 @@ Reload:
- `systemctl reload monlet-agent` перечитывает локальный TOML через `SIGHUP`.
- Ошибочный config не применяется, старый продолжает работать.
- Уже запущенный check при reload не убивается, а доживает до своего `timeout`.
- При смене расписания interval-check запускается после текущего in-flight run; cron-check ждёт следующий cron-slot.
- Для изменения `agent_id`, `state_dir`, `server.*`, `metrics.*` нужен restart.
## Server env
@@ -97,12 +108,14 @@ MONLET_EVENTS_RETENTION_MONTHS=36
MONLET_EVENTS_FUTURE_PARTITIONS=3
MONLET_PARTITION_MAINTENANCE_INTERVAL_SEC=3600
MONLET_OUTBOX_RETENTION_MAX_ROWS=50000
MONLET_OUTBOX_PRUNE_BATCH_SIZE=5000
MONLET_EVENT_DEDUP_RETENTION_DAYS=30
MONLET_EVENT_DEDUP_PRUNE_BATCH_SIZE=5000
MONLET_MAX_PENDING_AGENTS=10000
MONLET_PENDING_AGENT_TTL_DAYS=7
MONLET_PENDING_AGENT_PRUNE_BATCH_SIZE=1000
MONLET_ENABLE_NOTIFIER_WORKER=true
MONLET_NOTIFIER_TICK_SEC=5
@@ -154,10 +167,12 @@ Liveness-детектор:
- `MONLET_EVENTS_FUTURE_PARTITIONS` — сколько будущих месячных партиций создавать заранее (по умолчанию 3).
- `MONLET_PARTITION_MAINTENANCE_INTERVAL_SEC` — период обслуживания партиций и prune (по умолчанию 3600).
- `MONLET_OUTBOX_RETENTION_MAX_ROWS` — верхняя граница строк в outbox; лишние обрезаются (по умолчанию 50000).
- `MONLET_OUTBOX_PRUNE_BATCH_SIZE` — размер пакета при очистке terminal outbox-записей (по умолчанию 5000).
- `MONLET_EVENT_DEDUP_RETENTION_DAYS` — хранение записей идемпотентности `event_id` (по умолчанию 30).
- `MONLET_EVENT_DEDUP_PRUNE_BATCH_SIZE` — размер пакета при очистке dedup-записей (по умолчанию 5000).
- `MONLET_MAX_PENDING_AGENTS` — лимит новых непринятых агентов (по умолчанию 10000).
- `MONLET_PENDING_AGENT_TTL_DAYS` — очистка старых pending-ключей (по умолчанию 7).
- `MONLET_PENDING_AGENT_PRUNE_BATCH_SIZE` — размер пакета при очистке старых pending-ключей (по умолчанию 1000).
Notifier worker:
@@ -181,21 +196,29 @@ Notifier каналы:
## Web env
Веб-UI читает auth-настройки из env. Без них read-only страницы доступны локально, но admission-мутации возвращают `403`.
Веб-UI читает API/auth/timezone-настройки из env. Без auth-настроек read-only страницы доступны локально, но admission-мутации возвращают `403`.
```env
MONLET_API_BASE_URL=http://127.0.0.1:8000
MONLET_API_TOKEN=replace-with-server-token
MONLET_WEB_TIME_ZONE=UTC
MONLET_WEB_AUTH_USERNAME=admin
MONLET_WEB_AUTH_PASSWORD=replace-with-strong-password
MONLET_WEB_SESSION_SECRET=replace-with-random-32-bytes-min
MONLET_WEB_SESSION_TTL_SEC=604800
NEXT_PUBLIC_MONLET_POLL_MS=10000
# или вместо локального логина — доверие reverse-proxy:
# MONLET_WEB_TRUST_PROXY_AUTH=true
```
- `MONLET_API_BASE_URL` — базовый URL сервера для server-side fetch.
- `MONLET_API_TOKEN` — Bearer-токен для запросов UI к серверу; в браузер не передаётся.
- `MONLET_WEB_TIME_ZONE` — IANA timezone для отображения дат (по умолчанию `UTC`).
- `MONLET_WEB_AUTH_USERNAME` / `MONLET_WEB_AUTH_PASSWORD` — учётные данные локального cookie-логина.
- `MONLET_WEB_SESSION_SECRET` — секрет подписи сессии, минимум 32 байта; короче — UI считает auth неправильно настроенным.
- `MONLET_WEB_SESSION_TTL_SEC` — срок жизни сессии (по умолчанию 604800 = 7 дней).
- `MONLET_WEB_TRUST_PROXY_AUTH``true` включает доверие заголовку `X-Forwarded-User` от reverse-proxy. Несовместимо с локальным логином: одновременная настройка обоих режимов — ошибка (UI отвечает `500`).
- `NEXT_PUBLIC_MONLET_POLL_MS` — интервал браузерного polling в миллисекундах (по умолчанию 10000); для production-сборок Next.js это build-time переменная.
## Минимальная схема

View File

@@ -110,8 +110,8 @@ The server owns incident and notification lifecycle. The agent sends facts only.
The web UI is read-only for monitoring state; agent admission and blacklist
controls are the v1 mutation surface.
Public API contracts live in `api/openapi.yaml`. Design and operations docs live
under `docs/`.
Public API contracts live in `api/openapi.yaml`. Design and operations docs start
at `docs/index.md`.
## Non-Goals
@@ -134,6 +134,7 @@ Start with:
3. [server/README.md](server/README.md)
4. [web/README.md](web/README.md)
5. [api/README.md](api/README.md)
6. [docs/index.md](docs/index.md)
Dependency caches and virtual environments are kept inside the repository. See
the component READMEs for current local development commands.

View File

@@ -112,7 +112,7 @@ Server владеет lifecycle инцидентов и уведомлений.
controls - mutation surface для v1.
Публичный API contract лежит в `api/openapi.yaml`. Design и operations docs
лежат в `docs/`.
начинаются с `docs/index.md`.
## Не цели
@@ -135,6 +135,7 @@ Monlet 0.1.0 намеренно не является:
3. [server/README.md](server/README.md)
4. [web/README.md](web/README.md)
5. [api/README.md](api/README.md)
6. [docs/index.md](docs/index.md)
Dependency caches и virtual environments хранятся внутри репозитория. Актуальные
команды local development описаны в component README.

View File

@@ -14,7 +14,7 @@ Do not use `go install` or `go env -w`. Module cache, build cache, and tool bina
## Configuration
See `config.example.toml`. Required fields: top-level `state_dir`, at least one `[[checks]]` (`id`, `command`, `interval`, `timeout`), and at least one output enabled: `server.enabled = true` or `metrics.enabled = true`. `agent_id` is optional; if omitted, it defaults to the OS hostname. `hostname` is not a config key.
See `config.example.toml`. Required fields: top-level `state_dir`, at least one `[[checks]]` (`id`, `command`, `timeout`, and exactly one of `interval` or `cron`), and at least one output enabled: `server.enabled = true` or `metrics.enabled = true`. `agent_id` is optional; if omitted, it defaults to the OS hostname. `hostname` is not a config key.
| Key | Default | Notes |
|---|---|---|
@@ -25,10 +25,12 @@ See `config.example.toml`. Required fields: top-level `state_dir`, at least one
| `server.batch_interval` | `10s` | events flush cadence |
| `metrics.listen` | `127.0.0.1:9465` | low-cardinality only |
| `checks[].command` | required | Shell string executed as `/bin/sh -c`. Both forms accepted: `command = "check_disk --warn 80 --crit 90"` and triple-quoted `command = '''...'''`. Argv arrays are rejected. See [Command security contract](#command-security-contract). |
| `checks[].interval` | required unless `cron` is set | Go duration such as `60s`, `5m`, or `1h`. Interval checks run once immediately on start/reload, then on the interval. |
| `checks[].cron` | required unless `interval` is set | Standard 5-field cron expression, macros such as `@hourly`/`@daily`, and optional `CRON_TZ=...`. Seconds fields, `@every`, and `@reboot` are rejected. Cron checks wait for the next scheduled slot; they do not run immediately on start/reload. |
| `checks[].notifications_enabled` | `true` | Set `false` for checks that must not create server notifications. |
| `checks[].resource_limits` | `{}` | Optional per-check best-effort `ulimit` values: `cpu_time`, `memory`, `open_files`. |
Examples — one-line and multi-line forms:
Examples — interval, cron, and multi-line forms:
```toml
[[checks]]
@@ -37,6 +39,12 @@ command = "test $(cut -d. -f1 /proc/uptime) -gt 60"
interval = "60s"
timeout = "5s"
[[checks]]
id = "daily_backup"
command = "/usr/local/lib/monlet/check_backup_freshness.sh"
cron = "CRON_TZ=UTC 15 6 * * *"
timeout = "2m"
[[checks]]
id = "disk_root"
command = '''
@@ -81,16 +89,17 @@ executes it as `/bin/sh -c <command>` on the host where the agent runs.
Send `SIGHUP` or run `systemctl reload monlet-agent` to reload the local TOML file.
- Reload first validates the full new config. On error, the old config keeps running.
- Hot-reloadable: labels and checks, including `command`, `interval`, `timeout`,
`notifications_enabled`, `dedupe_key`, and `resource_limits`.
- Hot-reloadable: labels and checks, including `command`, `interval`, `cron`,
`timeout`, `notifications_enabled`, `dedupe_key`, and `resource_limits`.
- Restart required: `agent_id`, `state_dir`, `server.*`, and `metrics.*`.
- A check already running during reload is allowed to finish under its old config.
Removed or changed checks do not start new old-config runs; changed checks start
with the new config after the in-flight run finishes.
Removed or changed checks do not start new old-config runs. Changed interval
checks start with the new config after the in-flight run finishes; changed cron
checks wait for the next cron slot.
## Behaviour
- Per-check scheduler with no-overlap: a tick is skipped if the previous run is still in flight (`monlet_agent_check_skipped_total`).
- Per-check scheduler with no-overlap: a tick/cron slot is skipped if the previous run is still in flight (`monlet_agent_check_skipped_total`).
- Check commands run through `/bin/sh -c`; the old argv-array form is intentionally unsupported.
- Timeout via `exec.CommandContext` with `Setpgid` so the process group is killed on expiry; timeout → `critical`.
- Exit code mapping: 0=ok, 1=warning, 2=critical, 3+=unknown (ADR-0005).
@@ -115,7 +124,7 @@ Exposed on `metrics.listen` at `/metrics`. Labels are static (`check_id`, `statu
- `monlet_agent_check_exit_code{check_id}`
- `monlet_agent_check_last_run_timestamp_seconds{check_id}`
- `monlet_agent_check_last_success_timestamp_seconds{check_id}`
- `monlet_agent_check_interval_seconds{check_id}`
- `monlet_agent_check_interval_seconds{check_id}` (interval checks only)
- `monlet_agent_checks_configured`
- `monlet_agent_config_load_success`
- `monlet_agent_config_reload_total{result}`

View File

@@ -32,3 +32,11 @@ name = "Uptime probe"
command = "test $(cut -d. -f1 /proc/uptime) -gt 60"
interval = "60s"
timeout = "5s"
# Cron-style schedules are also supported. Use exactly one of interval or cron.
# [[checks]]
# id = "daily_backup"
# name = "Daily backup freshness"
# command = "/usr/local/lib/monlet/check_backup_freshness.sh"
# cron = "CRON_TZ=UTC 15 6 * * *"
# timeout = "2m"

View File

@@ -6,6 +6,8 @@ require (
github.com/BurntSushi/toml v1.6.0
github.com/google/uuid v1.6.0
github.com/prometheus/client_golang v1.23.2
github.com/prometheus/client_model v0.6.2
github.com/robfig/cron/v3 v3.0.1
)
require (
@@ -13,7 +15,6 @@ require (
github.com/cespare/xxhash/v2 v2.3.0 // indirect
github.com/kr/text v0.2.0 // indirect
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
github.com/prometheus/client_model v0.6.2 // indirect
github.com/prometheus/common v0.66.1 // indirect
github.com/prometheus/procfs v0.16.1 // indirect
go.yaml.in/yaml/v2 v2.4.2 // indirect

View File

@@ -31,6 +31,8 @@ github.com/prometheus/common v0.66.1 h1:h5E0h5/Y8niHc5DlaLlWLArTQI7tMrsfQjHV+d9Z
github.com/prometheus/common v0.66.1/go.mod h1:gcaUsgf3KfRSwHY4dIMXLPV0K/Wg1oZ8+SbZk/HH/dA=
github.com/prometheus/procfs v0.16.1 h1:hZ15bTNuirocR6u0JZ6BAHHmwS1p8B4P6MRqxtzMyRg=
github.com/prometheus/procfs v0.16.1/go.mod h1:teAbpZRB1iIAJYREa1LsoWUXykVXA1KlTmWl8x/U+Is=
github.com/robfig/cron/v3 v3.0.1 h1:WdRxkvbJztn8LMz/QEvLN5sBU+xKpSqwwUO1Pjr4qDs=
github.com/robfig/cron/v3 v3.0.1/go.mod h1:eQICP3HwyT7UooqI/z+Ov+PtYAWygg1TEWWzGIFLtro=
github.com/rogpeppe/go-internal v1.10.0 h1:TMyTOH3F/DB16zRVcYyreMH6GnZZrwQVAoYjRBZyWFQ=
github.com/rogpeppe/go-internal v1.10.0/go.mod h1:UQnix2H7Ngw/k4C5ijL5+65zddjncjaFoBhdsK/akog=
github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U=

View File

@@ -47,9 +47,7 @@ func (a *App) Run(ctx context.Context) error {
a.Metrics.ConfigLoadSuccess.Set(1)
a.Metrics.ChecksConfigured.Set(float64(len(cfg.Checks)))
a.Metrics.BuildInfo.WithLabelValues(a.Version, boolLabel(cfg.PushesToServer()), boolLabel(cfg.ExposesMetrics())).Set(1)
for _, c := range cfg.Checks {
a.Metrics.CheckInterval.WithLabelValues(c.ID).Set(c.Interval.Duration.Seconds())
}
a.recordCheckScheduleMetrics(cfg.Checks)
var wg sync.WaitGroup
@@ -256,9 +254,7 @@ func (a *App) Reload(newCfg *config.Config, newAgentID string) error {
a.Metrics.ConfigReloadTotal.WithLabelValues("success").Inc()
a.Metrics.ConfigLastReloadSuccess.Set(float64(time.Now().Unix()))
a.Metrics.ChecksConfigured.Set(float64(len(newCfg.Checks)))
for _, c := range newCfg.Checks {
a.Metrics.CheckInterval.WithLabelValues(c.ID).Set(c.Interval.Duration.Seconds())
}
a.recordCheckScheduleMetrics(newCfg.Checks)
a.schedulerMu.RLock()
mgr := a.scheduler
@@ -275,6 +271,16 @@ func (a *App) RecordReloadFailure() {
a.Metrics.ConfigReloadTotal.WithLabelValues("failure").Inc()
}
func (a *App) recordCheckScheduleMetrics(checks []config.CheckConfig) {
for _, c := range checks {
if c.UsesInterval() {
a.Metrics.CheckInterval.WithLabelValues(c.ID).Set(c.Interval.Duration.Seconds())
} else {
a.Metrics.CheckInterval.DeleteLabelValues(c.ID)
}
}
}
func (a *App) config() *config.Config {
a.cfgMu.RLock()
defer a.cfgMu.RUnlock()

View File

@@ -17,6 +17,7 @@ import (
"github.com/monlet/agent/internal/config"
"github.com/monlet/agent/internal/metrics"
"github.com/monlet/agent/internal/spool"
dto "github.com/prometheus/client_model/go"
)
func TestPrometheusOnlyDoesNotPush(t *testing.T) {
@@ -257,6 +258,92 @@ func TestReloadAcceptsLabelsAndChecks(t *testing.T) {
}
}
func TestReloadUpdatesIntervalMetricForCron(t *testing.T) {
oldCfg := &config.Config{
AgentID: "a1",
Hostname: "h",
StateDir: "/tmp/old",
Server: config.ServerConfig{
Enabled: boolPtr(true),
URL: "http://server",
HeartbeatInterval: config.Duration{Duration: 10 * time.Second},
BatchInterval: config.Duration{Duration: 10 * time.Second},
},
Metrics: config.MetricsConfig{Enabled: false},
Checks: []config.CheckConfig{{
ID: "c1",
Command: "true",
Interval: config.Duration{Duration: time.Second},
Timeout: config.Duration{Duration: time.Second},
}},
}
a := &App{
Cfg: oldCfg,
AgentID: "a1",
Metrics: metrics.New(),
}
a.recordCheckScheduleMetrics(oldCfg.Checks)
if got, ok := checkIntervalMetric(t, a, "c1"); !ok || got != 1 {
t.Fatalf("interval metric missing before reload: value=%v ok=%v", got, ok)
}
cronCfg := *oldCfg
cronCfg.Checks = []config.CheckConfig{{
ID: "c1",
Command: "true",
Cron: "0 * * * *",
Timeout: config.Duration{Duration: time.Second},
}}
if err := a.Reload(&cronCfg, "a1"); err != nil {
t.Fatal(err)
}
if got, ok := checkIntervalMetric(t, a, "c1"); ok {
t.Fatalf("interval metric must be deleted for cron check, got %v", got)
}
intervalCfg := cronCfg
intervalCfg.Checks = []config.CheckConfig{{
ID: "c1",
Command: "true",
Interval: config.Duration{Duration: 2 * time.Second},
Timeout: config.Duration{Duration: time.Second},
}}
if err := a.Reload(&intervalCfg, "a1"); err != nil {
t.Fatal(err)
}
if got, ok := checkIntervalMetric(t, a, "c1"); !ok || got != 2 {
t.Fatalf("interval metric missing after interval reload: value=%v ok=%v", got, ok)
}
}
func checkIntervalMetric(t *testing.T, a *App, checkID string) (float64, bool) {
t.Helper()
families, err := a.Metrics.Registry.Gather()
if err != nil {
t.Fatal(err)
}
for _, family := range families {
if family.GetName() != "monlet_agent_check_interval_seconds" {
continue
}
for _, metric := range family.GetMetric() {
if labelValue(metric.GetLabel(), "check_id") == checkID && metric.GetGauge() != nil {
return metric.GetGauge().GetValue(), true
}
}
}
return 0, false
}
func labelValue(labels []*dto.LabelPair, name string) string {
for _, label := range labels {
if label.GetName() == name {
return label.GetValue()
}
}
return ""
}
func boolPtr(v bool) *bool {
return &v
}

View File

@@ -11,6 +11,7 @@ import (
"unicode/utf8"
"github.com/BurntSushi/toml"
"github.com/robfig/cron/v3"
)
var idPattern = regexp.MustCompile(`^[A-Za-z0-9._:-]+$`)
@@ -58,6 +59,7 @@ type CheckConfig struct {
Name string `toml:"name"`
Command string `toml:"command"`
Interval Duration `toml:"interval"`
Cron string `toml:"cron"`
Timeout Duration `toml:"timeout"`
NotificationsEnabled *bool `toml:"notifications_enabled"`
DedupeKey string `toml:"dedupe_key"`
@@ -211,14 +213,24 @@ func (c *Config) Validate() error {
if strings.TrimSpace(ch.Command) == "" {
return fmt.Errorf("check %q: command is required", ch.ID)
}
if ch.Interval.Duration <= 0 {
return fmt.Errorf("check %q: interval must be > 0", ch.ID)
}
if ch.Timeout.Duration <= 0 {
return fmt.Errorf("check %q: timeout must be > 0", ch.ID)
}
if ch.Timeout.Duration > ch.Interval.Duration {
if ch.Interval.Duration < 0 {
return fmt.Errorf("check %q: interval must be > 0", ch.ID)
}
ch.Cron = strings.TrimSpace(ch.Cron)
hasInterval := ch.Interval.Duration > 0
hasCron := ch.Cron != ""
switch {
case hasInterval == hasCron:
return fmt.Errorf("check %q: exactly one of interval or cron is required", ch.ID)
case hasInterval && ch.Timeout.Duration > ch.Interval.Duration:
return fmt.Errorf("check %q: timeout must be <= interval", ch.ID)
case hasCron:
if _, err := ParseCronSchedule(ch.Cron); err != nil {
return fmt.Errorf("check %q: invalid cron: %w", ch.ID, err)
}
}
if len(ch.DedupeKey) > 256 {
return fmt.Errorf("check %q: dedupe_key too long", ch.ID)
@@ -253,6 +265,42 @@ func (c CheckConfig) NotificationsOn() bool {
return c.NotificationsEnabled == nil || *c.NotificationsEnabled
}
func (c CheckConfig) UsesInterval() bool {
return c.Interval.Duration > 0
}
func (c CheckConfig) UsesCron() bool {
return strings.TrimSpace(c.Cron) != ""
}
func ParseCronSchedule(spec string) (cron.Schedule, error) {
spec = strings.TrimSpace(spec)
if spec == "" {
return nil, fmt.Errorf("cron is empty")
}
withoutTZ := specWithoutTZ(spec)
lower := strings.ToLower(withoutTZ)
if strings.HasPrefix(lower, "@every") {
return nil, fmt.Errorf("@every is not supported; use interval")
}
if lower == "@reboot" {
return nil, fmt.Errorf("@reboot is not supported")
}
return cron.ParseStandard(spec)
}
func specWithoutTZ(spec string) string {
fields := strings.Fields(spec)
if len(fields) == 0 {
return ""
}
first := fields[0]
if strings.HasPrefix(first, "CRON_TZ=") || strings.HasPrefix(first, "TZ=") {
return strings.ToLower(strings.Join(fields[1:], " "))
}
return strings.ToLower(strings.Join(fields, " "))
}
func validateResourceLimits(checkID string, limits ResourceLimits) error {
if limits.CPUTime.Duration < 0 {
return fmt.Errorf("check %q: resource_limits.cpu_time must be > 0", checkID)

View File

@@ -48,9 +48,84 @@ func TestLoadMinimal(t *testing.T) {
}
}
func TestLoadCronCheck(t *testing.T) {
c, err := Load(writeTmp(t, `
agent_id = "host-1"
state_dir = "/tmp/x"
[server]
enabled = true
url = "http://localhost"
[[checks]]
id = "c1"
command = "true"
cron = "CRON_TZ=UTC 0 6 * * *"
timeout = "5m"
`))
if err != nil {
t.Fatal(err)
}
if !c.Checks[0].UsesCron() || c.Checks[0].UsesInterval() {
t.Fatalf("unexpected schedule mode: %+v", c.Checks[0])
}
}
func TestValidateCheckScheduleChoice(t *testing.T) {
for name, body := range map[string]string{
"both_interval_and_cron": `
interval = "10s"
cron = "0 * * * *"
timeout = "5s"
`,
"neither_interval_nor_cron": `
timeout = "5s"
`,
"invalid_cron": `
cron = "not a cron"
timeout = "5s"
`,
"seconds_cron": `
cron = "0 0 * * * *"
timeout = "5s"
`,
"every_cron": `
cron = "@every 10s"
timeout = "5s"
`,
"tz_every_cron": `
cron = "CRON_TZ=UTC @every 10s"
timeout = "5s"
`,
"reboot_cron": `
cron = "@reboot"
timeout = "5s"
`,
} {
t.Run(name, func(t *testing.T) {
_, err := Load(writeTmp(t, `
agent_id = "host-1"
state_dir = "/tmp/x"
[server]
enabled = true
url = "http://localhost"
[[checks]]
id = "c1"
command = "true"
`+body+`
`))
if err == nil {
t.Fatal("expected schedule validation error")
}
})
}
}
func TestLoadResourceLimits(t *testing.T) {
c, err := Load(writeTmp(t, `
agent_id = "host-1"
agent_id = "host-1"
state_dir = "/tmp/x"
[server]

View File

@@ -11,6 +11,8 @@ import (
"github.com/monlet/agent/internal/runner"
)
var parseCronSchedule = config.ParseCronSchedule
// Event is a contract-shaped check result emitted by the scheduler.
type Event struct {
EventID string `json:"event_id"`
@@ -207,8 +209,10 @@ func (w *checkWorker) loop(initial config.CheckConfig) {
w.wg.Done()
}()
cfg := initial
t := time.NewTicker(cfg.Interval.Duration)
t := time.NewTimer(time.Hour)
stopTimer(t)
defer t.Stop()
var timerCh <-chan time.Time
running := false
stopping := false
@@ -230,7 +234,26 @@ func (w *checkWorker) loop(initial config.CheckConfig) {
}()
}
start(cfg)
arm := func(c config.CheckConfig, immediateInterval bool) bool {
delay, err := nextDelay(c, time.Now(), immediateInterval)
if err != nil {
return false
}
resetTimer(t, delay)
timerCh = t.C
return true
}
disarm := func() {
stopTimer(t)
timerCh = nil
}
if !arm(cfg, true) {
if w.hooks.OnStopped != nil {
w.hooks.OnStopped(cfg.ID)
}
return
}
for {
select {
@@ -257,11 +280,21 @@ func (w *checkWorker) loop(initial config.CheckConfig) {
continue
}
cfg = next
t.Reset(cfg.Interval.Duration)
if cfg.UsesCron() {
runAfterCurrent = false
if !arm(cfg, false) {
return
}
continue
}
if running {
runAfterCurrent = true
disarm()
} else {
start(cfg)
if !arm(cfg, false) {
return
}
}
case <-w.doneCh:
running = false
@@ -281,26 +314,35 @@ func (w *checkWorker) loop(initial config.CheckConfig) {
}
if runAfterCurrent {
runAfterCurrent = false
startAfterDrain := true
for {
select {
case next := <-w.updateCh:
if !reflect.DeepEqual(cfg, next) {
cfg = next
t.Reset(cfg.Interval.Duration)
startAfterDrain = !cfg.UsesCron()
}
default:
start(cfg)
if startAfterDrain {
start(cfg)
}
if !arm(cfg, false) {
return
}
goto nextLoop
}
}
nextLoop:
continue
}
case <-t.C:
case <-timerCh:
if stopping {
continue
}
start(cfg)
if !arm(cfg, false) {
return
}
}
}
}
@@ -358,3 +400,41 @@ func incidentKey(agentID string, c config.CheckConfig) string {
}
return agentID + ":" + c.ID
}
func nextDelay(c config.CheckConfig, now time.Time, immediateInterval bool) (time.Duration, error) {
if c.UsesCron() {
schedule, err := parseCronSchedule(c.Cron)
if err != nil {
return 0, err
}
return delayUntil(now, schedule.Next(now)), nil
}
if immediateInterval {
return 0, nil
}
return c.Interval.Duration, nil
}
func delayUntil(now, next time.Time) time.Duration {
if next.Before(now) {
return 0
}
return next.Sub(now)
}
func resetTimer(t *time.Timer, d time.Duration) {
if d < 0 {
d = 0
}
stopTimer(t)
t.Reset(d)
}
func stopTimer(t *time.Timer) {
if !t.Stop() {
select {
case <-t.C:
default:
}
}
}

View File

@@ -8,6 +8,7 @@ import (
"time"
"github.com/monlet/agent/internal/config"
"github.com/robfig/cron/v3"
)
func runScheduler(ctx context.Context, agentID string, checks []config.CheckConfig, out chan<- Event, h Hooks) {
@@ -27,6 +28,32 @@ func mkCheck(id, cmd string, interval, timeout time.Duration) config.CheckConfig
}
}
type fakeCronSchedule struct {
delay time.Duration
}
func (s fakeCronSchedule) Next(t time.Time) time.Time {
return t.Add(s.delay)
}
func withFakeCron(t *testing.T, delay time.Duration) {
t.Helper()
old := parseCronSchedule
parseCronSchedule = func(string) (cron.Schedule, error) {
return fakeCronSchedule{delay: delay}, nil
}
t.Cleanup(func() { parseCronSchedule = old })
}
func mkCronCheck(id, cmd string, timeout time.Duration) config.CheckConfig {
return config.CheckConfig{
ID: id,
Command: cmd,
Cron: "0 * * * *",
Timeout: config.Duration{Duration: timeout},
}
}
func TestEmitsEvents(t *testing.T) {
ctx, cancel := context.WithTimeout(context.Background(), 500*time.Millisecond)
defer cancel()
@@ -45,6 +72,41 @@ func TestEmitsEvents(t *testing.T) {
}
}
func TestCronDoesNotRunImmediately(t *testing.T) {
withFakeCron(t, 200*time.Millisecond)
ctx, cancel := context.WithTimeout(context.Background(), 100*time.Millisecond)
defer cancel()
out := make(chan Event, 10)
m := NewManager(ctx, "a1", out, Hooks{})
m.Update([]config.CheckConfig{mkCronCheck("cron", "echo cron", 100*time.Millisecond)})
select {
case ev := <-out:
t.Fatalf("cron check ran before next slot: %+v", ev)
case <-ctx.Done():
}
m.Stop()
m.Wait()
}
func TestCronEmitsOnDueSlot(t *testing.T) {
withFakeCron(t, 30*time.Millisecond)
ctx, cancel := context.WithTimeout(context.Background(), time.Second)
defer cancel()
out := make(chan Event, 10)
m := NewManager(ctx, "a1", out, Hooks{})
m.Update([]config.CheckConfig{mkCronCheck("cron", "echo cron", 100*time.Millisecond)})
select {
case ev := <-out:
if ev.CheckID != "cron" || ev.Status != "ok" {
t.Fatalf("unexpected event: %+v", ev)
}
case <-ctx.Done():
t.Fatal("no cron event")
}
m.Stop()
m.Wait()
}
func TestNoOverlapSkips(t *testing.T) {
ctx, cancel := context.WithTimeout(context.Background(), 600*time.Millisecond)
defer cancel()
@@ -59,6 +121,23 @@ func TestNoOverlapSkips(t *testing.T) {
}
}
func TestCronNoOverlapSkips(t *testing.T) {
withFakeCron(t, 50*time.Millisecond)
ctx, cancel := context.WithTimeout(context.Background(), 250*time.Millisecond)
defer cancel()
out := make(chan Event, 10)
var skipped int32
hooks := Hooks{OnSkipped: func(string) { atomic.AddInt32(&skipped, 1) }}
m := NewManager(ctx, "a1", out, hooks)
m.Update([]config.CheckConfig{mkCronCheck("cron", "sleep 0.2", 500*time.Millisecond)})
<-ctx.Done()
m.Stop()
m.Wait()
if atomic.LoadInt32(&skipped) == 0 {
t.Fatal("expected at least one skipped cron slot")
}
}
func TestReportsFullEventChannel(t *testing.T) {
ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second)
defer cancel()

View File

@@ -1,9 +1,18 @@
# Alertmanager
Placeholder for optional Alertmanager integration examples.
Example Alertmanager configuration for the optional observability profile.
Planned examples:
`alertmanager.yml` is intentionally safe by default: it routes to a `null`
receiver, so alerts are grouped but dropped. Before using it outside local
development, add a real receiver and set `route.receiver` to that receiver.
- receiver config for Monlet-originated alerts;
- stable label conventions;
- routing notes for avoiding duplicate notifications.
Monlet can also send notifications to Alertmanager through the server notifier:
```env
MONLET_NOTIFIER_ALERTMANAGER_ENABLED=true
MONLET_NOTIFIER_ALERTMANAGER_URL=http://alertmanager:9093
```
Keep labels stable and low-cardinality. The example groups by `alertname`,
`agent_id`, and `check_id`; do not add timestamps, event IDs, or output text to
Alertmanager labels.

View File

@@ -1,11 +1,20 @@
# Grafana
Placeholder for Grafana dashboard examples.
Starter Grafana provisioning for the optional observability profile.
Planned dashboards:
Files:
- agents alive/stale/dead;
- check state counts;
- open incidents;
- notification outbox state;
- server ingestion and notifier failures.
- `provisioning/datasources/prometheus.yml` - adds Prometheus at `http://prometheus:9090`.
- `provisioning/dashboards/monlet.yml` - loads dashboards from `/var/lib/grafana/dashboards`.
- `dashboards/monlet-overview.json` - starter dashboard for agent status, open incidents, event ingest rate, and notification rate.
Run it with:
```sh
cd deploy
MONLET_AUTH_TOKEN=$(openssl rand -hex 16) docker compose --profile observability up
```
Grafana is exposed at <http://127.0.0.1:3001>. The local default is
`admin` / `admin` unless `GF_ADMIN_PASSWORD` is set. Change it before exposing
Grafana outside local development.

View File

@@ -1,9 +1,20 @@
# Prometheus
Placeholder for Prometheus examples.
Example scrape configuration for the optional observability profile in
`deploy/docker-compose.yml`.
Planned examples:
Run it with:
- scrape Monlet agent `/metrics`;
- scrape Monlet server `/metrics`;
- sample rules for checks that use `notifications_enabled = false`.
```sh
cd deploy
MONLET_AUTH_TOKEN=$(openssl rand -hex 16) docker compose --profile observability up
```
The example config scrapes:
- `server:8000/metrics` as `monlet-server`
- `host.docker.internal:9100/metrics` as `monlet-agent`
Adjust the agent target for your environment. In the showcase stack one demo
agent exposes metrics on host port `9465`; production agents usually expose
`metrics.listen` only on the host or a private monitoring network.

16
docs/index.md Normal file
View File

@@ -0,0 +1,16 @@
# Monlet Docs
This directory contains operator-facing docs that do not fit in the root or component READMEs.
## Start Here
- [First deployment checklist](ops/deployment-checklist.md) - step-by-step production-ish stand-up checklist.
- [Backup and restore](ops/backup-restore.md) - PostgreSQL backup, retention, and disaster recovery notes.
- [Token rotation](ops/token-rotation.md) - UI token overlap and agent-key rotation.
- [Deployment notes](ops/deployment.md) - deployment shape and production boundaries.
## Conventions
- Component setup lives in `agent/README.md`, `server/README.md`, `web/README.md`, and `deploy/README.md`.
- API contracts live in `api/openapi.yaml`; `api/README.md` only describes contract rules.
- Keep docs factual and current. Avoid placeholder docs when a real config or script already exists.

View File

@@ -1,68 +0,0 @@
# Deployment
This is an initial deployment plan, not a production runbook.
## Agent
Agent runs as a systemd service.
Expected paths:
- binary: `/usr/local/bin/monlet-agent`;
- config: `/etc/monlet/agent.toml`;
- state: `/var/lib/monlet-agent`;
- logs: journald.
The agent should support graceful shutdown and avoid overlapping runs for the same check.
## Server
Server runs in Docker.
Production storage:
- PostgreSQL.
Local/demo storage:
- SQLite may be supported later, but must be documented as non-production.
Required deployment features:
- TLS-ready reverse proxy;
- agent API tokens;
- request size limits;
- structured logs;
- `/health`;
- `/ready`;
- `/metrics`.
Docker image rule:
- build server dependencies inside the image;
- do not require Python packages installed on the host;
- do not mount host virtualenvs into the container;
- local non-Docker development uses `server/.venv` only.
## Web
Web is read-only for monitoring state in v1. Agent admission and blacklist controls are the explicit write exception.
Auth plan:
- local web login or equivalent gateway auth through a trusted reverse proxy;
- no full RBAC in v1.
## Observability
Optional examples:
- Prometheus scrape config;
- Alertmanager route config;
- Grafana dashboard stub.
Metrics must avoid high-cardinality labels.
## Backups
PostgreSQL backup/restore docs are required before first production use.

66
docs/ops/deployment.md Normal file
View File

@@ -0,0 +1,66 @@
# Deployment Notes
These notes describe the intended production shape. Use
[`docs/ops/deployment-checklist.md`](deployment-checklist.md) for the
step-by-step first stand-up checklist.
## Agent
The agent runs on monitored hosts as a systemd service.
Expected paths:
- binary: `/usr/local/bin/monlet-agent`
- config: `/etc/monlet/agent.toml`
- state: `/var/lib/monlet-agent`
- logs: journald
The agent reads only local config. It does not accept remote command execution or
remote config push from the server. Per-check scheduling is local to the agent,
and overlapping runs for the same check are skipped.
## Server
The server runs as a containerized FastAPI application backed by PostgreSQL 16+.
It owns inventory, current check state, incidents, notification outbox state,
retention, and public `/api/v1` contracts.
Required production boundaries:
- TLS and rate limiting at a reverse proxy
- PostgreSQL as canonical storage
- `MONLET_AUTH_TOKEN` stored in a secret manager
- request body limits enabled
- structured logs with auth material redacted
- `/api/v1/health`, `/api/v1/ready`, and `/metrics` exposed for operations
Docker images install dependencies inside the image. Do not mount host virtual
environments into server containers.
## Web
The web app is an operator dashboard. Monitoring pages are read-only; agent
admission and blacklist controls are the explicit v1 write surface.
Protect the UI with either local web login or a trusted auth proxy. If local
login is enabled, rate-limit `/login` at the reverse proxy. Monlet v1 does not
provide full RBAC.
## Observability
The local deploy stack includes optional Prometheus, Alertmanager, and Grafana
examples under `deploy/`. Production deployments should at minimum scrape:
- Monlet server `/metrics`
- each agent `/metrics`
- reverse-proxy and PostgreSQL metrics from the surrounding environment
Metric labels must stay low-cardinality. Do not put timestamps, request IDs, or
unbounded host-local values into check IDs or labels.
## Backups
PostgreSQL is the canonical state. Follow
[`docs/ops/backup-restore.md`](backup-restore.md) before first production
use. Agent spools are best-effort buffers and are not a substitute for database
backups.

View File

@@ -1,3 +1,12 @@
# Examples
Examples for checks, configs, and local demos.
Small payloads and scripts used by local smoke tests and manual API checks.
Files:
- `heartbeat.json` - example agent heartbeat request body.
- `event-batch.json` - example event batch request body.
- `checks/example_check.sh` - minimal shell check that follows the Monlet exit-code contract.
Use these examples as templates only. Real agents normally send heartbeat and
event payloads through the Go agent, not by hand-written curl requests.

View File

@@ -1,25 +1,47 @@
MONLET_AUTH_TOKEN=replace-me-with-random-token
MONLET_DATABASE_URL=postgresql+asyncpg://monlet:monlet@localhost:5432/monlet
MONLET_LOG_LEVEL=INFO
MONLET_HOST=0.0.0.0
MONLET_PORT=8000
MONLET_BODY_LIMIT_BYTES=1048576
MONLET_OUTPUT_MAX_BYTES=8192
MONLET_MAX_BATCH_EVENTS=200
MONLET_ENABLE_DETECTOR=true
MONLET_STALE_AFTER_SEC=20
MONLET_DEAD_AFTER_SEC=30
MONLET_DETECTOR_TICK_SEC=5
MONLET_ENABLE_DETECTOR=true
MONLET_DB_POOL_SIZE=10
MONLET_DB_MAX_OVERFLOW=10
MONLET_DB_POOL_TIMEOUT_SEC=30
MONLET_DB_POOL_RECYCLE_SEC=1800
MONLET_EVENTS_RETENTION_MONTHS=36
MONLET_EVENTS_FUTURE_PARTITIONS=3
MONLET_PARTITION_MAINTENANCE_INTERVAL_SEC=3600
MONLET_OUTBOX_RETENTION_MAX_ROWS=50000
MONLET_OUTBOX_PRUNE_BATCH_SIZE=5000
MONLET_EVENT_DEDUP_RETENTION_DAYS=30
MONLET_EVENT_DEDUP_PRUNE_BATCH_SIZE=5000
MONLET_MAX_PENDING_AGENTS=10000
MONLET_PENDING_AGENT_TTL_DAYS=7
MONLET_PENDING_AGENT_PRUNE_BATCH_SIZE=1000
MONLET_OUTBOX_RETENTION_MAX_ROWS=50000
MONLET_ENABLE_NOTIFIER_WORKER=true
MONLET_NOTIFIER_TICK_SEC=5
MONLET_NOTIFIER_BATCH_SIZE=20
MONLET_NOTIFIER_CONCURRENCY=4
MONLET_NOTIFIER_MAX_ATTEMPTS=8
MONLET_NOTIFIER_LEASE_SEC=300
MONLET_NOTIFIER_HTTP_TIMEOUT_SEC=10
MONLET_NOTIFIER_INCLUDE_OUTPUT=true
MONLET_NOTIFIER_MAX_OUTPUT_BYTES=1024
MONLET_NOTIFICATION_TIME_ZONE=UTC
MONLET_NOTIFIER_DEBUG_ENABLED=true
MONLET_NOTIFIER_TELEGRAM_ENABLED=false
MONLET_NOTIFIER_TELEGRAM_TOKEN=

View File

@@ -33,26 +33,38 @@ UV_PROJECT_ENVIRONMENT=.venv UV_CACHE_DIR=../.cache/uv uv run fastapi dev monlet
| `MONLET_AUTH_TOKEN` | required | UI/operator Bearer token. Space/comma-separated list; any accepted. `changeme` is rejected. |
| `MONLET_DATABASE_URL` | `postgresql+asyncpg://monlet:monlet@127.0.0.1:5432/monlet` | Async DSN |
| `MONLET_LOG_LEVEL` | `INFO` | Log level |
| `MONLET_HOST` | `0.0.0.0` | HTTP bind host for direct server runs |
| `MONLET_PORT` | `8000` | HTTP bind port for direct server runs |
| `MONLET_BODY_LIMIT_BYTES` | `1048576` | Request body limit (ADR-0005) |
| `MONLET_OUTPUT_MAX_BYTES` | `8192` | Per-event output limit |
| `MONLET_MAX_BATCH_EVENTS` | `200` | Maximum events per ingestion batch |
| `MONLET_STALE_AFTER_SEC` | `20` | Stale threshold |
| `MONLET_DEAD_AFTER_SEC` | `30` | Dead threshold |
| `MONLET_DETECTOR_TICK_SEC` | `5` | Detector tick |
| `MONLET_ENABLE_DETECTOR` | `true` | Toggle detector task |
| `MONLET_DB_POOL_SIZE` | `10` | Base async DB pool size |
| `MONLET_DB_MAX_OVERFLOW` | `10` | Extra DB connections above pool size |
| `MONLET_DB_POOL_TIMEOUT_SEC` | `30` | Seconds to wait for a DB connection |
| `MONLET_DB_POOL_RECYCLE_SEC` | `1800` | Recycle DB connections by age |
| `MONLET_EVENTS_RETENTION_MONTHS` | `36` | Months of `events` partitions to keep; `0` disables drop |
| `MONLET_EVENTS_FUTURE_PARTITIONS` | `3` | Future month partitions to pre-create |
| `MONLET_EVENT_DEDUP_RETENTION_DAYS` | `30` | Idempotency window for `event_id` |
| `MONLET_EVENT_DEDUP_PRUNE_BATCH_SIZE` | `5000` | Dedup rows pruned per maintenance tick |
| `MONLET_MAX_PENDING_AGENTS` | `10000` | Maximum pending agent keys before heartbeat rejects with 429 |
| `MONLET_PENDING_AGENT_TTL_DAYS` | `7` | Pending key cleanup TTL; `0` disables pruning |
| `MONLET_PENDING_AGENT_PRUNE_BATCH_SIZE` | `1000` | Pending cleanup rows per detector tick |
| `MONLET_PARTITION_MAINTENANCE_INTERVAL_SEC` | `3600` | Partition create/drop worker interval |
| `MONLET_OUTBOX_RETENTION_MAX_ROWS` | `50000` | Maximum retained terminal outbox rows; `0` disables pruning |
| `MONLET_OUTBOX_PRUNE_BATCH_SIZE` | `5000` | Outbox rows pruned per maintenance tick |
| `MONLET_ENABLE_NOTIFIER_WORKER` | `true` | Toggle outbox worker |
| `MONLET_NOTIFIER_TICK_SEC` | `5` | Worker poll interval |
| `MONLET_NOTIFIER_BATCH_SIZE` | `20` | Rows claimed per tick |
| `MONLET_NOTIFIER_CONCURRENCY` | `4` | Parallel notification deliveries per tick |
| `MONLET_NOTIFIER_MAX_ATTEMPTS` | `8` | Terminal-failure threshold (ADR-0005) |
| `MONLET_NOTIFIER_LEASE_SEC` | `300` | Recovery lease for stuck `sending` rows |
| `MONLET_NOTIFIER_HTTP_TIMEOUT_SEC` | `10` | Outbound HTTP timeout |
| `MONLET_NOTIFIER_INCLUDE_OUTPUT` | `true` | Include redacted check output in notifications |
| `MONLET_NOTIFIER_MAX_OUTPUT_BYTES` | `1024` | Maximum notification output bytes after redaction |
| `MONLET_NOTIFICATION_TIME_ZONE` | `UTC` | IANA timezone for human-readable notification timestamps |
| `MONLET_NOTIFIER_DEBUG_ENABLED` | `true` | Log-only notifier |
| `MONLET_NOTIFIER_TELEGRAM_ENABLED` | `false` | Telegram notifier on/off |

View File

@@ -41,6 +41,7 @@ Do not install Node packages globally. Project dependencies live in `web/node_mo
| `MONLET_WEB_SESSION_SECRET` | _(none)_ | Secret used to sign the web session cookie; required with local web login; at least 32 bytes |
| `MONLET_WEB_SESSION_TTL_SEC` | `604800` | Local web session lifetime |
| `MONLET_WEB_TRUST_PROXY_AUTH` | `false` | Trust `X-Forwarded-User` from an upstream auth proxy |
| `NEXT_PUBLIC_MONLET_POLL_MS` | `10000` | Browser polling interval in ms; build-time for production Next.js bundles |
The API token never reaches the browser — all server calls happen in Server Components / route handlers. Agent admission actions are blocked unless local web login is configured or trusted proxy auth is enabled.