Add cron schedules and sync docs
This commit is contained in:
24
.github/workflows/README.md
vendored
24
.github/workflows/README.md
vendored
@@ -1,16 +1,16 @@
|
||||
# CI Workflows
|
||||
|
||||
Placeholder for future GitHub Actions workflows.
|
||||
`ci.yml` runs on pushes to `main` and on pull requests. It cancels older runs for
|
||||
the same ref.
|
||||
|
||||
Planned jobs:
|
||||
Jobs:
|
||||
|
||||
- Go test for `agent/`;
|
||||
- `gofmt -l` and `go vet` for `agent/`;
|
||||
- `go test -race` for Go stage checkpoints or concurrency-heavy changes;
|
||||
- Python tests and migrations for `server/`;
|
||||
- PostgreSQL constraint tests for `server/`;
|
||||
- Agent-to-Server integration smoke;
|
||||
- TypeScript check, lint, and build for `web/`;
|
||||
- OpenAPI validation for `api/openapi.yaml`.
|
||||
- OpenAPI example payload validation for `examples/heartbeat.json` and `examples/event-batch.json`.
|
||||
- Docker Compose smoke for `/health`, `/ready`, `/metrics`, and example ingest.
|
||||
- `openapi` - lints `api/openapi.yaml` with pinned `@redocly/cli`.
|
||||
- `agent` - runs `gofmt`, `go vet`, `go test ./...`, and `go test -race ./...`.
|
||||
- `server` - installs with `uv`, then runs Ruff and pytest.
|
||||
- `web` - installs with npm, regenerates API types, then runs lint, typecheck,
|
||||
build, and Playwright smoke.
|
||||
- `stack-smoke` - runs `deploy/smoke.sh` after server and agent jobs pass.
|
||||
|
||||
Keep tool versions pinned in the workflow or in the component lockfiles. Do not
|
||||
replace pinned CI tools with `latest`.
|
||||
|
||||
@@ -36,6 +36,13 @@ interval = "60s"
|
||||
timeout = "10s"
|
||||
notifications_enabled = true
|
||||
resource_limits = { cpu_time = "5s", memory = "256MiB", open_files = 128 }
|
||||
|
||||
[[checks]]
|
||||
id = "daily_backup"
|
||||
name = "Daily backup freshness"
|
||||
command = "/usr/local/lib/monlet/check_backup_freshness.sh"
|
||||
cron = "CRON_TZ=UTC 15 6 * * *"
|
||||
timeout = "2m"
|
||||
```
|
||||
|
||||
Ключи:
|
||||
@@ -49,11 +56,14 @@ resource_limits = { cpu_time = "5s", memory = "256MiB", open_files = 128 }
|
||||
- `metrics.listen` — адрес `/metrics`.
|
||||
- `checks[].id` — стабильный ID проверки.
|
||||
- `checks[].command` — shell-команда строкой; однострочно в `"..."`, многострочно в `'''...'''`.
|
||||
- `checks[].interval` — период запуска.
|
||||
- `checks[].interval` — период запуска (`60s`, `5m`, `1h`); обязателен, если не задан `checks[].cron`.
|
||||
- `checks[].cron` — cron-style расписание; обязателен, если не задан `checks[].interval`. Поддерживаются стандартные 5 полей, `@hourly`/`@daily` и `CRON_TZ=...`; seconds-format, `@every` и `@reboot` отклоняются.
|
||||
- `checks[].timeout` — таймаут; при таймауте результат `critical`.
|
||||
- `checks[].notifications_enabled` — разрешить серверные уведомления по этой проверке.
|
||||
- `checks[].resource_limits` — опциональные лимиты процесса: CPU-время, virtual memory, open files.
|
||||
|
||||
Для каждой проверки нужно задать ровно один способ расписания: `interval` или `cron`. Interval-проверки запускаются сразу после старта/reload, затем по периоду. Cron-проверки ждут ближайший cron-slot и не запускаются немедленно при старте.
|
||||
|
||||
Exit code: `0=ok`, `1=warning`, `2=critical`, `3+=unknown`.
|
||||
|
||||
Env агента:
|
||||
@@ -65,6 +75,7 @@ Reload:
|
||||
- `systemctl reload monlet-agent` перечитывает локальный TOML через `SIGHUP`.
|
||||
- Ошибочный config не применяется, старый продолжает работать.
|
||||
- Уже запущенный check при reload не убивается, а доживает до своего `timeout`.
|
||||
- При смене расписания interval-check запускается после текущего in-flight run; cron-check ждёт следующий cron-slot.
|
||||
- Для изменения `agent_id`, `state_dir`, `server.*`, `metrics.*` нужен restart.
|
||||
|
||||
## Server env
|
||||
@@ -97,12 +108,14 @@ MONLET_EVENTS_RETENTION_MONTHS=36
|
||||
MONLET_EVENTS_FUTURE_PARTITIONS=3
|
||||
MONLET_PARTITION_MAINTENANCE_INTERVAL_SEC=3600
|
||||
MONLET_OUTBOX_RETENTION_MAX_ROWS=50000
|
||||
MONLET_OUTBOX_PRUNE_BATCH_SIZE=5000
|
||||
|
||||
MONLET_EVENT_DEDUP_RETENTION_DAYS=30
|
||||
MONLET_EVENT_DEDUP_PRUNE_BATCH_SIZE=5000
|
||||
|
||||
MONLET_MAX_PENDING_AGENTS=10000
|
||||
MONLET_PENDING_AGENT_TTL_DAYS=7
|
||||
MONLET_PENDING_AGENT_PRUNE_BATCH_SIZE=1000
|
||||
|
||||
MONLET_ENABLE_NOTIFIER_WORKER=true
|
||||
MONLET_NOTIFIER_TICK_SEC=5
|
||||
@@ -154,10 +167,12 @@ Liveness-детектор:
|
||||
- `MONLET_EVENTS_FUTURE_PARTITIONS` — сколько будущих месячных партиций создавать заранее (по умолчанию 3).
|
||||
- `MONLET_PARTITION_MAINTENANCE_INTERVAL_SEC` — период обслуживания партиций и prune (по умолчанию 3600).
|
||||
- `MONLET_OUTBOX_RETENTION_MAX_ROWS` — верхняя граница строк в outbox; лишние обрезаются (по умолчанию 50000).
|
||||
- `MONLET_OUTBOX_PRUNE_BATCH_SIZE` — размер пакета при очистке terminal outbox-записей (по умолчанию 5000).
|
||||
- `MONLET_EVENT_DEDUP_RETENTION_DAYS` — хранение записей идемпотентности `event_id` (по умолчанию 30).
|
||||
- `MONLET_EVENT_DEDUP_PRUNE_BATCH_SIZE` — размер пакета при очистке dedup-записей (по умолчанию 5000).
|
||||
- `MONLET_MAX_PENDING_AGENTS` — лимит новых непринятых агентов (по умолчанию 10000).
|
||||
- `MONLET_PENDING_AGENT_TTL_DAYS` — очистка старых pending-ключей (по умолчанию 7).
|
||||
- `MONLET_PENDING_AGENT_PRUNE_BATCH_SIZE` — размер пакета при очистке старых pending-ключей (по умолчанию 1000).
|
||||
|
||||
Notifier worker:
|
||||
|
||||
@@ -181,21 +196,29 @@ Notifier каналы:
|
||||
|
||||
## Web env
|
||||
|
||||
Веб-UI читает auth-настройки из env. Без них read-only страницы доступны локально, но admission-мутации возвращают `403`.
|
||||
Веб-UI читает API/auth/timezone-настройки из env. Без auth-настроек read-only страницы доступны локально, но admission-мутации возвращают `403`.
|
||||
|
||||
```env
|
||||
MONLET_API_BASE_URL=http://127.0.0.1:8000
|
||||
MONLET_API_TOKEN=replace-with-server-token
|
||||
MONLET_WEB_TIME_ZONE=UTC
|
||||
MONLET_WEB_AUTH_USERNAME=admin
|
||||
MONLET_WEB_AUTH_PASSWORD=replace-with-strong-password
|
||||
MONLET_WEB_SESSION_SECRET=replace-with-random-32-bytes-min
|
||||
MONLET_WEB_SESSION_TTL_SEC=604800
|
||||
NEXT_PUBLIC_MONLET_POLL_MS=10000
|
||||
# или вместо локального логина — доверие reverse-proxy:
|
||||
# MONLET_WEB_TRUST_PROXY_AUTH=true
|
||||
```
|
||||
|
||||
- `MONLET_API_BASE_URL` — базовый URL сервера для server-side fetch.
|
||||
- `MONLET_API_TOKEN` — Bearer-токен для запросов UI к серверу; в браузер не передаётся.
|
||||
- `MONLET_WEB_TIME_ZONE` — IANA timezone для отображения дат (по умолчанию `UTC`).
|
||||
- `MONLET_WEB_AUTH_USERNAME` / `MONLET_WEB_AUTH_PASSWORD` — учётные данные локального cookie-логина.
|
||||
- `MONLET_WEB_SESSION_SECRET` — секрет подписи сессии, минимум 32 байта; короче — UI считает auth неправильно настроенным.
|
||||
- `MONLET_WEB_SESSION_TTL_SEC` — срок жизни сессии (по умолчанию 604800 = 7 дней).
|
||||
- `MONLET_WEB_TRUST_PROXY_AUTH` — `true` включает доверие заголовку `X-Forwarded-User` от reverse-proxy. Несовместимо с локальным логином: одновременная настройка обоих режимов — ошибка (UI отвечает `500`).
|
||||
- `NEXT_PUBLIC_MONLET_POLL_MS` — интервал браузерного polling в миллисекундах (по умолчанию 10000); для production-сборок Next.js это build-time переменная.
|
||||
|
||||
## Минимальная схема
|
||||
|
||||
|
||||
@@ -110,8 +110,8 @@ The server owns incident and notification lifecycle. The agent sends facts only.
|
||||
The web UI is read-only for monitoring state; agent admission and blacklist
|
||||
controls are the v1 mutation surface.
|
||||
|
||||
Public API contracts live in `api/openapi.yaml`. Design and operations docs live
|
||||
under `docs/`.
|
||||
Public API contracts live in `api/openapi.yaml`. Design and operations docs start
|
||||
at `docs/index.md`.
|
||||
|
||||
## Non-Goals
|
||||
|
||||
@@ -134,6 +134,7 @@ Start with:
|
||||
3. [server/README.md](server/README.md)
|
||||
4. [web/README.md](web/README.md)
|
||||
5. [api/README.md](api/README.md)
|
||||
6. [docs/index.md](docs/index.md)
|
||||
|
||||
Dependency caches and virtual environments are kept inside the repository. See
|
||||
the component READMEs for current local development commands.
|
||||
|
||||
@@ -112,7 +112,7 @@ Server владеет lifecycle инцидентов и уведомлений.
|
||||
controls - mutation surface для v1.
|
||||
|
||||
Публичный API contract лежит в `api/openapi.yaml`. Design и operations docs
|
||||
лежат в `docs/`.
|
||||
начинаются с `docs/index.md`.
|
||||
|
||||
## Не цели
|
||||
|
||||
@@ -135,6 +135,7 @@ Monlet 0.1.0 намеренно не является:
|
||||
3. [server/README.md](server/README.md)
|
||||
4. [web/README.md](web/README.md)
|
||||
5. [api/README.md](api/README.md)
|
||||
6. [docs/index.md](docs/index.md)
|
||||
|
||||
Dependency caches и virtual environments хранятся внутри репозитория. Актуальные
|
||||
команды local development описаны в component README.
|
||||
|
||||
@@ -14,7 +14,7 @@ Do not use `go install` or `go env -w`. Module cache, build cache, and tool bina
|
||||
|
||||
## Configuration
|
||||
|
||||
See `config.example.toml`. Required fields: top-level `state_dir`, at least one `[[checks]]` (`id`, `command`, `interval`, `timeout`), and at least one output enabled: `server.enabled = true` or `metrics.enabled = true`. `agent_id` is optional; if omitted, it defaults to the OS hostname. `hostname` is not a config key.
|
||||
See `config.example.toml`. Required fields: top-level `state_dir`, at least one `[[checks]]` (`id`, `command`, `timeout`, and exactly one of `interval` or `cron`), and at least one output enabled: `server.enabled = true` or `metrics.enabled = true`. `agent_id` is optional; if omitted, it defaults to the OS hostname. `hostname` is not a config key.
|
||||
|
||||
| Key | Default | Notes |
|
||||
|---|---|---|
|
||||
@@ -25,10 +25,12 @@ See `config.example.toml`. Required fields: top-level `state_dir`, at least one
|
||||
| `server.batch_interval` | `10s` | events flush cadence |
|
||||
| `metrics.listen` | `127.0.0.1:9465` | low-cardinality only |
|
||||
| `checks[].command` | required | Shell string executed as `/bin/sh -c`. Both forms accepted: `command = "check_disk --warn 80 --crit 90"` and triple-quoted `command = '''...'''`. Argv arrays are rejected. See [Command security contract](#command-security-contract). |
|
||||
| `checks[].interval` | required unless `cron` is set | Go duration such as `60s`, `5m`, or `1h`. Interval checks run once immediately on start/reload, then on the interval. |
|
||||
| `checks[].cron` | required unless `interval` is set | Standard 5-field cron expression, macros such as `@hourly`/`@daily`, and optional `CRON_TZ=...`. Seconds fields, `@every`, and `@reboot` are rejected. Cron checks wait for the next scheduled slot; they do not run immediately on start/reload. |
|
||||
| `checks[].notifications_enabled` | `true` | Set `false` for checks that must not create server notifications. |
|
||||
| `checks[].resource_limits` | `{}` | Optional per-check best-effort `ulimit` values: `cpu_time`, `memory`, `open_files`. |
|
||||
|
||||
Examples — one-line and multi-line forms:
|
||||
Examples — interval, cron, and multi-line forms:
|
||||
|
||||
```toml
|
||||
[[checks]]
|
||||
@@ -37,6 +39,12 @@ command = "test $(cut -d. -f1 /proc/uptime) -gt 60"
|
||||
interval = "60s"
|
||||
timeout = "5s"
|
||||
|
||||
[[checks]]
|
||||
id = "daily_backup"
|
||||
command = "/usr/local/lib/monlet/check_backup_freshness.sh"
|
||||
cron = "CRON_TZ=UTC 15 6 * * *"
|
||||
timeout = "2m"
|
||||
|
||||
[[checks]]
|
||||
id = "disk_root"
|
||||
command = '''
|
||||
@@ -81,16 +89,17 @@ executes it as `/bin/sh -c <command>` on the host where the agent runs.
|
||||
Send `SIGHUP` or run `systemctl reload monlet-agent` to reload the local TOML file.
|
||||
|
||||
- Reload first validates the full new config. On error, the old config keeps running.
|
||||
- Hot-reloadable: labels and checks, including `command`, `interval`, `timeout`,
|
||||
`notifications_enabled`, `dedupe_key`, and `resource_limits`.
|
||||
- Hot-reloadable: labels and checks, including `command`, `interval`, `cron`,
|
||||
`timeout`, `notifications_enabled`, `dedupe_key`, and `resource_limits`.
|
||||
- Restart required: `agent_id`, `state_dir`, `server.*`, and `metrics.*`.
|
||||
- A check already running during reload is allowed to finish under its old config.
|
||||
Removed or changed checks do not start new old-config runs; changed checks start
|
||||
with the new config after the in-flight run finishes.
|
||||
Removed or changed checks do not start new old-config runs. Changed interval
|
||||
checks start with the new config after the in-flight run finishes; changed cron
|
||||
checks wait for the next cron slot.
|
||||
|
||||
## Behaviour
|
||||
|
||||
- Per-check scheduler with no-overlap: a tick is skipped if the previous run is still in flight (`monlet_agent_check_skipped_total`).
|
||||
- Per-check scheduler with no-overlap: a tick/cron slot is skipped if the previous run is still in flight (`monlet_agent_check_skipped_total`).
|
||||
- Check commands run through `/bin/sh -c`; the old argv-array form is intentionally unsupported.
|
||||
- Timeout via `exec.CommandContext` with `Setpgid` so the process group is killed on expiry; timeout → `critical`.
|
||||
- Exit code mapping: 0=ok, 1=warning, 2=critical, 3+=unknown (ADR-0005).
|
||||
@@ -115,7 +124,7 @@ Exposed on `metrics.listen` at `/metrics`. Labels are static (`check_id`, `statu
|
||||
- `monlet_agent_check_exit_code{check_id}`
|
||||
- `monlet_agent_check_last_run_timestamp_seconds{check_id}`
|
||||
- `monlet_agent_check_last_success_timestamp_seconds{check_id}`
|
||||
- `monlet_agent_check_interval_seconds{check_id}`
|
||||
- `monlet_agent_check_interval_seconds{check_id}` (interval checks only)
|
||||
- `monlet_agent_checks_configured`
|
||||
- `monlet_agent_config_load_success`
|
||||
- `monlet_agent_config_reload_total{result}`
|
||||
|
||||
@@ -32,3 +32,11 @@ name = "Uptime probe"
|
||||
command = "test $(cut -d. -f1 /proc/uptime) -gt 60"
|
||||
interval = "60s"
|
||||
timeout = "5s"
|
||||
|
||||
# Cron-style schedules are also supported. Use exactly one of interval or cron.
|
||||
# [[checks]]
|
||||
# id = "daily_backup"
|
||||
# name = "Daily backup freshness"
|
||||
# command = "/usr/local/lib/monlet/check_backup_freshness.sh"
|
||||
# cron = "CRON_TZ=UTC 15 6 * * *"
|
||||
# timeout = "2m"
|
||||
|
||||
@@ -6,6 +6,8 @@ require (
|
||||
github.com/BurntSushi/toml v1.6.0
|
||||
github.com/google/uuid v1.6.0
|
||||
github.com/prometheus/client_golang v1.23.2
|
||||
github.com/prometheus/client_model v0.6.2
|
||||
github.com/robfig/cron/v3 v3.0.1
|
||||
)
|
||||
|
||||
require (
|
||||
@@ -13,7 +15,6 @@ require (
|
||||
github.com/cespare/xxhash/v2 v2.3.0 // indirect
|
||||
github.com/kr/text v0.2.0 // indirect
|
||||
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
|
||||
github.com/prometheus/client_model v0.6.2 // indirect
|
||||
github.com/prometheus/common v0.66.1 // indirect
|
||||
github.com/prometheus/procfs v0.16.1 // indirect
|
||||
go.yaml.in/yaml/v2 v2.4.2 // indirect
|
||||
|
||||
@@ -31,6 +31,8 @@ github.com/prometheus/common v0.66.1 h1:h5E0h5/Y8niHc5DlaLlWLArTQI7tMrsfQjHV+d9Z
|
||||
github.com/prometheus/common v0.66.1/go.mod h1:gcaUsgf3KfRSwHY4dIMXLPV0K/Wg1oZ8+SbZk/HH/dA=
|
||||
github.com/prometheus/procfs v0.16.1 h1:hZ15bTNuirocR6u0JZ6BAHHmwS1p8B4P6MRqxtzMyRg=
|
||||
github.com/prometheus/procfs v0.16.1/go.mod h1:teAbpZRB1iIAJYREa1LsoWUXykVXA1KlTmWl8x/U+Is=
|
||||
github.com/robfig/cron/v3 v3.0.1 h1:WdRxkvbJztn8LMz/QEvLN5sBU+xKpSqwwUO1Pjr4qDs=
|
||||
github.com/robfig/cron/v3 v3.0.1/go.mod h1:eQICP3HwyT7UooqI/z+Ov+PtYAWygg1TEWWzGIFLtro=
|
||||
github.com/rogpeppe/go-internal v1.10.0 h1:TMyTOH3F/DB16zRVcYyreMH6GnZZrwQVAoYjRBZyWFQ=
|
||||
github.com/rogpeppe/go-internal v1.10.0/go.mod h1:UQnix2H7Ngw/k4C5ijL5+65zddjncjaFoBhdsK/akog=
|
||||
github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U=
|
||||
|
||||
@@ -47,9 +47,7 @@ func (a *App) Run(ctx context.Context) error {
|
||||
a.Metrics.ConfigLoadSuccess.Set(1)
|
||||
a.Metrics.ChecksConfigured.Set(float64(len(cfg.Checks)))
|
||||
a.Metrics.BuildInfo.WithLabelValues(a.Version, boolLabel(cfg.PushesToServer()), boolLabel(cfg.ExposesMetrics())).Set(1)
|
||||
for _, c := range cfg.Checks {
|
||||
a.Metrics.CheckInterval.WithLabelValues(c.ID).Set(c.Interval.Duration.Seconds())
|
||||
}
|
||||
a.recordCheckScheduleMetrics(cfg.Checks)
|
||||
|
||||
var wg sync.WaitGroup
|
||||
|
||||
@@ -256,9 +254,7 @@ func (a *App) Reload(newCfg *config.Config, newAgentID string) error {
|
||||
a.Metrics.ConfigReloadTotal.WithLabelValues("success").Inc()
|
||||
a.Metrics.ConfigLastReloadSuccess.Set(float64(time.Now().Unix()))
|
||||
a.Metrics.ChecksConfigured.Set(float64(len(newCfg.Checks)))
|
||||
for _, c := range newCfg.Checks {
|
||||
a.Metrics.CheckInterval.WithLabelValues(c.ID).Set(c.Interval.Duration.Seconds())
|
||||
}
|
||||
a.recordCheckScheduleMetrics(newCfg.Checks)
|
||||
|
||||
a.schedulerMu.RLock()
|
||||
mgr := a.scheduler
|
||||
@@ -275,6 +271,16 @@ func (a *App) RecordReloadFailure() {
|
||||
a.Metrics.ConfigReloadTotal.WithLabelValues("failure").Inc()
|
||||
}
|
||||
|
||||
func (a *App) recordCheckScheduleMetrics(checks []config.CheckConfig) {
|
||||
for _, c := range checks {
|
||||
if c.UsesInterval() {
|
||||
a.Metrics.CheckInterval.WithLabelValues(c.ID).Set(c.Interval.Duration.Seconds())
|
||||
} else {
|
||||
a.Metrics.CheckInterval.DeleteLabelValues(c.ID)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func (a *App) config() *config.Config {
|
||||
a.cfgMu.RLock()
|
||||
defer a.cfgMu.RUnlock()
|
||||
|
||||
@@ -17,6 +17,7 @@ import (
|
||||
"github.com/monlet/agent/internal/config"
|
||||
"github.com/monlet/agent/internal/metrics"
|
||||
"github.com/monlet/agent/internal/spool"
|
||||
dto "github.com/prometheus/client_model/go"
|
||||
)
|
||||
|
||||
func TestPrometheusOnlyDoesNotPush(t *testing.T) {
|
||||
@@ -257,6 +258,92 @@ func TestReloadAcceptsLabelsAndChecks(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestReloadUpdatesIntervalMetricForCron(t *testing.T) {
|
||||
oldCfg := &config.Config{
|
||||
AgentID: "a1",
|
||||
Hostname: "h",
|
||||
StateDir: "/tmp/old",
|
||||
Server: config.ServerConfig{
|
||||
Enabled: boolPtr(true),
|
||||
URL: "http://server",
|
||||
HeartbeatInterval: config.Duration{Duration: 10 * time.Second},
|
||||
BatchInterval: config.Duration{Duration: 10 * time.Second},
|
||||
},
|
||||
Metrics: config.MetricsConfig{Enabled: false},
|
||||
Checks: []config.CheckConfig{{
|
||||
ID: "c1",
|
||||
Command: "true",
|
||||
Interval: config.Duration{Duration: time.Second},
|
||||
Timeout: config.Duration{Duration: time.Second},
|
||||
}},
|
||||
}
|
||||
a := &App{
|
||||
Cfg: oldCfg,
|
||||
AgentID: "a1",
|
||||
Metrics: metrics.New(),
|
||||
}
|
||||
a.recordCheckScheduleMetrics(oldCfg.Checks)
|
||||
if got, ok := checkIntervalMetric(t, a, "c1"); !ok || got != 1 {
|
||||
t.Fatalf("interval metric missing before reload: value=%v ok=%v", got, ok)
|
||||
}
|
||||
|
||||
cronCfg := *oldCfg
|
||||
cronCfg.Checks = []config.CheckConfig{{
|
||||
ID: "c1",
|
||||
Command: "true",
|
||||
Cron: "0 * * * *",
|
||||
Timeout: config.Duration{Duration: time.Second},
|
||||
}}
|
||||
if err := a.Reload(&cronCfg, "a1"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got, ok := checkIntervalMetric(t, a, "c1"); ok {
|
||||
t.Fatalf("interval metric must be deleted for cron check, got %v", got)
|
||||
}
|
||||
|
||||
intervalCfg := cronCfg
|
||||
intervalCfg.Checks = []config.CheckConfig{{
|
||||
ID: "c1",
|
||||
Command: "true",
|
||||
Interval: config.Duration{Duration: 2 * time.Second},
|
||||
Timeout: config.Duration{Duration: time.Second},
|
||||
}}
|
||||
if err := a.Reload(&intervalCfg, "a1"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got, ok := checkIntervalMetric(t, a, "c1"); !ok || got != 2 {
|
||||
t.Fatalf("interval metric missing after interval reload: value=%v ok=%v", got, ok)
|
||||
}
|
||||
}
|
||||
|
||||
func checkIntervalMetric(t *testing.T, a *App, checkID string) (float64, bool) {
|
||||
t.Helper()
|
||||
families, err := a.Metrics.Registry.Gather()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, family := range families {
|
||||
if family.GetName() != "monlet_agent_check_interval_seconds" {
|
||||
continue
|
||||
}
|
||||
for _, metric := range family.GetMetric() {
|
||||
if labelValue(metric.GetLabel(), "check_id") == checkID && metric.GetGauge() != nil {
|
||||
return metric.GetGauge().GetValue(), true
|
||||
}
|
||||
}
|
||||
}
|
||||
return 0, false
|
||||
}
|
||||
|
||||
func labelValue(labels []*dto.LabelPair, name string) string {
|
||||
for _, label := range labels {
|
||||
if label.GetName() == name {
|
||||
return label.GetValue()
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func boolPtr(v bool) *bool {
|
||||
return &v
|
||||
}
|
||||
|
||||
@@ -11,6 +11,7 @@ import (
|
||||
"unicode/utf8"
|
||||
|
||||
"github.com/BurntSushi/toml"
|
||||
"github.com/robfig/cron/v3"
|
||||
)
|
||||
|
||||
var idPattern = regexp.MustCompile(`^[A-Za-z0-9._:-]+$`)
|
||||
@@ -58,6 +59,7 @@ type CheckConfig struct {
|
||||
Name string `toml:"name"`
|
||||
Command string `toml:"command"`
|
||||
Interval Duration `toml:"interval"`
|
||||
Cron string `toml:"cron"`
|
||||
Timeout Duration `toml:"timeout"`
|
||||
NotificationsEnabled *bool `toml:"notifications_enabled"`
|
||||
DedupeKey string `toml:"dedupe_key"`
|
||||
@@ -211,14 +213,24 @@ func (c *Config) Validate() error {
|
||||
if strings.TrimSpace(ch.Command) == "" {
|
||||
return fmt.Errorf("check %q: command is required", ch.ID)
|
||||
}
|
||||
if ch.Interval.Duration <= 0 {
|
||||
return fmt.Errorf("check %q: interval must be > 0", ch.ID)
|
||||
}
|
||||
if ch.Timeout.Duration <= 0 {
|
||||
return fmt.Errorf("check %q: timeout must be > 0", ch.ID)
|
||||
}
|
||||
if ch.Timeout.Duration > ch.Interval.Duration {
|
||||
if ch.Interval.Duration < 0 {
|
||||
return fmt.Errorf("check %q: interval must be > 0", ch.ID)
|
||||
}
|
||||
ch.Cron = strings.TrimSpace(ch.Cron)
|
||||
hasInterval := ch.Interval.Duration > 0
|
||||
hasCron := ch.Cron != ""
|
||||
switch {
|
||||
case hasInterval == hasCron:
|
||||
return fmt.Errorf("check %q: exactly one of interval or cron is required", ch.ID)
|
||||
case hasInterval && ch.Timeout.Duration > ch.Interval.Duration:
|
||||
return fmt.Errorf("check %q: timeout must be <= interval", ch.ID)
|
||||
case hasCron:
|
||||
if _, err := ParseCronSchedule(ch.Cron); err != nil {
|
||||
return fmt.Errorf("check %q: invalid cron: %w", ch.ID, err)
|
||||
}
|
||||
}
|
||||
if len(ch.DedupeKey) > 256 {
|
||||
return fmt.Errorf("check %q: dedupe_key too long", ch.ID)
|
||||
@@ -253,6 +265,42 @@ func (c CheckConfig) NotificationsOn() bool {
|
||||
return c.NotificationsEnabled == nil || *c.NotificationsEnabled
|
||||
}
|
||||
|
||||
func (c CheckConfig) UsesInterval() bool {
|
||||
return c.Interval.Duration > 0
|
||||
}
|
||||
|
||||
func (c CheckConfig) UsesCron() bool {
|
||||
return strings.TrimSpace(c.Cron) != ""
|
||||
}
|
||||
|
||||
func ParseCronSchedule(spec string) (cron.Schedule, error) {
|
||||
spec = strings.TrimSpace(spec)
|
||||
if spec == "" {
|
||||
return nil, fmt.Errorf("cron is empty")
|
||||
}
|
||||
withoutTZ := specWithoutTZ(spec)
|
||||
lower := strings.ToLower(withoutTZ)
|
||||
if strings.HasPrefix(lower, "@every") {
|
||||
return nil, fmt.Errorf("@every is not supported; use interval")
|
||||
}
|
||||
if lower == "@reboot" {
|
||||
return nil, fmt.Errorf("@reboot is not supported")
|
||||
}
|
||||
return cron.ParseStandard(spec)
|
||||
}
|
||||
|
||||
func specWithoutTZ(spec string) string {
|
||||
fields := strings.Fields(spec)
|
||||
if len(fields) == 0 {
|
||||
return ""
|
||||
}
|
||||
first := fields[0]
|
||||
if strings.HasPrefix(first, "CRON_TZ=") || strings.HasPrefix(first, "TZ=") {
|
||||
return strings.ToLower(strings.Join(fields[1:], " "))
|
||||
}
|
||||
return strings.ToLower(strings.Join(fields, " "))
|
||||
}
|
||||
|
||||
func validateResourceLimits(checkID string, limits ResourceLimits) error {
|
||||
if limits.CPUTime.Duration < 0 {
|
||||
return fmt.Errorf("check %q: resource_limits.cpu_time must be > 0", checkID)
|
||||
|
||||
@@ -48,9 +48,84 @@ func TestLoadMinimal(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestLoadCronCheck(t *testing.T) {
|
||||
c, err := Load(writeTmp(t, `
|
||||
agent_id = "host-1"
|
||||
state_dir = "/tmp/x"
|
||||
|
||||
[server]
|
||||
enabled = true
|
||||
url = "http://localhost"
|
||||
|
||||
[[checks]]
|
||||
id = "c1"
|
||||
command = "true"
|
||||
cron = "CRON_TZ=UTC 0 6 * * *"
|
||||
timeout = "5m"
|
||||
`))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !c.Checks[0].UsesCron() || c.Checks[0].UsesInterval() {
|
||||
t.Fatalf("unexpected schedule mode: %+v", c.Checks[0])
|
||||
}
|
||||
}
|
||||
|
||||
func TestValidateCheckScheduleChoice(t *testing.T) {
|
||||
for name, body := range map[string]string{
|
||||
"both_interval_and_cron": `
|
||||
interval = "10s"
|
||||
cron = "0 * * * *"
|
||||
timeout = "5s"
|
||||
`,
|
||||
"neither_interval_nor_cron": `
|
||||
timeout = "5s"
|
||||
`,
|
||||
"invalid_cron": `
|
||||
cron = "not a cron"
|
||||
timeout = "5s"
|
||||
`,
|
||||
"seconds_cron": `
|
||||
cron = "0 0 * * * *"
|
||||
timeout = "5s"
|
||||
`,
|
||||
"every_cron": `
|
||||
cron = "@every 10s"
|
||||
timeout = "5s"
|
||||
`,
|
||||
"tz_every_cron": `
|
||||
cron = "CRON_TZ=UTC @every 10s"
|
||||
timeout = "5s"
|
||||
`,
|
||||
"reboot_cron": `
|
||||
cron = "@reboot"
|
||||
timeout = "5s"
|
||||
`,
|
||||
} {
|
||||
t.Run(name, func(t *testing.T) {
|
||||
_, err := Load(writeTmp(t, `
|
||||
agent_id = "host-1"
|
||||
state_dir = "/tmp/x"
|
||||
|
||||
[server]
|
||||
enabled = true
|
||||
url = "http://localhost"
|
||||
|
||||
[[checks]]
|
||||
id = "c1"
|
||||
command = "true"
|
||||
`+body+`
|
||||
`))
|
||||
if err == nil {
|
||||
t.Fatal("expected schedule validation error")
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestLoadResourceLimits(t *testing.T) {
|
||||
c, err := Load(writeTmp(t, `
|
||||
agent_id = "host-1"
|
||||
agent_id = "host-1"
|
||||
state_dir = "/tmp/x"
|
||||
|
||||
[server]
|
||||
|
||||
@@ -11,6 +11,8 @@ import (
|
||||
"github.com/monlet/agent/internal/runner"
|
||||
)
|
||||
|
||||
var parseCronSchedule = config.ParseCronSchedule
|
||||
|
||||
// Event is a contract-shaped check result emitted by the scheduler.
|
||||
type Event struct {
|
||||
EventID string `json:"event_id"`
|
||||
@@ -207,8 +209,10 @@ func (w *checkWorker) loop(initial config.CheckConfig) {
|
||||
w.wg.Done()
|
||||
}()
|
||||
cfg := initial
|
||||
t := time.NewTicker(cfg.Interval.Duration)
|
||||
t := time.NewTimer(time.Hour)
|
||||
stopTimer(t)
|
||||
defer t.Stop()
|
||||
var timerCh <-chan time.Time
|
||||
|
||||
running := false
|
||||
stopping := false
|
||||
@@ -230,7 +234,26 @@ func (w *checkWorker) loop(initial config.CheckConfig) {
|
||||
}()
|
||||
}
|
||||
|
||||
start(cfg)
|
||||
arm := func(c config.CheckConfig, immediateInterval bool) bool {
|
||||
delay, err := nextDelay(c, time.Now(), immediateInterval)
|
||||
if err != nil {
|
||||
return false
|
||||
}
|
||||
resetTimer(t, delay)
|
||||
timerCh = t.C
|
||||
return true
|
||||
}
|
||||
disarm := func() {
|
||||
stopTimer(t)
|
||||
timerCh = nil
|
||||
}
|
||||
|
||||
if !arm(cfg, true) {
|
||||
if w.hooks.OnStopped != nil {
|
||||
w.hooks.OnStopped(cfg.ID)
|
||||
}
|
||||
return
|
||||
}
|
||||
|
||||
for {
|
||||
select {
|
||||
@@ -257,11 +280,21 @@ func (w *checkWorker) loop(initial config.CheckConfig) {
|
||||
continue
|
||||
}
|
||||
cfg = next
|
||||
t.Reset(cfg.Interval.Duration)
|
||||
if cfg.UsesCron() {
|
||||
runAfterCurrent = false
|
||||
if !arm(cfg, false) {
|
||||
return
|
||||
}
|
||||
continue
|
||||
}
|
||||
if running {
|
||||
runAfterCurrent = true
|
||||
disarm()
|
||||
} else {
|
||||
start(cfg)
|
||||
if !arm(cfg, false) {
|
||||
return
|
||||
}
|
||||
}
|
||||
case <-w.doneCh:
|
||||
running = false
|
||||
@@ -281,26 +314,35 @@ func (w *checkWorker) loop(initial config.CheckConfig) {
|
||||
}
|
||||
if runAfterCurrent {
|
||||
runAfterCurrent = false
|
||||
startAfterDrain := true
|
||||
for {
|
||||
select {
|
||||
case next := <-w.updateCh:
|
||||
if !reflect.DeepEqual(cfg, next) {
|
||||
cfg = next
|
||||
t.Reset(cfg.Interval.Duration)
|
||||
startAfterDrain = !cfg.UsesCron()
|
||||
}
|
||||
default:
|
||||
start(cfg)
|
||||
if startAfterDrain {
|
||||
start(cfg)
|
||||
}
|
||||
if !arm(cfg, false) {
|
||||
return
|
||||
}
|
||||
goto nextLoop
|
||||
}
|
||||
}
|
||||
nextLoop:
|
||||
continue
|
||||
}
|
||||
case <-t.C:
|
||||
case <-timerCh:
|
||||
if stopping {
|
||||
continue
|
||||
}
|
||||
start(cfg)
|
||||
if !arm(cfg, false) {
|
||||
return
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -358,3 +400,41 @@ func incidentKey(agentID string, c config.CheckConfig) string {
|
||||
}
|
||||
return agentID + ":" + c.ID
|
||||
}
|
||||
|
||||
func nextDelay(c config.CheckConfig, now time.Time, immediateInterval bool) (time.Duration, error) {
|
||||
if c.UsesCron() {
|
||||
schedule, err := parseCronSchedule(c.Cron)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
return delayUntil(now, schedule.Next(now)), nil
|
||||
}
|
||||
if immediateInterval {
|
||||
return 0, nil
|
||||
}
|
||||
return c.Interval.Duration, nil
|
||||
}
|
||||
|
||||
func delayUntil(now, next time.Time) time.Duration {
|
||||
if next.Before(now) {
|
||||
return 0
|
||||
}
|
||||
return next.Sub(now)
|
||||
}
|
||||
|
||||
func resetTimer(t *time.Timer, d time.Duration) {
|
||||
if d < 0 {
|
||||
d = 0
|
||||
}
|
||||
stopTimer(t)
|
||||
t.Reset(d)
|
||||
}
|
||||
|
||||
func stopTimer(t *time.Timer) {
|
||||
if !t.Stop() {
|
||||
select {
|
||||
case <-t.C:
|
||||
default:
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -8,6 +8,7 @@ import (
|
||||
"time"
|
||||
|
||||
"github.com/monlet/agent/internal/config"
|
||||
"github.com/robfig/cron/v3"
|
||||
)
|
||||
|
||||
func runScheduler(ctx context.Context, agentID string, checks []config.CheckConfig, out chan<- Event, h Hooks) {
|
||||
@@ -27,6 +28,32 @@ func mkCheck(id, cmd string, interval, timeout time.Duration) config.CheckConfig
|
||||
}
|
||||
}
|
||||
|
||||
type fakeCronSchedule struct {
|
||||
delay time.Duration
|
||||
}
|
||||
|
||||
func (s fakeCronSchedule) Next(t time.Time) time.Time {
|
||||
return t.Add(s.delay)
|
||||
}
|
||||
|
||||
func withFakeCron(t *testing.T, delay time.Duration) {
|
||||
t.Helper()
|
||||
old := parseCronSchedule
|
||||
parseCronSchedule = func(string) (cron.Schedule, error) {
|
||||
return fakeCronSchedule{delay: delay}, nil
|
||||
}
|
||||
t.Cleanup(func() { parseCronSchedule = old })
|
||||
}
|
||||
|
||||
func mkCronCheck(id, cmd string, timeout time.Duration) config.CheckConfig {
|
||||
return config.CheckConfig{
|
||||
ID: id,
|
||||
Command: cmd,
|
||||
Cron: "0 * * * *",
|
||||
Timeout: config.Duration{Duration: timeout},
|
||||
}
|
||||
}
|
||||
|
||||
func TestEmitsEvents(t *testing.T) {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 500*time.Millisecond)
|
||||
defer cancel()
|
||||
@@ -45,6 +72,41 @@ func TestEmitsEvents(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestCronDoesNotRunImmediately(t *testing.T) {
|
||||
withFakeCron(t, 200*time.Millisecond)
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 100*time.Millisecond)
|
||||
defer cancel()
|
||||
out := make(chan Event, 10)
|
||||
m := NewManager(ctx, "a1", out, Hooks{})
|
||||
m.Update([]config.CheckConfig{mkCronCheck("cron", "echo cron", 100*time.Millisecond)})
|
||||
select {
|
||||
case ev := <-out:
|
||||
t.Fatalf("cron check ran before next slot: %+v", ev)
|
||||
case <-ctx.Done():
|
||||
}
|
||||
m.Stop()
|
||||
m.Wait()
|
||||
}
|
||||
|
||||
func TestCronEmitsOnDueSlot(t *testing.T) {
|
||||
withFakeCron(t, 30*time.Millisecond)
|
||||
ctx, cancel := context.WithTimeout(context.Background(), time.Second)
|
||||
defer cancel()
|
||||
out := make(chan Event, 10)
|
||||
m := NewManager(ctx, "a1", out, Hooks{})
|
||||
m.Update([]config.CheckConfig{mkCronCheck("cron", "echo cron", 100*time.Millisecond)})
|
||||
select {
|
||||
case ev := <-out:
|
||||
if ev.CheckID != "cron" || ev.Status != "ok" {
|
||||
t.Fatalf("unexpected event: %+v", ev)
|
||||
}
|
||||
case <-ctx.Done():
|
||||
t.Fatal("no cron event")
|
||||
}
|
||||
m.Stop()
|
||||
m.Wait()
|
||||
}
|
||||
|
||||
func TestNoOverlapSkips(t *testing.T) {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 600*time.Millisecond)
|
||||
defer cancel()
|
||||
@@ -59,6 +121,23 @@ func TestNoOverlapSkips(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestCronNoOverlapSkips(t *testing.T) {
|
||||
withFakeCron(t, 50*time.Millisecond)
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 250*time.Millisecond)
|
||||
defer cancel()
|
||||
out := make(chan Event, 10)
|
||||
var skipped int32
|
||||
hooks := Hooks{OnSkipped: func(string) { atomic.AddInt32(&skipped, 1) }}
|
||||
m := NewManager(ctx, "a1", out, hooks)
|
||||
m.Update([]config.CheckConfig{mkCronCheck("cron", "sleep 0.2", 500*time.Millisecond)})
|
||||
<-ctx.Done()
|
||||
m.Stop()
|
||||
m.Wait()
|
||||
if atomic.LoadInt32(&skipped) == 0 {
|
||||
t.Fatal("expected at least one skipped cron slot")
|
||||
}
|
||||
}
|
||||
|
||||
func TestReportsFullEventChannel(t *testing.T) {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second)
|
||||
defer cancel()
|
||||
|
||||
@@ -1,9 +1,18 @@
|
||||
# Alertmanager
|
||||
|
||||
Placeholder for optional Alertmanager integration examples.
|
||||
Example Alertmanager configuration for the optional observability profile.
|
||||
|
||||
Planned examples:
|
||||
`alertmanager.yml` is intentionally safe by default: it routes to a `null`
|
||||
receiver, so alerts are grouped but dropped. Before using it outside local
|
||||
development, add a real receiver and set `route.receiver` to that receiver.
|
||||
|
||||
- receiver config for Monlet-originated alerts;
|
||||
- stable label conventions;
|
||||
- routing notes for avoiding duplicate notifications.
|
||||
Monlet can also send notifications to Alertmanager through the server notifier:
|
||||
|
||||
```env
|
||||
MONLET_NOTIFIER_ALERTMANAGER_ENABLED=true
|
||||
MONLET_NOTIFIER_ALERTMANAGER_URL=http://alertmanager:9093
|
||||
```
|
||||
|
||||
Keep labels stable and low-cardinality. The example groups by `alertname`,
|
||||
`agent_id`, and `check_id`; do not add timestamps, event IDs, or output text to
|
||||
Alertmanager labels.
|
||||
|
||||
@@ -1,11 +1,20 @@
|
||||
# Grafana
|
||||
|
||||
Placeholder for Grafana dashboard examples.
|
||||
Starter Grafana provisioning for the optional observability profile.
|
||||
|
||||
Planned dashboards:
|
||||
Files:
|
||||
|
||||
- agents alive/stale/dead;
|
||||
- check state counts;
|
||||
- open incidents;
|
||||
- notification outbox state;
|
||||
- server ingestion and notifier failures.
|
||||
- `provisioning/datasources/prometheus.yml` - adds Prometheus at `http://prometheus:9090`.
|
||||
- `provisioning/dashboards/monlet.yml` - loads dashboards from `/var/lib/grafana/dashboards`.
|
||||
- `dashboards/monlet-overview.json` - starter dashboard for agent status, open incidents, event ingest rate, and notification rate.
|
||||
|
||||
Run it with:
|
||||
|
||||
```sh
|
||||
cd deploy
|
||||
MONLET_AUTH_TOKEN=$(openssl rand -hex 16) docker compose --profile observability up
|
||||
```
|
||||
|
||||
Grafana is exposed at <http://127.0.0.1:3001>. The local default is
|
||||
`admin` / `admin` unless `GF_ADMIN_PASSWORD` is set. Change it before exposing
|
||||
Grafana outside local development.
|
||||
|
||||
@@ -1,9 +1,20 @@
|
||||
# Prometheus
|
||||
|
||||
Placeholder for Prometheus examples.
|
||||
Example scrape configuration for the optional observability profile in
|
||||
`deploy/docker-compose.yml`.
|
||||
|
||||
Planned examples:
|
||||
Run it with:
|
||||
|
||||
- scrape Monlet agent `/metrics`;
|
||||
- scrape Monlet server `/metrics`;
|
||||
- sample rules for checks that use `notifications_enabled = false`.
|
||||
```sh
|
||||
cd deploy
|
||||
MONLET_AUTH_TOKEN=$(openssl rand -hex 16) docker compose --profile observability up
|
||||
```
|
||||
|
||||
The example config scrapes:
|
||||
|
||||
- `server:8000/metrics` as `monlet-server`
|
||||
- `host.docker.internal:9100/metrics` as `monlet-agent`
|
||||
|
||||
Adjust the agent target for your environment. In the showcase stack one demo
|
||||
agent exposes metrics on host port `9465`; production agents usually expose
|
||||
`metrics.listen` only on the host or a private monitoring network.
|
||||
|
||||
16
docs/index.md
Normal file
16
docs/index.md
Normal file
@@ -0,0 +1,16 @@
|
||||
# Monlet Docs
|
||||
|
||||
This directory contains operator-facing docs that do not fit in the root or component READMEs.
|
||||
|
||||
## Start Here
|
||||
|
||||
- [First deployment checklist](ops/deployment-checklist.md) - step-by-step production-ish stand-up checklist.
|
||||
- [Backup and restore](ops/backup-restore.md) - PostgreSQL backup, retention, and disaster recovery notes.
|
||||
- [Token rotation](ops/token-rotation.md) - UI token overlap and agent-key rotation.
|
||||
- [Deployment notes](ops/deployment.md) - deployment shape and production boundaries.
|
||||
|
||||
## Conventions
|
||||
|
||||
- Component setup lives in `agent/README.md`, `server/README.md`, `web/README.md`, and `deploy/README.md`.
|
||||
- API contracts live in `api/openapi.yaml`; `api/README.md` only describes contract rules.
|
||||
- Keep docs factual and current. Avoid placeholder docs when a real config or script already exists.
|
||||
@@ -1,68 +0,0 @@
|
||||
# Deployment
|
||||
|
||||
This is an initial deployment plan, not a production runbook.
|
||||
|
||||
## Agent
|
||||
|
||||
Agent runs as a systemd service.
|
||||
|
||||
Expected paths:
|
||||
|
||||
- binary: `/usr/local/bin/monlet-agent`;
|
||||
- config: `/etc/monlet/agent.toml`;
|
||||
- state: `/var/lib/monlet-agent`;
|
||||
- logs: journald.
|
||||
|
||||
The agent should support graceful shutdown and avoid overlapping runs for the same check.
|
||||
|
||||
## Server
|
||||
|
||||
Server runs in Docker.
|
||||
|
||||
Production storage:
|
||||
|
||||
- PostgreSQL.
|
||||
|
||||
Local/demo storage:
|
||||
|
||||
- SQLite may be supported later, but must be documented as non-production.
|
||||
|
||||
Required deployment features:
|
||||
|
||||
- TLS-ready reverse proxy;
|
||||
- agent API tokens;
|
||||
- request size limits;
|
||||
- structured logs;
|
||||
- `/health`;
|
||||
- `/ready`;
|
||||
- `/metrics`.
|
||||
|
||||
Docker image rule:
|
||||
|
||||
- build server dependencies inside the image;
|
||||
- do not require Python packages installed on the host;
|
||||
- do not mount host virtualenvs into the container;
|
||||
- local non-Docker development uses `server/.venv` only.
|
||||
|
||||
## Web
|
||||
|
||||
Web is read-only for monitoring state in v1. Agent admission and blacklist controls are the explicit write exception.
|
||||
|
||||
Auth plan:
|
||||
|
||||
- local web login or equivalent gateway auth through a trusted reverse proxy;
|
||||
- no full RBAC in v1.
|
||||
|
||||
## Observability
|
||||
|
||||
Optional examples:
|
||||
|
||||
- Prometheus scrape config;
|
||||
- Alertmanager route config;
|
||||
- Grafana dashboard stub.
|
||||
|
||||
Metrics must avoid high-cardinality labels.
|
||||
|
||||
## Backups
|
||||
|
||||
PostgreSQL backup/restore docs are required before first production use.
|
||||
66
docs/ops/deployment.md
Normal file
66
docs/ops/deployment.md
Normal file
@@ -0,0 +1,66 @@
|
||||
# Deployment Notes
|
||||
|
||||
These notes describe the intended production shape. Use
|
||||
[`docs/ops/deployment-checklist.md`](deployment-checklist.md) for the
|
||||
step-by-step first stand-up checklist.
|
||||
|
||||
## Agent
|
||||
|
||||
The agent runs on monitored hosts as a systemd service.
|
||||
|
||||
Expected paths:
|
||||
|
||||
- binary: `/usr/local/bin/monlet-agent`
|
||||
- config: `/etc/monlet/agent.toml`
|
||||
- state: `/var/lib/monlet-agent`
|
||||
- logs: journald
|
||||
|
||||
The agent reads only local config. It does not accept remote command execution or
|
||||
remote config push from the server. Per-check scheduling is local to the agent,
|
||||
and overlapping runs for the same check are skipped.
|
||||
|
||||
## Server
|
||||
|
||||
The server runs as a containerized FastAPI application backed by PostgreSQL 16+.
|
||||
It owns inventory, current check state, incidents, notification outbox state,
|
||||
retention, and public `/api/v1` contracts.
|
||||
|
||||
Required production boundaries:
|
||||
|
||||
- TLS and rate limiting at a reverse proxy
|
||||
- PostgreSQL as canonical storage
|
||||
- `MONLET_AUTH_TOKEN` stored in a secret manager
|
||||
- request body limits enabled
|
||||
- structured logs with auth material redacted
|
||||
- `/api/v1/health`, `/api/v1/ready`, and `/metrics` exposed for operations
|
||||
|
||||
Docker images install dependencies inside the image. Do not mount host virtual
|
||||
environments into server containers.
|
||||
|
||||
## Web
|
||||
|
||||
The web app is an operator dashboard. Monitoring pages are read-only; agent
|
||||
admission and blacklist controls are the explicit v1 write surface.
|
||||
|
||||
Protect the UI with either local web login or a trusted auth proxy. If local
|
||||
login is enabled, rate-limit `/login` at the reverse proxy. Monlet v1 does not
|
||||
provide full RBAC.
|
||||
|
||||
## Observability
|
||||
|
||||
The local deploy stack includes optional Prometheus, Alertmanager, and Grafana
|
||||
examples under `deploy/`. Production deployments should at minimum scrape:
|
||||
|
||||
- Monlet server `/metrics`
|
||||
- each agent `/metrics`
|
||||
- reverse-proxy and PostgreSQL metrics from the surrounding environment
|
||||
|
||||
Metric labels must stay low-cardinality. Do not put timestamps, request IDs, or
|
||||
unbounded host-local values into check IDs or labels.
|
||||
|
||||
## Backups
|
||||
|
||||
PostgreSQL is the canonical state. Follow
|
||||
[`docs/ops/backup-restore.md`](backup-restore.md) before first production
|
||||
use. Agent spools are best-effort buffers and are not a substitute for database
|
||||
backups.
|
||||
@@ -1,3 +1,12 @@
|
||||
# Examples
|
||||
|
||||
Examples for checks, configs, and local demos.
|
||||
Small payloads and scripts used by local smoke tests and manual API checks.
|
||||
|
||||
Files:
|
||||
|
||||
- `heartbeat.json` - example agent heartbeat request body.
|
||||
- `event-batch.json` - example event batch request body.
|
||||
- `checks/example_check.sh` - minimal shell check that follows the Monlet exit-code contract.
|
||||
|
||||
Use these examples as templates only. Real agents normally send heartbeat and
|
||||
event payloads through the Go agent, not by hand-written curl requests.
|
||||
|
||||
@@ -1,25 +1,47 @@
|
||||
MONLET_AUTH_TOKEN=replace-me-with-random-token
|
||||
MONLET_DATABASE_URL=postgresql+asyncpg://monlet:monlet@localhost:5432/monlet
|
||||
MONLET_LOG_LEVEL=INFO
|
||||
MONLET_HOST=0.0.0.0
|
||||
MONLET_PORT=8000
|
||||
|
||||
MONLET_BODY_LIMIT_BYTES=1048576
|
||||
MONLET_OUTPUT_MAX_BYTES=8192
|
||||
MONLET_MAX_BATCH_EVENTS=200
|
||||
|
||||
MONLET_ENABLE_DETECTOR=true
|
||||
MONLET_STALE_AFTER_SEC=20
|
||||
MONLET_DEAD_AFTER_SEC=30
|
||||
MONLET_DETECTOR_TICK_SEC=5
|
||||
MONLET_ENABLE_DETECTOR=true
|
||||
|
||||
MONLET_DB_POOL_SIZE=10
|
||||
MONLET_DB_MAX_OVERFLOW=10
|
||||
MONLET_DB_POOL_TIMEOUT_SEC=30
|
||||
MONLET_DB_POOL_RECYCLE_SEC=1800
|
||||
|
||||
MONLET_EVENTS_RETENTION_MONTHS=36
|
||||
MONLET_EVENTS_FUTURE_PARTITIONS=3
|
||||
MONLET_PARTITION_MAINTENANCE_INTERVAL_SEC=3600
|
||||
MONLET_OUTBOX_RETENTION_MAX_ROWS=50000
|
||||
MONLET_OUTBOX_PRUNE_BATCH_SIZE=5000
|
||||
|
||||
MONLET_EVENT_DEDUP_RETENTION_DAYS=30
|
||||
MONLET_EVENT_DEDUP_PRUNE_BATCH_SIZE=5000
|
||||
|
||||
MONLET_MAX_PENDING_AGENTS=10000
|
||||
MONLET_PENDING_AGENT_TTL_DAYS=7
|
||||
MONLET_PENDING_AGENT_PRUNE_BATCH_SIZE=1000
|
||||
MONLET_OUTBOX_RETENTION_MAX_ROWS=50000
|
||||
|
||||
MONLET_ENABLE_NOTIFIER_WORKER=true
|
||||
MONLET_NOTIFIER_TICK_SEC=5
|
||||
MONLET_NOTIFIER_BATCH_SIZE=20
|
||||
MONLET_NOTIFIER_CONCURRENCY=4
|
||||
MONLET_NOTIFIER_MAX_ATTEMPTS=8
|
||||
MONLET_NOTIFIER_LEASE_SEC=300
|
||||
MONLET_NOTIFIER_HTTP_TIMEOUT_SEC=10
|
||||
MONLET_NOTIFIER_INCLUDE_OUTPUT=true
|
||||
MONLET_NOTIFIER_MAX_OUTPUT_BYTES=1024
|
||||
MONLET_NOTIFICATION_TIME_ZONE=UTC
|
||||
|
||||
MONLET_NOTIFIER_DEBUG_ENABLED=true
|
||||
MONLET_NOTIFIER_TELEGRAM_ENABLED=false
|
||||
MONLET_NOTIFIER_TELEGRAM_TOKEN=
|
||||
|
||||
@@ -33,26 +33,38 @@ UV_PROJECT_ENVIRONMENT=.venv UV_CACHE_DIR=../.cache/uv uv run fastapi dev monlet
|
||||
| `MONLET_AUTH_TOKEN` | required | UI/operator Bearer token. Space/comma-separated list; any accepted. `changeme` is rejected. |
|
||||
| `MONLET_DATABASE_URL` | `postgresql+asyncpg://monlet:monlet@127.0.0.1:5432/monlet` | Async DSN |
|
||||
| `MONLET_LOG_LEVEL` | `INFO` | Log level |
|
||||
| `MONLET_HOST` | `0.0.0.0` | HTTP bind host for direct server runs |
|
||||
| `MONLET_PORT` | `8000` | HTTP bind port for direct server runs |
|
||||
| `MONLET_BODY_LIMIT_BYTES` | `1048576` | Request body limit (ADR-0005) |
|
||||
| `MONLET_OUTPUT_MAX_BYTES` | `8192` | Per-event output limit |
|
||||
| `MONLET_MAX_BATCH_EVENTS` | `200` | Maximum events per ingestion batch |
|
||||
| `MONLET_STALE_AFTER_SEC` | `20` | Stale threshold |
|
||||
| `MONLET_DEAD_AFTER_SEC` | `30` | Dead threshold |
|
||||
| `MONLET_DETECTOR_TICK_SEC` | `5` | Detector tick |
|
||||
| `MONLET_ENABLE_DETECTOR` | `true` | Toggle detector task |
|
||||
| `MONLET_DB_POOL_SIZE` | `10` | Base async DB pool size |
|
||||
| `MONLET_DB_MAX_OVERFLOW` | `10` | Extra DB connections above pool size |
|
||||
| `MONLET_DB_POOL_TIMEOUT_SEC` | `30` | Seconds to wait for a DB connection |
|
||||
| `MONLET_DB_POOL_RECYCLE_SEC` | `1800` | Recycle DB connections by age |
|
||||
| `MONLET_EVENTS_RETENTION_MONTHS` | `36` | Months of `events` partitions to keep; `0` disables drop |
|
||||
| `MONLET_EVENTS_FUTURE_PARTITIONS` | `3` | Future month partitions to pre-create |
|
||||
| `MONLET_EVENT_DEDUP_RETENTION_DAYS` | `30` | Idempotency window for `event_id` |
|
||||
| `MONLET_EVENT_DEDUP_PRUNE_BATCH_SIZE` | `5000` | Dedup rows pruned per maintenance tick |
|
||||
| `MONLET_MAX_PENDING_AGENTS` | `10000` | Maximum pending agent keys before heartbeat rejects with 429 |
|
||||
| `MONLET_PENDING_AGENT_TTL_DAYS` | `7` | Pending key cleanup TTL; `0` disables pruning |
|
||||
| `MONLET_PENDING_AGENT_PRUNE_BATCH_SIZE` | `1000` | Pending cleanup rows per detector tick |
|
||||
| `MONLET_PARTITION_MAINTENANCE_INTERVAL_SEC` | `3600` | Partition create/drop worker interval |
|
||||
| `MONLET_OUTBOX_RETENTION_MAX_ROWS` | `50000` | Maximum retained terminal outbox rows; `0` disables pruning |
|
||||
| `MONLET_OUTBOX_PRUNE_BATCH_SIZE` | `5000` | Outbox rows pruned per maintenance tick |
|
||||
| `MONLET_ENABLE_NOTIFIER_WORKER` | `true` | Toggle outbox worker |
|
||||
| `MONLET_NOTIFIER_TICK_SEC` | `5` | Worker poll interval |
|
||||
| `MONLET_NOTIFIER_BATCH_SIZE` | `20` | Rows claimed per tick |
|
||||
| `MONLET_NOTIFIER_CONCURRENCY` | `4` | Parallel notification deliveries per tick |
|
||||
| `MONLET_NOTIFIER_MAX_ATTEMPTS` | `8` | Terminal-failure threshold (ADR-0005) |
|
||||
| `MONLET_NOTIFIER_LEASE_SEC` | `300` | Recovery lease for stuck `sending` rows |
|
||||
| `MONLET_NOTIFIER_HTTP_TIMEOUT_SEC` | `10` | Outbound HTTP timeout |
|
||||
| `MONLET_NOTIFIER_INCLUDE_OUTPUT` | `true` | Include redacted check output in notifications |
|
||||
| `MONLET_NOTIFIER_MAX_OUTPUT_BYTES` | `1024` | Maximum notification output bytes after redaction |
|
||||
| `MONLET_NOTIFICATION_TIME_ZONE` | `UTC` | IANA timezone for human-readable notification timestamps |
|
||||
| `MONLET_NOTIFIER_DEBUG_ENABLED` | `true` | Log-only notifier |
|
||||
| `MONLET_NOTIFIER_TELEGRAM_ENABLED` | `false` | Telegram notifier on/off |
|
||||
|
||||
@@ -41,6 +41,7 @@ Do not install Node packages globally. Project dependencies live in `web/node_mo
|
||||
| `MONLET_WEB_SESSION_SECRET` | _(none)_ | Secret used to sign the web session cookie; required with local web login; at least 32 bytes |
|
||||
| `MONLET_WEB_SESSION_TTL_SEC` | `604800` | Local web session lifetime |
|
||||
| `MONLET_WEB_TRUST_PROXY_AUTH` | `false` | Trust `X-Forwarded-User` from an upstream auth proxy |
|
||||
| `NEXT_PUBLIC_MONLET_POLL_MS` | `10000` | Browser polling interval in ms; build-time for production Next.js bundles |
|
||||
|
||||
The API token never reaches the browser — all server calls happen in Server Components / route handlers. Agent admission actions are blocked unless local web login is configured or trusted proxy auth is enabled.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user