Compare commits

...

2 Commits

Author SHA1 Message Date
Stanislav Rossovskii
d6f9335398 Add cron schedules and sync docs
Some checks failed
ci / openapi (push) Failing after 7s
ci / agent (push) Failing after 5s
ci / server (push) Failing after 6s
ci / stack-smoke (push) Has been skipped
ci / web (push) Failing after 5s
2026-06-23 19:18:01 +04:00
Stanislav Rossovskii
f5ec97fcb9 Fix README links and script limits 2026-06-23 18:11:28 +04:00
24 changed files with 652 additions and 141 deletions

View File

@@ -1,16 +1,16 @@
# CI Workflows # CI Workflows
Placeholder for future GitHub Actions workflows. `ci.yml` runs on pushes to `main` and on pull requests. It cancels older runs for
the same ref.
Planned jobs: Jobs:
- Go test for `agent/`; - `openapi` - lints `api/openapi.yaml` with pinned `@redocly/cli`.
- `gofmt -l` and `go vet` for `agent/`; - `agent` - runs `gofmt`, `go vet`, `go test ./...`, and `go test -race ./...`.
- `go test -race` for Go stage checkpoints or concurrency-heavy changes; - `server` - installs with `uv`, then runs Ruff and pytest.
- Python tests and migrations for `server/`; - `web` - installs with npm, regenerates API types, then runs lint, typecheck,
- PostgreSQL constraint tests for `server/`; build, and Playwright smoke.
- Agent-to-Server integration smoke; - `stack-smoke` - runs `deploy/smoke.sh` after server and agent jobs pass.
- TypeScript check, lint, and build for `web/`;
- OpenAPI validation for `api/openapi.yaml`. Keep tool versions pinned in the workflow or in the component lockfiles. Do not
- OpenAPI example payload validation for `examples/heartbeat.json` and `examples/event-batch.json`. replace pinned CI tools with `latest`.
- Docker Compose smoke for `/health`, `/ready`, `/metrics`, and example ingest.

View File

@@ -36,6 +36,13 @@ interval = "60s"
timeout = "10s" timeout = "10s"
notifications_enabled = true notifications_enabled = true
resource_limits = { cpu_time = "5s", memory = "256MiB", open_files = 128 } resource_limits = { cpu_time = "5s", memory = "256MiB", open_files = 128 }
[[checks]]
id = "daily_backup"
name = "Daily backup freshness"
command = "/usr/local/lib/monlet/check_backup_freshness.sh"
cron = "CRON_TZ=UTC 15 6 * * *"
timeout = "2m"
``` ```
Ключи: Ключи:
@@ -49,11 +56,14 @@ resource_limits = { cpu_time = "5s", memory = "256MiB", open_files = 128 }
- `metrics.listen` — адрес `/metrics`. - `metrics.listen` — адрес `/metrics`.
- `checks[].id` — стабильный ID проверки. - `checks[].id` — стабильный ID проверки.
- `checks[].command` — shell-команда строкой; однострочно в `"..."`, многострочно в `'''...'''`. - `checks[].command` — shell-команда строкой; однострочно в `"..."`, многострочно в `'''...'''`.
- `checks[].interval` — период запуска. - `checks[].interval` — период запуска (`60s`, `5m`, `1h`); обязателен, если не задан `checks[].cron`.
- `checks[].cron` — cron-style расписание; обязателен, если не задан `checks[].interval`. Поддерживаются стандартные 5 полей, `@hourly`/`@daily` и `CRON_TZ=...`; seconds-format, `@every` и `@reboot` отклоняются.
- `checks[].timeout` — таймаут; при таймауте результат `critical`. - `checks[].timeout` — таймаут; при таймауте результат `critical`.
- `checks[].notifications_enabled` — разрешить серверные уведомления по этой проверке. - `checks[].notifications_enabled` — разрешить серверные уведомления по этой проверке.
- `checks[].resource_limits` — опциональные лимиты процесса: CPU-время, virtual memory, open files. - `checks[].resource_limits` — опциональные лимиты процесса: CPU-время, virtual memory, open files.
Для каждой проверки нужно задать ровно один способ расписания: `interval` или `cron`. Interval-проверки запускаются сразу после старта/reload, затем по периоду. Cron-проверки ждут ближайший cron-slot и не запускаются немедленно при старте.
Exit code: `0=ok`, `1=warning`, `2=critical`, `3+=unknown`. Exit code: `0=ok`, `1=warning`, `2=critical`, `3+=unknown`.
Env агента: Env агента:
@@ -65,6 +75,7 @@ Reload:
- `systemctl reload monlet-agent` перечитывает локальный TOML через `SIGHUP`. - `systemctl reload monlet-agent` перечитывает локальный TOML через `SIGHUP`.
- Ошибочный config не применяется, старый продолжает работать. - Ошибочный config не применяется, старый продолжает работать.
- Уже запущенный check при reload не убивается, а доживает до своего `timeout`. - Уже запущенный check при reload не убивается, а доживает до своего `timeout`.
- При смене расписания interval-check запускается после текущего in-flight run; cron-check ждёт следующий cron-slot.
- Для изменения `agent_id`, `state_dir`, `server.*`, `metrics.*` нужен restart. - Для изменения `agent_id`, `state_dir`, `server.*`, `metrics.*` нужен restart.
## Server env ## Server env
@@ -97,12 +108,14 @@ MONLET_EVENTS_RETENTION_MONTHS=36
MONLET_EVENTS_FUTURE_PARTITIONS=3 MONLET_EVENTS_FUTURE_PARTITIONS=3
MONLET_PARTITION_MAINTENANCE_INTERVAL_SEC=3600 MONLET_PARTITION_MAINTENANCE_INTERVAL_SEC=3600
MONLET_OUTBOX_RETENTION_MAX_ROWS=50000 MONLET_OUTBOX_RETENTION_MAX_ROWS=50000
MONLET_OUTBOX_PRUNE_BATCH_SIZE=5000
MONLET_EVENT_DEDUP_RETENTION_DAYS=30 MONLET_EVENT_DEDUP_RETENTION_DAYS=30
MONLET_EVENT_DEDUP_PRUNE_BATCH_SIZE=5000 MONLET_EVENT_DEDUP_PRUNE_BATCH_SIZE=5000
MONLET_MAX_PENDING_AGENTS=10000 MONLET_MAX_PENDING_AGENTS=10000
MONLET_PENDING_AGENT_TTL_DAYS=7 MONLET_PENDING_AGENT_TTL_DAYS=7
MONLET_PENDING_AGENT_PRUNE_BATCH_SIZE=1000
MONLET_ENABLE_NOTIFIER_WORKER=true MONLET_ENABLE_NOTIFIER_WORKER=true
MONLET_NOTIFIER_TICK_SEC=5 MONLET_NOTIFIER_TICK_SEC=5
@@ -154,10 +167,12 @@ Liveness-детектор:
- `MONLET_EVENTS_FUTURE_PARTITIONS` — сколько будущих месячных партиций создавать заранее (по умолчанию 3). - `MONLET_EVENTS_FUTURE_PARTITIONS` — сколько будущих месячных партиций создавать заранее (по умолчанию 3).
- `MONLET_PARTITION_MAINTENANCE_INTERVAL_SEC` — период обслуживания партиций и prune (по умолчанию 3600). - `MONLET_PARTITION_MAINTENANCE_INTERVAL_SEC` — период обслуживания партиций и prune (по умолчанию 3600).
- `MONLET_OUTBOX_RETENTION_MAX_ROWS` — верхняя граница строк в outbox; лишние обрезаются (по умолчанию 50000). - `MONLET_OUTBOX_RETENTION_MAX_ROWS` — верхняя граница строк в outbox; лишние обрезаются (по умолчанию 50000).
- `MONLET_OUTBOX_PRUNE_BATCH_SIZE` — размер пакета при очистке terminal outbox-записей (по умолчанию 5000).
- `MONLET_EVENT_DEDUP_RETENTION_DAYS` — хранение записей идемпотентности `event_id` (по умолчанию 30). - `MONLET_EVENT_DEDUP_RETENTION_DAYS` — хранение записей идемпотентности `event_id` (по умолчанию 30).
- `MONLET_EVENT_DEDUP_PRUNE_BATCH_SIZE` — размер пакета при очистке dedup-записей (по умолчанию 5000). - `MONLET_EVENT_DEDUP_PRUNE_BATCH_SIZE` — размер пакета при очистке dedup-записей (по умолчанию 5000).
- `MONLET_MAX_PENDING_AGENTS` — лимит новых непринятых агентов (по умолчанию 10000). - `MONLET_MAX_PENDING_AGENTS` — лимит новых непринятых агентов (по умолчанию 10000).
- `MONLET_PENDING_AGENT_TTL_DAYS` — очистка старых pending-ключей (по умолчанию 7). - `MONLET_PENDING_AGENT_TTL_DAYS` — очистка старых pending-ключей (по умолчанию 7).
- `MONLET_PENDING_AGENT_PRUNE_BATCH_SIZE` — размер пакета при очистке старых pending-ключей (по умолчанию 1000).
Notifier worker: Notifier worker:
@@ -181,21 +196,29 @@ Notifier каналы:
## Web env ## Web env
Веб-UI читает auth-настройки из env. Без них read-only страницы доступны локально, но admission-мутации возвращают `403`. Веб-UI читает API/auth/timezone-настройки из env. Без auth-настроек read-only страницы доступны локально, но admission-мутации возвращают `403`.
```env ```env
MONLET_API_BASE_URL=http://127.0.0.1:8000
MONLET_API_TOKEN=replace-with-server-token
MONLET_WEB_TIME_ZONE=UTC
MONLET_WEB_AUTH_USERNAME=admin MONLET_WEB_AUTH_USERNAME=admin
MONLET_WEB_AUTH_PASSWORD=replace-with-strong-password MONLET_WEB_AUTH_PASSWORD=replace-with-strong-password
MONLET_WEB_SESSION_SECRET=replace-with-random-32-bytes-min MONLET_WEB_SESSION_SECRET=replace-with-random-32-bytes-min
MONLET_WEB_SESSION_TTL_SEC=604800 MONLET_WEB_SESSION_TTL_SEC=604800
NEXT_PUBLIC_MONLET_POLL_MS=10000
# или вместо локального логина — доверие reverse-proxy: # или вместо локального логина — доверие reverse-proxy:
# MONLET_WEB_TRUST_PROXY_AUTH=true # MONLET_WEB_TRUST_PROXY_AUTH=true
``` ```
- `MONLET_API_BASE_URL` — базовый URL сервера для server-side fetch.
- `MONLET_API_TOKEN` — Bearer-токен для запросов UI к серверу; в браузер не передаётся.
- `MONLET_WEB_TIME_ZONE` — IANA timezone для отображения дат (по умолчанию `UTC`).
- `MONLET_WEB_AUTH_USERNAME` / `MONLET_WEB_AUTH_PASSWORD` — учётные данные локального cookie-логина. - `MONLET_WEB_AUTH_USERNAME` / `MONLET_WEB_AUTH_PASSWORD` — учётные данные локального cookie-логина.
- `MONLET_WEB_SESSION_SECRET` — секрет подписи сессии, минимум 32 байта; короче — UI считает auth неправильно настроенным. - `MONLET_WEB_SESSION_SECRET` — секрет подписи сессии, минимум 32 байта; короче — UI считает auth неправильно настроенным.
- `MONLET_WEB_SESSION_TTL_SEC` — срок жизни сессии (по умолчанию 604800 = 7 дней). - `MONLET_WEB_SESSION_TTL_SEC` — срок жизни сессии (по умолчанию 604800 = 7 дней).
- `MONLET_WEB_TRUST_PROXY_AUTH``true` включает доверие заголовку `X-Forwarded-User` от reverse-proxy. Несовместимо с локальным логином: одновременная настройка обоих режимов — ошибка (UI отвечает `500`). - `MONLET_WEB_TRUST_PROXY_AUTH``true` включает доверие заголовку `X-Forwarded-User` от reverse-proxy. Несовместимо с локальным логином: одновременная настройка обоих режимов — ошибка (UI отвечает `500`).
- `NEXT_PUBLIC_MONLET_POLL_MS` — интервал браузерного polling в миллисекундах (по умолчанию 10000); для production-сборок Next.js это build-time переменная.
## Минимальная схема ## Минимальная схема

View File

@@ -59,6 +59,8 @@ Notifier outbox state:
## Features in 0.1.0 ## Features in 0.1.0
- Local command checks with `ok`, `warning`, `critical`, and `unknown` states. - Local command checks with `ok`, `warning`, `critical`, and `unknown` states.
- Per-check timeouts and best-effort script resource limits for CPU time,
virtual memory, and open files.
- Heartbeat-based agent liveness with alive, stale, and dead status. - Heartbeat-based agent liveness with alive, stale, and dead status.
- Durable on-disk agent spool for server outages. - Durable on-disk agent spool for server outages.
- Idempotent event ingestion and duplicate-safe replay. - Idempotent event ingestion and duplicate-safe replay.
@@ -108,8 +110,8 @@ The server owns incident and notification lifecycle. The agent sends facts only.
The web UI is read-only for monitoring state; agent admission and blacklist The web UI is read-only for monitoring state; agent admission and blacklist
controls are the v1 mutation surface. controls are the v1 mutation surface.
Public API contracts live in `api/openapi.yaml`. Design and operations docs live Public API contracts live in `api/openapi.yaml`. Design and operations docs start
under `docs/`. at `docs/index.md`.
## Non-Goals ## Non-Goals
@@ -127,11 +129,12 @@ Monlet 0.1.0 is intentionally not:
Start with: Start with:
1. `deploy/README.md` 1. [deploy/README.md](deploy/README.md)
2. `agent/README.md` 2. [agent/README.md](agent/README.md)
3. `server/README.md` 3. [server/README.md](server/README.md)
4. `web/README.md` 4. [web/README.md](web/README.md)
5. `api/README.md` 5. [api/README.md](api/README.md)
6. [docs/index.md](docs/index.md)
Dependency caches and virtual environments are kept inside the repository. See Dependency caches and virtual environments are kept inside the repository. See
the component READMEs for current local development commands. the component READMEs for current local development commands.

View File

@@ -60,6 +60,8 @@ Open и resolved incidents:
## Возможности в 0.1.0 ## Возможности в 0.1.0
- Локальные command checks со статусами `ok`, `warning`, `critical` и `unknown`. - Локальные command checks со статусами `ok`, `warning`, `critical` и `unknown`.
- Timeout для каждой проверки и best-effort resource limits для scripts: CPU
time, virtual memory и open files.
- Agent liveness по heartbeat: alive, stale и dead. - Agent liveness по heartbeat: alive, stale и dead.
- Durable on-disk spool в агенте на случай outage сервера. - Durable on-disk spool в агенте на случай outage сервера.
- Idempotent event ingestion и replay без дублей. - Idempotent event ingestion и replay без дублей.
@@ -110,7 +112,7 @@ Server владеет lifecycle инцидентов и уведомлений.
controls - mutation surface для v1. controls - mutation surface для v1.
Публичный API contract лежит в `api/openapi.yaml`. Design и operations docs Публичный API contract лежит в `api/openapi.yaml`. Design и operations docs
лежат в `docs/`. начинаются с `docs/index.md`.
## Не цели ## Не цели
@@ -128,11 +130,12 @@ Monlet 0.1.0 намеренно не является:
Начинать отсюда: Начинать отсюда:
1. `deploy/README.md` 1. [deploy/README.md](deploy/README.md)
2. `agent/README.md` 2. [agent/README.md](agent/README.md)
3. `server/README.md` 3. [server/README.md](server/README.md)
4. `web/README.md` 4. [web/README.md](web/README.md)
5. `api/README.md` 5. [api/README.md](api/README.md)
6. [docs/index.md](docs/index.md)
Dependency caches и virtual environments хранятся внутри репозитория. Актуальные Dependency caches и virtual environments хранятся внутри репозитория. Актуальные
команды local development описаны в component README. команды local development описаны в component README.

View File

@@ -14,7 +14,7 @@ Do not use `go install` or `go env -w`. Module cache, build cache, and tool bina
## Configuration ## Configuration
See `config.example.toml`. Required fields: top-level `state_dir`, at least one `[[checks]]` (`id`, `command`, `interval`, `timeout`), and at least one output enabled: `server.enabled = true` or `metrics.enabled = true`. `agent_id` is optional; if omitted, it defaults to the OS hostname. `hostname` is not a config key. See `config.example.toml`. Required fields: top-level `state_dir`, at least one `[[checks]]` (`id`, `command`, `timeout`, and exactly one of `interval` or `cron`), and at least one output enabled: `server.enabled = true` or `metrics.enabled = true`. `agent_id` is optional; if omitted, it defaults to the OS hostname. `hostname` is not a config key.
| Key | Default | Notes | | Key | Default | Notes |
|---|---|---| |---|---|---|
@@ -25,10 +25,12 @@ See `config.example.toml`. Required fields: top-level `state_dir`, at least one
| `server.batch_interval` | `10s` | events flush cadence | | `server.batch_interval` | `10s` | events flush cadence |
| `metrics.listen` | `127.0.0.1:9465` | low-cardinality only | | `metrics.listen` | `127.0.0.1:9465` | low-cardinality only |
| `checks[].command` | required | Shell string executed as `/bin/sh -c`. Both forms accepted: `command = "check_disk --warn 80 --crit 90"` and triple-quoted `command = '''...'''`. Argv arrays are rejected. See [Command security contract](#command-security-contract). | | `checks[].command` | required | Shell string executed as `/bin/sh -c`. Both forms accepted: `command = "check_disk --warn 80 --crit 90"` and triple-quoted `command = '''...'''`. Argv arrays are rejected. See [Command security contract](#command-security-contract). |
| `checks[].interval` | required unless `cron` is set | Go duration such as `60s`, `5m`, or `1h`. Interval checks run once immediately on start/reload, then on the interval. |
| `checks[].cron` | required unless `interval` is set | Standard 5-field cron expression, macros such as `@hourly`/`@daily`, and optional `CRON_TZ=...`. Seconds fields, `@every`, and `@reboot` are rejected. Cron checks wait for the next scheduled slot; they do not run immediately on start/reload. |
| `checks[].notifications_enabled` | `true` | Set `false` for checks that must not create server notifications. | | `checks[].notifications_enabled` | `true` | Set `false` for checks that must not create server notifications. |
| `checks[].resource_limits` | `{}` | Optional per-check best-effort `ulimit` values: `cpu_time`, `memory`, `open_files`. | | `checks[].resource_limits` | `{}` | Optional per-check best-effort `ulimit` values: `cpu_time`, `memory`, `open_files`. |
Examples — one-line and multi-line forms: Examples — interval, cron, and multi-line forms:
```toml ```toml
[[checks]] [[checks]]
@@ -37,6 +39,12 @@ command = "test $(cut -d. -f1 /proc/uptime) -gt 60"
interval = "60s" interval = "60s"
timeout = "5s" timeout = "5s"
[[checks]]
id = "daily_backup"
command = "/usr/local/lib/monlet/check_backup_freshness.sh"
cron = "CRON_TZ=UTC 15 6 * * *"
timeout = "2m"
[[checks]] [[checks]]
id = "disk_root" id = "disk_root"
command = ''' command = '''
@@ -81,16 +89,17 @@ executes it as `/bin/sh -c <command>` on the host where the agent runs.
Send `SIGHUP` or run `systemctl reload monlet-agent` to reload the local TOML file. Send `SIGHUP` or run `systemctl reload monlet-agent` to reload the local TOML file.
- Reload first validates the full new config. On error, the old config keeps running. - Reload first validates the full new config. On error, the old config keeps running.
- Hot-reloadable: labels and checks, including `command`, `interval`, `timeout`, - Hot-reloadable: labels and checks, including `command`, `interval`, `cron`,
`notifications_enabled`, `dedupe_key`, and `resource_limits`. `timeout`, `notifications_enabled`, `dedupe_key`, and `resource_limits`.
- Restart required: `agent_id`, `state_dir`, `server.*`, and `metrics.*`. - Restart required: `agent_id`, `state_dir`, `server.*`, and `metrics.*`.
- A check already running during reload is allowed to finish under its old config. - A check already running during reload is allowed to finish under its old config.
Removed or changed checks do not start new old-config runs; changed checks start Removed or changed checks do not start new old-config runs. Changed interval
with the new config after the in-flight run finishes. checks start with the new config after the in-flight run finishes; changed cron
checks wait for the next cron slot.
## Behaviour ## Behaviour
- Per-check scheduler with no-overlap: a tick is skipped if the previous run is still in flight (`monlet_agent_check_skipped_total`). - Per-check scheduler with no-overlap: a tick/cron slot is skipped if the previous run is still in flight (`monlet_agent_check_skipped_total`).
- Check commands run through `/bin/sh -c`; the old argv-array form is intentionally unsupported. - Check commands run through `/bin/sh -c`; the old argv-array form is intentionally unsupported.
- Timeout via `exec.CommandContext` with `Setpgid` so the process group is killed on expiry; timeout → `critical`. - Timeout via `exec.CommandContext` with `Setpgid` so the process group is killed on expiry; timeout → `critical`.
- Exit code mapping: 0=ok, 1=warning, 2=critical, 3+=unknown (ADR-0005). - Exit code mapping: 0=ok, 1=warning, 2=critical, 3+=unknown (ADR-0005).
@@ -115,7 +124,7 @@ Exposed on `metrics.listen` at `/metrics`. Labels are static (`check_id`, `statu
- `monlet_agent_check_exit_code{check_id}` - `monlet_agent_check_exit_code{check_id}`
- `monlet_agent_check_last_run_timestamp_seconds{check_id}` - `monlet_agent_check_last_run_timestamp_seconds{check_id}`
- `monlet_agent_check_last_success_timestamp_seconds{check_id}` - `monlet_agent_check_last_success_timestamp_seconds{check_id}`
- `monlet_agent_check_interval_seconds{check_id}` - `monlet_agent_check_interval_seconds{check_id}` (interval checks only)
- `monlet_agent_checks_configured` - `monlet_agent_checks_configured`
- `monlet_agent_config_load_success` - `monlet_agent_config_load_success`
- `monlet_agent_config_reload_total{result}` - `monlet_agent_config_reload_total{result}`

View File

@@ -32,3 +32,11 @@ name = "Uptime probe"
command = "test $(cut -d. -f1 /proc/uptime) -gt 60" command = "test $(cut -d. -f1 /proc/uptime) -gt 60"
interval = "60s" interval = "60s"
timeout = "5s" timeout = "5s"
# Cron-style schedules are also supported. Use exactly one of interval or cron.
# [[checks]]
# id = "daily_backup"
# name = "Daily backup freshness"
# command = "/usr/local/lib/monlet/check_backup_freshness.sh"
# cron = "CRON_TZ=UTC 15 6 * * *"
# timeout = "2m"

View File

@@ -6,6 +6,8 @@ require (
github.com/BurntSushi/toml v1.6.0 github.com/BurntSushi/toml v1.6.0
github.com/google/uuid v1.6.0 github.com/google/uuid v1.6.0
github.com/prometheus/client_golang v1.23.2 github.com/prometheus/client_golang v1.23.2
github.com/prometheus/client_model v0.6.2
github.com/robfig/cron/v3 v3.0.1
) )
require ( require (
@@ -13,7 +15,6 @@ require (
github.com/cespare/xxhash/v2 v2.3.0 // indirect github.com/cespare/xxhash/v2 v2.3.0 // indirect
github.com/kr/text v0.2.0 // indirect github.com/kr/text v0.2.0 // indirect
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
github.com/prometheus/client_model v0.6.2 // indirect
github.com/prometheus/common v0.66.1 // indirect github.com/prometheus/common v0.66.1 // indirect
github.com/prometheus/procfs v0.16.1 // indirect github.com/prometheus/procfs v0.16.1 // indirect
go.yaml.in/yaml/v2 v2.4.2 // indirect go.yaml.in/yaml/v2 v2.4.2 // indirect

View File

@@ -31,6 +31,8 @@ github.com/prometheus/common v0.66.1 h1:h5E0h5/Y8niHc5DlaLlWLArTQI7tMrsfQjHV+d9Z
github.com/prometheus/common v0.66.1/go.mod h1:gcaUsgf3KfRSwHY4dIMXLPV0K/Wg1oZ8+SbZk/HH/dA= github.com/prometheus/common v0.66.1/go.mod h1:gcaUsgf3KfRSwHY4dIMXLPV0K/Wg1oZ8+SbZk/HH/dA=
github.com/prometheus/procfs v0.16.1 h1:hZ15bTNuirocR6u0JZ6BAHHmwS1p8B4P6MRqxtzMyRg= github.com/prometheus/procfs v0.16.1 h1:hZ15bTNuirocR6u0JZ6BAHHmwS1p8B4P6MRqxtzMyRg=
github.com/prometheus/procfs v0.16.1/go.mod h1:teAbpZRB1iIAJYREa1LsoWUXykVXA1KlTmWl8x/U+Is= github.com/prometheus/procfs v0.16.1/go.mod h1:teAbpZRB1iIAJYREa1LsoWUXykVXA1KlTmWl8x/U+Is=
github.com/robfig/cron/v3 v3.0.1 h1:WdRxkvbJztn8LMz/QEvLN5sBU+xKpSqwwUO1Pjr4qDs=
github.com/robfig/cron/v3 v3.0.1/go.mod h1:eQICP3HwyT7UooqI/z+Ov+PtYAWygg1TEWWzGIFLtro=
github.com/rogpeppe/go-internal v1.10.0 h1:TMyTOH3F/DB16zRVcYyreMH6GnZZrwQVAoYjRBZyWFQ= github.com/rogpeppe/go-internal v1.10.0 h1:TMyTOH3F/DB16zRVcYyreMH6GnZZrwQVAoYjRBZyWFQ=
github.com/rogpeppe/go-internal v1.10.0/go.mod h1:UQnix2H7Ngw/k4C5ijL5+65zddjncjaFoBhdsK/akog= github.com/rogpeppe/go-internal v1.10.0/go.mod h1:UQnix2H7Ngw/k4C5ijL5+65zddjncjaFoBhdsK/akog=
github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U= github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U=

View File

@@ -47,9 +47,7 @@ func (a *App) Run(ctx context.Context) error {
a.Metrics.ConfigLoadSuccess.Set(1) a.Metrics.ConfigLoadSuccess.Set(1)
a.Metrics.ChecksConfigured.Set(float64(len(cfg.Checks))) a.Metrics.ChecksConfigured.Set(float64(len(cfg.Checks)))
a.Metrics.BuildInfo.WithLabelValues(a.Version, boolLabel(cfg.PushesToServer()), boolLabel(cfg.ExposesMetrics())).Set(1) a.Metrics.BuildInfo.WithLabelValues(a.Version, boolLabel(cfg.PushesToServer()), boolLabel(cfg.ExposesMetrics())).Set(1)
for _, c := range cfg.Checks { a.recordCheckScheduleMetrics(cfg.Checks)
a.Metrics.CheckInterval.WithLabelValues(c.ID).Set(c.Interval.Duration.Seconds())
}
var wg sync.WaitGroup var wg sync.WaitGroup
@@ -256,9 +254,7 @@ func (a *App) Reload(newCfg *config.Config, newAgentID string) error {
a.Metrics.ConfigReloadTotal.WithLabelValues("success").Inc() a.Metrics.ConfigReloadTotal.WithLabelValues("success").Inc()
a.Metrics.ConfigLastReloadSuccess.Set(float64(time.Now().Unix())) a.Metrics.ConfigLastReloadSuccess.Set(float64(time.Now().Unix()))
a.Metrics.ChecksConfigured.Set(float64(len(newCfg.Checks))) a.Metrics.ChecksConfigured.Set(float64(len(newCfg.Checks)))
for _, c := range newCfg.Checks { a.recordCheckScheduleMetrics(newCfg.Checks)
a.Metrics.CheckInterval.WithLabelValues(c.ID).Set(c.Interval.Duration.Seconds())
}
a.schedulerMu.RLock() a.schedulerMu.RLock()
mgr := a.scheduler mgr := a.scheduler
@@ -275,6 +271,16 @@ func (a *App) RecordReloadFailure() {
a.Metrics.ConfigReloadTotal.WithLabelValues("failure").Inc() a.Metrics.ConfigReloadTotal.WithLabelValues("failure").Inc()
} }
func (a *App) recordCheckScheduleMetrics(checks []config.CheckConfig) {
for _, c := range checks {
if c.UsesInterval() {
a.Metrics.CheckInterval.WithLabelValues(c.ID).Set(c.Interval.Duration.Seconds())
} else {
a.Metrics.CheckInterval.DeleteLabelValues(c.ID)
}
}
}
func (a *App) config() *config.Config { func (a *App) config() *config.Config {
a.cfgMu.RLock() a.cfgMu.RLock()
defer a.cfgMu.RUnlock() defer a.cfgMu.RUnlock()

View File

@@ -17,6 +17,7 @@ import (
"github.com/monlet/agent/internal/config" "github.com/monlet/agent/internal/config"
"github.com/monlet/agent/internal/metrics" "github.com/monlet/agent/internal/metrics"
"github.com/monlet/agent/internal/spool" "github.com/monlet/agent/internal/spool"
dto "github.com/prometheus/client_model/go"
) )
func TestPrometheusOnlyDoesNotPush(t *testing.T) { func TestPrometheusOnlyDoesNotPush(t *testing.T) {
@@ -257,6 +258,92 @@ func TestReloadAcceptsLabelsAndChecks(t *testing.T) {
} }
} }
func TestReloadUpdatesIntervalMetricForCron(t *testing.T) {
oldCfg := &config.Config{
AgentID: "a1",
Hostname: "h",
StateDir: "/tmp/old",
Server: config.ServerConfig{
Enabled: boolPtr(true),
URL: "http://server",
HeartbeatInterval: config.Duration{Duration: 10 * time.Second},
BatchInterval: config.Duration{Duration: 10 * time.Second},
},
Metrics: config.MetricsConfig{Enabled: false},
Checks: []config.CheckConfig{{
ID: "c1",
Command: "true",
Interval: config.Duration{Duration: time.Second},
Timeout: config.Duration{Duration: time.Second},
}},
}
a := &App{
Cfg: oldCfg,
AgentID: "a1",
Metrics: metrics.New(),
}
a.recordCheckScheduleMetrics(oldCfg.Checks)
if got, ok := checkIntervalMetric(t, a, "c1"); !ok || got != 1 {
t.Fatalf("interval metric missing before reload: value=%v ok=%v", got, ok)
}
cronCfg := *oldCfg
cronCfg.Checks = []config.CheckConfig{{
ID: "c1",
Command: "true",
Cron: "0 * * * *",
Timeout: config.Duration{Duration: time.Second},
}}
if err := a.Reload(&cronCfg, "a1"); err != nil {
t.Fatal(err)
}
if got, ok := checkIntervalMetric(t, a, "c1"); ok {
t.Fatalf("interval metric must be deleted for cron check, got %v", got)
}
intervalCfg := cronCfg
intervalCfg.Checks = []config.CheckConfig{{
ID: "c1",
Command: "true",
Interval: config.Duration{Duration: 2 * time.Second},
Timeout: config.Duration{Duration: time.Second},
}}
if err := a.Reload(&intervalCfg, "a1"); err != nil {
t.Fatal(err)
}
if got, ok := checkIntervalMetric(t, a, "c1"); !ok || got != 2 {
t.Fatalf("interval metric missing after interval reload: value=%v ok=%v", got, ok)
}
}
func checkIntervalMetric(t *testing.T, a *App, checkID string) (float64, bool) {
t.Helper()
families, err := a.Metrics.Registry.Gather()
if err != nil {
t.Fatal(err)
}
for _, family := range families {
if family.GetName() != "monlet_agent_check_interval_seconds" {
continue
}
for _, metric := range family.GetMetric() {
if labelValue(metric.GetLabel(), "check_id") == checkID && metric.GetGauge() != nil {
return metric.GetGauge().GetValue(), true
}
}
}
return 0, false
}
func labelValue(labels []*dto.LabelPair, name string) string {
for _, label := range labels {
if label.GetName() == name {
return label.GetValue()
}
}
return ""
}
func boolPtr(v bool) *bool { func boolPtr(v bool) *bool {
return &v return &v
} }

View File

@@ -11,6 +11,7 @@ import (
"unicode/utf8" "unicode/utf8"
"github.com/BurntSushi/toml" "github.com/BurntSushi/toml"
"github.com/robfig/cron/v3"
) )
var idPattern = regexp.MustCompile(`^[A-Za-z0-9._:-]+$`) var idPattern = regexp.MustCompile(`^[A-Za-z0-9._:-]+$`)
@@ -58,6 +59,7 @@ type CheckConfig struct {
Name string `toml:"name"` Name string `toml:"name"`
Command string `toml:"command"` Command string `toml:"command"`
Interval Duration `toml:"interval"` Interval Duration `toml:"interval"`
Cron string `toml:"cron"`
Timeout Duration `toml:"timeout"` Timeout Duration `toml:"timeout"`
NotificationsEnabled *bool `toml:"notifications_enabled"` NotificationsEnabled *bool `toml:"notifications_enabled"`
DedupeKey string `toml:"dedupe_key"` DedupeKey string `toml:"dedupe_key"`
@@ -211,14 +213,24 @@ func (c *Config) Validate() error {
if strings.TrimSpace(ch.Command) == "" { if strings.TrimSpace(ch.Command) == "" {
return fmt.Errorf("check %q: command is required", ch.ID) return fmt.Errorf("check %q: command is required", ch.ID)
} }
if ch.Interval.Duration <= 0 {
return fmt.Errorf("check %q: interval must be > 0", ch.ID)
}
if ch.Timeout.Duration <= 0 { if ch.Timeout.Duration <= 0 {
return fmt.Errorf("check %q: timeout must be > 0", ch.ID) return fmt.Errorf("check %q: timeout must be > 0", ch.ID)
} }
if ch.Timeout.Duration > ch.Interval.Duration { if ch.Interval.Duration < 0 {
return fmt.Errorf("check %q: interval must be > 0", ch.ID)
}
ch.Cron = strings.TrimSpace(ch.Cron)
hasInterval := ch.Interval.Duration > 0
hasCron := ch.Cron != ""
switch {
case hasInterval == hasCron:
return fmt.Errorf("check %q: exactly one of interval or cron is required", ch.ID)
case hasInterval && ch.Timeout.Duration > ch.Interval.Duration:
return fmt.Errorf("check %q: timeout must be <= interval", ch.ID) return fmt.Errorf("check %q: timeout must be <= interval", ch.ID)
case hasCron:
if _, err := ParseCronSchedule(ch.Cron); err != nil {
return fmt.Errorf("check %q: invalid cron: %w", ch.ID, err)
}
} }
if len(ch.DedupeKey) > 256 { if len(ch.DedupeKey) > 256 {
return fmt.Errorf("check %q: dedupe_key too long", ch.ID) return fmt.Errorf("check %q: dedupe_key too long", ch.ID)
@@ -253,6 +265,42 @@ func (c CheckConfig) NotificationsOn() bool {
return c.NotificationsEnabled == nil || *c.NotificationsEnabled return c.NotificationsEnabled == nil || *c.NotificationsEnabled
} }
func (c CheckConfig) UsesInterval() bool {
return c.Interval.Duration > 0
}
func (c CheckConfig) UsesCron() bool {
return strings.TrimSpace(c.Cron) != ""
}
func ParseCronSchedule(spec string) (cron.Schedule, error) {
spec = strings.TrimSpace(spec)
if spec == "" {
return nil, fmt.Errorf("cron is empty")
}
withoutTZ := specWithoutTZ(spec)
lower := strings.ToLower(withoutTZ)
if strings.HasPrefix(lower, "@every") {
return nil, fmt.Errorf("@every is not supported; use interval")
}
if lower == "@reboot" {
return nil, fmt.Errorf("@reboot is not supported")
}
return cron.ParseStandard(spec)
}
func specWithoutTZ(spec string) string {
fields := strings.Fields(spec)
if len(fields) == 0 {
return ""
}
first := fields[0]
if strings.HasPrefix(first, "CRON_TZ=") || strings.HasPrefix(first, "TZ=") {
return strings.ToLower(strings.Join(fields[1:], " "))
}
return strings.ToLower(strings.Join(fields, " "))
}
func validateResourceLimits(checkID string, limits ResourceLimits) error { func validateResourceLimits(checkID string, limits ResourceLimits) error {
if limits.CPUTime.Duration < 0 { if limits.CPUTime.Duration < 0 {
return fmt.Errorf("check %q: resource_limits.cpu_time must be > 0", checkID) return fmt.Errorf("check %q: resource_limits.cpu_time must be > 0", checkID)

View File

@@ -48,6 +48,81 @@ func TestLoadMinimal(t *testing.T) {
} }
} }
func TestLoadCronCheck(t *testing.T) {
c, err := Load(writeTmp(t, `
agent_id = "host-1"
state_dir = "/tmp/x"
[server]
enabled = true
url = "http://localhost"
[[checks]]
id = "c1"
command = "true"
cron = "CRON_TZ=UTC 0 6 * * *"
timeout = "5m"
`))
if err != nil {
t.Fatal(err)
}
if !c.Checks[0].UsesCron() || c.Checks[0].UsesInterval() {
t.Fatalf("unexpected schedule mode: %+v", c.Checks[0])
}
}
func TestValidateCheckScheduleChoice(t *testing.T) {
for name, body := range map[string]string{
"both_interval_and_cron": `
interval = "10s"
cron = "0 * * * *"
timeout = "5s"
`,
"neither_interval_nor_cron": `
timeout = "5s"
`,
"invalid_cron": `
cron = "not a cron"
timeout = "5s"
`,
"seconds_cron": `
cron = "0 0 * * * *"
timeout = "5s"
`,
"every_cron": `
cron = "@every 10s"
timeout = "5s"
`,
"tz_every_cron": `
cron = "CRON_TZ=UTC @every 10s"
timeout = "5s"
`,
"reboot_cron": `
cron = "@reboot"
timeout = "5s"
`,
} {
t.Run(name, func(t *testing.T) {
_, err := Load(writeTmp(t, `
agent_id = "host-1"
state_dir = "/tmp/x"
[server]
enabled = true
url = "http://localhost"
[[checks]]
id = "c1"
command = "true"
`+body+`
`))
if err == nil {
t.Fatal("expected schedule validation error")
}
})
}
}
func TestLoadResourceLimits(t *testing.T) { func TestLoadResourceLimits(t *testing.T) {
c, err := Load(writeTmp(t, ` c, err := Load(writeTmp(t, `
agent_id = "host-1" agent_id = "host-1"

View File

@@ -11,6 +11,8 @@ import (
"github.com/monlet/agent/internal/runner" "github.com/monlet/agent/internal/runner"
) )
var parseCronSchedule = config.ParseCronSchedule
// Event is a contract-shaped check result emitted by the scheduler. // Event is a contract-shaped check result emitted by the scheduler.
type Event struct { type Event struct {
EventID string `json:"event_id"` EventID string `json:"event_id"`
@@ -207,8 +209,10 @@ func (w *checkWorker) loop(initial config.CheckConfig) {
w.wg.Done() w.wg.Done()
}() }()
cfg := initial cfg := initial
t := time.NewTicker(cfg.Interval.Duration) t := time.NewTimer(time.Hour)
stopTimer(t)
defer t.Stop() defer t.Stop()
var timerCh <-chan time.Time
running := false running := false
stopping := false stopping := false
@@ -230,7 +234,26 @@ func (w *checkWorker) loop(initial config.CheckConfig) {
}() }()
} }
start(cfg) arm := func(c config.CheckConfig, immediateInterval bool) bool {
delay, err := nextDelay(c, time.Now(), immediateInterval)
if err != nil {
return false
}
resetTimer(t, delay)
timerCh = t.C
return true
}
disarm := func() {
stopTimer(t)
timerCh = nil
}
if !arm(cfg, true) {
if w.hooks.OnStopped != nil {
w.hooks.OnStopped(cfg.ID)
}
return
}
for { for {
select { select {
@@ -257,11 +280,21 @@ func (w *checkWorker) loop(initial config.CheckConfig) {
continue continue
} }
cfg = next cfg = next
t.Reset(cfg.Interval.Duration) if cfg.UsesCron() {
runAfterCurrent = false
if !arm(cfg, false) {
return
}
continue
}
if running { if running {
runAfterCurrent = true runAfterCurrent = true
disarm()
} else { } else {
start(cfg) start(cfg)
if !arm(cfg, false) {
return
}
} }
case <-w.doneCh: case <-w.doneCh:
running = false running = false
@@ -281,26 +314,35 @@ func (w *checkWorker) loop(initial config.CheckConfig) {
} }
if runAfterCurrent { if runAfterCurrent {
runAfterCurrent = false runAfterCurrent = false
startAfterDrain := true
for { for {
select { select {
case next := <-w.updateCh: case next := <-w.updateCh:
if !reflect.DeepEqual(cfg, next) { if !reflect.DeepEqual(cfg, next) {
cfg = next cfg = next
t.Reset(cfg.Interval.Duration) startAfterDrain = !cfg.UsesCron()
} }
default: default:
if startAfterDrain {
start(cfg) start(cfg)
}
if !arm(cfg, false) {
return
}
goto nextLoop goto nextLoop
} }
} }
nextLoop: nextLoop:
continue continue
} }
case <-t.C: case <-timerCh:
if stopping { if stopping {
continue continue
} }
start(cfg) start(cfg)
if !arm(cfg, false) {
return
}
} }
} }
} }
@@ -358,3 +400,41 @@ func incidentKey(agentID string, c config.CheckConfig) string {
} }
return agentID + ":" + c.ID return agentID + ":" + c.ID
} }
func nextDelay(c config.CheckConfig, now time.Time, immediateInterval bool) (time.Duration, error) {
if c.UsesCron() {
schedule, err := parseCronSchedule(c.Cron)
if err != nil {
return 0, err
}
return delayUntil(now, schedule.Next(now)), nil
}
if immediateInterval {
return 0, nil
}
return c.Interval.Duration, nil
}
func delayUntil(now, next time.Time) time.Duration {
if next.Before(now) {
return 0
}
return next.Sub(now)
}
func resetTimer(t *time.Timer, d time.Duration) {
if d < 0 {
d = 0
}
stopTimer(t)
t.Reset(d)
}
func stopTimer(t *time.Timer) {
if !t.Stop() {
select {
case <-t.C:
default:
}
}
}

View File

@@ -8,6 +8,7 @@ import (
"time" "time"
"github.com/monlet/agent/internal/config" "github.com/monlet/agent/internal/config"
"github.com/robfig/cron/v3"
) )
func runScheduler(ctx context.Context, agentID string, checks []config.CheckConfig, out chan<- Event, h Hooks) { func runScheduler(ctx context.Context, agentID string, checks []config.CheckConfig, out chan<- Event, h Hooks) {
@@ -27,6 +28,32 @@ func mkCheck(id, cmd string, interval, timeout time.Duration) config.CheckConfig
} }
} }
type fakeCronSchedule struct {
delay time.Duration
}
func (s fakeCronSchedule) Next(t time.Time) time.Time {
return t.Add(s.delay)
}
func withFakeCron(t *testing.T, delay time.Duration) {
t.Helper()
old := parseCronSchedule
parseCronSchedule = func(string) (cron.Schedule, error) {
return fakeCronSchedule{delay: delay}, nil
}
t.Cleanup(func() { parseCronSchedule = old })
}
func mkCronCheck(id, cmd string, timeout time.Duration) config.CheckConfig {
return config.CheckConfig{
ID: id,
Command: cmd,
Cron: "0 * * * *",
Timeout: config.Duration{Duration: timeout},
}
}
func TestEmitsEvents(t *testing.T) { func TestEmitsEvents(t *testing.T) {
ctx, cancel := context.WithTimeout(context.Background(), 500*time.Millisecond) ctx, cancel := context.WithTimeout(context.Background(), 500*time.Millisecond)
defer cancel() defer cancel()
@@ -45,6 +72,41 @@ func TestEmitsEvents(t *testing.T) {
} }
} }
func TestCronDoesNotRunImmediately(t *testing.T) {
withFakeCron(t, 200*time.Millisecond)
ctx, cancel := context.WithTimeout(context.Background(), 100*time.Millisecond)
defer cancel()
out := make(chan Event, 10)
m := NewManager(ctx, "a1", out, Hooks{})
m.Update([]config.CheckConfig{mkCronCheck("cron", "echo cron", 100*time.Millisecond)})
select {
case ev := <-out:
t.Fatalf("cron check ran before next slot: %+v", ev)
case <-ctx.Done():
}
m.Stop()
m.Wait()
}
func TestCronEmitsOnDueSlot(t *testing.T) {
withFakeCron(t, 30*time.Millisecond)
ctx, cancel := context.WithTimeout(context.Background(), time.Second)
defer cancel()
out := make(chan Event, 10)
m := NewManager(ctx, "a1", out, Hooks{})
m.Update([]config.CheckConfig{mkCronCheck("cron", "echo cron", 100*time.Millisecond)})
select {
case ev := <-out:
if ev.CheckID != "cron" || ev.Status != "ok" {
t.Fatalf("unexpected event: %+v", ev)
}
case <-ctx.Done():
t.Fatal("no cron event")
}
m.Stop()
m.Wait()
}
func TestNoOverlapSkips(t *testing.T) { func TestNoOverlapSkips(t *testing.T) {
ctx, cancel := context.WithTimeout(context.Background(), 600*time.Millisecond) ctx, cancel := context.WithTimeout(context.Background(), 600*time.Millisecond)
defer cancel() defer cancel()
@@ -59,6 +121,23 @@ func TestNoOverlapSkips(t *testing.T) {
} }
} }
func TestCronNoOverlapSkips(t *testing.T) {
withFakeCron(t, 50*time.Millisecond)
ctx, cancel := context.WithTimeout(context.Background(), 250*time.Millisecond)
defer cancel()
out := make(chan Event, 10)
var skipped int32
hooks := Hooks{OnSkipped: func(string) { atomic.AddInt32(&skipped, 1) }}
m := NewManager(ctx, "a1", out, hooks)
m.Update([]config.CheckConfig{mkCronCheck("cron", "sleep 0.2", 500*time.Millisecond)})
<-ctx.Done()
m.Stop()
m.Wait()
if atomic.LoadInt32(&skipped) == 0 {
t.Fatal("expected at least one skipped cron slot")
}
}
func TestReportsFullEventChannel(t *testing.T) { func TestReportsFullEventChannel(t *testing.T) {
ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second) ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second)
defer cancel() defer cancel()

View File

@@ -1,9 +1,18 @@
# Alertmanager # Alertmanager
Placeholder for optional Alertmanager integration examples. Example Alertmanager configuration for the optional observability profile.
Planned examples: `alertmanager.yml` is intentionally safe by default: it routes to a `null`
receiver, so alerts are grouped but dropped. Before using it outside local
development, add a real receiver and set `route.receiver` to that receiver.
- receiver config for Monlet-originated alerts; Monlet can also send notifications to Alertmanager through the server notifier:
- stable label conventions;
- routing notes for avoiding duplicate notifications. ```env
MONLET_NOTIFIER_ALERTMANAGER_ENABLED=true
MONLET_NOTIFIER_ALERTMANAGER_URL=http://alertmanager:9093
```
Keep labels stable and low-cardinality. The example groups by `alertname`,
`agent_id`, and `check_id`; do not add timestamps, event IDs, or output text to
Alertmanager labels.

View File

@@ -1,11 +1,20 @@
# Grafana # Grafana
Placeholder for Grafana dashboard examples. Starter Grafana provisioning for the optional observability profile.
Planned dashboards: Files:
- agents alive/stale/dead; - `provisioning/datasources/prometheus.yml` - adds Prometheus at `http://prometheus:9090`.
- check state counts; - `provisioning/dashboards/monlet.yml` - loads dashboards from `/var/lib/grafana/dashboards`.
- open incidents; - `dashboards/monlet-overview.json` - starter dashboard for agent status, open incidents, event ingest rate, and notification rate.
- notification outbox state;
- server ingestion and notifier failures. Run it with:
```sh
cd deploy
MONLET_AUTH_TOKEN=$(openssl rand -hex 16) docker compose --profile observability up
```
Grafana is exposed at <http://127.0.0.1:3001>. The local default is
`admin` / `admin` unless `GF_ADMIN_PASSWORD` is set. Change it before exposing
Grafana outside local development.

View File

@@ -1,9 +1,20 @@
# Prometheus # Prometheus
Placeholder for Prometheus examples. Example scrape configuration for the optional observability profile in
`deploy/docker-compose.yml`.
Planned examples: Run it with:
- scrape Monlet agent `/metrics`; ```sh
- scrape Monlet server `/metrics`; cd deploy
- sample rules for checks that use `notifications_enabled = false`. MONLET_AUTH_TOKEN=$(openssl rand -hex 16) docker compose --profile observability up
```
The example config scrapes:
- `server:8000/metrics` as `monlet-server`
- `host.docker.internal:9100/metrics` as `monlet-agent`
Adjust the agent target for your environment. In the showcase stack one demo
agent exposes metrics on host port `9465`; production agents usually expose
`metrics.listen` only on the host or a private monitoring network.

16
docs/index.md Normal file
View File

@@ -0,0 +1,16 @@
# Monlet Docs
This directory contains operator-facing docs that do not fit in the root or component READMEs.
## Start Here
- [First deployment checklist](ops/deployment-checklist.md) - step-by-step production-ish stand-up checklist.
- [Backup and restore](ops/backup-restore.md) - PostgreSQL backup, retention, and disaster recovery notes.
- [Token rotation](ops/token-rotation.md) - UI token overlap and agent-key rotation.
- [Deployment notes](ops/deployment.md) - deployment shape and production boundaries.
## Conventions
- Component setup lives in `agent/README.md`, `server/README.md`, `web/README.md`, and `deploy/README.md`.
- API contracts live in `api/openapi.yaml`; `api/README.md` only describes contract rules.
- Keep docs factual and current. Avoid placeholder docs when a real config or script already exists.

View File

@@ -1,68 +0,0 @@
# Deployment
This is an initial deployment plan, not a production runbook.
## Agent
Agent runs as a systemd service.
Expected paths:
- binary: `/usr/local/bin/monlet-agent`;
- config: `/etc/monlet/agent.toml`;
- state: `/var/lib/monlet-agent`;
- logs: journald.
The agent should support graceful shutdown and avoid overlapping runs for the same check.
## Server
Server runs in Docker.
Production storage:
- PostgreSQL.
Local/demo storage:
- SQLite may be supported later, but must be documented as non-production.
Required deployment features:
- TLS-ready reverse proxy;
- agent API tokens;
- request size limits;
- structured logs;
- `/health`;
- `/ready`;
- `/metrics`.
Docker image rule:
- build server dependencies inside the image;
- do not require Python packages installed on the host;
- do not mount host virtualenvs into the container;
- local non-Docker development uses `server/.venv` only.
## Web
Web is read-only for monitoring state in v1. Agent admission and blacklist controls are the explicit write exception.
Auth plan:
- local web login or equivalent gateway auth through a trusted reverse proxy;
- no full RBAC in v1.
## Observability
Optional examples:
- Prometheus scrape config;
- Alertmanager route config;
- Grafana dashboard stub.
Metrics must avoid high-cardinality labels.
## Backups
PostgreSQL backup/restore docs are required before first production use.

66
docs/ops/deployment.md Normal file
View File

@@ -0,0 +1,66 @@
# Deployment Notes
These notes describe the intended production shape. Use
[`docs/ops/deployment-checklist.md`](deployment-checklist.md) for the
step-by-step first stand-up checklist.
## Agent
The agent runs on monitored hosts as a systemd service.
Expected paths:
- binary: `/usr/local/bin/monlet-agent`
- config: `/etc/monlet/agent.toml`
- state: `/var/lib/monlet-agent`
- logs: journald
The agent reads only local config. It does not accept remote command execution or
remote config push from the server. Per-check scheduling is local to the agent,
and overlapping runs for the same check are skipped.
## Server
The server runs as a containerized FastAPI application backed by PostgreSQL 16+.
It owns inventory, current check state, incidents, notification outbox state,
retention, and public `/api/v1` contracts.
Required production boundaries:
- TLS and rate limiting at a reverse proxy
- PostgreSQL as canonical storage
- `MONLET_AUTH_TOKEN` stored in a secret manager
- request body limits enabled
- structured logs with auth material redacted
- `/api/v1/health`, `/api/v1/ready`, and `/metrics` exposed for operations
Docker images install dependencies inside the image. Do not mount host virtual
environments into server containers.
## Web
The web app is an operator dashboard. Monitoring pages are read-only; agent
admission and blacklist controls are the explicit v1 write surface.
Protect the UI with either local web login or a trusted auth proxy. If local
login is enabled, rate-limit `/login` at the reverse proxy. Monlet v1 does not
provide full RBAC.
## Observability
The local deploy stack includes optional Prometheus, Alertmanager, and Grafana
examples under `deploy/`. Production deployments should at minimum scrape:
- Monlet server `/metrics`
- each agent `/metrics`
- reverse-proxy and PostgreSQL metrics from the surrounding environment
Metric labels must stay low-cardinality. Do not put timestamps, request IDs, or
unbounded host-local values into check IDs or labels.
## Backups
PostgreSQL is the canonical state. Follow
[`docs/ops/backup-restore.md`](backup-restore.md) before first production
use. Agent spools are best-effort buffers and are not a substitute for database
backups.

View File

@@ -1,3 +1,12 @@
# Examples # Examples
Examples for checks, configs, and local demos. Small payloads and scripts used by local smoke tests and manual API checks.
Files:
- `heartbeat.json` - example agent heartbeat request body.
- `event-batch.json` - example event batch request body.
- `checks/example_check.sh` - minimal shell check that follows the Monlet exit-code contract.
Use these examples as templates only. Real agents normally send heartbeat and
event payloads through the Go agent, not by hand-written curl requests.

View File

@@ -1,25 +1,47 @@
MONLET_AUTH_TOKEN=replace-me-with-random-token MONLET_AUTH_TOKEN=replace-me-with-random-token
MONLET_DATABASE_URL=postgresql+asyncpg://monlet:monlet@localhost:5432/monlet MONLET_DATABASE_URL=postgresql+asyncpg://monlet:monlet@localhost:5432/monlet
MONLET_LOG_LEVEL=INFO MONLET_LOG_LEVEL=INFO
MONLET_HOST=0.0.0.0
MONLET_PORT=8000
MONLET_BODY_LIMIT_BYTES=1048576 MONLET_BODY_LIMIT_BYTES=1048576
MONLET_OUTPUT_MAX_BYTES=8192 MONLET_OUTPUT_MAX_BYTES=8192
MONLET_MAX_BATCH_EVENTS=200
MONLET_ENABLE_DETECTOR=true
MONLET_STALE_AFTER_SEC=20 MONLET_STALE_AFTER_SEC=20
MONLET_DEAD_AFTER_SEC=30 MONLET_DEAD_AFTER_SEC=30
MONLET_DETECTOR_TICK_SEC=5 MONLET_DETECTOR_TICK_SEC=5
MONLET_ENABLE_DETECTOR=true
MONLET_DB_POOL_SIZE=10
MONLET_DB_MAX_OVERFLOW=10
MONLET_DB_POOL_TIMEOUT_SEC=30
MONLET_DB_POOL_RECYCLE_SEC=1800
MONLET_EVENTS_RETENTION_MONTHS=36 MONLET_EVENTS_RETENTION_MONTHS=36
MONLET_EVENTS_FUTURE_PARTITIONS=3
MONLET_PARTITION_MAINTENANCE_INTERVAL_SEC=3600
MONLET_OUTBOX_RETENTION_MAX_ROWS=50000
MONLET_OUTBOX_PRUNE_BATCH_SIZE=5000
MONLET_EVENT_DEDUP_RETENTION_DAYS=30
MONLET_EVENT_DEDUP_PRUNE_BATCH_SIZE=5000
MONLET_MAX_PENDING_AGENTS=10000 MONLET_MAX_PENDING_AGENTS=10000
MONLET_PENDING_AGENT_TTL_DAYS=7 MONLET_PENDING_AGENT_TTL_DAYS=7
MONLET_PENDING_AGENT_PRUNE_BATCH_SIZE=1000 MONLET_PENDING_AGENT_PRUNE_BATCH_SIZE=1000
MONLET_OUTBOX_RETENTION_MAX_ROWS=50000
MONLET_ENABLE_NOTIFIER_WORKER=true MONLET_ENABLE_NOTIFIER_WORKER=true
MONLET_NOTIFIER_TICK_SEC=5 MONLET_NOTIFIER_TICK_SEC=5
MONLET_NOTIFIER_BATCH_SIZE=20 MONLET_NOTIFIER_BATCH_SIZE=20
MONLET_NOTIFIER_CONCURRENCY=4
MONLET_NOTIFIER_MAX_ATTEMPTS=8 MONLET_NOTIFIER_MAX_ATTEMPTS=8
MONLET_NOTIFIER_LEASE_SEC=300 MONLET_NOTIFIER_LEASE_SEC=300
MONLET_NOTIFIER_HTTP_TIMEOUT_SEC=10 MONLET_NOTIFIER_HTTP_TIMEOUT_SEC=10
MONLET_NOTIFIER_INCLUDE_OUTPUT=true
MONLET_NOTIFIER_MAX_OUTPUT_BYTES=1024
MONLET_NOTIFICATION_TIME_ZONE=UTC MONLET_NOTIFICATION_TIME_ZONE=UTC
MONLET_NOTIFIER_DEBUG_ENABLED=true MONLET_NOTIFIER_DEBUG_ENABLED=true
MONLET_NOTIFIER_TELEGRAM_ENABLED=false MONLET_NOTIFIER_TELEGRAM_ENABLED=false
MONLET_NOTIFIER_TELEGRAM_TOKEN= MONLET_NOTIFIER_TELEGRAM_TOKEN=

View File

@@ -33,26 +33,38 @@ UV_PROJECT_ENVIRONMENT=.venv UV_CACHE_DIR=../.cache/uv uv run fastapi dev monlet
| `MONLET_AUTH_TOKEN` | required | UI/operator Bearer token. Space/comma-separated list; any accepted. `changeme` is rejected. | | `MONLET_AUTH_TOKEN` | required | UI/operator Bearer token. Space/comma-separated list; any accepted. `changeme` is rejected. |
| `MONLET_DATABASE_URL` | `postgresql+asyncpg://monlet:monlet@127.0.0.1:5432/monlet` | Async DSN | | `MONLET_DATABASE_URL` | `postgresql+asyncpg://monlet:monlet@127.0.0.1:5432/monlet` | Async DSN |
| `MONLET_LOG_LEVEL` | `INFO` | Log level | | `MONLET_LOG_LEVEL` | `INFO` | Log level |
| `MONLET_HOST` | `0.0.0.0` | HTTP bind host for direct server runs |
| `MONLET_PORT` | `8000` | HTTP bind port for direct server runs |
| `MONLET_BODY_LIMIT_BYTES` | `1048576` | Request body limit (ADR-0005) | | `MONLET_BODY_LIMIT_BYTES` | `1048576` | Request body limit (ADR-0005) |
| `MONLET_OUTPUT_MAX_BYTES` | `8192` | Per-event output limit | | `MONLET_OUTPUT_MAX_BYTES` | `8192` | Per-event output limit |
| `MONLET_MAX_BATCH_EVENTS` | `200` | Maximum events per ingestion batch |
| `MONLET_STALE_AFTER_SEC` | `20` | Stale threshold | | `MONLET_STALE_AFTER_SEC` | `20` | Stale threshold |
| `MONLET_DEAD_AFTER_SEC` | `30` | Dead threshold | | `MONLET_DEAD_AFTER_SEC` | `30` | Dead threshold |
| `MONLET_DETECTOR_TICK_SEC` | `5` | Detector tick | | `MONLET_DETECTOR_TICK_SEC` | `5` | Detector tick |
| `MONLET_ENABLE_DETECTOR` | `true` | Toggle detector task | | `MONLET_ENABLE_DETECTOR` | `true` | Toggle detector task |
| `MONLET_DB_POOL_SIZE` | `10` | Base async DB pool size |
| `MONLET_DB_MAX_OVERFLOW` | `10` | Extra DB connections above pool size |
| `MONLET_DB_POOL_TIMEOUT_SEC` | `30` | Seconds to wait for a DB connection |
| `MONLET_DB_POOL_RECYCLE_SEC` | `1800` | Recycle DB connections by age |
| `MONLET_EVENTS_RETENTION_MONTHS` | `36` | Months of `events` partitions to keep; `0` disables drop | | `MONLET_EVENTS_RETENTION_MONTHS` | `36` | Months of `events` partitions to keep; `0` disables drop |
| `MONLET_EVENTS_FUTURE_PARTITIONS` | `3` | Future month partitions to pre-create | | `MONLET_EVENTS_FUTURE_PARTITIONS` | `3` | Future month partitions to pre-create |
| `MONLET_EVENT_DEDUP_RETENTION_DAYS` | `30` | Idempotency window for `event_id` | | `MONLET_EVENT_DEDUP_RETENTION_DAYS` | `30` | Idempotency window for `event_id` |
| `MONLET_EVENT_DEDUP_PRUNE_BATCH_SIZE` | `5000` | Dedup rows pruned per maintenance tick |
| `MONLET_MAX_PENDING_AGENTS` | `10000` | Maximum pending agent keys before heartbeat rejects with 429 | | `MONLET_MAX_PENDING_AGENTS` | `10000` | Maximum pending agent keys before heartbeat rejects with 429 |
| `MONLET_PENDING_AGENT_TTL_DAYS` | `7` | Pending key cleanup TTL; `0` disables pruning | | `MONLET_PENDING_AGENT_TTL_DAYS` | `7` | Pending key cleanup TTL; `0` disables pruning |
| `MONLET_PENDING_AGENT_PRUNE_BATCH_SIZE` | `1000` | Pending cleanup rows per detector tick | | `MONLET_PENDING_AGENT_PRUNE_BATCH_SIZE` | `1000` | Pending cleanup rows per detector tick |
| `MONLET_PARTITION_MAINTENANCE_INTERVAL_SEC` | `3600` | Partition create/drop worker interval | | `MONLET_PARTITION_MAINTENANCE_INTERVAL_SEC` | `3600` | Partition create/drop worker interval |
| `MONLET_OUTBOX_RETENTION_MAX_ROWS` | `50000` | Maximum retained terminal outbox rows; `0` disables pruning | | `MONLET_OUTBOX_RETENTION_MAX_ROWS` | `50000` | Maximum retained terminal outbox rows; `0` disables pruning |
| `MONLET_OUTBOX_PRUNE_BATCH_SIZE` | `5000` | Outbox rows pruned per maintenance tick |
| `MONLET_ENABLE_NOTIFIER_WORKER` | `true` | Toggle outbox worker | | `MONLET_ENABLE_NOTIFIER_WORKER` | `true` | Toggle outbox worker |
| `MONLET_NOTIFIER_TICK_SEC` | `5` | Worker poll interval | | `MONLET_NOTIFIER_TICK_SEC` | `5` | Worker poll interval |
| `MONLET_NOTIFIER_BATCH_SIZE` | `20` | Rows claimed per tick | | `MONLET_NOTIFIER_BATCH_SIZE` | `20` | Rows claimed per tick |
| `MONLET_NOTIFIER_CONCURRENCY` | `4` | Parallel notification deliveries per tick |
| `MONLET_NOTIFIER_MAX_ATTEMPTS` | `8` | Terminal-failure threshold (ADR-0005) | | `MONLET_NOTIFIER_MAX_ATTEMPTS` | `8` | Terminal-failure threshold (ADR-0005) |
| `MONLET_NOTIFIER_LEASE_SEC` | `300` | Recovery lease for stuck `sending` rows | | `MONLET_NOTIFIER_LEASE_SEC` | `300` | Recovery lease for stuck `sending` rows |
| `MONLET_NOTIFIER_HTTP_TIMEOUT_SEC` | `10` | Outbound HTTP timeout | | `MONLET_NOTIFIER_HTTP_TIMEOUT_SEC` | `10` | Outbound HTTP timeout |
| `MONLET_NOTIFIER_INCLUDE_OUTPUT` | `true` | Include redacted check output in notifications |
| `MONLET_NOTIFIER_MAX_OUTPUT_BYTES` | `1024` | Maximum notification output bytes after redaction |
| `MONLET_NOTIFICATION_TIME_ZONE` | `UTC` | IANA timezone for human-readable notification timestamps | | `MONLET_NOTIFICATION_TIME_ZONE` | `UTC` | IANA timezone for human-readable notification timestamps |
| `MONLET_NOTIFIER_DEBUG_ENABLED` | `true` | Log-only notifier | | `MONLET_NOTIFIER_DEBUG_ENABLED` | `true` | Log-only notifier |
| `MONLET_NOTIFIER_TELEGRAM_ENABLED` | `false` | Telegram notifier on/off | | `MONLET_NOTIFIER_TELEGRAM_ENABLED` | `false` | Telegram notifier on/off |

View File

@@ -41,6 +41,7 @@ Do not install Node packages globally. Project dependencies live in `web/node_mo
| `MONLET_WEB_SESSION_SECRET` | _(none)_ | Secret used to sign the web session cookie; required with local web login; at least 32 bytes | | `MONLET_WEB_SESSION_SECRET` | _(none)_ | Secret used to sign the web session cookie; required with local web login; at least 32 bytes |
| `MONLET_WEB_SESSION_TTL_SEC` | `604800` | Local web session lifetime | | `MONLET_WEB_SESSION_TTL_SEC` | `604800` | Local web session lifetime |
| `MONLET_WEB_TRUST_PROXY_AUTH` | `false` | Trust `X-Forwarded-User` from an upstream auth proxy | | `MONLET_WEB_TRUST_PROXY_AUTH` | `false` | Trust `X-Forwarded-User` from an upstream auth proxy |
| `NEXT_PUBLIC_MONLET_POLL_MS` | `10000` | Browser polling interval in ms; build-time for production Next.js bundles |
The API token never reaches the browser — all server calls happen in Server Components / route handlers. Agent admission actions are blocked unless local web login is configured or trusted proxy auth is enabled. The API token never reaches the browser — all server calls happen in Server Components / route handlers. Agent admission actions are blocked unless local web login is configured or trusted proxy auth is enabled.