Add cron schedules and sync docs
Some checks failed
ci / openapi (push) Failing after 7s
ci / agent (push) Failing after 5s
ci / server (push) Failing after 6s
ci / stack-smoke (push) Has been skipped
ci / web (push) Failing after 5s

This commit is contained in:
Stanislav Rossovskii
2026-06-23 19:18:01 +04:00
parent f5ec97fcb9
commit d6f9335398
24 changed files with 638 additions and 131 deletions

View File

@@ -14,7 +14,7 @@ Do not use `go install` or `go env -w`. Module cache, build cache, and tool bina
## Configuration
See `config.example.toml`. Required fields: top-level `state_dir`, at least one `[[checks]]` (`id`, `command`, `interval`, `timeout`), and at least one output enabled: `server.enabled = true` or `metrics.enabled = true`. `agent_id` is optional; if omitted, it defaults to the OS hostname. `hostname` is not a config key.
See `config.example.toml`. Required fields: top-level `state_dir`, at least one `[[checks]]` (`id`, `command`, `timeout`, and exactly one of `interval` or `cron`), and at least one output enabled: `server.enabled = true` or `metrics.enabled = true`. `agent_id` is optional; if omitted, it defaults to the OS hostname. `hostname` is not a config key.
| Key | Default | Notes |
|---|---|---|
@@ -25,10 +25,12 @@ See `config.example.toml`. Required fields: top-level `state_dir`, at least one
| `server.batch_interval` | `10s` | events flush cadence |
| `metrics.listen` | `127.0.0.1:9465` | low-cardinality only |
| `checks[].command` | required | Shell string executed as `/bin/sh -c`. Both forms accepted: `command = "check_disk --warn 80 --crit 90"` and triple-quoted `command = '''...'''`. Argv arrays are rejected. See [Command security contract](#command-security-contract). |
| `checks[].interval` | required unless `cron` is set | Go duration such as `60s`, `5m`, or `1h`. Interval checks run once immediately on start/reload, then on the interval. |
| `checks[].cron` | required unless `interval` is set | Standard 5-field cron expression, macros such as `@hourly`/`@daily`, and optional `CRON_TZ=...`. Seconds fields, `@every`, and `@reboot` are rejected. Cron checks wait for the next scheduled slot; they do not run immediately on start/reload. |
| `checks[].notifications_enabled` | `true` | Set `false` for checks that must not create server notifications. |
| `checks[].resource_limits` | `{}` | Optional per-check best-effort `ulimit` values: `cpu_time`, `memory`, `open_files`. |
Examples — one-line and multi-line forms:
Examples — interval, cron, and multi-line forms:
```toml
[[checks]]
@@ -37,6 +39,12 @@ command = "test $(cut -d. -f1 /proc/uptime) -gt 60"
interval = "60s"
timeout = "5s"
[[checks]]
id = "daily_backup"
command = "/usr/local/lib/monlet/check_backup_freshness.sh"
cron = "CRON_TZ=UTC 15 6 * * *"
timeout = "2m"
[[checks]]
id = "disk_root"
command = '''
@@ -81,16 +89,17 @@ executes it as `/bin/sh -c <command>` on the host where the agent runs.
Send `SIGHUP` or run `systemctl reload monlet-agent` to reload the local TOML file.
- Reload first validates the full new config. On error, the old config keeps running.
- Hot-reloadable: labels and checks, including `command`, `interval`, `timeout`,
`notifications_enabled`, `dedupe_key`, and `resource_limits`.
- Hot-reloadable: labels and checks, including `command`, `interval`, `cron`,
`timeout`, `notifications_enabled`, `dedupe_key`, and `resource_limits`.
- Restart required: `agent_id`, `state_dir`, `server.*`, and `metrics.*`.
- A check already running during reload is allowed to finish under its old config.
Removed or changed checks do not start new old-config runs; changed checks start
with the new config after the in-flight run finishes.
Removed or changed checks do not start new old-config runs. Changed interval
checks start with the new config after the in-flight run finishes; changed cron
checks wait for the next cron slot.
## Behaviour
- Per-check scheduler with no-overlap: a tick is skipped if the previous run is still in flight (`monlet_agent_check_skipped_total`).
- Per-check scheduler with no-overlap: a tick/cron slot is skipped if the previous run is still in flight (`monlet_agent_check_skipped_total`).
- Check commands run through `/bin/sh -c`; the old argv-array form is intentionally unsupported.
- Timeout via `exec.CommandContext` with `Setpgid` so the process group is killed on expiry; timeout → `critical`.
- Exit code mapping: 0=ok, 1=warning, 2=critical, 3+=unknown (ADR-0005).
@@ -115,7 +124,7 @@ Exposed on `metrics.listen` at `/metrics`. Labels are static (`check_id`, `statu
- `monlet_agent_check_exit_code{check_id}`
- `monlet_agent_check_last_run_timestamp_seconds{check_id}`
- `monlet_agent_check_last_success_timestamp_seconds{check_id}`
- `monlet_agent_check_interval_seconds{check_id}`
- `monlet_agent_check_interval_seconds{check_id}` (interval checks only)
- `monlet_agent_checks_configured`
- `monlet_agent_config_load_success`
- `monlet_agent_config_reload_total{result}`