Harden production workflows and agent admission

docs/ops/token-rotation.md

@@ -1,20 +1,33 @@
 # Token rotation
 
-Monlet v1 uses a single shared Bearer token (`MONLET_AUTH_TOKEN`) for all agent and UI endpoints (ADR-0007). Per-agent tokens and dual-token acceptance are out of scope for v1.
+Monlet uses split Bearer-token auth (ADR-0007):
 
-## Why rotation is disruptive in v1
+- **UI token** — `MONLET_AUTH_TOKEN` env. Accepted by every UI/operator endpoint. The variable holds one or more space/comma-separated tokens; the server accepts any of them, so rotation is zero-loss.
+- **Agent key** — generated locally by each agent and stored in `state_dir/agent.key`. It is not configured on the server; operators accept, block, remove, or blacklist keys through the Agents UI/API.
 
-The server validates exactly one token at a time. Agents that present the old token after the server has switched receive `401`, which the agent treats as a **non-retryable** failure and drops the affected batch from its local spool (`agent/internal/client/client.go`, `agent/internal/app/app.go`). Events flushed during the mismatch window are lost — they are not re-spooled and not re-sent.
+Rotate UI tokens independently from agent admission. Changing `MONLET_AUTH_TOKEN` does not affect accepted agents.
 
-Plan rotations during a low-traffic window and either stop the agents first (so spooled events persist on disk until they restart with the new token) or accept the event-loss window.
+## How multi-token overlap works
+
+`MONLET_AUTH_TOKEN` is a list: tokens separated by spaces and/or commas. Examples:
+
+```sh
+MONLET_AUTH_TOKEN="single-token"
+MONLET_AUTH_TOKEN="old-token new-token"
+MONLET_AUTH_TOKEN="old-token,new-token"
+```
+
+All listed tokens are accepted. The middleware compares in constant time across the full list, so timing-side-channels do not leak which token matched.
+
+Logs and metrics never include token values.
 
 ## When to rotate
 
-- A suspected leak (token committed, log exposure, lost laptop with `config.toml`).
+- Suspected UI token leak.
 - Operator turnover.
 - Routine schedule (e.g., quarterly).
 
-## Procedure
+## Zero-loss procedure
 
 1. **Generate a new token**:
 
@@ -22,24 +35,39 @@ Plan rotations during a low-traffic window and either stop the agents first (so
    openssl rand -hex 32
    ```
 
-2. **Stop or pause agents** (recommended). Either:
-   - `systemctl stop monlet-agent` on each host; spooled events remain on disk and replay after restart, or
-   - skip this step and accept that any events sent during steps 3–4 are lost on the `401`.
+2. **Add the new token alongside the old one** in the server environment:
 
-3. **Restart server** with the new `MONLET_AUTH_TOKEN`. Verify:
+   ```sh
+   MONLET_AUTH_TOKEN="$OLD $NEW"
+   ```
+
+   Restart (or reload) the server. The old token still works, the new one is now accepted as well.
+
+3. **Move clients to the new token**:
+   - Update Web UI `MONLET_API_TOKEN` and restart.
+   - Update any operator scripts or curl examples to use the new value.
+
+4. **Verify everything uses the new token**:
 
    ```sh
    curl -sf -H "Authorization: Bearer $NEW" http://server:8000/api/v1/agents | jq .
+   # Optional: tail server logs for unauthorized attempts during the window.
    ```
 
-4. **Update agent `config.toml`** (`[server].token`) on each host and start agents.
+5. **Remove the old token** from the server environment:
 
-5. **Update Web UI** (`MONLET_API_TOKEN`) and restart.
+   ```sh
+   MONLET_AUTH_TOKEN="$NEW"
+   ```
 
-6. **Revoke the old token**: remove the old value from secret stores, CI variables, and operator notes.
+   Restart the server. From this point the old token is revoked.
+
+6. **Scrub the old value** from secret stores, CI variables, and operator notes.
 
 ## Notes
 
-- The token is the only auth in v1; protect it like a database password.
-- The token never appears in metrics labels, incident keys, or notification labels (ADR-0007, redaction policy).
+- Tokens are the only auth in v1; protect them like a database password.
+- Tokens never appear in metrics labels, incident keys, or notification labels (ADR-0007, redaction policy).
 - The server redacts `Authorization`/`Cookie`-style headers before logging (`monlet_server/redaction.py`).
+- The middleware uses `hmac.compare_digest` against every accepted token; rotation does not introduce a timing oracle.
+- To rotate an individual agent key, delete `state_dir/agent.key` on that host and restart the agent. The server will show a pending replacement for the same hostname; accept it after verifying the host.