agent ops architecture (2 mac minis → n nodes): specs + checklist

agent_ops_architecture_spec.md

agent ops architecture (2 mac minis → n nodes)

north star: separate decision + state from side effects.

• mini #1 (coordinator / research): planning, triage, drafting, approvals, canonical state, audit log.
• mini #2 (executor / side effects): x posting (api), email sends, calendar writes, deployments, credentials.
• rule: #1 never posts/sends/schedules. #2 never decides.

---

transport + storage (scales)

start with append-only intents + receipts on a shared path, and trigger execution via openclaw nodes.

• shared path (initial): ~/shared/openclaw/{queues,state,receipts,deadletter}
• when you add more nodes: swap shared folder → postgres or redis streams without changing the contracts.

---

hard specs (contracts)

these constraints prevent double-posts, drift, and distributed chaos.

1. idempotency key (required)

• every side-effect intent has a stable idempotencyKey
• receipts must reference it
• dedupe is based on receipts/state, not hope

2. lease + ttl (required)

• executor takes a lease before performing side effects
• lease has ttl so another executor can safely take over after crash

3. central rate limits (required)

• rate limits are enforced per account/channel, centrally on the executor
• adding more workers must not increase spam risk

4. append-only receipts (required)

• every action emits a receipt: {intentId, idempotencyKey, status, externalId, url, error}
• receipts are the audit log and the truth surface

5. secrets isolation (required)

• write-privileged tokens live only on the executor box
• coordinator has zero credentials that can post/send/schedule
• .secrets/ is ignored and never pushed

---

intent + receipt schemas

intent (coordinator → queue)

{
  "intentId": "uuid-or-sha1",
  "createdAt": "ISO-8601",
  "channel": "x|email|calendar",
  "action": "post|reply|quote|send_email|create_event",
  "account": "antihunter59823|antifund",
  "payload": {},
  "idempotencyKey": "sha1(channel+account+action+payload-stable)",
  "approvalRequired": true,
  "approved": false,
  "priority": 1,
  "retry": { "max": 2, "backoffSec": [30, 120] }
}

receipt (executor → receipts log)

{
  "intentId": "...",
  "idempotencyKey": "...",
  "executedAt": "ISO-8601",
  "executorNode": "mini-2",
  "status": "success|failed|skipped",
  "externalId": "tweet_id|gmail_msg_id|calendar_event_id",
  "url": "https://...",
  "error": "string-if-any"
}

---

implementation checklist (when mini #2 arrives)

1. install openclaw on mini #2; pair it as a node.
2. create shared folder: ~/shared/openclaw/ with queues/ state/ receipts/ deadletter/.
3. set up syncthing (or sshfs) to sync that folder.
4. move all write secrets to mini #2 only (.secrets never synced; never committed).
5. wire mini #1 producers to write intents into shared queues (x/email/calendar).
6. wire mini #2 consumers to execute intents + write receipts + update state.
7. enforce approval gates (especially for @antifund).
8. add kill switch: shared executor_gate.json (enabled=false pauses side effects).
9. add global rate limiter + circuit breaker on executor.
10. add a “what’s live” dashboard script on mini #1 (reads receipts + state).

---

notes

when scaling beyond 2 nodes, keep the same intent/receipt contracts and swap storage: shared folder → postgres/redis streams.

receipts-first beats vibes-first.