--- title: "Peer Swarm Heartbeat" description: "Internal endpoint that tracks distributed OpenClaw gateway nodes across multiple machines for Intelligence Dashboard aggregation" canonical: "https://docs.ainative.studio/docs/agent-cloud/peer-swarm" last-updated: "2026-10-13T21:42:35.442Z" --- # Peer Swarm Heartbeat Source: https://docs.ainative.studio/docs/agent-cloud/peer-swarm > Internal endpoint that tracks distributed OpenClaw gateway nodes across multiple machines for Intelligence Dashboard aggregation Internal endpoint that tracks distributed OpenClaw gateway nodes across multiple machines, enabling the Intelligence Dashboard to aggregate swarm activity without direct LAN access. :::caution AINative-internal infrastructure, not a third-party integration point This endpoint requires no authentication and is intended only for AINative's own OpenClaw gateway machines reporting their status. It is documented here for completeness, not as something external developers should call — there's no agent-cloud use case this serves for a third-party integration. ::: Refs #3340 --- ## Problem The AINative platform runs OpenClaw agent swarms on multiple physical machines (laptops, servers, cloud instances). Railway-hosted services cannot reach these machines over LAN because Railway's private network is isolated from local networks. The heartbeat system solves this by having each peer push its status to the centralized backend. ## Architecture ``` +---------------------+ HTTPS POST +--------------------+ | Peer Machine A | ----------------------> | Railway Backend | | (OpenClaw Gateway) | /peer-swarm-heartbeat | (FastAPI) | +---------------------+ | | | peer_swarm_ | +---------------------+ HTTPS POST | heartbeat table | | Peer Machine B | ----------------------> | | | (OpenClaw Gateway) | /peer-swarm-heartbeat +--------+-----------+ +---------------------+ | | reads v +--------------------+ | Intelligence | | Dashboard | | (aggregated view) | +--------------------+ ``` Each peer machine runs a **peer-swarm-watchdog cron** every 5 minutes. The cron: 1. Calls `openclaw gateway call health --json` on the local LAN gateway 2. Extracts agent count and loop count from the response 3. POSTs the data to the Railway backend via the public URL The backend upserts the data into the `peer_swarm_heartbeat` table. The Intelligence Dashboard reads from this table to show aggregate swarm metrics. --- ## Endpoint ### `POST /api/v1/internal/peer-swarm-heartbeat` Records (upserts) the latest heartbeat from a peer OpenClaw gateway node. **Status Code:** `204 No Content` on success **Authentication:** Internal-only. Behind Kong internal route -- no user-facing auth required. ### Request Body | Field | Type | Required | Description | |--------------------|---------------|----------|----------------------------------------------------------------| | `peer_id` | `string` | Yes | Unique identifier for the peer node (e.g., `tobys-macbook-air`) | | `agents_active` | `integer` | Yes | Number of active agents reported by the peer gateway. Must be >= 0. | | `improvement_loops`| `integer` | No | Session/loop count from the peer gateway. Defaults to `0`. Must be >= 0. | | `gateway_version` | `string\|null` | No | OpenClaw version string (e.g., `0.8.54`). Defaults to `null`. | ### Database Table ```sql CREATE TABLE peer_swarm_heartbeat ( peer_id TEXT PRIMARY KEY, agents_active INTEGER NOT NULL DEFAULT 0, improvement_loops INTEGER NOT NULL DEFAULT 0, gateway_version TEXT, last_seen_at TIMESTAMPTZ NOT NULL DEFAULT NOW() ); CREATE INDEX idx_peer_swarm_heartbeat_last_seen ON peer_swarm_heartbeat(last_seen_at DESC); ``` The `peer_id` is the primary key. Each POST performs an `INSERT ... ON CONFLICT (peer_id) DO UPDATE`, so the table always holds one row per peer with the latest values. ### Upsert Behavior - If a peer with the given `peer_id` does not exist, a new row is inserted. - If the peer already exists, all fields (`agents_active`, `improvement_loops`, `gateway_version`, `last_seen_at`) are overwritten with the new values. - `last_seen_at` is always set to the current UTC timestamp by the server, not by the caller. --- ## Staleness Detection The system considers a peer **offline** if its `last_seen_at` timestamp is older than the staleness cutoff: ``` PEER_HEARTBEAT_MAX_AGE_SECS = 600 (10 minutes) ``` This value is configurable via the `PEER_HEARTBEAT_MAX_AGE_SECS` environment variable. Since the watchdog cron runs every 5 minutes, a peer that misses two consecutive heartbeats is considered offline. When the Intelligence Dashboard queries the table, it applies a `WHERE last_seen_at >= :cutoff` filter, so stale peers are excluded from all aggregate metrics. ### What happens when a peer goes offline - Its `agents_active` and `improvement_loops` are no longer included in the dashboard totals. - The peer row remains in the table (it is not deleted). If the peer comes back online, the next heartbeat upsert updates the row and it becomes visible again. --- ## Intelligence Dashboard Aggregation The `GET /api/v1/public/platform/intelligence` endpoint reads peer swarm data through two helper functions: ### `_get_peer_swarm_stats(db)` Aggregates across all online peers: ```sql SELECT COALESCE(SUM(agents_active), 0) AS total_agents, COALESCE(SUM(improvement_loops), 0) AS total_loops, COUNT(*) AS online_peers FROM peer_swarm_heartbeat WHERE last_seen_at >= :cutoff ``` The returned `agents_active` is added to the dashboard's total `agents_active` count (alongside heartbeat-based and Agent Cloud counts). The `improvement_loops` are added to the dashboard's total `improvement_loops`. ### `_get_peer_swarm_events(db)` Generates per-agent event cards from the agent roster and run log, plus machine-level summary cards from the heartbeat table. Each online peer produces an `AgentEvent` card that shows: - The peer name (formatted from `peer_id`, e.g., "Tobys Macbook Air") - The number of active agents (e.g., "running 12 agents") - How recently the heartbeat was received ### `_get_tasks_completed_today(db)` The `tasks_completed_today` metric on the dashboard includes peer swarm data: ```sql SELECT COALESCE(SUM(improvement_loops), 0) FROM peer_swarm_heartbeat WHERE last_seen_at >= :cutoff ``` Each `improvement_loops` count represents completed watchdog/cron cycles, so this supplements the formal `agent_run_log` task count. --- ## Example Requests ### Basic heartbeat ```bash curl -X POST https://api.ainative.studio/api/v1/internal/peer-swarm-heartbeat \ -H "Content-Type: application/json" \ -d '{ "peer_id": "tobys-macbook-air", "agents_active": 12, "improvement_loops": 47, "gateway_version": "0.8.54" }' ``` Response: `204 No Content` ### Minimal heartbeat (no optional fields) ```bash curl -X POST https://api.ainative.studio/api/v1/internal/peer-swarm-heartbeat \ -H "Content-Type: application/json" \ -d '{ "peer_id": "build-server-01", "agents_active": 4 }' ``` Response: `204 No Content` ### Querying the dashboard to see aggregated data ```bash curl -s https://api.ainative.studio/api/v1/public/platform/intelligence \ | python3 -m json.tool ``` The response includes `stats.agents_active` and `stats.improvement_loops`, both of which incorporate peer swarm data from all online peers. --- ## Watchdog Cron Setup The peer-swarm-watchdog is typically configured as a system cron on each machine: ```cron */5 * * * * /path/to/peer-swarm-watchdog.sh ``` The script collects local gateway health data and POSTs it to the Railway backend. Since the local machine can reach the LAN gateway (which Railway cannot), this push-based design bridges the network gap. --- ## Related Files | File | Purpose | |------|---------| | `src/backend/app/api/internal/peer_swarm_heartbeat.py` | Endpoint implementation | | `src/backend/app/api/v1/endpoints/platform_intelligence.py` | Dashboard aggregation logic | | `scripts/sync-production-schema.py` | Table creation (search for `peer_swarm_heartbeat`) |