Synapse Ingestion Engine

Synapse is a brokerless, edge-first IoT ingestion engine using a symmetric P2P mesh. Using ZeroMQ, SQLite timeseries database, and Uber H3 spatial analysis, it offers decentralized telemetry ingestion. Every node is identical with no central brokers, no monitors, and no single points of failure. Nodes discover each other automatically via Zero-Touch Provisioning (mDNS/Zeroconf), validate sensor readings locally through H3 spatial corroboration, compress payloads using pluggable strategies (zlib, lz4, zstd), and secure communication via CurveZMQ and Caddy TLS.

Prerequisites

• Python 3.11+
• Docker (or rootless Podman auto-detected by the Makefile)
• Docker Compose v2

1. Start the P2P Mesh (Recommended)

The Makefile is the primary interface. make start builds and deploys a simulated local mesh of 4 identical nodes, each alongside its secure Caddy proxy:

2. Accessing the Dashboard

Each node serves its own localized dashboard via HTTPS through the Caddy sidecar. Open your browser and navigate to the secure endpoint bypass the local development certificate warning once:

• Green Hexagons: All active sensors in the H3 cell are healthy and congruent.
• Yellow/Orange Hexagons: One or more sensors are FAULTY or DEAD.
• Red Hexagons: The majority of sensors in the cell have failed local corroboration.
• Connected Peers sidebar: Navigate between the perspectives of all peers in the mesh without centralized aggregation.

3. Manual Python Development Setup

To develop and run a single node directly in Python without Docker:

Synapse v0.6.0 adopts a fully Symmetric P2P Mesh model. Unlike hub-and-spoke topologies that depend on a central broker or monitor daemon, every participant is an identical Node that simultaneously senses data, ingests peer telemetry, validates anomalies, and serves its own operator dashboard.

Core Components

Data Flow

1. Geospatial Mesh via H3 Grid

Geographic coordinates are transformed into a single 64-bit integer H3 cell representing a hexagonal region. The default resolution of 7 covers an average area of 4.3 km² with an edge length of approximately 1.2 km. Peers are defined strictly as nodes occupying the same cell. Subscriptions extend to the k-ring 1 neighborhood (the 7 immediately adjacent cells) for cross-cell validation.

2. Node State Machine

Each node in the registry can be in exactly one of three states:

Any state → ALIVE on next valid ping (recovery is instantaneous).

3. Pluggable Outlier Detection (Leave-One-Out Corroboration)

When a sensor publishes telemetry, the node triggers a Leave-One-Out (LOO) test. For node n reporting value x_n inside cell C, the peer comparison set is:

Corroboration only runs if |P_n| ≥ CORROBORATION_MIN_PEERS (default 3). The decision rule is pluggable via CORROBORATION_METHOD.

Method: zscore Classic Z-Score

Uses sample mean (μ) and sample standard deviation (σ) over the peer set P_n:

Caveat: μ and σ are non-robust. A single high-variance faulty peer in P_n can skew the baseline and mask real outliers. Prefer mad for small cells.

Method: mad Modified Z-Score via Median Absolute Deviation DEFAULT

Uses the median (m) and the Median Absolute Deviation (MAD) of the peer set, providing a robust estimator unaffected by up to 50% compromised peers:

The constant 0.6745 scales the denominator to be equivalent to standard Z-scores under normality. If the modified Z exceeds ANOMALY_ZSCORE_THRESHOLD (default 2.0), the node status changes to FAULTY.

Method: both AND-Hybrid

A node is marked FAULTY only if both the zscore and mad methods independently flag it at the same threshold. Maximizes precision at the cost of recall highly conservative.

4. ZeroMQ Hybrid Binary Framing

ZeroMQ subscriptions perform spatial topic filtering in clear-text using the H3 cell prefix. The payload is separated by a pipe character and compressed using the selected strategy:

This layout enables edge nodes to leverage ZeroMQ native subscription matching at the socket level without decompressing headers, drastically reducing CPU cycles for unrelated traffic.

5. Spatial Multi-Hop Relaying & Loop Prevention

To extend telemetry propagation beyond direct range (k-ring 1), nodes act as spatial relays by republishing messages under their local topic prefix while keeping coordinates and sender IDs unchanged. To prevent infinite forwarding loops and broadcast storms, the message schema introduces a path vector:

When a node n receives a relayed packet, the loop prevention rule is enforced:

6. Historical Duplicate Suppression

To avoid parsing or storing redundant messages propagated from multiple relay paths, the ingestion layer compares incoming timestamps against the node registry historical cache. A packet is discarded immediately at the ingest boundary if:
timestamp_incoming ≤ timestamp_registry_last_seen

Each node exposes a versioned JSON REST API at http://<node-host>:8080/api/v1/ (or HTTPS via Caddy on port 8441–844N). The API is read-only and reflects the local node's perspective of the network not a centralized view.

NodeEntry Schema

Prometheus Metrics

Example: GET /api/v1/nodes

[
  {
    "node_id": "node-a1b2",
    "type": "mock",
    "status": "ALIVE",
    "last_seen": 1748389082.5,
    "h3_cell": "871fb4670ffffff",
    "lat": 45.4642,
    "lon": 9.1895,
    "last_value": 24.5
  },
  {
    "node_id": "node-c3d4",
    "type": "mock",
    "status": "FAULTY",
    "last_seen": 1748389079.1,
    "h3_cell": "871fb4670ffffff",
    "lat": 45.4640,
    "lon": 9.1890,
    "last_value": 999.0
  }
]

Example: POST /api/v1/otaa/enroll

Send enrollment requests with a node identifier and public key Z85 string to configure peer authentication dynamically.

{
  "node_id": "node-c3d4",
  "public_key": "t7.d&F-gV=m]W5hR#c(P/y?Xq+eL*Z@wK8oNU2iv"
}

Example: GET /api/v1/timeseries

Query historical node telemetry readings archived in the local store. Filters: node_id, h3_cell, start_time, end_time, and limit.

[
  {
    "timestamp": 1748389082.5,
    "node_id": "node-a1b2",
    "h3_cell": "871fb4670ffffff",
    "lat": 45.4642,
    "lon": 9.1895,
    "value": 24.5,
    "status": "ALIVE"
  },
  {
    "timestamp": 1748389079.1,
    "node_id": "node-a1b2",
    "h3_cell": "871fb4670ffffff",
    "lat": 45.4642,
    "lon": 9.1895,
    "value": 24.1,
    "status": "ALIVE"
  }
]

ZeroMQ transmits structured telemetry using lightweight JSON framing. Each message is prefixed with the sender's h3_cell as the ZMQ topic, allowing subscribers to filter spatially with zero overhead.

PING Payload Schema

{
  "schema_version": 1,          // forward-compat version; listener accepts missing as v1
  "node_id": "node-a1b2",         // resolved from NODE_ID env or hostname
  "type": "mock",
  "timestamp": 1748389082.12,
  "status": "PING",
  "h3_cell": "871fb4670ffffff",      // also used as ZMQ topic prefix
  "relayed_by": ["node-x9y0"],  // list of node IDs that relayed this message
  "payload": {
    "value": 22.45,             // required: used in corroboration
    "lat": 45.4642,
    "lon": 9.1895
  }
}

Environment Configuration

Synapse v0.6.0 implements a layered security model: Caddy TLS at the HTTP/dashboard layer, CurveZMQ (Curve25519) with ZAP authentication at the ZMQ data channel layer, and Dynamic OTAA key enrollment.

Layer 1: Caddy TLS Reverse Proxy Sidecar

Every Synapse node in the Docker Compose and Helm deployments is paired with a Caddy proxy sidecar container within the same pod/service. Caddy automatically generates and rotates a self-signed certificate for local development, and can be configured for ACME/Let's Encrypt certificates in production.

Layer 2: CurveZMQ Elliptic Curve Transport Encryption

The ZMQ data channel can be encrypted and authenticated with CurveZMQ (Elliptic Curve Cryptography based on Curve25519). Generate keys and container-specific environment configurations automatically:

Layer 4: Dynamic Over-The-Air Activation (OTAA)

To bypass manual keypair pre-distribution, nodes run symmetrically authorized REST handshakes via POST /api/v1/otaa/enroll upon discovery. Public Curve keys are automatically registered on the target node, allowing handshake validation at the ZeroMQ ZAP authentication phase.

Layer 3: Rate Limiting via Token-Bucket

To mitigate DoS attempts by faulty or flooded nodes, the registry enforces a per-node token-bucket rate limiter:

• Burst Capacity: Maximum burst allowance (RATE_LIMIT_BURST, default 20 tokens).
• Replenish Rate: Token restoration per second (RATE_LIMIT_MSG_PER_SEC, default 100).
• Action: Packets depleting the bucket are dropped and counted in Synapse_rate_limited_total.

Synapse follows strict TDD practices. The full verification suite ruff auto-fix, pytest, mypy, Helm lint, and chaos smoke integration runs as a single make verify command.

Running the Full Verification Suite

Test Suite Overview

Linter & Type Checking

Synapse includes a native, container-overhead-free benchmark tool (tools/benchmark.py) that orchestrates up to 50 in-process simulated nodes via subprocess.Popen with static peer discovery. It measures mesh convergence time and aggregate message throughput under intense stress tests.

Key Performance Metrics (N=50 nodes)

Running Benchmarks

Synapse ships two complementary chaos tools: the Chaos Monkey (tools/chaos_monkey.py) for application-level failure injection, and Pumba for kernel-level network degradation via tc/netem.

Chaos Monkey Application-Level Failure Injection

Pumba Network-Level Chaos (Latency & Packet Loss)

Pumba intercepts the network stack of node containers and injects degradation via the Linux kernel's tc/netem subsystem. This validates the ZeroMQ P2P mesh's resilience to physical network impairment.

Option 1: Docker Compose (Local / Development)

The Makefile wraps all Docker Compose operations and auto-detects rootless Podman sockets. A 4-node mesh with Caddy sidecars starts in seconds:

Option 2: Kubernetes via Helm (Edge Production)

For production deployment on physical edge clusters (K3s, MicroK8s), Synapse provides an official Helm chart implementing a DaemonSet topology guaranteeing exactly one Synapse instance per physical node:

Helm Chart: Key Values (charts/synapse/values.yaml)

Synapse uses GitHub Actions for fully automated quality assurance and release publishing. No secrets are stored PyPI publishing uses OIDC Trusted Publishing.

Synapse is an evolving research project exploring edge ingestion paradigms.