Scale design

The prototype runs as keyed stages: ingest per sensor, fusion per transmitter MAC, detection per client / AP / channel. The keys and the state per key carry over from today's batch run to a partitioned stream. The detectors do not carry over unchanged: today they read the whole capture (end-of-capture checks, medians over everything); as stream jobs they need time windows, watermarks for late frames (sensor clocks are up to 1.2 s off) and state that expires. Incidents that span many APs run as one small global stage on per-key summaries.

Partition keys

sensor(ingest, clock alignment)
transmitter MAC(fusion, de-duplication)
client / AP / channel(detectors, incidents)

One processmeasured

≥ 12k frames/s

full pipeline on the server, measured before de-dup was skipped for one-sensor-per-channel layouts

Per sensormeasured

78 frames/s

Tesla's capture: 82 % beacons, 0.24 % data. A busy factory channel: 1,000–5,000 frames/s

Busy sensors per processderived

≈ 6–12

at 1,000–2,000 frames/s per sensor. Beacon summaries do not help here: the load is data and ACK headers

2,000-sensor sitederived

≈ 170–330 cores

keyed by transmitter MAC. Next factor: counters per client on the sensor instead of every header

Measured: the prototype on Tesla's 8 captures (30 minutes, ≈ 1.1 M frames) on one laptop. Derived: the same rate divided across sensors, with beacons summarised on the sensor instead of sent one by one.

Pipeline

6 keyed stages, each shards independently

#	Stage	Runs	Key	What it does	Tech
1	Edge ingest	on every sensor	per sensor	tshark/libpcap dissects 802.11 + 802.1X headers on the sensor and emits compact header events (~60 bytes instead of full frames). Raw pcap stays in a ring buffer on the sensor, pulled only on demand.	Rust or Go agent, protobuf, mTLS
2	Event bus	site cluster	partition = transmitter MAC	All sensors publish to one topic per site. Partitioning by transmitter MAC puts every copy of the same frame (heard by several sensors) into the same partition - fusion needs no cross-node traffic.	Kafka / Redpanda
3	Fusion	stream workers	per TA, 10 ms window	Clock alignment from shared beacons (continuously re-estimated), de-duplication into one air event with RSSI per sensor → location estimate. Exactly the code that runs in this prototype, just windowed.	Flink or Python (Bytewax/Faust) workers
4	Sequence detectors	stateful stream jobs	per client / BSSID / channel	Per-client state machines (auth → assoc → EAP → 4-way → connected), AP beacon watchdogs confirmed by ≥2 sensors, channel load windows, security rules. State is small per key, so it shards linearly.	Flink keyed state, RocksDB
5	Store & serve	central	time-partitioned	Events and findings in a columnar time-series store for drill-down and trends; findings + device inventory in Postgres; dashboard and API as today.	ClickHouse / TimescaleDB, Postgres, FastAPI, Next.js
6	Alert & act	central	per area / device class	Routing by device class and location: torque tool on line 3 fails its handshake → line lead's phone within seconds; canteen congestion → IT ticket. AI explanation attached. Deduplicated per root cause, not per frame.	Alertmanager, Teams/SMS, ServiceNow

Sensor lifecycle

Thousands of sensors are managed as a fleet.

Registry: every sensor has an identity (certificate), location, channel plan and firmware version in one inventory.
Zero-touch onboarding: sensor boots, enrols via mTLS, pulls its config (channels, area, filters) as code from Git.
Signed OTA updates in waves: canary (1 %) → one hall → site → all sites, automatic rollback if heartbeat, event rate or clock sync degrade.
Self-monitoring: the same fusion that finds bad APs finds bad sensors - a sensor that stops hearing what its neighbours hear is flagged (see "sensor blind spot" findings).

Design choices

What keeps the path to production short.

Edge first: headers only, aggregated on the sensor → 10-100x less backhaul, no payload leaves the sensor (privacy, works council).
Key by transmitter: duplicates from overlapping sensors (same channel) meet in one partition - fusion is local, horizontally scalable. With one sensor per channel there are no duplicates and fusion only aligns the clocks.
Findings, not frames: alerts are per root cause (AP down, handshake broken) and per device class: 3 alerts for the line lead instead of 30,000 frames.
Same keys batch and stream: the partitioning and the per-key state carry over; each detector becomes a windowed stream job with a watermark.