How We Built an IoT Platform That Handles 30 Million Concurrent Connections — With a Team of 10

DGIOT is an open-source industrial IoT platform that can handle 30 million concurrent connections with just 10 team members. The platform was developed to address the challenges faced by Daqing Oil Field, China's largest oil producer, which had 928 industrial gateways from various vendors and 114,809 sensor points speaking 15 different protocols. The existing system was a patchwork of vendor-specific tools, causing operators to log into 8 different systems to check on the status of a single pump.

DGIOT is built as a universal translator for industrial protocols, using an Erlang/OTP-based architecture that treats industrial protocols as state machines. This allows the platform to handle hundreds of protocols concurrently with almost no overhead. The platform consists of four layers: DATA, LOGIC, ACTION, and EDGE.

The DATA layer includes the Ingestion Parse Server, which handles device metadata, user authentication, and tenant isolation. Data is then stored in TDengine, a time-series database with 10:1 compression, and processed by EMQX for MQTT message routing at 1M+ messages per second. In-memory caching is provided by Mnesia/ETS for hot data.

The LOGIC layer, or Ontology Engine, is the platform's secret weapon. It uses a 252-entity OWL ontology to model industrial equipment, such as pumps, bearings, and components. Rules are compiled to Erlang pattern matches, enabling the platform to not only display data but also understand relationships and generate work orders based on alerts and failure modes.

The ACTION layer consists of Shadow State Machines, which mirror the real-world state of physical devices. These in-memory processes handle network interruptions and maintain the last known state when a device is unreachable. The dashboard only shows a device as disconnected if it has been unreachable for 2+ minutes.

The EDGE layer is implemented using iotStudio, a Python + Vue 3 edge agent that runs on ARM gateways. This agent handles protocol adaptation, stream computation, offline autonomy, and OTA updates.

Erlang was chosen for its built-in concurrency model, preemptive scheduling, hot code reloading, and let-it-crash philosophy. These features allow the platform to handle massive concurrency, high availability, and soft real-time requirements. DGIOT is open-source and licensed under Apache 2.0, allowing the industrial IoT community to contribute and benefit from each other's work.