Processing Heterogeneous IoT Logs with Fluent Bit and Elasticsearch
A lightweight two-stage approach for parsing, preprocessing and storing semi-structured home automation data The full configuration example is available on GitHub. Home automation systems tend to generate a surprisingly heterogeneous collection of logs and measurements. In this project, most data originates from Domoticz, but additional information is produced by ESP8266/D1 devices, alarm…
Home automation systems generate diverse logs and measurements, often in various formats. To process this heterogeneous data, a two-stage approach using Fluent Bit and Elasticsearch is employed. Fluent Bit performs initial parsing and preprocessing, while Elasticsearch handles final JSON decoding and document transformation.
The configuration begins with several parsers for different log formats, such as Domoticz and ESP8266/D1 devices. Fluent Bit extracts timestamps, metadata, and decodes JSON where available, forwarding the processed records to Elasticsearch. This separation keeps Fluent Bit configuration simple and isolates Elasticsearch for final document transformation.
Fluent Bit tail inputs monitor individual log files, assigning each source a unique tag for easy record tracking. Each source receives its own persistent database. The Domoticz records are processed with specific parsers, while other records are covered by additional parsers, including a fallback for pure JSON.
Preprocessed data is sent to Elasticsearch via an output filter. The resulting records are indexed in a common index, such as "dom_v3." An ingest pipeline in Elasticsearch decodes any remaining JSON in the log field and adds the decoded fields to the document root. This allows for further analysis and visualization using tools like Kibana and Grafana.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
