D-MO (Data Micro-Optimizer)

Software development and data analysis often require repetitive file preparation and cleaning tasks, especially when dealing with rigid layouts, misaligned rows, and changing column names. This friction can be resolved through a robust, client-side ETL (Extract, Transform, Load) tool, which is the subject of this report: D-MO (Data Micro-Optimizer).

D-MO was developed with the primary goal of maintaining strict data security, particularly when handling sensitive banking or financial reports. By processing information within the local browser memory, the tool eliminates the risk of transferring data to external platforms, thus ensuring zero server overhead and enhanced privacy.

The data processing pipeline in D-MO operates sequentially through a decoupled flow, allowing users to transform complex files into production-ready datasets with a single click. The Pipeline includes the following components:

1. DropZone: Validates file extension and size

2. File Parser: Detects delimiters and dynamic headers

3. ETL Engine: Implements business rules, column alias mapping, and custom filters

D-MO's core architecture is built using Next.js 14 (App Router) and TypeScript, dividing its internal logic into three main components:

1. Interface and Coordination (page.tsx): Acts as the data lifecycle orchestrator. It captures uploaded files through a drag-and-drop interface, invokes parsing utilities, feeds the transformation engine with current state, and dynamically updates UI metrics.

2. Parsing and Serialization Layer (file-parser.ts): Utilizes the xlsx (SheetJS) library to analyze workbooks, normalizing raw data and dynamically resolving header rows in complex Excel files. It also implements papaparse for fast streaming memory reads and automatic delimiter detection in CSV files.

3. Flexible Rule Engine (etl-engine.ts): The brain of D-MO, which employs a robust string normalization matching system. This enables the engine to identify dynamic headers using a flexible catalog of alias mappings, processing column layouts with subtle naming differences without breaking the transformation flow.

The system's technical components are complemented by real-time control panels for comprehensive traceability:

1. LogConsole: Provides real-time visual audit trails of each validation and transformation executed by the engine, categorized by states with a clean design.

2. StatsBar: Offers immediate analytical visualization of processed vs. exported rows, resulting column count, and precise pipeline execution time.

In comparison to traditional server-based processing approaches, D-MO offers several advantages, particularly in terms of privacy and cost-effectiveness. While traditional server-based processing is subject to data transfer latency and infrastructure costs, D-MO processes data directly in memory, ensuring instant performance and zero server overhead.

In conclusion, D-MO showcases that heavy ETL data pipelines do not necessarily require delegation to complex backend architectures. By transferring the logic directly to the client using TypeScript, the tool not only saves on infrastructure costs but also addresses critical issues like regulatory compliance and data information protection.