From Informatica to Apache SeaTunnel: A Financial-Grade ETL Migration in Practice

The report discusses a successful migration of a financial technology company's core ETL system from Informatica PowerCenter to a domestic ETL platform called Apache SeaTunnel. This migration, which involved over 200 workflows processing terabytes of data daily, took five months and was completed without any data incidents.

Informatica, a long-standing leader in enterprise data integration, was chosen as the ETL solution due to its visual development environment, reliable scheduling engine, and comprehensive metadata management. However, the company faced challenges such as high licensing costs, a closed technology stack, and slow response to customization needs.

After evaluating three mainstream domestic ETL tools - Kettle, DataX, and SeaTunnel - the company selected Apache SeaTunnel for three main reasons: its support for both Spark and Flink engines, a plugin-based architecture that facilitates easy extension for custom data sources, and an active Chinese-language community for quick issue resolution.

The migration strategy employed a hybrid approach combining phased migration with parallel-run validation. The Informatica workflows were decoupled into three independent modules for extraction, transformation, and loading. Complex transformation logic was rebuilt using Spark SQL, and scheduling dependencies were orchestrated with Apache DolphinScheduler.

The report details the key steps in the migration process, including metadata migration using a metadata parsing tool, refactoring complex transformations, and performance tuning in practice. They faced challenges such as an ETL job containing 20 joins that ran into an OOM error on SeaTunnel, which was resolved through optimizations like analysis of the execution plan, enabling dynamic partition pruning, adjusting the broadcast threshold, and forcing broadcast joins for dimension tables.

The company ensured data consistency through a three-level validation framework, including record-level validation using CRC32 fingerprints, business metric comparison, and user acceptance testing. The migration was carried out incrementally by business line, with particular focus on monitoring data latency, resource utilization, and error logs.

The lessons learned from this migration emphasize the importance of team training and optimizing transformation logic during the migration process. By doing so, the company was able to successfully migrate its core ETL system to a domestic solution that offered cost savings, flexibility for customization, and support for future real-time data warehouse strategies.