Open-source tool: Cross-database universal "field-level" data lineage analysis and visualization(Part 3 of 3)

This is Part 3 of a three-part article detailing the development of an open-source tool for cross-database universal field-level data lineage analysis and visualization. In the previous parts, the parsing and saving of script field-level data lineage information was accomplished.

The next step in this project involves creating a web application to visualize the data lineage. The web application, named web_data_lineage_show, is built using the Django framework in Python. The project name is web_data_lineage_show.

Upon launching the web application, users are greeted with a graphical display of script content. This includes a list of parsed scripts, the option to select a script for generating a data lineage graph, selecting a table to view field logic, and connecting lines that represent association methods and conditions.

One of the key features of this web application is the field lineage tracing function. Users can select any column, click the Trace Upward button, and view the lineage path of the column. They can also view the columns of the data lineage node, which provides a comprehensive understanding of the data lineage.

The web application also supports cross-script file field lineage tracing. Users can select "Cross-file traceability," choose any source table from the left column, and select any field. By clicking the Trace Upward button, the system traces the field to another script file. If the traced script is selected, the system automatically continues to trace based on the fields traced in the previous layer, enabling users to achieve full-chain tracing for a specific field.

Finally, the article provides a graphical demonstration of complex scripts. After thorough testing, the parsed field-level lineage is found to be completely accurate. The source code for the web project web_data_lineage_show is available in the open-source community link: https://github.com/zgl-20053779/ZGLanguage/tree/main/project/Field-level%20data%20lineage%20analysis%20and%20visualization.