The Dawn of the Data Intelligence Engineer

In the evolving landscape of data teams, a new role has emerged as AI continues to reshape the way information workers approach decision-making. This role can be aptly named the Data Intelligence Engineer.

Nick Ursa, the lead data engineer and co-founder of MotherDuck, emphasizes that while AI has reduced the backlog in his career, it has also transformed his responsibilities. Instead of focusing on ensuring the accuracy of the numbers he computes, his job now involves verifying the correctness of the insights generated by Large Language Models (LLMs) or agents. This shift signifies a departure from the traditional ticket-based approach, where data teams worked to resolve issues one by one.

Data teams, at their core, are tasked with empowering businesses to make informed decisions. The speed and accuracy of these decisions hinge on the quality and completeness of the data available. However, the journey from a question to an answer has often been hindered by a series of handoffs. A marketing leader might notice a spike in churn, file a ticket with the data team, and wait for an analyst to investigate, only to discover missing data sources and initiate a ticket for the data engineer.

This chain of communication results in delays, context loss, and a gradual erosion of the effectiveness of the answers provided.

The introduction of AI technologies, such as LLMs and agents, has the potential to streamline this process. These tools can now understand business logic, generate accurate SQL queries, and build rich visualizations with minimal supervision. By allowing business stakeholders to interact with data using prompts, rather than relying on traditional BI tools, the Data Intelligence Engineer facilitates self-service analytics.

This empowers decision-makers to ask follow-up questions, explore different scenarios, and make confident decisions faster, even without deep technical expertise.

However, despite the automation enabled by AI, the role of humans in the data ecosystem remains crucial. Data Intelligence Engineers must ensure that the data model is clean, well-documented, and consistently applied throughout the organization. They serve as accountability functions, validating the accuracy of the insights generated by AI and ensuring that stakeholders without a "Data" title can effectively leverage the power of data to solve problems.

In this new paradigm, humans are still indispensable in maintaining the integrity and trustworthiness of the data-driven decision-making process.