Netflix Open-Sources Agentic Workflow for Causal Inference

Netflix has released an open-source agentic workflow designed to streamline Observational Causal Inference (OCI) analysis. This workflow automates error-prone or repetitive tasks, such as sensitivity analysis and iteration tracking, while leaving higher-level tasks like framing questions and evaluating results to human users. The Netflix team evaluated the workflow on the Atlantic Causal Inference Conference (ACIC) competition dataset, finding it competitive against benchmark systems.

The agent workflow relies on an actor-critic loop, where the actor estimates causality, writes a report, and suggests next steps. The critic reviews the output, rates its satisfaction, and recommends changes. The actor agent uses a templated Jupyter notebook and analysis plan provided by the human user to execute the analysis. The critic agent checks the results, flags issues such as potential bias or failed tests, and suggests improvements.

Netflix demonstrated the workflow's capabilities by estimating the impact of new entertainment types on user retention. When compared to a baseline approach using a Claude model, the oci-agent workflow produced an estimated effect that was 25% of the baseline. The critic agent identified several issues, including early adopter bias and a failed placebo test. The workflow automates multiple analyses with tweaked parameters, providing process audits and human oversight throughout the workflow.

The open-source oci-agent source code is available on GitHub, enabling others to learn from and critique the workflow. Experts can verify the results and ensure the transparency of the process. This approach highlights Netflix's focus on improving AI workflows rather than just better models to enhance the overall AI ecosystem.