Python Automation Cookbook, Part 2: Using AI to make decisions in your automation pipelines

Part 2 of the Python Automation Cookbook delves into the complex task of incorporating AI into decision-making processes within automation pipelines. While Part 1 introduced a collection of 25 essential automation scripts, this section focuses on the more intricate challenge of managing decision points within these pipelines. Traditionally, decision-making in such pipelines was achieved through if/elif statements, which work well when the rules are straightforward and unchanging.

However, as the complexity of decision-making increases—especially in cases that require judgment, such as parsing ambiguous outputs, classifying content, or handling edge cases—the if/elif approach becomes less flexible and more prone to errors.

The solution proposed is to leverage the Python scripts from the Cookbook for the more deterministic tasks (like HTTP calls, file I/O, scheduling, and retries) and to integrate a small AI prompt for the more subjective decision-making tasks. This approach ensures that each part of the pipeline is optimized for its specific strengths: deterministic scripts handle the predictable and structured tasks, while AI prompts are employed for the more nuanced, judgment-based decisions.

This combination allows the pipeline to handle a wide range of inputs and scenarios more efficiently and accurately.

The structure of an AI-in-the-loop pipeline is demonstrated through a three-script example before and after the addition of an AI decision layer. Initially, the pipeline consists of pure script chaining, where each step follows the other sequentially. This basic setup works well for straightforward processes, such as downloading files, converting file formats, and archiving them.

However, when faced with files that have malformed data, unexpected schema changes, or are empty or error responses, the script chain can fail silently or stop the entire pipeline abruptly.

The introduction of an AI triage step resolves these issues by classifying files into categories like PROCESS (well-formed and ready for conversion), REVIEW (needs human review due to unusual structure), or SKIP (empty, error response, or duplicate). This classification is based on a concise AI prompt that instructs the AI to make a simple, single-word decision (PROCESS, REVIEW, or SKIP) without providing any explanation or additional context.

The prompt is designed to be straightforward and unambiguous to ensure that the classification is accurate and can be reliably parsed by downstream scripts.

The AI triage step is implemented using a script named `prompt_runner.py`, which is a lightweight wrapper around the AI API. This script handles initialization, token budgeting, error handling, output parsing, and retry logic, making it easier to incorporate AI decisions into the pipeline without dealing with the complexities of the underlying API directly.

The usage of `prompt_runner.py` is straightforward, with options for both single-file processing and batch processing of multiple files. Batch processing allows for parallel execution of the AI calls, significantly speeding up the classification of large numbers of files.

To decide whether to use AI prompts or write custom code for decision-making, the author suggests a set of guidelines. Code should be used for decisions that are fully deterministic, inputs are structured and predictable, and an auditable record of decisions is needed. On the other hand, AI prompts are more suitable for situations where the input is unstructured, the decision requires handling variability and edge cases, and manual intervention would be impractical.

The author emphasizes that while code handles the predictable and straightforward path, AI prompts are designed to handle the unpredictable and complex aspects of the pipeline.

A practical application of this pattern is illustrated through an example of automated PR review triage, where the goal is to distinguish between PRs that require deep review and those that can be auto-approved. This application showcases how AI can be effectively used to handle the long tail of decision-making tasks that are not easily automated through deterministic scripts.

By using AI to handle the more complex and nuanced decisions, the pipeline can be made more robust, efficient, and capable of handling a wider variety of inputs with greater accuracy.