You Won’t Know How Much to Delegate to AI Until You Use It Extensively

Determining the extent of AI delegation and necessary human review depends on the task's complexity and the AI model used. Tasks with a limited range of possible solutions and easy reversal are simpler to offload to AI. Conversely, tasks with multiple viable approaches, significant product-policy implications, or long-term maintenance require human decision-making alongside AI.

This intuition-based process involves using models from various vendors repeatedly to develop a feel for each one's reliability in your work. AI and review depth are not quantifiable metrics, but through experience, you can gauge how much to trust each model for your specific tasks.

When faced with a task, consider the number of plausible solutions first. For instance, changing the color of a UI element from white to gray has minimal ambiguity, making it easy to delegate to AI. While checking the affected screen and the change's diff is advisable, a deep product-level review is typically unnecessary. If several implementation options are plausible, AI can generate these options and trade-offs, allowing you to compare them against existing design and product future needs.

A task with countless possible solutions does not immediately warrant AI implementation. Instead, AI serves as a thinking partner to help articulate your thinking, compare alternatives, and identify potential oversight.

For instance, when building KIBAKO, an online service for creating board game prototypes, many product decisions required AI's assistance. Determining the scope of improvement notes, where and how users should access them, which participants should view different notes, and how to handle multiple play rooms belonging to the same project were all challenging.

Though the final design stored improvement notes across the project, spanning multiple play rooms and the production room, this decision was not handed over to AI. Instead, AI helped explore various options, and the decision was ultimately made by the human using AI as a thinking partner.

Another factor to consider is the reversibility of changes. Database schema changes, particularly those impacting user-entered data, can be difficult or impossible to reverse. For these types of tasks, review depth goes beyond mere file execution checks. You must assess:

1. Whether the migration can be rolled back before user data exists.

2. The data migration required after user data enters the system.

3. The degree of rollback possible if something goes wrong.

4. The constraints this imposes for future features and maintenance.

The greater the potential impact on the product's future direction or maintainability, the more thorough the review needs to be.

To manage costs effectively, start by finding an affordable default model that offers the best efficiency for your typical work. In this case, the author uses the Codex's $100 Pro plan, setting the default to around 5.6 Luna High. For tasks with multiple solutions, database schema design reviews, or significant product impact, switch to more powerful models like Terra or Sol.

Adjust effort levels and create the Markdown specification together with AI, starting with the uncertainty and impact of the task in mind before choosing the model and effort level. During implementation, review critical code portions with AI, asking questions about data scope, trade-offs, and potential alternatives. This iterative process allows you to maintain sufficient review depth while optimizing for cost-effectiveness.