Testing and Verification in AI SDLC, Technical Debt Management and Guardrails | Weekly Reflections #1

This week, I focused on two closely related tasks: developing an end-to-end test setup for a project and implementing a technical debt management process for my team. In the past, manual testing was the norm, but I aimed to replace it with automated tests that could be run by an AI agent before a risky release. To create this setup, I used Pytest and Playwright, a browser automation library, which work well for testing contained services like backend APIs and admin panels, such as Django Admin.

By running the tests from my local machine, I ensured that AI agents could execute the suite independently and validate their own changes. This approach eliminated the need for manual tests for this project's most crucial feature. When a new risky release approached, we ran the suite, and if additional coverage was necessary, we added test cases.

The authentication process for the service was initially limited to inside the cluster, but I added an API key authenticator specifically for local and QA environments, preserving production security. This process took less than a day and proved to be highly beneficial. While working on both tasks simultaneously, I noticed that the lack of harness and guardrails was becoming a significant source of technical debt.

Engineering principles for preventing and fixing debt can still be applied to AI, but we now delegate the implementation to agents rather than writing it ourselves. This shift enables faster development, but we must still prioritize the time invested. I believe that having a robust technical debt management process is easier now than ever, thanks to proper skills and agents integrated with the ticket system.

With guardrails for software architecture, design, security, and performance, we can avoid or mitigate new debt while still benefiting from the increased development speed provided by coding agents. To maximize the efficiency of coding agents, we need to focus on generating high-quality software and reducing the time spent on other aspects of the software development life cycle (SDLC), such as review, testing, and deployment.

These have become new bottlenecks, and guardrails play a crucial role in reducing them. I also reflected on the definition of technical debt, which refers to the hidden consequences of past decisions that hinder software change. Customers notice features and defects, but developers are the only ones aware of the debt, and it remains unnoticed until the system requires modification.

To measure the impact of technical debt, I estimated the hours lost each time a debt item becomes problematic, using a fixed scale (e.g., 15 minutes, 1 hour, 4 hours, 1 day, 2+ days). By calculating the interest burden per time period and dividing it by the repayment effort (fixing time), I determined the return on investment (ROI) in weeks.

For example, a 3-hour debt item with a frequency of four times per week would result in a 12-hour weekly expense, making a 40-hour fix worthwhile within just over three weeks. Implementing a tech debt budget for each development cycle or sprint is essential, ensuring that a 0% debt rate is not maintained as a permanent state. Technical debt that becomes more expensive to fix over time, such as an API with debt, requires careful consideration, especially when data changes are involved.

Even with agentic coding, data changes and migrations remain challenging. Lastly, I pondered how to build performance and race condition checks that an agent can run locally, similar to how it executes the end-to-end suite. Additionally, I questioned how to ensure data changes and migrations are safer when agents perform more changes.

To build guardrails effectively, I must address these concerns and explore ways to mitigate the hidden costs of technical debt.