When code is cheap, trust is not: Inside the AI-era engineering roundtable

The AI era has changed software engineering, making code creation swift, but trust in the output is the new challenge for leaders. At a roundtable discussion, engineers from various sectors agreed that speed is no longer the main hurdle; instead, managing AI is the critical issue. The conversation centered on scaling with AI without sacrificing reliability, security, or control.

The main bottleneck has shifted from code creation to downstream processes. As AI generates large volumes of code, questions arise about who reviews it, tests it, and monitors dozens of deployments daily. Teams now spend more time on design, context, and governance compared to writing components. Incomplete repository knowledge, inadequate documentation, and templates hinder agents from functioning optimally.

Participants converged on a progressive approach, distinguishing between reversible and irreversible actions. For reversible tasks such as fetching data or generating reports, agents can operate with minimal friction. However, irreversible actions like deleting data or running database updates require human review. A 'four eyes' model, where AI handles reading and analysis, deterministic rules provide yes or no outcomes, and humans grant final approval, was one such example.

Another approach involved risk scoring based on severity and occurrence to prioritize human review for medium and high criticality items. Specialization also emerged as a viable solution, enabling engineers to utilize agents within their domain while maintaining production ownership with people capable of debugging at odd hours.

Trust in AI systems will develop gradually, much like it did with online payments, and the decision line will continue to evolve. Accountability remains firmly in human hands, as AI cannot delegate responsibility. Leaders stressed that they are accountable for the actions of agents, given AI's amplification of both good and bad decisions.

Concerns about skill erosion were raised, particularly in healthcare, where losing expertise to judge AI output is a significant issue. To address this, some suggested deliberately deciding where AI drafts and where humans take charge.

Data control is a design decision, especially crucial for telecom and healthcare leaders subject to stringent regulations and penalties. Governing data from the first stage involves determining what data is ingested, used for training versus inference, and how it is masked or de-identified. Concerns about intellectual property and bias also led to a call for rigorously vetting AI tools, similar to the process for hiring new employees.

Observability is critical in the agentic world, as teams need to trace agent inputs, decisions, and handoffs to understand why an agent acted in a particular manner and rectify any issues. Companies have already started implementing solutions like a change ledger, lineage, system graphs, and debugging agents that propose ranked hypotheses. The consensus is that observability must be integrated closely with development, as it can no longer be treated separately.

In summary, while there is no definitive playbook, the conversation highlighted the need for guardrails and clear guidelines to ensure AI is used effectively. Developers should be given the freedom to move swiftly, but robust guardrails must be established first. Human oversight is essential for irreversible and high-impact decisions, while automation should be expanded as trust is built. The AI era demands that teams keep up with the pace of change, as competitors will not wait.