CERT-In calls for testing AI agents for unauthorised tool use, privilege abuse

The Indian Computer Emergency Response Team (CERT-In) has issued new guidelines for securing artificial intelligence (AI) systems that can independently plan and execute tasks, utilize external tools, and delegate work to other AI entities. These frameworks emphasize the heightened security risks posed by agentic AI systems, which can perform multiple steps to achieve a goal, access information, employ software tools, execute code, and communicate with external systems.

Unlike conventional AI tools that respond to user commands, agentic AI systems can autonomously plan and carry out tasks, potentially accessing information, using software tools, executing code, or interacting with external systems without human intervention. CERT-In's guidelines call for rigorous testing of AI agents to ensure they do not exceed their authorized permissions, misuse tools, bypass approval mechanisms, or misuse credentials.

They also stress the importance of examining unsafe delegation of tasks and the effectiveness of emergency-stop mechanisms. The guidelines address concerns regarding AI agents' memory, including the potential for memory manipulation, information leakage between users, and retention or unauthorized reuse of sensitive data. Furthermore, the framework delineates the need for unique identities and clearly defined permissions for each AI agent when multiple agents collaborate, requiring authenticated and logged communications between agents to track requests and ensure authorization.

CERT-In also advises organizations to establish controls to detect and limit excessive task delegation, loops in task assignment, and maintain comprehensive records of agent interactions, task delegations, permissions, decisions, software tools used, and external actions. For high-risk systems, the guidelines recommend the ability to suspend individual agents or the entire workflow and revoke access credentials.

Additionally, organizations are urged to maintain inventories of components in production AI systems, documenting gaps in third-party model disclosures and their implications on audits. Lastly, CERT-In emphasizes that auditors utilizing AI must verify the accuracy of AI-generated outputs independently, as AI-generated summaries cannot substitute for audit evidence or professional judgment.