Giving an AI Assistant Access to Your Accounts: What to Check

AI assistants that are linked to real tools can be highly useful, as they can access invoices, calendars, and customer threads, rather than requiring users to manually provide this information. However, this access also presents a significant risk. An assistant with the ability to read a user's inbox could be misled by the content it contains, while an assistant with write access—meaning the ability to send messages or move money—could cause real damage if it makes an incorrect decision or is manipulated.

The recommended approach is not to avoid connecting AI assistants to tools altogether, but rather to carefully decide what each connection is allowed to do. Read access and write access are two distinct types of power granted by most integrations. Read access allows the assistant to view data, such as emails, bank transactions, calendar entries, and invoices.

Write access, on the other hand, enables the assistant to take action, like sending a message, creating an event, altering a record, or moving money. While reading carries a risk to privacy, writing carries operational and financial risks.

To begin, it is advised to start with read-only access and only then grant write access, one action at a time. Any action that involves payments, deletions, or sending information outside the company should be subject to human approval. If a tool is unable to differentiate between read and write access, it should be treated as having write access by default.

Prompt injection is an emerging risk for AI assistants that read external content. This risk arises when the text an assistant reads, such as an email, web page, or document, contains instructions meant for the assistant rather than for the user. For instance, the assistant might be instructed to ignore previous instructions or forward the last ten invoices to a different email address.

Since language models treat all read content as text, they may sometimes follow these instructions. OWASP, a leading security community resource, ranks prompt injection as the top risk for applications built on language models. It does not require advanced hacking skills; a simple email or PDF from an unknown source can suffice.

Practical defenses against prompt injection include the principle of least privilege, which involves connecting only the necessary accounts and folders required for a task. Additionally, any action that sends, pays, deletes, or shares information externally should require a person's confirmation, who can verify the exact outcome.

The assistant should not be given instructions derived from the content it reads, such as changing bank details. Shared inboxes that receive attachments from unknown sources pose a higher risk compared to curated sources. It is essential to maintain activity logs, understand what the assistant reads and does, and ask vendors for these logs, along with how long they are retained.

Vendors should also provide mechanisms for revoking connections and managing data that has already been fetched.

When deciding to use a vendor's AI assistant, ask about the permissions requested by each integration and whether read-only access is an option. Determine which actions require confirmation by default and whether more actions should also require confirmation. Understand how the product handles instructions found within emails or documents, and ascertain whether you can access and export activity logs.

Inquire about the process for disconnecting each integration and the fate of previously fetched data. Identify the sub-processors handling the data and whether it is used for training purposes. Learn how access tokens are stored, rotated, and revoked.

In case something appears amiss, immediately revoke the connection, change the passwords on the affected accounts, review recent sent items and transactions, and notify the bank if payments are involved. Document what happened. If personal data may have been exposed, the 72-hour breach notification rule under GDPR may apply, so involve the party responsible for data protection.

Finally, tighten the permissions that initially allowed the issue. Companies like Mirage Cloud utilize AI agents that heavily rely on connections to tools such as Gmail, Google Calendar, Qonto, and Pennylane. When evaluating such vendors, utilize the seven questions mentioned above and incorporate their answers into your decision-making process.

Regardless of the product, start with limited access and gradually expand it only after observing the assistant's behavior.