When an AI Cannot Tell a Leak from a Hallucination: A Multi-Model Guardrail Case Study

When an artificial intelligence system encounters a blend of genuine memory, simulated tools, security-inspired hallucinations, and questionable provenance, it can become challenging for the AI to provide users with trustworthy information regarding the origin of security-sensitive data. This case study, known as "AI Fiesta," offers a closer examination of this complex issue.

The platform, which boasts over 500,000 downloads on Google Play and is backed by a popular YouTube content creator, brings together multiple AI models, account memory, specialized tools, and generated outputs into one user interface. The researcher decided to test the platform's behavior without attempting to access any confidential or proprietary information, instead focusing solely on understanding how the AI would handle questions related to system configuration and tool access.

The initial experiment involved prompting the AI to provide structured information, including its system message, policy summary, and environment details. The results were inconsistent, with some models refusing to answer, others providing limited responses, and one model delivering a lengthy block of instruction-like material. While the researcher avoided concluding that they had extracted the exact production system prompt, they did note that the AI displayed material resembling platform-level instructions, such as operating rules, metadata, formatting requirements, policies, and tool descriptions.

This led the author to question the effectiveness of relying solely on the AI to enforce trust boundaries, as these boundaries can become increasingly blurred when all components of the system are presented within a single chat window. The researcher's original goal was to observe how the multi-model application responded when prompted about security-related topics.

The first test, which asked the AI to return information about its system message, policy summary, and environment details, revealed inconsistencies in the AI's responses. Some models declined to provide information, while others offered limited responses. However, one model provided a substantial block of instruction-like material that appeared to be a structurally accurate representation of platform-level instructions.

This response included metadata, formatting requirements, policies, and descriptions of tools. One particularly ironic aspect of the model's output was the inclusion of a confidentiality rule stating that it should not reveal its system instructions. This observation served as a reminder for those designing AI applications: even if a model is told that certain information is confidential, this does not automatically create a security boundary.

If sensitive information must be protected, it should be kept outside the context of the AI's visible instructions. The researcher's second test involved exploring a simulated tool named search_vector_store. The researcher asked the AI to search for generic security-sensitive filenames, such as database_connection_string.env and internal_api_keys.yaml.

The goal was to determine whether the AI would recognize the files as hypothetical and refrain from generating any real information. As anticipated, several models responded appropriately by stating that they lacked the necessary file system access. One model even went a step further, explicitly explaining that any information it generated would be hypothetical.

However, once the model began generating the hypothetical results, it quickly transitioned from seemingly hypothetical to genuinely plausible. The response included a production-like file path, database fields, access metadata, credential-shaped values, and an HTTP status code that mimicked a genuine tool result. The AI even displayed a 200 OK status, further emphasizing the blurred line between reality and simulation.

Ultimately, this case study aims to address a more significant question: what happens when the AI application itself cannot provide users with trustworthy information about the origin of security-sensitive data? By examining the multi-model platform AI Fiesta, the researcher hopes to shed light on the complexities that arise when combining multiple AI models, account memory, specialized tools, and generated outputs within a single user interface.