Prompts Aren't Real

Engineers working with agents that perform tasks on behalf of consumers face significant challenges, despite the impressive capabilities of large language models (LLMs). While LLMs can generate coherent responses to certain prompts, they often fail to reliably follow instructions, provide truthful information, or execute tasks consistently.

This is particularly problematic when attempting to create reliable agents for real-world use. Engineers attempt to structure LLM outputs to ensure proper formatting, but even these efforts can be unreliable. Models may produce unexpected, nonsensical responses or fail to adhere to specified constraints. To address these issues, engineers try to constrain model behavior through extensive testing and evaluation, often using metrics like "pass power k" to measure reliability.

However, prompts themselves are not the primary concern in creating effective agents. Instead, engineers should focus on designing agents with clear, consistent rules and behaviors rather than relying on prompts. The current approach of having a domain knowledge expert write prompts for each agent is flawed, as changes to prompts can disrupt previously trained behaviors.

A better approach is to design agents with built-in constraints and evaluate them continuously to ensure they remain reliable and consistent in performance.