Building AI Agents in Production: What MCP Rejections, .env Leaks, and 70-Line Loops Taught Me
Originally published on tamiz.pro . Most AI agent architectures fail not because the model is wrong, but because the engineering around it is naive. I learned this the hard way when my first production agent loop crashed the billing service after a Model Context Protocol rejection cascaded into an environment variable leak and a 70-line execution trace that never terminated. This article isn't…
Title: Building AI Agents in Production: Lessons from MCP Rejections, .env Leaks, and Endless Loops
Most AI agent architectures fail not because the underlying model is flawed, but due to naive engineering practices. My first production agent loop experienced a catastrophic crash when a Model Context Protocol (MCP) rejection cascaded into an environment variable leak, resulting in a non-terminating 70-line execution trace. This article focuses on the foundational plumbing that distinguishes toy demos from robust, production-ready AI systems.
The Death of the Single-Hop Agent
Designing production AI agents requires moving beyond the simplistic notion of a single LLM call followed by a tool invocation. Real-world systems demand stateful, multi-step reasoning with explicit failure boundaries. The MCP protocol was conceived to address interoperability challenges, yet in practice, MCP rejections disclose deeper architectural weaknesses in agent reliability.
MCP rejections are not mere tool call failures; they signify the agent has encountered an unsupported context. Common scenarios leading to MCP rejections include:
- Unsupported tool invocation order by the MCP server
- Context windows exceeding the MCP server's capacity
- Expired authentication tokens during a session
- Hit rate limits without proper backoff mechanisms
Treat MCP rejections as first-class events rather than exceptions. Implement a rejection handler to convert MCP failures into recoverable state transitions, preventing silent crashes.
```typescript
interface MCPRejectionHandler {
handle(rejection: MCPError): Promise<AgentStateTransition>;
}
class ResilientAgentController implements MCPRejectionHandler {
async handle(rejection: MCPError): Promise<AgentStateTransition> {
switch (rejection.code) {
case 'CONTEXT_OVERFLOW':
return await this.compressContext(rejection);
case 'RATE_LIMITED':
return await this.exponentialBackoff(rejection);
case 'UNAUTHENTICATED':
return await this.refreshCredentials(rejection);
default:
throw new RecoveryFailure(`Unhandled MCP rejection: ${rejection.code}`, rejection);
}
}
}
```
The .env Leak that Cost $47,000
Environment variable management in AI systems differs significantly from traditional applications. Dynamic prompts, external API calls, and sensitive output generation make environment variables a critical concern. A single misconfigured variable can expose API keys, database credentials, or customer PII through the agent's output stream.
In my case, a typo in Docker Compose configuration mistakenly granted an agent access to the production database connection string. The LLM, programmed to provide comprehensive diagnostic information, included the full connection string in its response. The solution lies in adhering to the principle of least environment: grant the agent only the necessary environment variables, scoped per deployment.
Additionally, fetch credentials dynamically through secure endpoints validated against the agent's role and permissions, and implement output sanitization to detect and redact sensitive patterns.
The 70-Line Loop Problem
The most insidious bug in production AI agents is the infinite reasoning loop. Agents that call tools without making progress consume resources linearly while delivering no value. A 70-line execution trace that never terminates poses a more severe risk than an immediate crash. Common culprits include overly permissive tool schemas, enabling agents to cycle through tool invocations relentlessly.
To mitigate this issue, enforce explicit termination conditions:
- Set a maximum iteration bound for tool invocations per turn
- Implement progress detection to verify meaningful state changes before allowing further tool calls
- Maintain a hash of recent tool calls and reject duplicates within a sliding window
By incorporating these safeguards, AI agents can operate reliably, preventing runaway costs and ensuring predictable latency in production environments.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
