Serverless MCP server for AI agent data lake access on Amazon ECS Fargate

Organizations developing AI-powered applications struggle with granting large language model (LLM) agents access to enterprise data lakes without duplicating integration logic for each agent and application. Initially, developers connected agents directly to databases and APIs, leading to custom code creation for each new dataset and the need for manual permission enforcement for every new data source. This resulted in a multi-day engineering effort when onboarding new datasets or agents.

To address this issue, a team built a serverless Model Context Protocol (MCP) server on Amazon Elastic Container Service (Amazon ECS) with AWS Fargate. This server functions as a centralized, agent-agnostic gateway between AI agents and the enterprise data lake. Adding a new dataset only requires a JSON metadata file, which the server automatically discovers during the next pipeline sync, eliminating the need for per-dataset integration code.

The architecture comprises four layers: client, MCP server, tools, and knowledge-base refresh. The metadata catalog, underlying the tools layer, stores schema descriptions, column-level semantics, and table relationships. AI agents reference this catalog when generating SQL queries, eliminating the need for hardcoding table structures or redeploying when schemas change. The knowledge-base refresh layer maintains the catalog current by indexing new dataset metadata as it arrives.

The request flow from client applications to the data layer includes user query submission, DNS resolution through Amazon Route 53, HTTPS routing via Application Load Balancer, JWT authentication with Amazon Cognito, and session storage in Amazon DynamoDB. The client layer handles session state and forwards requests to the MCP server, which includes AI agent orchestration using Amazon Bedrock, Amazon Bedrock Guardrails, and tools backends such as Amazon Athena, Amazon Redshift, and Amazon Bedrock Knowledge Bases.