AWS Serverless Patterns and Anti-Patterns: What Works, What Breaks, and When to Use What

Serverless architecture on AWS is not just about using Lambda. It is a design philosophy that emphasizes leveraging AWS-managed infrastructure, paying for actual usage, and utilizing managed services that scale independently. However, the patterns that work in serverless differ significantly from traditional architectures, and learning the anti-patterns (those that lead to increased costs or outages) is crucial.

The AWS serverless stack consists of several key components:

1. Compute: Lambda, Fargate (serverless containers)

2. API: API Gateway (REST/HTTP/WebSocket), AppSync (GraphQL)

3. Orchestration: Step Functions, EventBridge Scheduler

4. Messaging: SQS, SNS, EventBridge

5. Storage: S3, DynamoDB, Aurora Serverless

6. Streaming: Kinesis, DynamoDB Streams, MSK Serverless

7. Authentication: Cognito, IAM, Lambda Authorizers

8. Observability: CloudWatch, X-Ray, Application Signals

The underlying principle is to compose applications from managed services, with Lambda serving as the glue connecting them. The most common serverless pattern is a synchronous API (request/response) where a client sends a request to API Gateway, which forwards it to a Lambda function. The Lambda function then interacts with services like DynamoDB or Aurora Serverless to process the data and returns the response back through the same pipeline.

Best practices for this pattern include:

- Using API Gateway HTTP API instead of REST API for better cost-efficiency, speed, and simplicity.

- Assigning one Lambda function per route to maintain a single responsibility.

- Keeping Lambda functions warm by using provisioned concurrency for latency-sensitive endpoints.

- Utilizing DynamoDB for simple access patterns to benefit from automatic scaling and avoid connection pooling.

- Employing Aurora Serverless v2 for complex queries, but make sure to use RDS Proxy to manage connections.

When deciding between HTTP API and REST API, consider the following:

- HTTP API has a lower cost per million requests ($1.00) and latency (around 10ms) compared to REST API ($3.50/million requests and ~30ms latency).

- HTTP API is suitable for standard APIs where cost is a primary concern.

- REST API is preferable when you need advanced features such as WAF, throttling plans, API keys, caching, or request validation.

For asynchronous event processing, events trigger Lambda functions, allowing independent processing from the caller. Common examples include:

- S3 Upload → Lambda: Process image → S3 (store thumbnail)

- SQS Message → Lambda: Process order → DynamoDB (update status)

- EventBridge → Lambda: Handle event → SNS (send notification)

Best practices for this pattern include:

- Using Dead Letter Queues (DLQ) to handle failed events and prevent data loss.

- Designing for idempotency to handle potential duplicate event deliveries.

- Processing messages in batches using SQS Lambda trigger to optimize costs.

- Setting reserved concurrency limits to prevent a single function from consuming all account concurrency.

- Using event filtering to invoke the Lambda function only for relevant messages, thereby reducing unnecessary invocations.

For workflow orchestration, Step Functions provide a visual workflow designer for multi-step processes involving branching, retries, and error handling. You can create a step function with various states, such as:

- Validate input

- Process payment (Lambda)

- Success → Reserve inventory (Lambda)

- Failure → Notify customer (SNS) → End

- Ship order (Lambda)

- Wait 7 days

- Send follow-up email (Lambda)

Best practices for Step Functions include choosing between Express and Standard modes based on your workflow requirements, duration, and pricing considerations. Express mode allows for longer durations (up to 1 year) and costs per state transition, while Standard mode offers at-least-once execution and shorter durations (up to 5 minutes).

For streaming and real-time processing, Kinesis can handle continuous data ingestion, while Lambda processes the data in real-time. Firehose can be used for batched delivery to S3, Redshift, or OpenSearch without writing any custom code. Lambda can also be used to bisect and isolate failed batches for further analysis.

Best practices for this pattern include:

- Utilizing Kinesis for ordering and replay capabilities.

- Using Firehose for batched delivery to S3, Redshift, or OpenSearch without writing custom code.

- Implementing tumbling windows in Lambda to aggregate data over time efficiently.

- Bisecting on error to handle failed batches separately.

For fan-out or scatter-gather patterns, one trigger can initiate several parallel processes, with results aggregated later. API Gateway can serve as the trigger, spawning multiple Lambda functions to process items concurrently. The results are then combined and returned as a response.

Best practices for this pattern include:

- Utilizing Step Functions Distributed Map for parallel processing of millions of items with up to 10,000 concurrent executions.

- Applying this pattern for batch processing large datasets from S3, parallel API calls to external services, or large-scale data transformations.

Lastly, AppSync (AWS AppSync) is a GraphQL API service that enables flexible client-driven queries. It integrates with DynamoDB, Lambda, Aurora, and even external APIs, allowing clients to fetch only the necessary data, reducing over-fetching. AppSync also supports real-time subscriptions through WebSocket and provides direct DynamoDB/Aurora resolvers.

Best practices for AppSync include:

- Fetching only the required data by clients to avoid unnecessary data transfer.

- Leveraging real-time subscriptions (WebSocket) for live updates.

- Using direct DynamoDB and Aurora resolvers to reduce latency and eliminate the need for Lambda for simple CRUD operations.