Before You Add Kafka, Redis, and Elasticsearch: Try One Postgres First

Last month, the author was building out their AI agent infrastructure, which required three components: a job queue, a cache for repeated lookups, and search functionality for article text. Their instinct was to use RabbitMQ or Redis Streams for the queue, Redis for the cache, and Elasticsearch or Meilisearch for search. However, this created five separate services with their own Docker images, failure modes, and monitoring requirements. Intrigued by the idea of using Postgres for all three jobs, they decided to try it out.

They created a table with SKIP LOCKED for the queue, an unlogged table for the cache, and a tsvector column with a GIN index for search. The result was a single schema, backup strategy, and connection pool. This approach resonated with other companies like Contentful and Instacart, who have also successfully implemented Postgres for various tasks.

Pattern 1: The job queue with SKIP LOCKED is implemented using a jobs table where workers claim jobs using SELECT ... FOR UPDATE SKIP LOCKED. Each worker locks the rows they grab, preventing others from claiming the same job. The claim query is executed using a native query in Spring Data, which updates the status and locks the row.

A poller, scheduled to run every 2 seconds, fetches a batch of jobs and processes them. This provides durable job queueing with minimal overhead compared to deploying and configuring a separate queueing system like RabbitMQ.

Pattern 2: The cache is implemented as an unlogged table. This table type skips the write-ahead log, resulting in faster writes and lower latency, similar to a cache. The table is created with a primary key on the key, a jsonb field for the value, and an expires_at timestamp. The Spring repository provides a native query to retrieve values from the cache.

TTL handling can be managed with a scheduled job that deletes expired entries. This approach provides a simple and lightweight caching solution that can be scaled down as needed.

These Postgres-based patterns provide elegant solutions to common challenges in building microservices and offer a more streamlined approach compared to deploying separate specialized services. While these solutions may not be suitable for large-scale, high-performance systems, they can be highly effective for smaller projects and teams looking to minimize complexity and overhead.