O que é, e quando usar Kafka?

Imagine um e-commerce que recebe milhares de pedidos por hora. To guarantee seamless operation, it is crucial to orchestrate events like orders, payments, inventory updates, and notifications. This is where the power of messaging comes into play.

In scenarios with numerous sensors, such as smart farms, factories, or medical device networks, IoT devices constantly transmit measurements. These data points are then sent to a message broker like Kafka. Subsequently, other devices react to this information. For instance, a sensor measures soil moisture, temperature, and light levels in a specific area of a plantation.

It sends this data to a message broker, after which an irrigation service reads the moisture data and decides whether to activate the sprinklers in that area.

Messaging is a method by which various systems can exchange information with one another without one service calling the other directly. They communicate by sending and receiving messages through an intermediary, such as Kafka, RabbitMQ, etc. Messaging helps solve several common issues in modern systems:

- Decoupling services: Ensures microservices communicate without creating dependency.

- Processing tasks asynchronously: Operations occur in the background without blocking the user.

- Distributing work among multiple consumers: Makes the system more resilient, as messages are stored if a service fails and are processed once it returns.

Apache Kafka is an open-source messaging system that stores data streams in a durable, ordered, and distributed manner. It enables the creation of systems that collect logs, track user activity, and integrate various components in a resilient manner. Kafka distinguishes itself from most brokers by retaining messages even after a consumer reads and confirms processing.

Advantages include handling high volumes of messages, real-time processing, and durability and fault tolerance. Kafka also ensures data durability and supports multiple consumers.

However, there are also some disadvantages: It becomes excessive when there is low data volume (less than 1000 messages per second). It requires complex configuration and operation. It comes with a high maintenance cost and requires specialized personnel. It has a steep learning curve.