A Kafka Streams topology starts in a sandbox, not on your cluster
Somebody owns the Kafka cluster you need, and it is not you. You have a topology in mind: read orders , keep the big ones, write them somewhere. You have a predicate in mind too, and you are fairly sure it is right. "Fairly sure" is the problem. To find out, you need to run it, and the only place to run it is the shared cluster: a consumer group in everyone's list, internal topics with your name…
A Kafka Streams topology begins in a sandbox, not on the production cluster. The concept is to test the topology first before deploying it on the shared cluster. The main objective is to ensure that the predicate used in the topology works as intended, without causing any harm to the actual data and other consumers downstream.
To achieve this, developers need to create a stack of their own, which they can safely break and use for testing. Along with this, they also need a method to bring the findings back to the production cluster without any data transfer. The primary challenge is not in writing the topology, but in determining whether the predicate accurately represents the developer's intentions based on the actual records arriving.
The author highlights two key factors that significantly reduce the loop between having an idea and finding out whether the predicate works as expected. First, the records being evaluated must be real. Second, the evaluation should be the same as the one that will be running after deployment. Lastly, it is crucial that there are no repercussions from being wrong, as anything wrong would have already been acted upon by other downstream consumers.
The post continues by discussing the importance of having a 'stack of your own,' which is free to break without any consequences. It uses Docker Compose to set up a single-node Kafka, Schema Registry, and a console that runs Kafka Streams topologies. The entire setup is isolated and self-contained, ensuring that it does not affect any other system or data. The playground topics are pre-seeded with real Avro records and a pre-running topology, providing a canvas for developers to experiment with.
The author also emphasizes the need for a 'sandboxed topic' to test the topology, and how to set it up using JSON Schema values. The topic name must match the actual production topic, ensuring that the design being tested carries over to the real-world scenario. The sample records are crucial, as they help prove the logic against the actual data shape rather than volume. This method allows developers to test the predicate's performance without causing any harm to the production data or other consumers.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
