Kafka series 1

Gain insight through storytelling

1. Apache Kafka is an open-source distributed event streaming platform
2. It was developed as a publish-subscribe messaging system to handle mass amounts of data
3. Kafka is used to collect, process, store, and integrate data at scale
4. Kafka is primarily used to build real-time streaming data pipelines and applications that adapt to the data streams
5. It has numerous use cases including distributed streaming, stream processing, data integration, and pub/sub messaging

Storytime

Once upon a time, in the land of data, there was a busy city named Kafka. This city was unique because it was built to facilitate the smooth flow of information.

Broker:

In the heart of Kafka city, there was a busy marketplace known as the Broker. The Broker was the central hub where all the information was exchanged. It was always buzzing with activity, as it stored and categorized all the messages that flowed through the city. (A Kafka broker is a server that runs the Kafka software and stores messages. Each broker is identified by an ID and contains certain topic partitions. The brokers form a network of machines called a cluster)

Zookeeper:

Overseeing the operations of the Broker was the wise old Zookeeper. The Zookeeper was responsible for maintaining order and ensuring that everything ran smoothly. (Zookeeper is a centralized service used to maintain naming and configuration data and to provide flexible and robust synchronization within distributed systems. It keeps track of the status of the Kafka cluster nodes and also keeps track of Kafka topics, partitions, etc)

Producers:

The Producers were the creators of information. They generated messages based on various events and sent them to the Broker. Each message was like a piece of news, an update, or a command that needed to be shared with the rest of the city.(Producers in Kafka are client applications that publish (write) events to a Kafka cluster. They generate messages based on various events and send them to the broker.)

Consumers:

On the other side were the Consumers. They were the recipients of the information. They subscribed to the Broker to receive specific types of messages. Once a message arrived at the Broker, it was immediately sent to all the Consumers who were interested in it. (Consumers in Kafka are client applications that subscribe to (read and process) events. They subscribe to the broker to receive specific types of messages.)

The Producers and Consumers didn’t need to know each other. They just trusted the Broker and the Zookeeper to deliver the right messages at the right time. This way, the city of Kafka thrived as a hub of seamless and efficient communication.

Kafka provides three main functions to its users:

• Publish and subscribe to streams of records
• Effectively store streams of records in the order in which records were generated
• Process streams of records in real time