Kafka Producers and Consumers
Producers write data on the topic. Topics are made up of partitions in Kafka. And producers know which partitions they will be writing to.
How is Data sent from producers?
— Data is sent using the Round Robin method (0, 1 & 2). Producers can choose to send a key with the message.
— If the key is null, then they follow the round-robin approach.
— And in case the key is not null, then all messages with that particular key will go to the same partitions(Hashing technique is used for that).
Kafka Message created by user Comprises of —
- Key and value: In binary format. They both can be null.
2. Compression type to the message if any.
3. Headers add metadata to your message. They are optional.
4. Partition And Offset you would like to provide if the topic is multi-partitioned.
5. Timestamp System or set it up by the user.
Concept of Serialization
Kafka only understands the language of bytes.
Kafka only accepts bytes as input from producers and sends bytes as output to consumers.
What that means is your message in object form will be serialized to bytes?
It provides common serializers for different formats(JSON, Avro, etc.), which are applied to your key-value pair while sending data.
Kafka Consumers
— Consumers read data from a topic.
— They will be reading it from the offset in increasing order.
— Consumers hold a tendency to recover from broker failures.
Example topic with three partitions —
Partition 1 has five offsets
Partition 2 has three offsets
Partition 3 has two offsets
Read from 0 to 4
Read from 0 to 2
Read from 0 to 1
Consumer Deserializer
— To get back your data deserialized to object.
— Deserializers are used on the key and value of the message.
— And its type must remain the same during the topic lifecycle.
