Need for Kafka- Theory for Beginners
DISTRIBUTED MESSAGE STREAMING PLATFORM
Streaming means flow here.
Publish & Subscribe Mechanism to stream the records.
Publish means write and subscribe means read streams(flow) of events.
Now you will say, what does the word distribute mean here?
I am coming to this point.
Example: Let us say you have 1000 Records stored in one location. If the system gets crashes, everything is lost.
Solution: To put it in a distributed system.
Now you thought to insert your data in a distributed way in five systems(200 Records each). But, again, if anyone’s system gets crashed, data is lost.
Solution: Put complete records(data) in distributed systems instead of dividing the data. Replication is done at multiple locations. Space is used more, but data is safe now.
Why does it follow the messaging system?
Let us say you have two source services and three destination services. One source needs to send data to the three destination services, and similarly, the second source service needs to send data to the other three destination services. Total six connections (n*m).
Disadvantages
- Increased Bottleneck.
- Highly coupled systems
Now there comes the need for a messaging system.
The message is served to the messaging system, and the destination service will consume it from there.
Total Connections(n + m).
Advantages:
- Loosely Coupled system.
- Notifications to all the destination services.
- Messaging systems keep destination services keep consuming data.
For example — Please Check Pic2 and Pic3.
— Any queries, hit me up in the comment sections.
— Do let me know in the comments if any correction is required(If anyone of you know prior).
