NRT – The Storm solution

This solution captures the high–level streaming data in real–time and routes it through some Queue/broker: Kafka or RabbitMQ. Then, the distributed processing part is handled through Storm topology, and once the insights are computed, they can be written to a fast write data store like Cassandra or some other queue like Kafka for further real–time downstream processing:

As per the figure, we collect real–time streaming data from diverse data sources, through push/pull collection agents like Flume, Logstash, FluentD, or Kafka adapters. Then, the data is written to Kafka partitions, Storm topologies pull/read the streaming data from Kafka and processes this flowing data in its topology, and writes the insights/results to Cassandra or some other real–time dashboards.