Broker

One of the fundamental architectural principles is the decoupling of various components. Broker is precisely the component in NRT architecture that not only decouples the data collection component and processing unit but also provides elasticity to hold data in a queue when there are sudden surges in traffic.

Amongst the vast variety of tools and technologies available under this segment, the key ones we would like to touch on are:

• Apache Kafka: Kafka is used for building real-time data pipelines and streaming apps. It is horizontally scalable, fault-tolerant, wickedly fast, and runs in production in thousands of companies. (Source: https://kafka.apache.org/). The salient features of this broker component are:
  • It's highly scalable
  • It's fail safe; it provides for fault tolerance and high availability
  • It is open source and extensible
  • It's disk-based and can store vast amounts of data (this is a USP of Kafka that enables it to virtually cater for any amount of data)
  • It allows replication and partitioning of data

• ActiveMQ: Apache ActiveMQ is fast, supports many cross language clients and protocols, and comes with easy-to-use enterprise integration patterns and many advanced features while fully supporting JMS 1.1 and J2EE 1.4. Apache ActiveMQ is released under the Apache 2.0 License. Key aspects of this protocol are:
  • It supports a variety of clients and protocols
  • It supports JMS 1.1 and J2EE 1.4
  • High performance
  • Support persistence
  • It exposes technology agnostic layer of web services
• RabbitMQ: This is an in-memory, durable, and persistent, low-latency, distributed queue system that has the following salient features:
  • It's robust
  • It's easy to use
  • It supports all major operating systems
  • It is available in both open source and commercial versions

The following table captures the critical aspects of the two close contenders for the broker: