Understanding and Tailing Data Streams

This chapter does a deep dive on the most critical aspect of the real-time application, which is about getting the streaming data from the source to the compute component. We will discuss the expectations and choices which are available. We will also walk the reader through which ones are more appropriate for certain use cases and scenarios. We will give high-level setup and some basic use cases for each of them. In this chapter, we will also introduce technologies related to data ingestion for the use cases.

The following is the list of components:

• Understanding data streams
• Setting up infrastructure for data ingestion
• Taping data from the source to the processor: expectations and caveats
• Comparing and choosing what works best for your use case
• Do it yourself