How it works...

The data lake is a valuable source of information. Elasticsearch is uniquely suited for indexing this coarse-grained time series information. Kibana is the data visualization plugin for Elasticsearch. Kibana is a great tool for creating dashboards containing statistics about the events in the data lake and to perform ad hoc searches to troubleshoot system problems based on the contents of the events.

In this recipe, we are using Kinesis Firehose because it performs the heavy lifting of writing the events to Elasticsearch. It provides buffering based on time and size, hides the complexity of the Elasticsearch bulk index API, provides error handling, and supports index rotation. In the custom elasticsearch Serverless plugin, we create the index template that defines the index_patterns and the timestamp field used to affect the index rotation.

This recipe defines one delivery stream, because in this cookbook, our stream topology consists of only one stream with ${cf:cncb-event-stream-${opt:stage}.streamArn}. In practice, your topology will consist of multiple streams and you will define one Firehose delivery stream per Kinesis stream to ensure that all events are indexed.