Logstash

Download Logstash version 5.2.1 from https://artifacts.elastic.co/downloads/logstash/logstash-5.2.1.tar.gz on your local machine. Once the logstash-5.2.1.tar.gz file is downloaded, extract the files using the following command:

    cp logstash-5.2.1.tar.gz ~/demo/.
    tar -xvf ~/demo/logstash-5.2.1.tar.gz

The following folders and files will be extracted as seen in the following screenshot:

We will repeat the same example explained for NiFi, that is, reading from a file and pushing an event on a Kafka topic. Let's create topic on Kafka for Logstash:

    ./kafka-topics.sh --create --topic logstash-example --zookeeper localhost:2181 --partitions 1 --replication-factor 1

To run the example in Logstash, we have to create a configuration file, which will define the input, filter, and output. Here, we will not apply any filters, so the configuration file will contain two components: input as reading from file and output as writing into Kafka topic. The following is the configuration file required to execute the example:

input { 
    file {
              path => "/home/ubuntu/demo/files/test"
              start_position => "beginning"
         } 
    }
    output {
              stdout { codec => plain }
              kafka { 
                      codec => plain {format => "%{message}"}
                      topic_id => "logstash-example"
                      client_id => "logstash-client"
                      bootstrap_servers => "localhost:9092"
                     }
            }

Create a file with the name logstash-kafka-example.conf and paste the previous configuration into that file. Create an input file of test in /home/ubuntu/demo/files and add the following content:

hello 
this 
is 
logstash 
kafka 
integration 

Before running Logstash Pipeline, start a process from the console to read from the Kafka topic logstash-example, using the following command:

    /bin/kafka-console-consumer.sh --topic logstash-example --bootstrap-server localhost:9092 --from-beginning

Now, run the example using the following command:

    /bin/logstash -f config/logstash-kafka-example.conf

The output on Kafka will be as shown in the following screenshot: