Introducing the example projects

This is a good time to introduce the example projects that we are going to work on in the book. The three examples are varied in the type of data they are presenting, in the hope that the examples will present the user with different ways of visualizing and working with different data. It may be worth getting the data indexed so that you can start to get an idea of what we will be working with.

NASA HTTP data

Although the data is a little old, I think it can give an interesting insight into the web traffic for the NASA website. The data is from 1995 and contains two traces of two months of all HTTP requests to the web server at the Kennedy Space Centre in Florida. The log files are Squid proxy logs and provide details on the host making the request, timestamp, request being made, HTTP reply code, and bytes in the reply.

A download of the data can be found at the following location: http://ita.ee.lbl.gov/html/contrib/NASA-HTTP.html.

The example projects will help analyze the web traffic hitting the NASA website and provide visualization and insights into the site's usage. The data will allow us to start with basic visualizations within the SplunkWeb Framework.

Game of life

If you have been working in development, even for a short period of time, I am sure you will have heard of Conway's Game of Life. Even though it's called a game, it's more of a simulation of biological cells, where we can watch the cells evolve to either live or fail. The cells are governed by a set of rules that determines if they live or die through each generation or step in the simulation:

1. Any live cell with fewer than two live neighbors will die, as if caused by under-population.
2. Any live cell with two or three live neighbors lives on to the next generation.
3. Any live cell with more than three live neighbors dies, as if caused by overpopulation.
4. Any dead cell with exactly three live neighbors becomes a live cell, as if caused by reproduction.

The logs presented here are random, but will consist of the grid where the cells will live, a timestamp, and the cells that are present through each generation of the life cycle (https://en.wikipedia.org/wiki/Conway%27s_Game_of_Life).

I have created a GitHub repository with a basic example of Conway's Game Of Life, but I have also produced logs for the script for 2 hours to give you some sample data that can be worked with through the examples. The sample Python script and log file can be found by going to the following link:  https://github.com/vincesesto/game_of_life_splunk.

From here, you can index the file called game_of_life.log. If you are using at least version 6 of Splunk, the logs will be indexed correctly with the events separated correctly for each date and timestamp. The sample log file will look similar to the following image:

The example data that we have will allow us to analyze the simulation of cells, and although the data is not very complex, we should hopefully provide some interesting visualizations and take our skills with the Splunk Web Framework further.

Historical stock market data

Yahoo! Finance provides an API that allows people to download historical stock market data directly to their environment. In our example, we will take a few different companies and download their historical data for the year 2015, displaying the date stamp, opening value for the day, highest value of the day, lowest value for the day, closing value, volume traded for the day, and adjusted close value of the stock. The sample data will be in CSV form and the API call will be similar to the following URL:  http://ichart.finance.yahoo.com/table.csv?s=YHOO&d=0&e=28&f=2016&g=d&a=3&b=12&c=2015&ignore=.csv.

The API call is pretty straightforward and the commands are listed here:

• s: Company symbol (Yahoo!)
• d: To month -1
• e: To day
• f: To year
• g: Set up of date (d for day, m for month, y for yearly)
• a: From month -1
• b: From day (two digits)
• c: From year

For more details on different company symbols and more explanations of the data that the API can provide, go to the Yahoo! Finance site at https://finance.yahoo.com/.

The data presented is an interesting sample is varied, allowing for interesting trend analysis. This is where we will take our skills further and start to use more of the advanced features of the Splunk Web Framework.