Downloading Hadoop

Once you have completed the prerequisites and SSH keyless entry with all the necessary nodes, you are good to download the Hadoop release. You can download Apache Hadoop from http://www.apache.org/dyn/closer.cgi/hadoop/common/. Hadoop provides two options for downloading—you can either download the source code of Apache Hadoop or you can download binaries. If you download the source code, you need to compile it and create binaries out of it. We will proceed with downloading binaries.

One important question that often arises while downloading Hadoop involves which version to choose. You will find many alpha and beta versions, as well as stable versions. Currently, the stable Hadoop version is 2.9.1, however this may change by the time you read this book. The answer to such a question depends upon usage. For example, if you are evaluating Hadoop for the first time, you may choose to go with the latest Hadoop version (3.1.0) with all-new features, so as to keep yourself updated with the latest trends and skills.

However, if you are looking to set up a production-based cluster, you may need to choose a version of Hadoop that is stable (such as 2.9.1), as well as established, to ensure peaceful project execution. In our case, we will download Hadoop 3.1.0, as shown in the following screenshot:

You can download the binary (tar.gz) from Apache's website, and you can untar it with following command:

hadoop@base0:/$ tar xvzf <hadoop-downloaded-file>.tar.gz

The preceding command will extract the file in a given location. When you list the directory, you should see the following folders:

• The bin/ folder contains all executable for Hadoop
• sbin/ contains all scripts to start or stop clusters
• etc/ contains all configuration pertaining to Hadoop
• share/ contains all the documentation and examples
• Other folders such as include/, lib/, and libexec/ contain libraries and other dependencies