Preparing hardware for Hadoop

One important aspect of Hadoop setup is defining the hardware requirements and sizing before the start of a project. Although Apache Hadoop can run on commodity hardware, most of the implementations utilize server-class hardware for their Hadoop cluster. (Look at powered by Hadoop or go through the Facebook Data warehouse research paper in SIGMOD-2010 for more information).

There is no rule of thumb regarding the minimum hardware requirements for setting up Hadoop, but we would recommend the following configurations while running Hadoop to ensure reasonable performance:

• CPU ≥ 2 Core 2.5 GHz or more frequency
• Memory ≥ 8 GB RAM
• Storage ≥ 100 GB of free space, for running programs and processing data
• Good internet connection

There is an official Cloudera blog for cluster sizing information if you need more detail. If you are setting up a virtual machine, you can always opt for dynamically sized disks that can be increased based on your needs. We will look at how to size the cluster in the upcoming Hadoop cluster section.