- Hadoop MapReduce v2 Cookbook（Second Edition）
- Thilina Gunarathne
- 224字
- 2021-07-23 20:32:53
Running the WordCount program in a distributed cluster environment
This recipe describes how to run a MapReduce computation in a distributed Hadoop v2 cluster.
Getting ready
Start the Hadoop cluster by following the Setting up HDFS recipe or the Setting up Hadoop ecosystem in a distributed cluster environment using a Hadoop distribution recipe.
How to do it...
Now let's run the WordCount sample in the distributed Hadoop v2 setup:
- Upload the
wc-inputdirectory in the source repository to the HDFS filesystem. Alternatively, you can upload any other set of text documents as well.$ hdfs dfs -copyFromLocal wc-input . - Execute the WordCount example from the
HADOOP_HOMEdirectory:$ hadoop jar hcb-c1-samples.jar \ chapter1.WordCount \ wc-input wc-output
- Run the following commands to list the output directory and then look at the results:
$hdfs dfs -ls wc-output Found 3 items -rw-r--r-- 1 joesupergroup0 2013-11-09 09:04 /data/output1/_SUCCESS drwxr-xr-x - joesupergroup0 2013-11-09 09:04 /data/output1/_logs -rw-r--r-- 1 joesupergroup1306 2013-11-09 09:04 /data/output1/part-r-00000 $ hdfs dfs -cat wc-output/part*
How it works...
When we submit a job, YARN would schedule a MapReduce ApplicationMaster to coordinate and execute the computation. ApplicationMaster requests the necessary resources from the ResourceManager and executes the MapReduce computation using the containers it received from the resource request.
There's more...
You can also see the results of the WordCount application through the HDFS monitoring UI by visiting http://NAMANODE:50070.
推薦閱讀
- Data Visualization with D3 4.x Cookbook（Second Edition）
- Vue 3移動Web開發與性能調優實戰
- Practical Data Science Cookbook（Second Edition）
- Python程序設計（第3版）
- INSTANT Weka How-to
- 區塊鏈：以太坊DApp開發實戰
- Visual Basic程序設計上機實驗教程
- Frank Kane's Taming Big Data with Apache Spark and Python
- Managing Microsoft Hybrid Clouds
- 程序員的成長課
- Learning Kotlin by building Android Applications
- AngularJS UI Development
- Web開發新體驗
- Computer Vision with Python 3
- 中小企業網站建設與管理（靜態篇）