- Hadoop MapReduce v2 Cookbook（Second Edition）
- Thilina Gunarathne
- 460字
- 2021-07-23 20:32:56
Using EMR bootstrap actions to configure VMs for the Amazon EMR jobs
EMR bootstrap actions provide us a mechanism to configure the EC2 instances before running our MapReduce computations. Examples of bootstrap actions include providing custom configurations for Hadoop, installing any dependent software, distributing a common dataset, and so on. Amazon provides a set of predefined bootstrap actions as well as allowing us to write our own custom bootstrap actions. EMR runs the bootstrap actions in each instance before Hadoop cluster services are started.
In this recipe, we are going to use a stop words list to filter out the common words from our WordCount sample. We download the stop words list to the workers using a custom bootstrap action.
How to do it...
The following steps show you how to download a file to all the EC2 instances of an EMR computation using a bootstrap script:
- Save the following script to a file named
download-stopwords.sh. Upload the file to a Blob container in the Amazon S3. This custom bootstrap file downloads a stop words list to each instance and copies it to a pre-designated directory inside the instance.#!/bin/bash set -e wget http://www.textfixer.com/resources/common-english-words-with-contractions.txt mkdir –p /home/hadoop/stopwords mv common-english-words-with-contractions.txt /home/hadoop/stopwords
- Complete steps 1 to 10 of the Running Hadoop MapReduce computations using Amazon Elastic MapReduce recipe in this chapter.
- Select the Add Bootstrap Actions option in the Bootstrap Actions tab. Select Custom Action in the Add Bootstrap Actions drop-down box. Click on Configure and add. Give a name to your action in the Name textbox and provide the S3 path of the location where you uploaded the
download-stopwords.shfile in the S3 location textbox. Click on Add.
- Add Steps if needed.
- Click on the Create Cluster button to launch instances and to deploy the MapReduce cluster.
- Click on Refresh in the EMR console and go to your Cluster Details page to view the details of the cluster.
There's more...
Amazon provides us with the following predefined bootstrap actions:
configure-daemons: This allows us to set Java Virtual Machine (JVM) options for the Hadoop daemons, such as the heap size and garbage collection behavior.configure-hadoop: This allows us to modify the Hadoop configuration settings. Either we can upload a Hadoop configuration XML or we can specify individual configuration options as key-value pairs.memory-intensive: This allows us to configure the Hadoop cluster for memory-intensive workloads.run-if: This allows us to run bootstrap actions based on a property of an instance. This action can be used in scenarios where we want to run a command only in the Hadoop master node.
You can also create shutdown actions by writing scripts to a designated directory in the instance. Shutdown actions are executed after the job flow is terminated.
Refer to http://docs.amazonwebservices.com/ElasticMapReduce/latest/DeveloperGuide/Bootstrap.html for more information.
- JBoss Weld CDI for Java Platform
- Learning Single：page Web Application Development
- Practical Data Analysis Cookbook
- Oracle 11g從入門到精通（第2版） （軟件開發視頻大講堂）
- Learning Selenium Testing Tools with Python
- 新一代通用視頻編碼H.266/VVC：原理、標準與實現
- 劍指Offer（專項突破版）：數據結構與算法名企面試題精講
- Python 深度學習
- SAS數據統計分析與編程實踐
- 小學生C++創意編程（視頻教學版）
- Python數據結構與算法（視頻教學版）
- 劍指大數據：企業級數據倉庫項目實戰（在線教育版）
- 零基礎學C語言第2版
- JSP程序設計實例教程（第2版）
- Java Web應用開發項目教程