- Hadoop 2.x Administration Cookbook
- Gurmukh Singh
- 270字
- 2021-07-09 20:10:30
Distcp usage
In Hadoop, we deal with large data, so performing a simple copy operation might not be the optimal thing to do. Imagine copying a 1 TB file from one cluster to another, or within the same cluster to a different path, and after 50% of the copy operation it times out. In this situation, the copy has to be started from the beginning.
Getting ready
This recipe shows the steps needed to copy files within and across the cluster. Ensure that the user has a running cluster with YARN configured to run MapReduce, as discussed in Chapter 1, Hadoop Architecture and Deployment.
For this recipe, there is no configuration needed to run Distcp; just make sure HDFS and YARN is up and running.
How to do it...
- ssh to Namenode or the edge node and execute the following command to copy the
projectsdirectory to thenewdirectory:$ hadoop distcp /projects /new - The preceding command will submit a MapReduce job to the cluster, and once the job finishes we can see the data copied at the destination.
- We can perform an incremental copy as well by using the following command:
- The copy can be performed across clusters as a backup, or simply to move data from one cluster to another:
$ hadoop distcp hdfs://master1.cyrus.com:9000/projects hdfs://nn1.cluster1.com:9000/projects
推薦閱讀
- Clojure Data Analysis Cookbook
- Excel 2007函數(shù)與公式自學(xué)寶典
- 精通Windows Vista必讀
- Julia 1.0 Programming
- 自動檢測與傳感技術(shù)
- Docker High Performance（Second Edition）
- Enterprise PowerShell Scripting Bootcamp
- 實用網(wǎng)絡(luò)流量分析技術(shù)
- PLC與變頻技術(shù)應(yīng)用
- 工業(yè)機器人力覺視覺控制高級應(yīng)用
- INSTANT Adobe Story Starter
- 設(shè)計模式
- 案例解說Delphi典型控制應(yīng)用
- 新一代人工智能與語音識別
- 暗戰(zhàn)強人：黑客攻防入門全程圖解