- Hadoop MapReduce v2 Cookbook(Second Edition)
- Thilina Gunarathne
- 428字
- 2021-07-23 20:32:53
HDFS command-line file operations
HDFS is a distributed filesystem, and just like any other filesystem, it allows users to manipulate the filesystem using shell commands. This recipe introduces some of these commands and shows how to use the HDFS shell commands.
It is worth noting that some of the HDFS commands have a one-to-one correspondence with the mostly used Unix commands. For example, consider the following command:
$ bin/hdfs dfs –cat /user/joe/foo.txt
The command reads the /user/joe/foo.txt
file and prints it to the screen, just like the cat
command in a Unix system.
Getting ready
Start the HDFS server by following the Setting up HDFS recipe or the Setting up Hadoop ecosystem in a distributed cluster environment using a Hadoop distribution recipe.
How to do it...
- Run the following command to list the content of your HDFS home directory. If your HDFS home directory does not exist, please follow step 9 of the Setting up Hadoop ecosystem in a distributed cluster environment using a Hadoop distribution recipe to create your HDFS home directory.
$ hdfs dfs -ls
- Run the following command to create a new directory called
test
inside your home directory in HDFS:$ hdfs dfs -mkdir test
- The HDFS filesystem has
/
as the root directory. Run the following command to list the content of the newly created directory in HDFS:$ hdfs dfs -ls test
- Run the following command to copy the local readme file to
test
:$ hdfs dfs -copyFromLocal README.txt test
- Run the following command to list the
test
directory:$ hdfs dfs -ls test Found 1 items -rw-r--r-- 1 joesupergroup1366 2013-12-05 07:06 /user/joe/test/README.txt
- Run the following command to copy the
/test/README.txt
file back to a local directory:$ hdfs dfs –copyToLocal \ test/README.txt README-NEW.txt
How it works...
When the command is issued, the HDFS client will talk to HDFS NameNode on our behalf and carry out the operation. The client will pick up the NameNode from the configurations in the HADOOP_HOME/etc/hadoop/conf
directory.
However, if needed, we can use a fully qualified path to force the client to talk to a specific NameNode. For example, hdfs://bar.foo.com:9000/data
will ask the client to talk to NameNode running on bar.foo.com
at the port 9000
.
There's more...
HDFS supports most of the Unix commands such as cp
, mv
, and chown
, and they follow the same pattern as the commands discussed earlier. The following command lists all the available HDFS shell commands:
$ hdfs dfs -help
Using a specific command with help
will display the usage of that command.
$ hdfs dfs –help du
- Vue.js 3.x快速入門
- Designing Hyper-V Solutions
- 基于Swift語言的iOS App 商業實戰教程
- Instant RubyMotion App Development
- Instant Ext.NET Application Development
- Visualforce Developer’s guide
- TMS320LF240x芯片原理、設計及應用
- Visual Studio 2015高級編程(第6版)
- 打開Go語言之門:入門、實戰與進階
- Domain-Driven Design in PHP
- SQL Server 2008 R2數據庫技術及應用(第3版)
- C語言程序設計實踐
- Effective C++:改善程序與設計的55個具體做法(第三版)中文版(雙色)
- Roslyn Cookbook
- PHP高性能開發:基礎、框架與項目實戰