官术网_书友最值得收藏!

HDFS command-line file operations

HDFS is a distributed filesystem, and just like any other filesystem, it allows users to manipulate the filesystem using shell commands. This recipe introduces some of these commands and shows how to use the HDFS shell commands.

It is worth noting that some of the HDFS commands have a one-to-one correspondence with the mostly used Unix commands. For example, consider the following command:

$ bin/hdfs dfs –cat /user/joe/foo.txt

The command reads the /user/joe/foo.txt file and prints it to the screen, just like the cat command in a Unix system.

Getting ready

Start the HDFS server by following the Setting up HDFS recipe or the Setting up Hadoop ecosystem in a distributed cluster environment using a Hadoop distribution recipe.

How to do it...

  1. Run the following command to list the content of your HDFS home directory. If your HDFS home directory does not exist, please follow step 9 of the Setting up Hadoop ecosystem in a distributed cluster environment using a Hadoop distribution recipe to create your HDFS home directory.
    $ hdfs dfs -ls
    
  2. Run the following command to create a new directory called test inside your home directory in HDFS:
    $ hdfs dfs -mkdir test
    
  3. The HDFS filesystem has / as the root directory. Run the following command to list the content of the newly created directory in HDFS:
    $ hdfs dfs -ls test
    
  4. Run the following command to copy the local readme file to test:
    $ hdfs dfs -copyFromLocal README.txt test
    
  5. Run the following command to list the test directory:
    $ hdfs dfs -ls test
    Found 1 items
    -rw-r--r-- 1 joesupergroup1366 2013-12-05 07:06 /user/joe/test/README.txt
    
  6. Run the following command to copy the /test/README.txt file back to a local directory:
    $ hdfs dfs –copyToLocal \
    test/README.txt README-NEW.txt
    

How it works...

When the command is issued, the HDFS client will talk to HDFS NameNode on our behalf and carry out the operation. The client will pick up the NameNode from the configurations in the HADOOP_HOME/etc/hadoop/conf directory.

However, if needed, we can use a fully qualified path to force the client to talk to a specific NameNode. For example, hdfs://bar.foo.com:9000/data will ask the client to talk to NameNode running on bar.foo.com at the port 9000.

There's more...

HDFS supports most of the Unix commands such as cp, mv, and chown, and they follow the same pattern as the commands discussed earlier. The following command lists all the available HDFS shell commands:

$ hdfs dfs -help

Using a specific command with help will display the usage of that command.

$ hdfs dfs –help du
主站蜘蛛池模板: 安陆市| 敦化市| 廉江市| 怀安县| 宜章县| 清徐县| 石河子市| 金山区| 丰台区| 科尔| 沂水县| 泸溪县| 鹤岗市| 安阳县| 承德市| 都江堰市| 斗六市| 乳源| 武汉市| 红安县| 甘孜县| 章丘市| 灵丘县| 石泉县| 龙陵县| 白河县| 峨山| 滕州市| 泰兴市| 仁寿县| 兴海县| 香格里拉县| 日照市| 富平县| 彰化市| 九寨沟县| 浮梁县| 盘山县| 临海市| 梁河县| 南川市|