- Hadoop 2.x Administration Cookbook
- Gurmukh Singh
- 392字
- 2021-07-09 20:10:29
HDFS health and FSCK
The health of the filesystem is very important for data retrieval and optimal performance. In a distributed system, it becomes more critical to maintain the good health of the HDFS filesystem so as to ensure block replication and near-parallel streaming of data blocks.
In this recipe, we will see how to check the health of the filesystem and do repairs, if any are needed.
Getting ready
Make sure you have a running cluster that has already been up for a few days with data. We can run the commands on a new cluster as well, but for the sake of this lab, it will give you more insights if it is run on a cluster with a large dataset.
How to do it...
- ssh to the
master1.cyrus.comNamenode and change the user tohadoop. - To check the HDFS root filesystem, execute the
hdfs fsck /command, as shown in the following screenshot:
- We can also check the status of just one file instead of the entire filesystem, as shown in the following screenshot:
- The output of the
fsckcommand will show the blocks for a file, the replication status, whether blocks are corrupted, and many more details, as shown in the following screenshot:
- We can also look at how the blocks of a file are laid across the cluster using the commands as shown in the following screenshot:
- In the cluster named
cyrus, you can see that there are some corrupt blocks. We can simulate this by manually deleting a block of a file on the lower filesystem. Each of the HDFS blocks, is a file at the lower filesystem such as EXT4.
- The corrupt blocks can be fixed by deleting them, and for an under replicated block we can use the
hdfs dfs -setrep 2 /input/new.txtcommand, so that a particular file is set to the desired number of replications. If we need to set many files to a specified number of replications, just loop through the list and do asetrepon them.
How it works...
The hdfs fsck /command is similar to the Linux fsck command. In Hadoop, it does not repair the filesystem automatically and needs a manual intervention. To see what options there are for this command, please use the hdfs fsck –help help command.
See also
- The Configuring rack awareness recipe
- AutoCAD快速入門與工程制圖
- Getting Started with MariaDB
- HBase Design Patterns
- 流處理器研究與設(shè)計(jì)
- 樂(lè)高創(chuàng)意機(jī)器人教程（中級(jí) 下冊(cè) 10~16歲） (青少年iCAN+創(chuàng)新創(chuàng)意實(shí)踐指導(dǎo)叢書)
- 西門子S7-200 SMART PLC實(shí)例指導(dǎo)學(xué)與用
- CentOS 8 Essentials
- 運(yùn)動(dòng)控制系統(tǒng)應(yīng)用與實(shí)踐
- Mastering pfSense
- 從零開(kāi)始學(xué)SQL Server
- 基于ARM9的小型機(jī)器人制作
- Hands-On Dashboard Development with QlikView
- PowerMill 2020五軸數(shù)控加工編程應(yīng)用實(shí)例
- 算法設(shè)計(jì)與分析
- Hands-On Agile Software Development with JIRA