- Hadoop 2.x Administration Cookbook
- Gurmukh Singh
- 196字
- 2021-07-09 20:10:30
Control block report storm
When Datanodes come up in large clusters with more than 200 nodes, Namenode will be overwhelmed by the block reports and this can cause Namenode to become unresponsive.
Getting ready
This recipe makes more sense for large clusters, not in terms of the number of nodes, but the number of blocks in the cluster.
How to do it...
- ssh to Namenode and edit the
hdfs-site.xmlfile to add the following property to it:<property> <name>dfs.blockreport.initialDelay</name> <value>20</value> </property>
- Copy
hdfs-site.xmlacross all nodes in the cluster. - Restart HDFS daemons across the nodes for the property to take effect:
$ stop-dfs.sh $ start-dfs.sh
How it works...
The dfs.blockreport.initialDelay parameter specifies the time in seconds. This is the upper limit of the allotted time, and it is chosen randomly by all Datanodes. What it means is that a few Datanodes can take the value of 1, others may be 2, and a few others 10, but the maximum values are capped at 20 seconds.
Now, instead of the block report being sent immediately when the Datanodes come up, it will be delayed by the specified number of seconds. This will reduce the block advertisement storm on Namenode.
- AutoCAD快速入門與工程制圖
- Linux Mint System Administrator’s Beginner's Guide
- 3D Printing with RepRap Cookbook
- 計算機應用復習與練習
- Photoshop CS4經典380例
- 條碼技術及應用
- 可編程控制器技術應用（西門子S7系列）
- Maya極速引擎：材質篇
- Windows Server 2003系統安全管理
- SAP Business Intelligence Quick Start Guide
- 電子設備及系統人機工程設計（第2版）
- PowerMill 2020五軸數控加工編程應用實例
- 數字多媒體技術基礎
- Cortex-M3嵌入式處理器原理與應用
- Microsoft 365 Mobility and Security：Exam Guide MS-101