How to fix: org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot create directory. Name node is in safe mode.

I guess by now you know that Apache Hadoop has a safe mode that enforces the node to have only read-only disk permissions. This is usually because your node lacks sufficient available disk space or memory. When you run Hadoop MapReduce jobs or Apache Spark jobs, you usually write (at least your failover checkpoints) to HDFS even if you perform your job in-memory. Problem becomes visible when your node needs to write to HDFS but since its automatically in safe mode, it halts the process.

Solution

If you use Hadoop version 2.6.1 and earlier, you can switch the safe mode off with a command:

$HADOOP_HOME/hdfs dfsadmin -safemode leave

while the command works, it complains that its depreciated. Also, using this command is problematic if you deploy via docker, too. Problem being it needs the node to be running already.

I actually it because I was running Hadoop in a Docker container and that command magically fails when run in the container. I checked doc and found the HDFS property called dfs.safemode.threshold.pct in documentation that says:

Specifies the percentage of blocks that should satisfy the minimal replication requirement defined by dfs.replication.min. Values less than or equal to 0 mean not to wait for any particular percentage of blocks before exiting safemode. Values greater than 1 will make safe mode permanent.

so I changed the hdfs-site.xml into the following (In older Hadoop versions, apparently you need to do it in hdfs-default.xml:

<configuration>
<property>
    <name>dfs.safemode.threshold.pct</name>
    <value>0</value>
</property>

This way you can easily copy the config with Docker COPY and if not using docker, you can just easily add the property to your hdfs-site.xml.

NOTE: In earlier versions of hadoop this configuration is called hdfs-default.xml.

Now I guess you are done! Post a comment or send a tweet to @_ambodi if you needed more help on this.

The Rstats tag of this blog is added to R Bloggers