Hadoop previous.checkpoint location

Question

I tried hadoop-1.0.3 as well as 1.0.4. Both under pseudo cluster mode.

My understanding is that the previous.checkpoint directory should be created under secondary name node designated by "fs.checkpoint.dir"? On all occasions I am finding it under the namenode directory designated by "dfs.name.dir". It this something to do with Pseudo mode or my understanding is wrong? Can someone please help!

Below is my hdfs-site.xml file.

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>
    <property>
        <name>dfs.replication</name>
        <value>1</value>
    </property>
    <property>
        <name>dfs.name.dir</name>
        <value>/home/hadoop/lab/hdfs/namenode</value>
    </property>
    <property>
        <name>dfs.data.dir</name>
        <value>/home/hadoop/lab/hdfs/datanode</value>
    </property>
    <property>
        <name>fs.checkpoint.dir</name>
        <value>/home/hadoop/lab/hdfs/secnamenode</value>
    </property>
</configuration>

The below is high level directory structure for the daemons

hadoop@ubuntu:~/lab/hdfs$ ls -l
total 12
drwxr-xr-x 6 hadoop hadoop 4096 Mar 14 03:41 datanode
drwxrwxr-x 5 hadoop hadoop 4096 Mar 14 03:41 namenode
drwxrwxr-x 4 hadoop hadoop 4096 Mar 14 04:46 secnamenode

Below is the NameNode directory details

hadoop@ubuntu:~/lab/hdfs$ ls -l namenode
total 12
drwxrwxr-x 2 hadoop hadoop 4096 Mar 14 04:46 current
drwxrwxr-x 2 hadoop hadoop 4096 Mar 14 03:13 image
-rw-rw-r-- 1 hadoop hadoop    0 Mar 14 03:41 in_use.lock
drwxrwxr-x 2 hadoop hadoop 4096 Mar 14 03:34 previous.checkpoint

Below is the SNN directory details

hadoop@ubuntu:~/lab/hdfs$ ls -l secnamenode
total 8
drwxrwxr-x 2 hadoop hadoop 4096 Mar 14 04:46 current
drwxrwxr-x 2 hadoop hadoop 4096 Mar 14 03:46 image
-rw-rw-r-- 1 hadoop hadoop    0 Mar 14 03:41 in_use.lock

If you need any further details please let me know.

Thanks Rags

Rags Rags · Accepted Answer · 2013-09-02T09:28:56

I have done an extensive and desperate search about this. There seems to be a long pending Bug HDFS-1839 which removes previous.checkpoint directory from the SecondaryNameNode. The same bug might be responsible for having this directory created under the NameNode.

I have seen all the versions of hadoop so far and under all these the previous.checkpoint directory is consistently being created under the NameNode.

Hope soon this bug will be fixed or Apache Hadoop clarifies why directory is created under NameNode

Hadoop previous.checkpoint location

1 Answers