Hadoop namenode secondarynode stops abruptly

0
votes

I have a single node where i run MR jobs frequently. The system up and running fine from two days. Suddently all the hadoop process were stoped. It is annoying that my all running jobs were failed since then. I have gotten the logs saying

for secondary namenode:

java.io.ioexception: cannot lock storage /hdfs/namesecondary. the directory is already locked

for namenode :

java.io.ioexception: cannot lock storage /hdfs/name. the directory is already locked

I tried to come out of the safemode and tried formatting the namenode but this also throws the same exception.

How can i start the Hadoop process. There were no disk space issue. its a 900GB disk and 300GB is free at the time of this shutdown.

What should i verify now? Have not found any thread on this world.

Thanks

1
javahadoophbase
Did you try re-starting the NameNode? - Manjunath Ballur

1 Answers

0
votes

I solved it by removing the inuse_lock file in /hdfs/name/ and /hdfs/secondarynamenode. After i formatted both namenode and secondarynamenode.