hbase hdfs goes to safemode on process restart ( probably under replication reported? )

Question

UPDATE You need to give hdfs-site.xml to hbase/conf so hbase can use the correct target replica, else it uses default 3.

That fixes the message. But my namenode is always in safemode during every process restart. The fsck is all fine with no errors, no under replicated etc.

I see no logs after:

2012-10-17 13:15:13,278 INFO org.apache.hadoop.hdfs.StateChange: STATE* Safe mode ON. The ratio of reported blocks 0.0000 has not reached the threshold 0.9990. Safe mode will be turned off automatically.

2012-10-17 13:15:14,228 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/127.0.0.1:50010 2012-10-17 13:15:14,238 INFO org.apache.hadoop.hdfs.StateChange: BLOCK
NameSystem.processReport: from 127.0.0.1:50010, blocks: 20, processing time: 0 msecs

Any suggestions ?

I have dfs.replication set to 1. hbase is in distributed mode.

First write goes through, but when I restart namenode always reports blocks as under reported.

Output from hadoop fsck /hbase

/hbase/tb1/.tableinfo.0000000003:  Under replicated blk_-6315989673141511716_1029. Target Replicas is 3 but found 1 replica(s).

.
/hbase/tb1/83859abf0f46016485814a5941b16de5/.oldlogs/hlog.1350414672838:  Under replicated blk_-7364606700173135939_1027. Target Replicas is 3 but found 1 replica(s).
.

/hbase/tb1/83859abf0f46016485814a5941b16de5/.regioninfo:  Under replicated blk_788178851601564156_1027. Target Replicas is 3 but found 1 replica(s).

Status: HEALTHY

 Total size:    8731 B
 Total dirs:    34
 Total files:   25 (Files currently being written: 1)
 Total blocks (validated):      25 (avg. block size 349 B) (Total open file blocks (not validated): 1)
Minimally replicated blocks:   25 (100.0 %)
Over-replicated blocks:        0 (0.0 %)
 Under-replicated blocks:       25 (100.0 %)
Mis-replicated blocks:         0 (0.0 %)
 Default replication factor:    1
Average block replication:     1.0
Corrupt blocks:                0
Missing replicas:              50 (200.0 %)
Number of data-nodes:          1
Number of racks:               1

FSCK ended at Tue Oct 16 13:23:55 PDT 2012 in 0 milliseconds

Why does it say target replica is 3 but default replication factor is clearly 1. Anyone please advice.

My versions are hadoop 1.0.3 and hbase 0.94.1

Thanks!

can you explain why you want dfs.replication set to 1? If one disk goes bad you can lose nearly everything in hbase (corrupt root / meta tables) and it can be very difficult to get back to where you can use the system again. — David
@David We set dfs.replication to 1 as this is a developer environment. In production it is set to 3. As an update to the problem hdfs always goes to safemode [ and never exits ] after process restart [ we restart after hbase creates schema, to test restarts etc ] — Paul Arer

Balaji Balaji · Accepted Answer · 2012-10-18T08:18:37

4

votes

To force the Hdfs to exit from safemode.
Type this:

hadoop dfsadmin -safemode leave

hbase hdfs goes to safemode on process restart ( probably under replication reported? )

1 Answers