How to flush the table in HBase

Question

Am using HBase (0.98 in Client with server as HBase 1.1.2) with underlying datastore as HDFS.

I tried to flush the table using the following code and am able to see the data flushed to HFile location in Hadoop.

            htable.put(puts);
            htable.close();
            admin.flush(tableName);

Data location in Hadoop

./hadoop fs -du /hbase/data/default/tableName/

When I do the power shut down and restarted the node, restarted the Hadoop and HBase am able to see the data got corrupted in HDFS.

If the data is properly flushed to HFile, why it is getting corrupted during power shutdown.

Do I need to make any change in the code for flushing the table?

Thanks, Har

A few questions : 1. Is this a cluster or a single node installation? 2. Do you gracefully shutdown namenode services before powering down the machine? 3. How do you know the data is corrupt? — Venkat

Michał Zaborowski Michał Zaborowski · Accepted Answer · 2017-01-21T13:38:16

I got something like that years ago, and that was due to sync problem. I can see resolution for that. Here is another description, with sequence diagram for put operation.

What happens in your case? Perhaps that put is quite small, and ends in memory storage, not in HFile, where you are looking to check if it is "corrupted", or not.

Try to write 25MB, or more - since that is page size for hadoop, and that would trigger, all writes. That way you can simply eliminate other problems. If that works - then you can play with storing policy or simply wait more. Silly advice, but note that in normal system there will be more writes, so full writes to HFile will be triggered anyway. Other option is to force it, but your production can go bad with too many writes.

How to flush the table in HBase

1 Answers