Google Cloud Bigtable/HBase Result Iterable

Question

I am testing performance using Google Cloud Bigtable with a table with 10M rows using the Java HBase API client. For the following commands the performance is relatively fast

Scan scan = new Scan()
ResultScanner scanner = table.getScanner(scan);

(1e-3, or 1e-2 second range)

However when I use the foreach loop to get the contents of the row from the iterable:

for (Result row : scanner) {
    byte[] valueBytes = row.getRow();

(The above commands are generic as I'm pretty sure I used the correct rowkey filtering methods, as well as accounting for regions/tablets)

The row.getRow()/ row.getValue() is super fast

However, traversing the ResultScanner is super slow (like > 30 seconds)

I've been reading in the docs that using setCaching() should help speed things up, however according to Google Cloud Bigtable docs setCaching() is not implemented. From the docs it reads:

org.apache.hadoop.hbase.client.Scan#setCaching(int caching) are ignored

My question is how can I increase speeds to read actual content from row objects (especially when traversing rows)?

Solomon Duskis Solomon Duskis · Accepted Answer · 2017-11-06T12:49:31

This performance is about what you can expect from a single scan on your table. FWIW, there is already caching done under the covers, and the Cloud Bigtable client ignores settings on HBase Scan object.

If you want to speed up performance, you will have to read the table in parallel scans. You can get some hints about your keyspace from RegionLocator. You can further split the range for each region by calling Bytes.split(). Once you have those ranges, you can construct parallel reads.

Google Cloud Bigtable/HBase Result Iterable

1 Answers