I have a very large hbase/phoenix table that I am trying to query using sqlline. I am getting the following exception. I tried changing various settings both on client and server:
phoenix.query.keepAliveMs
phoenix.query.timeoutMs
dfs.client.socket-timeout
I tried setting all the above to 6000000 but still getting the same issue. Any ideas? It seems the original exception is the HBaseClient.CallTimeoutException.
sqlline version 1.1.8 0: jdbc:phoenix:nn.lan> select count(*) from OUTPUT_TEST_TABLE;
+------------------------------------------+
| COUNT(1) |
+------------------------------------------+
java.lang.RuntimeException: org.apache.phoenix.exception.PhoenixIOException: org.apache.phoenix.exception.PhoenixIOException: Failed after attempts=36, exceptions:
Tue Sep 22 09:22:40 BST 2015, null, java.net.SocketTimeoutException: callTimeout=60000, callDuration=60302: row '' on table 'OUTPUT_TEST_TABLE' at region=OUTPUT_TEST_TABLE,,1442867748366.5e70471615adc084cd0a60f943a654b0., hostname=d2.lan,16020,1442909697689, seqNum=194461
at sqlline.IncrementalRows.hasNext(IncrementalRows.java:73)
at sqlline.TableOutputFormat.print(TableOutputFormat.java:33)
at sqlline.SqlLine.print(SqlLine.java:1653)
at sqlline.Commands.execute(Commands.java:833)
at sqlline.Commands.sql(Commands.java:732)
at sqlline.SqlLine.dispatch(SqlLine.java:808)
at sqlline.SqlLine.begin(SqlLine.java:681)
at sqlline.SqlLine.start(SqlLine.java:398)
at sqlline.SqlLine.main(SqlLine.java:292)
0: jdbc:phoenix:nn.lan> 15/09/22 09:22:44 WARN client.ScannerCallable: Ignore, probably already closed
java.io.IOException: Call to d2.lan/192.168.0.30:16020 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=100, waitTime=60001, operationTimeout=60000 expired.
at org.apache.hadoop.hbase.ipc.RpcClientImpl.wrapException(RpcClientImpl.java:1232)
at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1200)
at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:213)
at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:287)
at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.scan(ClientProtos.java:32651)
at org.apache.hadoop.hbase.client.ScannerCallable.close(ScannerCallable.java:355)
at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:195)
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:142)
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:61)
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:320)
at org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:258)
at org.apache.hadoop.hbase.client.ClientScanner.possiblyNextScanner(ClientScanner.java:241)
at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:532)
at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:364)
at org.apache.phoenix.iterate.ScanningResultIterator.next(ScanningResultIterator.java:55)
at org.apache.phoenix.iterate.TableResultIterator.next(TableResultIterator.java:107)
at org.apache.phoenix.iterate.SpoolingResultIterator.<init>(SpoolingResultIterator.java:125)
at org.apache.phoenix.iterate.SpoolingResultIterator.<init>(SpoolingResultIterator.java:83)
at org.apache.phoenix.iterate.SpoolingResultIterator.<init>(SpoolingResultIterator.java:62)
at org.apache.phoenix.iterate.SpoolingResultIterator$SpoolingResultIteratorFactory.newIterator(SpoolingResultIterator.java:78)
at org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:109)
at org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:100)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.apache.phoenix.job.JobManager$InstrumentedJobFutureTask.run(JobManager.java:183)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=100, waitTime=60001, operationTimeout=60000 expired.
at org.apache.hadoop.hbase.ipc.Call.checkAndSetTimeout(Call.java:70)
at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1174)
... 26 more
15
