Apache Drill connect from code

2
votes

I have start embedded apache drill instance following this tutorial https://drill.apache.org/docs/using-the-jdbc-driver/ I can query drill from console. But now I need to connect it from my code. 

Connection conn = null;
conn = DriverManager.getConnection("jdbc:drill:zk=localhost:31010");

When I run it I get following output in my IDE:

18:01:04.970 [main-SendThread(localhost:31010)] INFO  org.apache.zookeeper.ClientCnxn - Opening socket connection to server localhost/0:0:0:0:0:0:0:1:31010. Will not attempt to authenticate using SASL (unknown error)
18:01:04.970 [main-SendThread(localhost:31010)] INFO  org.apache.zookeeper.ClientCnxn - Socket connection established to localhost/0:0:0:0:0:0:0:1:31010, initiating session
18:01:04.971 [main-SendThread(localhost:31010)] DEBUG org.apache.zookeeper.ClientCnxn - Session establishment request sent on localhost/0:0:0:0:0:0:0:1:31010
18:01:04.972 [main-SendThread(localhost:31010)] INFO  org.apache.zookeeper.ClientCnxn - Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect
18:01:05.331 [main] DEBUG org.apache.curator.RetryLoop - Retrying operation
18:01:06.721 [main-SendThread(localhost:31010)] INFO  org.apache.zookeeper.ClientCnxn - Opening socket connection to server localhost/127.0.0.1:31010. Will not attempt to authenticate using SASL (unknown error)
18:01:06.721 [main-SendThread(localhost:31010)] INFO  org.apache.zookeeper.ClientCnxn - Socket connection established to localhost/127.0.0.1:31010, initiating session
18:01:06.721 [main-SendThread(localhost:31010)] DEBUG org.apache.zookeeper.ClientCnxn - Session establishment request sent on localhost/127.0.0.1:31010
18:01:06.722 [main-SendThread(localhost:31010)] INFO  org.apache.zookeeper.ClientCnxn - Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect
 18:01:07.495 [main-SendThread(localhost:31010)] INFO  org.apache.zookeeper.ClientCnxn - Opening socket connection to server localhost/0:0:0:0:0:0:0:1:31010. Will not attempt to authenticate using SASL (unknown error)
18:01:07.495 [main-SendThread(localhost:31010)] INFO  org.apache.zookeeper.ClientCnxn - Socket connection established to localhost/0:0:0:0:0:0:0:1:31010, initiating session
18:01:07.495 [main-SendThread(localhost:31010)] DEBUG org.apache.zookeeper.ClientCnxn - Session establishment request sent on localhost/0:0:0:0:0:0:0:1:31010
18:01:07.496 [main-SendThread(localhost:31010)] INFO  org.apache.zookeeper.ClientCnxn - Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect
18:01:09.269 [main-SendThread(localhost:31010)] INFO  org.apache.zookeeper.ClientCnxn - Opening socket connection to server localhost/127.0.0.1:31010. Will not attempt to authenticate using SASL (unknown error)
18:01:09.269 [main-SendThread(localhost:31010)] INFO  org.apache.zookeeper.ClientCnxn - Socket connection established to localhost/127.0.0.1:31010, initiating session
18:01:09.269 [main-SendThread(localhost:31010)] DEBUG org.apache.zookeeper.ClientCnxn - Session establishment request sent on localhost/127.0.0.1:31010
18:01:09.270 [main-SendThread(localhost:31010)] INFO  org.apache.zookeeper.ClientCnxn - Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect
18:01:09.392 [main-SendThread(localhost:31010)] INFO  org.apache.zookeeper.ClientCnxn - Opening socket connection to server localhost/0:0:0:0:0:0:0:1:31010. Will not attempt to authenticate using SASL (unknown error)
18:01:09.392 [main-SendThread(localhost:31010)] INFO  org.apache.zookeeper.ClientCnxn - Socket connection established to localhost/0:0:0:0:0:0:0:1:31010, initiating session
18:01:09.392 [main-SendThread(localhost:31010)] DEBUG org.apache.zookeeper.ClientCnxn - Session establishment request sent on localhost/0:0:0:0:0:0:0:1:31010
18:01:09.393 [main-SendThread(localhost:31010)] INFO  org.apache.zookeeper.ClientCnxn - Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect
18:01:10.355 [main] ERROR org.apache.curator.ConnectionState - Connection timed out for connection string (localhost:31010) and timeout (5000) / elapsed (6199)
 org.apache.curator.CuratorConnectionLossException: KeeperErrorCode = ConnectionLoss
at        org.apache.curator.ConnectionState.checkTimeouts(ConnectionState.java:198) [drill-jdbc-all-1.0.0.jar:na]
at org.apache.curator.ConnectionState.getZooKeeper(ConnectionState.java:88) [drill-jdbc-all-1.0.0.jar:na]
at org.apache.curator.CuratorZookeeperClient.getZooKeeper(CuratorZookeeperClient.java:115) [drill-jdbc-all-1.0.0.jar:na]
at org.apache.curator.utils.EnsurePath$InitialHelper$1.call(EnsurePath.java:148) [drill-jdbc-all-1.0.0.jar:na]
at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:107) [drill-jdbc-all-1.0.0.jar:na]
at org.apache.curator.utils.EnsurePath$InitialHelper.ensure(EnsurePath.java:140) [drill-jdbc-all-1.0.0.jar:na]
at org.apache.curator.utils.EnsurePath.ensure(EnsurePath.java:99) [drill-jdbc-all-1.0.0.jar:na]
at org.apache.curator.framework.imps.NamespaceImpl.fixForNamespace(NamespaceImpl.java:74) [drill-jdbc-all-1.0.0.jar:na]
at org.apache.curator.framework.imps.NamespaceImpl.newNamespaceAwareEnsurePath(NamespaceImpl.java:87) [drill-jdbc-all-1.0.0.jar:na]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.newNamespaceAwareEnsurePath(CuratorFrameworkImpl.java:468) [drill-jdbc-all-1.0.0.jar:na]
at org.apache.curator.framework.recipes.cache.PathChildrenCache.<init>(PathChildrenCache.java:223) [drill-jdbc-all-1.0.0.jar:na]
at org.apache.curator.framework.recipes.cache.PathChildrenCache.<init>(PathChildrenCache.java:182) [drill-jdbc-all-1.0.0.jar:na]
at org.apache.curator.x.discovery.details.ServiceCacheImpl.<init>(ServiceCacheImpl.java:65) [drill-jdbc-all-1.0.0.jar:na]
at org.apache.curator.x.discovery.details.ServiceCacheBuilderImpl.build(ServiceCacheBuilderImpl.java:47) [drill-jdbc-all-1.0.0.jar:na]
at org.apache.drill.exec.coord.zk.ZKClusterCoordinator.<init>(ZKClusterCoordinator.java:104) [drill-jdbc-all-1.0.0.jar:na]
at org.apache.drill.exec.client.DrillClient.connect(DrillClient.java:185) [drill-jdbc-all-1.0.0.jar:na]
at org.apache.drill.jdbc.DrillConnectionImpl.<init>(DrillConnectionImpl.java:126) [drill-jdbc-1.0.0.jar:1.0.0]
at org.apache.drill.jdbc.DrillJdbc41Factory$DrillJdbc41Connection.<init>(DrillJdbc41Factory.java:97) [drill-jdbc-1.0.0.jar:1.0.0]
at org.apache.drill.jdbc.DrillJdbc41Factory.newDrillConnection(DrillJdbc41Factory.java:60) [drill-jdbc-1.0.0.jar:1.0.0]
at org.apache.drill.jdbc.DrillJdbc41Factory.newDrillConnection(DrillJdbc41Factory.java:46) [drill-jdbc-1.0.0.jar:1.0.0]
at org.apache.drill.jdbc.DrillFactory.newConnection(DrillFactory.java:54) [drill-jdbc-1.0.0.jar:1.0.0]
at net.hydromatic.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:126) [drill-jdbc-all-1.0.0.jar:na]
at java.sql.DriverManager.getConnection(DriverManager.java:571) [na:1.7.0_67]
at java.sql.DriverManager.getConnection(DriverManager.java:233) [na:1.7.0_67]
at DrillConnectTest.testConnect(DrillConnectTest.java:34) [classes/:na]
at DrillConnectTest.main(DrillConnectTest.java:55) [classes/:na]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.7.0_67]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[na:1.7.0_67]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_67]
at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_67]
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:134) [idea_rt.jar:na]
18:01:10.355 [main] DEBUG org.apache.curator.RetryLoop - Retry-able exception received
org.apache.curator.CuratorConnectionLossException: KeeperErrorCode = ConnectionLoss
at org.apache.curator.ConnectionState.checkTimeouts(ConnectionState.java:198) ~[drill-jdbc-all-1.0.0.jar:na]
at   org.apache.curator.ConnectionState.getZooKeeper(ConnectionState.java:88) ~[drill-jdbc-all-1.0.0.jar:na]
at   org.apache.curator.CuratorZookeeperClient.getZooKeeper(CuratorZookeeperClient.java:115) ~[drill-jdbc-all-1.0.0.jar:na]
at   org.apache.curator.utils.EnsurePath$InitialHelper$1.call(EnsurePath.java:148) ~[drill-jdbc-all-1.0.0.jar:na]
at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:107) ~[drill-jdbc-all-1.0.0.jar:na]
at    org.apache.curator.utils.EnsurePath$InitialHelper.ensure(EnsurePath.java:140) [drill-jdbc-all-1.0.0.jar:na]
at org.apache.curator.utils.EnsurePath.ensure(EnsurePath.java:99) [drill-jdbc-all-1.0.0.jar:na]
at org.apache.curator.framework.imps.NamespaceImpl.fixForNamespace(NamespaceImpl.java:74) [drill-jdbc-all-1.0.0.jar:na]
at org.apache.curator.framework.imps.NamespaceImpl.newNamespaceAwareEnsurePath(NamespaceImpl.java:87) [drill-jdbc-all-1.0.0.jar:na]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.newNamespaceAwareEnsurePath(CuratorFrameworkImpl.java:468) [drill-jdbc-all-1.0.0.jar:na]
at org.apache.curator.framework.recipes.cache.PathChildrenCache.<init>(PathChildrenCache.java:223) [drill-jdbc-all-1.0.0.jar:na]
at org.apache.curator.framework.recipes.cache.PathChildrenCache.<init>(PathChildrenCache.java:182) [drill-jdbc-all-1.0.0.jar:na]
at org.apache.curator.x.discovery.details.ServiceCacheImpl.<init>(ServiceCacheImpl.java:65) [drill-jdbc-all-1.0.0.jar:na]
at org.apache.curator.x.discovery.details.ServiceCacheBuilderImpl.build(ServiceCacheBuilderImpl.java:47) [drill-jdbc-all-1.0.0.jar:na]
at org.apache.drill.exec.coord.zk.ZKClusterCoordinator.<init>(ZKClusterCoordinator.java:104) [drill-jdbc-all-1.0.0.jar:na]
at org.apache.drill.exec.client.DrillClient.connect(DrillClient.java:185) [drill-jdbc-all-1.0.0.jar:na]
at org.apache.drill.jdbc.DrillConnectionImpl.<init>(DrillConnectionImpl.java:126) [drill-jdbc-1.0.0.jar:1.0.0]
at org.apache.drill.jdbc.DrillJdbc41Factory$DrillJdbc41Connection.<init>(DrillJdbc41Factory.java:97) [drill-jdbc-1.0.0.jar:1.0.0]
at org.apache.drill.jdbc.DrillJdbc41Factory.newDrillConnection(DrillJdbc41Factory.java:60) [drill-jdbc-1.0.0.jar:1.0.0]
at org.apache.drill.jdbc.DrillJdbc41Factory.newDrillConnection(DrillJdbc41Factory.java:46) [drill-jdbc-1.0.0.jar:1.0.0]
at org.apache.drill.jdbc.DrillFactory.newConnection(DrillFactory.java:54) [drill-jdbc-1.0.0.jar:1.0.0]
at net.hydromatic.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:126) [drill-jdbc-all-1.0.0.jar:na]
at java.sql.DriverManager.getConnection(DriverManager.java:571) [na:1.7.0_67]
at java.sql.DriverManager.getConnection(DriverManager.java:233) [na:1.7.0_67]
at DrillConnectTest.testConnect(DrillConnectTest.java:34) [classes/:na]
at DrillConnectTest.main(DrillConnectTest.java:55) [classes/:na]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.7.0_67]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[na:1.7.0_67]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_67]
at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_67]
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:134) [idea_rt.jar:na]
18:01:10.860 [main] DEBUG org.apache.curator.RetryLoop - Retrying operation

And this repeats in cycle. And in server logs also I see following: 

2015-06-22 17:53:20,465 [UserServer-1] INFO  o.a.d.exec.rpc.ProtobufLengthDecoder - Channel is closed, discarding remaining 48 byte(s) in buffer.
2015-06-22 17:53:21,232 [UserServer-1] ERROR o.a.d.exec.rpc.RpcExceptionHandler - Exception in RPC communication.  Connection: /127.0.0.1:31010 <--> /127.0.0.1:55704 (user client).  Closing connection.
 io.netty.handler.codec.CorruptedFrameException: Received a message of length 0.
at   org.apache.drill.exec.rpc.ProtobufLengthDecoder.decode(ProtobufLengthDecoder.java:74) ~[drill-java-exec-1.0.0-rebuffed.jar:1.0.0]
at   org.apache.drill.exec.rpc.user.UserProtobufLengthDecoder.decode(UserProtobufLengthDecoder.java:37) ~[drill-java-exec-1.0.0-rebuffed.jar:1.0.0]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:315) ~[netty-codec-4.0.27.Final.jar:4.0.27.Final]
at     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:229) ~[netty-codec-4.0.27.Final.jar:4.0.27.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339) [netty-transport-4.0.27.Final.jar:4.0.27.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324) [netty-transport-4.0.27.Final.jar:4.0.27.Final]
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) [netty-transport-4.0.27.Final.jar:4.0.27.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339) [netty-transport-4.0.27.Final.jar:4.0.27.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324) [netty-transport-4.0.27.Final.jar:4.0.27.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:847) [netty-transport-4.0.27.Final.jar:4.0.27.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131) [netty-transport-4.0.27.Final.jar:4.0.27.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511) [netty-transport-4.0.27.Final.jar:4.0.27.Final]
at   io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) [netty-transport-4.0.27.Final.jar:4.0.27.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) [netty-transport-4.0.27.Final.jar:4.0.27.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) [netty-transport-4.0.27.Final.jar:4.0.27.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) [netty-common-4.0.27.Final.jar:4.0.27.Final]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_67]
2015-06-22 17:53:21,232 [UserServer-1] INFO  o.a.d.exec.rpc.ProtobufLengthDecoder - Channel is closed, discarding remaining 48 byte(s) in buffer.
2015-06-22 17:53:23,324 [UserServer-1] ERROR o.a.d.exec.rpc.RpcExceptionHandler - Exception in RPC communication.  Connection: /0:0:0:0:0:0:0:1:31010 <--> /0:0:0:0:0:0:0:1:55706 (user client).  Closing connection.
 io.netty.handler.codec.CorruptedFrameException: Received a message of length 0.
at org.apache.drill.exec.rpc.ProtobufLengthDecoder.decode(ProtobufLengthDecoder.java:74) ~[drill-java-exec-1.0.0-rebuffed.jar:1.0.0]
at org.apache.drill.exec.rpc.user.UserProtobufLengthDecoder.decode(UserProtobufLengthDecoder.java:37) ~[drill-java-exec-1.0.0-rebuffed.jar:1.0.0]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:315) ~[netty-codec-4.0.27.Final.jar:4.0.27.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:229) ~[netty-codec-4.0.27.Final.jar:4.0.27.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339) [netty-transport-4.0.27.Final.jar:4.0.27.Final]
at    io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324) [netty-transport-4.0.27.Final.jar:4.0.27.Final]
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) [netty-transport-4.0.27.Final.jar:4.0.27.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339) [netty-transport-4.0.27.Final.jar:4.0.27.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324) [netty-transport-4.0.27.Final.jar:4.0.27.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:847) [netty-transport-4.0.27.Final.jar:4.0.27.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131) [netty-transport-4.0.27.Final.jar:4.0.27.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511) [netty-transport-4.0.27.Final.jar:4.0.27.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) [netty-transport-4.0.27.Final.jar:4.0.27.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) [netty-transport-4.0.27.Final.jar:4.0.27.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) [netty-transport-4.0.27.Final.jar:4.0.27.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) [netty-common-4.0.27.Final.jar:4.0.27.Final]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_67]
2015-06-22 17:53:23,325 [UserServer-1] INFO  o.a.d.exec.rpc.ProtobufLengthDecoder - Channel is closed, discarding remaining 48 byte(s) in buffer.
2015-06-22 17:53:23,460 [UserServer-1] ERROR o.a.d.exec.rpc.RpcExceptionHandler - Exception in RPC communication.  Connection: /127.0.0.1:31010 <--> /127.0.0.1:55707 (user client).  Closing connection.
 io.netty.handler.codec.CorruptedFrameException: Received a message of length 0.

Any suggestion why this can happen? May be I need some extra setup of local drill?

2
apache-drill

2 Answers

1
votes

You've used zk= in your JDBC URL, but the right hand side of that portion of the URL is pointing to 31010, which is the Drillbit's "user" port, per the docs:

http://drill.apache.org/docs/ports-used-by-drill/

For embedded mode you can try either:

jdbc:drill:zk=local

or

jdbc:drill:drillbit=localhost:31010

Hope that helps.

0
votes

Other way to execute query by Calling Apache Drill REST API through code

Prerequisite:

1.Start Apache Drill 2.Add/enable Storage Plugin in http://localhost:8047/storage 3.Execute below code

public class RestAPI {

private static Log logger = LogFactory.getLog(RestAPI.class);
public static void main(String[] args) {
    getQueryResponce();
}

private static void getQueryResponce(){
    Client client = null;
    WebTarget target = null;
    try {
        logger.info("---------Started execution-----------");
        RequestQuery query = new RequestQuery();
        query.setQueryType("SQL");
        //MySQL
        //query.setQuery("SELECT * FROM MYSQL.foodmart.collections");
        //query.setQuery("SELECT * FROM MYSQL.foodmart.collections limit 100");
        //query.setQuery("SELECT payment_due_from,NumItems FROM MYSQL.foodmart.collections limit 10");
        //query.setQuery("SELECT count(SPORTS_PREFERENCE) FROM MYSQL.foodmart.collections group by SPORTS_PREFERENCE");
        //query.setQuery("SELECT DISTINCT payment_due_from FROM MYSQL.foodmart.collections ");

        //Mongo DB
        //query.setQuery("select * from mongo.apache_drill.pt_BMS_preferences_data where SPORTS_PREFERENCE = 'Cricket' limit 10");
        //query.setQuery("SELECT COUNT(SPORTS_PREFERENCE) FROM mongo.apache_drill.pt_BMS_preferences_data GROUP BY SPORTS_PREFERENCE");
        query.setQuery("SELECT DISTINCT(SPORTS_PREFERENCE) FROM mongo.apache_drill.pt_BMS_preferences_data ");
        client = ClientBuilder.newClient();
        target = client.target("http://localhost:8047/query.json");
        //target = target.path(path);
        Response response = target.request().accept(MediaType.APPLICATION_JSON)
                .post(Entity.json(query), Response.class);
        if (response.getStatus() != 200) {
            throw new RuntimeException("Failed : HTTP error code : " + response.getStatus());
        }
        String string = response.readEntity(String.class);
        logger.info(query.getQueryType()+"->"+query.getQuery());
        logger.info("Responce:\n"+string);
        logger.info("---------End execution-----------");
    } catch (Exception e) {
        e.printStackTrace();
        logger.error(e.getMessage(),e);
    }
}

Hope this help...