When I run:
SELECT anum as anum, COUNT(*) as callCount, SUM(dur*1) as callDuration, COUNT(distinct bnum) as voiceBnumCount, COUNT(distinct cdrref) as longCallCount
FROM hdfs.`/data/files/{171221}*` as t1 WHERE `t1`.`direction` IN ('up') AND `t1`.`anum_code` IN ('FIRSTANUM')
AND CONCAT(sdate, ' ', stime) BETWEEN '17.12.21 14.39.58' AND '17.12.21 17.40.00'
GROUP BY (anum) HAVING COUNT(*) >= 30 AND SUM(dur*1)/60 >= 150
LIMIT 10
It raise
org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR: IllegalStateException: Record count not set for this vector container Fragment 0:0 [Error Id: 1c199e3b-d680-4264-88c5-437a97516a75 on master:31010]
simple err log:
2018-11-29 20:38:42,552 [2400141c-c721-c327-9199-19ec95ea5b62:foreman] INFO o.a.drill.exec.work.foreman.Foreman - Query text for query id 2400141c-c721-c327-9199-19ec95ea5b62: SELECT anum as anum, COUNT(*) as callCount, SUM(dur*1) as callDuration, COUNT(distinct bnum) as voiceBnumCount, COUNT(distinct cdrref) as longCallCount
FROM hdfs.`/data/files/{171221}*` as t1 WHERE `t1`.`direction` IN ('moc') AND `t1`.`anum_code` IN ('62_XL')
AND CONCAT(sdate, ' ', stime) BETWEEN '17.12.21 14.39.58' AND '17.12.21 17.40.00'
GROUP BY (anum) HAVING COUNT(*) >= 30 AND SUM(dur*1)/60 >= 150
LIMIT 10
2018-11-29 20:38:42,665 [2400141c-c721-c327-9199-19ec95ea5b62:frag:4:1] INFO o.a.d.e.w.fragment.FragmentExecutor - 2400141c-c721-c327-9199-19ec95ea5b62:4:1: State change requested AWAITING_ALLOCATION --> RUNNING
...
2018-11-29 20:38:42,759 [2400141c-c721-c327-9199-19ec95ea5b62:frag:5:3] INFO o.a.d.e.w.f.FragmentStatusReporter - 2400141c-c721-c327-9199-19ec95ea5b62:5:3: State to report: FINISHED
2018-11-29 20:38:42,759 [2400141c-c721-c327-9199-19ec95ea5b62:frag:0:0] INFO o.a.d.e.p.impl.join.HashJoinBatch - Memory limit 40 MB bytes
2018-11-29 20:38:42,759 [2400141c-c721-c327-9199-19ec95ea5b62:frag:0:0] INFO o.a.d.e.p.impl.join.HashJoinBatch - Memory limit 40 MB bytes
2018-11-29 20:38:42,759 [2400141c-c721-c327-9199-19ec95ea5b62:frag:0:0] INFO o.a.d.e.w.fragment.FragmentExecutor - 2400141c-c721-c327-9199-19ec95ea5b62:0:0: State change requested AWAITING_ALLOCATION --> RUNNING
2018-11-29 20:38:42,759 [2400141c-c721-c327-9199-19ec95ea5b62:frag:0:0] INFO o.a.d.e.w.f.FragmentStatusReporter - 2400141c-c721-c327-9199-19ec95ea5b62:0:0: State to report: RUNNING
...
2018-11-29 20:38:44,218 [2400141c-c721-c327-9199-19ec95ea5b62:frag:2:0] INFO o.a.d.e.w.f.FragmentStatusReporter - 2400141c-c721-c327-9199-19ec95ea5b62:2:0: State to report: FINISHED
2018-11-29 20:38:44,232 [2400141c-c721-c327-9199-19ec95ea5b62:frag:0:0] INFO o.a.d.e.p.i.u.UnorderedReceiverBatch - Informing senders of request to terminate sending.
2018-11-29 20:38:44,232 [2400141c-c721-c327-9199-19ec95ea5b62:frag:0:0] WARN o.a.d.e.w.b.ControlMessageHandler - Dropping request for early fragment termination for path 2400141c-c721-c327-9199-19ec95ea5b62:1:0 -> 2400141c-c721-c327-9199-19ec95ea5b62:0:0 as path to executor unavailable.
2018-11-29 20:38:44,406 [2400141c-c721-c327-9199-19ec95ea5b62:frag:6:1] INFO o.a.d.e.w.fragment.FragmentExecutor - 2400141c-c721-c327-9199-19ec95ea5b62:6:1: State change requested RUNNING --> FINISHED
2018-11-29 20:38:44,406 [2400141c-c721-c327-9199-19ec95ea5b62:frag:6:1] INFO o.a.d.e.w.f.FragmentStatusReporter - 2400141c-c721-c327-9199-19ec95ea5b62:6:1: State to report: FINISHED
...
2018-11-29 20:38:44,963 [2400141c-c721-c327-9199-19ec95ea5b62:frag:0:0] INFO o.a.d.e.w.fragment.FragmentExecutor - 2400141c-c721-c327-9199-19ec95ea5b62:0:0: State change requested RUNNING --> FAILED
2018-11-29 20:38:44,965 [2400141c-c721-c327-9199-19ec95ea5b62:frag:0:0] INFO o.a.d.e.w.fragment.FragmentExecutor - 2400141c-c721-c327-9199-19ec95ea5b62:0:0: State change requested FAILED --> FINISHED
2018-11-29 20:38:44,965 [2400141c-c721-c327-9199-19ec95ea5b62:frag:0:0] ERROR o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: IllegalStateException: Record count not set for this vector container
Fragment 0:0
[Error Id: 1c199e3b-d680-4264-88c5-437a97516a75 on master:31010]
org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: IllegalStateException: Record count not set for this vector container
Fragment 0:0
[Error Id: 1c199e3b-d680-4264-88c5-437a97516a75 on master:31010]
at org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:633) ~[drill-common-1.14.0.jar:1.14.0]
at org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:361) [drill-java-exec-1.14.0.jar:1.14.0]
at org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:216) [drill-java-exec-1.14.0.jar:1.14.0]
at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:327) [drill-java-exec-1.14.0.jar:1.14.0]
at org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38) [drill-common-1.14.0.jar:1.14.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_181]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_181]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_181]
Caused by: java.lang.IllegalStateException: Record count not set for this vector container
at com.google.common.base.Preconditions.checkState(Preconditions.java:173) ~[guava-18.0.jar:na]
at org.apache.drill.exec.record.VectorContainer.getRecordCount(VectorContainer.java:394) ~[drill-java-exec-1.14.0.jar:1.14.0]
at org.apache.drill.exec.record.RecordBatchSizer.<init>(RecordBatchSizer.java:714) ~[drill-java-exec-1.14.0.jar:1.14.0]
at org.apache.drill.exec.record.RecordBatchSizer.<init>(RecordBatchSizer.java:698) ~[drill-java-exec-1.14.0.jar:1.14.0]
at org.apache.drill.exec.physical.impl.common.HashTableTemplate$BatchHolder.getActualSize(HashTableTemplate.java:442) ~[drill-java-exec-1.14.0.jar:1.14.0]
at org.apache.drill.exec.physical.impl.common.HashTableTemplate.getActualSize(HashTableTemplate.java:901) ~[drill-java-exec-1.14.0.jar:1.14.0]
at org.apache.drill.exec.physical.impl.common.HashTableTemplate.makeDebugString(HashTableTemplate.java:910) ~[drill-java-exec-1.14.0.jar:1.14.0]
at org.apache.drill.exec.physical.impl.common.HashPartition.makeDebugString(HashPartition.java:580) ~[drill-java-exec-1.14.0.jar:1.14.0]
at org.apache.drill.exec.physical.impl.join.HashJoinBatch.makeDebugString(HashJoinBatch.java:1017) ~[drill-java-exec-1.14.0.jar:1.14.0]
at org.apache.drill.exec.physical.impl.join.HashJoinBatch.executeBuildPhase(HashJoinBatch.java:829) ~[drill-java-exec-1.14.0.jar:1.14.0]
at org.apache.drill.exec.physical.impl.join.HashJoinBatch.innerNext(HashJoinBatch.java:415) ~[drill-java-exec-1.14.0.jar:1.14.0]
at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:172) ~[drill-java-exec-1.14.0.jar:1.14.0]
at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119) ~[drill-java-exec-1.14.0.jar:1.14.0]
at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109) ~[drill-java-exec-1.14.0.jar:1.14.0]
at org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext(AbstractUnaryRecordBatch.java:63) ~[drill-java-exec-1.14.0.jar:1.14.0]
at org.apache.drill.exec.physical.impl.limit.LimitRecordBatch.innerNext(LimitRecordBatch.java:87) ~[drill-java-exec-1.14.0.jar:1.14.0]
at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:172) ~[drill-java-exec-1.14.0.jar:1.14.0]
at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119) ~[drill-java-exec-1.14.0.jar:1.14.0]
at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109) ~[drill-java-exec-1.14.0.jar:1.14.0]
at org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext(AbstractUnaryRecordBatch.java:63) ~[drill-java-exec-1.14.0.jar:1.14.0]
at org.apache.drill.exec.physical.impl.limit.LimitRecordBatch.innerNext(LimitRecordBatch.java:87) ~[drill-java-exec-1.14.0.jar:1.14.0]
at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:172) ~[drill-java-exec-1.14.0.jar:1.14.0]
at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119) ~[drill-java-exec-1.14.0.jar:1.14.0]
at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109) ~[drill-java-exec-1.14.0.jar:1.14.0]
at org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext(AbstractUnaryRecordBatch.java:63) ~[drill-java-exec-1.14.0.jar:1.14.0]
at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:172) ~[drill-java-exec-1.14.0.jar:1.14.0]
at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119) ~[drill-java-exec-1.14.0.jar:1.14.0]
at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109) ~[drill-java-exec-1.14.0.jar:1.14.0]
at org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext(AbstractUnaryRecordBatch.java:63) ~[drill-java-exec-1.14.0.jar:1.14.0]
at org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:142) ~[drill-java-exec-1.14.0.jar:1.14.0]
at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:172) ~[drill-java-exec-1.14.0.jar:1.14.0]
at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119) ~[drill-java-exec-1.14.0.jar:1.14.0]
at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109) ~[drill-java-exec-1.14.0.jar:1.14.0]
at org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext(AbstractUnaryRecordBatch.java:63) ~[drill-java-exec-1.14.0.jar:1.14.0]
at org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:142) ~[drill-java-exec-1.14.0.jar:1.14.0]
at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:172) ~[drill-java-exec-1.14.0.jar:1.14.0]
at org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:103) ~[drill-java-exec-1.14.0.jar:1.14.0]
at org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.innerNext(ScreenCreator.java:83) ~[drill-java-exec-1.14.0.jar:1.14.0]
at org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:93) ~[drill-java-exec-1.14.0.jar:1.14.0]
at org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:294) ~[drill-java-exec-1.14.0.jar:1.14.0]
at org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:281) ~[drill-java-exec-1.14.0.jar:1.14.0]
at java.security.AccessController.doPrivileged(Native Method) ~[na:1.8.0_181]
at javax.security.auth.Subject.doAs(Subject.java:422) ~[na:1.8.0_181]
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) ~[hadoop-common-2.7.1.jar:na]
at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:281) [drill-java-exec-1.14.0.jar:1.14.0]
... 4 common frames omitted
2018-11-29 20:38:44,969 [qtp717308094-635] ERROR o.a.d.e.server.rest.QueryResources - Query from Web UI Failed
org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR: IllegalStateException: Record count not set for this vector container
Fragment 0:0
[Error Id: 1c199e3b-d680-4264-88c5-437a97516a75 on master:31010]
at org.apache.drill.exec.server.rest.QueryWrapper.run(QueryWrapper.java:115) ~[drill-java-exec-1.14.0.jar:1.14.0]
at org.apache.drill.exec.server.rest.QueryResources.submitQueryJSON(QueryResources.java:72) ~[drill-java-exec-1.14.0.jar:1.14.0]
at org.apache.drill.exec.server.rest.QueryResources.submitQuery(QueryResources.java:87) ~[drill-java-exec-1.14.0.jar:1.14.0]
at sun.reflect.GeneratedMethodAccessor189.invoke(Unknown Source) ~[na:na]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_181]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_181]
at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81) [jersey-server-2.8.jar:na]
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:151) [jersey-server-2.8.jar:na]
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:171) [jersey-server-2.8.jar:na]
at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:195) [jersey-server-2.8.jar:na]
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:104) [jersey-server-2.8.jar:na]
at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:387) [jersey-server-2.8.jar:na]
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:331) [jersey-server-2.8.jar:na]
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:103) [jersey-server-2.8.jar:na]
at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:269) [jersey-server-2.8.jar:na]
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) [jersey-common-2.8.jar:na]
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) [jersey-common-2.8.jar:na]
at org.glassfish.jersey.internal.Errors.process(Errors.java:315) [jersey-common-2.8.jar:na]
at org.glassfish.jersey.internal.Errors.process(Errors.java:297) [jersey-common-2.8.jar:na]
at org.glassfish.jersey.internal.Errors.process(Errors.java:267) [jersey-common-2.8.jar:na]
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:297) [jersey-common-2.8.jar:na]
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:252) [jersey-server-2.8.jar:na]
at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1023) [jersey-server-2.8.jar:na]
at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:372) [jersey-container-servlet-core-2.8.jar:na]
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:382) [jersey-container-servlet-core-2.8.jar:na]
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:345) [jersey-container-servlet-core-2.8.jar:na]
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:220) [jersey-container-servlet-core-2.8.jar:na]
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:738) [jetty-servlet-9.1.5.v20140505.jar:9.1.5.v20140505]
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:551) [jetty-servlet-9.1.5.v20140505.jar:9.1.5.v20140505]
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:219) [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1111) [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:478) [jetty-servlet-9.1.5.v20140505.jar:9.1.5.v20140505]
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:183) [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1045) [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
at org.eclipse.jetty.server.Server.handle(Server.java:462) [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:279) [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:232) [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:534) [jetty-io-9.1.5.v20140505.jar:9.1.5.v20140505]
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607) [jetty-util-9.1.5.v20140505.jar:9.1.5.v20140505]
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536) [jetty-util-9.1.5.v20140505.jar:9.1.5.v20140505]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_181]
2018-11-29 20:38:44,973 [2400141c-c721-c327-9199-19ec95ea5b62:frag:0:0] WARN o.a.d.e.w.f.QueryStateProcessor - Dropping request to move to COMPLETED state as query is already at FAILED state (which is terminal).
full err log:
https://pastebin.com/1AGTVSjx
And that's only in development machine. Which I assume because memory configuration problem only (?).
In production mode, with same query the error is (cannot provide more, not have access):
org.apache.drill.common.exceptions.UserRemoteException: RESOURCE ERROR: One or more nodes ran out of memory while executing the query. Unable to allocate buffer of size 8192 due to memory limit (41943040). Current allocation: 41943040 Fragment 3:0 [Error Id: 45ab7e51-b563-4fff-9908-33208b895ee6 on MASTER:31010]
if I do something like this (dividing the column into join) it works:
SELECT t1.anum as anum, COUNT(*) as callCount, SUM(dur*1) as callDuration, COUNT(distinct bnum) as voiceBnumCount, t2.longCallCount
FROM hdfs4.`/cdr2p2/{181129}*` as t1
JOIN(
SELECT anum as anum, COUNT(distinct cdrref) as longCallCount
FROM hdfs4.`/cdr2p2/{181129}*` as t1 WHERE `t1`.`direction` IN ('UP') AND `t1`.`anum_code` IN ('FIRSTANUM')
AND CONCAT(sdate, ' ', stime) BETWEEN '18.11.29 14.39.58' AND '18.11.29 17.40.00'
GROUP BY (anum) HAVING COUNT(*) >= 30 AND SUM(dur*1)/60 >= 150
) as t2 ON t2.anum = t1.anum
WHERE `t1`.`direction` IN ('UP') AND `t1`.`anum_code` IN ('FIRSTANUM')
AND CONCAT(sdate, ' ', stime) BETWEEN '18.11.29 14.39.58' AND '18.11.29 17.40.00'
GROUP BY (t1.anum, t2.longCallCount) HAVING COUNT(*) >= 30 AND SUM(dur*1)/60 >= 150
LIMIT 10
What cause this? Is because too many aggregation while only 1 grouping? How do I solve it?
