Apache Drill cannot read partitioned parquet files

0
votes

I have created on azure blob storage a parquet file structure with Apache Spark on HD Insight. This is the structure:

 /root
      /sitename=www.site1.com
        /datekey=20160101
          log-01-file.parquet
      /sitename=www.site2.com
        /datekey=29160192

We want to use Apache Drill in order to run queries againts this parquet structure but we found a few issues.

When running this query

SELECT datekey FROM azure.root.`./root` WHERE sitename='www.mysite.com' GROUP BY datekey

We get this error

"org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR: NumberFormatException: www.trovaprezzi.it Fragment 2:2"

What could be the cause of the error?

Again, when running queries without WHERE clause seems that partitions key are seen as null values.

SELECT sitename, COUNT(*) as N FROM azure.root.`./root` GROUP BY sitename

|sitename|N
|NULL    |100000|

Does anyone experimented this issue? Any help will be really appreciated.

Thanks Rob

2
apache-sparkazure-hdinsightapache-drill

2 Answers

0
votes

HDInsight does not support Drill today. Hive (on Tez) should also be able to leverage the Parquet format, maybe you can try that instead?

0
votes

At the moment of writing this post drill 1.6 seems work in this way.

Whatever partition scheme you will use DRILL will cal you parition directories structure using: dir0, dir1, etc.etc..

For instance, if we partitioned our data by hostname and date we obtain

|dir0|dir1|... |host1|20160101|.... |host2|20160101|....