Does all of three: Presto, hive and impala support Avro data format?

0
votes

I am clear about the Serde available in Hive to support Avro schema for data formats. Comfortable in using avro with hive.

AvroSerDe

for say, I have found this issue against presto. https://github.com/prestodb/presto/issues/5009

I need to choose components for fast execution cycle. Presto and impala provide much smaller execution cycle. So, Anyone please let me clarify that which would be better in different data formats. Primarily, I am looking for avro support with Presto now.

However, lets consider following data formats stored on HDFS:

  1. Avro format
  2. Parquet format
  3. Orc format

Which is the best to use with high performance on different data formats. ?? please suggest.

1
hadoophiveimpalapresto
Impala is a querying engine that is mostly used to query over tables that are already present in Hive. - philantrovert
my question is about what are the better ways to choose query engine and tech stack. I know impala is query engine provided from cloudera only. However, I am looking for comparative study here. - shashankS
for say, hiveQL is taking 23secs for select count(*) from table, however, presto SQL is taking 2 secs for same cluster size. So, what does impala's performance and how does the different data formats performance for same dataset, - shashankS
svds.com/dataformats - philantrovert
Thanks!! this gave little answer but still I am looking for presto insights. - shashankS

1 Answers

1
votes

  • Impala can read Avro data but can not write it. Please refer to this documentaion page describing the file formats supported by Impala.

  • Hive supports both reading and writing Avro files.

  • Presto's Hive Connector supports Avro as well. Thanks to David Phillips for pointing out this documentaion page.

There are different benchmarks on the internet about performance, but I would not like to link to a specific one as results heavily depend on the exact use case benchmarked.