spark query execution time

2
votes

I have a local hadoop single node and hive installed and I have some hive tables stored in hdfs. Then I configure Hive with MySQL Metastore. And now I installed spark and Im doing some queries over hive tables like this (in scala):

var hiveContext = new org.apache.spark.sql.hive.HiveContext(sc)
result = hiveContext.sql("SELECT * FROM USERS");
result.show

Do you know how to configure spark to show to the execution time of the query? Because for default it is not showing..

1
scalaapache-sparkapache-spark-sql
Possible duplicate of this stackoverflow.com/questions/34629313/… – charles gomes
OMG, isn't there anyone who finally doesn't suggest to visit a web browser to get the execution time?! can't believe it. This has to also be possible programmatically. – Nadjib Mami

1 Answers

1
votes

Use spark.time().

var hiveContext = new org.apache.spark.sql.hive.HiveContext(sc) result = hiveContext.sql("SELECT * FROM USERS"); spark.time(result.show)

https://db-blog.web.cern.ch/blog/luca-canali/2017-03-measuring-apache-spark-workload-metrics-performance-troubleshooting