I have a hive table which have about 2500 columns, when I read it by spark sql like this:
val df = spark.sql("select * from dbName.tableName")
It takes about 3 hours to finish read the hive table, while I use hive sql to read this table, it only table me about a few seconds.
Anyone who knows why the performance is so difference between spark sql and hive sql? Thanks a lot!
Assuming you are using Hive and not Impala: the Hive QL query select * from table is not even a Map only job, but an edge case, namely a "fetch task" is created rather than a MR task which just dumps the data as it is without doing anything on it. I seem to remember noting that in the past as well when I first started out. I.e. an edge case, it need not count, sum, filter, etc. and can do it this way.
val df ... on the other hand is building a dataframe and it needs to complete via shuffling, resource allocation (via YARN) etc. We know nothing of the environment as you do not state, the scheduling policies in place if Spark Standalone or YARN.
In any event select even with MR is relatively fast to return a first set of rows as per above and I suspect it has little to do with the wide columns, except of course more work, but ever so slightly different thrust to the way you pose the question.
