Spark SQL performance is very bad

Question

I want to use SPARK SQL. I found the performance is very bad.

In my first solution: when each SQL query coming, load the data from hbase entity to a dataRDD, then register this dataRDD to SQLcontext. at last execute spark SQL query. Apparently the solution is very bad because it needs to load data every time.

So I improved the first solution.
In my second solution don't consider hbase data updates and inserts:
When app starts, load the current data from HBASE entity to a dataRDD, named cachedDataRDD.
Register cachedDataRDD to SQLcontext
When each SQL query coming, execute spark SQL query. The performance is very good.

But some entity needs to consider the updates and inserts.
So I changed the solution base on the second solution.

In my third solution need to consider the hbase data updates and inserts:
When app starts, load the current data from HBASE entity to a dataRDD, named cachedDataRDD.
When SQL query coming, load the new updates and inserts data to another dataRDD, named newDataRDD.
Then set cachedDataRDD = cachedDataRDD.union(dataRDD);
Register cachedDataRDD to SQLcontext
At last execute spark SQL query.
But I found the union transformation will cause the collect action for getting the query result is very slow. Much slow than the hbase api query.

Is there any way to tune the third solution performance?
Usually under what condition to use the spark SQL is better?
Is any good use case of using spark SQL?
Thanks

Please post the relevant code so we can see what exactly it is that you're doing. — Mike Park

user3342239 user3342239 · Accepted Answer · 2015-08-12T05:40:35

Consider creating a new table for newDataRDD and do the UNION on the Spark SQL side. So for example, instead of unioning the RDD, do:

SELECT * FROM data
UNION
SELECT * FROM newData

This should provide more information to the query optimizer and hopefully help make your query faster.

Spark SQL performance is very bad

1 Answers