Spark SQL join on two RDDs created using java

Question

I have created two RDDs and persisted them using java. I have submitted the jar to the spark master. Now when the spark shell prompts me to enter the query, I have given an SQL statement join condition on the two RDDs; then it throws an exception while performing collect() on the result set.

JOIN Condition : select a.ID from TABLE_1 as a JOIN TABLE_2 b ON a.NAME = b.NAME;

Exception : Null pointer Exception on resultSet.collect()

PS: I have checked if the result set is null before resultSet.collect(); but still it enters the condition and throws an NPE. I am using spark-sql-1.1.1 jar(latest).

Is it some issue with JOIN in SPARK SQL?

Answer 1

It looks like the Java spark API has issues with certain JOIN operations.It worked when I tried the same with scala