GeoSpark show SQL results fails

Question

I am using GeoSpark 1.3.1 where I am trying to find all geo points that are contained in a POLYGON. I use the sql command:

val result = spark.sql(
    |SELECT *
    |FROM spatial_trace, streetCrossDf
    |WHERE ST_Within (streetCrossDf.geometry, spatial_trace.geometry)
    """.stripMargin)

result.show()

The query works fine but, fails when I try to show the result. Seems like an output issue from the library. I am doing this in zeppelin notebook. Can someone please tell me what I am doing wrong.? I get error below:

org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 10.0 failed 4 times, most recent failure: Lost task 0.3 in stage 10.0 (TID 15, 10.42.22.236, executor 3): java.lang.ClassCastException: org.apache.spark.unsafe.types.UTF8String cannot be cast to org.apache.spark.sql.catalyst.util.ArrayData at org.apache.spark.sql.geosparksql.strategy.join.TraitJoinQueryExec$$anonfun$toSpatialRdd$1.apply(TraitJoinQueryExec.scala:164) at org.apache.spark.sql.geosparksql.strategy.join.TraitJoinQueryExec$$anonfun$toSpatialRdd$1.apply(TraitJoinQueryExec.scala:163) at scala.collection.Iterator$$anon$11.next(Iterator.scala:410) at scala.collection.Iterator$class.foreach(Iterator.scala:891) at scala.collection.AbstractIterator.foreach(Iterator.scala:1334) at scala.collection.TraversableOnce$class.foldLeft(TraversableOnce.scala:157) at scala.collection.AbstractIterator.foldLeft(Iterator.scala:1334) at scala.collection.TraversableOnce$class.aggregate(TraversableOnce.scala:214) at scala.collection.AbstractIterator.aggregate(Iterator.scala:1334) at org.apache.spark.rdd.RDD$$anonfun$aggregate$1$$anonfun$22.apply(RDD.scala:1122) at org.apache.spark.rdd.RDD$$anonfun$aggregate$1$$anonfun$22.apply(RDD.scala:1122) at org.apache.spark.SparkContext$$anonfun$36.apply(SparkContext.scala:2157) at org.apache.spark.SparkContext$$anonfun$36.apply(SparkContext.scala:2157) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90) at org.apache.spark.scheduler.Task.run(Task.scala:121) at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:402) at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:408) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)

sidequesting sidequesting · Accepted Answer · 2020-12-16T21:11:10

I know I'm a bit late, but this is addressed by the developer here. The geometries need to be converted using a constructor

Example fix:

WHERE ST_Within (ST_GeomFromWKT(streetCrossDf.geometry), ST_GeomFromWKT(spatial_trace.geometry))

GeoSpark show SQL results fails

1 Answers