Apache spark querying columnar data

0
votes

I have been looking at Apache Spark 2.0 and trying to use Spark SQL to process some data in text files. The data is structured as follows

1
Bob
London
2014

2
Robert
Paris
2016

3
Sam
Rome
2011

How would I run SparkSQL queries on data schema such as this.

I also realize spark provides various options to read data sources.

spark.read.

csv,jdbc,load,options,parquet,table,textFile
format,json,option,orc,schema,text

Could anyone of these be used?

1
scalaapache-spark
I recommend you convert before the file to CSV format - Umberto Griffo
The files are too large 45 to 60 gigs - Lawan subba
See here: mapr.com/blog/… - Alfredo Gimenez

1 Answers

0
votes

This solved the task for me

spark.sparkContext.hadoopConfiguration.set("textinputformat.record.delimiter","\n\n")
spark.sparkContext.textFile("File.txt")