0
votes

I was trying to read a csv file using the latest built of spark (2.1.0 (Dec 28 2016) for Windows in Scala. I did notice that when reading the csv file and inferring the schema, the parsing of the file will fail. The inferschema option would fail and some of the fields that were supposed to be double are listed as strings. I could force the columns to be a specif type of data, but still parsing rows would leave null values. I believe this happened because there were commas inside the string column.

1

1 Answers

0
votes

I changed the spark built to 2.0.1 (Oct 3 2016) the infer schema worked all right. So maybe there is an error in the new spark built version for windows (2.1.0 (Dec 28 2016).