TypeError: ArrayType(DoubleType,true) can not accept object u'..'

Question

I cannot create a DataFrame because of coordinates. This field does not fit the schema type ArrayType(DoubleType()).

my_schema = StructType(
        [
            StructField('alarm_id', StringType()),
            StructField('coordinates',ArrayType(DoubleType()))
        ])

df = spark.createDataFrame(rows, my_schema) 

I get this error:

TypeError: ArrayType(DoubleType,true) can not accept object u'[[[1.7594273000000102, 41.82814869999999], [1.7594281999999908, 41.828104700000004]]]' in type <type 'unicode'>

Is there any workaround?

Answer 1

It seems that your data is type of string.

You can use the ast lib in order to make it a list.

import ast

rows = '[[[1.7594273000000102, 41.82814869999999], [1.7594281999999908, 41.828104700000004]]]'

rows_li = ast.literal_eval(rows)

More on literal_eval