Date Conversion - Spark SQL

0
votes

I have a column which is a date in this format;

Fri Mar 08 14:12:32 +0000 2013

And I would like to see that data in this format;

2013-03-08 14:12:32.000000

I've tried some functions for conversion such as to_utc_timestamp(timestamp, string timezone), however I got null results.

I need to use it with spark.sql("") like;

spark.sql("select TO_DATE(CAST(UNIX_TIMESTAMP('08/26/2016', 'MM/dd/yyyy') AS TIMESTAMP)) AS newdate ").show(20, False)

Sorry for my english, Thanks in advance!

1
pythonsql-serverapache-sparkapache-spark-sql

1 Answers

1
votes

You kind of already answered this, you just needed to head to the java date format options.

spark.createDataFrame(List((1,"Fri Mar 08 14:12:32 +0000 2013")))
res0.createOrReplaceTempView("test")
spark.sql("SELECT FROM_UNIXTIME(UNIX_TIMESTAMP(_2, 'EEE MMM dd HH:mm:ss Z yyyy'),'yyyy-MM-dd HH:mm:ss.SSSS') FROM test").collect

The only thing is that once it's in a time_stamp format then you use FROM_UNIXTIME to get it into the string you want.