Convert string to date using pyspark

Question

I have a pyspark dataframe with a string column in the format of YYYYMMDD and I am attempting to convert this into a date column (I should have a final date ISO 8061). The field is named deadline and is formatted as follows:

deadline
20190530

I tried the following solutions:

from pyspark.sql.functions import unix_timestamp, col
from pyspark.sql.types import TimestampType
from pyspark.sql.types import StringType
from pyspark.sql.functions import from_unixtime
from pyspark.sql.types import DateType

df.select(to_date(df.deadline).alias('dt')).show()

df.withColumn('new_date',to_date(unix_timestamp(df.deadline, 'YYYYMMDD').cast('timestamp'))).show()

orders_concat.select(unix_timestamp(orders_concat.deadline, 'YYYYMMDD')).show()

df.select(unix_timestamp(df.ts_string, 'yyyy/MM/dd HH:mm:ss').cast(TimestampType()).alias("timestamp")).show()


df.select(unix_timestamp(df.deadline, 'yyyy/MM/dd HH:mm:ss').cast(TimestampType()).alias("timestamp")).show()

df.select(to_date(cast(unix_timestamp('deadline', 'YYYYMMDD').alias('timestamp').show()

 ndf = df.withColumn('_1', df['deadline'].cast(DateType()))

 df2 = df.select('deadline', from_unixtime(unix_timestamp('deadline', 'YYYYMMDD')).alias('date'))

I always get null values.

Does anyone has suggestions?

Possible duplicate of Convert date from String to Date format in Dataframes — Steven
@Steven I tried the solution posted in that post but none worked. I will edit the code because I did not notice I posted the old version (in the new one I tried the correct data format) — Greenfox

Steven Steven · Accepted Answer · 2019-06-13T09:57:18

with the right format yyyyMMdd, it works fine :

from pyspark.sql import functions as F

df.withColumn('new_date',F.to_date(F.unix_timestamp(df.deadline, 'yyyyMMdd').cast('timestamp'))).show() 

    +--------+----------+                                                           
    |deadline|  new_date|
    +--------+----------+
    |20190530|2019-05-30|
    +--------+----------+

Convert string to date using pyspark

1 Answers