Pyspark: Getting current_timestamp in dynamic hive query

0
votes

I am preparing Spark with python program which inserts data from 2 tables based on joins. The last column of the target table has a timestamp field which will have the value of create timestamp.

I tried current_timestamp and from_unixtime(unix_timestamp()). Both the functions does not seem to work. I tried now().

e.g., HiveContext(sc).sql("SELECT " + from_unixtime(unix_timestamp()) + " ") This statement errors in pyspark with "NameError: name 'from_unixtime' is not defined" I have imported the pyspark.sql.function

Is there a way to insert timestamp value to the target table? My query contains insert with select from 2 tables which I am running in HiveContext.

Thank in advance!!!

1
pysparkhivecontext
I tried the below code and I am getting: TypeError: 'Column' object is not callable import pyspark.sql.functions as func HiveContext(sc).sql("SELECT " + func.from_unixtime(func.unix_timestamp()) + " "). Could someone say what is wrong with this query. Thanks!!! - Aavik

1 Answers

0
votes

Used within double quotes as below and it worked:

HiveContext(sc).sql("SELECT from_unixtime(unix_timestamp())")