Rename column names when select from dataframe

Question

I have 2 dataframes : df1 and df2 and I am left joining both of them on id column and saving it to another dataframe named df3. Below is the code that I am using, which works fine as expected.

val df3 = df1.alias("tab1").join(df2.alias("tab2"),Seq("id"),"left_outer").select("tab1.*","tab2.name","tab2.dept","tab2.descr");

I would like to rename the tab2.descr column to dept_full_description within the above statement.

I am aware that I could create a seq val like below and use toDF method

val columnsRenamed = Seq("id", "empl_name", "name","dept","dept_full_description") ;
df4 = df3.toDF(columnsRenamed: _*);

Is there any other way to to aliasing in the first statement itself. My end goal is not to list about 30-40 columns explicitly .

Alper t. Turker Alper t. Turker · Accepted Answer · 2017-08-01T20:50:47

I'd rename before join:

df1.alias("tab1").join(
   df2.withColumnRenamed("descr", "dept_full_description").alias("tab2"),
   Seq("id"), "left_outer")

Rename column names when select from dataframe

1 Answers