Spark Dataframe column with last character of other column

9
votes

I'm looking for a way to get the last character from a string in a dataframe column and place it into another column.

I have a Spark dataframe that looks like this:

    animal
    ======
    cat
    mouse
    snake

I want something like this:

    lastchar
    ========
    t
    e
    e

Right now I can do this with a UDF that looks like:

    def get_last_letter(animal):
        return animal[-1]

    get_last_letter_udf = udf(get_last_letter, StringType())

    df.select(get_last_letter_udf("animal").alias("lastchar")).show()

I'm mainly curious if there's a better way to do this without a UDF. Thanks!

2
apache-sparkpysparkapache-spark-sqlpyspark-sql

2 Answers

22
votes

Just use the substring function

from pyspark.sql.functions import substring
df.withColumn("b", substring(col("columnName"), -1, 1))
4
votes

One way is by using Column substr() function:

df = df.withColumn("lastchar", df.animal.substr(-1,1))

See documentation: https://spark.apache.org/docs/2.1.0/api/python/pyspark.sql.html#pyspark.sql.Column.substr