delete row from table using pyspark on Notebook

Question

i'm writing pyspark script on Databricks notebook to insert/update/query cassandra tables, however I cannot find a way to delete rows from table, i tried spark sql:

spark.sql("DELETE from users_by_email where email_address IN ('[email protected]')")

I also don't see it's possible to delete data using dataframe. is there any workaround?

Can you check if this is of any help stackoverflow.com/questions/28563809/… or datastax-oss.atlassian.net/browse/SPARKC-392 — Joby
Instead of dropping that row you can just filter out that rows — Mahesh Gupta

Shadowtrooper Shadowtrooper · Accepted Answer · 2019-12-17T08:49:37

You can load the dataframe and filter it:

import pyspark.sql.functions as f

df = spark.sql("SELECT * from users_by_email")
df_filtered = df.filter(f.col("email_address") == "[email protected]")

Then you can save the dataframe with the overwrite option or, also, in a new table.

delete row from table using pyspark on Notebook

2 Answers