What will happen to a Spark broadcast variable if one of the node is full

0
votes

I have just started working on spark and is a novice.

Q. Suppose we have broadcast a small table and at the same time some of the nodes are full and have no space left.

If Spark cannot copy the broadcast variable due to the memory error then how is broadcast beneficial?

How will spark handle this situation?

1
apache-sparkpysparkapache-spark-sql
Why do you want to broadcast small table? – pvy4917
So that it uses Boradcast join instead of sort merge join and hence reduce the shuffle. – Dhruv

1 Answers

-1
votes

One hit that scenario, Spark will just crash with OOM. In this, it is not really different compared to case when data is passed through closure.