Pyspark - fpgrowth - association rules - StackOverflow error

1
votes

I have a huge dataframe (5 million rows) , each row is a basket of items, and I am trying to get the frequent item sets and association rules. But it's giving me StackOverflowErrors, I tried setting the checkpoint directory but it didn't solve the issue. Any idea how to solve this? Thanks a lot in advance

fpGrowth = FPGrowth(itemsCol="ARFeatures", minSupport=0.8, minConfidence=0.9)

model = fpGrowth.fit(completeDf)

java.lang.StackOverflowError at java.lang.reflect.InvocationTargetException.(InvocationTargetException.java:72) at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:1028) at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1496) at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432) at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178) at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548) at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509) at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432) at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178) at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348) at scala.collection.mutable.HashMap$$anonfun$writeObject$1.apply(HashMap.scala:138) at scala.collection.mutable.HashMap$$anonfun$writeObject$1.apply(HashMap.scala:136) at scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:230) at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40) at scala.collection.mutable.HashTable$class.serializeTo(HashTable.scala:125) at scala.collection.mutable.HashMap.serializeTo(HashMap.scala:40) at scala.collection.mutable.HashMap.writeObject(HashMap.scala:136) at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:1028) at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1496) at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432) at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178) at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548) at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509) at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432) at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178) at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348) at scala.collection.mutable.HashMap$$anonfun$writeObject$1.apply(HashMap.scala:138)

1
apache-sparkpysparkstack-overflowpyspark-sql
I don't know the internals of FPGrowth but have you tried increasing the stack size? (JVM option -Xss) – Traian
Is this one of the command line arguments for spark-submit? – soulless

1 Answers

0
votes

Increase the driver stack size. It depends on how you execute the application, you need to correctly pass the driver JVM option.

For spark-submit you can add it as a cmd line arg:

--conf "spark.driver.extraJavaOptions=-Xss10m"

Take a look at these for more details: