Spark Mllib - Frequent Pattern Mining - Association Rules - Not getting the expected results

0
votes

I've the following dataset:

[A,D]
[C,A,B]
[A]
[A,E,D]
[B,D]

And I am trying to extract some association rules using Frequent Pattern Mining using Spark Mllib. For that I've the following code:

val transactions = sc.textFile("/user/cloudera/teste")

import org.apache.spark.mllib.fpm.AssociationRules
import org.apache.spark.mllib.fpm.FPGrowth.FreqItemset

val freqItemsets = transactions.repartition(10).map(_.split(",")).flatMap(xs => 
    (xs.combinations(1) ++ xs.combinations(2) ++ xs.combinations(3) ++ xs.combinations(4) ++ xs.combinations(5)).filter(_.nonEmpty).map(x => (x.toList, 1L))   ).reduceByKey(_ + _).map{case (xs, cnt) => new FreqItemset(xs.toArray, cnt)}

val ar = new AssociationRules().setMinConfidence(0.8)

val results = ar.run(freqItemsets)

results.collect().foreach { rule =>
  println("[" + rule.antecedent.mkString(",")
    + "=>"
    + rule.consequent.mkString(",") + "]," + rule.confidence)}

But all the rules extracted have confidence equal to 1:

[[C=>A],1.0
[[C=>B]],1.0
[A,B]=>[C],1.0
[E=>D]],1.0
[E=>[A],1.0
[A=>B]],1.0
[A=>[C],1.0
[[C,A=>B]],1.0
[[A=>D]],1.0
[E,D]=>[A],1.0
[[A,E=>D]],1.0
[[C,B]=>A],1.0
[[B=>D]],1.0
[B]=>A],1.0
[B]=>[C],1.0

I really not understanding the issue that I've in my code... Anyone knows what is the error that I have to calculate the confidence?

Many thanks!

1
scaladata-miningapache-spark-mllibpattern-mining

1 Answers

0
votes

Your data set is too tiny. The maximum frequency of any item in your data is 3. So you can have confidences 0, 1/3, 1/2, 2/3, 1. Only 1 is larger than 0.8.

Try setting minimum confidence to 0.6, then you can actually get

[A]=>[D] confidence 0.666