Scala Nested Map to Spark RDD

Question

I'm trying to convert a list of maps (Seq[Map[String, Map[String, String]]) into an RDD table/tuple where each key -> value pair in the map is flat mapped into a tuple with the outer map's key. For example

Map(
 1 -> Map('k' -> 'v', 'k1' -> 'v1')
)

becomes

(1, 'k', 'v')
(1, 'k1', 'v1')

I've tried the following approach, but it seems to fail on concurrency issues. I have two worker nodes, and it duplicates the key -> value twice(which I assume is because i'm doing this wrong)

Lets assume I hold my map type in a case class 'Records'

  val rdd = sc.parallelize(1 to records.length)
    val recordsIt = records.iterator
      val res: RDD[(String, String, String)] = rdd.flatMap(f => {
        val currItem = recordsIt.next()
        val x: immutable.Iterable[(String, String, String)] = currItem.mapData.map(v => {
          (currItem.identifier, v._1, v._2)
        })
        x
      }).sortBy(r => r)

Is there a way to paralleize this work without running into serious concurrency issues(as I suspect is happening?

example duplicated output

(201905_001ac172c2751c1d4f4b4cb0affb42ef_gFF0dSg4iw,CID,B13131608623827542)
(201905_001ac172c2751c1d4f4b4cb0affb42ef_gFF0dSg4iw,CID,B13131608623827542)
(201905_001ac172c2751c1d4f4b4cb0affb42ef_gFF0dSg4iw,ROD,19190321)
(201905_001ac172c2751c1d4f4b4cb0affb42ef_gFF0dSg4iw,ROD,19190321)
(201905_001b3ba44f6d1f7505a99e2288108418_mSfAfo31f8,CID,339B4C3C03DDF96AAD)
(201905_001b3ba44f6d1f7505a99e2288108418_mSfAfo31f8,CID,339B4C3C03DDF96AAD)
(201905_001b3ba44f6d1f7505a99e2288108418_mSfAfo31f8,ROD,19860115)
(201905_001b3ba44f6d1f7505a99e2288108418_mSfAfo31f8,ROD,19860115)

user11548303 user11548303 · Accepted Answer · 2019-05-24T05:17:19

Spark parallelize is very efficient from the beginning (since you already start storing data in memory it is much less expensive to just iterate over it locally), nonetheless a more idiomatic approach would be a simple flatMap:

sc.parallelize(records.toSeq)
  .flatMapValues(identity)
  .map { case (k1, (k2, v)) => (k1, k2, v) }

Scala Nested Map to Spark RDD

1 Answers