how to merge 2 different rdd in spark using scala

0
votes

I m trying to merge 2 rdds to one. If my rdd1 consists of 2 records of 2 elements both are strings ex: key_A:value_A and Key_B:value_B

rdd2 also consists of 1 record of 2 elements both of which are strings key_C:value_c

my final rdd would look like this: key_A :value_A , Key_B :value_B , key_C :value_c

we can use union method of rdd but its not working . Plz kindly help while using union of 2 rdds should the row of the 2 differnt rdd contain the same no of elments or there size can differ.......??

2
scalaapache-sparkrdd
"we can use union method of rdd but its not working" - show the code you tried, and explain what's "not working" (show expected result vs. actual result / exception) - Tzach Zohar
@Swaroop : Since you said both are String's union should work properly. - Shankar

2 Answers

2
votes

Try with join:

join(otherDataset, [numTasks])

When called on datasets of type (K, V) and (K, W), returns a dataset of (K, (V, W)) pairs with all pairs of elements for each key. Outer joins are supported through leftOuterJoin, rightOuterJoin, and fullOuterJoin.

See the associated section of the docs

0
votes

union is working.

Sample code is:

val rdd = sparkContext.parallelize(1 to 10, 3)
    val pairRDD = rdd.map { x => (x, x) }

    val rdd1 = sparkContext.parallelize(11 to 20, 3)
    val pairRDD1 = rdd1.map { x => (x, x) }

    pairRDD.union(pairRDD1).foreach(tuple => {
      println(tuple._1)
      println(tuple._2)
    })