Using values of two RDD to generate a new RDD

0
votes

I have a text file with each line having 2 integer values separated by space

Suppose i have two RDDs. One with a pair of values (x,y) where x and y are 2 integer values. Another RDD (rdd2 -> (a,b)) which has just one pair with two integer values a,b.

rdd1 = textfile.map(lambda line: line.split())
...

My aim is to subtract each value in rdd1 with its corresponding value in rdd2

The result would look like 

x1-a,y1-b
x2-a,y2-b
x3-a,y3-b
...

How can i achieve this

1
pythonscalaapache-sparkpyspark

1 Answers

1
votes

If second RDD only contains one tuple, i.e. value (a:Int, b:Int) -which doesn't make sense at all-, and - means subtraction. This should do the deed:

val res = rdd2.collect()
val a = res(0)(0)
val b = res(0)(1)
val rdd3 = rdd1.map(tuple => (tuple._1 - a, tuple._2 - b) )