Spark - How to create a map with selective fields from a flatMap

Question

I have 8 fields(field1,field2,...,field8) in a flatMap and I would like to create a map object in runtime. I would like to create a map with dynamic key and value elements.

For eg. I have key elements like field1, field2,field3 and value elements like field6,field7. The following snippet works perfectly fine.

val myMap = myFlatMap.map(line1 => line1 match {
      case (field1,field2,field3,field4,field5,field6,field7,field8) => {

        ((field1,field2,field3) -> ( field6, field7))
      }

But, the key and value elements are obtained dynamically (like command line argument). How can I do a similar map if I have the key elements in a data structure like an array of String or Seq()

Input: For example, myFlatMap has the following data : (field1_row1,field2_row1,field3_row1,field4_row1,field5_row1,field6_row1,field7_row1,field8_row1) (field1_row2,field2_row2,field3_row2,field4_row2,field5_row2,field6_row2,field7_row2,field8_row2) (field1_row3,field2_row3,field3_row3,field4_row3,field5_row3,field6_row3,field7_row3,field8_row3)

Output: A map of key,value pairs as below.

(field1_row1,field2_row1,field3_row1) -> ( field6_row1, field7_row1) (field1_row2,field2_row2,field3_row2) -> ( field6_row2, field7_row2) (field1_row3,field2_row3,field3_row3) -> ( field6_row3, field7_row3)

In the above example the key of the output map is (field1,field2,field3) and the value is (field6,field7). This is not the case for every run of the job.

In the second run, I may have to map just (field1,field2) -> field8 and ignore all other fields in the input flatMap. In the third run, I would like to create the output map with (field2,field4) -> (field7,field8)

I have the input fields in a variable, keyFields=List("field1","field3","field5"). Is there an elegant way to get the Some of only these fields?

Any help would be greatly appreciated.

ra2085 ra2085 · Accepted Answer · 2015-12-16T02:52:29

You should have a look at this question Is there way to create tuple from list(without codegeneration)?

I would follow dhg's advice though. You should really try to code your application by knowing the arity in advance.

You can also look into spark sql and dataframes. You'll be able to group by any amount keys, any amout of fields.

Spark - How to create a map with selective fields from a flatMap

1 Answers