My objective is to join two tables, where the second table is normal and the first one is nested structure table. The join key is available inside the nested structure in first table. In this case, how to Join these two tables using dataflow java code. WithKeys (org.apache.beam.sdk.transforms.WithKeys) accepting direct column name and it does not allow like firstTable.columnname. Could some one to help to solve this case.
0
votes
Is there a reason to have this nested structure in the output table? Can we just flatten the output to be: Cusname, Custno, headerid, cr.date, lineid, item desc? It will make it much easier tbh ..
- RomanM
I have answered same problem here which is in python with apache Beam, might be helpful to you. in ParDo addkeysnested function, i am getting key values from nested field. by using this code you don't need to unnest the fields. so we don't require to change the schema structure of table.
- Dhara Dhruve
1 Answers
1
votes
If both tables are equally large consider using the CoGroupByKey transform described here. You will have to transform your data into two PCollections keyed by the proper key before this operation.
If one table is significantly smaller than the other, feeding the smaller PCollection as a side input to a ParDo over the larger PCollection as described here might be a better option.
