0
votes

It seems that group is not supported in nested FOREACH statements. I have the following schema:

data2: {group: chararray,data1: {(lt: chararray,ln: chararray)}}

on which I want to flatten data1, group all pairs of (lt, ln), count, order DESC, and finally limit 1.

The idea is to extract the most probable pair of (lt, ln) for each group. How would you recommend me to do that?

1

1 Answers

1
votes

For fastest execution a UDF would be best. In pure Apache Pig it would be nice if you could do count on (group, lt, ln) before creating a relation with your schema. It woul be something like this (it is just a pseudo script, might need some debugging)

Assuming load schema is (id, lt, ln)

inpt = LOAD ....... as (id, lt : chararray, ln : chararray);
grp1 = GROUP inpt BY (id, lt, ln);
data1 = FOREACH grp FLATTEN(inpt), COUNT(data1) as cnt;
data2 = GROUP data_wtih_count BY id;
--data2: {group: chararray,data1: {(id, lt: chararray,ln: chararray, cnt : int)}}
most_probable_pair = FOREACH data2 {
  ord = ORDER data1 BY cnt ASC;
  top = LIMIT ord 1;
  GENERATE group, top.(ln, lt);
}

Or you could flatten data2 and data1 and start with grp1.