Apache Pig - Nested loop within specific boundaries

Question

Suppose you are iterating through a bag using the foreach construct. Would you be able to, for each element of the bag except the first one, iterate through all previous elements up to the current one?

To make it easier to visualize, consider a foreach loop that has reached the indicated position in the following bag:

(1, Element1)
(2, Element2)
(3, Element3)
(4, Element4)
(5, Element5) <-- You are here
(6, Element6)
(7, Element7)

What I would like to do at this point in the loop is run another loop through elements 1 to 4 then have the outer loop proceed to element 6 and the inner loop go through elements 1 to 5 and so on.

I highly doubt that this functionality could be achieved using some built in construct. Could this achieved by implementing a UDF which takes the current index and the bag as arguments, does the desired processing and returns the results to Pig?

Michał Tartanus Michał Tartanus · Accepted Answer · 2017-02-13T15:22:38

I'm afraid you can't do this in 'vanilla' Pig. You have to write custom UDF, here is some starting point:

public class MyUDF extends EvalFunc<Long> 
    public Long exec(Tuple input) throws IOException {
    Object values = input.get(0);
    if (values instanceof DataBag) {
        // you logic goes here
    }
    return ...
}

Apache Pig - Nested loop within specific boundaries

1 Answers