Combinations using FOREACH in Pig

1
votes

I would like to generate combinations in pig with the help of FOREACH. Is there any possible way to do this ?

My Input:
A
B
C

Objective:
A,B
A,C
B,C

Here's the sample which I have tried. This sample shows " Syntax error, unexpected symbol at or near '$0' ".

A = load '/test';
B = foreach A generate $0;
Combination = Cross A, B;
Combination_Filter = foreach Combination generate $0 < $1;

Please help me in resolving this. Thanks in Advance

2
apache-pigcombinations

2 Answers

3
votes

Can you try the below options?

input:

A
B
C

Option1:

A = LOAD 'input' AS(f1:chararray);
B = LOAD 'input' AS(f2:chararray);
C = CROSS A,B;
D = FILTER C BY A::f1 < B::f2;
DUMP D;

Option2:

A = LOAD 'input' AS (f1:chararray);
B = FOREACH A GENERATE f1 AS (f2:chararray);
C = CROSS A,B;
D = FILTER C BY A::f1 < B::f2;
DUMP D;

Output:

(A,B)
(A,C)
(B,C)
1
votes

It is not possible to do that using only foreach, the only way to achieve something like that would be with Sivasakthi's answer, or using a custom UDF. You can put all the registers in a bag with a group all, and then run the UDF.

The UDF is in this other question: How to turn (A, B, C) into (AB, AC, BC) with Pig?

The code would be something like:

A = load '/test';
A_grouped = group A all;
A_combinations = foreach A_grouped generate CombinationsUDF(A);