Generate arbitrarily nested tuples in Pig?

1
votes

I have an input relation X that looks like this:

a:chararray, b:chararray, c:(d:chararray, e:(f:chararray, g:chararray))

I'd like to generate that exact tuple again, syntactically, in a single generate statement. This doesn't work, but hopefully it gives a good example of what I'm after:

foreach x generate a as a, b as b, (d as d, (f as f, g as g) as e) as c

In reality, I'm generating code to make small edits to the type, so an answer like:

foreach x generate *

...isn't useful here.

Is there a syntax that will let me generate arbitrarily nested structures like this in a single generate statement in Pig? It's fair to assume a fully FLATTEN-ed type -- that is, arbitrarily nested tuples without bags, maps, etc.

Thank you for your help!

EDIT: I've gotten a little closer. This almost works:

x = LOAD 'data.json' USING JsonLoader('a:chararray, b:chararray') AS a:chararray, b:chararray;
y = foreach x generate (a, b, (a, b, (a, b))) as (a:chararray, b:chararray, tuple1:(a:chararray, b:chararray, tuple2:(a:chararray, b:chararray)));

Semantically, this is The Right Thing and it's syntactically valid, but it gives me this error:

Incompatable schema: left is "a:chararray,b:chararray,tuple1:tuple(a:chararray,b:chararray,tuple2:tuple(a:chararray,b:chararray))", right is "org.apache.pig.builtin.totuple_b_3:tuple(a:chararray,b:chararray,org.apache.pig.builtin.totuple_b_2:tuple(a:chararray,b:chararray,org.apache.pig.builtin.totuple_b_1:tuple(a:chararray,b:chararray)))"

...which seems strange to me, because either (a) the two schemas should match, or (b) there is an additional tuple "layer" on the right which is syntactically impossible to match.

1
hadoopapache-pig

1 Answers

2
votes

One way could be 

y = foreach x generate a,b,
    TOTUPLE(c.d,TOTUPLE(c.e.f,c.e.g)) 
         as c:(d:chararray,e:(f:chararray,g:chararray));