Pig: Accessing inner tuples' fields for filtering

1
votes

I have a custom Pig loader like this:

A = LOAD 'myfile' USING myudf_loader()

A contains:

((key1, val1), (key2, val2), (key3, val3), ...)

That is A has an outer tuple that contains key-value pairs stored in inner tuples.

I am not using maps because maps require that key values within a relation must be unique. The keys I have do not necessarily have to be unique.

The keys are chararrays, while the values can be chararrays, ints, and floats.

I would like to access A's inner tuples, as well as the (key, value) pairs within those tuples.

For example, I want to FILTER the keys of A such that the only fields remaining are key = "city" and value = "New York City".

Example input:

DUMP A;
(("city", "New York City"), ("city", "Boston"),
 ("city", "Washington, D.C."), ("non-city-key", "non-city-value"),
 ("city", "New York City"), ("non-city-key", "non-city-value"))

Example output of the filtering, which is stored into B:

DUMP B;
("city", "New York City")
("city", "New York City")
1
hadoopmapreduceapache-pig

1 Answers

-1
votes

I dont have your full pig latin script.

But you can achieve using the below idea

grouped_records = GROUP records By Key;

filtered_records = FILTER grouped_records By group='CITY'

Dump filtered_records

Cheers Nag