Pig JsonLoader() bug?

1
votes

I am trying to load a json file with pig. I can load the file succesfully, but i found a bug.

schema:
id,name,brand,color

data:

{"id":2561,"name":"abc","brand":"Levis","color":"Blue"}
{"id":2562,"brand":"Adidas","color":"Black"}
{"id":2563,"name":"edf","brand":"Nike","color":"White"}

code:

raw = LOAD '$INPUT_PATH' USING JsonLoader('
id:chararray,
name:chararray,
brand:chararray,
color:chararray
');

x = foreach raw generate id,brand;
dump x;

And result is wrong if the particular raw does no contains all the fields defined in the schema: (the second raw should be Adidas instead of black)

(2561,Levis)
(2562,Black)
(2563,Nike)

Is there any workaround for the above?

thanks in advance

1
jsonapache-pig

1 Answers

2
votes

I suggest you to use elephantbird instead of JsonLoader. Elephantbird will store the input json in the form of key/value pair(i.e map) and it will be easy to extract the required fields even-though some fields are missing in the input json.

Download the two jars files(elephant-bird-pig-4.1.jar and elephant-bird-hadoop-compat-4.1.jar) and try the below approach.

input.json

{"id":2561,"name":"abc","brand":"Levis","color":"Blue"}
{"id":2562,"brand":"Adidas","color":"Black"}
{"id":2563,"name":"edf","brand":"Nike","color":"White"}

PigScript:

REGISTER /tmp/elephant-bird-pig-4.1.jar;
REGISTER /tmp/elephant-bird-hadoop-compat-4.1.jar;

A = LOAD 'input.json' USING com.twitter.elephantbird.pig.load.JsonLoader() AS myMap;
B = FOREACH A GENERATE myMap#'id' AS ID,myMap#'brand' AS brand;
DUMP B;

Output:

(2561,Levis)
(2562,Adidas)
(2563,Nike)