Loading protobuf format file into pig script using loadfunc pig UDF

4
votes

I have very little knowledge of pig. I have protobuf format data file. I need to load this file into a pig script. I need to write a LoadFunc UDF to load it. say function is Protobufloader().

my PIG script would be 

A = LOAD 'abc_protobuf.dat' USING Protobufloader() as (name, phonenumber, email);

All i wish to know is How do i get the file input stream. Once i get hold of file input stream, i can parse the data from protobuf format to PIG tuple format.

PS: thanks in advance

1
hadoopapache-pighadoop-pluginsprotocol-buffers

1 Answers

6
votes

Twitter's open source library elephant bird has many such loaders: https://github.com/kevinweil/elephant-bird

You can use LzoProtobufB64LinePigLoader and LzoProtobufBlockPigLoader. https://github.com/kevinweil/elephant-bird/tree/master/src/java/com/twitter/elephantbird/pig/load

To use it, you just need to do:

define ProtoLoader com.twitter.elephantbird.pig.load.LzoProtobufB64LineLoader('your.proto.class.name');
a = load '/your/file' using ProtoLoader;
b = foreach a generate
  field1, field2;

After loading, it will be automatically translated to pig tuples with proper schema.

However, they assume you write your data in serialized protobuffer and compressed by lzo.

They have corresponding writers as well, in package com.twitter.elephantbird.pig.store. If your data format is a bit different, you can adapt their code to your custom loader.