semi structured data processing using pig/hive

Question

I have semi structured data like below:

col1 col2 col3 col4
1    2    3    [name#aa, address#[perminentaddress#abc,currentaddress#xyg]]
5    9    8    [address#[perminentaddress#dev,currentaddress#pqr],name#bb]
3    4    9    [name#cc,mobile#111,id#66 address#[perminentaddress#abc,currentaddress#xyg]]

first three column's are fixed and the 4th column can have any unknown data with key value pairs. Key values pairs can be nested as shown in the above example. And most importantly keys position in the 4th column is not fixed and can have unlimited number of keys.

is it possible to process this data using pig/hive?

for example how to get currentaddress value from all above rows? (Please observe, keys position is not fixed and address key has nested keys)

Thanks.

Answer 1

You can represent the 4th column using nested Data Maps, see the overview at http://wiki.apache.org/pig/PigLatin

Then you will be able to access currentaddress as col4#'address'#'currentaddress'

To represent your data this way, you might need to write a custom load function.