pig - parsing string with regex

3
votes

I'm stuck on string parsing in Pig.

I have looked at the documentation around regex_extract and regex_extract_all and hoped to use one of those functions.

I have file '/logs/test.log':

cat '/logs/test.log'
user=242562&friend=6226&friend=93856&age=35&friend=35900

I want to extract the friend tags from the url, and in this case, I have 3 identical tags. regex_extract seems to only work for the first instance, which is what I expected, and for regex_extract_all, it seems like I have know the whole string pattern, which changes on each row of the source file.

It looked ok with regex_extract, but this option only gives me the first one.

 [root@test]# pig -x local
 A = LOAD './test.log';
 B = FOREACH A GENERATE REGEX_EXTRACT($0, 'friend=([0-9]*)',1);
 dump B;
 (6226)

The examples I see for regex_extract_all show regex where you seek out all the tags:

  B = FOREACH A GENERATE FLATTEN(REGEX_EXTRACT_ALL($0, 'user=([0-9]+?)&friend=([0-9]+?)&friend=([0-9]+?)&.+?'));
 dump B;
 (242562,6226,93856)

That seems to work, but I really just want to extract the friends - (6226,93856,35900). I also have cases where there might be more-than or less-than 3 friends per user.

Any ideas?

Also looking at using something like FLATTEN(TOKENIZE($0,'&')) and then somehow only filtering on the SUBSTRING($0,0,INDEXOF($0,'=')) == 'friend' or something like that, but wanted to see if anyone knew a good regex approach.

2
regexapache-pig
Kinda gave up on this and just used STREAM to a quick script I put together. Didn't want to have to maintain yet another script, but much, much quicker solution. Regret spending so much time trying to get it to work natively - jeff

2 Answers

2
votes

This can be achieved by simple string manipulations:

inputs = LOAD 'input' AS (line: chararray);
tokenized = FOREACH inputs GENERATE FLATTEN(TOKENIZE(line, '&')) AS parameter;
filtered = FILTER tokenized BY INDEXOF(parameter, 'friend=') != -1;
result = FOREACH filtered GENERATE SUBSTRING(parameter, 7, (int)SIZE(parameter)) AS   friend_number;
DESCRIBE tokenized;
DUMP tokenized;
DESCRIBE filtered;
DUMP filtered;
DESCRIBE result;
DUMP result;

result:

tokenized: {parameter: chararray}
(user=242562)
(friend=6226)
(friend=93856)
(age=35)
(friend=35900)
filtered: {parameter: chararray}
(friend=6226)
(friend=93856)
(friend=35900)
result: {friend_number: chararray}
(6226)
(93856)
(35900)
0
votes

Try this:

a = LOAD '/logs/test.log' USING PigStorage('&') as (f1, f2, f3, f4, f5);
b = FOREACH a GENERATE REGEX_EXTRACT(f2,'friend=([0-9]*)', 1), 
                       REGEX_EXTRACT(f3,'friend=([0-9]*)', 1),
                       REGEX_EXTRACT(f5,'friend=([0-9]*)', 1);
DUMP b;