RegexpError: Stack overflow in regexp matcher

1
votes

I have small problem with a simple tokenizer regex:

def test_tokenizer_regex_limit
   string = '<p>a</p>' * 400
   tokens = string.scan(/(<\s*tag:.*?\/?>)|((?:[^<]|\<(?!\s*tag:.*?\/?>))+)/)
end

Basically it runs through the text and gets pairs of [ matched_tag , other_text ]. Here's an example: http://rubular.com/r/f88JBjfzFh

Works fine for smaller sets. If you run in under ruby 1.8.7 it will blow up. 1.9.2 works fine.

Any ideas how to simplify / improve this? My regex-fu is weak

1
rubyregex
I'm not really (x)html parsing though. Just need to tokenize text like this: text | <tag:blah> | more text | <tag:meh>. As a matter of fact it could be a string like this: '<a href="<tag:url>">' Not really parsable. - Grocery

1 Answers

0
votes

This is a bit more simplified but not much:

(<[^<]*:[^<]*>)|((?:[^<]|<[^:]*>)+)

(<.*?>|[^<>]+)