python - nltk regular expression tokenizer

Question

I tried to implement a regular expression tokenizer with nltk in python, but the result is this:

>>> import nltk
>>> text = 'That U.S.A. poster-print costs $12.40...'
>>> pattern = r'''(?x)    # set flag to allow verbose regexps
...     ([A-Z]\.)+        # abbreviations, e.g. U.S.A.
...   | \w+(-\w+)*        # words with optional internal hyphens
...   | \$?\d+(\.\d+)?%?  # currency and percentages, e.g. $12.40, 82%
...   | \.\.\.            # ellipsis
...   | [][.,;"'?():-_`]  # these are separate tokens; includes ], [
... '''
>>> nltk.regexp_tokenize(text, pattern)
[('', '', ''), ('', '', ''), ('', '-print', ''), ('', '', ''), ('', '', '')]

But the wanted result is this:

['That', 'U.S.A.', 'poster-print', 'costs', '$12.40', '...']

Why? Where is the mistake?

Try from nltk.tokenize import RegexpTokenizer, tokenizer = RegexpTokenizer(pattern) and then tokenizer.tokenize(text) — Wiktor Stribiżew
It is returning ['That', 'U.S.A.', 'poster-print', 'costs', '$12.40', '...'] in my notebook. Maybe a version issue? (3.0.4) — ayhan
i tried with python 3.5 but the result is this yet: [('', '', ''), ('', '', ''), ('', '-print', ''), ('', '', ''), ('', '', '')] — Juan Menashsheh
Aha, you should turn all capturing groups to non-capturing. ([A-Z]\.)+ > (?:[A-Z]\.)+, \w+(-\w+)* -> \w+(?:-\w+)* and \$?\d+(\.\d+)?%? to \$?\d+(?:\.\d+)?%? — Wiktor Stribiżew
Please see github.com/nltk/nltk/issues/1206#issuecomment-156470847 — alvas

Wiktor Stribiżew Wiktor Stribiżew · Accepted Answer · 2016-04-01T10:30:55

You should turn all capturing groups to non-capturing:

([A-Z]\.)+ > (?:[A-Z]\.)+
\w+(-\w+)* -> \w+(?:-\w+)*
\$?\d+(\.\d+)?%? to \$?\d+(?:\.\d+)?%?

The issue is that regexp_tokenize seems to be using re.findall that returns capture tuple lists when multiple capture groups are defined in the pattern. See this nltk.tokenize package reference:

pattern (str) – The pattern used to build this tokenizer. (This pattern must not contain capturing parentheses; Use non-capturing parentheses, e.g. (?:...), instead)

Also, I am not sure you wanted to use :-_ that matches a range including all uppercase letters, put the - to the end of the character class.

Thus, use

pattern = r'''(?x)          # set flag to allow verbose regexps
        (?:[A-Z]\.)+        # abbreviations, e.g. U.S.A.
      | \w+(?:-\w+)*        # words with optional internal hyphens
      | \$?\d+(?:\.\d+)?%?  # currency and percentages, e.g. $12.40, 82%
      | \.\.\.              # ellipsis
      | [][.,;"'?():_`-]    # these are separate tokens; includes ], [
    '''

python - nltk regular expression tokenizer

1 Answers