Very slow regular expression search

Question

I\'m not sure I completely understand what is going on with the following regular expression search:

>>> import re
>>> template = re.compil         


        

Answer 1

The slowness is caused by backtracking of the engine:

(\w+)+\.

Backtracking will naturally occur with this pattern if there's no . at the end of your string. The engine will first attempt to match as many \w as possible and backtracks when it finds out that more characters need to be matched before the end of your string.

(a x 59) .
(a x 58) .
...
(a) .

Finally it will fail to match. However, the second + in your pattern causes the engine to inspect (n-1)! possible paths, so:

(a x 58) (a) .
(a x 57) (a) (a) .
(a x 57) (a x 2) .
...
(a) (a) (a) (a) (a) (a) (a) ...

Removing the + will prevent an abnormal amount of backtracking:

(\w+)\.

Some implementations will also support possessive quantifiers, which might be more ideal in this particular scenario:

(\w++)\.

Answer 2

The second plus is causing issues:

template = re.compile("(\w+)\.")

works fine for me. To see the parse tree for the regex, pass in re.DEBUG as the 2nd argument to compile:

import re

re.compile("(\w+)+\.", re.DEBUG)
print "\n\n"
re.compile("(\w+)\.", re.DEBUG)


max_repeat 1 65535
  subpattern 1
    max_repeat 1 65535
      in
        category category_word
literal 46


subpattern 1
  max_repeat 1 65535
    in
      category category_word
literal 46

Process finished with exit code 0

That proves that the second plus is adding a loop, which the python regex parser must cap at 65535. That somewhat proves my theory.

Note that to run that you will want a fresh python interpreter for each execution. re.compile memoizes the values passed in, so it will no re-compile the same regex twice, repeatedly running that in ipython for instance does not print out the parse tree after the first time you run it.

Answer 3

Understanding this problem requires understanding how NFA works under RegExp.

Elaborating the definition of NFA may be a mission too heavy for me. Search NFA on wiki it will gives you a better explanation. Here just think NFA is a robot finding patterns you give.

Crudely implemented NFA is somewhat dumb, it just looks ahead one or two tokens you give. So in the synthetic example you give, NFA just looks \w+ at first (not parenthesis for grouping).

Because + is a greedy quantifier, that is, matches as many characters as possible, so NFA dumbly continues to consume characters in target. After 30 as, NFA encounters the end of string. After then does NFA realize that he needs to match other tokens in template. The next token is +. NFA has matched it so it proceeds to \.. This time it fails.

What NFA does next is to step one character back, trying to match the pattern by truncating the submatching of \w+. So NFA split the target in to two groups, 29 as for one \w+, and one trailing a. NFA first tries to consume the trailing a by matching it against the second +, but it still fails when NFA meeting \.. NFA continues the process above until it gets a full match, otherwise it will tries all possible partitions.

So (\w+)+\. instructs NFA to group target in such manner: target is partitioned into one or more groups, at least one character per group, and target is end with a period '.'. As long as the period is not matched. NFA tries all partitions possible. So how many partitions are there? 2^n, the exponential of 2. (JUst think inserting separator between a). Like below

aaaaaaa a
aaaaaa aa
aaaaaa a a
.....
.......
aa a a ... a
a a a a a .... a

If NFA matches \., it won't hurt much. But when it fails to match, this expression is doomed to be never-ending exponential .

I'm not advertising but Mastering Regular Expression is a good book to understand mechanism under RegExp.