I have a very crazy regex that I'm trying to diagnose. It is also very long, but I have cut it down to just the following script. Run using Strawberry Perl v5.26.2.
use strict; use warnings; my $text = "M Y H A P P Y T E X T"; my $regex = '(?i)(?<!(Mon|Fri|Sun)day |August )abcd(?-i)'; if ($text =~ m/$regex/){ print "true\n"; } else { print "false\n"; }
This gives the error "Variable length lookbehind not implemented in regex."
I am hoping you can help with several issues:
- I don't see why this error would occur, because all of the possible lookbehind values are 7 characters: "Monday ", "Friday ", "Sunday ", "August ".
- I did not write this regex myself, and I am not sure how to interpret the syntax
(?i) and (?-i). When I get rid of the (?i) the error actually goes away. How will perl interpret this part of the regex? I would think the first two characters are evaluated to "optional literal parentheses" except that the parentheses isn't escaped and also in that case I would get a different syntax error because the closing parentheses would then not be matched. - This behavior starts somewhere between Perl 5.16.3_64 and 5.26.1_64, at least in Strawberry Perl. The former version is fine with the code, the latter is not. Why did it start?
I have reduced your problem to this:
my $text = 'M Y H A P P Y T E X T'; my $regex = '(?<!st)A'; print ($text =~ m/$regex/i ? "true\n" : "false\n");
Due to presence of /i (case insensitive) modifier and presence of certain character combinations such as "ss" or "st" that can be replaced by a Typographic_ligature causing it to be a variable length (/August/i matches for instance on both AUGUST (6 characters) and
However if we remove /i (case insensitive) modifier then it works because typographic ligatures are not matched.
Solution: Use aa modifiers i.e.:
/(?<!st)A/iaa
Or in your regex:
my $text = 'M Y H A P P Y T E X T'; my $regex = '(?<!(Mon|Fri|Sun)day |August )abcd'; print ($text =~ m/$regex/iaa ? "true\n" : "false\n");
From perlre:
To forbid ASCII/non-ASCII matches (like "k" with "\N{KELVIN SIGN}"), specify the "a" twice, for example /aai or /aia. (The first occurrence of "a" restricts the \d, etc., and the second occurrence adds the "/i" restrictions.) But, note that code points outside the ASCII range will use Unicode rules for /i matching, so the modifier doesn't really restrict things to just ASCII; it just forbids the intermixing of ASCII and non-ASCII.
See a closely related discussion here
That's because st can be a ligature. The same happens to fi and ff:
So imagine something like
st could be represented in a 1-character stylistic ligature as
Quickly finding perl's full list of 2→1-character ligatures using a bash command:
$ perl -e 'print $^V' v5.26.2 $ for lig in {a..z}{a..z}; do \ perl -e 'print if /(?<!'$lig')x/i' 2>/dev/null || echo $lig; done ff fi fl ss st
These respectively represent the
(st and it does not match ft.)
Perl also supports the remaining stylistic ligatures, ffi and ffl, though this isn't noteworthy in this context since lookbehinds already have issues with
Future releases of perl may include more stylistic ligatures, though all that remain are font-specific (e.g. Linux Libertine has stylistic ligatures for ct and ch) or debatably stylistic (such as the Dutch ij or the obsolete Spanish ll). It doesn't seem appropriate to have this treatment for ligatures that are not entirely interchangeable (nobody would accept does), though there are other scenarios, such as including SS.
Perl 5.16.3 (and similarly old versions) only stumble on ss (for
Perl 5.14 introduced ligature support, so earlier versions don't have this problem.
Workarounds
Workarounds for /(?<!August)x/i (only the first will properly avoid
/(?<!Augus[t])(?<!Augu(?=st).)x/i (absolutely comprehensive)/(?<!Augu(?aa:st))x/i (just the st/(?<!(?aa)August)x/i/(?<!August)x/iaa/(?<!Augus[t])x/i/(?<!Augus.)x/i (slightly different, matches more)/(?<!Augu(?-i:st))x/i (case-sensitive st in lookbehind, won't match AugusTx)
These toy with removing the case-insensitive modifierASCII-safe modifier
The first variation (which is the only comprehensive one) matches the variable widths with two lookbehinds: first for the six character version (no ligatures as noted in the first quote below) and second for any ligatures, employing a forward lookahead (which has zero width!) for st (including the ligatures) and then accounting for its single character width with a .
Two segments of the perlre man page:
/i & ligatures
There are a number of Unicode characters that match a sequence of multiple characters under /i. For example, "LATIN SMALL LIGATURE FI" should match the sequence fi. Perl is not currently able to do this when the multiple characters are in the pattern and are split between groupings, or when one or more are quantified. Thus
"\N{LATIN SMALL LIGATURE FI}" =~ /fi/i; # Matches [in perl 5.14+] "\N{LATIN SMALL LIGATURE FI}" =~ /[fi][fi]/i; # Doesn't match! "\N{LATIN SMALL LIGATURE FI}" =~ /fi*/i; # Doesn't match! "\N{LATIN SMALL LIGATURE FI}" =~ /(f)(i)/i; # Doesn't match!
/aa (perl 5.14+)
To forbid ASCII/non-ASCII matches (like k with \N{KELVIN SIGN}), specify the a twice, for example /aai or /aia. (The first occurrence of a restricts the \d, etc., and the second occurrence adds the /i restrictions.) But, note that code points outside the ASCII range will use Unicode rules for /i matching, so the modifier doesn't really restrict things to just ASCII; it just forbids the intermixing of ASCII and non-ASCII.
To summarize, this modifier provides protection for applications that don't wish to be exposed to all of Unicode. Specifying it twice gives added protection.
Put (?i) after lookbehind:
(?<!(Mon|Fri|Sun)day |August )(?i)abcd(?-i)
or
(?<!(Mon|Fri|Sun)day |August )(?i:abcd)
To me it seems to be a bug.
