“Variable length lookbehind not implemented” but it isn't variable length

Question

I have a very crazy regex that I\'m trying to diagnose. It is also very long, but I have cut it down to just the following script. Run using Strawberry Perl v5.26.2.

Answer 1

st could be represented in a 1-character stylistic ligature as st or ſt, so its length could be 2 or 1.

Quickly finding perl's full list of 2→1-character ligatures using a bash command:

$ perl -e 'print $^V'
v5.26.2
$ for lig in {a..z}{a..z}; do \
    perl -e 'print if /(?/dev/null || echo $lig; done

ff fi fl ss st

These respectively represent the ff, fi, fl, ß, and st/ſt ligatures.
^{(ſt represents ſt, using the obsolete long s character; it matches st and it does not match ft.)}

Perl also supports the remaining stylistic ligatures, ffi and ffl for ffi and ffl, though this isn't noteworthy in this context since lookbehinds already have issues with ff and fi/fl separately.

Future releases of perl may include more stylistic ligatures, though all that remain are font-specific (e.g. Linux Libertine has stylistic ligatures for ct and ch) or debatably stylistic (such as the Dutch ij for ij or the obsolete Spanish ꝇ for ll). It doesn't seem appropriate to have this treatment for ligatures that are not entirely interchangeable (nobody would accept dœs for does), though there are other scenarios, such as including ß thanks to its uppercase form being SS.

Perl 5.16.3 (and similarly old versions) only stumble on ss (for ß) and fail to expand the other ligatures in lookbehinds (they have fixed width and will not match). I didn't seek out the bugfix to itemize exactly which versions are affected.

Perl 5.14 introduced ligature support, so earlier versions don't have this problem.

Workarounds

Workarounds for /(? (only the first will properly avoid August):

• /(? (absolutely comprehensive)
• /(? (just the st in the lookbehind is "ASCII-safe" ²)
• /(? (the whole the lookbehind is "ASCII-safe" ²)
• /(? (the whole regex is "ASCII-safe" ²)
• /(? (breaks ligature seeking ¹)
• /(? (slightly different, matches more)
• /(? (case-sensitive st in lookbehind, won't match AugusTx)

These toy with removing the case-insensitive modifier¹ or adding the ASCII-safe modifier² in various places, often requiring the regex writer to specifically know of the variable-width ligature.

The first variation (which is the only comprehensive one) matches the variable widths with two lookbehinds: first for the six character version (no ligatures as noted in the first quote below) and second for any ligatures, employing a forward lookahead (which has zero width!) for st (including the ligatures) and then accounting for its single character width with a .

Two segments of the perlre man page:

¹ Case-insensitive modifier /i & ligatures

There are a number of Unicode characters that match a sequence of multiple characters under /i. For example, "LATIN SMALL LIGATURE FI" should match the sequence fi. Perl is not currently able to do this when the multiple characters are in the pattern and are split between groupings, or when one or more are quantified. Thus

"\N{LATIN SMALL LIGATURE FI}" =~ /fi/i;          # Matches [in perl 5.14+]
"\N{LATIN SMALL LIGATURE FI}" =~ /[fi][fi]/i;    # Doesn't match!
"\N{LATIN SMALL LIGATURE FI}" =~ /fi*/i;         # Doesn't match!
"\N{LATIN SMALL LIGATURE FI}" =~ /(f)(i)/i;      # Doesn't match!

² ASCII-safe modifier /aa (perl 5.14+)

To forbid ASCII/non-ASCII matches (like k with \N{KELVIN SIGN}), specify the a twice, for example /aai or /aia. (The first occurrence of a restricts the \d, etc., and the second occurrence adds the /i restrictions.) But, note that code points outside the ASCII range will use Unicode rules for /i matching, so the modifier doesn't really restrict things to just ASCII; it just forbids the intermixing of ASCII and non-ASCII.

To summarize, this modifier provides protection for applications that don't wish to be exposed to all of Unicode. Specifying it twice gives added protection.