Best way to test for existing string against a large list of comparables

Question

Suppose you have a list of acronym's that define a value (ex. AB1,DE2,CC3) and you need to check a string value (ex. "Happy:DE2|234") to see if an acronym is found in the string. For a short list of acronym's I would usually create a simple RegEx that used a separator (ex. (AB1|DE2|CC3) ) and just look for a match.

But how would I tackle this if there are over 30 acronym's to match against? Would it make sense to use the same technique (ugly) or is there a more effecient and elegant way to accomplish this task?

Keep in mind the example acronym list and example string is not the actual data format that I am working with, rather just a way to express my challenge.

BTW, I read a SO related question but didn't think it applied to what I was trying to accomplish.

EDIT: I forgot to include my need to capture the matched value, hence the choice to use Regular Expressions...

Answer 1

Personally I don't think 30 is particularly large for a regex so I wouldn't be too quick to rule it out. You can create the regex with a single line of code:

var acronyms = new[] { "AB", "BC", "CD", "ZZAB" };
var regex = new Regex(string.Join("|", acronyms), RegexOptions.Compiled);
for (var match = regex.Match("ZZZABCDZZZ"); match.Success; match = match.NextMatch())
    Console.WriteLine(match.Value);
// returns AB and CD

So the code is relatively elegant and maintainable. If you know the upper bound for the number of acronyms I would to some testing, who knows what kind of optimizations there are already built into the regex engine. You'll also be able to benefit for free from future regex engine optimizations. Unless you have reason to believe performance will be an issue keep it simple.

On the other hand regex may have other limitations e.g. by default if you have acronyms AB, BC and CD then it'll only return two of these as a match in "ABCD". So its good at telling you there is an acronym but you need to be careful about catching multiple matches.

When performance became an issue for me (> 10,000 items) I put the 'acronyms' in a HashSet and then searched each substring of the text (from min acronym length to max acronym length). This was ok for me because the source text was very short. I'd not heard of it before, but at first look the Aho-Corasick algorithm, referred to in the question you reference, seems like a better general solution to this problem.

Answer 2

If acronym's have fixed size (like in above example), you could calculate a hash for all of them (could be done once per application life) and then split the string in such overlapped pieces and calculate hashes for them too. Then all you'd have to do is to search for values from one array into another one.

You probably could create a suffix/prefix tree or something similar from acronyms and search using this information, there's plenty of algorithms in Wikipedia to do just that.

You could also create an deterministic automata for each of acronyms but it's very similar to previous approach.

Answer 3

Why not simply split the string and compare the returned list? It seems like needless overhead to use a REGEX in this case. I know your format may differ, but it would seem that you could:

• Split the string based on the 'title separator', in your case a colon :
• Take the 2nd half of the result, the acronym string, and split it based on the acronym separator, in this case a pipe |
• Finally, iterate over the newly split list of acronyms and compare each to your list of candidates with a nested for loop

EDIT: If you only need to know if a particular acronym or set of acronyms exist inside a string, use the .Search() method instead of .Match().

Answer 4

The regex approach seems efficient and elegant enough. Of course, you'll have to watch out for unescaped characters when building the expression, or a failure to compile it because of complexity or size limitations.

Another way to do this would be to construct a trie data structure to represent all the acronyms (this may somewhat duplicate what the regex matcher is doing). As you step through each character in the string, you would create a new pointer to the root of the trie, and advance existing pointers to the appropriate child (if any). You get a match when any pointer reaches a leaf.

Answer 5

Here is what I came up with. I would appreciate any constructive criticism that you could offer...

First, create an enum that holds each of my acronym's:

enum acronym
{ AB1,DE2,CC3 }

Next I create a string array of the enum:

string[] acronyms = Enum.GetNames(typeof(acronym));

Finally I loop through the string array and peform the regex.match method:

foreach (string a in acronyms)
{
    Match aMatch = Regex.Match(input, a.ToString(), RegexOptions.None);
    if (aMatch.Success)
    {
        ...<do something>...
        break;
    }
}

See anything wrong with that?