Specifying and using a NGramTokenizer with the C# NEST client for Elastic Search

Question

Updated to show a working sample

I am trying to do a partial search on a collection of usernames in ElasticSearch.

Searching around has pointed me in the nGram Tokenizer direction but I am stumped at proper implementation and fail to get any results.

This is the the relevant code stripped from the project I'm working on.

I have tried different combinations and search types to no avail.

setup.cs

var client = new ElasticClient(settings.ConnectionSettings);

// (Try and) Setup the nGram tokenizer.
var indexSettings = new IndexSettings();
var custonAnalyzer = new CustomAnalyzer();

customAnalyzer.Tokenizer = "mynGram";
customAnalyzer.Filter = new List<string> { "lowercase" };

indexSettings.Analysis.Analyzers.Add("mynGram", customAnalyzer);

indexSettings.Analysis.Tokenizers.Add("mynGram", new NGramTokenizer
                                                    {
                                                        MaxGram = 10,
                                                        MinGram = 2
                                                    });

client.CreateIndex(settings.ConnectionSettings.DefaultIndex, indexSettings);

client.MapFromAttributes<Profile>();

// Create and add a new profile object.
var profile = new Profile
                  {
                      Id = "1",
                      Username = "Russell"
                  };


client.IndexAsync(profile);

// Do search for object
var s = new SearchDescriptor<Profile>().Query(t => t.Term(c => c.Username, "russ"));

var results = client.Search<Profile>(s);

Profile.cs

public class Profile
{
    public string Id { get; set; }

    [ElasticProperty(IndexAnalyzer = "mynGram")]
    public string Username { get; set; }
}

Any tips would be much appreciated.

Answer 1

Take a look at this from the es docs on nGram token filters:

    "settings" : {
        "analysis" : {
            "analyzer" : {
                "my_ngram_analyzer" : {
                    "tokenizer" : "my_ngram_tokenizer"
                }
            },
            "tokenizer" : {
                "my_ngram_tokenizer" : {
                    "type" : "nGram",
                    "min_gram" : "2",
                    "max_gram" : "3",
                    "token_chars": [ "letter", "digit" ]
                }
            }
        }
    }

A few things to note

1. You need to add mynGram to your analyzer or it won't be used. They way it works is like this. Each indexed field has an analyzer applied to it, an analyzer is one tokenizer followed by zero or more token filters. You have defined a nice nGram tokenizer (mynGram) to use, but you did not use it in customAnalyzer, it is using the standard tokenizer. (Basically you are just defining but never using mynGram.)

2. You need to tell elasticsearch to use your customAnalyzer in your mapping: "properties": {"string_field": {"type": "string", "index_analyzer": customAnalyzer" }}

3. You should change the maxGram to a bigger number (maybe 10), otherwise 4 letter searches will not behave exactly as autocomplete (or could return nothing, depends on the search-time analyzer).

4. Use the _analyze api endpoint to test your analyzer. Something line this should work.

   curl -XGET 'http://yourserver.com:9200?index_name/_analyze?analyzer=customAnalyzer' -d 'rlewis'

Good luck!