NEST Search whole document C# Elasticsearch

Question

I want to make a query over a million documents in Elasticsearch using Nest. My code:

var response = client.Search<MyObject>(s => s
          .Index("test")
          .Type("one")
          .Query(q => q.
                Term(
                    t => t.name, "A"
                )
          )
          .Size(10000)
          .Scroll("10m")
          .Pretty()
        );

My MyObject class:

public class MyObject
    {
        public int id { get; set; }
        public int age { get; set; }
        public string lastname { get; set; }
        public string name { get; set; }
    }

The problem is when this query is not found in the first 10k documents, it won't continue searching the rest of the results scroll API.

My question is how to achieve this (i.e moving through the whole pages in Scroll API despite there is no hits..)?

Answer 1

The query will search all documents, but will only return you the top .Size number of documents.

You can paginate results using .From() and .Size(), however, deep pagination is likely a concern when paginating over a million documents. For this, you would be better to use the scroll API to efficiently retrieve 1 million documents. NEST has an observable helper ScrollAll() to help with this

var client = new ElasticClient();

// number of slices in slice scroll
var numberOfSlices = 4;

var scrollObserver = client.ScrollAll<MyObject>("1m", numberOfSlices, s => s
    .MaxDegreeOfParallelism(numberOfSlices)
    .Search(search => search
        .Index("test")
        .Type("one")
        .Term(t => t.name, "A")
    )
).Wait(TimeSpan.FromMinutes(60), r =>
{
    // do something with documents from a given response.
    var documents = r.SearchResponse.Documents;
});