Google App Engine Datastore - Testing Queries fails

Question

I am currently trying to test a piece of my code that runs a query on the datastore before putting in a new entity to ensure that duplicates are not created. The code I wrote works fine in the context of the app, but the tests I wrote for that methods are failing. It seems that I cannot access data put into the datastore through queries in the context of the testing package.

One possibility might lie in the output from goapp test which reads: Applying all pending transactions and saving the datastore. This line prints out after both the get and put methods are called (I verified this with log statements).

I tried closing the context and creating a new one for the different operations, but unfortunately that didn't help either. Below is a simple test case that Puts in an object and then runs a query on it. Any help would be appreciated.

type Entity struct {
    Value string
}

func TestEntityQuery(t *testing.T) {
    c, err := aetest.NewContext(nil)
    if err != nil {
        t.Fatal(err)
    }
    defer c.Close()

    key := datastore.NewIncompleteKey(c, "Entity", nil)
    key, err = datastore.Put(c, key, &Entity{Value: "test"})
    if err != nil {
        t.Fatal(err)
    }

    q := datastore.NewQuery("Entity").Filter("Value =", "test")
    var entities []Entity
    keys, err := q.GetAll(c, &entities)
    if err != nil {
        t.Fatal(err)
    }
    if len(keys) == 0 {
        t.Error("No keys found in query")
    }
    if len(entities) == 0 {
        t.Error("No entities found in query")
    }
}

Answer 1

There is nothing wrong with your test code. The issue lies in the Datastore itself. Most queries in the HR Datastore are not "immediately consistent" but eventually consistent. You can read more about this in the Datastore documentation.

So basically what happens is that you put an entity into the Datastore, and the SDK's Datastore "simulates" the latency that you can observe in production, so if you run a query right after that (which is not an ancestor query), the query result will not include the new entity you just saved.

If you put a few seconds sleep between the datastore.Put() and q.GetAll(), you will see the test passes. Try it. In my test it was enough to sleep just 100ms, and the test always passed. But when writing tests for such cases, use the StronglyConsistentDatastore: true option as can be seen in JonhGB's answer.

You would also see the test pass without sleep if you'd use Ancestor queries because they are strongly consistent.

Answer 2

The way to do this is to force the datastore to be strongly consistent by setting up the context like this:

c, err := aetest.NewContext(&aetest.Options{StronglyConsistentDatastore: true})
    if err != nil {
        t.Fatal(err)
    }

Now the datastore won't need any sleep to work, which is faster, and better practice in general.

---

Update: This only works with the old aetest package which was imported via appengine/aetest. It does not work with the newer aetest package which is imported with google.golang.org/appengine/aetest. App Engine has changed from using an appengine.Context to using a context.Context, and consequently the way that the test package now works is quite different.

Answer 3

To compliment @JohnGB's answer in the latest version of aetest, there are more steps to get a context with strong consistency. First create an instance, then create a request from that instance, which you can use to produce a context.

inst, err := aetest.NewInstance(
&aetest.Options{StronglyConsistentDatastore: true})

if err != nil {
    t.Fatal(err)
}
defer inst.Close()

req, err := inst.NewRequest("GET", "/", nil)
if err != nil {
    t.Fatal(err)
}

ctx := appengine.NewContext(req)