Google Datastore Emulator using Java (Not using GAE)

Question

I am using Google Cloud\'s Datastore Client Library for Java to access the Cloud Datastore.

Note: I am not using App Engine to deploy my applicatio

Answer 1

I assume that you want to test against the Datastore Emulator. In that case it is not necessary to start the Datastore Emulator from the shell. There is a LocalDatastoreHelper in the gcloud library that allows you to create, start, reset and stop a local Datastore Emulator with ease.

I didn't found any documentation on that, so I created this testcases for you:

import com.google.cloud.datastore.Datastore;
import com.google.cloud.datastore.DatastoreOptions;
import com.google.cloud.datastore.Entity;
import com.google.cloud.datastore.KeyFactory;
import com.google.cloud.datastore.testing.LocalDatastoreHelper;
import org.junit.*;

import java.io.IOException;

import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertNotNull;
import static org.junit.Assert.assertNull;

/**
 * This testcase demonstrate the use of the datastore emulator in JUnit test cases.
 *
 * from @link https://www.kontaktlinsen-preisvergleich.de
 */
public class DatastoreEmulatorTest {

    protected static LocalDatastoreHelper localDatastoreHelper;

    protected Datastore datastore;

    protected KeyFactory keyFactory;

    @BeforeClass
    public static void setUpClass() throws InterruptedException, IOException {
        // create and start a local datastore emulator on a random free port
        // this also means that you probably can run tests like this concurrently.
        System.out.println("[Datastore-Emulator] start");
        localDatastoreHelper = LocalDatastoreHelper.create();
        localDatastoreHelper.start();
        System.out.println("[Datastore-Emulator] listening on port: " + localDatastoreHelper.getPort());

        // set the system property to tell the gcloud lib to use the datastore emulator
        System.setProperty("DATASTORE_EMULATOR_HOST","localhost:" + localDatastoreHelper.getPort());
    }

    @Before
    public void setUp() {
        // create the datastore instance
        // because of the system property set it in setUpClass() this
        // datastore will be connected with the datastore emulator.
        datastore = DatastoreOptions.getDefaultInstance().getService();
        keyFactory = datastore.newKeyFactory().setKind("TestEntity");
    }

    @After
    public void tearDown() throws IOException {
        System.out.println("[Datastore-Emulator] reset");
        // this resets the datastore after every test
        localDatastoreHelper.reset();
    }

    @AfterClass
    public static void tearDownClass() throws InterruptedException, IOException {
        System.out.println("[Datastore-Emulator] stop");
        // this stops the datastore emulator after all tests are done
        localDatastoreHelper.stop();
    }

    @Test
    public void test1() {
        // stores an entity in the datastore and retrieves it later

        // create an Entity "TestEntity"
        Entity.Builder builder = Entity.newBuilder(keyFactory.newKey(42));
        builder.set("name", "Test1");

        // store it in datastore
        datastore.put(builder.build());

        // retrieve entity by key
        Entity entity = datastore.get(keyFactory.newKey(42));
        assertNotNull(entity);
        assertEquals("Test1", entity.getString("name"));
    }

    @Test
    public void test2() {
        // try to access the entity created in test1, shouldn't work because
        // of calling reset in tearDown() after each test.

        // try to retrieve entity by key
        Entity entity = datastore.get(keyFactory.newKey(42));
        assertNull(entity);
    }
}

The LocalDatastoreHelper creates a Datastore Emulator instance on a free port and doesn't store to disk - when you stop the testcase with the debugger and look for processes, you'll find something like this:

$ ps ax | grep CloudDatastore
2614   ??  R      0:01.39 /usr/bin/java -cp /Users/marco/google-cloud-sdk/platform/cloud-datastore-emulator/CloudDatastore.jar com.google.cloud.datastore.emulator.CloudDatastore /Users/marco/google-cloud-sdk/platform/cloud-datastore-emulator/cloud_datastore_emulator start --host=localhost --port=57640 --store_on_disk=False --consistency=0.9 --allow_remote_shutdown /var/folders/ky/c126qk_161159ltyrbpdxv8w0000gn/T/gcd2141205756617995044

That also means you should also be able to run tests in parallel, too.

Answer 2

The line below always connects to the remote datastore. Uses the default options (e.g. project, auth credentials) from gcloud settings.

Datastore datastore = DatastoreOptions.defaultInstance().service();

To connect to the local datastore, try the below:

@Test
public void test1() throws IOException, InterruptedException {
    Datastore ds = DatastoreOptions.builder().host("http://localhost:9999").projectId("my-project").build().service();
    com.google.cloud.datastore.Key key = ds.newKeyFactory().kind("MyEntity").newKey("mykey");
    com.google.cloud.datastore.Entity entity = com.google.cloud.datastore.Entity.builder(key).set("p1", "Hello World!").build();
    entity = ds.put(entity);
    entity = ds.get(key);
    System.out.println(entity);
}

I started my Datastore Emulator on localhost:9999. Set that as the host when building the DatastoreOptions.

I've confirmed that the Emulator console shows requests are received and entities are persisted. I've also checked the data file (local_db.bin) and it shows the data (of course it is not a plain text file).

The one thing I don't know is - if there is a way to manage the local datastore using a browser interface. I could not find much documentation on how to administer the local datastore just like how we do the remote one from Cloud Console. Perhaps someone else can help with this.