How to talk to aws elasticsearch service using elastic java client?

Question

I have set up a elasticsearch server using AWS elasticsearch service (Not EC2). It gave me an endpoint https://xxx-xxxxxxxx.us-west-2.es.amazonaws.com/ and if I click this e

Answer 1

Believe it or not, AWS doesn't launch Elasticsearch using 9200 and 9300. It's launched via plain old port 80.

So, to demonstrate, try this...

curl -XPOST "http://xxx-xxxxxxxx.us-west-2.es.amazonaws.com:80/myIndex/myType" -d '["name":"Edmond"}'

Or

curl -XPOST "https://xxx-xxxxxxxx.us-west-2.es.amazonaws.com:443/myIndex/myType" -d '["name":"Edmond"}'

It should respond with: {"_index":"myIndex","_type":"myType","_id":"SOME_ID_#","_version":1,"created":true}

Check in Kibana and you'll see it's there.

So, then in your code, it should be:

Client client = TransportClient.builder().build()
    .addTransportAddress(new InetSocketTransportAddress(InetAddress.getByName("xxx-xxxxxxxx.us-west-2.es.amazonaws.com"), 80));

Unfortunately, I don't off-hand know how to transmit encrypted via SSL/HTTPS using the transport client. You could try using regular REST calls instead using JERSEY.

Finally, make sure your Elasticsearch access policy is configured properly. Something along the lines of:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": "*",
      "Action": "es:*",
      "Resource": "arn:aws:es:us-east-1:yyyyyyy:domain/myDomain/*"
    },
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": "*",
      "Action": "es:*",
      "Resource": "arn:aws:es:us-east-1:yyyyyyyyy:domain/myDomain"
    }
  ]
}

NOTE: The above access policy is completely wide open and is not recommended for anything remotely close to production. Just so you know....

Answer 2

After lot of search i found an example which used a GET request, so I made minor changes to it for allowing POST requests so that complex queries can be submitted via POST body. The implementation is available at https://github.com/dy10/aws-elasticsearch-query-java

Apart from properly configuring access to you AWS ES (i.e. dont open it to Public), make sure to use https (the above code uses http; just replace http with https in the code and it will work).

Another useful looking but partial implementation is at https://github.com/aws/aws-sdk-java/issues/861

Answer 3

Managed elastic search service in AWS does not provide the port for the transport protocol until now.
This question has been answered here ,

Elastic Transport client on AWS Managed ElasticSearch1

There is also a discussion in the AWS forum regarding the transport protocol. Here is the link

What is the port for the transport protocol ?