How to setup a 3-node Elasticsearch cluster on a single AWS EC2 instance?

问题

I am currently trying to deploy a 3-node Elasticsearch cluster on a single EC2 instance (i.e. using ONE instance only) using a docker-compose file. The problem is I could not get the 3 nodes to communicate with each other to form the cluster.

On my Windows 10 machine, I used the official Elasticsearch:6.4.3 image while for AWS EC2, I am using a custom Elasticsearch:6.4.3 image with ec2-discovery plugin installed where I build using the "docker build -t mdasri/eswithec2disc ." command. Refer to dockerfile below.

The dockerfile:

FROM docker.elastic.co/elasticsearch/elasticsearch:6.4.3

RUN /usr/share/elasticsearch/bin/elasticsearch-plugin install --batch discovery-ec2

I was successful in setting up the 3-node Elasticsearch cluster locally using docker-compose on my Windows 10 machine. In my docker-compose file, I have 3 different Elasticsearch services to make up the 3-nodes: es01, es02, es03. I was hoping to use the same docker-compose file to set up the cluster on AWS EC2 instance but I was hit with error.

I am using the "ecs-cli compose -f docker-compose.yml up" command to deploy to AWS EC2. The status of the ecs-cli compose was: "Started container...".

So to check the cluster status, I typed x.x.x.x/_cluster/health?pretty, but was hit with this error:

{
  "error" : {
    "root_cause" : [
      {
        "type" : "master_not_discovered_exception",
        "reason" : null
      }
    ],
    "type" : "master_not_discovered_exception",
    "reason" : null
  },
  "status" : 503
}

When I assess each of the docker container logs in the EC2 instance after I ssh in, this is the error I face in ALL 3 containers:

[2019-06-24T06:19:43,880][WARN ][o.e.d.z.UnicastZenPing ] [es01] failed to resolve host [es02]

This is my docker-compose file for the respective AWS EC2 service:

version: '2'
services:

es01:
image: mdasri/eswithec2disc
container_name: es01
cpu_shares: 100
mem_limit: 2147482548

ulimits:
  memlock:
    soft: -1
    hard: -1
  nofile:
    soft: 65536
    hard: 65536

ports:
  - "9200:9200"
  - "9300:9300"

environment:
  - "cluster.name=aws-cluster"
  - "node.name=es01"
  - "node.master=true"
  - "node.data=false"
  - "discovery.zen.hosts_provider=ec2"
  - "discovery.zen.ping.unicast.hosts=es01, es02"
  - "discovery.zen.minimum_master_nodes=2"
  - "ES_JAVA_OPTS= -Xmx256m -Xms256m"
  - "bootstrap.memory_lock=true"

volumes:
  - /usr/share/elasticsearch/data

networks:
  - esnet

es02:
image: mdasri/eswithec2disc
container_name: es02
cpu_shares: 100
mem_limit: 2147482548

ulimits:
  memlock:
    soft: -1
    hard: -1
  nofile:
    soft: 65536
    hard: 65536

environment:
  - "cluster.name=aws-cluster"
  - "node.name=es02"
  - "node.master=true"
  - "node.data=false"
  - "discovery.zen.hosts_provider=ec2"
  - "discovery.zen.ping.unicast.hosts=es01, es02"
  - "ES_JAVA_OPTS= -Xmx256m -Xms256m"
  - "bootstrap.memory_lock=true"

volumes:
  - /usr/share/elasticsearch/data

networks:
  - esnet
es03:
image: mdasri/eswithec2disc
container_name: es03
cpu_shares: 100
mem_limit: 2147482548

ulimits:
  memlock:
    soft: -1
    hard: -1
  nofile:
    soft: 65536
    hard: 65536

environment:
  - "cluster.name=aws-cluster"
  - "node.name=es03"
  - "node.master=false"
  - "node.data=true"
  - "discovery.zen.hosts_provider=ec2"
  - "discovery.zen.ping.unicast.hosts=es01,es02"
  - "ES_JAVA_OPTS= -Xmx256m -Xms256m"
  - "bootstrap.memory_lock=true"

volumes:
  - /usr/share/elasticsearch/data

networks:
  - esnet

networks:
  esnet:

Please help me as I've been stuck on this problem for the past 1-2 weeks. P.S: Please let me know what other information do you guys need. Thanks!

回答1:

you need to configure links in your docker-compose to be able to resolvable:

from docker-compose Docs:

Link to containers in another service. Either specify both the service name and a link alias (SERVICE:ALIAS), or just the service name.

web:
  links:
   - db
   - db:database
   - redis

and see the comment also from @Mishi.Srivastava

来源：https://stackoverflow.com/questions/56730966/how-to-setup-a-3-node-elasticsearch-cluster-on-a-single-aws-ec2-instance