How to configure hosts file for Hadoop ecosystem

Question

The question may seem pretty obvious, but I have faced it many times, due to bad configuration of hosts file on a hadoop cluster.

Can anyone describe how to setup ho

Answer 1

For UBUNTU

Hosts File and other configuration for Hadoop Cluster

Provide hostname to all cluster machines, to do so add hostname in /etc/hostname file as

hostname-of-machine

On all the host, hosts file should be like this:

hosts

127.0.0.1       localhost
#127.0.1.1      localhost

    FQDN                hostname    other_name
172.26.43.10    cdh4hdm.domain.com  cdh4hdm     kdc1
172.26.43.11    cdh4hbm.domain.com  cdh4hbm
172.26.43.12    cdh4s1.domain.com   cdh4s1
172.26.43.13    cdh4s2.domain.com   cdh4s2
172.26.43.14    cdh4s3.domain.com   cdh4s3
172.26.43.15    cdh4s4.domain.com   cdh4s4

Note: Make sure to comment line 127.0.1.1 localhost it may create problem in zookeeper and cluster.

Add DNS server IP in /etc/resolv.conf

resolve.conf

search domain.com
nameserver 10.0.1.1

to verify configuration check hostfile and your should be able to ping all the machines by their hostname

To check hostname and FQDN on all machines run following commands:

hostname        //should return the hostname
hostname -f     //Fully Qualified Hostname
hostname -d     //Domain name

All commands will be same for RHEL except the hostname.

Source1 and Source2