org.apache.hadoop.hbase.PleaseHoldException: Master is initializing

问题:

I am trying to setup the multinode cluster of Hbase. When i do the jps on slave i get

5780 Jps 5558 HQuorumPeer 5684 HRegionServer 1963 DataNode 2093 TaskTracker 

similarly on master i get

4254 SecondaryNameNode 15226 Jps 14982 HMaster 3907 NameNode 14921 HQuorumPeer 4340 JobTracker 

EVerything is runnnig properly. But when i try to create table on hbase shell. It gives an error

ERROR: org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing 

regionserver log of my slave(where region server is running):

2013-06-11 13:09:53,119 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Attempting connect to Master server at localhost,60000,137093$ 2013-06-11 13:10:53,190 WARN org.apache.hadoop.hbase.regionserver.HRegionServer: Unable to connect to master. Retrying. Error was: org.apache.hadoop.hbase.ipc.HBaseClient$FailedServerException: This server is in the failed servers list: localhost/127.0.0.1:60000         at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:425)         at org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:1124)         at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:974)         at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:86)         at $Proxy8.getProtocolVersion(Unknown Source)         at org.apache.hadoop.hbase.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:138)         at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:208)         at org.apache.hadoop.hbase.regionserver.HRegionServer.getMaster(HRegionServer.java:2037)         at org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:2083)         at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:744)         at java.lang.Thread.run(Thread.java:722) 2013-06-11 13:10:53,391 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Attempting connect to Master server at localhost,60000,137093$ 

FYI, i have also took care of /etc/hosts file on both master and slave.

127.0.0.1       localhost 127.0.0.1       naresh-PC 

I again did changes in /etc/hosts file 127.0.1.1 to naresh-PC. But still getting this error

2013-06-11 14:51:17,781 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Attempting connect to Master server at naresh-pc,60000,137094$ 2013-06-11 14:52:17,817 WARN org.apache.hadoop.hbase.regionserver.HRegionServer: Unable to connect to master. Retrying. Error was: java.net.UnknownHostException: unknown host: naresh-pc         at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.<init>(HBaseClient.java:276)         at org.apache.hadoop.hbase.ipc.HBaseClient.createConnection(HBaseClient.java:255)         at org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:1111)         at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:974)         at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:86)         at $Proxy8.getProtocolVersion(Unknown Source)         at org.apache.hadoop.hbase.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:138)         at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:208)         at org.apache.hadoop.hbase.regionserver.HRegionServer.getMaster(HRegionServer.java:2037)         at org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:2083)         at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:744)         at java.lang.Thread.run(Thread.java:722) 

回答1:

Try clearing all the states in Zookeeper.

• stop Zookeeper
• wipe the Zookeeper data directory
• restart it

I was getting the same issue, and followed this approach and it worked fine.

回答2:

You need to change the configuration on the slave node to point at the master. It is currently pointing to localhost and not connecting to the actual master:

"org.apache.hadoop.hbase.ipc.HBaseClient$FailedServerException: This server is in the failed servers list: localhost/127.0.0.1:60000 at "

回答3:

I'm hosting my own cluster inside Docker. Here's what worked in my case. I grepped the HBase log file for errors and found "Master passed us a different hostname to use"

`[root@docker-iop bin]# grep ERROR /var/log/hbase/hbase-hbase-regionserver-bi-mgmt01.local.log 2016-10-06 00:05:29,816 ERROR [regionserver/bi-mgmt01.local/111.11.2.3:16020] regionserver.HRegionServer: Master passed us a different hostname to use; was=my-host-name, but now=111.22.33.444' 

I mapped my-host-name to 111.22.333.444 in my hosts file, restarted HBase and it worked.

回答4:

I also had the same issue with a fully distributed hbase cluster with the configuration below.

1. Master Node (Node-A)
2. Backup Masters ($HBASE_HOME/conf/backup-masters) (Node-B & Node-C)
3. 3 Replication servers (Node-A, Node-B & Node-C)

RCA: The backup-masters nodes were attempted to be started when the cluster started.

Solution I removed the backup masters by making $HBASE_HOME/conf/backup-masters empty in all hbase nodes.

So I had a cluster running without backup masters.

I wonder if the master node and master nodes must not also function as regionservers? The HBase documentation says otherwise though.

文章来源: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing