用到HBase的背景也是因为数据量到达了一定的量级，传统的关系型数据库存储处理有些乏力，固转为HBase这种扩展性更强、处理大数据更强的存储方式。
try { Connection con = DriverManager.getConnection("jdbc:phoenix:hb-proxy-pub1,hb-proxy-pub2,hb-proxy-pub3", props); Statement stmt = con.createStatement(); stmt.execute("drop table if exists test"); stmt.execute("create table test (mykey integer not null primary key, mycolumn varchar)"); stmt.execute("create index test_idx on test(mycolumn)"); stmt.executeUpdate("upsert into test values (1,'World!')"); stmt.executeUpdate("upsert into test values (2,'Hello')"); stmt.executeUpdate("upsert into test values (3,'World!')"); con.commit(); PreparedStatement statement = con.prepareStatement("select mykey from test where mycolumn='Hello'"); ResultSet rset = statement.executeQuery(); while (rset.next()) { System.out.println(rset.getInt(1)); } rset.close(); con.close(); } catch (SQLException e) { e.printStackTrace(); } 当检索大量数据的时候，可能会在等待一段时间后报错：
call(HashJoinPlan.java:168) at org.apache.phoenix.execute.HashJoinPlan$1.call(HashJoinPlan.java:164) at java.util.concurrent.FutureTask.run(Unknown Source) at org.apache.phoenix.job.JobManager$InstrumentedJobFutureTask.run(JobManager.java:183) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Caused by: org.apache.phoenix.exception.PhoenixIOException: Failed after attempts=36, exceptions: Tue May 29 20:12:28 AWST 2018, null, java.net.SocketTimeoutException: callTimeout=60000, callDuration=60208: row on table 'IDX_CONTRACT_CREATE_TIME' at region=IDX_CONTRACT_CREATE_TIME,\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00,1524044495952.56c349853653a18d1d7cbb3eedd3f45f., hostname=hb-bp1ese450t849odg0-003.hbase.rds.aliyuncs.com,16020,1523854368775, seqNum=2 at org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:120) at org.apache.phoenix.iterate.TableResultIterator.next(TableResultIterator.java:173) at org.apache.phoenix.iterate.LookAheadResultIterator$1.advance(LookAheadResultIterator.java:47) at org.apache.phoenix.iterate.LookAheadResultIterator.init(LookAheadResultIterator.java:59) at org.apache.phoenix.iterate.LookAheadResultIterator.peek(LookAheadResultIterator.java:73) at org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:128) at org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:113) ... 5 more Caused by: org.apache.phoenix.exception.PhoenixIOException: Failed after attempts=36, exceptions: Tue May 29 20:12:28 AWST 2018, null, java.net.SocketTimeoutException: callTimeout=60000, callDuration=60208: row on table 'IDX_CONTRACT_CREATE_TIME' at region=IDX_CONTRACT_CREATE_TIME,\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00,1524044495952.56c349853653a18d1d7cbb3eedd3f45f., hostname=hb-bp1ese450t849odg0-003.hbase.rds.aliyuncs.com,16020,1523854368775, seqNum=2 at org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:120) at org.apache.phoenix.iterate.ScanningResultIterator.next(ScanningResultIterator.java:121) at org.apache.phoenix.iterate.TableResultIterator.next(TableResultIterator.java:166) ... 10 more Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=36, exceptions: Tue May 29 20:12:28 AWST 2018, null, java.net.SocketTimeoutException: callTimeout=60000, callDuration=60208: row on table 'IDX_CONTRACT_CREATE_TIME' at region=IDX_CONTRACT_CREATE_TIME,\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00,1524044495952.56c349853653a18d1d7cbb3eedd3f45f., hostname=hb-bp1ese450t849odg0-003.hbase.rds.aliyuncs.com,16020,1523854368775, seqNum=2 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.throwEnrichedException(RpcRetryingCallerWithReadReplicas.java:271) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:210) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:60) at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:210) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:327) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:413) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:371) at org.apache.phoenix.iterate.ScanningResultIterator.next(ScanningResultIterator.java:112) ... 11 more Caused by: java.net.SocketTimeoutException: callTimeout=60000, callDuration=60208: row on table 'IDX_CONTRACT_CREATE_TIME' at region=IDX_CONTRACT_CREATE_TIME,\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00,1524044495952.56c349853653a18d1d7cbb3eedd3f45f., hostname=hb-bp1ese450t849odg0-003.hbase.rds.aliyuncs.com,16020,1523854368775, seqNum=2 at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:169) at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:65) ... 3 more Caused by: java.io.IOException: Call to hb-proxy-pub-bp1ese450t849odg0-003.hbase.rds.aliyuncs.com/101.37.143.39:16020 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=76, waitTime=60049, operationTimeout=59999 expired. at org.apache.hadoop.hbase.ipc.AbstractRpcClient.wrapException(AbstractRpcClient.java:284) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1238) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:223) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:328) at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.scan(ClientProtos.java:32831) at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:215) at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:63) at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:210) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:364) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:338) at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:136) ... 4 more Caused by: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=76, waitTime=60049, operationTimeout=59999 expired. at org.apache.hadoop.hbase.ipc.Call.checkAndSetTimeout(Call.java:70) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1212) ... 13 more [WARN ][phoenix-1-thread-0] 2018-05-29 20:12:47.833 - ScannerCallable.close(367) | Ignore, probably already closed org.apache.hadoop.hbase.UnknownScannerException: org.apache.hadoop.hbase.UnknownScannerException: Name: 66596 ,scanner = {"loadColumnFamiliesOnDemand":null,"startRow":"","stopRow":"","batch":-1,"cacheBlocks":true,"totalColumns":0,"maxResultSize":-1,"attributes":"{}","families":{},"caching":-1,"maxVersions":1,"timeRange":[0,9223372036854775807]} ,HRegion = 559038737, already closed? at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2865) at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32295) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2394) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:174) at org.apache.hadoop.hbase.ipc.RpcExecutor$CallHandler.run(RpcExecutor.java:178) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source) at java.lang.reflect.Constructor.newInstance(Unknown Source) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:327) at org.apache.hadoop.hbase.client.ScannerCallable.close(ScannerCallable.java:364) at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:197) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:145) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:60) at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:210) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:327) at org.apache.hadoop.hbase.client.ClientScanner.close(ClientScanner.java:748) at org.apache.phoenix.iterate.ScanningResultIterator.close(ScanningResultIterator.java:67) at org.apache.phoenix.iterate.TableResultIterator.close(TableResultIterator.java:145) at org.apache.phoenix.iterate.LookAheadResultIterator$1.close(LookAheadResultIterator.java:42) at org.apache.phoenix.iterate.BaseResultIterators.close(BaseResultIterators.java:1091) at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:1003) at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:890) at org.apache.phoenix.iterate.RoundRobinResultIterator.getIterators(RoundRobinResultIterator.java:176) at org.apache.phoenix.iterate.RoundRobinResultIterator.next(RoundRobinResultIterator.java:91) at org.apache.phoenix.iterate.DelegateResultIterator.next(DelegateResultIterator.java:44) at org.apache.phoenix.execute.TupleProjectionPlan$1.next(TupleProjectionPlan.java:62) at org.apache.phoenix.execute.HashJoinPlan$HashSubPlan.execute(HashJoinPlan.java:433) at org.apache.phoenix.execute.HashJoinPlan$1.call(HashJoinPlan.java:168) at org.apache.phoenix.execute.HashJoinPlan$1.call(HashJoinPlan.java:164) at java.util.concurrent.FutureTask.run(Unknown Source) at org.apache.phoenix.job.JobManager$InstrumentedJobFutureTask.run(JobManager.java:183) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.UnknownScannerException): org.apache.hadoop.hbase.UnknownScannerException: Name: 66596 ,scanner = {"loadColumnFamiliesOnDemand":null,"startRow":"","stopRow":"","batch":-1,"cacheBlocks":true,"totalColumns":0,"maxResultSize":-1,"attributes":"{}","families":{},"caching":-1,"maxVersions":1,"timeRange":[0,9223372036854775807]} ,HRegion = 559038737, already closed? at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2865) at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32295) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2394) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:174) at org.apache.hadoop.hbase.ipc.RpcExecutor$CallHandler.run(RpcExecutor.java:178) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1234) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:223) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:328) at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.scan(ClientProtos.java:32831) at org.apache.hadoop.hbase.client.ScannerCallable.close(ScannerCallable.java:362) ... 24 more [INFO ][phoenix-1-thread-0] 2018-05-29 20:12:47.839 - BaseResultIterators.close(1096) | Failed to execute task during cancel java.util.concurrent.ExecutionException: org.apache.phoenix.exception.PhoenixIOException: Failed after attempts=36, exceptions: Tue May 29 20:12:28 AWST 2018, null, java.net.SocketTimeoutException: callTimeout=60000, callDuration=60202: row on table 'IDX_CONTRACT_CREATE_TIME' at region=IDX_CONTRACT_CREATE_TIME,\x03\x00\x00\x00\x00\x00\x00\x00\x00\x00,1524044495952.098bab978fb0bcfd2c4c31940cc1357a., hostname=hb-bp1ese450t849odg0-004.hbase.rds.aliyuncs.com,16020,1523854274747, seqNum=2 at java.util.concurrent.FutureTask.report(Unknown Source) at java.util.concurrent.FutureTask.get(Unknown Source) at org.apache.phoenix.iterate.BaseResultIterators.close(BaseResultIterators.java:1090) at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:1003) at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:890) at org.apache.phoenix.iterate.RoundRobinResultIterator.getIterators(RoundRobinResultIterator.java:176) at org.apache.phoenix.iterate.RoundRobinResultIterator.next(RoundRobinResultIterator.java:91) at org.apache.phoenix.iterate.DelegateResultIterator.next(DelegateResultIterator.java:44) at org.apache.phoenix.execute.TupleProjectionPlan$1.next(TupleProjectionPlan.java:62) at org.apache.phoenix.execute.HashJoinPlan$HashSubPlan.execute(HashJoinPlan.java:433) at org.apache.phoenix.execute.HashJoinPlan$1.call(HashJoinPlan.java:168) at org.apache.phoenix.execute.HashJoinPlan$1.call(HashJoinPlan.java:164) at java.util.concurrent.FutureTask.run(Unknown Source) at org.apache.phoenix.job.JobManager$InstrumentedJobFutureTask.run(JobManager.java:183) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Caused by: org.apache.phoenix.exception.PhoenixIOException: Failed after attempts=36, exceptions: Tue May 29 20:12:28 AWST 2018, null, java.net.SocketTimeoutException: callTimeout=60000, callDuration=60202: row on table 'IDX_CONTRACT_CREATE_TIME' at region=IDX_CONTRACT_CREATE_TIME,\x03\x00\x00\x00\x00\x00\x00\x00\x00\x00,1524044495952.098bab978fb0bcfd2c4c31940cc1357a., hostname=hb-bp1ese450t849odg0-004.hbase.rds.aliyuncs.com,16020,1523854274747, seqNum=2 at org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:120) at org.apache.phoenix.iterate.TableResultIterator.next(TableResultIterator.java:173) at org.apache.phoenix.iterate.LookAheadResultIterator$1.advance(LookAheadResultIterator.java:47) at org.apache.phoenix.iterate.LookAheadResultIterator.init(LookAheadResultIterator.java:59) at org.apache.phoenix.iterate.LookAheadResultIterator.peek(LookAheadResultIterator.java:73) at org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:128) at org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:113) ... 5 more Caused by: org.apache.phoenix.exception.PhoenixIOException: Failed after attempts=36, exceptions: Tue May 29 20:12:28 AWST 2018, null, java.net.SocketTimeoutException: callTimeout=60000, callDuration=60202: row on table 'IDX_CONTRACT_CREATE_TIME' at region=IDX_CONTRACT_CREATE_TIME,\x03\x00\x00\x00\x00\x00\x00\x00\x00\x00,1524044495952.098bab978fb0bcfd2c4c31940cc1357a., hostname=hb-bp1ese450t849odg0-004.hbase.rds.aliyuncs.com,16020,1523854274747, seqNum=2 at org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:120) at org.apache.phoenix.iterate.ScanningResultIterator.next(ScanningResultIterator.java:121) at org.apache.phoenix.iterate.TableResultIterator.next(TableResultIterator.java:166) ... 10 more Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=36, exceptions: Tue May 29 20:12:28 AWST 2018, null, java.net.SocketTimeoutException: callTimeout=60000, callDuration=60202: row on table 'IDX_CONTRACT_CREATE_TIME' at region=IDX_CONTRACT_CREATE_TIME,\x03\x00\x00\x00\x00\x00\x00\x00\x00\x00,1524044495952.098bab978fb0bcfd2c4c31940cc1357a., hostname=hb-bp1ese450t849odg0-004.hbase.rds.aliyuncs.com,16020,1523854274747, seqNum=2 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.throwEnrichedException(RpcRetryingCallerWithReadReplicas.java:271) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:210) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:60) at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:210) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:327) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:413) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:371) at org.apache.phoenix.iterate.ScanningResultIterator.next(ScanningResultIterator.java:112) ... 11 more Caused by: java.net.SocketTimeoutException: callTimeout=60000, callDuration=60202: row on table 'IDX_CONTRACT_CREATE_TIME' at region=IDX_CONTRACT_CREATE_TIME,\x03\x00\x00\x00\x00\x00\x00\x00\x00\x00,1524044495952.098bab978fb0bcfd2c4c31940cc1357a., hostname=hb-bp1ese450t849odg0-004.hbase.rds.aliyuncs.com,16020,1523854274747, seqNum=2 at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:169) at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:65) ... 3 more Caused by: java.io.IOException: Call to hb-proxy-pub-bp1ese450t849odg0-004.hbase.rds.aliyuncs.com/101.37.8.84:16020 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=88, waitTime=60043, operationTimeout=60000 expired. at org.apache.hadoop.hbase.ipc.AbstractRpcClient.wrapException(AbstractRpcClient.java:284) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1238) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:223) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:328) at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.scan(ClientProtos.java:32831) at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:215) at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:63) at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:210) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:364) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:338) at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:136) ... 4 more Caused by: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=88, waitTime=60043, operationTimeout=60000 expired. at org.apache.hadoop.hbase.ipc.Call.checkAndSetTimeout(Call.java:70) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1212) ... 13 more [WARN ][phoenix-1-thread-0] 2018-05-29 20:12:47.993 - ScannerCallable.close(367) | Ignore, probably already closed org.apache.hadoop.hbase.UnknownScannerException: org.apache.hadoop.hbase.UnknownScannerException: Name: 16764878 ,scanner = {"loadColumnFamiliesOnDemand":null,"startRow":"","stopRow":"","batch":-1,"cacheBlocks":true,"totalColumns":0,"maxResultSize":-1,"attributes":"{}","families":{},"caching":-1,"maxVersions":1,"timeRange":[0,9223372036854775807]} ,HRegion = 559038737, already closed? at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2865) at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32295) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2394) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:174) at org.apache.hadoop.hbase.ipc.RpcExecutor$CallHandler.run(RpcExecutor.java:178) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source) at java.lang.reflect.Constructor.newInstance(Unknown Source) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:327) at org.apache.hadoop.hbase.client.ScannerCallable.close(ScannerCallable.java:364) at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:197) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:145) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:60) at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:210) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:327) at org.apache.hadoop.hbase.client.ClientScanner.close(ClientScanner.java:748) at org.apache.phoenix.iterate.ScanningResultIterator.close(ScanningResultIterator.java:67) at org.apache.phoenix.iterate.TableResultIterator.close(TableResultIterator.java:145) at org.apache.phoenix.iterate.LookAheadResultIterator$1.close(LookAheadResultIterator.java:42) at org.apache.phoenix.iterate.BaseResultIterators.close(BaseResultIterators.java:1091) at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:1003) at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:890) at org.apache.phoenix.iterate.RoundRobinResultIterator.getIterators(RoundRobinResultIterator.java:176) at org.apache.phoenix.iterate.RoundRobinResultIterator.next(RoundRobinResultIterator.java:91) at org.apache.phoenix.iterate.DelegateResultIterator.next(DelegateResultIterator.java:44) at org.apache.phoenix.execute.TupleProjectionPlan$1.next(TupleProjectionPlan.java:62) at org.apache.phoenix.execute.HashJoinPlan$HashSubPlan.execute(HashJoinPlan.java:433) at org.apache.phoenix.execute.HashJoinPlan$1.call(HashJoinPlan.java:168) at org.apache.phoenix.execute.HashJoinPlan$1.call(HashJoinPlan.java:164) at java.util.concurrent.FutureTask.run(Unknown Source) at org.apache.phoenix.job.JobManager$InstrumentedJobFutureTask.run(JobManager.java:183) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.UnknownScannerException): org.apache.hadoop.hbase.UnknownScannerException: Name: 16764878 ,scanner = {"loadColumnFamiliesOnDemand":null,"startRow":"","stopRow":"","batch":-1,"cacheBlocks":true,"totalColumns":0,"maxResultSize":-1,"attributes":"{}","families":{},"caching":-1,"maxVersions":1,"timeRange":[0,9223372036854775807]} ,HRegion = 559038737, already closed? at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2865) at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32295) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2394) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:174) at org.apache.hadoop.hbase.ipc.RpcExecutor$CallHandler.run(RpcExecutor.java:178) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1234) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:223) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:328) at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.scan(ClientProtos.java:32831) at org.apache.hadoop.hbase.client.ScannerCallable.close(ScannerCallable.java:362) ... 24 more [WARN ][phoenix-1-thread-0] 2018-05-29 20:12:48.025 - ScannerCallable.close(367) | Ignore, probably already closed org.apache.hadoop.hbase.UnknownScannerException: org.apache.hadoop.hbase.UnknownScannerException: Name: 16764880 ,scanner = {"loadColumnFamiliesOnDemand":null,"startRow":"","stopRow":"","batch":-1,"cacheBlocks":true,"totalColumns":0,"maxResultSize":-1,"attributes":"{}","families":{},"caching":-1,"maxVersions":1,"timeRange":[0,9223372036854775807]} ,HRegion = 559038737, already closed? at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2865) at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32295) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2394) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:174) at org.apache.hadoop.hbase.ipc.RpcExecutor$CallHandler.run(RpcExecutor.java:178) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source) at java.lang.reflect.Constructor.newInstance(Unknown Source) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:327) at org.apache.hadoop.hbase.client.ScannerCallable.close(ScannerCallable.java:364) at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:197) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:145) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:60) at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:210) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:327) at org.apache.hadoop.hbase.client.ClientScanner.close(ClientScanner.java:748) at org.apache.phoenix.iterate.ScanningResultIterator.close(ScanningResultIterator.java:67) at org.apache.phoenix.iterate.TableResultIterator.close(TableResultIterator.java:145) at org.apache.phoenix.iterate.LookAheadResultIterator$1.close(LookAheadResultIterator.java:42) at org.apache.phoenix.iterate.BaseResultIterators.close(BaseResultIterators.java:1091) at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:1003) at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:890) at org.apache.phoenix.iterate.RoundRobinResultIterator.getIterators(RoundRobinResultIterator.java:176) at org.apache.phoenix.iterate.RoundRobinResultIterator.next(RoundRobinResultIterator.java:91) at org.apache.phoenix.iterate.DelegateResultIterator.next(DelegateResultIterator.java:44) at org.apache.phoenix.execute.TupleProjectionPlan$1.next(TupleProjectionPlan.java:62) at org.apache.phoenix.execute.HashJoinPlan$HashSubPlan.execute(HashJoinPlan.java:433) at org.apache.phoenix.execute.HashJoinPlan$1.call(HashJoinPlan.java:168) at org.apache.phoenix.execute.HashJoinPlan$1.call(HashJoinPlan.java:164) at java.util.concurrent.FutureTask.run(Unknown Source) at org.apache.phoenix.job.JobManager$InstrumentedJobFutureTask.run(JobManager.java:183) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.UnknownScannerException): org.apache.hadoop.hbase.UnknownScannerException: Name: 16764880 ,scanner = {"loadColumnFamiliesOnDemand":null,"startRow":"","stopRow":"","batch":-1,"cacheBlocks":true,"totalColumns":0,"maxResultSize":-1,"attributes":"{}","families":{},"caching":-1,"maxVersions":1,"timeRange":[0,9223372036854775807]} ,HRegion = 559038737, already closed? at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2865) at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32295) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2394) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:174) at org.apache.hadoop.hbase.ipc.RpcExecutor$CallHandler.run(RpcExecutor.java:178) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1234) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:223) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:328) at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.scan(ClientProtos.java:32831) at org.apache.hadoop.hbase.client.ScannerCallable.close(ScannerCallable.java:362) ... 24 more [WARN ][phoenix-1-thread-0] 2018-05-29 20:12:48.053 - ScannerCallable.close(367) | Ignore, probably already closed org.apache.hadoop.hbase.UnknownScannerException: org.apache.hadoop.hbase.UnknownScannerException: Name: 16764882 ,scanner = {"loadColumnFamiliesOnDemand":null,"startRow":"","stopRow":"","batch":-1,"cacheBlocks":true,"totalColumns":0,"maxResultSize":-1,"attributes":"{}","families":{},"caching":-1,"maxVersions":1,"timeRange":[0,9223372036854775807]} ,HRegion = 559038737, already closed? at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2865) at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32295) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2394) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:174) at org.apache.hadoop.hbase.ipc.RpcExecutor$CallHandler.run(RpcExecutor.java:178) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source) at java.lang.reflect.Constructor.newInstance(Unknown Source) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:327) at org.apache.hadoop.hbase.client.ScannerCallable.close(ScannerCallable.java:364) at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:197) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:145) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:60) at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:210) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:327) at org.apache.hadoop.hbase.client.ClientScanner.close(ClientScanner.java:748) at org.apache.phoenix.iterate.ScanningResultIterator.close(ScanningResultIterator.java:67) at org.apache.phoenix.iterate.TableResultIterator.close(TableResultIterator.java:145) at org.apache.phoenix.iterate.LookAheadResultIterator$1.close(LookAheadResultIterator.java:42) at org.apache.phoenix.iterate.BaseResultIterators.close(BaseResultIterators.java:1091) at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:1003) at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:890) at org.apache.phoenix.iterate.RoundRobinResultIterator.getIterators(RoundRobinResultIterator.java:176) at org.apache.phoenix.iterate.RoundRobinResultIterator.next(RoundRobinResultIterator.java:91) at org.apache.phoenix.iterate.DelegateResultIterator.next(DelegateResultIterator.java:44) at org.apache.phoenix.execute.TupleProjectionPlan$1.next(TupleProjectionPlan.java:62) at org.apache.phoenix.execute.HashJoinPlan$HashSubPlan.execute(HashJoinPlan.java:433) at org.apache.phoenix.execute.HashJoinPlan$1.call(HashJoinPlan.java:168) at org.apache.phoenix.execute.HashJoinPlan$1.call(HashJoinPlan.java:164) at java.util.concurrent.FutureTask.run(Unknown Source) at org.apache.phoenix.job.JobManager$InstrumentedJobFutureTask.run(JobManager.java:183) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.UnknownScannerException): org.apache.hadoop.hbase.UnknownScannerException: Name: 16764882 ,scanner = {"loadColumnFamiliesOnDemand":null,"startRow":"","stopRow":"","batch":-1,"cacheBlocks":true,"totalColumns":0,"maxResultSize":-1,"attributes":"{}","families":{},"caching":-1,"maxVersions":1,"timeRange":[0,9223372036854775807]} ,HRegion = 559038737, already closed? at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2865) at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32295) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2394) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:174) at org.apache.hadoop.hbase.ipc.RpcExecutor$CallHandler.run(RpcExecutor.java:178) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1234) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:223) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:328) at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.scan(ClientProtos.java:32831) at org.apache.hadoop.hbase.client.ScannerCallable.close(ScannerCallable.java:362) ... 24 more [INFO ][phoenix-1-thread-0] 2018-05-29 20:12:48.054 - BaseResultIterators.close(1096) | Failed to execute task during cancel java.util.concurrent.ExecutionException: org.apache.phoenix.exception.PhoenixIOException: Failed after attempts=36, exceptions: Tue May 29 20:12:28 AWST 2018, null, java.net.SocketTimeoutException: callTimeout=60000, callDuration=60206: row on table 'IDX_CONTRACT_CREATE_TIME' at region=IDX_CONTRACT_CREATE_TIME,\x07\x00\x00\x00\x00\x00\x00\x00\x00\x00,1524044495952.afa2f91cfec3726f168ae1777152d1e1., hostname=hb-bp1ese450t849odg0-003.hbase.rds.aliyuncs.com,16020,1523854368775, seqNum=2 at java.util.concurrent.FutureTask.report(Unknown Source) at java.util.concurrent.FutureTask.get(Unknown Source) at org.apache.phoenix.iterate.BaseResultIterators.close(BaseResultIterators.java:1090) at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:1003) at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:890) at org.apache.phoenix.iterate.RoundRobinResultIterator.getIterators(RoundRobinResultIterator.java:176) at org.apache.phoenix.iterate.RoundRobinResultIterator.next(RoundRobinResultIterator.java:91) at org.apache.phoenix.iterate.DelegateResultIterator.next(DelegateResultIterator.java:44) at org.apache.phoenix.execute.TupleProjectionPlan$1.next(TupleProjectionPlan.java:62) at org.apache.phoenix.execute.HashJoinPlan$HashSubPlan.execute(HashJoinPlan.java:433) at org.apache.phoenix.execute.HashJoinPlan$1.call(HashJoinPlan.java:168) at org.apache.phoenix.execute.HashJoinPlan$1.call(HashJoinPlan.java:164) at java.util.concurrent.FutureTask.run(Unknown Source) at org.apache.phoenix.job.JobManager$InstrumentedJobFutureTask.run(JobManager.java:183) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Caused by: org.apache.phoenix.exception.PhoenixIOException: Failed after attempts=36, exceptions: Tue May 29 20:12:28 AWST 2018, null, java.net.SocketTimeoutException: callTimeout=60000, callDuration=60206: row on table 'IDX_CONTRACT_CREATE_TIME' at region=IDX_CONTRACT_CREATE_TIME,\x07\x00\x00\x00\x00\x00\x00\x00\x00\x00,1524044495952.afa2f91cfec3726f168ae1777152d1e1., hostname=hb-bp1ese450t849odg0-003.hbase.rds.aliyuncs.com,16020,1523854368775, seqNum=2 at org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:120) at org.apache.phoenix.iterate.TableResultIterator.next(TableResultIterator.java:173) at org.apache.phoenix.iterate.LookAheadResultIterator$1.advance(LookAheadResultIterator.java:47) at org.apache.phoenix.iterate.LookAheadResultIterator.init(LookAheadResultIterator.java:59) at org.apache.phoenix.iterate.LookAheadResultIterator.peek(LookAheadResultIterator.java:73) at org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:128) at org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:113) ... 5 more Caused by: org.apache.phoenix.exception.PhoenixIOException: Failed after attempts=36, exceptions: Tue May 29 20:12:28 AWST 2018, null, java.net.SocketTimeoutException: callTimeout=60000, callDuration=60206: row on table 'IDX_CONTRACT_CREATE_TIME' at region=IDX_CONTRACT_CREATE_TIME,\x07\x00\x00\x00\x00\x00\x00\x00\x00\x00,1524044495952.afa2f91cfec3726f168ae1777152d1e1., hostname=hb-bp1ese450t849odg0-003.hbase.rds.aliyuncs.com,16020,1523854368775, seqNum=2 at org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:120) at org.apache.phoenix.iterate.ScanningResultIterator.next(ScanningResultIterator.java:121) at org.apache.phoenix.iterate.TableResultIterator.next(TableResultIterator.java:166) ... 10 more Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=36, exceptions: Tue May 29 20:12:28 AWST 2018, null, java.net.SocketTimeoutException: callTimeout=60000, callDuration=60206: row on table 'IDX_CONTRACT_CREATE_TIME' at region=IDX_CONTRACT_CREATE_TIME,\x07\x00\x00\x00\x00\x00\x00\x00\x00\x00,1524044495952.afa2f91cfec3726f168ae1777152d1e1., hostname=hb-bp1ese450t849odg0-003.hbase.rds.aliyuncs.com,16020,1523854368775, seqNum=2 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.throwEnrichedException(RpcRetryingCallerWithReadReplicas.java:271) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:210) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:60) at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:210) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:327) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:413) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:371) at org.apache.phoenix.iterate.ScanningResultIterator.next(ScanningResultIterator.java:112) ... 11 more Caused by: java.net.SocketTimeoutException: callTimeout=60000, callDuration=60206: row on table 'IDX_CONTRACT_CREATE_TIME' at region=IDX_CONTRACT_CREATE_TIME,\x07\x00\x00\x00\x00\x00\x00\x00\x00\x00,1524044495952.afa2f91cfec3726f168ae1777152d1e1., hostname=hb-bp1ese450t849odg0-003.hbase.rds.aliyuncs.com,16020,1523854368775, seqNum=2 at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:169) at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:65) ... 3 more Caused by: java.io.IOException: Call to hb-proxy-pub-bp1ese450t849odg0-003.hbase.rds.aliyuncs.com/101.37.143.39:16020 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=80, waitTime=60046, operationTimeout=59999 expired. at org.apache.hadoop.hbase.ipc.AbstractRpcClient.wrapException(AbstractRpcClient.java:284) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1238) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:223) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:328) at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.scan(ClientProtos.java:32831) at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:215) at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:63) at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:210) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:364) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:338) at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:136) ... 4 more Caused by: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=80, waitTime=60046, operationTimeout=59999 expired. at org.apache.hadoop.hbase.ipc.Call.checkAndSetTimeout(Call.java:70) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1212) ... 13 more [WARN ][phoenix-1-thread-0] 2018-05-29 20:12:48.083 - ScannerCallable.close(367) | Ignore, probably already closed org.apache.hadoop.hbase.UnknownScannerException: org.apache.hadoop.hbase.UnknownScannerException: Name: 16764879 ,scanner = {"loadColumnFamiliesOnDemand":null,"startRow":"","stopRow":"","batch":-1,"cacheBlocks":true,"totalColumns":0,"maxResultSize":-1,"attributes":"{}","families":{},"caching":-1,"maxVersions":1,"timeRange":[0,9223372036854775807]} ,HRegion = 559038737, already closed? at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2865) at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32295) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2394) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:174) at org.apache.hadoop.hbase.ipc.RpcExecutor$CallHandler.run(RpcExecutor.java:178) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source) at java.lang.reflect.Constructor.newInstance(Unknown Source) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:327) at org.apache.hadoop.hbase.client.ScannerCallable.close(ScannerCallable.java:364) at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:197) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:145) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:60) at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:210) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:327) at org.apache.hadoop.hbase.client.ClientScanner.close(ClientScanner.java:748) at org.apache.phoenix.iterate.ScanningResultIterator.close(ScanningResultIterator.java:67) at org.apache.phoenix.iterate.TableResultIterator.close(TableResultIterator.java:145) at org.apache.phoenix.iterate.LookAheadResultIterator$1.close(LookAheadResultIterator.java:42) at org.apache.phoenix.iterate.BaseResultIterators.close(BaseResultIterators.java:1091) at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:1003) at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:890) at org.apache.phoenix.iterate.RoundRobinResultIterator.getIterators(RoundRobinResultIterator.java:176) at org.apache.phoenix.iterate.RoundRobinResultIterator.next(RoundRobinResultIterator.java:91) at org.apache.phoenix.iterate.DelegateResultIterator.next(DelegateResultIterator.java:44) at org.apache.phoenix.execute.TupleProjectionPlan$1.next(TupleProjectionPlan.java:62) at org.apache.phoenix.execute.HashJoinPlan$HashSubPlan.execute(HashJoinPlan.java:433) at org.apache.phoenix.execute.HashJoinPlan$1.call(HashJoinPlan.java:168) at org.apache.phoenix.execute.HashJoinPlan$1.call(HashJoinPlan.java:164) at java.util.concurrent.FutureTask.run(Unknown Source) at org.apache.phoenix.job.JobManager$InstrumentedJobFutureTask.run(JobManager.java:183) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.UnknownScannerException): org.apache.hadoop.hbase.UnknownScannerException: Name: 16764879 ,scanner = {"loadColumnFamiliesOnDemand":null,"startRow":"","stopRow":"","batch":-1,"cacheBlocks":true,"totalColumns":0,"maxResultSize":-1,"attributes":"{}","families":{},"caching":-1,"maxVersions":1,"timeRange":[0,9223372036854775807]} ,HRegion = 559038737, already closed? at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2865) at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32295) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2394) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:174) at org.apache.hadoop.hbase.ipc.RpcExecutor$CallHandler.run(RpcExecutor.java:178) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1234) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:223) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:328) at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.scan(ClientProtos.java:32831) at org.apache.hadoop.hbase.client.ScannerCallable.close(ScannerCallable.java:362) ... 24 more [WARN ][phoenix-1-thread-0] 2018-05-29 20:12:48.110 - ScannerCallable.close(367) | Ignore, probably already closed org.apache.hadoop.hbase.UnknownScannerException: org.apache.hadoop.hbase.UnknownScannerException: Name: 66600 ,scanner = {"loadColumnFamiliesOnDemand":null,"startRow":"","stopRow":"","batch":-1,"cacheBlocks":true,"totalColumns":0,"maxResultSize":-1,"attributes":"{}","families":{},"caching":-1,"maxVersions":1,"timeRange":[0,9223372036854775807]} ,HRegion = 559038737, already closed? at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2865) at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32295) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2394) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:174) at org.apache.hadoop.hbase.ipc.RpcExecutor$CallHandler.run(RpcExecutor.java:178) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source) at java.lang.reflect.Constructor.newInstance(Unknown Source) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:327) at org.apache.hadoop.hbase.client.ScannerCallable.close(ScannerCallable.java:364) at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:197) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:145) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:60) at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:210) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:327) at org.apache.hadoop.hbase.client.ClientScanner.close(ClientScanner.java:748) at org.apache.phoenix.iterate.ScanningResultIterator.close(ScanningResultIterator.java:67) at org.apache.phoenix.iterate.TableResultIterator.close(TableResultIterator.java:145) at org.apache.phoenix.iterate.LookAheadResultIterator$1.close(LookAheadResultIterator.java:42) at org.apache.phoenix.iterate.BaseResultIterators.close(BaseResultIterators.java:1091) at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:1003) at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:890) at org.apache.phoenix.iterate.RoundRobinResultIterator.getIterators(RoundRobinResultIterator.java:176) at org.apache.phoenix.iterate.RoundRobinResultIterator.next(RoundRobinResultIterator.java:91) at org.apache.phoenix.iterate.DelegateResultIterator.next(DelegateResultIterator.java:44) at org.apache.phoenix.execute.TupleProjectionPlan$1.next(TupleProjectionPlan.java:62) at org.apache.phoenix.execute.HashJoinPlan$HashSubPlan.execute(HashJoinPlan.java:433) at org.apache.phoenix.execute.HashJoinPlan$1.call(HashJoinPlan.java:168) at org.apache.phoenix.execute.HashJoinPlan$1.call(HashJoinPlan.java:164) at java.util.concurrent.FutureTask.run(Unknown Source) at org.apache.phoenix.job.JobManager$InstrumentedJobFutureTask.run(JobManager.java:183) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.UnknownScannerException): org.apache.hadoop.hbase.UnknownScannerException: Name: 66600 ,scanner = {"loadColumnFamiliesOnDemand":null,"startRow":"","stopRow":"","batch":-1,"cacheBlocks":true,"totalColumns":0,"maxResultSize":-1,"attributes":"{}","families":{},"caching":-1,"maxVersions":1,"timeRange":[0,9223372036854775807]} ,HRegion = 559038737, already closed? at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2865) at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32295) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2394) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:174) at org.apache.hadoop.hbase.ipc.RpcExecutor$CallHandler.run(RpcExecutor.java:178) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1234) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:223) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:328) at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.scan(ClientProtos.java:32831) at org.apache.hadoop.hbase.client.ScannerCallable.close(ScannerCallable.java:362) ... 24 more [WARN ][phoenix-1-thread-0] 2018-05-29 20:12:48.138 - ScannerCallable.close(367) | Ignore, probably already closed org.apache.hadoop.hbase.UnknownScannerException: org.apache.hadoop.hbase.UnknownScannerException: Name: 66598 ,scanner = {"loadColumnFamiliesOnDemand":null,"startRow":"","stopRow":"","batch":-1,"cacheBlocks":true,"totalColumns":0,"maxResultSize":-1,"attributes":"{}","families":{},"caching":-1,"maxVersions":1,"timeRange":[0,9223372036854775807]} ,HRegion = 559038737, already closed? at org.apache.hadoop.hbase.regionserver.RSRpcServices.s