1. 技术文章
  2. [Hive]I got “ArrayIndexOutOfBoundsException” while I query the hive database

[Hive]I got “ArrayIndexOutOfBoundsException” while I query the hive database

由 ℡╲_俬逩灬. 提交于 2021-02-10 09:25:21

问题


I always get "ArrayIndexOutOfBoundsException" while I query the hive base(both hive-0.11.0 and hive-0.12.0), but sometimes not. Here is the error

java.lang.RuntimeException: Hive Runtime Error while closing operators: java.lang.ArrayIndexOutOfBoundsException: 0
at org.apache.hadoop.hive.ql.exec.mr.ExecReducer.close(ExecReducer.java:313)
at org.apache.hadoop.io.IOUtils.cleanup(IOUtils.java:232)
at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:539)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:421)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.ArrayIndexOutOfBoundsException: 0
    at org.apache.hadoop.hive.ql.exec.persistence.RowContainer.first(RowContainer.java:231)
    at org.apache.hadoop.hive.ql.exec.persistence.RowContainer.first(RowContainer.java:74)
    at org.apache.hadoop.hive.ql.exec.CommonJoinOperator.genUniqueJoinObject(CommonJoinOperator.java:645)
    at org.apache.hadoop.hive.ql.exec.CommonJoinOperator.checkAndGenObject(CommonJoinOperator.java:758)
    at org.apache.hadoop.hive.ql.exec.JoinOperator.endGroup(JoinOperator.java:257)
    at org.apache.hadoop.hive.ql.exec.mr.ExecReducer.close(ExecReducer.java:298)
    ... 8 more
Caused by: java.lang.ArrayIndexOutOfBoundsException: 0
    at org.apache.hadoop.hive.ql.exec.persistence.RowContainer.first(RowContainer.java:220)
    ... 13 more

Could someone help me?

Update my code:

    Select distinct jabberUseragent.gameID,agentPlayInfo.gameLabel,jabberUseragent.userAgent,CONCAT(CONCAT(CONCAT(triggerUsageStart.generateDate,' '),triggerUsageStart.timezone),CONCAT(' ',triggerUsageStart.generateTime)) as generateDate,(unix_timestamp(CONCAT(CONCAT(triggerUsageStop.generateDate,' '),triggerUsageStop.generateTime)) - unix_timestamp(CONCAT(CONCAT(triggerUsageStart.generateDate,' '),triggerUsageStart.generateTime))) from
    (Select gameSession,gameID,userAgent from(Select distinct regexp_extract(t.payload,'playRequestId:(.*), playRequest',1) as gameSession,regexp_extract(t.payload,'gameId:(.*), userAgent:',1) as gameID,regexp_extract(t.payload,', userAgent:(.*), agentLocation',1) as userAgent,payload from (select * from ${hiveconf:DATA_BASE} p where p.dt >= '${hiveconf:LOW_DATE}' and p.dt <= '${hiveconf:UPPER_DATE}')  t where CONCAT(t.generatedate,t.generatetime) >= CONCAT('${hiveconf:LOW_DATE}','${hiveconf:LOW_TIME}') and CONCAT(t.generatedate,t.generatetime) <= CONCAT('${hiveconf:UPPER_DATE}','${hiveconf:UPPER_TIME}'))jabberUseragent where jabberUseragent.gameSession!='' and jabberUseragent.userAgent!='') jabberUseragent
join
    (Select gameID,gameLabel from(Select distinct regexp_extract(t.payload,'gameId=(.*),gameLabel=.*,configFilePath',1) as gameID,regexp_extract(t.payload,'gameId=.*,gameLabel=(.*),configFilePath',1) as gameLabel,payload from (select * from ${hiveconf:DATA_BASE} p where p.dt >= '${hiveconf:LOW_DATE}' and p.dt <= '${hiveconf:UPPER_DATE}')  t where CONCAT(t.generatedate,t.generatetime) >= CONCAT('${hiveconf:LOW_DATE}','${hiveconf:LOW_TIME}') and CONCAT(t.generatedate,t.generatetime) <= CONCAT('${hiveconf:UPPER_DATE}','${hiveconf:UPPER_TIME}'))agentPlayInfo where agentPlayInfo.gameID!='' and agentPlayInfo.gameLabel!='') agentPlayInfo
join
    (Select gameSession,generateDate,generateTime,timezone,payload from(Select distinct regexp_extract(t.payload,'GAME_SESSION=.*((.{8})-(.{4})-(.{4})-(.{4})-(.{12}))\" USAGE=\"([\\w\\-\\(\\)\\.]*,){41}9.*\"',1) as gameSession,generateDate,generateTime,timezone,payload from (select * from ${hiveconf:DATA_BASE} p where p.dt >= '${hiveconf:LOW_DATE}' and p.dt <= '${hiveconf:UPPER_DATE}')  t where t.payload like '%[e] usage_record%' and CONCAT(t.generatedate,t.generatetime) <= CONCAT('${hiveconf:UPPER_DATE}','${hiveconf:UPPER_TIME}') and CONCAT(t.generatedate,t.generatetime) >= CONCAT('${hiveconf:LOW_DATE}','${hiveconf:LOW_TIME}'))triggerStart where triggerStart.gameSession!='')triggerUsageStart
join
    (Select gameSession,generateDate,generateTime,timezone,payload from(Select distinct regexp_extract(t.payload,'GAME_SESSION=.*((.{8})-(.{4})-(.{4})-(.{4})-(.{12}))\" USAGE=\"([\\w\\-\\(\\)\\.]*,){41}[1-5].*\"',1) as gameSession,generateDate,generateTime,timezone,payload from (select * from ${hiveconf:DATA_BASE} p where p.dt >= '${hiveconf:LOW_DATE}' and p.dt <= '${hiveconf:UPPER_DATE}')  t where t.payload like '%[e] usage_record%' and CONCAT(t.generatedate,t.generatetime) <= CONCAT('${hiveconf:UPPER_DATE}','${hiveconf:UPPER_TIME}') and CONCAT(t.generatedate,t.generatetime) >= CONCAT('${hiveconf:LOW_DATE}','${hiveconf:LOW_TIME}'))triggerStop where triggerStop.gameSession!='')triggerUsageStop
on (jabberUseragent.gameSession = triggerUsageStart.gameSession and triggerUsageStart.gameSession = triggerUsageStop.gameSession and jabberUseragent.gameID = agentPlayInfo.gameID) order by generateDate

Sorry, I can't share my samples. By the way, I've also got this exception before I got "ArrayIndexOutOfBoundException"

javax.jdo.JDODataStoreException: Error executing SQL query "select PARTITIONS.PART_ID from PARTITIONS  inner join TBLS on PARTITIONS.TBL_ID = TBLS.TBL_ID   inner join DBS on TBLS.DB_ID = DBS.DB_ID  where TBLS.TBL_NAME = ? and DBS.NAME = ? and PARTITIONS.PART_NAME in (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)".
at org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:451)
at org.datanucleus.api.jdo.JDOQuery.executeWithArray(JDOQuery.java:321)
at org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getPartitionsViaSqlFilterInternal(MetaStoreDirectSql.java:181)
at org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getPartitionsViaSqlFilter(MetaStoreDirectSql.java:82)
at org.apache.hadoop.hive.metastore.ObjectStore.getPartitionsByNamesInternal(ObjectStore.java:1717)
at org.apache.hadoop.hive.metastore.ObjectStore.getPartitionsByNames(ObjectStore.java:1700)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)......

NestedThrowablesStackTrace:
org.postgresql.util.PSQLException: ERROR: relation "partitions" does not exist
    at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:1591)
    at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1340)
    at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:192)
    at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:471)
    at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:373)
    at org.postgresql.jdbc2.AbstractJdbc2Statement.executeQuery(AbstractJdbc2Statement.java:258)......

回答1:


Based on the information provided, only this can be a sensible solution to your problem.

I have put the method definition for reference. Please go through it to understand

If you run down the source code closely, there are two areas / possibilities where ArrayIndexOutOfBoundException can be thrown.

  • Accessing the array values of input splits from Configuration
  • Reading the Row from currentReadBlock array (this is mostly not the case for the exception since it's size is greater than 0)

Please check your set of input files for job because InputFormat#split() method returns an array of InputSplit type. Each InputSplit is then assigned to an individual Mapper for processing. Mostly, the exception occurs while accessing this InputSplit[] array.



来源：https://stackoverflow.com/questions/20896587/hivei-got-arrayindexoutofboundsexception-while-i-query-the-hive-database

标签
java
Hive
易学教程内所有资源均来自网络或用户发布的内容，如有违反法律规定的内容欢迎反馈！
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!