Hive query execution: Failed with exception java.io.IOException:org.apache.avro.AvroTypeException: Found double, expecting union

问题

I am trying to execute a simple

select * from table limit 1;

Statement in hive on an external table. But facing failue with execption:

java.io.IOException:org.apache.avro.AvroTypeException: Found double, expecting union.

Can Someone help me understand what this means? I have checked the schema file and the "default":null is already given.

What is the exact reason for this exception occuring?

I tried understanding an already existing discussion.

The schema looks something like this:

{"type":"record",
 "name":"VISIBILITY",
 "namespace":"pentaho_etl",
 "fields":[ {"name":"ID",               "type":["null","long"],     "default":null},
            {"name":"VERSION_ID",       "type":["null","long"],     "default":null},
            {"name":"IMP_CLASS",        "type":["null","string"],   "default":null},
            {"name":"NAME",             "type":["null","string"],   "default":null},
            {"name":"DL_SCHEMA_VERSN",  "type":["null","long"],     "default":null},
            {"name":"DL_FEED_TM",       "type":["null","string"],   "default":null},
            {"name":"DL_WFID",          "type":["null","long"],     "default":null},
            {"name":"DL_ENT_NM",        "type":["null","string"],   "default":null},
            {"name":"DL_JID",           "type":["null","long"],     "default":null},
            {"name":"DL_DATASET_ID",    "type":["null","long"],     "default":null},
            {"name":"DL_FD_DT",         "type":["null","string"],   "default":null}
        ]
}

回答1:

I know, it is old question, but it is unanswered, hence, adding little information that I know about this issue, so that, it helps others.

This is very common issue, we face due to mismatch between the table schema and schema contained in the avro file.

Though, avro file is binary, you can open it in any text editor and get its schema from initial few lines. This schema is not matching with the schema of the table, which you can get using show create table table_name. Then you can compare these 2 schemas column by column to find any discrepancies e.g. your avro is showing a column as double but your schema might have it as string.

来源：https://stackoverflow.com/questions/45865899/hive-query-execution-failed-with-exception-java-io-ioexceptionorg-apache-avro

标签

java

apache

Hadoop

Hive