Apache beam and BigQuery

问题

I'm trying to execute apache beam sdk 2.4 and the libraries com.google.cloud.bigquery but thow exception

Exception in thread main java.lang.NoSuchMethodError com.google.api.client.googleapis.services.json.AbstractGoogleJsonClient$Builder.setBatchPath(LjavalangString;)LcomgoogleapiclientgoogleapisservicesAbstractGoogleClient$Builder;
at com.google.api.services.bigquery.Bigquery$Builder.setBatchPath(Bigquery.java3519)

import com.google.cloud.bigquery.*;

<dependency>
    <groupId>com.google.cloud.dataflow</groupId>
    <artifactId>google-cloud-dataflow-java-sdk-all</artifactId>
    <version>[2.4.0, 2.99)</version>
</dependency>

<!-- slf4j API frontend binding with JUL backend -->
<dependency>
    <groupId>org.slf4j</groupId>
    <artifactId>slf4j-api</artifactId>
    <versi

on>1.7.14</version>
    </dependency>

   <dependency>
        <groupId>org.slf4j</groupId>
        <artifactId>slf4j-jdk14</artifactId>
        <version>1.7.14</version>
    </dependency>

    <dependency>
        <groupId>com.google.cloud</groupId>
        <artifactId>google-cloud-bigquery</artifactId>
        <version>1.35.0</version>
    </dependency>

</dependencies>

the code example apache beam 2.4 using

import com.google.api.services.bigquery.model.TableFieldSchema;
import com.google.api.services.bigquery.model.TableRow;
import com.google.api.services.bigquery.model.TableSchema;

If I use the the libraries om.google.api.services.bigquery. the code example run successfully .

What library is it using internally?

What library should I use? com.google.api.services.bigquery or com.google.cloud.bigquery?

回答1:

Beam 2.4 is using com.google.api.services.bigquery v2-rev374-1.22.0. This is also the library that you should use with Beam 2.4. The com.google.api.services.bigquery library is out of date and there is an open issue to upgrade it. Unfortunately other components in Beam depend on libraries which are incompatible with com.google.cloud.bigquery, and those need to be upgraded before the new library can be used.

If you aren't already doing so, you should take a look at the BigQuery IO, which provides a wrapper to common BigQuery operations for Beam.

来源：https://stackoverflow.com/questions/51080969/apache-beam-and-bigquery

标签

java

google-bigquery

apache-beam