Unable to create Pipeline with PubsubIO.Read, Set up of resource … failed

问题

My test pipeline is really simple. It attempts to read from a topic created via the console.

public static void main(String[] args) throws IOException {

    Options options = PipelineOptionsFactory.fromArgs(args).
            withValidation().as(Options.class);

    options.setStreaming(true);

    Pipeline pipeline = Pipeline.create(options);

    PCollection<String> input = pipeline
            .apply(PubsubIO.Read.topic(options.getPubsubTopic()))
            .apply(ParDo.of(new ExtractEvents()));

    pipeline.run();

}

When I attempt to execute this pipeline I get the following error:

Workflow failed. Causes: (de5f777e2e08c1d9): Step setup_resource_additionaltopic.subscription-375367840492394866711: Set up of resource additionaltopic.subscription-3753678404923948667 failed

The Dataflow console also reports an internal error:

I can't find anything in the documentation and my trial and error attempts at resolving this haven't been successful.

Solution

To run a Dataflow job, a project must enable the following Google Cloud Platform APIs:

Google Cloud Dataflow API
Compute Engine API (Google Compute Engine)
Google Cloud Logging API
Google Cloud Storage
Google Cloud Storage JSON API
BigQuery API
Google Cloud Pub/Sub
Google Cloud Datastore API

You can use the Google Cloud Platform Console to enable all the required APIs at once.

回答1:

It looks like your project doesn't have the Pubsub API enabled. Have you gone through the instructions in the getting started guide, especially the part on APIs. There is a link there that should enable everything you need to get going.

You may also need to verify that the APIs and permissions are properly setup for the project that the Dataflow job is running in to access the Pubsub topic you are trying to subscribe to.

来源：https://stackoverflow.com/questions/36187665/unable-to-create-pipeline-with-pubsubio-read-set-up-of-resource-failed

标签

google-cloud-dataflow

google-cloud-pubsub