How to stop a streaming pipeline in google cloud dataflow

问题

I have a Streaming dataflow running to read the PUB/SUB subscription.

After a period of a time or may be after processing certain amount of data, i want the pipeline to stop by itself. I don't want my compute engine instance to be running indefinitely.

When i cancel the job through dataflow console, it is shown as failed job.

Is there a way to achieve this? am i missing something ? Or that feature is missing in the API.

回答1:

Could you do something like this?

Pipeline pipeline = ...;
... (construct the streaming pipeline) ...
final DataflowPipelineJob job =
    DataflowPipelineRunner.fromOptions(pipelineOptions)
                          .run(pipeline);
Thread.sleep(your timeout);
job.cancel();

回答2:

I was able to drain (canceling a job without losing data) a running streaming job on data flow with Rest API.

See my answer

Use Rest Update method, with this body :

{ "requestedState": "JOB_STATE_DRAINING" }

来源：https://stackoverflow.com/questions/30859829/how-to-stop-a-streaming-pipeline-in-google-cloud-dataflow