Perform action after Dataflow pipeline has processed all data

Question

Is it possible to perform an action once a batch Dataflow job has finished processing all data? Specifically, I\'d like to move the text file that the pipeline just processe

Answer 1

I think two options can help you here:

1) Use TextIO to write to the bucket or folder you want, specifying the exact GCS path (for e.g. gs://sandbox/other-bucket)

2) Use Object Change Notifications in combination with Cloud Functions. You can find a good primer on doing this here and the SDK for GCS in JS here. What you will do in this option is basically setting up a trigger when something drops in a certain bucket, and move it to another one using your self-written Cloud Function.