Writing to Google Cloud Storage from PubSub using Cloud Dataflow using DoFn

Question

I am trying write Google PubSub messages to Google Cloud Storage using Google Cloud Dataflow. I know that TextIO/AvroIO do not support streaming pipelines. However, I read i

Answer 1

There's a gotcha here, which is that you'll need a GroupByKey in order for the panes to be aggregated appropriate. The Spotify example references this as "Materialization of panes is done in “Aggregate Events” transform which is nothing else than a GroupByKey transform", but it's a subtle point. You'll need to provide a key in order to do this, and in your case, it appears a constant value will work.

  PCollection<String> streamData = p.apply(readFromPubsub);
  PCollection<KV<String, String>> keyedStream =
        streamData.apply(WithKeys.of(new SerializableFunction<String, String>() {
           public Integer apply(String s) { return "constant"; } }));

At this point, you can apply your windowing function, and then a final GroupByKey to get the desired behavior:

  PCollection<String, Iterable<String>> keyedWindows = keyedStream.apply(...)
       .apply(GroupByKey.create());
  PCollection<Iterable<String>> windows = keyedWindows
       .apply(Values.<Iterable<String>>create());

Now the elements in processElement will be Iterable<String>, with size 100 or more.

We've filed https://issues.apache.org/jira/browse/BEAM-184 to make this behavior clearer.

Answer 2

As of Beam 2.0, TextIO/AvroIO do support writing unbounded collections - see documentation, in particular, you have to specify withWindowedWrites().