apache-beam

I need to read a file from a GCS bucket. I know I'll have to use GCS API/Client Libraries but I cannot find any example related to it. I have been referring to this link in the GCS documentation: GCS Client Libraries . But couldn't really make a dent. If anybody can provide an example that would really help. Thanks. OK. If you want to simply read files from GCS, not as a PCollection but as regular files, and if you are having trouble with the GCS Java client libraries, you can also use the Apache Beam FileSystems API: First, you need to make sure that you have a Maven dependency in your pom

I have a directory on GCS or another supported filesystem to which new files are being written by an external process. I would like to write an Apache Beam streaming pipeline that continuously watches this directory for new files and reads and processes each new file as it arrives. Is this possible? This is possible starting with Apache Beam 2.2.0. Several APIs support this use case: If you're using TextIO or AvroIO , they support this explicitly via TextIO.read().watchForNewFiles() and the same on readAll() , for example: PCollection<String> lines = p.apply(TextIO.read() .from("gs://path/to

Suppose I have a PCollection<Foo> and I want to write it to multiple BigQuery tables, choosing a potentially different table for each Foo . How can I do this using the Apache Beam BigQueryIO API? This is possible using a feature recently added to BigQueryIO in Apache Beam. PCollection<Foo> foos = ...; foos.apply(BigQueryIO.write().to(new SerializableFunction<ValueInSingleWindow<Foo>, TableDestination>() { @Override public TableDestination apply(ValueInSingleWindow<Foo> value) { Foo foo = value.getValue(); // Also available: value.getWindow(), getTimestamp(), getPane() String tableSpec = ...;