apache-beam

问题 Is there a way to export Google Cloud Storage files to Google Drive using Python? I was doing the Google Dataflow tutorial using the Google Shell, which is basically a single apache_beam command. I noticed that the command takes an output destination which is a Google Storage location. I wanted to know if, after the command is run, does Google provide a way to take the output and export it to Google Drive. 回答1: Google does not have any mechanism in place for developers to allow for movement

问题 I have built a small function in Apache Beam to perform a lookup/join: given a collection mapping A to B, and another collection mapping B to C, return a collection mapping A to C. class Main { private static <A,B,C> PCollection<KV<A,C>> lookup( PCollection<KV<A,B>> collection, PCollection<KV<B,C>> lookup ){ var leftTag = new TupleTag<A>(); var rightTag = new TupleTag<C>(); return KeyedPCollectionTuple.of(leftTag, collection.apply(KvSwap.create())) .and(rightTag, lookup) .apply(CoGroupByKey

问题 I'm writing data to BigQuery and successfully gets written there. But I'm concerned with the format in which it is getting written. Below is the format in which the data is shown when I execute any query in BigQuery : Check the first row, the value of SalesComponent is CPS_H but its showing 'BeamRecord [dataValues=[CPS_H' and In the ModelIteration the value is ended with a square braket. Below is the code that is used to push data to BigQuery from BeamSql: TableSchema tableSchema = new

问题 My goal is to benchmark the latency and the throughput of Apache Beam on a streaming data use-case with different window queries. I want to create my own data with an on-the-fly data generator to control the data generation rate manually and consume this data directly from a pipeline without a pub/sub mechanism, i.e. I don't want to read the data from a broker, etc. to avoid bottlenecks. Is there a way of doing something similar to what I want to achieve? or is there any source code for such

问题 This is my code to read csv //DataflowPipelineOptions options = PipelineOptionsFactory.as(DataflowPipelineOptions.class); PipelineOptions options=PipelineOptionsFactory.create(); //options.setProject("ProjectId"); //options.setStagingLocation("gs://bucketname/Object"); options.setRunner(DirectRunner.class); options.setTempLocation("gs://bucketname/Object"); Pipeline p = Pipeline.create(options); p.apply(FileIO.match().filepattern("gs://bucketname/objectname.csv")).apply(FileIO.readMatches())

问题 I've been working with the Go Beam SDK (v2.13.0) and can't get the wordcount example working on GCP Dataflow. It enters crash loop trying to start the org.apache.beam.runners.dataflow.worker.DataflowRunnerHarness . The example is executing correctly when run locally using the Direct runner. The example was completely unmodified from the original example given above. The stack trace is: org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.InvalidProtocolBufferException: Protocol message had

问题 How to write messages received from PubSub to a text file in GCS using TextIO in Apache Beam? Saw some methods like withWindowedWrites() and withFilenamePolicy() but couldn't find any example of it in the documentation. 回答1: Here is an example provided you are using the Java SDK (BEAM 2.1.0). PipelineOptions options = PipelineOptionsFactory.fromArgs(args) .withValidation() .as(PipelineOptions.class); Pipeline pipeline = Pipeline.create(options); pipeline.begin() .apply("PubsubIO",PubsubIO

问题 Thanks in advance! [+] Issue: I have a lot of files on google cloud, for every file I have to: get the file Make a bunch of Google-Cloud-Storage API calls on each file to index it(e.g. name = blob.name, size = blob.size) unzip it search for stuff in there put the indexing information + stuff found inside file in a BigQuery Table I've been using python2.7 and the Google-Cloud-SDK. This takes hours if I run it linearly. I was suggested Apache Beam/DataFlow to process in parallel. [+] What I've

问题 I don't found how get tables from a dataset specified. I want use Dataflow for migrate tables since Dataset US to dataset location EU. I would like get all tables in paralel process of dataset US and write the tables in dataset EU. Beam 2.4 is using com.google.api.services.bigquery v2-rev374-1.22.0. This is also the library that you should use with Beam 2.4. The code run successfully with DirectRunner but If I run with DataflowRunner doesn't run and throw the error un 29, 2018 1:52:48 PM com

问题 Currently I am using the gcs-text-to-bigquery google provided template and feeding in a transform function to transform my jsonl file. The jsonl is pretty nested and i wanted to be able to output multiple rows per one row of the newline delimited json by doing some transforms. For example: {'state': 'FL', 'metropolitan_counties':[{'name': 'miami dade', 'population':100000}, {'name': 'county2', 'population':100000}…], 'rural_counties':{'name': 'county1', 'population':100000}, {'name': 'county2