apache-beam

问题 Say we have one worker with 4 CPU cores. How does parallelism configured in Dataflow worker machines? Do we parallelize beyond # of cores? Where would this type of information be available? 回答1: One worker thread is used per core, and each worker thread independently processes a chunk of the input space. 来源： https://stackoverflow.com/questions/47777639/google-cloud-dataflow-worker-threading

问题 As I have a working static Dataflow running, I'd like to create a template from this one to let me easily reuse the Dataflow without any command line typing. Following the Creating Templates tutorial from the official doesn't provide a sample for templatable output. My Dataflow ends with a BigQuery sink which takes a few arguments like the target table for storage. This exact parameter is the one I'd like to make available in my template allowing me to choose the target storage after running

问题 I'm using Dataflow 0.5.5 Python. Ran into the following error in very simple code: print(len(row_list)) row_list is a list. Exactly the same code, same data and same pipeline runs perfectly fine on DirectRunner, but throws the following exception on DataflowRunner. What does it mean and how I can solve it? job name: `beamapp-root-0216042234-124125` (f14756f20f567f62): Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/dataflow_worker/batchworker.py", line 544, in

问题 I have tried the example code of SortValues transform using DirectRunner on local machine (Windows) PCollection<KV<String, KV<String, Integer>>> input = ... PCollection<KV<String, Iterable<KV<String, Integer>>>> grouped = input.apply(GroupByKey.<String, KV<String, Integer>>create()); PCollection<KV<String, Iterable<KV<String, Integer>>>> groupedAndSorted = grouped.apply(SortValues.<String, String, Integer>create(BufferedExternalSorter.options())); but I got the error

问题 In Eclipse i am running WordCount Dataflow pipeline. Running locally works. But switching to Cloud i get the error: Caused by: java.lang.IllegalArgumentException: Missing object or bucket in path: 'gs://tough-shard-129113/', did you mean: 'gs://some-bucket/tough-shard-129113'? Of course the bucket exists. Any suggestion? I use Java8. thanks. 回答1: Ok. I managed to make it work. I unchecked the flag "Use Default Dataflow options". Thanks all for support. – 回答2: After uncheck flag "Use Default

问题 If we have a stream that looks like this Person { … OrganizationID } that we want to join with another stream Organization { ID … } to create a composite record like so: Person { … Organization { ID … } } What is the most idiomatic and efficient way to do so in the Apache Beam programming model ? NB: have seen side input s recommended as a solution to similar problems like this, but it is not applicable here since the effect we are after is that every change to either Person or Organization

问题 I have a Beam pipeline that starts off with reading multiple text files where each line in a file represents a row that gets inserted into Bigtable later in the pipeline. The scenario requires confirming that the count of rows extracted from each file & count of rows later inserted into Bigtable match. For this I am planning to develop a custom Windowing strategy so that lines from a single file get assigned to a single window based on the file name as the key that will be passed to the

问题 I have a Beam pipeline that starts off with reading multiple text files where each line in a file represents a row that gets inserted into Bigtable later in the pipeline. The scenario requires confirming that the count of rows extracted from each file & count of rows later inserted into Bigtable match. For this I am planning to develop a custom Windowing strategy so that lines from a single file get assigned to a single window based on the file name as the key that will be passed to the

问题 I'm trying to execute apache beam sdk 2.4 and the libraries com.google.cloud.bigquery but thow exception Exception in thread main java.lang.NoSuchMethodError com.google.api.client.googleapis.services.json.AbstractGoogleJsonClient$Builder.setBatchPath(LjavalangString;)LcomgoogleapiclientgoogleapisservicesAbstractGoogleClient$Builder; at com.google.api.services.bigquery.Bigquery$Builder.setBatchPath(Bigquery.java3519) import com.google.cloud.bigquery.*; <dependency> <groupId>com.google.cloud

问题 To Join two nested structure PCollection, we need to Unnest the PCollection before doing join, as getting challenges (refer my other stackoverflow case a link). So want to know how to unnest the PCollection. It would be good if some one give idea either Join two nested table or how to unnest PCollections. I just noted that we have PTransform "Unnest" (link) for unnesting collection from the nested one. But I could not find any sample on net. However I just tried to implement it with below