google-cloud-dataflow

问题 Is there a way to delete a bigquery table only after all the steps in a batch dataflow pipeline have succeeded? 回答1: You can use DataflowPipelineJob.waitToFinish(...) to wait for your job to finish, check that the returned state was DONE, and then use the BigQuery API to delete the table. 来源： https://stackoverflow.com/questions/41774664/delete-a-bigquery-table-after-all-the-steps-in-dataflow-job-have-completed

问题 I would like to know if there is a way to find out total no rows in a file using google dataflow. Any code sample and pointer will be great help. Basically, I have a method as int getCount(String fileName) {} So, above method will return total count of rows and its implementation will be dataflow code. Thanks 回答1: Seems like your use case is one that doesn't require distributed processing, because the file is compressed and hence can not be read in parallel. However, you may still find it

问题 I'm making dataflow pipeline with python. I want to share global variables across pipeline transform and across worker nodes like global variables (across multiple workers). Is there any way to support this? thanx in advance 回答1: Stateful processing may be of use for sharing state between workers of a specific node (would not be able to share between transforms though): https://beam.apache.org/blog/2017/02/13/stateful-processing.html 来源： https://stackoverflow.com/questions/44432556/is-there

问题 I noticed in the java sdk, there is a function that allows you to write the headers of a csv file. https://cloud.google.com/dataflow/java-sdk/JavaDoc/com/google/cloud/dataflow/sdk/io/TextIO.Write.html#withHeader-java.lang.String- Is this features mirrored on the python skd? 回答1: You can now write to a text and specify a header using the text sink. From the documentation: class apache_beam.io.textio.WriteToText(file_path_prefix, file_name_suffix='', append_trailing_newlines=True, num_shards=0,

问题 purpose: I want to load stream data, then add a key and then count them by key. problem: Apache Beam Dataflow pipline gets a memory error when i try to load and group-by-key a big-size data using streaming approach (unbounded data) . Because it seems that data is accumulated in group-by and it does not fire data earlier with triggering of each window. If I decrease the elements size (elements count will not change) it works! because actually group-by step waits for all the data to be grouped

问题 We migrated from Google Dataflow 1.9 to Apache Beam 0.6. We are noticing a change in the behavior to the timestamps after applying the globalwindow. In Google Dataflow 1.9, we would get the correct timestamps in the DoFn after windowing/combine function. Now we get some huge value for the timestamp e.g. 9223371950454775, Did the default behavior for the globalwindow change in Apache Beam version? input.apply(name(id, "Assign To Shard"), ParDo.of(new AssignToTest())) .apply(name(id, "Window"),

问题 We migrated from Google Dataflow 1.9 to Apache Beam 0.6. We are noticing a change in the behavior to the timestamps after applying the globalwindow. In Google Dataflow 1.9, we would get the correct timestamps in the DoFn after windowing/combine function. Now we get some huge value for the timestamp e.g. 9223371950454775, Did the default behavior for the globalwindow change in Apache Beam version? input.apply(name(id, "Assign To Shard"), ParDo.of(new AssignToTest())) .apply(name(id, "Window"),

问题 I'm using Dataflow SDK 2.X Java API ( Apache Beam SDK) to write data into mysql. I've created pipelines based on Apache Beam SDK documentation to write data into mysql using dataflow. It inserts single row at a time where as I need to implement bulk insert. I do not find any option in official documentation to enable bulk inset mode. Wondering, if it's possible to set bulk insert mode in dataflow pipeline? If yes, please let me know what I need to change in below code. .apply(JdbcIO.<KV

问题 I have a Dataflow job to write to BigQuery. It works well for non-nested schema, however fails for the nested schema. Here is my Dataflow pipeline: pipeline_options = PipelineOptions() p = beam.Pipeline(options=pipeline_options) wordcount_options = pipeline_options.view_as(WordcountTemplatedOptions) schema = 'url: STRING,' \ 'ua: STRING,' \ 'method: STRING,' \ 'man: RECORD,' \ 'man.ip: RECORD,' \ 'man.ip.cc: STRING,' \ 'man.ip.city: STRING,' \ 'man.ip.as: INTEGER,' \ 'man.ip.country: STRING,'

问题 I have a Dataflow job to write to BigQuery. It works well for non-nested schema, however fails for the nested schema. Here is my Dataflow pipeline: pipeline_options = PipelineOptions() p = beam.Pipeline(options=pipeline_options) wordcount_options = pipeline_options.view_as(WordcountTemplatedOptions) schema = 'url: STRING,' \ 'ua: STRING,' \ 'method: STRING,' \ 'man: RECORD,' \ 'man.ip: RECORD,' \ 'man.ip.cc: STRING,' \ 'man.ip.city: STRING,' \ 'man.ip.as: INTEGER,' \ 'man.ip.country: STRING,'