google-cloud-dataflow

问题 Take the official document 'Creating Templates' as an example: https://cloud.google.com/dataflow/docs/templates/creating-templates class WordcountOptions(PipelineOptions): @classmethod def _add_argparse_args(cls, parser): # Use add_value_provider_argument for arguments to be templatable # Use add_argument as usual for non-templatable arguments parser.add_value_provider_argument( '--input', default='gs://dataflow-samples/shakespeare/kinglear.txt', help='Path of the file to read from') parser

问题 Take the official document 'Creating Templates' as an example: https://cloud.google.com/dataflow/docs/templates/creating-templates class WordcountOptions(PipelineOptions): @classmethod def _add_argparse_args(cls, parser): # Use add_value_provider_argument for arguments to be templatable # Use add_argument as usual for non-templatable arguments parser.add_value_provider_argument( '--input', default='gs://dataflow-samples/shakespeare/kinglear.txt', help='Path of the file to read from') parser

问题 We are building data pipeline using Beam Python SDK and trying to run on Dataflow, but getting the below error, A setup error was detected in beamapp-xxxxyyyy-0322102737-03220329-8a74-harness-lm6v. Please refer to the worker-startup log for detailed information. But could not find detailed worker-startup logs. We tried increasing memory size, worker count etc, but still getting the same error. Here is the command we use, python run.py \ --project=xyz \ --runner=DataflowRunner \ --staging

问题 We are building data pipeline using Beam Python SDK and trying to run on Dataflow, but getting the below error, A setup error was detected in beamapp-xxxxyyyy-0322102737-03220329-8a74-harness-lm6v. Please refer to the worker-startup log for detailed information. But could not find detailed worker-startup logs. We tried increasing memory size, worker count etc, but still getting the same error. Here is the command we use, python run.py \ --project=xyz \ --runner=DataflowRunner \ --staging

问题 I experiment a weird error with my Dataflow pipeline when I want to use specific library from PyPI. I need jsonschema in a ParDo, so, in my requirements.txt file, I added jsonschema==3.2.0 . I launch my pipeline with the command line below: python -m gcs_to_all \ --runner DataflowRunner \ --project <my-project-id> \ --region europe-west1 \ --temp_location gs://<my-bucket-name>/temp/ \ --input_topic "projects/<my-project-id>/topics/<my-topic>" \ --network=<my-network> \ --subnetwork=<my-subnet

问题 I experiment a weird error with my Dataflow pipeline when I want to use specific library from PyPI. I need jsonschema in a ParDo, so, in my requirements.txt file, I added jsonschema==3.2.0 . I launch my pipeline with the command line below: python -m gcs_to_all \ --runner DataflowRunner \ --project <my-project-id> \ --region europe-west1 \ --temp_location gs://<my-bucket-name>/temp/ \ --input_topic "projects/<my-project-id>/topics/<my-topic>" \ --network=<my-network> \ --subnetwork=<my-subnet

问题 My use case involves fetching the job id of all streaming dataflow jobs present in my project and cancel it. Update the sources for my dataflow job and re-run it. I am trying to achieve this using python. I did not come across any useful documentation until now. I thought of using python's library subprocess to execute the gcloud commands as a workaround. But again I was not able to store the result and use it. Can somebody please guide me as what is the best way of doing this. 回答1: In

问题 We host elatsicsearch cluster on Elastic Cloud and call it from dataflow (GCP). Job works fine in dev but when we deploy to prod we're seeing lots of connection timeout on the client side. Traceback (most recent call last): File "apache_beam/runners/common.py", line 1213, in apache_beam.runners.common.DoFnRunner.process File "apache_beam/runners/common.py", line 570, in apache_beam.runners.common.SimpleInvoker.invoke_process File "main.py", line 159, in process File "/usr/local/lib/python3.7

问题 I need to apply error handling to my Dataflow for multiple inserts to Spanner with the same primary key. The logic being that an older message may be received after the current message and I do not want to overwrite the saved values. Therefore I will create my mutation as an insert and throw an error when a duplicate insert is attempted. I have seen several examples of try blocks within DoFn's that write to a side output to log any errors. This is a very nice solution but I need to apply

问题 I need to apply error handling to my Dataflow for multiple inserts to Spanner with the same primary key. The logic being that an older message may be received after the current message and I do not want to overwrite the saved values. Therefore I will create my mutation as an insert and throw an error when a duplicate insert is attempted. I have seen several examples of try blocks within DoFn's that write to a side output to log any errors. This is a very nice solution but I need to apply