apache-beam-io

问题 I have 2 approaches to initialize the HttpClient in order to make an API call from a ParDo in Apache Beam. Approach 1: Initialise the HttpClient object in the StartBundle and close the HttpClient in FinishBundle . The code is as follows: public class ProcessNewIncomingRequest extends DoFn<String, KV<String, String>> { @StartBundle public void startBundle() { HttpClient client = HttpClient.newHttpClient(); HttpRequest request = HttpRequest.newBuilder() .uri(URI.create(<Custom_URL>)) .build();

问题 Current beam pipeline is reading files as stream using FileIO.matchAll().continuously() . This returns PCollection . I want to write these files back with the same names to another gcs bucket i.e each PCollection is one file metadata/readableFile which should be written back to another bucket after some processing. Is there any sink that i should use to achieve writing each PCollection item back to GCS or are there any ways to do it ? Is it possible to create a window per element and then use

问题 I am trying to run a very simple program in Apache Beam to try out how it works. import apache_beam as beam class Split(beam.DoFn): def process(self, element): return element with beam.Pipeline() as p: rows = (p | beam.io.ReadAllFromText( "input.csv") | beam.ParDo(Split())) While running this, I get the following errors .... some more stack.... File "/home/raheel/code/beam-practice/lib/python2.7/site-packages/apache_beam/transforms/util.py", line 565, in expand windowing_saved = pcoll

问题 I'm using Dataflow SDK 2.X Java API ( Apache Beam SDK) to write data into mysql. I've created pipelines based on Apache Beam SDK documentation to write data into mysql using dataflow. It inserts single row at a time where as I need to implement bulk insert. I do not find any option in official documentation to enable bulk inset mode. Wondering, if it's possible to set bulk insert mode in dataflow pipeline? If yes, please let me know what I need to change in below code. .apply(JdbcIO.<KV

问题 Unable to set triggering_frequency for Dataflow Streaming job. transformed | 'Write' >> beam.io.WriteToBigQuery( known_args.target_table, schema=schema, create_disposition=beam.io.BigQueryDisposition.CREATE_IF_NEEDED, write_disposition=beam.io.BigQueryDisposition.WRITE_APPEND, method=bigquery.WriteToBigQuery.Method.FILE_LOADS, triggering_frequency=5 ) Error: triggering_frequency can only be used with FILE_LOADS method of writing to BigQuery 回答1: This is a bug. The WriteToBigQuery transform

问题 I have a streaming pipeline developed in Apache Beam (using Spark Runner) which reads from kinesis stream. I am looking out for options in Apache Beam to manage kinesis checkpointing (i.e. stores periodically the current position of kinesis stream) so as it allows the system to recover from failures and continue processing where the stream left off. Is there a provision available for Apache Beam to support kinesis checkpointing as similar to Spark Streaming (Reference link - https://spark

问题 Step1 : AssumeRole public static AWSCredentialsProvider getCredentials() { if (roleARN.length() > 0) { STSAssumeRoleSessionCredentialsProvider credentialsProvider = new STSAssumeRoleSessionCredentialsProvider .Builder(roleARN, Constants.SESSION_NAME) .withStsClient(AWSSecurityTokenServiceClientBuilder.defaultClient()) .build(); return credentialsProvider; } return new ProfileCredentialsProvider(); } Step 2 : Set Credentials to pipeline credentials = getCredentials(); pipeline.getOptions().as

来源： https://stackoverflow.com/questions/62775435/does-gcp-dataflow-support-kafka-io-in-python

问题 I have an use case where, I read in the newline json elements stored in google cloud storage and start processing each json. While processing each json, I have to call an external API for doing de-duplication whether that json element was discovered previously. I'm doing a ParDo with a DoFn on each json. I haven't seen any online tutorial saying how to call an external API endpoint from apache beam DoFn Dataflow. I'm using JAVA SDK of Beam. Some of the tutorial I studied explained that using

问题 I have an use case where, I read in the newline json elements stored in google cloud storage and start processing each json. While processing each json, I have to call an external API for doing de-duplication whether that json element was discovered previously. I'm doing a ParDo with a DoFn on each json. I haven't seen any online tutorial saying how to call an external API endpoint from apache beam DoFn Dataflow. I'm using JAVA SDK of Beam. Some of the tutorial I studied explained that using