apache-beam

来源： https://stackoverflow.com/questions/46854167/dataflow-streaming-using-python-sdk-transform-for-pubsub-messages-to-bigquery-o

来源： https://stackoverflow.com/questions/62696580/write-avro-files-with-logicaltype-timestamp-millis-on-date-timestamps-in-java

来源： https://stackoverflow.com/questions/62775435/does-gcp-dataflow-support-kafka-io-in-python

来源： https://stackoverflow.com/questions/63382761/dataflow-streaming-job-error-writing-to-bigquery

问题 In my pipeline I use WriteToBigQuery something like this: | beam.io.WriteToBigQuery( 'thijs:thijsset.thijstable', schema=table_schema, write_disposition=beam.io.BigQueryDisposition.WRITE_APPEND, create_disposition=beam.io.BigQueryDisposition.CREATE_IF_NEEDED) This returns a Dict as described in the documentation as follows: The beam.io.WriteToBigQuery PTransform returns a dictionary whose BigQueryWriteFn.FAILED_ROWS entry contains a PCollection of all the rows that failed to be written. How

问题 In my pipeline I use WriteToBigQuery something like this: | beam.io.WriteToBigQuery( 'thijs:thijsset.thijstable', schema=table_schema, write_disposition=beam.io.BigQueryDisposition.WRITE_APPEND, create_disposition=beam.io.BigQueryDisposition.CREATE_IF_NEEDED) This returns a Dict as described in the documentation as follows: The beam.io.WriteToBigQuery PTransform returns a dictionary whose BigQueryWriteFn.FAILED_ROWS entry contains a PCollection of all the rows that failed to be written. How

问题 I have an use case where, I read in the newline json elements stored in google cloud storage and start processing each json. While processing each json, I have to call an external API for doing de-duplication whether that json element was discovered previously. I'm doing a ParDo with a DoFn on each json. I haven't seen any online tutorial saying how to call an external API endpoint from apache beam DoFn Dataflow. I'm using JAVA SDK of Beam. Some of the tutorial I studied explained that using

问题 I have an use case where, I read in the newline json elements stored in google cloud storage and start processing each json. While processing each json, I have to call an external API for doing de-duplication whether that json element was discovered previously. I'm doing a ParDo with a DoFn on each json. I haven't seen any online tutorial saying how to call an external API endpoint from apache beam DoFn Dataflow. I'm using JAVA SDK of Beam. Some of the tutorial I studied explained that using

问题 I have been testing an Apache beam pipeline within Apache beam notebooks provided by GCP using a Kafka instance as a input and Bigquery as output. I have been able to successfully use the pipeline via Interactive runner, but when I deploy the same pipeline to Dataflow runner it seems to never actually read from the Kafka topic that has been defined. Looking into the logs gives me the error: Failed to read inputs in the data plane. Traceback (most recent call last): File /usr/local/lib/python3

问题 How do you log error messages while reading or writing to topic. We would be using Apache Beam API to read or write to topic. So I any exception is generated how do we log it. Can I send my data to other topic? PubsubIO.writeMessages() PubsubIO.readMessages() Can I write this DoFn and add debug logs log.debug("Publishing json message to pubsub topic"); PubsubIO.Write message = PubsubIO.writeMessages().to(pipelineOptions.getPubsubEnpEventTopic()); log.debug("Message published to pubsub"); 回答1: