apache-beam

When I applied ParDo.of(new ParDoFn()) to PCollection named textInput , The program throws this Exception. But The Program is normal when I delete .apply(ParDo.of(new ParDoFn())) . //SparkRunner private static void testHadoop(Pipeline pipeline){ Class<? extends FileInputFormat<LongWritable, Text>> inputFormatClass = (Class<? extends FileInputFormat<LongWritable, Text>>) (Class<?>) TextInputFormat.class; @SuppressWarnings("unchecked") //hdfs://localhost:9000 HadoopIO.Read.Bound<LongWritable, Text> readPTransfom_1 = HadoopIO.Read.from("hdfs://localhost:9000/tmp/kinglear.txt", inputFormatClass,

I have tried to follow this documentation in the most precise way I could: https://beam.apache.org/documentation/sdks/javadoc/2.0.0/org/apache/beam/sdk/io/xml/XmlIO.html Please find below my codes : public static void main(String args[]) { DataflowPipelineOptions options=PipelineOptionsFactory.as(DataflowPipelineOptions.class); options.setTempLocation("gs://balajee_test/stagging"); options.setProject("test-1-130106"); Pipeline p=Pipeline.create(options); PCollection<XMLFormatter> record= p.apply(XmlIO.<XMLFormatter>read() .from("gs://balajee_test/sample_3.xml") .withRootElement("book")

I'm trying to read a RabbitMQ queue using Apache Beam. I've written some transformation code to have messages written to Kafka. I've even tested my scenario using simple text messages. Now I try to deploy it with the effective messages this transformer is made to run on. These are JSON message of a quite moderate size. Strangely, when i try to read "production" messages, I get this exception stack trace. java.lang.IllegalArgumentException: Unable to encode element 'ValueWithRecordId{id=[], value=org.apache.beam.sdk.io.rabbitmq.RabbitMqMessage@f179a7f}' with coder 'ValueWithRecordId

Say we have one worker with 4 CPU cores. How does parallelism configured in Dataflow worker machines? Do we parallelize beyond # of cores? Where would this type of information be available? One worker thread is used per core, and each worker thread independently processes a chunk of the input space. 来源： https://stackoverflow.com/questions/47777639/google-cloud-dataflow-worker-threading

I have a folder structure of the type year/month/day/hour/* , and I'd like the beam to read this as an unbounded source in chronological order. Specifically, this means reading in all the files in the first hour on record and adding their contents for processing. Then, add the file contents of the next hour for processing, up until the current time where it waits for new files to arrive in the latest year/month/day/hour folder. Is it possible to do this with apache beam? So what I would do is to add timestamps to each element according to the file path. As a test I used the following example.

As I have a working static Dataflow running, I'd like to create a template from this one to let me easily reuse the Dataflow without any command line typing. Following the Creating Templates tutorial from the official doesn't provide a sample for templatable output. My Dataflow ends with a BigQuery sink which takes a few arguments like the target table for storage. This exact parameter is the one I'd like to make available in my template allowing me to choose the target storage after running the flow. But, I'm not able to get this working. Below I paste some code snippets which could help