apache-beam

I'm using Dataflow 0.5.5 Python. Ran into the following error in very simple code: print(len(row_list)) row_list is a list. Exactly the same code, same data and same pipeline runs perfectly fine on DirectRunner, but throws the following exception on DataflowRunner. What does it mean and how I can solve it? job name: `beamapp-root-0216042234-124125` (f14756f20f567f62): Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/dataflow_worker/batchworker.py", line 544, in do_work work_executor.execute() File "dataflow_worker/executor.py", line 973, in dataflow_worker

What is the recommended way to do expensive one-off initialization in a Beam Python DoFn? The Java SDK has DoFn.Setup , but there doesn't appear to be an equivalent in Beam Python. Is the best way currently to attach objects to threading.local() in the DoFn initializer? Setup and teardown have now been added to the Python SDK and are the recommended way to do expensive one-off initialization in a Beam Python DoFn. Dataflow Python is not particularly transparent about the optimal method for initializing expensive objects. There are a few mechanisms by which objects can be instantiated

What is the difference between these two annotations? DoFn.Setup Annotation for the method to use to prepare an instance for processing bundles of elements. Uses the word "bundle", takes zero arguments. DoFn.StartBundle Annotation for the method to use to prepare an instance for processing a batch of elements. Uses the word "batch", takes zero or one arguments ( StartBundleContext , a way to access PipelineOptions ). What I'm trying to do I need to initialize a library within the DoFn instance, then use that library for every element in the "batch" or "bundle". I wouldn't normally split hairs

We are attempting to use fixed windows on an Apache Beam pipeline (using DirectRunner ). Our flow is as follows: Pull data from pub/sub Deserialize JSON into Java object Window events w/ fixed windows of 5 seconds Using a custom CombineFn , combine each window of Event s into a List<Event> For the sake of testing, simply output the resulting List<Event> Pipeline code: pipeline // Read from pubsub topic to create unbounded PCollection .apply(PubsubIO .<String>read() .topic(options.getTopic()) .withCoder(StringUtf8Coder.of()) ) // Deserialize JSON into Event object .apply("ParseEvent", ParDo .of

I was going through the Apache posts and found a new term called Beam. Can anybody explain what exactly Apache Beam is? I tried to google out but unable to get a clear answer. Apache Beam is an open source, unified model for defining and executing both batch and streaming data-parallel processing pipelines, as well as a set of language-specific SDKs for constructing pipelines and runtime-specific Runners for executing them. History: The model behind Beam evolved from a number of internal Google data processing projects, including MapReduce , FlumeJava , and Millwheel . This model was

I am trying to read a table from a Google spanner database, and write it to a text file to do a backup, using google dataflow with the python sdk. I have written the following script: from __future__ import absolute_import import argparse import itertools import logging import re import time import datetime as dt import logging import apache_beam as beam from apache_beam.io import iobase from apache_beam.io import WriteToText from apache_beam.io.range_trackers import OffsetRangeTracker, UnsplittableRangeTracker from apache_beam.metrics import Metrics from apache_beam.options.pipeline_options

I have tried the example code of SortValues transform using DirectRunner on local machine (Windows) PCollection<KV<String, KV<String, Integer>>> input = ... PCollection<KV<String, Iterable<KV<String, Integer>>>> grouped = input.apply(GroupByKey.<String, KV<String, Integer>>create()); PCollection<KV<String, Iterable<KV<String, Integer>>>> groupedAndSorted = grouped.apply(SortValues.<String, String, Integer>create(BufferedExternalSorter.options())); but I got the error PipelineExecutionException: java.lang.NoClassDefFoundError: org/apache/hadoop/io/Writable . Does this mean this transform

Input PCollection is http requests, which is a bounded dataset. I want to make async http call (Java) in a ParDo , parse response and put results into output PCollection. My code is below. Getting exception as following. I cound't figure out the reason. need a guide.... java.util.concurrent.CompletionException: java.lang.IllegalStateException: Can't add element ValueInGlobalWindow{value=streaming.mapserver.backfill.EnrichedPoint@2c59e, pane=PaneInfo.NO_FIRING} to committed bundle in PCollection Call Map Server With Rate Throttle/ParMultiDo(ProcessRequests).output [PCollection] Code : public