apache-beam

I'm using Flink 1.4.1 and Beam 2.3.0, and would like to know is it possible to have metrics available in Flink WebUI (or anywhere at all), as in Dataflow WebUI ? I've used counter like: import org.apache.beam.sdk.metrics.Counter; import org.apache.beam.sdk.metrics.Metrics; ... Counter elementsRead = Metrics.counter(getClass(), "elements_read"); ... elementsRead.inc(); but I can't find "elements_read" counts available anywhere (Task Metrics or Accumulators) in Flink WebUI. I thought this will be straightforward after BEAM-773 . Once you have selected a job in your dashboard, you will see the

Apache Beam has recently introduced state cells, through StateSpec and the @StateId annotation, with partial support in Apache Flink and Google Cloud Dataflow. My question is about state garbage collection, in the case where a stateful DoFn is used on a windowed stream. Typically, state is removed (garbage collected) by the runner when the window expires (i.e. the watermark passes the end of the window). However, consider the case where window panes are triggered early, and the fired panes are discarded: input.apply(Window.<MyElement>into(CalendarWindows.days(1)) .triggering(AfterWatermark

I tried using the following TextIO.Read.from("gs://xyz.abc/xxx_{2017-06-06,2017-06-06}.csv") That pattern didn't work, as I get java.lang.IllegalStateException: Unable to find any files matching StaticValueProvider{value=gs://xyz.abc/xxx_{2017-06-06,2017-06-06}.csv} Even though those 2 files do exist. And I tried with a local file using a similar expression TextIO.Read.from("somefolder/xxx_{2017-06-06,2017-06-06}.csv") And that did work just fine. I would've thought there would be support for all kinds of globs for files in GCS, but nope. Why is that? is there away to accomplish what I'm

I am using Apache-Beam to run some data transformation, which including data extraction from txt, csv, and different sources of data. One thing I noticed, is the difference of results when using beam.Map and beam.ParDo In the next sample: I am reading csv data, and in the first case pass it to a DoFn using a beam.ParDo , which extracts the first element which is the date, then print it. In the second case, I directly use beam.Map to do the same thing, then print it. class Printer(beam.DoFn): def process(self,data_item): print data_item class DateExtractor(beam.DoFn): def process(self,data_item

How can I get GroupByKey to trigger early results, rather than wait for all the data to arrive (which in my case is a pretty long time).I tried to split my input PCollection into windows with an early trigger, but it just doesn`t work. It still waits for all the data to arrive before giving out the results. PCollection<List<String>> input = ... PCollection<KV<Integer,List<String>>> keyedInput = input.apply(ParDo.of(new AddArbitraryKey())) keyedInput.apply(Window<KV<Integer,List<String>>>into( FixedWindows.of(Duration.standardSeconds(1))) .triggering(Repeatedly.forever(AfterWatermark

I'm setting up a slow-changing lookup Map in my Apache-Beam pipeline. It continuously updates the lookup map. For each key in lookup map, I retrieve the latest value in the global window with accumulating mode. But it always meets Exception : org.apache.beam.sdk.Pipeline$PipelineExecutionException: java.lang.IllegalArgumentException: Duplicate values for mykey Is anything wrong with this snippet code? If I use .discardingFiredPanes() instead, I will lose information in the last emit. pipeline .apply(GenerateSequence.from(0).withRate(1, Duration.standardMinutes(1L))) .apply( Window.<Long>into

Problem Context I am trying to generate a total (linear) order of event items per key from a real-time stream where the order is event time (derived from the event payload). Approach I had attempted to implement this using streaming as follows: 1) Set up a non overlapping sequential windows, e.g. duration 5 minutes 2) Establish an allowed lateness - it is fine to discard late events 3) Set accumulation mode to retain all fired panes 4) Use the "AfterwaterMark" trigger 5) When handling a triggered pane, only consider the pane if it is the final one 6) Use GroupBy.perKey to ensure all events in

I'm running a job using Beam KafkaIO source in Google Dataflow and cannot find an easy way to persist offsets across job restarts (job update option is not enough, i need to restart the job) Comparing Beam's KafkaIO against PubSubIO (or to be precise comparing PubsubCheckpoint with KafkaCheckpointMark) I can see that checkpoint persistence is not implemented in KafkaIO (KafkaCheckpointMark.finalizeCheckpoint method is empty) whereas it's implemented in PubsubCheckpoint.finalizeCheckpoint which does acknowledgement to PubSub. Does this mean I have no means of reliably managing Kafka offsets on