apache-flink

I'm using Flink 1.4.1 and Beam 2.3.0, and would like to know is it possible to have metrics available in Flink WebUI (or anywhere at all), as in Dataflow WebUI ? I've used counter like: import org.apache.beam.sdk.metrics.Counter; import org.apache.beam.sdk.metrics.Metrics; ... Counter elementsRead = Metrics.counter(getClass(), "elements_read"); ... elementsRead.inc(); but I can't find "elements_read" counts available anywhere (Task Metrics or Accumulators) in Flink WebUI. I thought this will be straightforward after BEAM-773 . Once you have selected a job in your dashboard, you will see the

Let's say we want to compute the sum and average of the items, and can either working with states or windows (time). Example working with windows - https://ci.apache.org/projects/flink/flink-docs-release-0.10/apis/streaming_guide.html#example-program Example working with states - https://github.com/dataArtisans/flink-training-exercises/blob/master/src/main/java/com/dataartisans/flinktraining/exercises/datastream_java/ride_speed/RideSpeed.java Can I ask what would be the reasons to make decision? Can I infer that if the data arrives very irregularly (50% comes in the defined window length and

I am using Apache Flink v1.6.0 and I am trying to write to Elasticsearch v6.4.0, which is hosted in Elastic Cloud . I am having issue when authenticating to the Elastic Cloud cluster. I have been able to get Flink to write to a local Elasticsearch v6.4.0 node, which does not have encryption using the following code: /* Elasticsearch Configuration */ List<HttpHost> httpHosts = new ArrayList<>(); httpHosts.add(new HttpHost("127.0.0.1", 9200, "http")); // use a ElasticsearchSink.Builder to create an ElasticsearchSink ElasticsearchSink.Builder<ObjectNode> esSinkBuilder = new ElasticsearchSink

We plan to use Flink CEP for processing a big amount of events according to some dynamic templates. The system must recognize chains of events (sometimes complicated chains with conditions and grouping). The templates will be created by user. In other words we have to create complicated templates without touching the code. Is it possible to use Apache Flink for solving this problem? Does Filnk support dynamic-templates? At the moment Flink's CEP library does not support this kind of dynamic rule adaption. However, there is no fundamental reason which makes it impossible to implement. In fact,

I have two stream one is of Int and other is of json .In The json Schema there is one key which is some int .So i need to filter the json stream via key comparison with the other integer stream so Is it possible in Flink? Yes, you can do this kind of stream processing with Flink. The basic building blocks you need from Flink are connected streams, and stateful functions -- here's an example using a RichCoFlatMap: import org.apache.flink.api.common.state.ValueState; import org.apache.flink.api.common.state.ValueStateDescriptor; import org.apache.flink.api.common.typeinfo.TypeHint; import org

It's a follow up question on Flink Scala API "not enough arguments" . I'd like to be able to pass Flink's DataSet s around and do something with it, but the parameters to the dataset are generic. Here's the problem I have now: import org.apache.flink.api.scala.ExecutionEnvironment import org.apache.flink.api.scala._ import scala.reflect.ClassTag object TestFlink { def main(args: Array[String]) { val env = ExecutionEnvironment.getExecutionEnvironment val text = env.fromElements( "Who's there?", "I think I hear them. Stand, ho! Who's there?") val split = text.flatMap { _.toLowerCase.split("\\W+"

I have a stream that I want to partition using a certain key, and then run through several transformations, each using a state. When I call keyBy() I get a KeyedStream and the next transformation can access a partitioned state correctly, but another transformation chained after that gets an exception when trying to access a partitioned state. The exception is: State key serializer has not been configured in the config. This operation cannot use partitioned state It seems that the key information is only passed to the first transformation and not further down the chain. The code I try to run is