apache-flink

Could anyone compare Flink and Spark as platforms for machine learning? Which is potentially better for iterative algorithms? Link to the general Flink vs Spark discussion: What is the difference between Apache Spark and Apache Flink? Fabian Hueske Disclaimer: I'm a PMC member of Apache Flink. My answer focuses on the differences of executing iterations in Flink and Spark. Apache Spark executes iterations by loop unrolling. This means that for each iteration a new set of tasks/operators is scheduled and executed. Spark does that very efficiently because it is very good at low-latency task

I'm going through following samples using Scala / sbt : flink / elasticsearch / kibana flink tutorial My built.sbt includes following versions: libraryDependencies ++= Seq( "org.apache.flink" %% "flink-scala" % "1.3.1" % "provided", "org.apache.flink" %% "flink-streaming-scala" % "1.3.1" % "provided", "org.apache.flink" %% "flink-clients" % "1.3.1" % "provided", "joda-time" % "joda-time" % "2.9.9", "com.google.guava" % "guava" % "22.0", "com.typesafe" % "config" % "1.3.0", "org.apache.flink" % "flink-connector-kafka-0.10_2.10" % "1.2.0", "org.elasticsearch" % "elasticsearch" % "5.5.1", "org

I have written a simple pattern like this Pattern<JoinedEvent, ?> pattern = Pattern.<JoinedEvent>begin("start") .where(new SimpleCondition<JoinedEvent>() { @Override public boolean filter(JoinedEvent streamEvent) throws Exception { return streamEvent.getRRInterval()>= 10 ; } }).within(Time.milliseconds(WindowLength)); and it executes well in IntellijIdea. I am using Flink 1.3.2 both in the dashboard and in IntelliJ-Idea. While I was building Flink from source, I have seen a lot of warning messages which led me to believe that iterative condition classes have not been included in a jar as error

Apache flink - job simple windowing problem - java.lang.RuntimeException: segment has been freed Hi, I am a flink newbee and in my job, I am trying to use windowing to simply aggregate elements to enable delayed processing: src = src.timeWindowAll(Time.milliseconds(1000)).process(new BaseDelayingProcessAllWindowFunctionImpl()); processwindow function simply collects input elements: public class BaseDelayingProcessAllWindowFunction<IN> extends ProcessAllWindowFunction<IN, IN, TimeWindow> { private static final long serialVersionUID = 1L; protected Logger logger; public

I want to configure filesystem state backend and zookeeper recovery mode: state.backend: filesystem state.backend.fs.checkpointdir: ??? recovery.mode: zookeeper recovery.zookeeper.storageDir: ??? As you can see I should specify checkpointdir and storageDir parameters, but I don't have any file systems supported by Apache Flink (like HDFS or Amazon S3). But I have installed Riak CS cluster (seems like it compatible with S3 ). So, can I use Riak CS together with Apache Flink? If it is possible: how to configure Apache Flink to work with Riak CS? Answer: How to join Apache Flink and Riak CS? Riak

While running Flink 1.5.0 with a local environment I was trying to get latency metrics via REST (with something similar to http://localhost:8081/jobs/e779dbbed0bfb25cd02348a2317dc8f1/vertices/e70bbd798b564e0a50e10e343f1ac56b/metrics ) but there isn't any reference to latency . All of this while the latency tracking is enabled which I confirmed by checking with the debugger that the LatencyMarksEmitter is emiting the marks. What can I be doing wrong? In 1.5 latency metrics aren't exposed for tasks but for jobs instead, the reasoning being that latency metrics inherently contain information

I want to share a HashMap across every node in Flink and allow the nodes to update that HashMap. I have this code so far: object ParallelStreams { val env = StreamExecutionEnvironment.getExecutionEnvironment //Is there a way to attach a HashMap to this config variable? val config = new Configuration() config.setClass("HashMap", Class[CustomGlobal]) env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime) class CustomGlobal extends ExecutionConfig.GlobalJobParameters { override def toMap: util.Map[String, String] = { new HashMap[String, String]() } } class MyCoMap extends RichCoMapFunction