apache-flink

问题 It may seem stupid but it is my very first post here. Sorry for doing anything wrong. I am currently building a simple ML pipeline with TFX 0.11 (i.e. tfdv-tft-tfserving) and tensorflow 1.11, using python2.7. I currently have a apache-flink cluster and I want to use that for TFX. I know the framework behind TFX is apache-beams 2.8, and it (apache-beams) supports flink with python SDK currently through a portable runner layer. But the problem is how I can code in TFX (tfdv-tft) using apache

问题 My Question is that, if we have two raw event streams i.e Smoke and Temperature and we want to find out if complex event i.e Fire has happened by applying operators to raw streams, can we do this in Flink? I am asking this question because all the examples that I have seen till now for Flink CEP include only one input stream. Please correct me if I am wrong. 回答1: Short Answer - Yes, you can read and process multiple streams and fire rules based on your event types from the different stream

问题 My Question is that, if we have two raw event streams i.e Smoke and Temperature and we want to find out if complex event i.e Fire has happened by applying operators to raw streams, can we do this in Flink? I am asking this question because all the examples that I have seen till now for Flink CEP include only one input stream. Please correct me if I am wrong. 回答1: Short Answer - Yes, you can read and process multiple streams and fire rules based on your event types from the different stream

问题 Suppose that I have 2 different types of data streams, one providing weather data and the other providing vehicle data, and I would like to use Flink to do complex event processing on the data. Which method in Flink 1.3.x is the correct method to use? I saw different methods like Union, Connect, Window Join. Basically I just want to try a simple CEP like this: IF weather is wet AND vehicle speed > 60 WITHIN the last 10 seconds THEN raise alert Thanks! 回答1: In my opinion, there are two ways

问题 When I am trying to run the flink job in standalone cluster I get this error: java.lang.Exception: TaskManager was lost/killed: ResourceID{resourceId='2961948b9ac490c11c6e41b0ec197e9f'} @ localhost (dataPort=55795) at org.apache.flink.runtime.instance.SimpleSlot.releaseSlot(SimpleSlot.java:217) at org.apache.flink.runtime.instance.SlotSharingGroupAssignment.releaseSharedSlot(SlotSharingGroupAssignment.java:533) at org.apache.flink.runtime.instance.SharedSlot.releaseSlot(SharedSlot.java:192)

问题 By reading the documentation I understood that with Apache Flink 1.3 I should be able to use Elasticsearch 5.x. However, in my pom.xml : <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-connector-elasticsearch5_2.10</artifactId> <version>1.3.0</version> </dependency> I got this : Dependency "org.apache.flink:flink-connector-elasticsearch5_2.10:1.3.0" not found Any idea why this dependency is unfound ? 回答1: This was a bug in the 1.3.0 release and is being fixed for 1.3.1

问题 I am trying to read AVRO data from a Kafka topic using Flink 1.0.3. I just know that this particular Kafka topic is having AVRO encoded message and I am having the AVRO schema file. My Flink code: public static void main(String[] args) throws Exception { StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); Properties properties = new Properties(); properties.setProperty("bootstrap.servers", "dojo3xxxxx:9092,dojoxxxxx:9092,dojoxxxxx:9092"); properties

问题 Got stuck a bit with CoFlatMapFunction . It seems to work fine if I place it on the DataStream before window but fails if placed after window's “apply” function. I was testing two streams, main “Features” on flatMap1 constantly ingesting data and control stream “Model” on flatMap2 changing the model on request. I am able to set and see b0/b1 properly set in flatMap2 , but flatMap1 always see b0 and b1 as was set to 0 at the initialization. Am I missing something obvious here? public static

问题 We plan to use Flink CEP for processing a big amount of events according to some dynamic templates. The system must recognize chains of events (sometimes complicated chains with conditions and grouping). The templates will be created by user. In other words we have to create complicated templates without touching the code. Is it possible to use Apache Flink for solving this problem? Does Filnk support dynamic-templates? 回答1: At the moment Flink's CEP library does not support this kind of

问题 I have a stream that I want to partition using a certain key, and then run through several transformations, each using a state. When I call keyBy() I get a KeyedStream and the next transformation can access a partitioned state correctly, but another transformation chained after that gets an exception when trying to access a partitioned state. The exception is: State key serializer has not been configured in the config. This operation cannot use partitioned state It seems that the key