apache-flink

问题 Does flink support authentication of flink web UI like any username/password way of authentication. If yes, then does this require any third party applications such as key cloak etc 回答1: Flink does not directly support authenticating access to the web UI, but you can always put something like nginx in front of it. 来源： https://stackoverflow.com/questions/49254307/flink-web-ui-authentication

问题 I am looking for a solution how I can change a source function in Flink while execution is in progress: StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); SourceFunction<String> mySource = ...; // this a function that I want to change during runtime; DataStream<String> stream = env.addSource(mySource); stream.map(...).print(); // creating my stream env.execute("sample"); I am thinking about creating a wrapper around a real implementation of SourceFunction

问题 I've been trying different methods but unable to get right. what's wrong in this code? def main(args: Array[String]): Unit = { val TEMPERATURE_THRESHOLD: Double = 50.00 val see: StreamExecutionEnvironment = StreamExecutionEnvironment.getExecutionEnvironment val properties = new Properties() properties.setProperty("bootstrap.servers", "localhost:9092") properties.setProperty("zookeeper.connect", "localhost:2181") val src = see.addSource(new FlinkKafkaConsumer010("broadcast", new

问题 I am trying to perform a map operation on a KeyedStream in Flink: stream.map(new JsonToMessageObjectMapper()) .keyBy("keyfield") .map(new MessageProcessorStateful()) The output of the JsonToObjectMapper operator is a POJO of class MessageObject which has a String field ' keyfield '. The stream is then keyed on this field. The MessageProcessorStateful is a RichMapFunction like this: public class MessageAdProcessorStateful extends RichMapFunction<MessageObject, Tuple2<String, String>> { private

问题 I have this code below: import java.util.Properties import com.google.gson._ import com.typesafe.config.ConfigFactory import org.apache.flink.cep.scala.pattern.Pattern import org.apache.flink.cep.scala.CEP import org.apache.flink.streaming.api.TimeCharacteristic import org.apache.flink.streaming.api.scala._ import org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer010 import org.apache.flink.streaming.util.serialization.SimpleStringSchema object WindowedWordCount { val

问题 After reading the flink docs, (relevant part noted below) I still did not completely understand the atomicity and key distribution. i.e consider a graph consisting of keyby->flatmap(containing a map state), and parallelism set as 1 with 4 task slots, does flink ensure that each key exists only once (in one task slot) in the distributed environment, and is it the atomic unit? Thanks in advance to all helpers. You can think of Keyed State as Operator State that has been partitioned, or sharded,

问题 I have created a simple Apache Flink project that will read data from a Kafka topic and write that data to an S3 bucket. I do not receive any errors when I run the project and it successfully reads each message from the Kafka topic, but nothing is written to my S3 bucket. There are no errors so it is difficult to try and debug what is going on. Below is my project and my configurations. This is only occurring when I am using a StreamExecutionEnviornment. If I try to just produce to S3 using a

问题 I have the following method that computes the probability of a value in a DataSet : /** * Compute the probabilities of each value on the given [[DataSet]] * * @param x single colum [[DataSet]] * @return Sequence of probabilites for each value */ private[this] def probs(x: DataSet[Double]): Seq[Double] = { val counts = x.groupBy(_.doubleValue) .reduceGroup(_.size.toDouble) .name("X Probs") .collect val total = counts.sum counts.map(_ / total) } The problem is that when I submit my flink job,

问题 I am trying to deserialize an avro byte stream into a scala case class object. Basically, i had a kafka stream with avro encoded data flowing and now there is an addition to the schema and i am trying to update the scala case class to include the new field. The case class looks like this /** Case class to hold the Device data. */ case class DeviceData(deviceId: String, sw_version: String, timestamp: String, reading: Double, new_field: Option[String] = None ) { this() = this("na", "na", "na",

问题 I am pretty new in the world of Streams and I am facing some issues in my first try. More specifically, I am trying to implement a count and groupBy functionality in a sliding window using Flink. I 've done it in a normal DateStream but I am not able to make it work in a WindowedStream . Do you have any suggestion on how can I do it? val parsedStream: DataStream[(String, Response)] = stream .mapWith(_.decodeOption[Response]) .filter(_.isDefined) .map { record => ( s"${record.get.group.group