apache-flink

I have created a writer for BucketingSink. The sink and writer works without error but when it comes to the writer writing avro genericrecord to parquet, the file was created from in-progress, pending to complete. But the files are empty with 0 bytes. Can anyone tell me what is wrong with the code ? I have tried placing the initialization of AvroParquetWriter at the open() method, but result still the same. When debugging the code, I confirm that writer.write(element) does executed and element contain the avro genericrecord data Streaming Data BucketingSink<DataEventRecord> sink = new

My objective is to setup a high throughput cluster using Kafka as source & Flink as the stream processing engine. Here's what I have done. I have setup a 2-node cluster the following configuration on the master and the slave. Master flink-conf.yaml jobmanager.rpc.address: <MASTER_IP_ADDR> #localhost jobmanager.rpc.port: 6123 jobmanager.heap.mb: 256 taskmanager.heap.mb: 512 taskmanager.numberOfTaskSlots: 50 parallelism.default: 100 Slave flink-conf.yaml jobmanager.rpc.address: <MASTER_IP_ADDR> #localhost jobmanager.rpc.port: 6123 jobmanager.heap.mb: 512 #256 taskmanager.heap.mb: 1024 #512

Im using Flink (latest via git) to stream from kafka to cassandra. To ease unit testing Im adding dependency injection via Dagger. The ObjectGraph seems to be setting itself up properly but the 'inner objects' are being flagged as 'not serializable' by Flink. If I include these objects directly they work - so what's the difference? Class in question implements MapFunction and @Inject a module for cassandra and one for reading config files. Is there a way to build this so I can use late binding or does Flink make this impossible? Edit: fwiw - Dependency injection (via dagger) and

In Apache Flink I have a stream of tuples. Let's assume a really simple Tuple1<String> . The tuple can have an arbitrary value in it's value field (e.g. 'P1', 'P2', etc.). The set of possible values is finite but I don't know the full set beforehand (so there could be a 'P362'). I want to write that tuple to a certain output location depending on the value inside of the tuple. So e.g. I would like to have the following file structure: /output/P1 /output/P2 In the documentation I only found possibilities to write to locations that I know beforehand (e.g. stream.writeCsv("/output/somewhere") ),

What is the difference between mini-batch vs real time streaming in practice (not theory)? In theory, I understand mini batch is something that batches in the given time frame whereas real time streaming is more like do something as the data arrives but my biggest question is why not have mini batch with epsilon time frame (say one millisecond) or I would like to understand reason why one would be an effective solution than other? I recently came across one example where mini-batch (Apache Spark) is used for Fraud detection and real time streaming (Apache Flink) used for Fraud Prevention.