hazelcast-jet

Hazelcast Jet and Node.JS client serialization issue

由 ＊爱你＆永不变心＊ 提交于 2021-02-10 06:26:13
问题 I am having Jet 0.6 as a backup and some Node.JS process with hazelcast-nodejs-client 0.8.0 installed. I am trying to push to map an object from Node process that is exactly a reflection of similar one on the Jet side. However I don't understand how to make sure on Jet's side this JS object will be serialized/deserialized respectively. I have a feeling I need to indicate to Jet that this JSON object is meant to be Data POJO and proper serialization/deserialization should be used. On the node

How much overhead is usual while distributing processing?

由 做~自己de王妃 提交于 2020-04-11 05:00:11
问题 For impatient readers: this is a work in progress, where I ask for help, during the process. Please do not judge the tools by my temporary data, as they can change while I try to get better results. We are in the middle of the decision process on the architecture for a tool to analyse the output from co-simulations. As part of that process I was asked to write a benchmark tool, and get data on the speeds of several distributed processing frameworks. The frameworks I tested are: Apache Spark,

Creating a new Jet custom Partitioner

由 若如初见. 提交于 2019-12-25 01:44:52
问题 My use case requires to read messages from a Kafka topics and process the messages in the natural order as they were published into the Kafka. The Kafka producer is responsible to publish each group of messages sorted in a single kafka topic-partition, and I need to process each group of message in the same Vertex-Processor in the same order. The image above represents the basic idea. There a few KafkaSource-Processors reading from Kafka. And one edge connected to a vertex to decode the kafka

Hazelcast jet pipeline created on an app with multiple instace causing problem

由 可紊 提交于 2019-12-11 10:24:58
问题 I have an app where i have created Jet instance and pipeline job to aggregate result of an streaming data. I am running multiple instances of such app. The problem i am facing is since there are 2 instaces it is running 2 pipeline job and hence the result is computed twice and incorrect but it figures out that both jet instance are part of the same cluster. Does jet pipeline do not check the pipeline job and if same just consider it as one just like kafka stream does it with its topology? 回答1

hazelcast-jet deployment and data ingestion

由 限于喜欢 提交于 2019-12-08 02:03:59
问题 I have a distributed system running on AWS EC2 instances. My cluster has around 2000 nodes. I want to introduce a stream processing model which can process metadata being periodically published by each node (cpu usage, memory usage, IO and etc..). My system only cares about the latest data. It is also OK with missing a couple of data points when the processing model is down. Thus, I picked hazelcast-jet which is an in-memory processing model with great performance. Here I have a couple of

hazelcast-jet deployment and data ingestion

由 岁酱吖の 提交于 2019-12-06 11:41:28
I have a distributed system running on AWS EC2 instances. My cluster has around 2000 nodes. I want to introduce a stream processing model which can process metadata being periodically published by each node (cpu usage, memory usage, IO and etc..). My system only cares about the latest data. It is also OK with missing a couple of data points when the processing model is down. Thus, I picked hazelcast-jet which is an in-memory processing model with great performance. Here I have a couple of questions regarding the model: What is the best way to deploy hazelcast-jet to multiple ec2 instances? How

Hazelcast Jet Query

由 ∥☆過路亽．° 提交于 2019-12-02 15:01:24
问题 I have following query regarding Hazelcast Jet The use-case as follows There is one application (Application 'A', deployed in cluster) uses Hazelcast IMDG and puts millions of records / transactions in hazelcast IMap. The Event Journal has been configured for this IMap. There is another application (Application B, deployed in cluster) instantiates JetInstance and runs the job individually on each node to process the records. Currently, this job reads data from event journal and adds into
订阅 hazelcast-jet