topic

Killing node with __consumer_offsets leads to no message consumption at consumers

匿名 (未验证) 提交于 2019-12-03 01:34:02
可以将文章内容翻译成中文,广告屏蔽插件可能会导致该功能失效(如失效,请关闭广告屏蔽插件后再试): 问题: I have 3 node(nodes0,node1,node2) Kafka cluster(broker0, broker1, broker2) with replication factor 2 and Zookeeper(using zookeeper packaged with Kafka tar) running on a different node (node 4). I had started broker 0 after starting zookeper and then remaining nodes. It is seen in broker 0 logs that it is reading __consumer_offsets and seems they are stored on broker 0. Below are sample logs: Kafka Version: kafka_2.10-0.10.2.0 2017-06-30 10:50:47,381] INFO [GroupCoordinator 0]: Loading group metadata for console-consumer-85124 with generation

RabbitMQ: persistent message with Topic exchange

匿名 (未验证) 提交于 2019-12-03 01:29:01
可以将文章内容翻译成中文,广告屏蔽插件可能会导致该功能失效(如失效,请关闭广告屏蔽插件后再试): 由 翻译 强力驱动 问题: I am very new to RabbitMQ. I have set up a 'topic' exchange. The consumers may be started after the publisher. I'd like the consumers to be able to receive messages that have been sent before they were up, and that were not consumed yet. The exchange is set up with the following parameters: exchange_type => 'topic' durable => 1 auto_delete => 0 passive => 0 The messages are published with this parameter: delivery_mode => 2 Consumers use get() to retrieve the messages from the exchange. Unfortunately, any message published before

Verify that local mosquitto MQTT Broker is bridged to test.mosquitto.org

匿名 (未验证) 提交于 2019-12-03 01:26:01
可以将文章内容翻译成中文,广告屏蔽插件可能会导致该功能失效(如失效,请关闭广告屏蔽插件后再试): 问题: I have a local (OSX 11) mosquitto broker (1.4.7) running with the following config: connection test address test.mosquitto.org topic in_topic in 0 topic out_topic out 0 try_private false notifications false bridge_attempt_unsubscribe true I got the conf settings form here: bridge local mosquitto to cloud broker (thanks) I now have 3 terminals. A : subscribed to test.mosquitto.org directly using a ruby script and ruby-mqtt . B : subscribed to local broker with mosquitto_sub -t in_topic C : publishes using this command: mosquitto_pub -h test

Phoenix Channels - Multiple channels per socket

匿名 (未验证) 提交于 2019-12-03 01:26:01
可以将文章内容翻译成中文,广告屏蔽插件可能会导致该功能失效(如失效,请关闭广告屏蔽插件后再试): 问题: I'm writing an application using Elixir Channels to handle realtime events. I understand that there will be 1 socket open per client and can multiplex multiple channels over it. So my app is a chat application where users are part of multiple group chats. I have 1 Phoenix Channel called MessageChannel where the join method will handle dyanamic topics. def join("groups:" <> group_id, payload, socket) do .... Let's say John joins groups/topics A and B while Bob only join group/topic B. When john sends a message to group/topic A, broadcast!/3

Kafka: Get broker host from ZooKeeper

匿名 (未验证) 提交于 2019-12-03 01:20:02
可以将文章内容翻译成中文,广告屏蔽插件可能会导致该功能失效(如失效,请关闭广告屏蔽插件后再试): 由 翻译 强力驱动 问题: For particular reasons I need to use both - ConsumerGroup (a.k.a. high-level consumer) and SimpleConsumer (a.k.a. low-level consumer) to read from Kafka. For ConsumerGroup I use ZooKeeper-based config and am completely satisfied with it, but SimpleConsumer requires seed brokers to be instantiated. I don't want to keep list of both - ZooKeeper and broker hosts. Thus, I'm looking for a way to automatically discover brokers for a particular topic from ZooKeeper . Because of some indirect information I belief that these data is stored

android - Paho MQTT service for publishing

匿名 (未验证) 提交于 2019-12-03 01:17:01
可以将文章内容翻译成中文,广告屏蔽插件可能会导致该功能失效(如失效,请关闭广告屏蔽插件后再试): 问题: I am new to Android and services. My aim is to be able to set-up subscriptions and do publications on topic strings. The topic strings and client ID are set-up after parsing input of text fields. I am using the Paho MQTT service (downloaded the source and built the JAR). The following causes a Null Pointer Exception at c.publish() . The logcat shows the exception at the IMqttDeliveryToken publish(String topic, MqttMessage message, Object userContext, IMqttActionListener callback) method in MqttAndroidClient where a delivery token is being

org.apache.kafka.common.KafkaException: Failed to construct kafka consumer

匿名 (未验证) 提交于 2019-12-03 01:06:02
可以将文章内容翻译成中文,广告屏蔽插件可能会导致该功能失效(如失效,请关闭广告屏蔽插件后再试): 问题: I am manually starting Zookeeper, then Kafka server and finally the Kafka-Rest server with their respective properties file. Next, I am deploying my Spring Boot application on tomcat In the Tomcat log trace, I am getting the Error org.springframework.context.ApplicationContextException: Failed to start bean 'org.springframework.kafka.config.internalKafkaListenerEndpointRegistry'; nested exception is org.apache.kafka.common.KafkaException: Failed to construct kafka consumer and my application is failing to startup Error Log 25-Dec-2017 15:00

Python: how to mock a kafka topic for unit tests?

匿名 (未验证) 提交于 2019-12-03 00:53:01
可以将文章内容翻译成中文,广告屏蔽插件可能会导致该功能失效(如失效,请关闭广告屏蔽插件后再试): 问题: We have a message scheduler that generates a hash-key from the message attributes before placing it on a Kafka topic queue with the key. This is done for de-duplication purposes. However, I am not sure how I could possibly test this deduplication without actually setting up a local cluster and checking that it is performing as expected. Searching online for tools for mocking a Kafka topic queue has not helped, and I am concerned that I am perhaps thinking about this the wrong way. Ultimately, whatever is used to mock the Kafka queue, should

Kafka安装配置

匿名 (未验证) 提交于 2019-12-03 00:39:02
环境 hadoop11 hadoop12 hadoop13 zookeeper 集群 hadoop11:2181 ; hadoop12:2181 ; hadoop13:2181 kafka安装包 2.11 1.0安装包上传到虚拟机 /export/service 2.0更改配置文件 2.1 vi server.properties broker.id=0 每个机器的id不同 log.dirs=/export/servers/kafka_2.11-1.0.0/logs 文件路径自己创建 zookeeper.connect=hadoop11:2181,hadoop12:2181,hadoop13:2181 zookeeper集群 delete.topic.enable=true host.name=hadoop13 3.0kafka启动 先启动zookeeper集群 启动kafka 三台机器启动 nohup bin/kafka-server-start.sh config/server.properties > /dev/null 2>&1 & hadoop11创建topic ./kafka-topics.sh --create --partitions 3 --replication-factor 2 --topic test --zookeeper hadoop11:2181

MQTT入门

匿名 (未验证) 提交于 2019-12-03 00:39:02
什么是MQTT     MQTT(Message Queuing Telemetry Transport,消息队列遥测传输)是IBM开发的一个即时通讯协议,有可能成为物联网的重要组成部分。该协议支持所有平台,几乎可以把所有联网物品和外部连接起来,是一种基于发布/订阅(publish/subscribe)模式的“轻量级”通讯协议,该协议构建于TCP/IP协议上,被用来当做传感器和制动器(比如通过Twitter让房屋联网)的通信协议。 MQTT特点     MQTT协议是为大量计算能力有限,且工作在低带宽、不可靠的网络的远程传感器和控制设备通讯而设计的协议,它具有以下主要的几项特性:     1、使用发布/订阅消息模式,提供一对多的消息发布,解除应用程序耦合;     2、对负载内容屏蔽的消息传输;     3、使用 TCP/IP 提供网络连接;     4、有三种消息发布服务质量: “至多一次”,消息发布完全依赖底层 TCP/IP 网络。会发生消息丢失或重复。这一级别可用于如下情况,环境传感器数据,丢失一次读记录无所谓,因为不久后还会有第二次发送。 “至少一次”,确保消息到达,但消息重复可能会发生。 “只有一次”,确保消息到达一次。这一级别可用于如下情况,在计费系统中,消息重复或丢失会导致不正确的结果。     5、小型传输,开销很小(固定长度的头部是 2 字节),协议交换最小化