spring-cloud-stream

spring.cloud.stream.kafka.binder.headers not working as expected

我怕爱的太早我们不能终老 提交于 2019-12-06 11:35:33
I am trying to use spring.cloud.stream.kafka.binder.headers to transport a custom header that I am setting based upon a previous question . I have read in the documentation where... spring.cloud.stream.kafka.binder.headers The list of custom headers that will be transported by the binder. Default: empty. seems to suggest that setting a list (comma separated?) will cause a custom header to get transported in the Message<> , but the header is lost as soon as the kafka write is completed. My annotation creates the header as a part of the call to the MessagingGateway: @MessagingGateway(name =

Manual Acknowledgement of Messages : Spring Cloud Stream Kafka

岁酱吖の 提交于 2019-12-06 11:34:55
问题 The scenario i want to implement is consume a message from Kafka , process it, if some condition fails i do not wish to acknowledge the message. For this i found in the spring cloud stream reference documentation, autoCommitOffset Whether to autocommit offsets when a message has been processed. If set to false, an Acknowledgment header will be available in the message headers for late acknowledgment. Default: true. My question is after setting autoCommitOffset to false, how can i acknowledge

Spring Cloud Stream App VS Spring Cloud Task App

被刻印的时光 ゝ 提交于 2019-12-06 10:49:06
I'm exploring the the SCDF for orchestrating the existing batch task for my system. I noticed that the SC Stream App Starters already provides some OOTB ready to use streaming applications, which suits my use case of: Extracting xml file from FTP -> the ftp source Customize transformation Upload the transformed xml file to another FTP -> the ftp sink But, my system only requires the above mentioned batch job to be executed at every 10 minutes interval. Implementing it using streams app is fine but it requires the stream app to be running 24-7 (using SCDF with Kubernetes). My concern is that I

spring-cloud-stream request-reply messaging pattern

99封情书 提交于 2019-12-06 04:36:20
问题 Is there a request-reply pattern that one should use with spring-cloud-stream? All the documentation I can find on spring-cloud-stream is geared toward the MessageChannel.send fire-and-forget type of producer, and I'm familiar with @MessagingGateway from spring-integration, but I'm not sure how that would work with spring-cloud-stream. That would be useful when you have a REST POST endpoint that saves an entity with an assigned identifier and you need to return the assigned identifier to the

jackson cannot deserialize (spring cloud stream kafka)

守給你的承諾、 提交于 2019-12-06 04:14:40
I am trying to read a json message from kafka and got an exception, which says Jackson cannot deserialize the json to POJO. The json is like {"code":"500","count":22,"from":1528343820000,"to":1528343880000} , which is an output of kafka stream. The POJO declares all attributes of the json, and is exactly the same POJO to produce the json message. So I have no idea why it would happen. I am using spring cloud stream 2.0.0.RELEASE. Any help would be appreciated. Thanks. POJO: public class CodeCount { private String code; private long count; private Date from; private Date to; @Override public

Spring Cloud Stream dynamic channels

99封情书 提交于 2019-12-06 02:35:34
问题 I am using Spring Cloud Stream and want to programmatically create and bind channels. My use case is that during application startup I receive the dynamic list of Kafka topics to subscribe to. How can I then create a channel for each topic? 回答1: I ran into similar scenario recently and below is my sample of creating SubscriberChannels dynamically. ConsumerProperties consumerProperties = new ConsumerProperties(); consumerProperties.setMaxAttempts(1); BindingProperties bindingProperties = new

Manual ack with Spring Cloud Stream RabbitMQ

夙愿已清 提交于 2019-12-06 00:24:48
I'm trying to setup a scenario where the listeners will manually send ACK to RabbitMQ. I have this in my application.properties spring.cloud.stream.rabbit.bindings.input.consumer.acknowledge-mode=MANUAL How do I send an ack from the listener? I see from Manual Acknowledgement of Messages : Spring Cloud Stream Kafka that for Kafka, we create an Acknowledgement object and call acknowledge() in it. Is there an example for RabbitMQ to do the same? See the Spring Integration Reference Manual . The message has two headers, AmqpHeaders.CHANNEL and AmqpHeaders.DELIVERY_TAG . You need to call basicAck

Kafka Source in Spring Cloud Data Flow

ε祈祈猫儿з 提交于 2019-12-05 02:40:27
问题 I am migrating from Spring XD to Spring Cloud Data Flow. When I am looking for module list I realised that some of the sources are not listed in Spring Cloud Flow - One of them is KAFKA source. My question is why KAFKA source is removed from standard sources list in spring cloud data flow ? 回答1: When I am looking for module list I realised that some of the sources are not listed in Spring Cloud Flow Majority of the applications are ported over and the remaining are incrementally prioritized -

How to unit test Spring Cloud Stream with Kafka DSL

可紊 提交于 2019-12-05 02:20:31
问题 I am trying to (unit) test a Spring Cloud Stream Kafka processor that uses Kafka DSL, but receives the following error " Connection to node -1 could not be established. Broker may not be available. ". Besides, the test does not shut down. I tried both EmbeddedKafka and TestBinder and yet I have the same behavior. I tried to start from the reponse given by Spring Cloud Team (which works) and I adapted the application to use Kafka DSL and left the test class pretty much as is. Does

@MessagingGateway, Spring Cloud Stream, and error handling across both

若如初见. 提交于 2019-12-04 20:14:44
Regarding the answer posted for How can @MessagingGateway be configured with Spring Cloud Stream MessageChannels? , what is the correct way to handle errors at the @MessagingGateway that can be returned from Spring Cloud Stream services? To recap, I have a @MessagingGateway that provides synchronous access to asynchronous services built using Spring Cloud Stream. When an error occurs within my Spring Cloud Stream service layer, I create an error response and send it through a SubscribableChannel to other @StreamListener services that process the errors. For example, when an account is created,