How to use “li-apache-kafka-clients” in spring boot app to send large message (above 1MB) from Kafka producer?

问题

How to use li-apache-kafka-clients in spring boot app to send large message (above 1MB) from Kafka producer to Kafka Consumer? Below is the GitHub link of li-apache-kafka-clients: https://github.com/linkedin/li-apache-kafka-clients

I have imported .jar file of li-apache-kafka-clients and put the below configuration for producer:

props.put("large.message.enabled", "true");
props.put("max.message.segment.bytes", 1000 * 1024);
props.put("segment.serializer", DefaultSegmentSerializer.class.getName());

and for consumer:

message.assembler.buffer.capacity, 
max.tracked.messages.per.partition, 
exception.on.message.dropped, 
segment.deserializer.class

but still getting error for large message. Please help me to solve error.

Below is my code, please let me know where I need to create LiKafkaProducer:

@Configuration
public class KafkaProducerConfig {

    @Value("${kafka.boot.server}")
    private String kafkaServer;


    @Bean
    public ProducerFactory<String, String> producerFactory() {
        return new DefaultKafkaProducerFactory<>(producerConfig());
    }

    @Bean
    public KafkaTemplate<String, String> kafkaTemplate() {
        return new KafkaTemplate<String, String>(producerFactory());
    }

    @Bean
    public Map<String, Object> producerConfig() {
        // TODO Auto-generated method stub
        Map<String, Object> config = new HashMap<>();
        config.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
        config.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
        config.put("bootstrap.servers", "localhost:9092");
        config.put("acks", "all");
        config.put("retries", 0);
        config.put("batch.size", 16384);
        config.put("linger.ms", 1);
        config.put("buffer.memory", 33554432);

        // The following properties are used by LiKafkaProducerImpl
        config.put("large.message.enabled", "true");
        config.put("max.message.segment.bytes", 1000 * 1024);
        config.put("segment.serializer", DefaultSegmentSerializer.class.getName());
        config.put("auditor.class", LoggingAuditor.class.getName());

        return config;
    }
}

@RestController
@RequestMapping("/kafkaProducer")
public class KafkaProducerController {

    @Autowired
    private KafkaSender sender;

    @PostMapping
    public ResponseEntity<List<Student>> sendData(@RequestBody List<Student> student){

        sender.sendData(student);
        return new ResponseEntity<List<Student>>(student, HttpStatus.OK);
    }
}

@Service
public class KafkaSender {

    private static final Logger LOGGER = LoggerFactory.getLogger(KafkaSender.class);

    @Autowired
    private KafkaTemplate<String, String> kafkaTemplate;

    @Value("${kafka.topic.name}")
    private String topicName;

    public void sendData(List<Student> student) {

        // TODO Auto-generated method stub
        Map<String, Object> headers = new HashMap<>();
        headers.put(KafkaHeaders.TOPIC, topicName);
        headers.put("payload", student.get(0));

        // Construct a JSONObject from a Map.
        JSONObject HeaderObject = new JSONObject(headers);
        System.out.println("\nMethod-2: Using new JSONObject() ==> " + HeaderObject);
        final String record = HeaderObject.toString();

        Message<String> message = MessageBuilder.withPayload(record).setHeader(KafkaHeaders.TOPIC, topicName)
                .setHeader(KafkaHeaders.MESSAGE_KEY, "Message")
                .build();

        kafkaTemplate.send(topicName, message.toString());
    }
}

回答1:

You would need to implement your own ConsumerFactory and ProducerFactory to create the LiKafkaConsumer and LiKafkaProducer respectively.

You should be able to subclass the default factories provided by the framework.

来源：https://stackoverflow.com/questions/57978818/how-to-use-li-apache-kafka-clients-in-spring-boot-app-to-send-large-message-a