Error reading Kafka SSL client truststore file from Spark streaming

末鹿安然 提交于 2021-02-10 18:29:23

问题


I have a Spark streaming application reading from Kafka. I am running it from EMR. Recently I implemented Kafka SSL. I am creating the Kafka client as shown below. I am getting a strange error running the application when it tries to read the truststore file. Error is:

- Caused by: org.apache.kafka.common.KafkaException: org.apache.kafka.common.KafkaException: java.io.FileNotFoundException: /tmp/kafka.client.truststore.jks (No such file or directory)

What is causing this issue?

DataStreamReader df = session.readStream()
    .format("kafka")
    .option("kafka.bootstrap.servers",kafka_server)
    .option("subscribe", kafka_topic)
    .option("failOnDataLoss", "false")
    .option("group.id", kafka_group)
    .option("kafka.security.protocol","SSL")
    .option("kafka.ssl.truststore.location","/tmp/kafka.client.truststore.jks")
    .option("kafka.ssl.truststore.password","clientpass");

回答1:


That file must exist on the executor nodes of the cluster, not just on the machine you submit the code from

Otherwise, you must pass the --files option to spark-submit to pass them from your machine to the driver

Also note: group.id isn't allowed in structured streaming



来源:https://stackoverflow.com/questions/59274128/error-reading-kafka-ssl-client-truststore-file-from-spark-streaming

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!