1. 背景
写的Spark Structured SQL 任务，然后用的spark版本为
<spark.version>2.3.0.cloudera2</spark.version>
<!--spark-->
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_${scala.compat.version}</artifactId>
<version>${spark.version}</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql-kafka-0-10_${scala.compat.version}</artifactId>
<version>${spark.version}</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-hive_${scala.compat.version}</artifactId>
<version>${spark.version}</version>
</dependency>
实际用户的环境为
CDH ： cdh5.13.3.p0.316101
SPARK : SPARK2-2.3.0.cloudera2-1.cdh5.13.3.p0.316101
spark版本引入的稍微有些问题。
使用报错
19/08/26 19:24:20 INFO storage.BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
19/08/26 19:24:20 INFO storage.BlockManager: external shuffle service port = 7337
19/08/26 19:24:20 INFO storage.BlockManager: Initialized BlockManager: BlockManagerId(driver, shuqi2.test.dtwave, 33721, None)
19/08/26 19:24:21 INFO scheduler.EventLoggingListener: Logging events to hdfs://cdh-master:8020/user/spark/spark2ApplicationHistory/application_1566807509826_0024
19/08/26 19:24:21 INFO spark.SparkContext: Registered listener com.cloudera.spark.lineage.NavigatorAppListener
19/08/26 19:24:21 INFO internal.SharedState: loading hive config file: file:/etc/spark2/conf.cloudera.spark2_on_yarn/yarn-conf/hive-site.xml
19/08/26 19:24:21 INFO internal.SharedState: spark.sql.warehouse.dir is not set, but hive.metastore.warehouse.dir is set. Setting spark.sql.warehouse.dir to the value of hive.metastore.warehouse.dir ('/user/hive/warehouse').
19/08/26 19:24:21 INFO internal.SharedState: Warehouse path is '/user/hive/warehouse'.
19/08/26 19:24:21 INFO state.StateStoreCoordinatorRef: Registered StateStoreCoordinator endpoint
19/08/26 19:24:21 INFO TopoSparkSubmitter:
系统参数如下:
spark.lineage.log.dir -> /var/log/spark2/lineage
spark.serializer -> org.apache.spark.serializer.KryoSerializer
spark.sql.warehouse.dir -> /user/hive/warehouse
spark.yarn.jars -> local:/opt/cloudera/parcels/SPARK2-2.3.0.cloudera2-1.cdh5.13.3.p0.316101/lib/spark2/jars/*
spark.driver.host -> shuqi2.test.dtwave
spark.eventLog.enabled -> true
spark.driver.port -> 38075
spark.shuffle.service.enabled -> true
spark.extraListeners -> com.cloudera.spark.lineage.NavigatorAppListener
spark.driver.extraLibraryPath -> /opt/cloudera/parcels/CDH-5.15.2-1.cdh5.15.2.p0.3/lib/hadoop/lib/native
spark.lineage.enabled -> true
spark.yarn.queue -> root.dev
spark.jars -> file:/opt/workspace/cheetah-node/libs/spark-structured-streaming.jar
spark.yarn.historyServer.address -> http://cdh-master:18089
spark.ui.enabled -> true
spark.app.name -> demo_kboxing_structuredsql_61551209005188302019082619240991521
spark.ui.killEnabled -> true
spark.sql.hive.metastore.jars -> ${env:HADOOP_COMMON_HOME}/../hive/lib/*:${env:HADOOP_COMMON_HOME}/client/*
spark.dynamicAllocation.schedulerBacklogTimeout -> 1
spark.yarn.am.extraLibraryPath -> /opt/cloudera/parcels/CDH-5.15.2-1.cdh5.15.2.p0.3/lib/hadoop/lib/native
spark.yarn.config.gatewayPath -> /opt/cloudera/parcels
spark.sql.queryExecutionListeners -> com.cloudera.spark.lineage.NavigatorQueryListener
spark.executor.id -> driver
spark.yarn.config.replacementPath -> {{HADOOP_COMMON_HOME}}/../../..
spark.submit.deployMode -> client
spark.shuffle.service.port -> 7337
spark.master -> yarn
spark.authenticate -> false
spark.ui.filters -> org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
spark.network.crypto.enabled -> false
spark.executor.extraLibraryPath -> /opt/cloudera/parcels/CDH-5.15.2-1.cdh5.15.2.p0.3/lib/hadoop/lib/native
spark.io.encryption.enabled -> false
spark.eventLog.dir -> hdfs://cdh-master:8020/user/spark/spark2ApplicationHistory
spark.dynamicAllocation.enabled -> true
spark.sql.catalogImplementation -> hive
spark.driver.appUIAddress -> http://shuqi2.test.dtwave:4040
spark.org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter.param.PROXY_HOSTS -> cdh-master
spark.dynamicAllocation.minExecutors -> 0
spark.org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter.param.PROXY_URI_BASES -> http://cdh-master:8088/proxy/application_1566807509826_0024
spark.dynamicAllocation.executorIdleTimeout -> 60
spark.app.id -> application_1566807509826_0024
spark.sql.hive.metastore.version -> 1.1.0
19/08/26 19:24:21 INFO kafka.KafkaStructureSource: schema内容为：{"type":"record","name":"entity_positions_qudao","namespace":"CHEETAH_SCHEMA","doc":"record","fields":[{"name":"table","type":"string"},{"name":"timestamp","type":"string"},{"name":"transactionID","type":"string"},{"name":"entryType","type":"string"},{"name":"transactionUser","type":"string"},{"name":"beforeImage","type":{"type":"record","name":"beforeImage","namespace":"beforeImage","doc":"record","fields":[{"name":"sysdt","type":"string"},{"name":"systm","type":"string"},{"name":"channel_seq","type":"string"},{"name":"money","type":"string"}]}},{"name":"afterImage","type":{"type":"record","name":"afterImage","namespace":"afterImage","doc":"record","fields":[{"name":"sysdt","type":"string"},{"name":"systm","type":"string"},{"name":"channel_seq","type":"string"},{"name":"money","type":"string"}]}}]}
root
|-- table: string (nullable = false)
|-- timestamp: string (nullable = false)
|-- transactionID: string (nullable = false)
|-- entryType: string (nullable = false)
|-- transactionUser: string (nullable = false)
|-- beforeImage: struct (nullable = false)
| |-- sysdt: string (nullable = false)
| |-- systm: string (nullable = false)
| |-- channel_seq: string (nullable = false)
| |-- money: string (nullable = false)
|-- afterImage: struct (nullable = false)
| |-- sysdt: string (nullable = false)
| |-- systm: string (nullable = false)
| |-- channel_seq: string (nullable = false)
| |-- money: string (nullable = false)
19/08/26 19:24:22 INFO consumer.ConsumerConfig: ConsumerConfig values:
metric.reporters = []
metadata.max.age.ms = 300000
value.deserializer = class org.apache.kafka.common.serialization.ByteArrayDeserializer
group.id = spark-kafka-source-710a601f-b97d-4a7b-95b1-25fa9e1f9613--2131045480-driver-0
partition.assignment.strategy = [org.apache.kafka.clients.consumer.RangeAssignor]
reconnect.backoff.ms = 50
sasl.kerberos.ticket.renew.window.factor = 0.8
max.partition.fetch.bytes = 1048576
bootstrap.servers = [cdh-master:9092, cdh-node1:9092, cdh-node2:9092]
retry.backoff.ms = 100
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
ssl.keystore.type = JKS
ssl.trustmanager.algorithm = PKIX
enable.auto.commit = false
ssl.key.password = null
fetch.max.wait.ms = 500
sasl.kerberos.min.time.before.relogin = 60000
connections.max.idle.ms = 540000
ssl.truststore.password = null
session.timeout.ms = 30000
metrics.num.samples = 2
client.id =
ssl.endpoint.identification.algorithm = null
key.deserializer = class org.apache.kafka.common.serialization.ByteArrayDeserializer
ssl.protocol = TLS
check.crcs = true
request.timeout.ms = 40000
ssl.provider = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
ssl.keystore.location = null
heartbeat.interval.ms = 3000
auto.commit.interval.ms = 5000
receive.buffer.bytes = 65536
ssl.cipher.suites = null
ssl.truststore.type = JKS
security.protocol = PLAINTEXT
ssl.truststore.location = null
ssl.keystore.password = null
ssl.keymanager.algorithm = SunX509
metrics.sample.window.ms = 30000
fetch.min.bytes = 1
send.buffer.bytes = 131072
auto.offset.reset = earliest
19/08/26 19:24:22 WARN consumer.ConsumerConfig: The configuration max.poll.records = 1 was supplied but isn't a known config.
at org.apache.spark.sql.kafka010.SubscribeStrategy.createConsumer(ConsumerStrategy.scala:63)
at org.apache.spark.sql.kafka010.KafkaOffsetReader.createConsumer(KafkaOffsetReader.scala:314)
at org.apache.spark.sql.kafka010.KafkaOffsetReader.<init>(KafkaOffsetReader.scala:78)
at org.apache.spark.sql.kafka010.KafkaSourceProvider.createContinuousReader(KafkaSourceProvider.scala:130)
at org.apache.spark.sql.kafka010.KafkaSourceProvider.createContinuousReader(KafkaSourceProvider.scala:43)
at org.apache.spark.sql.streaming.DataStreamReader.load(DataStreamReader.scala:185)
at com.dtwave.cheetah.node.spark.structured.streaming.source.kafka.KafkaStructureSource.createKafkaSourceStream(KafkaStructureSource.scala:85)
at com.dtwave.cheetah.node.spark.structured.streaming.runtime.SparkStreamHandler.handleSource(SparkStreamHandler.scala:51)
at com.dtwave.cheetah.node.spark.structured.streaming.runtime.TopoSparkSubmitter$.submit(TopoSparkSubmitter.scala:115)
at com.dtwave.cheetah.node.spark.structured.streaming.StructureStreamingExecutor$.main(StructureStreamingExecutor.scala:48)
at com.dtwave.cheetah.node.spark.structured.streaming.StructureStreamingExecutor.main(StructureStreamingExecutor.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:892)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:197)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:227)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:136)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
19/08/26 19:24:22 INFO spark.SparkContext: Invoking stop() from shutdown hook
19/08/26 19:24:22 INFO server.AbstractConnector: Stopped Spark@456abb66{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}
19/08/26 19:24:22 INFO ui.SparkUI: Stopped Spark web UI at http://shuqi2.test.dtwave:4040
19/08/26 19:24:22 INFO spark.MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
19/08/26 19:24:22 INFO storage.BlockManager: BlockManager stopped
19/08/26 19:24:22 INFO scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
19/08/26 19:24:22 INFO spark.SparkContext: Successfully stopped SparkContext
END-EOF-END-EOF
来源：https://blog.csdn.net/qq_21383435/article/details/100084581