spark-structured-streaming

问题 I want to do Spark Structured Streaming (Spark 2.4.x) from a Kafka source to a MariaDB with Python (PySpark). I want to use the streamed Spark dataframe and not the static nor Pandas dataframe. It seems that one has to use foreach or foreachBatch since there are no possible database sinks for streamed dataframes according to https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#output-sinks. Here is my try: from pyspark.sql import SparkSession import pyspark.sql

问题 I have a Static DataFrame with millions of rows as follows. Static DataFrame : -------------- id|time_stamp| -------------- |1|1540527851| |2|1540525602| |3|1530529187| |4|1520529185| |5|1510529182| |6|1578945709| -------------- Now in every batch, a Streaming DataFrame is being formed which contains id and updated time_stamp after some operations like below. In first Batch : -------------- id|time_stamp| -------------- |1|1540527888| |2|1540525999| |3|1530529784| -------------- Now in every

问题 I have a Static DataFrame with millions of rows as follows. Static DataFrame : -------------- id|time_stamp| -------------- |1|1540527851| |2|1540525602| |3|1530529187| |4|1520529185| |5|1510529182| |6|1578945709| -------------- Now in every batch, a Streaming DataFrame is being formed which contains id and updated time_stamp after some operations like below. In first Batch : -------------- id|time_stamp| -------------- |1|1540527888| |2|1540525999| |3|1530529784| -------------- Now in every

问题 I was going through the Spark structured streaming - Kafka integration guide here. It is told at this link that enable.auto.commit: Kafka source doesn’t commit any offset. So how do I manually commit offsets once my spark application has successfully processed each record? 回答1: Current Situation (Spark 2.4.5) This feature seems to be under discussion in the Spark community https://github.com/apache/spark/pull/24613. In that Pull Request you will also find a possible solution for this at https

问题 In my scenario I have several dataSet that comes every now and then that i need to ingest in our platform. The ingestion processes involves several transformation steps. One of them being Spark. In particular I use spark structured streaming so far. The infrastructure also involve kafka from which spark structured streaming reads data. I wonder if there is a way to detect when there is nothing else to consume from a topic for a while to decide to stop the job. That is i want to run it for the

问题 I am building a Spark structured streaming job that does the below, Streaming source, val small_df = spark.readStream .format("kafka") .load() small_df.createOrReplaceTempView("small_df") A dataframe - Phoenix load val phoenixDF = spark.read.format("org.apache.phoenix.spark") .option("table", "my_table") .option("zkUrl", "zk") .load() phoenixDF.createOrReplaceTempView("phoenix_tbl") Then, spark sql statement to join(on primary_key) with another small dataframe to filter records. val

问题 I am building a Spark structured streaming job that does the below, Streaming source, val small_df = spark.readStream .format("kafka") .load() small_df.createOrReplaceTempView("small_df") A dataframe - Phoenix load val phoenixDF = spark.read.format("org.apache.phoenix.spark") .option("table", "my_table") .option("zkUrl", "zk") .load() phoenixDF.createOrReplaceTempView("phoenix_tbl") Then, spark sql statement to join(on primary_key) with another small dataframe to filter records. val

问题 I am trying to run Python Spark Structured Streaming + Kafka, when I run the command Master@MacBook-Pro spark-3.0.0-preview2-bin-hadoop2.7 % bin/spark-submit --packages org.apache.spark:spark-sql-kafka-0-10_2.12:2.4.5 \ examples/src/main/python/sql/streaming/structured_kafka_wordcount.py \ /Users/Master/Projects/bank_kafka_spark/spark_job1.py localhost:9092 transaction receiving next 20/04/22 13:06:04 WARN Utils: Your hostname, MacBook-Pro.local resolves to a loopback address: 127.0.0.1;

问题 Why does the following structured query run multiple SQL queries as can be seen in web UI's SQL tab? import org.apache.spark.sql.streaming.{OutputMode, Trigger} import scala.concurrent.duration._ val rates = spark. readStream. format("rate"). option("numPartitions", 1). load. writeStream. format("console"). option("truncate", false). option("numRows", 10). trigger(Trigger.ProcessingTime(10.seconds)). queryName("rate-console"). start 来源： https://stackoverflow.com/questions/46162143/why-does-a

问题 Why does the following structured query run multiple SQL queries as can be seen in web UI's SQL tab? import org.apache.spark.sql.streaming.{OutputMode, Trigger} import scala.concurrent.duration._ val rates = spark. readStream. format("rate"). option("numPartitions", 1). load. writeStream. format("console"). option("truncate", false). option("numRows", 10). trigger(Trigger.ProcessingTime(10.seconds)). queryName("rate-console"). start 来源： https://stackoverflow.com/questions/46162143/why-does-a