问题
I am trying to implement streaming input updates in Postgresql. Specifically , I would like to use Postgresql as datasource in stream input into spark.
Looking at the document, I was not sure if this is possible or not.
https://spark.apache.org/docs/latest/streaming-programming-guide.html
Would it be possible to stream input from PostgresQL, perhaps as a micro batch ?
回答1:
To stream your PSQL Data as a micro batch, Kafka is the best way. You can use Kafka connect (as a source) to established a connection between your Brokers and your Database.
See Kafka streaming integration with Spark Streaming: https://spark.apache.org/docs/2.2.0/streaming-kafka-0-10-integration.html
Kafka connect documentation: https://docs.confluent.io/current/connect/index.html
By this way, you can face to faults, concurrency and optimize your computation capabilities.
来源：https://stackoverflow.com/questions/58949914/how-to-use-external-database-postgresql-as-input-in-streaming-query