How to save/insert each DStream into a permanent table

Question

I\'ve been facing a problem with \"Spark Streaming\" about the insertion of output Dstream into a permanent SQL table. I\'d like to insert every output DStream (com

Answer 1

Instead of using external connectors better go for spark structured streaming .

Answer 2

Vanilla Spark does not provide a way to persist data unless you've downloaded the version packaged with HDFS (although they appear to be playing with the idea in Spark 2.0). One way to store the results to a permanent table and query those results later is to use one of the various databases in the Spark Database Ecosystem. There are pros and cons to each and your use case matters. I'll provide something close to a master list. These are segmented by:

Type of data managment, form data is stored in, connection to Spark

Database, SQL, Integrated

• SnappyData

Database, SQL, Connector

• MemSQL
• Hana
• Kudu
• FiloDB
• DB2
• SQLServer (JDBC)
• Oracle (JDBC)
• MySQL (JDBC)

Database, NoSQL, Connector

• Cassandra
• HBase
• Druid
• Ampool
• Riak
• Aerospike
• Cloudant

Database, Document, Connector

• MongoDB
• Couchbase

Database, Graph, Connector

• Neo4j
• OrientDB

Search, Document, Connector

• Elasticsearch
• Solr

Data grid, SQL, Connector

• Ignite

Data grid, NoSQL, Connector

• Infinispan
• Hazelcast
• Redis

File System, Files, Integrated

• HDFS

File System, Files, Connector

• S3
• Alluxio

Datawarehouse, SQL, Connector

• Redshift
• Snowflake
• BigQuery
• Aster