Using pyspark to connect to PostgreSQL

Question

I am trying to connect to a database with pyspark and I am using the following code:

sqlctx = SQLContext(sc)
df = sqlctx.load(
    url = "jdbc:postgresql         


        

Answer 1

You normally need either:

1. to install the Postgres Driver on your cluster,
2. to provide the Postgres driver jar from your client with the --jars option
3. or to provide the maven coordinates of the Postgres driver with --packages option.

If you detail how are you launching pyspark, we may give you more details.

Some clues/ideas:

spark-cannot-find-the-postgres-jdbc-driver

Not able to connect to postgres using jdbc in pyspark shell